-
Notifications
You must be signed in to change notification settings - Fork 1.9k
fix: Retry KVM_CREATE_VM on EINTR #5046
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Merged
zulinx86
merged 2 commits into
firecracker-microvm:main
from
zulinx86:EINTR_on_KVM_CREATE_VM
Feb 24, 2025
Merged
fix: Retry KVM_CREATE_VM on EINTR #5046
zulinx86
merged 2 commits into
firecracker-microvm:main
from
zulinx86:EINTR_on_KVM_CREATE_VM
Feb 24, 2025
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
The error variant is used when creating a new VM from KVM FD via KVM_CREATE_VM call. On success, VM FD is returned. The previous error message doesn't tell a lie but is a bit unintuitive. Signed-off-by: Takahiro Itazuri <[email protected]>
Question:
|
f292393
to
dd04187
Compare
roypat
reviewed
Feb 21, 2025
Codecov ReportAttention: Patch coverage is
Additional details and impacted files@@ Coverage Diff @@
## main #5046 +/- ##
==========================================
- Coverage 83.20% 83.19% -0.02%
==========================================
Files 247 247
Lines 26631 26641 +10
==========================================
+ Hits 22158 22163 +5
- Misses 4473 4478 +5
Flags with carried forward coverage won't be shown. Click here to find out more. ☔ View full report in Codecov by Sentry. |
dd04187
to
52127c1
Compare
Manciukic
reviewed
Feb 21, 2025
fb23c4f
to
be26374
Compare
ShadowCurse
previously approved these changes
Feb 21, 2025
roypat
previously approved these changes
Feb 24, 2025
It is known that KVM_CREATE_VM fails with EINTR on heavily loaded machines with many VMs. It might be a kernel bug but apparently has not been fixed. To mitigate it, QEMU does an infinitely retry on EINTR. Similar, do retries up to 5 times. Signed-off-by: Takahiro Itazuri <[email protected]>
be26374
to
5e0efad
Compare
roypat
approved these changes
Feb 24, 2025
ShadowCurse
approved these changes
Feb 24, 2025
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Reason & Changes
It is known that KVM_CREATE_VM occasionally fails with EINTR on heavily loaded machines
with many VMs.
The behavior itself that KVM_CREATE_VM can return EINTR is intentional. This is because
the KVM_CREATE_VM path includes mm_take_all_locks() that is CPU intensive and all CPU
intensive syscalls should check for pending signals and return EINTR immediately to allow
userland to remain interactive.
https://lists.nongnu.org/archive/html/qemu-devel/2014-01/msg01740.html
However, it is empirically confirmed that, even though there is no pending signal,
KVM_CREATE_VM returns EINTR.
https://lore.kernel.org/qemu-devel/[email protected]/
To mitigate it, QEMU does an inifinite retry on EINTR that greatly improves reliabiliy:
Similarly, we do retries up to 5 times. Although Firecracker clients are also able to
retry, they have to start Firecracker from scratch. Doing retries in Firecracker makes
recovery faster and improves reliability.
License Acceptance
By submitting this pull request, I confirm that my contribution is made under
the terms of the Apache 2.0 license. For more information on following Developer
Certificate of Origin and signing off your commits, please check
CONTRIBUTING.md
.PR Checklist
tools/devtool checkstyle
to verify that the PR passes theautomated style checks.
how they are solving the problem in a clear and encompassing way.
[ ] I have updated any relevant documentation (both in code and in the docs)in the PR.
[ ] I have mentioned all user-facing changes inCHANGELOG.md
.[ ] If a specific issue led to this PR, this PR closes the issue.[ ] When making API changes, I have followed theRunbook for Firecracker API changes.
integration tests.
TODO
.rust-vmm
.