Avoid subprocess memory copy when c library supports posix_spawn (#87958)

* use posix spawn on alpine

* Avoid subprocess memory copy when c library supports posix_spawn

By default python 3.10 will use the fork() which has to
copy all the memory of the parent process (in our case
this can be huge since Home Assistant core can use
hundreds of megabytes of RAM). By using posix_spawn
this is avoided.

In python 3.11 vfork will also be available
https://github.com/python/cpython/issues/80004#issuecomment-1093810689
https://github.com/python/cpython/pull/11671 but we won't
always be able to use it and posix_spawn is considered safer
https://bugzilla.kernel.org/show_bug.cgi?id=215813#c14

The subprocess library doesn't know about musl though
even though it supports posix_spawn https://git.musl-libc.org/cgit/musl/log/src/process/posix_spawn.c
so we have to teach it since it only has checks for glibc
1b736838e6/Lib/subprocess.py (L745)

The constant is documented as being able to be flipped here:
https://docs.python.org/3/library/subprocess.html#disabling-use-of-vfork-or-posix-spawn

* Avoid subprocess memory copy when c library supports posix_spawn

By default python 3.10 will use the fork() which has to
copy memory of the parent process (in our case
this can be huge since Home Assistant core can use
hundreds of megabytes of RAM). By using posix_spawn
this is avoided and subprocess creation does not
get discernibly slow the larger the Home Assistant
python process grows.

In python 3.11 vfork will also be available
https://github.com/python/cpython/issues/80004#issuecomment-1093810689
https://github.com/python/cpython/pull/11671 but we won't
always be able to use it and posix_spawn is considered safer
https://bugzilla.kernel.org/show_bug.cgi?id=215813#c14

The subprocess library doesn't know about musl though
even though it supports posix_spawn https://git.musl-libc.org/cgit/musl/log/src/process/posix_spawn.c
so we have to teach it since it only has checks for glibc
1b736838e6/Lib/subprocess.py (L745)

The constant is documented as being able to be flipped here:
https://docs.python.org/3/library/subprocess.html#disabling-use-of-vfork-or-posix-spawn

* missed some

* adjust more tests

* coverage
This commit is contained in:
J. Nick Koston 2023-02-13 08:02:51 -06:00 committed by GitHub
parent cf2fcdfba1
commit 03eea7bd3f
No known key found for this signature in database
GPG key ID: 4AEE18F83AFDEB23
14 changed files with 84 additions and 9 deletions

View file

@ -137,3 +137,22 @@ async def test_unhandled_exception_traceback(hass, caplog):
assert "Task exception was never retrieved" in caplog.text
assert "This is unhandled" in caplog.text
assert "_unhandled_exception" in caplog.text
def test__enable_posix_spawn():
"""Test that we can enable posix_spawn on Alpine."""
def _mock_alpine_exists(path):
return path == "/etc/alpine-release"
with patch.object(runner.subprocess, "_USE_POSIX_SPAWN", False), patch.object(
runner.os.path, "exists", _mock_alpine_exists
):
runner._enable_posix_spawn()
assert runner.subprocess._USE_POSIX_SPAWN is True
with patch.object(runner.subprocess, "_USE_POSIX_SPAWN", False), patch.object(
runner.os.path, "exists", return_value=False
):
runner._enable_posix_spawn()
assert runner.subprocess._USE_POSIX_SPAWN is False