Pick better default for mp_context

#4333 changes how mp_context is chosen from:

spikeinterface/src/spikeinterface/core/job_tools.py

Lines 419 to 427 in d8f49ec

    
           if pool_engine == "process": 
        
               if mp_context is None: 
        
                   mp_context = recording.get_preferred_mp_context() 
        
               if mp_context is not None and platform.system() == "Windows": 
        
                   assert mp_context != "fork", "'fork' mp_context not supported on Windows!" 
        
               elif mp_context == "fork" and platform.system() == "Darwin": 
        
                   warnings.warn('As of Python 3.8 "fork" is no longer considered safe on macOS') 
        
           self.mp_context = mp_context

to

spikeinterface/src/spikeinterface/core/job_tools.py

Lines 418 to 440 in 81e89fd

    
           if pool_engine == "process": 
        
               if mp_context is None: 
        
                   # auto choice 
        
                   if platform.system() == "Windows": 
        
                       mp_context = "spawn" 
        
                   elif platform.system() == "Linux": 
        
                       mp_context = "fork" 
        
                   elif platform.system() == "Darwin": 
        
                       # We used to force spawn for macos, this is sad but in some cases fork in macos 
        
                       # is very unstable and lead to crashes. 
        
                       mp_context = "spawn" 
        
                   else: 
        
                       mp_context = "spawn" 
        
               preferred_mp_context = recording.get_preferred_mp_context() 
        
               if preferred_mp_context is not None and preferred_mp_context != mp_context: 
        
                   warnings.warn( 
        
                       f"Your processing chain using pool_engine='process' and mp_context='{mp_context}' is not possible." 
        
                       f"So use mp_context='{preferred_mp_context}' instead" 
        
                   ) 
        
                   mp_context = preferred_mp_context 
        
           self.mp_context = mp_context

Before the PR; job_kwargs.mp_context=None and recording.get_preferred_mp_context()=None (the default and most common setup) will call mp_context = multiprocessing.get_context(None) that returned the default context for that specific machine (spawn for windows & Mac and fork/forkserver for Linux). Now, the behavior has changed for Linux: it always picks fork. Default mp_context for Linux from python 3.14 onwards is forkserver (a thread-safe, less problematic alternative to fork) docs.

When running kilosort4 in python 3.14. fork did not play nice with some multi-threaded calls inside scikit learn, particularly the process hangs forever in this line:
https://github.com/MouseLand/Kilosort/blob/7a19a57bef39f3e07ab7fa1edf5aa8635c69a850/kilosort/spikedetect.py#L72
This was run inside a kubernetes pod with GPUs but I couldn't replicate it locally: 3.13 and 3.14 both worked with fork or forkserver :/ so not sure if it will be a problem for other people.

In any case, it might be worth considering making forkserver (rather than fork) the Linux default (it's been around since python 3.4) or allow multiprocessing to pick the best default based on system and python version as it did before this PR, i.e., let mp_context=None with a comment saying what that means).

For our case on linux, I still think that fork is a good choice.
in linux fork must be avoided when thread are started before the fork, this is never the case for use.
The fork onlinux is instantaneous and do not need to send varaible wich is super important for many case in components.

Did you made tests including the spawn time ?

Sam only believes in fork. Haha.

@ecobost

I raised this back in 2023: the default should not be something that could be unsafe, and people who want the performance of fork should opt-in rather than everyone else having to opt-out. Python 3.14 has now made the switch to forkserver on Linux, and once we add 3.14 support (#4173) we'll be actively overriding the language default. Before #4333 we could at least claim innocence via the act-omission distinction, but now the override is an explicit choice and the blame is on us.

We're already seeing concrete consequences. #4469 is a false-positive warning caused by forkserver workers not inheriting global state. And as @ecobost reported above, fork caused a hang with Kilosort4 + scikit-learn in a Kubernetes environment (though it wasn't reproducible locally, which is exactly the kind of intermittent, environment-dependent failure that makes fork dangerous).

The CPython maintainers were clear about why they made this change:

The runtime itself is unsafe after fork(), regardless of whether your own code starts threads. C extensions (OpenBLAS, MKL) can spawn thread pools silently, so there is no way to guarantee a clean fork.
PyPy hit deadlocks where "literally compiling a single file with parallel compileall could hang."
PyTorch silently deadlocks using method=fork.
Free-threading (no GIL) makes it worse: fork() can leave Python data structures in undefined states.

The performance arguments are real. fork avoids serialization and gives instant copy-on-write, which matters for our large-array pipelines. But that's exactly why it should be an opt-in for power users who understand the trade-off, not the default for everyone.

I think we should:

Stop overriding Python's default. Let multiprocessing.get_context(None) pick forkserver on Python 3.14+ and fork on older versions, as it did before Improve multithreading and multi processing : lock + nogil + mp_context #4333.
Document mp_context="fork" as the performance option for users who want it.

If we can't get behind that, at minimum we should add a CI job that runs tests with forkserver so we catch issues like #4469 before release.

We should not prioritize power users over the median user. The median user expects Python defaults to work without any side effects.

cc @alejoe91 @chrishalcrow @JoeZiminski

I agree with the idea that the default should be safe and mirror the Python default, and users should opt-in to better performance. In some sense we already do this by having n_jobs=1 as the default: users currently have to opt-in to multiprocessing.

@h-mayorquin

I think we should:

Stop overriding Python's default. Let multiprocessing.get_context(None) pick forkserver on Python 3.14+ and fork on older versions, as it did before Improve multithreading and multi processing : lock + nogil + mp_context #4333.

Document mp_context="fork" as the performance option for users who want it.

If we can't get behind that, at minimum we should add a CI job that runs tests with forkserver so we catch issues like #4469 before release.

We should not prioritize power users over the median user. The median user expects Python defaults to work without any side effects.

Thanks @h-mayorquin for the in-depth recap! :) I agree with the plan. Let's use python defaults and document forks usage for power-users.

Also, do you guys know what is the performance gain of fork vs forkserver? I'll do some benchmarks on my end

I am more or less agree but we need to re-think the serialization to children process because sometime it is very very big (long spiketrains, template or drifting templates, ...)

Seems like people are in agreement. Are we good with opening a PR to this end (or has a PR already been done?)

Navigation Menu

Repository navigation

Pick better default for mp_context #4422

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

	if pool_engine == "process":
	if mp_context is None:
	mp_context = recording.get_preferred_mp_context()
	if mp_context is not None and platform.system() == "Windows":
	assert mp_context != "fork", "'fork' mp_context not supported on Windows!"
	elif mp_context == "fork" and platform.system() == "Darwin":
	warnings.warn('As of Python 3.8 "fork" is no longer considered safe on macOS')

	self.mp_context = mp_context

	if pool_engine == "process":
	if mp_context is None:
	# auto choice
	if platform.system() == "Windows":
	mp_context = "spawn"
	elif platform.system() == "Linux":
	mp_context = "fork"
	elif platform.system() == "Darwin":
	# We used to force spawn for macos, this is sad but in some cases fork in macos
	# is very unstable and lead to crashes.
	mp_context = "spawn"
	else:
	mp_context = "spawn"

	preferred_mp_context = recording.get_preferred_mp_context()
	if preferred_mp_context is not None and preferred_mp_context != mp_context:
	warnings.warn(
	f"Your processing chain using pool_engine='process' and mp_context='{mp_context}' is not possible."
	f"So use mp_context='{preferred_mp_context}' instead"
	)
	mp_context = preferred_mp_context

	self.mp_context = mp_context

Pick better default for mp_context #4422

Description

Activity

samuelgarcia commented on Mar 11, 2026

zm711 commented on Mar 31, 2026

h-mayorquin commented on Apr 9, 2026

chrishalcrow commented on Apr 13, 2026

alejoe91 commented on Apr 13, 2026

samuelgarcia commented on Apr 15, 2026

zm711 commented on Jul 7, 2026

Metadata

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Issue actions