mpi4py.util.pool

Added in version 4.0.0.

See also

This module intends to be a drop-in replacement for the multiprocessing.pool interface from the Python standard library. The Pool class exposed here is implemented as a thin wrapper around MPIPoolExecutor.

Note

The mpi4py.futures package offers a higher level interface for asynchronously pushing tasks to MPI worker process, allowing for a clear separation between submitting tasks and waiting for the results.

class mpi4py.util.pool.Pool

Pool using MPI processes as workers.

__init__(processes=None, initializer=None, initargs=(), **kwargs)

Initialize a new Pool instance.

Parameters:

• processes (int | None) – Number of worker processes.

• initializer (Callable[[...], None] | None) – An callable used to initialize workers processes.

• initargs (Iterable[Any]) – A tuple of arguments to pass to the initializer.

• kwargs (Any)

Return type:

None

Note

Additional keyword arguments are passed down to the MPIPoolExecutor constructor.

Warning

The maxtasksperchild and context arguments of multiprocessing.pool.Pool are not supported. Specifying maxtasksperchild or context with a value other than None will issue a warning of category UserWarning.

apply(func, args=(), kwds={})

Call func with arguments args and keyword arguments kwds.

Equivalent to func(*args, **kwds).

Parameters:

• func (Callable[[...], T])

• args (Iterable[Any])

• kwds (Mapping[str, Any])

Return type:

T

apply_async(func, args=(), kwds={}, callback=None, error_callback=None)

Asynchronous version of apply() returning ApplyResult.

Parameters:

• func (Callable[..., T])

• args (Iterable[Any])

• kwds (Mapping[str, Any])

• callback (Callable[[T], object] | None)

• error_callback (Callable[[BaseException], object] | None)

Return type:

AsyncResult[T]

map(func, iterable, chunksize=None)

Apply func to each element in iterable.

Equivalent to list(map(func, iterable)).

Block until all results are ready and return them in a list.

The iterable is choped into a number of chunks which are submitted as separate tasks. The (approximate) size of these chunks can be specified by setting chunksize to a positive integer.

Consider using imap() or imap_unordered() with explicit chunksize for better efficiency.

Parameters:

• func (Callable[[S], T])

• iterable (Iterable[S])

• chunksize (int | None)

Return type:

list[T]

map_async(func, iterable, chunksize=None, callback=None, error_callback=None)

Asynchronous version of map() returning MapResult.

Parameters:

• func (Callable[[S], T])

• iterable (Iterable[S])

• chunksize (int | None)

• callback (Callable[[T], None] | None)

• error_callback (Callable[[BaseException], None] | None)

Return type:

MapResult[T]

imap(func, iterable, chunksize=1)

Like map() but return an iterator.

Equivalent to map(func, iterable).

Parameters:

• func (Callable[[S], T])

• iterable (Iterable[S])

• chunksize (int)

Return type:

Iterator[T]

imap_unordered(func, iterable, chunksize=1)

Like imap() but ordering of results is arbitrary.

Parameters:

• func (Callable[[S], T])

• iterable (Iterable[S])

• chunksize (int)

Return type:

Iterator[T]

starmap(func, iterable, chunksize=None)

Apply func to each argument tuple in iterable.

Equivalent to list(itertools.starmap(func, iterable)).

Block until all results are ready and return them in a list.

The iterable is choped into a number of chunks which are submitted as separate tasks. The (approximate) size of these chunks can be specified by setting chunksize to a positive integer.

Consider using istarmap() or istarmap_unordered() with explicit chunksize for better efficiency.

Parameters:

• func (Callable[[...], T])

• iterable (Iterable[Iterable[Any]])

• chunksize (int | None)

Return type:

list[T]

starmap_async(func, iterable, chunksize=None, callback=None, error_callback=None)

Asynchronous version of starmap() returning MapResult.

Parameters:

• func (Callable[..., T])

• iterable (Iterable[Iterable[Any]])

• chunksize (int | None)

• callback (Callable[[T], None] | None)

• error_callback (Callable[[BaseException], None] | None)

Return type:

MapResult[T]

istarmap(func, iterable, chunksize=1)

Like starmap() but return an iterator.

Equivalent to itertools.starmap(func, iterable).

Parameters:

• func (Callable[[...], T])

• iterable (Iterable[Iterable[Any]])

• chunksize (int)

Return type:

Iterator[T]

istarmap_unordered(func, iterable, chunksize=1)

Like istarmap() but ordering of results is arbitrary.

Parameters:

• func (Callable[[...], T])

• iterable (Iterable[Iterable[Any]])

• chunksize (int)

Return type:

Iterator[T]

close()

Prevent any more tasks from being submitted to the pool.

Return type:

None

terminate()

Stop the worker processes without completing pending tasks.

Return type:

None

join()

Wait for the worker processes to exit.

Return type:

None

class mpi4py.util.pool.ThreadPool

Bases: Pool

Pool using threads as workers.

class mpi4py.util.pool.AsyncResult

Asynchronous result.

get(timeout=None)

Return the result when it arrives.

If timeout is not None and the result does not arrive within timeout seconds then raise TimeoutError.

If the remote call raised an exception then that exception will be reraised.

Parameters:

timeout (float | None)

Return type:

T

wait(timeout=None)

Wait until the result is available or timeout seconds pass.

Parameters:

timeout (float | None)

Return type:

None

ready()

Return whether the call has completed.

Return type:

bool

successful()

Return whether the call completed without raising an exception.

If the result is not ready then raise ValueError.

Return type:

bool

class mpi4py.util.pool.ApplyResult

Bases: AsyncResult

Result type of apply_async().

class mpi4py.util.pool.MapResult

Bases: AsyncResult

Result type of map_async() and starmap_async().