1.2 - Introduction to the `multiprocessing` module

Key Concept

The multiprocessing module allows you to run code in parallel using multiple processor cores, enabling faster execution of computationally intensive tasks. It achieves this by creating and managing separate processes, each with its own memory space.

The `multiprocessing` module in Python provides a way to bypass the limitations of the Global Interpreter Lock (GIL) and achieve true parallelism, allowing programs to utilize multiple CPU cores simultaneously. This is particularly beneficial for CPU-bound tasks, where the execution time is dominated by computations rather than waiting for I/O operations. Instead of running code sequentially, the `multiprocessing` module allows you to distribute work across multiple processes, effectively speeding up your program.

Topics

Process Creation: Creating new processes to execute tasks concurrently.

At its core, the `multiprocessing` module enables the creation of independent processes. A process is essentially a separate instance of the Python interpreter, with its own memory space and execution context. This isolation is crucial because it allows processes to run concurrently without interfering with each other. You create a process using the `Process` class, passing a function to be executed within that process. The `start()` method then initiates the execution of this function in the new process. The `Process` class also provides methods for retrieving the return value of the function executed in the process, and for handling exceptions that occur within the process. For example, you can define a function and then create a `Process` object that runs that function, and then retrieve the result when the process completes.

Process Communication: Mechanisms for processes to exchange data (e.g., queues, pipes).

Since processes have separate memory spaces, they cannot directly access each other's variables. Therefore, a mechanism for communication is required. The `multiprocessing` module offers several ways to facilitate this communication. These include:

Process Pools: Simplified way to manage a group of worker processes.

Creating and managing individual processes can be cumbersome, especially when dealing with a large number of tasks. The multiprocessing module provides the Pool class, which simplifies the process of distributing tasks across a group of worker processes. A Pool object creates a pool of worker processes and allows you to submit tasks to the pool, which will then be executed by the worker processes. The Pool class provides methods for map, apply_async, and imap, which allow you to submit tasks to the pool in different ways. mapapplies a function to each element of an iterable and returns a list of results. apply_async submits a single task to the pool and returns an AsyncResult object, which can be used to retrieve the result of the task. imap is similar to map, but it returns an iterator that yields results as they become available.

Global Interpreter Lock (GIL): Understanding its limitations on true parallelism in Python.

The Global Interpreter Lock (GIL) is a mechanism in the Python interpreter that allows only one thread to execute Python bytecode at a time, even on multi-core processors. This limitation can hinder the performance of multi-threaded programs, especially those that are CPU-bound. The multiprocessing module circumvents the GIL by creating separate processes, each with its own interpreter and memory space. This allows multiple processes to execute Python bytecode concurrently, effectively utilizing all available CPU cores. While the GIL prevents true parallelism within a single process, the multiprocessing module enables parallelism across multiple processes, leading to significant performance improvements for CPU-intensive tasks. It's important to note that communication between processes can introduce overhead, so it's crucial to carefully consider the trade-offs between parallelism and communication costs.

Exercise

Briefly consider a scenario where you could benefit from parallel processing. What tasks would be suitable?

Answer: Imagine you’re working with large satellite images (e.g., 10,000 × 10,000 pixels). You need to apply a filter (like edge detection or Gaussian blur).

Parallel Image Tiles Example

from multiprocessing import Process, Queue
import numpy as np

def process_tile(tile, q, idx):
    # Example: simple inversion filter
    q.put((idx, 255 - tile))

if __name__ == "__main__":
    img = np.random.randint(0, 256, (2000, 2000), dtype=np.uint8)  # fake grayscale image
    tiles = np.array_split(img, 4)  # split into 4 chunks
    q = Queue()
    procs = []

    for i, tile in enumerate(tiles):
        p = Process(target=process_tile, args=(tile, q, i))
        p.start()
        procs.append(p)

    results = [q.get() for _ in range(len(tiles))]
    results.sort()  # sort by index
    img_out = np.vstack([r[1] for r in results])  # stitch back

    for p in procs:
        p.join()

    print("Processed image shape:", img_out.shape)

Common Pitfalls

Best Practices