Note: The following was generated by AI with the materials from the workshop (slides and activities).

Joe showed how to speed up Python code by spreading work across multiple CPU cores.

Slides (Google Slides)

Why parallelize?

Single-core CPU speeds have plateaued, so modern CPUs ship with more and more cores. Most Python code runs one instruction at a time. Parallelizing splits the work into processes that run at the same time.

Three ways to do it

  • NumPy (and SciPy, PyTorch, scikit-learn, etc.) already multithreads many operations in compiled code. It’s the easiest option but gives you the least control.
  • multiprocessing is in the standard library. It sidesteps the Global Interpreter Lock (GIL) by spawning separate Python processes, at the cost of some startup overhead. Scripts run like normal Python, but on HPC they’re limited to a single node.
  • MPI via mpi4py is the standard used by heavy simulation codes and samplers (implementations include OpenMPI and MPICH). It gives you the most control and is the hardest to use. Scripts run on every rank at once and are launched with:
mpirun -n <N> python -m mpi4py <myscript.py>

Activity

  1. Estimate π by Monte Carlo integration using multiprocessing.Pool: exercise, solution
  2. Compute the Mandelbrot set with mpi4py: exercise, solution

Useful references: Pool.map tutorial, multiprocessing docs, mpi4py tutorial, mpi4py install.