Shekhar Koirala: Parallel Python @ PyCon Ireland 2024

Parallel Python: Embracing the Future with Sub-Interpreters and Free Threading - Shekhar Koirala We will start by examining the new per-interpreter GIL introduced in Python 3.12, which allows sub-interpreters to run Python code concurrently. We'll explore the enhancements in Python 3.13, which include support for sub-interpreters in the standard library, making parallel execution more accessible. Additionally, we will discuss the revolutionary NoGIL compile option, paving the way for Python threads to run truly in parallel, potentially becoming the default in the future. The talk will cover the traditional multiprocessing approach, comparing it with the newer methods of sub-interpreters and free-threading. We will assess the pros and cons of each approach, considering their complexities and suitability for various types of problems, whether CPU/IO bound. Thread safety remains a crucial aspect of parallel programming. We will address common pitfalls such as race conditions and provide practical solutions using synchronization primitives to ensure thread-safe code. Lastly, we will explore a real-world application combining sub-interpreters and free-threading in a multi-worker, multi-thread ASGI web server, benchmarking its performance against traditional web server architectures.