Parallel programming and Cython — David Woods

[EuroPython 2025 — South Hall 2A on 2025-07-16] 🎤 *Parallel programming and Cython by David Woods* 🔗 https://ep2025.europython.eu/session/parallel-programming-and-cython 📝 Abstract: Cython is a well-established tool that lets you generate low-level, quick C/C++ code from Python-like syntax, letting you blur the boundary between an optimized C/C++ layer and the Python code that uses it. Cython has long had the ability to write parallel code and this has become increasingly important with recent support for both subinterpreters and for freethreaded Python. Firstly in this talk I aim to provide a general overview of what's available in Cython and how you can use it. * Most obvious is Cython's inbuilt "prange" mechanism - a high-performance way of handling loops with independent iterations but not a general parallelization solution. * Second (and often neglected) you can use Python's own `threading` module - the key here is to ensure that the Cython code that it calls is performant. * Third, you have easy access to the C++ standard library and the threading tools that this exposes. My advice here is that the synchronization tools are nice, but using C++ to create new threads requires care. * Finally there are “subinterpreters” which provides isolation between parallel processes both as a pro and a con. Since this is new to Cython (and not yet widely used in Python) there are limitations and pitfalls. I'll also discuss future plans, and some details of the current implementation in Cython. Secondly in this talk I will cover "what's new". This is mostly improvements targeting freethreaded Python although many are useful generally. Most significant is `cython.critical_section` which provides a convenient syntax around Python's internal `Py_BEGIN/END_CRITICAL_SECTION` – essentially a very localized Interpreter Lock around a single object. I will compare the differences and similarities to the Global Interpreter Lock (GIL) of regular Python since understanding this is necessary to write code that works well in both. Finally, I’ll discuss Cython’s typical attitude towards thread-safety in the freethreaded build (together will some illustrative examples from the Cython internals), and what that means for writing your own performant and thread-safe --- This work is licensed under a Creative Commons Attribution-NonCommercial-ShareAlike 4.0 International License: https://creativecommons.org/licenses/by-nc-sa/4.0/