Graham Dumpleton (of mod_wsgi fame) pointed out a glaring omission and subtle inaccuracy in my post about Python’s state structures.
When discussing what I called “Pythonic threads”, which are threads created and managed by Python, which have a PyThreadState structure allocated to them and that are able to call into the Python API and run Python code, I said: “theoretically threads can be created which will not be under the interpreter’s control; these threads won’t have a PyThreadState structure and must never call a Python API; this is not very common“. This is sort of correct, but misleading: such threads are quite obvious to have in a non-Python applications that has a Python interpreter embedded in it (something Graham probably knows a thing or two about :). These ‘foreign’ (I don’t know of formal term) threads indeed can’t call the Python API and run Python code unless they’re ‘migrated’ (don’t know of a formal term) to come under the control of a Python interpreter, having the correct bookkeeping structures allocated and initialized for them.
Graham also noted correctly that such ‘migration’ is easier when there’s just one Python interpreter in the process, and harder if there’s more than one. If you have just one interpreter, you can use the (somewhat confusingly named) PyGILState_Ensure and PyGILState_Release calls to do the job. These calls allow a thread to ensure it has the GIL (blocking if necessary until it can be acquired), but importantly can be called by a thread that isn’t under Python’s control at all; the necessary work to allow the thread to call Python’s API will be done for you. These calls assume there is only one Python interpreter (interp_head points to a PyInterpreterState which points to NULL), hence they mustn’t be used in a multi-interpreter process, if that is your case you’ll have to ‘migrate’ the threat manually with lower level primitives.
As per my errata policy, I’ve chosen to update the original post as if the error never happened and write this post note the mistake for future reference and for those subscribed to the RSS feed of the Errata category.
I’d like to thank Graham for the important correction.
Comments