pyactivemq does threads
After employing various debugging techniques (mostly some RTFM and printf debugging), I was able to get C++ threads to call back into Python code successfully. In pyactivemq, this allows you to register a MessageListener on a Session and have the thread associated with the Session call into Python whenever a message arrives.
All the info I needed to solve the problem was contained in PEP 311: Simplified Global Interpreter Lock Acquisition for Extensions and Thread State and the Global Interpreter Lock. My main mistake was that I didn’t call PyEval_InitThreads() in my module’s initialization code, which seems to turn attempts to acquire the GIL into a no-op, which meant that I had multiple threads calling into the interpreter at the same time, with the associated “interesting” results.
June 21st, 2007 at 8:04 pm
Do you have any unit tests etc that were able to demonstrate bugs in multi-threaded code? (Sounds tricky to test that kind of “interesting results”?)
June 21st, 2007 at 11:12 pm
By “interesting results” I mean segfaults within a few seconds of the threads calling into Python without holding the lock. Writing tests for these was pretty easy. I guess these tests don’t prove that the code is right, but it no longer crashes and Valgrind is happy, so I think it might be right.
Another issue I had was that I didn’t release the GIL in cases where the Python callback threw an exception instead of completing normally. Again, this was pretty easy to test.
I suspect there might still be few reference counting bugs lurking. sys.getrefcount(…) is handy to do basic checks in this regard. There’s also some special ways to build Python to make it do extra checks at runtime, but I haven’t tried these yet.
As an added bonus, the test suite for pyactivemq turned up a few race conditions in ActiveMQ-CPP.