Hyperic SIGAR (System Information Gatherer and Reporter) is a cross-platform, cross-language library and command-line tool for accessing operating system and hardware information in C, Java, Perl and C#. SIGAR is licensed under the GPL version 2. Not quite sure what this implies for Java projects, but anyway.
Just spotted this library as part of the upcoming GridGain 2.0 release.
I had high hopes for the OpenMP support introduced in GCC 4.2, and then came this:
Reported by Nathan Bell on the NumPy mailing list.
Intel has released Threading Building Blocks (TBB), their library for doing multithreaded programming with C++, under the GPL. If you’re stuck in C++ land but yearn for the power of all those shiny new cores (with 4 Core 2 Duo cores now within reach of anyone with a modest budget), this might be for you. As an added bonus, the documentation looks pretty impressive.
They’re also running a contest to promote TBB.
I’m currently working on some algorithms for which I’ve found it very useful to be able to generate the same random numbers in Python (using NumPy’s numpy.random module), C++ (using the Boost Random Number Library) and MATLAB (using the rand function).
I use the random numbers in tests, where being able to generate the same random numbers everywhere has been very useful. It allows me to compare the output of my algorithms exactly. For example, I might implement an algorithm in Python with NumPy first, write some tests, and then rewrite it in C++ (for the cases where I need the extra speed).
Getting MATLAB and NumPy to generate the same random numbers is straight forward. It seems NumPy uses the Mersenne Twister algorithm. MATLAB can be configured to do the same. Specifically, the following code generates the exact same 10 numbers with NumPy 1.0.3:
numpy.random.seed(0); numpy.random.rand(10,1)
and MATLAB R2006b:
rand('twister',0); rand(10,1)
However, the Mersenne Twister combined with a uniform distribution generator from the Boost Random Number Library don’t generate this same sequence. A quick glance through the NumPy and Boost code seemed to indicate that they use the same magic constants. I used the following C++ code with Boost 1.34.0:
#include <boost/random.hpp>
boost::mt19937 rng;
rng.seed(0U);
boost::uniform_real<> uni_dist(0, 1);
boost::variate_generator<boost::mt19937&, boost::uniform_real<> > uni(rng, uni_dist);
I’m guessing that NumPy and Boost simply do slightly different things with the bits that come out of the MT algorithm to generate the respective uniform distributions.
To work around this problem, I cooked up a Boost.Python wrapper for the Boost Random Library classes so that I could generate exactly the same numbers in Python and C++. You can take a look at the code here (the tests are still a work in progress):
The interface exposed by this wrapper behaves more like the stuff in numpy.random, compared to the wrapper in pyboost, which could be considered a very literal translation. It probably also wouldn’t be too hard to wrap these classes with MEX for MATLAB, guaranteeing you the “same” random numbers across all three platforms.
Disclaimer: I am not an expert on anything related to random numbers (except making them up in my head).
Unbeknownst to me, libstdc++ has some very cool debugging features that can be enabled by defining some or all of _GLIBCXX_DEBUG, _GLIBCXX_DEBUG_PEDANTIC or _GLIBCXX_CONCEPT_CHECKS. You can read more about these defines in Debugging schemes and strategies and Concept checkers — new and improved! Combined with Valgrind, this provides most of the tools you need to find bugs in many C++ programs.
Along these lines, you might also be interested in libcwd, which is “a thread-safe, full-featured debugging support library for C++ developers.”
Spotted a mention of the libstdc++ defines in a comment to Memory Checker Tools For C++? on Slashdot.
Update: I think the documentation mentions this somewhere, but I still got bitten by this issue: you can’t mix code compiled with these defines with “normal” code. In my case, I was trying to link some debug code against CppUnit installed on my system, with strange results (for example, you’ll see a destructor in the normal code segfault as it tries to deallocate a debug std::string instance).
After employing various debugging techniques (mostly some RTFM and printf debugging), I was able to get C++ threads to call back into Python code successfully. In pyactivemq, this allows you to register a MessageListener on a Session and have the thread associated with the Session call into Python whenever a message arrives.
All the info I needed to solve the problem was contained in PEP 311: Simplified Global Interpreter Lock Acquisition for Extensions and Thread State and the Global Interpreter Lock. My main mistake was that I didn’t call PyEval_InitThreads() in my module’s initialization code, which seems to turn attempts to acquire the GIL into a no-op, which meant that I had multiple threads calling into the interpreter at the same time, with the associated “interesting” results.
After searching far and wide (actually, after Googling for the right phrase and clicking on the 13th result), I found a blog entry that deals with the issues you can run into when allocating and deallocating memory across module boundaries on Windows. Read it at The Old New Thing: Allocating and freeing memory across module boundaries.
I have sent pyactivemq 0.0.2 out into the world. pyactivemq is a wrapper for ActiveMQ message broker. pyactivemq uses Boost.Python to wrap the ActiveMQ C++ library.
Update: Since making this release, I’ve done a few more hours of work on pyactivemq. I’ve almost gotten multiple consumers with message listeners in separate sessions to receive messages, i.e. multiple threads calling back into Python code almost works. I need to stick a lock or grab onto the GIL somewhere, but I’m not exactly sure where yet.
I was recently writing some tests for a Python extension module that uses Boost.Python. To keep things agile (i.e. keep compile times down) I have a few #ifdefs in my code that allows me to disable various parts of the module when I’m only interested in other parts. This greatly reduces the time of edit-compile-test cycle when your module starts getting really large. I wanted to skip certain tests in my test suite depending on whether a part of the module was available. Here’s what I came up with:
import unittest
class _test_foo:
def setUp(self): pass
def test_baz(self):
self.assert_(True)
def tearDown(self): pass
try:
from extmodule import foo
class test_foo(_test_foo, unittest.TestCase):
pass
except ImportError:
print 'skipping foo tests'
if __name__ == '__main__':
unittest.main()
GCC 4.2.0 has been released (after some delays). The most interesting feature for me is that OpenMP is now supported for the C, C++ and Fortran compilers. OpenMP is already supported by the Intel and Microsoft Visual Studio 2005 compilers (but not the Express editions of the latter). OpenMP allows you to make use of all the spankin’ new cores on your brand new CPU, without requiring any thread programming on your part. See also Wikipedia: OpenMP and the OpenMP tutorial.
I’m still a little unsure about what happens when other threaded code calls into OpenMP threaded code (is it even safe to do this?). I’ll see if I can find out.
According to Ulrich Drepper in The Growing Importance of Parallel Programming, the OpenMP bits have already been backported to GCC 4.1 in Fedora Core 6 and RHEL 5. This is why I love the Fedora guys.
By the way, you can get recent versions of MATLAB (everything from R2006a and up) to make use of multiple cores through OpenMP by setting the OMP_NUM_THREADS environment variable to some number greater than 1 before running MATLAB. You’ll see multiple cores being used for operations like matrix-matrix multiplication.
Also look out for the new -fstrict-overflow option in GCC. I haven’t looked at this in detail yet, but it’s been mentioned on the numpy-discussion mailing list as something to take notice of.
Update: See also Linux.com: New GCC 4.2.0 — boon to developers, bore to distros and Google Video: GCC: Current Topics and Future Directions.