libffi

February 1st, 2008

It seems there is some renewed interest in libffi, the library used by ctypes and JNA to call functions in native libraries from Python and Java, respectively. Until recently, the efforts around libffi have been very fragmented, with various patches only being available in only ctypes or only JNA (or only elsewhere).

On the ctypes side, there are some patches to build libffi with Visual Studio, which are useful for Windows junkies like myself. There is also a patch for Win64 support, which really needs to get into JNA (Java on Windows Server 2003 x64 rocks!). Timothy Wall of JNA fame has also produced some patches. A lot of this work has featured on the gcc-patches mailing list.

Anthony Green is also doing some work on libffi, but this seems to be happening separately from the work of the gcc-patches folks. Hopefully all these disparate efforts can be unified so that we can all benefit from a single libffi that works on many platforms (including Win64, please!).

Hadoop does WebDAV

July 25th, 2007

There’s recently been some movement on HADOOP-496, the issue for exposing the Hadoop Distributed File System as a WebDAV store. Maximum respect to Enis Soztutar for sorting this out.

The patch hasn’t made it into SVN yet, but I hope it will show up there in a few weeks. Getting files onto your HDFS has never been this easy: add a new network place in Windows Explorer, open the folder, drag and drop! I’m pretty sure your favourite Linux desktop environment will have similar functionality.

I’ve also been running the litmus WebDAV server protocol compliance test suite against this Hadoop/WebDAV combination to test the patch. There’s still a few issues to fix, but things are looking good.

In case you want to add a WebDAV interface to your Java application, you might consider building it on top of Apache Jackrabbit, as this patch does. I took a brief look at doing what this patch does with Jakarta Slide, but even with the WebDAV Construction Kit to help you, Slide is a rather large beast.

Intel Threading Building Blocks open sourced

July 25th, 2007

Intel has released Threading Building Blocks (TBB), their library for doing multithreaded programming with C++, under the GPL. If you’re stuck in C++ land but yearn for the power of all those shiny new cores (with 4 Core 2 Duo cores now within reach of anyone with a modest budget), this might be for you. As an added bonus, the documentation looks pretty impressive.

They’re also running a contest to promote TBB.

More from Eclipse and Java land

July 4th, 2007

My new favourite Eclipse 3.3 keyboard shortcut is Ctrl+3. It allows you to search through almost anything you can do in the IDE. If you’re a BASH user, Ctrl+3 will remind you of Ctrl+R (reverse-i-search).

InfoQ has an article on New Concurrency Features for Java SE 7, with a link to Doug Lea’s A Java Fork/Join Framework. Along these lines, I have been playing with CompletionServices and Executors quite a bit the past 3 weeks. I hope to present the fruits of my labour here soon.

Eclipse 3.3 First Impressions

July 1st, 2007

Eclipse 3.3 (aka Eclipse Europa) was released a few days ago and I decided to give it a spin. Here are some notes in no particular order:

  • The probably isn’t a new feature, but you can configure your default web browser under Window | Web Browser.
  • You can now do “Sort Members” on a whole project. You can also “Organize Imports” in the same way. Between Eclipse’s code formatter, cleanups on save, the ability to export settings for these features and The Checkstyle Plug-in for Eclipse there is no reason I can think of for all the Java code in your project not to look exactly the same.
  • Libraries references by a Java project are all grouped together under a “Referenced Libraries” item in the tree view. Now you don’t have to fiddle with the filters so often.
  • I gave the Dynamic Languages Toolkit a quick spin. I was able to get up and running with JRuby in seconds.
  • The Help system uses Jetty now. Nice. But for some reason it wants to listen on all my interfaces, causing Windows to pop up a Firewall allow/block dialog the first time you launch the Help system. What’s up with that?
  • The Java compiler’s warnings with regards to generics have been improved. Way back in the 3.3Mx days I found that it rejected some invalid generics code that would compile with 3.2, so it looks like there’s been some improvements.
  • The new Run/Debug launching is weird. The default is now to have Eclipse launch what it thinks you want to launch, instead of launching the last thing you launched. You can turn off this behavior under Window | Preferences | Run/Debug | Launching by selecting “Always launch the previously launched application”.
  • The new interface for the Rename refactoring is annoying. Previously I could Alt+Shift+R and a dialog would pop up with the previous name selected, so that if I typed something, it would overwrite the old name. Now I get an editing thing that takes my cursor position in the name I’m trying to refactor into account, with no obvious way of completely nuking the old name so that I can just type the new one quickly.
  • The existing FindBugs plugin still works. Nice. Still need to install a Subversion plugin. Maybe it’s time to switch to Subversive?
  • The Remote System Explorer is very useful.

Now for my only major gripe so far: the platform proxy settings. Bug 154100 - Platform level proxy settings has the details. These proxy settings were clearly not designed by someone who actually uses a proxy on a daily basis, especially with a laptop. Here are some hints:

  • As I move around with my laptop between networks, my proxy settings usually need to change. The proxy settings need some kind of “profile” feature. I guess I could switch workspaces, but that’s just annoying.
  • As far as I can tell, I can’t exclude ranges of IPs or hostnames with wildcards (e.g., *.foo.com or 192.168.0.0/255.255.0.0 or 192.168.0.0/16). I guess I could type in the host name or IP address of every machine on my network, but that’s going to take a while.

And if you’re going to fix this second issue, don’t be stupid like Firefox. If I add 192.168.0.0/255.255.0.0 to the list of IPs to exclude, I actually also mean any host name that resolves to an IP in that range.

Update: Tried Subversive. No, thank you. I’ll stick with Subclipse for the moment.

Random Java Stuff

June 28th, 2007

Java Genetic Algorithms Package (JGAP) is (surprise) a genetic algorithm package for Java. JGAP 3.2 was recently released. Coolest feature: you can use JGAP to evolve a Robocode robot (more about Robocode).

After reading this post by Bill Pugh to OpenJDK’s quality-discuss mailing list, I immediately gave FindBugs a try. The Eclipse plugin for FindBugs is highly recommended. Getting FindBugs going is Maven is a bit more… interesting. I think you want the findbugs-maven-plugin, although there’s also a maven-findbugs-plugin. It seems findbugs-maven-plugin includes support for FindBugs 1.2.0, but I wasn’t able to consistently convince it to use that instead of 1.0.0. Good luck.

I promptly unleashed FindBugs on Apache ActiveMQ and found some minor issues and some more serious issues. Maximum respect to Rob Davies for fixing the latter collection.

Update: I unleashed FindBugs on JRuby. See JRUBY-1173.

Java Thread vs Executor

June 19th, 2007

Quote of the day from Java Concurrency in Practice: “The primary abstraction for task execution in the Java class libraries is not Thread, but Executor.”

First pyactivemq bug report

June 16th, 2007

I received what I am choosing to call the first pyactivemq bug report. World domination is now but a mere formality.

I recently ported one of the JMS examples from the Java EE 5 tutorial to Python and pyactivemq. Check it out on the Examples page. Everything works, except the final call to cancel the subscription. This should work once AMQCPP-132 gets fixed.

Random Numbers with Python, C++ and MATLAB

June 13th, 2007

I’m currently working on some algorithms for which I’ve found it very useful to be able to generate the same random numbers in Python (using NumPy’s numpy.random module), C++ (using the Boost Random Number Library) and MATLAB (using the rand function).

I use the random numbers in tests, where being able to generate the same random numbers everywhere has been very useful. It allows me to compare the output of my algorithms exactly. For example, I might implement an algorithm in Python with NumPy first, write some tests, and then rewrite it in C++ (for the cases where I need the extra speed).

Getting MATLAB and NumPy to generate the same random numbers is straight forward. It seems NumPy uses the Mersenne Twister algorithm. MATLAB can be configured to do the same. Specifically, the following code generates the exact same 10 numbers with NumPy 1.0.3:

numpy.random.seed(0); numpy.random.rand(10,1)

and MATLAB R2006b:

rand('twister',0); rand(10,1)

However, the Mersenne Twister combined with a uniform distribution generator from the Boost Random Number Library don’t generate this same sequence. A quick glance through the NumPy and Boost code seemed to indicate that they use the same magic constants. I used the following C++ code with Boost 1.34.0:

#include <boost/random.hpp>
boost::mt19937 rng;
rng.seed(0U);
boost::uniform_real<> uni_dist(0, 1);
boost::variate_generator<boost::mt19937&, boost::uniform_real<> > uni(rng, uni_dist);

I’m guessing that NumPy and Boost simply do slightly different things with the bits that come out of the MT algorithm to generate the respective uniform distributions.

To work around this problem, I cooked up a Boost.Python wrapper for the Boost Random Library classes so that I could generate exactly the same numbers in Python and C++. You can take a look at the code here (the tests are still a work in progress):

The interface exposed by this wrapper behaves more like the stuff in numpy.random, compared to the wrapper in pyboost, which could be considered a very literal translation. It probably also wouldn’t be too hard to wrap these classes with MEX for MATLAB, guaranteeing you the “same” random numbers across all three platforms.

Disclaimer: I am not an expert on anything related to random numbers (except making them up in my head).

libstdc++ debugging features and libcwd

June 6th, 2007

Unbeknownst to me, libstdc++ has some very cool debugging features that can be enabled by defining some or all of _GLIBCXX_DEBUG, _GLIBCXX_DEBUG_PEDANTIC or _GLIBCXX_CONCEPT_CHECKS. You can read more about these defines in Debugging schemes and strategies and Concept checkers — new and improved! Combined with Valgrind, this provides most of the tools you need to find bugs in many C++ programs.

Along these lines, you might also be interested in libcwd, which is “a thread-safe, full-featured debugging support library for C++ developers.”

Spotted a mention of the libstdc++ defines in a comment to Memory Checker Tools For C++? on Slashdot.

Update: I think the documentation mentions this somewhere, but I still got bitten by this issue: you can’t mix code compiled with these defines with “normal” code. In my case, I was trying to link some debug code against CppUnit installed on my system, with strange results (for example, you’ll see a destructor in the normal code segfault as it tries to deallocate a debug std::string instance).