Creating fancy images with Matplotlib

I have to give a short presentation at SOSP next week, and for it, I needed to have some nice pictures representing a distributed array. After trying out several tools for trying to create these, I began to lament and cry over the state of Linux drawing software. But that’s a different story. I ended up writing a simple matplotlib script to generate the pictures I needed, and since it worked out pretty well, I thought I’d share it here.

Here’s the kind of picture I’m referring to:

array

It turns out this is pretty straightforward using matplotlib. Here’s the basic function:

def draw_array(a, target=None):
    fig = pylab.gcf()
    fig.frameon = False

    ax = fig.gca()
    #ax.set_axis_off()

    ax.patch.set_facecolor('white')
    ax.set_aspect('equal', 'box')
    ax.xaxis.set_major_locator(plt.NullLocator())
    ax.yaxis.set_major_locator(plt.NullLocator())

    size = 1.0
    z_scale = 1.4
    i = 0
    for z in reversed(range(a.shape[2])):
        for (x,y),v in np.ndenumerate(a[:, :, z]):
            i += 2
            alpha = a['transparency'][x,y,z]
            color = tuple(a['color'][x,y,z])
            off_x = 0.01 + x + size + z / z_scale
            off_y = y + size + z / z_scale

            rect = pylab.Rectangle([off_x, off_y], size, size,
                                   facecolor=color, edgecolor=(0,0,0),
                                   zorder = i, alpha = alpha)
            ax.add_patch(rect)

            cx = off_x + size/2
            cy = off_y + size/2

            # sigh
            label = str(a['name'][x,y,z])
            w, h = pylab.matplotlib.text.TextPath((0,0), label).get_extents().size / 30

            #print w, h

            text = pylab.Text(cx - w / 2, cy - h / 2, label, zorder=i+1)
            ax.add_artist(text)

    ax.autoscale_view()
    if target is not None:
        pylab.savefig(target)
    return ax

The first part of this just turns off the various lines for the axes. We then iterate through the elements of the array and create a Rectangle() for each one; each “layer” (z-axis) is shifted off to the right a little bit from the previous, to give our illusion of depth. (We don’t want a normal perspective projection, as it would hide too much of the deeper layers).

The “sigh” comment is where I’m using a hack to determine the size of the text we’re going to put in so I can center it in the array cell. I couldn’t find an easier way to do this, and no, I don’t know why I have to divide the result by 30.

The input array has 3 fields which specify how to render each rectangle:

dtype=([('color', 'f,f,f'), ('name', 'i'), ('transparency', 'f')]))

Now we can construct an arbitrary array and feed it into our function:

shape = (3,3,5)
a = np.ndarray(shape, dtype=([('color', 'f,f,f'), ('name', 'i'), ('transparency', 'f')]))
a['name'] = np.arange(np.prod(shape)).reshape(shape)
a['transparency'] = 1.0
a['color'] = (1,1,1)
return a

draw_array(a, target='array.pdf')

Once we have the basics out of the way, we can do some fancy rendering really easily. First, let’s make a little helper class to draw slices:

class draw_slice(object):
    def __init__(self, a, target=None):
        self.a = a
        self.target = target

    def __getitem__(self, slc):
        slice_z = np.copy(self.a)
        slice_z['color'][slc] = (0.9, 0.5, 0.3)
        slice_z['transparency'] = 0.9
        draw_array(slice_z, self.target)

We can wrap an array in draw_slice() to make it easy to construct pictures of slices:

draw_slice(a)[:,:,1]

slice-z

We can be fancier if we like too, drawing the results of a filter operation:,

draw_slice(a)[a['name'] <= 1]

filter

If you are interested, the full code for creating these figures is here: https://gist.github.com/rjpower/7249729. All you need is matplotlib and numpy.

statically linking shared libraries with libtool

I run a lot of experiments on our local cluster.  Unfortunately, over time, the library versions on the cluster tend to diverge from those on my local machine. As a result, I’ve gotten used to seeing this:

/usr/bin/python: /lib/x86_64-linux-gnu/libc.so.6: version `GLIBC_2.17' not found 
(required by /home/power/w/spartan/build/.libs/libspartan.so.0)

If I was using Go this wouldn’t be a problem, as they statically link everything.  Despite the crowd of people who think shared libraries are the bees knees, I agree with this approach — it’s just far simpler than trying to deal with errors like the above. You can solve this most of the time by simply statically linking everything (–enable-static).  Sadly, if you’re trying to build a shared library (in my case, an extension module for Python), you can’t really go this route. (Statically linking Python API calls into a library which is then hoisted into Python is going to end very, very badly). What am I supposed to do with this error? If you trawl around the web, you find that depending on the exact error, you should either:

  • find an old version of GLIBC and link against that
  • insert assembly directives to indicate the old symbol

If you happen to have a single symbol that’s pulling the newer version, the latter is an easy fix (though it’s a bit annoying to ensure you’ve always got the directive declared before you use a function). If you’ve somehow acquired a dependency on the whole library things become more annoying. In my case, this dependency seemed to result from the chain:

_spartan_wrap.so -> libspartan.so -> libstdc++.so -> libc.so

Oddly enough, depending directly on libstdc++ isn’t the problem. If I remove the dependency on libspartan (just linking directly against all of the objects), we’re fine:

ldd -v .libs/_spartan_wrap.so
...

        Version information:
        .libs/_spartan_wrap.so:
                librt.so.1 (GLIBC_2.2.5) => /lib/x86_64-linux-gnu/librt.so.1
                libgcc_s.so.1 (GCC_3.0) => /lib/x86_64-linux-gnu/libgcc_s.so.1
                libm.so.6 (GLIBC_2.2.5) => /lib/x86_64-linux-gnu/libm.so.6
                libc.so.6 (GLIBC_2.15) => /lib/x86_64-linux-gnu/libc.so.6
                libc.so.6 (GLIBC_2.14) => /lib/x86_64-linux-gnu/libc.so.6
                libc.so.6 (GLIBC_2.4) => /lib/x86_64-linux-gnu/libc.so.6
                libc.so.6 (GLIBC_2.3.2) => /lib/x86_64-linux-gnu/libc.so.6
                libc.so.6 (GLIBC_2.3.4) => /lib/x86_64-linux-gnu/libc.so.6
                libc.so.6 (GLIBC_2.2.5) => /lib/x86_64-linux-gnu/libc.so.6
                libpthread.so.0 (GLIBC_2.3.2) => /lib/x86_64-linux-gnu/libpthread.so.0
                libpthread.so.0 (GLIBC_2.2.5) => /lib/x86_64-linux-gnu/libpthread.so.0
                libstdc++.so.6 (GLIBCXX_3.4.14) => /usr/lib/x86_64-linux-gnu/libstdc++.so.6
                libstdc++.so.6 (GLIBCXX_3.4.15) => /usr/lib/x86_64-linux-gnu/libstdc++.so.6
                libstdc++.so.6 (GLIBCXX_3.4.10) => /usr/lib/x86_64-linux-gnu/libstdc++.so.6
                libstdc++.so.6 (CXXABI_1.3) => /usr/lib/x86_64-linux-gnu/libstdc++.so.6
                libstdc++.so.6 (GLIBCXX_3.4) => /usr/lib/x86_64-linux-gnu/libstdc++.so.6

Notice, no reference to GLIBC_2.17. Luckily in this case there’s a simple solution: make all of the helper libraries into convenience libraries. This causes those libraries to be statically linked and avoids pulling in the extra dependencies:

# old
# lib_LTLIBRARIES = _wrap.la liba.la libb.la

# new 
noinst_LTLIBRARIES = liba.la libb.la
lib_LTLIBRARIES = _wrap.la

If we weren’t lucky — we’re actually using the symbol that’s from a later version — than we can force static linking of your dependent library; you do this by listing it explicitly by name in your LIBADD variable:

_spartan_wrap_la_LIBADD = -module -lrt /usr/lib/gcc/x86_64-linux-gnu/4.8/libstdc++.a

libtool will complain that linking against a static library isn’t portable (which is true), but it should work correctly as long as the static library was built with -fPIC.