malloc47

LaTeX Snippet: (Literal) One Liners

Jarrell Waggoner (malloc47@gmail.com) — Wed, 03 Apr 2013 00:00:00 -0700

There are some truly impressive LaTeX solutions for doing PowerPoint-style font-resizing to fit into a fixed width box. I recently had need of something more simple: print text on one line only, scaling the size down instead of allowing it to wrap. The following LaTeX snippet does exactly this, triggered only if the font width (before wrapping) exceeds \textwidth.

{
  \def\formattedtext{The no-wrap text to scale}%
  \newdimen{\namewidth}%
  \setlength{\namewidth}{\widthof{\formattedtext}}%
  \ifthenelse{\lengthtest{\namewidth < \textwidth}}%
  {\formattedtext}% do nothing if shorter than text width
  {\resizebox{\textwidth}{!}{\formattedtext}}% scale down
}

This requires

\usepackage{xifthen}
\usepackage{graphicx}

to handle the \ifthenelse, \lengthtest, and \resizebox statements.

It works like you might expect: check the width of the text, and then use \resizebox to scale it down, if needed. Such logic isn’t always obvious in LaTeX: arbitrary defs cannot store length information, so you have to set the type of the \namewidth variable as a dimension before you can assign/test it as a length.

As with most helpful things in LaTeX, we can wrap it up in a reusable macro:

\newcommand{\oneline}[1]{%
  \newdimen{\namewidth}%
  \setlength{\namewidth}{\widthof{#1}}%
  \ifthenelse{\lengthtest{\namewidth < \textwidth}}%
  {#1}%
  {\resizebox{\textwidth}{!}{#1}}%
}

which allows

\oneline{\Huge{The no-wrap text to scale}}

\oneline{\Huge{The quick brown fox jumped over the lazy dog, over and over and over and over again.}}

On any reasonable-sized page width, these two lines will not wrap, but the longer line will be stretched horizontally to fit in the available space.

You can find a fully-working (as of TeXLive 2012) gist here.

Tech Company Locations (North America)

Jarrell Waggoner (malloc47@gmail.com) — Sun, 24 Feb 2013 00:00:00 -0800

Since I'm soon to be on the technology job market, I decided to get a handle on where the major technology companies were located. After spending some quality time going back and forth bettween the "Career" sections of different companies and Google Maps, I assembled this:

There are a number of disclaimers here:

This is geared primarily toward locations that emphasize software engineering roles. I tried to filter accordingly if there are few/no engineering jobs listed for a paricular location, but that may not be perfect either.
Few companies provided the exact address (Microsoft being one of the exceptions) for every one of their locations, so if Google Maps searches failed, then I just approximated.
This is certainly not exhaustive; many companies (namely Amazon) have many subsidiaries under different branding, which may or may not have any engineering roles.
This doesn't include data centers, retail stores, etc.

Let me know if you see any gross errors, or would like me to include any other companies with more than one location. You can get a better view on Google Maps.

Laptops are the Stenotypes of Software Engineers

Jarrell Waggoner (malloc47@gmail.com) — Sun, 17 Feb 2013 00:00:00 -0800

Increasingly, I’ve been asked variants on the question “what will happen to desktops/laptops,” particularly in light of the proliferation of smartphones and tablets. This has resulted in several good conversations, and I’ve begun to use the following analogy when discussing this with colleagues in non-engineering disciplines:

Laptop computers will become the stenotypes of software engineers.

The stenotype is a niche device used by stenographers (most prominently, court reporters) to transcribe dialog in real-time at blindingly-fast words-per-minute rates. Fellow Emacs users might appreciate how the stenotype works: instead of typing single letters, multiple keys are “chorded” together to allow many more combinations with many fewer keys. And instead of producing single letters, many of these combinations produce syllables or whole words. These physical optimizations are coupled with conventions among stenographers to wantonly omit or abbreviate words where there is little ambiguity of meaning, which further improves efficiency—the fewer characters put to a page, the less chance of typos. This leads to the output of the stenotype being difficult to read by those who are not well-versed in the conventions (called “theories”) used by stenographers.

One could describe the typical QWERTY keyboard as the antithesis of the stenotype. Designed to reduce jams in the old typewriting systems—a constraint clearly nonexistent in modern hardware—mainstream keyboards are considered more accessible compared with a stenotype. However, even considering later iterations, such as the Dvorak layout, these keyboards cannot hope to match the ruthless efficiency of a stenotype when used in real-time transcription work.

Few would reach for a stenotype to write a letter to their mother (unless she was a stenographer herself!), and no trained court reporter would care to wonder into the courtroom with a QWERTY keyboard. Despite digital recorders making inroads into court reporting and closed caption transcription, stenotypes are still available (hint: ebay), persisting thanks to an ingrained base of stenographers who remain ruthlessly efficient at these highly-specialized tasks.

Which brings us back to computing.

Software engineers are not your average user. We don’t have an average computing workload and have a completely disparate set of tools and conventions. Dell didn’t even try to lampshade this fact with their Sputnik Ubuntu laptop aimed squarely at developers. And, while tablets have proven to be capable of everything from typical computing tasks to basic software development,

If it doesn’t have a keyboard, I feel that my thoughts are being forced out through a straw.
—Joey Hess

The immanent death of the laptop is greatly exaggerated; after all, the stenotype lives on to this day. However, the fate of the laptop as we know it—available in every imaginable color, style, shape, size, and brandishing shiny logos to reinforce its reputation as a status symbol—is less certain. This fact was made more stark for me when I realized that installing Android x86 on my Eee PC gave it more in common with modern computing platforms than my primary development laptop (Thinkpad x120e with Arch Linux and the minimal xmonad window manager). And I know I’m not the only developer who has, consciously or unconsciously, increased the specialization of my desktops/laptops while using smartphones/tablets for non-work related activities. Who really wants to pull out and boot up a laptop for light internet reading when you’ve got an instant-on smartphone or tablet within reach? I’m becoming more convinced that this is what the real “death” of the laptop looks like.

For software engineers, the role of laptops (and desktops even moreso) is slowly morphing into one similar to that of stenotypes. It’s arguable that this is inevitable: artists don’t use college-rule paper, chemists don’t conduct titrations in a coffee mug, and firefighters don’t roll up in a corolla to put out house fires. Professions evolve better and more efficient methodology, and when professionals outgrow the prevailing tools available to consumers, they develop new tools. This has already been well underway on the software side—you won’t find lawyers writing briefs with vim, after all.

If consumer computing devices become more recalcitrant for software engineers, it makes sense that a professional-grade tool should fill in the gap. The most natural candidate is the form factors we already have: laptops and desktops. But that certainly doesn’t limit innovation to current devices—a positive side effect of this “death” of laptops is that it provides a great opportunity to rethink what sort of device would benefit developers most, without being strictly constrained by a 1980s design that was intended for general computing. And this prospect may not be as far away as it might seem, given the recent rise of hardware startups.

The next time you pull out a laptop in a coffee shop, I don’t anticipate you’ll get the same quizzical looks you might receive if you brought a stenotype with you. But, like the stenotype, I do think that the proliferation of tablets and other more consumer-oriented devices will necessitate a professional class of devices that are less common and more specialized. And in the meantime, that might mean the stylishness of laptops will begin to wane. I’m okay with that.

[Incidentally, you can turn a conventional keyboard into a stenotype-like device with Plover, an open-source stenotype software package.]

Producing LaTeX from NumPy Arrays

Jarrell Waggoner (malloc47@gmail.com) — Tue, 29 Jan 2013 20:00:00 -0800

For my comprehensive exam, I needed to quickly convert some NumPy arrays into nice-looking LaTeX array elements. The TeX Stack Exchange site has a good answer for tabular environments, but wasn’t quite suited to the array environment. The usual answer here would be Pweave but, being short on time, I ended up rolling my own function instead:

def to_latex(a,label='A'):
    sys.stdout.write('\[ '
                     + label
                     + ' = \\left| \\begin{array}{' 
                     + ('c'*a.shape[1]) 
                     + '}\n' )
    for r in a:
        sys.stdout.write(str(r[0]))
        for c in r[1:]:
            sys.stdout.write(' & '+str(c))
        sys.stdout.write('\\\\\n')
    sys.stdout.write('\\end{array} \\right| \]\n')

Here’s an incomplete snippet of it in action, where I convolve an array t with four different filters, producing a latex formula for each result:

filters = (('A \\oplus H_1',h1)
           , ('A \\oplus H_2',h2)
           , ('A \\oplus H_3',h3)
           , ('A \\oplus H_4',h4))

for label,f in filters:
    t2 = scipy.signal.convolve(t,f,'same')
    to_latex(t2.astype('uint8'),label=label)

I’ll likely get around to expanding this into a full package sometime in the future, since there’s a lot that is hard coded (the \[ \] environment, stringification of the array, the fact that all columns are centered, etc.). A gist of the function is available here.

pythonbrew+opencv+debian

Jarrell Waggoner (malloc47@gmail.com) — Sat, 24 Nov 2012 15:00:00 -0800

There are a number of ways to go about building a modern development environment for scientific computing and computer vision in python. If you’re used to developing on bleeding-edge, however, the latest debian stable makes it a chore to get started with the latest and greatest. It ships with python2.6 instead of 2.7, and opencv is notoriously out of date in a number of distributions, debian included. I typically use Arch, but the server-class machines I have access to were running debian, so I had to bootstrap my setup into this environment.

Challenge accepted.

Thankfully, pythonbrew (or pythonz) comes to the rescue by making it easy to handle multiple pythons for a single account (without having to install them system-wide) as well as providing wrappers around virtualenv. However, not everything is rosy. The python you choose has to be built with shared libraries if you want to install opencv later:

pythonbrew install --configure="--enable-shared" 2.7.3

After this, you can bootstrap a virtualenv as usual

pythonbrew venv init
pythonbrew venv create debian
pythonbrew venv use debian

and install any requisite stuff you might need (minimum numpy/scipy)

pip install numpy
pip install scipy
pip install pymorph
pip install matplotlib
pip install distutils

Unfortunately, there’s no such pip package for opencv. Thankfully, the debian installation guide isn’t too far out of date, and many of the listed packages to apt-get are still relevant.

wget http://downloads.sourceforge.net/project/opencvlibrary/opencv-unix/2.4.3/OpenCV-2.4.3.tar.bz2
tar xjvf OpenCV-2.4.3.tar.bz2
cd OpenCV-2.4.3
mkdir {build,release}
cd release

At this point, we need to delve into where pythonbrew puts all its related files to configure opencv correctly. First, your installed python will be available in one of two places (here python 2.7.3 is used as an example):

~/.pythonbrew/venvs/Python-2.7.3/{venv-name}/bin/python
~/.pythonbrew/pythons/Python-2.7.3/bin/python

All virtualenvs based on a particular version of python will have a copy of that python binary for use in their own isolated environment. In addition, the virtualenv has an include directory that you should use, since all your additional packages installed into the virtualenv will place their headers in this directory:

~/.pythonbrew/venvs/Python-2.7.3/{venv-name}/include/python2.7

The hitch, however, is that the virtualenv does not have a copy/symlink of the shared library we specifically built when first compiling python using pythonbrew, unlike a typical native python install. This means that cmake’s approach to locate this library will fail. Thus we must point opencv to this

~/.pythonbrew/pythons/Python-2.7.3/lib/libpython2.7.so

for it to build corectly.

Speaking of cmake, there is a bug in the cmake included in debian that prevents it from building opencv correctly. I was lazy and simply grabbed a binary of the latest cmake,

wget http://www.cmake.org/files/v2.8/cmake-2.8.9-Linux-i386.tar.gz

which worked on my debian build, but it’s better to compile it if you plan to continue using it for more than a one-off build.

Finally, understanding opencv’s cmake flags is important for getting everything stitched together:

PYTHON_EXECUTABLE=~/.pythonbrew/venvs/Python-2.7.3/{venv-name}/bin/python
PYTHON_INCLUDE_DIR=~/.pythonbrew/venvs/Python-2.7.3/debian/include/python2.7
PYTHON_LIBRARY=~/.pythonbrew/pythons/Python-2.7.3/lib/libpython2.7.so

Additionally, if you find that numpy isn’t autodetected, you can specify

PYTHON_NUMPY_INCLUDE_DIR=~/.pythonbrew/venvs/Python-2.7.3/debian/lib/python2.7/site-packages/numpy/core/include

You can also specify your virtualenv path to install the python libraries

PYTHON_PACKAGES_PATH=~/.pythonbrew/venvs/Python-2.7.3/{venv-name}/lib/python2.7/site-packages

or just symlink/copy the resulting cv2.so and cv.py files there later.

Putting it all together, I used this command to generate the makefile which compiles correctly against pythonbrew’s python (where debian is my virtualenv name):

~/cmake-2.8.9-Linux-i386/bin/cmake \
-D CMAKE_INSTALL_PREFIX=../build \
-D BUILD_NEW_PYTHON_SUPPORT=ON \
-D BUILD_PYTHON_SUPPORT=ON \
-D BUILD_EXAMPLES=OFF \
-D PYTHON_EXECUTABLE=~/.pythonbrew/venvs/Python-2.7.3/debian/bin/python \
-D PYTHON_INCLUDE_DIR=~/.pythonbrew/venvs/Python-2.7.3/debian/include/python2.7 \
-D PYTHON_LIBRARY=~/.pythonbrew/pythons/Python-2.7.3/lib/libpython2.7.so \
-D PYTHON_NUMPY_INCLUDE_DIR=~/.pythonbrew/venvs/Python-2.7.3/debian/lib/python2.7/site-packages/numpy/core/include \
-D PYTHON_PACKAGES_PATH=~/.pythonbrew/venvs/Python-2.7.3/debian/lib/python2.7/site-packages \
../
make
make install

Depending on what you’re doing, there may be other tricks with LD_LIBRARY_PATH to make specific things work, but your pythonbrewed python should be primed to access opencv from here.

Anatomy of a Chrome Extension

Jarrell Waggoner (malloc47@gmail.com) — Sat, 15 Sep 2012 00:00:00 -0700

I launched nonpartisan.me a few weeks back, which exists primarily in the form of a Google Chrome extension (there’s a Firefox add-on too). Since I released it with all of the source, this makes it a great time to dissect the (very simple) code. As you will notice from the site and the small bit of press it picked up, nonpartisan.me has a very simple premise: filter out political keywords from the various newsfeeds (specifically Facebook, Twitter, and Google+).

This was my first attempt at a Chrome extension, and it’s surprisingly straightforward. All such extensions require a manifest.json, which looks like this for nonpartisan.me:

{
    "name"             : "nonpartisan.me",
    "version"          : "0.2.1",
    "manifest_version" : 2,
    "description"      : "Removes partisanship from your news feeds",
    "icons"            : { "16": "icon16.png",
                           "48": "icon48.png",
                          "128": "icon128.png" },
    "homepage_url"     : "http://nonpartisan.me",
    "page_action"      : {"default_icon" : "icon48.png",
                          "default_title": "nonpartisan'ed" },
    "permissions"      : ["tabs",
                          "http://www.facebook.com/",
                          "http://www.twitter.com/",
                          "http://plus.google.com/"],
    "options_page"     : "options.html",
    "content_scripts"  : [
    {
        "matches": ["*://*.facebook.com/*"],
        "js"     : ["jquery.js","common.js","fb.js","nonpartisan.js"],
        "run_at" : "document_end"
    },
    {
        "matches": ["*://twitter.com/*"],
        "js"     : ["jquery.js","common.js","tw.js","nonpartisan.js"],
        "run_at" : "document_end"
    },
    {
        "matches": ["*://plus.google.com/*"],
        "js"     : ["jquery.js","common.js","gp.js","nonpartisan.js"],
        "run_at" : "document_end"
    }],
    "background": {"scripts"   : ["common.js","background.js"],
                   "persistent": false }
}

The real meat here is content_scripts, which lists the javascript we wish to trigger after a page is loaded, greasemonkey-style. A particularly nice feature of content scripts are that they work in an isolated environment separate from any javascript that the page itself may include. Thus we can add jquery to the list of javascript that is run without fear of clashing with a page’s global namespace.

You can think of every element in the "js" array as a separate <script> tag in an HTML page, so the files are loaded in the given order, all into a single namespace. Rather clumsily, I chose to simply put a callback module (which is called plugin here) in the individual fb.js, tw.js, and gp.js files which is then used by the core component, nonpartisan.js, as a simple means of avoiding any hard-coded per-site values in the actual filtering code.

With this, and the pseudo-regex "matches" field that specifies which pages trigger the content script, we can run arbitrary code on websites we specify. For nonpartisan.me, the filtering code looks like this:

"use strict";
var nonpartisan = function(plugin) {

    function nonpartisan (watch,parent,keywords) {
        function kill (parent,removeList){
            $(parent).each(function () {
                var el = $(this);
                if(el.css('display') !== 'none') {
                    el.find('*').each(function () {
                        var toCheck = $(this).text().toLowerCase();
                        if(toCheck.length > 0 &&
                           (removeList.some(function (value) {
                               return (toCheck.search("\\b"+value.toLowerCase()+"\\b") >=0);
                           }))
                          ) {
                            el.css({'display':'none'});
                            return false;
                        }
                    });
                }
            });
        }

        if($(parent) && $(watch)) {
            var numChildren = $(parent).children().length;
            setInterval(function () {
                var newNumChildren = $(parent).children().length;
                if(numChildren !== newNumChildren) {
                    kill(parent,keywords);
                    numChildren = newNumChildren;
                }
            },
                        500);
            kill(parent,keywords);
        }
    }

    // get parameters from plugin and trigger nonpartisan() here...

}(plugin);

The first chunk–the kill function–works as advertised: given a parent element and a set of keywords, the function iterates over every child element and determines if any of the nested elements within (i.e. el.find('*')) contains any of the keywords. Instead of deleting DOM nodes, which may break the page’s own javascript (I discovered this the hard way), it’s easier to instead call el.css({'display':'none}); to simply hide unwanted elements. For efficiency, the forEach terminates as soon any any nested child returns a match, potentially saving a small amount of needless searching.

The second chunk starts a timer (if indeed the parent is even found on the current page) that checks if the number of children of the parent element has changed and, if so, re-triggers the filtering process to determine if there are any new children to be hidden. This helps handle AJAX-driven sites, like the “infinite scrolling” facebook newsfeed, which may mutate the DOM at any time. Both of these functions are wrapped up into another easy-to-call function inside of the high-level nonpartisan module.

And that really is all there is to a typical greasemonkey-like Chrome extension, but that’s certainly not the end of what a complete and helpful extension can provide. The trickier bit is persisting configuration options. The downside of sandboxing content scripts is that they exist in a transient execution context, meaning there’s no localStorage to persist program options. The details of the plumbing used to kick-off the process and handle options were omitted from the above snippet, so we’ll dig more into this now to illustrate how to handle persistent options.

Chrome provides a nice solution to the problem of not having localStorage available to content scripts by providing a background script which does have its own localStorage, which it can transmit to a content script via the chrome.extension.onMessage listener. We can then fill in the omitted component of the above snippet with:

chrome.extension.sendMessage({method: "config"}, function (response) {
    if(!response.sites[plugin.site]) return;
    var l = response.filter;
    if(l && l.length>0) {
        plugin.cb(l,nonpartisan);
    }
    // get default values from common.js
    else {
        l = [];
        for(var index in choices) {
            l = l.concat(choices[index]);
        }
        plugin.cb(l,nonpartisan);
    }
});

This sends a message, requesting "config" from the background.js script, which returns, among other things, the list of keywords we wish to filter. This list was saved in localStorage in background.js’s execution context. Recall that plugin is the module that specifies the particular settings for the page being filtered. Thus we pass along the list of words to filter and the nonpartisan() callback function to the plugin module, and it subsequently executes nonpartisan() on the appropriate elements on the DOM. The background.js file used in nonpartisan.me is a bit more involved, but it nonetheless essentially acts as a broker, converting Chrome’s internal message-passing API calls to localStorage requests.

Of course, there’s only so much utility to be gained from localStorage without supplying the user with the ability to configure the various options that may be saved in therein. This is done by a typical html page, specified by "options_page". Since there’s not much magic there–it’s just a plain html page with enough javascript to persist the settings–I will omit the gory details, which you can poke around yourself in the repository, if you’re so inclined.

So that’s an extension. Writing the above was literally a matter of minutes and some quality time with the Chrome API specifications. As is always the case (especially when I’m working outside of my area of expertise, say with making the amateurish logo), the real work is doing the little bits of spit-and-polish to handle the various configuration options, throwing together the webpage, creating the icons and promotional images for the Chrome Web Store, etc. But it’s still good to know that the Chrome team has made the extension-building process as simple and well documented as they have.

Poor Man's LDAP

Jarrell Waggoner (malloc47@gmail.com) — Sat, 11 Aug 2012 00:00:00 -0700

In addition to being a researcher and backend web developer, I’ve also worn the system administrator hat for a number of years. While the likes of LDAP, Active Directory, NIS, and their ilk can work quite well for managing medium-to-large networks, I’ve more often been tasked with managing small-scale (< 20 machines) heterogeneous Linux networks, where deploying LDAP with full Kerberos authentication would be overkill. Typical requirements I’ve encountered in small lab settings are simple user account and home folder sharing, and (relatively) similar package installations.

With this in mind, I did what probably every sysadmin in the same situation would do: scrape together a simple set of scripts to handle basic file synchronization for me. Specifically, I noticed two prevalent requirements among config files being synced:

machines and/or distros have a common header or footer that must be included (e.g., a list of system users in /etc/passwd), and
specific machines (e.g., servers) shouldn’t have some files synced with the rest of the machines (e.g., file shares might be different on a server).

Thus, Poor Man's LDAP was born.

While nothing more than a collection of scripts–no different than what many other sysadmins have implemented, in all likelihood–they will hopefully be of use for those who, like me, are graduate students or otherwise non-full-time sysadmins that don’t have time to do things the “right” way.

I’m dogfooding pmldap on my research lab’s network, where we (currently) have 5 Fedora machines (various versions between 10 and 16) and 5 Debian machines (all on stable). Since my recent patch, pmldap now supports groups, which are useful for running yum commands only on the Fedora machines and apt commands on only the Debian boxes. Files being synchronized include: fstab, group, hosts, hosts.allow, hosts.deny, passwd, shadow, and sudoers.

Also in the repo are a few convenience tools that I’ve found useful:

authorize-machine bootstraps a machine by setting up ssh keys
setup bootstraps config files from a remote machine so they can be merged with the desired additions
cmd runs an arbitrary command on all machines (or a particular group of machines)
useradd is a feature-incomplete reimplementation of the native useradd command that works on local passwd, shadow, and group files to add new users that can later be synchronized across the network

Since I hadn’t stumbled across something of this scope to fit the small-scale-network use case, I’m hopeful that pmldap will be of use to anyone in a similar situation.

You’ll find it on gitub here.

Git Dotfile Versioning Across Systems

Jarrell Waggoner (malloc47@gmail.com) — Sun, 29 Jul 2012 00:00:00 -0700

For users of unix-like operating systems, treating your dotfiles like real code and keeping them in a repository is a supremely good idea. While there are a myriad of ways to go about this, the typical (albeit destructive) way to do this is by symlinking files in the repository to the home folder:

#!/bin/bash
DEST=$HOME
FILES=$(git ls-files | grep -v .gitignore | grep -v ^$(basename $0)$)
for f in $FILES ; do
    [ -n "$(dirname $f)" \ 
      -a "$(dirname $f)" != "." \
      -a ! -d "$DEST/$(dirname $f)" ] \ 
    && mkdir -p $DEST/$(dirname $f)
    ln -sf $(pwd)/$f $DEST/$f
done

I specifically chose to have FILES populated using git ls-files to prevent any unversioned files from sneaking into the home folder, additionally filtering out both the .gitignore file, and the current script name (so it can be safely checked in as well). After this, we loop over the files, creating appropriate directories if they do not exist, effectively symlinking the entire repo to the home folder, clobbering any files that are already there (without asking!).

While most dotfiles won’t care what system they are on, certain scripts or settings may be machine-dependent. To accommodate this, I include a ~/.sys/`hostname`/ folder for every machine with system-specific files. Then, when symlinking, we favor files listed in the ~/.sys/`hostname`/ folder rather than the top-level files:

if [ -e ".sys/$(hostname)/$f" ] ; then
    ln -sf $(pwd)/.sys/$(hostname)/$f $DEST/$f
else
    ln -sf $(pwd)/$f $DEST/$f
fi

Thus, for example, given machine1 and machine2 and a repo in the ~/dotfiles directory with these files:

~/dotfiles/.gitconfig
~/dotfiles/.sys/machine2/.gitconfig

machine1 will get a symlink from

~/dotfiles/.gitconfig

to ~/.gitconfig, while machine2 will instead get a symlink from

~/dotfiles/.sys/machine2/.gitconfig

to ~/.gitconfig. This variant of the script doesn’t explicitly ignore the .sys folder itself so it will be added to the home folder as well. Which, as an aside, can be useful by including something like this

[ -d ~/.sys/`hostname`/bin ] && export PATH=~/.sys/`hostname`/bin:$PATH

in the .bashrc file such that specific scripts will be on the PATH for individual machines.

So the final script, with a bit of input checking, looks like this:

#!/bin/bash
set -e
EXPECTED_ARGS=1
if [ $# -lt $EXPECTED_ARGS ]
then
    echo "Usage: `basename $0` directory"
    echo "WILL clobber existing files without permission: Use at your own risk"
    exit 65 
fi

DEST=$1
FILES=$(git ls-files | grep -v .gitignore | grep -v ^$(basename $0)$)

for f in $FILES ; do
    echo $f
    if [ -n "$(dirname $f)" -a "$(dirname $f)" != "." -a ! -d "$DEST/$(dirname $f)" ] ; then
        mkdir -p $DEST/$(dirname $f)
    fi
	
    if [ -e ".sys/$(hostname)/$f" ] ; then
        ln -sf $(pwd)/.sys/$(hostname)/$f $DEST/$f
    else
        ln -sf $(pwd)/$f $DEST/$f
    fi
done

By making DEST a command-line parameter, a dry-run can be done by simply giving it an empty folder. There’s no issue doing this inside the repo’s working tree, as only checked-in files will be transferred to the target directory:

> mkdir tmp
> ./deploy tmp/

Doing this, the contents of the tmp/ directory can be verified with ls -al to see exactly what the script will do to your home folder. Once satisfied, it can be run again with

> ./deploy ~

to symlink all the files to the home folder proper.

Feel free to grab an up-to-date version of this script from my own dotfile repo here.

Beating Interview Problems to Death with Parallel Haskell

Jarrell Waggoner (malloc47@gmail.com) — Sun, 08 Jul 2012 00:00:00 -0700

Like anyone for whom graduation is becoming more immanent, I’ve been taking a look at the latest trends in the typical technology interview process. While many of the Fizz Buzzes being thrown around aren’t exactly exciting highlights of problem solving… well, you can always just beat them to death.

The Run Length Encoding algorithm is a nice, compact, and slightly real-world interview problem that has been making the rounds for years now. The basic idea being that “runs” of data, e.g. aaaabbbbbbb, are compressed into tuples, e.g. 4a7b, which may be a smaller representation if there is a large amount of adjacent repeated information. While real-world use cases for such a naïve compression scheme aren’t abundant, the algorithm is straightforward and can be implemented in a dozen lines or so in most languages. If you’ve got regexes or similar libraries at your disposal, you can manage even fewer lines. In Haskell’s case, one:

rleMap l = map (\e -> (head e, length e)) (group l)

which converts a string (or any arbitrary list of items) into a list of tuples, each of which has the character and its count. The function has type

rleMap :: (Eq a) => [a] -> [(a, Int)]

Simple and easy. But where’s the fun in calling it quits now? Let’s MapReduce our RLE algorithm to make it easier to parallelize and potentially Hadoop-friendly. We’ve already got our map function, so lets create a reduce:

rleReduce :: [(Char,Int)] -> [(Char,Int)] -> [(Char,Int)]
rleReduce [] [] = []
rleReduce a  [] = a
rleReduce [] b  = b
rleReduce a b
          | (fst $ last a ) == (fst $ head b) = 
                 init a ++  [(fst(last(a)),snd(last(a)) + snd(head(b)))] ++ tail b
          | otherwise = a ++ b

This is a less common component of RLE implementations (I haven’t spotted this particular bit of code anywhere else, so there’s likely room for improvement), but no less straightforward: simply join two RLE‘d lists together if their tail and head are not the same character; if they are, merge the head and tail tuple (updating the count) and combine the rest of the list as normal.

Now, it’s simply a matter of splitting the RLE target into pieces, maping over pieces, and reducing them back into a cohesive RLE-encoded document:

splitRLE n s = foldl rleReduce [] $ map rleMap $ chunkn n s

(chunkn is a simple hand-rolled routine that splits a string into n similar-sized pieces) As expected, splitting the list apart and recombining is needless overhead without parallelization:

# No splitting (rleMap s)
> ghc -O2 prle --make
> /usr/bin/time -f '%E' ./prle large.txt 1>/dev/null
0:02.68

# Nonparallel splitting (foldl rleReduce [] $ map rleMap $ chunkn n s)
> ghc -O2 prle --make
> /usr/bin/time -f '%E' ./prle large.txt 1>/dev/null
0:06.51

If we parallelize it using a simple parMap,

parallelRLE n s = foldl rleReduce [] $ (parMap rdeepseq) rleMap $ chunkn n s

we might expect some improvement:

# parallelRLE n s = foldl rleReduce [] $ (parMap rdeepseq) rleMap $ chunkn n s
> ghc -O2 prle --make -threaded -rtsopts

# Parallel map 1 core
> /usr/bin/time -f '%E' ./prle large.txt +RTS -N1 1>/dev/null
0:06.31

# Parallel map 2 cores 
> /usr/bin/time -f '%E' ./prle large.txt +RTS -N2 1>/dev/null
0:08.50

# Parallel map 4 cores
/usr/bin/time -f '%E' ./prle large.txt +RTS -N4 1>/dev/null
0:11.00

Unfortunately, the bookkeeping and garbage collection overwhelm the problem very quickly, never achieving better performance.

I’m running the above on a randomly-generated 12MB text file, and no amount of coaxing could make the parallelized version do any better. While we could have written our RLE algorithm in plain C without much more trouble and not have encountered such performance obstacles, one does not simply parallelize C by swapping in a parMap either (see also: this). Thus, we deep-dive into some Haskell optimization to get a performant version.

There is one particularly painful bottleneck: Haskell list monads are not ideal for handling bulk data of the sort we need because Haskell’s String type is represented as a [Char]. Since there’s no reason to use a boxed linked list just to scan over characters, we instead turn to Data.ByteString for reading the input, and to Data.Sequence to handle the RLE-encoded tuples. Data.Sequence specifically removes the large penalty when concatenating the lists together in the reduce step, as adding to either end of a Seq is a constant time operation. This is in contrast to [], where only adding an element to a list head is constant time. Importing these

import Data.ByteString.Lazy.Char8 as BL 
       (ByteString
       ,length
       , take
       , null
       , splitAt
       , group
       , head
       , pack
       , readFile)
import Data.Sequence as S

lets us rewrite our map

rleMap :: BL.ByteString -> Seq (Char,Int)
rleMap l = fromList $ P.zip (map BL.head c) (map (fromIntegral . BL.length) c)
       where
        c = BL.group $ l

and reduce

rleReduce :: Seq (Char,Int) -> Seq (Char,Int) -> Seq (Char,Int)
rleReduce a b = rleReduce' (viewr a) (viewl b)
             where
              rleReduce' EmptyR EmptyL = S.empty
              rleReduce' EmptyR _ = b
              rleReduce' _ EmptyL = a
              rleReduce' (rs :> r) (l :< ls)
                         | (fst r) == (fst l) = 
                           (rs |> (fst(r),snd(r) + snd(l))) >< ls
                         | otherwise = a >< b

Optionally, Data.Sequence can be expressed with ViewPatterns. Rewriting with these in mind allows the new reduce to resemble the old one fairly closely:

{-# LANGUAGE ViewPatterns #-}
rleReduce (viewr -> EmptyR) (viewl -> EmptyL) = S.empty
rleReduce (viewr -> EmptyR) b = b
rleReduce a (viewl -> EmptyL) = a
rleReduce a@(viewr -> (rs :> r)) b@(viewl -> (l :< ls))
           | (fst r) == (fst l) = 
             (rs |> (fst(r),snd(r) + snd(l))) >< ls
           | otherwise = a >< b

Now we finally define a new parallelRLE

parallelRLE :: Int -> BL.ByteString -> Seq (Char, Int)
parallelRLE n s = foldl rleReduce empty $ (parMap rseq) rleMap $ chunkn n s

and wrap it all up in a IO monad

main :: IO()
main = do
     [fileName] <- getArgs
     s <- (BL.readFile fileName)
     print (parallelRLE (numCapabilities) s)

With an improved algorithm and IO wrapper, it’s time for a more complete benchmark:

This was run on a 0.5GB file, as the smaller 12MB file used above runs so fast that is essentially instant. Between 2 and 5 processors, we get a nicely ramped speedup. After 5 processors, the bookkeeping overhead rears its ugly head again reversing the trend, and around 48 processors (my system maximum), the parallelization ends up running as slowly as the unparallelized version. This is certainly not the end of possible optimizations, but we have to stop sometime.

While I’m no Haskell expert, parallelization which costs no more than swapping in a parMap and paying homage to the Big O gods is a very compelling reason to hammer out any other toy interview questions with Haskell in the future.

Get the code here. Feedback welcome.

Google Completion in Emacs

Jarrell Waggoner (malloc47@gmail.com) — Sat, 19 May 2012 00:00:00 -0700

While many an emacs include dabbrev-expand for within-buffer completion, I’ve always wanted (purely for reasons of amusement) to take it further: completion via Google’s search suggestions. I was going to do this as a weekend project, but an ugly version was surprisingly simple.

Conveniently, curl is all we need to fetch the completions for a query string as JSON:

> echo -en $(curl -H "Accept: application/json" \
  "http://suggestqueries.google.com/complete/search?client=firefox&q=query")

["query",["query","query xiv","query letter","query_posts","query shark","query access","query tracker","query string","query letter sample","queryperformancecounter"]]

using a (very platform dependent) echo trick to convert the escaped unicode sequences to their proper characters.

With this, a quick hack in elisp is all that’s necessary to parse the results into emacs and insert it into the current buffer:

(defun google-request (query)
 (shell-command-to-string 
  (format
   "echo -en $(curl -H \"Accept: application/json\" \"http://suggestqueries.google.com/complete/search?client=firefox&q=%s\" 2>/dev/null)"
   query)))

(defun google-preprocess (query)
 (let ((l (split-string 
	   (apply 'string 
		  (removel 
		   '(?\" ?\[ ?\])
		   (string-to-list query)))
	   ",")))
  (if (> (length (car (cdr l))) 0)
    (remove (car l) (cdr l))
   nil)))
   
(defun google-complete ()
 (interactive)
 (end-of-thing 'word)
 (let ((s (thing-at-point 'word)))
  (when s
   (let ((q (google-preprocess (google-request s))))
    (when q
     (insert (substring
	       (car q)
	       (length s))))))))
		   
(defun removel (el l)
 (cond (el (removel (cdr el) (remove (car el) l)))
       (t l)))

Since it went more swiftly than anticipated, I generalized the code to parsing any delimited shell output and wrapped it in a minor mode with some key bindings and customize variables. Right now, I’m uncreatively calling it shell-parse.el.

After activating shell-parse-mode, it has support for scrolling through the list of completions forwards (shell-parse-complete) and backwards (shell-parse-complete-backwards) with the C-Tab and C-Shift-Tab keys, respectively. Using M-x customize-mode <Enter> shell-parse-mode, you can swap out the curl command with any shell snippet that will kick back completions, and change the delimiter as well.

You can grab shell-parse.el on github. Simply load-file the shell-parse.el script in .emacs and it should be ready to go.

It has a few todos scattered through it yet, and is not very idiomatic emacs or portable, but that’s what github’s issue tracker is for, after all.

Preprocessors and Graphviz

Jarrell Waggoner (malloc47@gmail.com) — Sat, 07 Apr 2012 00:00:00 -0700

Graphviz is a useful toolset for describing and rendering graphs. One of the features the graphviz language doesn’t have, however, is a C-like preprocessor to #include files. Which, granted, isn’t a particularly common use case when building graphs, but one I found desirable when dealing with a large number of graphs with a shared subset of nodes, differing by how they are connected.

Initially, I grafted together an unwieldy script that used an ugly grep+sed combination to grab the line number and substitute the included file contents: essentially a poor man’s preprocessor. Thankfully, to the rescue was a particularly useful gist (initially illustrated with JavaScript) I serendipitously found on Reddit. The key call being this:

cpp -P -undef -Wundef -std=c99 -nostdinc -Wtrigraphs \
    -fdollars-in-identifiers -C < input.dot > output.dot

In your input .dot file, standard C include syntax

#include "common.doth"

will work as expected, despite the fact that it is a completely different language.

This solution is highly verbose if you don’t drop it in a build chain and forget about it. Which is straightforward using a standard makefile:

SOURCES = $(wildcard *.dot)
OBJECTS = $(SOURCES:.dot=.doto)
IMAGES  = $(SOURCES:.dot=.png)

all: $(IMAGES)

%.png: %.doto
	dot -Tpng $< > $@

%.doto: %.dot
	cpp -P -undef -Wundef -std=c99 -nostdinc -Wtrigraphs \
		-fdollars-in-identifiers -C < $< | \
		gvpr -c 'N[$$.degree==0]{delete(NULL,$$)}' > $@

clean:
	-rm $(IMAGES) $(OBJECTS) *~

In the above example, I introduced an intermediate doto file (analogous to an object file) and doth (header file) to recreate a C-like build process.

Another ingredient above is the piped invocation of gvpr, which removes nodes of degree 1 (so that included nodes that are not attached to anything in the current file will be ignored). Remember that in makefiles, the $ must be escaped by using $$. Unfortunately, the delete function in gvpr is broken in a number of Debian-based distros (at least), but the latest version works bug-free in Arch.

So, given a nodes.doth file with these contents:

    node1 [label = "1"];
    node2 [label = "2"];
    node3 [label = "3"];

and a graph.dot file as such:

graph g {
    #include "nodes.doth"

    node1 -- node2;
    node2 -- node3;
    node3 -- node1;
}

the makefile will use cpp to generate the following intermediate file,

graph g {
    node1 [label = "1"];
    node2 [label = "2"];
    node3 [label = "3"];
    node1 -- node2;
    node2 -- node3;
    node3 -- node1;
}

which will then be compiled by graphviz’s dot command into an image. While obviously not necessary with this toy example, scaling up to more nodes shared by multiple graphs is much more pleasant when the nodes don’t have to be duplicated in each graph.

Very little of this is exclusive to graphviz, and is reasonable to extrapolate to other problems fairly easily. And, since this literally uses the C preprocessor to do the job, there’s many more tricks to be explored.

Quite helpful to have on hand when the need arises, and a testament to the quality of cpp that it can be used for arbitrary metaprogramming in other languages.

QR Codes En Masse

Jarrell Waggoner (malloc47@gmail.com) — Sat, 24 Mar 2012 23:00:00 -0700

For the upcoming POSSCON here in Columbia, we had need of QR codes for the brochure. Lots of them. And while there are some great online resources, I wanted to create QR codes in batch.

Of course, the online services could be batch processed with a dose of curl magic, but there is a more UNIX way: libqrencode.

Creating a discrete QR code image is straightforward with the qrencode command:

qrencode -o output.png -s 50 "http://www.malloc47.com"

The -s parameter controls the size of the QR “dots” and therefore the output resolution.

This is great if you don’t need a vectorized format, but for print-quality work where you may not know the eventual DPI, having vectorized output (e.g. eps, and svg) is preferable.

Again, the vast repositories of libre software come to the rescue here: potrace is designed for exactly this. Annoyingly, it only handles bitmap (or the easy-to-generate pnm) format, so a little imagemagick will take care of this:

convert output.png output.bmp

Now we can convert to a vector format easily:

potrace -e -o output.eps output.bmp # -e is for EPS
potrace -s -o output.svg output.bmp # -s is for SVG

We can wrap it all up into a nice (bash) script:

#!/bin/bash
set -e
qrencode -o $1.png -s 50 "$2"
convert $1.png $1.bmp
potrace -e -o $1.eps $1.bmp
rm $1.png $1.bmp

which takes a file name prefix and a string to be encoded. To generate a large set of QR codes with this script, simply create a file with file prefix-URL(or whatever data is to be encoded) pairs, each on a separate line,

google http://www.google.com
amazon http://www.amazon.com
reddit http://www.reddit.com
....

and then loop over this file, line-by-line:

while read line ; do ./create-qr-code.sh $line ; done < list.text

which conveniently gives us google.eps, amazon.eps, and reddit.eps files for their respective URLs.

If there is uncertainty that your URLs are good (i.e. don’t kick back 404s), then you can augment the above script with this nice curl snippet (courtesy of this post on SO):

#!/bin/bash
set -e
curl -s --head $2 | head -n 1 | grep "HTTP/1.[01] [23].." > /dev/null
if [ $? -eq 0 ] ; then
    qrencode -o $1.png -s 50 "$2"
    convert $1.png $1.bmp
    potrace -e -o $1.eps $1.bmp
    rm $1.png $1.bmp
else
    echo "URL error: $2" 1>&2
fi

This will let you know which URLs don’t come back with clean headers so you can give them further attention. It won’t capture everything that might go wrong, but it does give you a programmatic way to verify that all is well.

Incidentally, all the tools used here can be installed on Arch with

pacman -S qrencode potrace imagemagick curl

Not exactly the prettiest shell glue, but it certainly beats slowly copy & pasting in and out of a browser.

Flirting with the Front End

Jarrell Waggoner (malloc47@gmail.com) — Tue, 20 Mar 2012 00:00:00 -0700

Every few years, when I’m not teaching introductory web programming, I revisit front end development, oftentimes in the form of retooling my site. Last time, it was a Wordpress-driven theme with cobbled-together JavaScript snippets for random bits of functionality:

Serviceable, at least.

Before this, I used a generic Wordpress theme, the specifics of which I don’t recall. Rolling back all the way to the mid-90s, I had a fortunecity site, which was–as typical of sites in the 90s–equal parts bland and garish:

Yes, it had a Christmas theme for the title page. And yes, the header, navigation bar, and footer (on individual pages) are all java applets. Not exactly, a usability panacea.

And now, I’ve transitioned to Jekyll, for a few reasons:

It’s hard to get faster than static pages for serving content.
Github can handle more traffic than the shared hosting I was using previously.
A Jekyll deploy on github can’t use external plugins. Which is, by most accounts, a downside, but it forces me to find front-end solutions for what I want rather than turning to the back end for everything.
I wanted to build everything from scratch. The limited Liquid DSL used by Jekyll is leaner than full-blown PHP, and more satisfying for building from the ground-up (all my Wordpress themes started from–at minimum–a skeleton theme, just to cover the essentials needed by Wordpress).
Having everything in a git repo is both satisfying for my current work flow and avoids the pain of database backups.

So here it is. I avoided jQuery (convenient as it is) to keep things lean and loading quickly, and rampantly bludgeoned the site with HTML5/CSS3 without much regard for backwards compatibility. To further optimize queries, I used Liquid includes to aggregate all the js and css into single files. For JavaScript:

---
---
(function () {
    "use strict";
{% include cookies.js %}
{% include mini-clock.js %}
{% include check-time.js %}
{% include event-handler.js %}
}());

you can wrap everything with "use strict" to get some extra exception goodness. Doing this may cause JSLint to complain about indentation issues, and if you don’t add event handlers with JavaScript (e.g. you use the onclick or onload events in your HTML tags), you may run into scope issues as well. All of this together provided a nearly 20-point speed bump on Google page speed.

I opted for a dual-themed site, determined by the time of day. The clock drawn in the HTML5 Canvas element in the upper-left shows when the transition will occur, or you can override it with the icon next to the clock.

All in all, a good transition so far.

Jedi Knight: Dark Forces II -- A coding retrospective

Jarrell Waggoner (malloc47@gmail.com) — Tue, 28 Feb 2012 00:00:00 -0800

Years ago, before I got my hands on a real compiler, I satisfied my coding compulsion by modding Jedi Knight: Dark Forces II (which, yes, has the subtitle backwards from normal). As detailed elsewhere, Jedi Knight (abbreviated JK) is one of the earlier examples of a game that used a simple scripting language to handle interaction with the game engine. COG script, as it was called was, in hindsight, designed as much to be easy to compile as it was to write. Of course, the attractive thing to any 13-year-old who desperately wants to code in a “real” language was COG’s (superficial) similarity to C.

The structure of a COG script is straightforward: it starts with a symbols section, where all variables are declared, followed by a code section broken into various messages (denoted by goto-like labels, which must be declared as variables in the symbol section to be valid) that represent events that could be triggered in-game. Aside from the usual expressions and flow control, there are library functions that form an elementary API for the game engine itself. Variables are typed, and have one of two scopes. The “local” variables are bound within the scope of the script itself. All non-local variables must be bound to external entities in the game environment–think of them as free parameters that must be specified outside the script. Variable types include a few atomic types (e.g., int, flex) and some game-specific entities (e.g., things, surfaces), and event types (e.g., messages). Have a look:

symbols

surface downsect
message activated
thing ghosttel1
thing ghosttel2
thing player local
surface upsuf1
end

code

activated:
if (GetSenderRef() == upsuf1)
{
    player = GetLocalPlayerThing();
    TeleportThing(player, ghosttel1);
    return;
}

if (GetSenderRef() == downsect)
{
    player = GetLocalPlayerThing();
    TeleportThing(player, ghosttel2);
    return;
}
return;

end

The above code works as you would expect–it simulates a “ladder” by simply teleporting the player to the top of the ladder if they “activate” (by pressing the spacebar while standing near) a surface at the bottom, or teleports them back down if they activate the ladder surface at the top.

With no reference whatsoever–beyond the COGs included with the game itself–it took me a while to get used to writing COG. But given that it had one of the more fun compilers around (though billed as a scripting language, COG was actually compiled to a stack-based language at load-time), I had little trouble finding motivation to toy with it.

Aside from COG, JK had a variety of other specialized file types that could be (relatively) easily created or modified: .3do for 3D models, .mat and .bm for images (bitmaps, essentially), .jk for level meta information, .pup (puppet) files that link keyframed animations to specific pose states for a character, .key files which are the keyframed animations for the characters, .snd files that link sound to a character, .ai files that expose variables for how the characters react (simple artificial intelligence), .gob files which are essentially .zip files for storing all the various resources in a level, and many many more file types I’m probably forgetting.

But it was always the COGs that were most interesting to me, despite the fact that I wrote some ugly and embarrassing code in it. Really embarrassing.

And I won’t even get started on my very unrefined sense of aesthetic design in the level I ended up building. Yes, I built an entire level, mainly to facilitate being able to write fun little scripts in COG. Which is not nearly as involved as it sounds thanks to JED, a community-supported level editor. I built a very basic plotline with a number of cutscenes (9, I claim in the readme); a weapon mod that added a phaser to your arsenal (you know, the Star Trek sort–I never claimed any of this made sense except to my 13-year-old self); two new types of enemies, as I never thought the original lineup was difficult enough; a few secret areas, including one with a rather famous character; a zoo (again, never said anything about making sense); a few RPG-style quests; and committed some very horrible texture misuse, a fact that the contemporary reviews will back up.

I did learn some rather important software engineering lessons along the way. I recently ran across my issue-tracking system: a notepad with 50+ pages of bugs and fixes that I did on the level for the year or so it was in development. Everything from enemies falling through floors, cutscenes being mistimed, and impossible-to-win scenarios, were among the issues I recorded. I simply couldn’t keep track of the number of issues I would notice on every playthrough without writing them down.

On my nostalgic adventure, I decided to actually get Jedi Knight up and running again. I think my PC was perfectly adequate to run the game, according to its own system requirements analyzer:

Though it was incorrect about the "Windows 95" part.

Thankfully, a kind soul rewrote the 16-bit installer (here) so, if you happen to have the Jedi Knight CDs on hand, you can actually install it on a 64-bit Windows (virtual) machine.

Ah, nostalgia.

Getting any new level up and running is typically as simple as placing the .gob file in the EPISODES folder. So, of course, I loaded up my old creation. I definitely had a lot of fun as a kid.

Completely forgot about the in-game menu system (so you could bypass the rather dry "Story" cutscene I included, among other things).

A planetarium. On a star destroyer, for some reason.

The aforementioned zoo.

Oddly, I also included a kitchen. Complete with an oven you have to crawl into to complete a mission objective.

The obligatory "dress up as the enemy to get through a checkpoint" plot point.

A sewer at the bottom of a detention center. Even star destroyers have to take care of their waste somehow.

Grand Admiral Thrawn himself, wielding a phaser. And fairly deadly at that--I died a dozen times or so before beating him.

Barney makes an appearance. And yes, you can obliterate him.

While my handiwork hasn’t aged well (and neither has JK, for that matter), I’ve placed the entire monstrosity on my github for posterity’s sake: https://github.com/malloc47/return-of-thrawn.

Reflecting back, COG certainly wasn’t the most stretching or important language I learned (by a long-shot), nor was building my level the most disciplined software engineering project I’ve undertaken, but it was one thing: it was enthralling enough to keep a kid hooked on coding and building cool things. Before the era of web apps and mobile apps, or code academies and khan academies, or Scratch and Alice, I feel rather lucky to have stumbled on something that was simultaneously fun and empowering. JK wasn’t just a game. It was an ecosystem. It was an IDE. And it was fun.

Import Links into Google Plus as +1s

Jarrell Waggoner (malloc47@gmail.com) — Sat, 18 Feb 2012 00:00:00 -0800

I’ve been accumulating helpful and interesting articles for a number of years now. At first, they existed solely as starred articles in Google Reader. Eventually, I migrated to Delicious, and finally to diigo. While diigo has served me well, I finally decided to consolidate services and begin using Google’s +1 feature, as it is fairly ubiquitous and is associated with an account I am already logged into (and appears nicely on my Google+ profile).

While Google’s Takeout is very useful on the exporting front (a factor I consciously consider before migrating to any system), I’ve never run across a good way to import a list of links as +1s on my Google+ profile. So here’s what I came up with last night:

#!/bin/sh
echo "<html><head>" > page.html
echo "<script type=\"text/javascript\" src=\"https://apis.google.com/js/plusone.js\"></script>" >> page.html
echo "</head><body>" >> page.html
grep -o http[^\"\)\']* $1 | xargs -I{} echo "<g:plusone href=\"{}\"></g:plusone>" >> page.html
echo "</body></html>" >> page.html

Since diigo will happily let you export in a variety of formats, I chose the csv file. My goal was simply to create a webpage with the links and the +1 button next to each. The process of clicking the buttons themselves could be automated, but I decided to manually click on the buttons, since I wanted to vet the links I transferred. The very simple (read: not iron-clad) regex (http[^\"\)\']*) is used to pull out the links. It looks for an instance of a string “http” and then continues grabbing characters until it hits a quote. It should work in a number of contexts, aside from just CSV files. xargs kindly loops over all of these addresses and outputs each in HTML form. The rest of the script just adds the usual HTML boilerplate, as well as the +1 script necessary for the buttons to work.

One more wrinkle: the latest crop of browsers sandbox what javascript is allowed to do to the local filesystem (and rightfully so), so you will need to upload the generated page to a non-local path (or simply copy and paste it into a website that will let you edit html live). Once you do, just click on all the +1 buttons everywhere, and your links will be +1’d accordingly.

Have a look.

Man Page Readability

Jarrell Waggoner (malloc47@gmail.com) — Sat, 11 Feb 2012 00:00:00 -0800

Man pages are one of the staples of a healthy *nix diet, but having grown up with them, it didn’t occur to me until recently to wonder how readable they really are. The de facto standard for readability has–for better or worse–converged to the Flesch–Kincaid test, which (a particular variant) ranks readability as a “grade level,” roughly corresponding to an American school grade. Getting a readability score for a manpage is as simple as piping our man page to the GNU style program (not installed on many distros, I discovered, and typically available in the “diction” package).

> man /usr/share/man/man1/git.1.gz | style
readability grades:
        Kincaid: 7.0
        ARI: 4.4
        Coleman-Liau: 4.0
        ...

Notice that man can read in the (tarzipped) man source file (typically located in /usr/share/man/man? folders) rather than having to type the executable name.

Since we’re only concerned with the Kincaid score, we can apply a smattering of grep+awk to extract it.

man /usr/share/man/man1/git.1.gz | style | grep Kincaid | awk '{print $2'}

And finally, looping over all the installed man pages (+sed to trim out short sentences, headers, etc.) gives us one big file, from which we can get readability statistics.

for i in `ls -d -1 /usr/share/man/man?/*` ; do echo -n "$i " ; man $i | tr '\n' ' ' | sed 's/\./\.\n/g' | sed -e 's/^[ \t]*//' | sed '/.\{3\}/!d' | grep Kincaid | awk '{print $2}' ; done > ~/flesch-kincaid

More awk magic will give us an average and standard deviation.

> awk '{avg+=$2} END {print avg/NR}' ~/flesch-kincaid
9.08134
> awk '{sum+=$2; sumsq+=$2*$2} END {print sqrt(sumsq/NR - (sum/NR)**2)}' ~/flesch-kincaid
10.3857

Perhaps surprisingly, the average readability of the man pages on my machine is below the college level. More unexpectedly, the standard deviation is very high, indicating that there’s a wide range of readability from one man page to another. The page with the most absurdly large readability score

grep `awk '$2>m{m=$2}END{print m}' ~/flesch-kincaid` ~/flesch-kincaid

consist almost entirely of code and API documentation.

Of course, this does not take into account a myriad of confounding factors: some non-English language pages crept into the list, which kick out bogus scores by the Flesch–Kincaid metric, man pages have non-standard formatting (e.g. command switches) which aren’t considered in the metric, etc. But knowing the general (or at least average) education level required to comprehend man pages is worth considering as more mainstream distributions bring with them an influx of younger and less experienced users.

vim -> emacs

Jarrell Waggoner (malloc47@gmail.com) — Wed, 18 May 2011 00:00:00 -0700

After having grown up in the terminal with vim at my side, I’ve been transitioning to emacs. There’s no shortage of guides for making the leap, but there’s a number of features in vim that operate very differently in emacs, or do not map cleanly to a single feature. There’s nothing like not knowing how to get that one feature you rely on when making the vim->emacs leap to make you want to run back to familiar territory. Here’s a few of the discoveries I’ve made along the way.

dd -> C-Shift-Backspace

C-k is more often cited as the analog for dd (and it is indeed more flexible in many situations), but as far as raw functionality is concerned, sometimes you just want the current line to go away, regardless of where the cursor is. Unfortunately, the** C-Shift-Backspace** keybinding may not be triggered when you’re running emacs in a screen or xterm (emacs -nw), but for the majority of standard emacs use, it’s a feature-compatible substitute.

% -> C-M-n, C-M-p

Traversing pairs of grouping characters (e.g. (), [], {}, etc.) with vim’s % key is invaluable, but emacs not only replicates this feature, but adds a few new bits of functionality too. In general C-n and C-p will traverse to the next or previous line, respectively, but C-M-n and C-M-p will traverse forward or backward over the current list. Since emacs has a major mode related to most languages you could find yourself working in (and if not, it’s not difficult to find or make one) that will do a rudimentary parse of the code into proper tokens, C-M-n and C-M-p can intelligently jump to the beginning or end of parenthetical structures, words, or blocks with equal effectiveness. As a bonus, you also get C-M-d (if your window manager hasn’t stolen that keybinding) and C-M-u that go down or up (respectively) in the current structure—i.e., C-M-d will place the cursor inside the parenthetical statement, ready to iterate over the items inside the parentheses with C-M-n, while C-M-u will place the cursor at the beginning of the parenthesis or block structure enclosing the cursor. Check the major mode you are in for even more context-sensitive commands (e.g., M-a and M-e to go the beginning or ending of a statement in the C++ major mode). Another instance where it seems like emacs adds overhead to the preciously minimalistic vim keybindings, but having two keys instead of one adds immeasurable navigational flexibility.

f, F, t, T -> C-s, C-r

This may not be immediately intuitive, but the best mapping for vim’s nifty “forward (or backward) to character” feature is emacs’s standard search function. While, yes, the concession is that it requires three keystrokes (you must press enter after searching in emacs to position the cursor) instead of one, you get the same functionality, with the added benefit that you can search more than one character easily, and you don’t have to retype the character you want to jump to, as you would in vim.

[n]gg, :[n] -> M-g g

Though emacs integrates so well into most REPLs that you don’t often need to jump to specific line numbers manually, it’s still trivial to hop to specific line numbers with ease. Again, you’re sacrificing more keystrokes but, as with everything in emacs, you can remap anything you use often. Alternately, consider using the faster C-[n] M-g g variant to prepend the line number instead of having to specify it (and hit enter) in the mini-buffer.

"[register]p -> C-y M-y

The kill ring in emacs isn’t remotely complex, so it’s hardly worth mentioning except that it varies greatly from vim’s model of using registers to store yanked text. Instead of having to specify a register, as you do in vim, you simply paste whatever text happens to be in the ring into your document/code with C-y, then loop through the various yanks with M-y. Coming from a vim world, where every yank requires a register that contains only one snippet of text, the value of having yanks (or kills) you can scroll through (in the exact context you wish to paste them) is quite clear (and can actually result in fewer keystrokes for complicated yanks). As a bonus, only having to use C-w or M-w to yank text requires fewer keystrokes than having to use “[reg]y in vim too.

q[register] ... q -> C-x ( ... C-x ) M-x name-last-kbd-macro [name]

The only surface difference between vim and emacs for keyboard macros is that vim assigns keyboard macros to a register by default, but emacs requires M-x name-last-kbd-macro to cache more than one macro at a time. The difference becomes more stark once you explore multiple macros, as you can apply a named keyboard macro by using M-x [name], since emacs saves the keyboard macro as a standard emacs command. As a bonus you can use the M-x insert-kbd-macro to save the macro to your .emacs file for future use.

u, C-r -> C-/

This is another emacs feature that’s very simple to understand, but where vim logic may trip you up. Most vim->emacs guides will clue you in that vim’s u maps to emacs C-/ (C-_ or C-x u, but those are more of a hassle), which is indeed correct. But the underlying “redo” logic is a bit different. Instead of vim’s separate C-r redo command, emacs lets you “undo your undos” (again, think yank ring). So after a series of consecutive C-/s, all you need do is interrupt the sequence with a command that does not produce any undo history (I typically use a movement command like C-f), and then use C-/ again, which will then, essentially, have become a “redo” command. This will become second nature quickly, but it’s a very different model from the inferior method used by vim (and a large number of other editors).

: -> M-:, M-x

Only thing worth mentioning here is the difference between executing direct elisp code (M-:) and emacs commands (M-x), which is not a distinction that vim has to make.

:tabn, gf -> C-x b

Vim actually does allow for “hidden” buffers, so it can operate in a very emacsish way, if desired, but it a rarely-used feature; one that emacs adopts by default. Emacs doesn’t have “tabs” to switch between, per se, but the way it handles buffers is vastly more powerful than any tab or hidden buffer in vim, minus the “tab” aesthetic (and ido-mode or C-x C-b is more than capable of providing a list if that is what is needed).

Screen Line Movement -> Logical Line Movement

Emacs’s C-n and C-p move over screen lines (the line breaks you see on the screen) by default, while vim’s j,k moves over logical lines (line breaks that are actually in the file). While vim’s logical line movement can cause problems with long lines (you have to use gj and gk to switch over to screen lines), you can switch emacs to logical line movement, if you prefer it, with

(setp line-move-visual nil)

which can be easily entered with M-: in emacs to try it out, or added to your .emacs file to save the setting.

Once Upon an After

Jarrell Waggoner (malloc47@gmail.com) — Wed, 15 Dec 2010 00:00:00 -0800

“Well…” said I.

“Yes?” said it.

“I don’t understand what you mean by ‘happy endings’ at all.”

“Every single of your stories is one.” It seemed insistent. It couldn’t have meant what it sounded like it did; its slightly strange phrasing being (I assumed) an artifact of adapting to our complex language.

“Surely you don’t believe all of our stories have a happy ending?” I added, trying to clarify things.

“Surely I do,” was its reply.

“Why?”

“Because they end, happily.”

I knew I wasn’t going to get anywhere without examples. “But what of stories that leave all the characters expired or cadaverous?”

“But they end.” Its reply so very matter-of-fact.

“Well, yes. But what of stories of lost love and unhappiness?”

“They still end.”

The monotonous reply, coupled with its already colorless voice, was starting to grate me wrongly. “But what still of stories where the miscreant escapes justice?”

“They come to an end, still.”

Now it was just trying to get under my skin. “You can’t be serious! You can’t call a villain triumphant anything but a travesty; a champion dead anything but a loss!”

It raised one of its arms. “Certainly the events are allowed less than happiness, but the story itself is still happy.”

I didn’t know what it was getting at. “What are you getting at?”

“Perhaps I misspoke,” it said, clearly recoiling from my words (it understood verbal frustration so much more clearly than nonverbal frustration). “What I mean is that the nature of your stories is happy, by virtue of the fact that they end.”

“Ah, now we are getting somewhere,” I uttered, relieved. “I thought you meant that our stories end in a happy way, while you actually meant that our stories end, which is happy… right?”

“Yes, that would be it, if I understand the nuances of your language correctly.” It was considerably calmer now.

“But… why?”

“Because they happily end.”

“No, no, no… I mean, ummm…” I sputtered, trying to get into words what I was asking of it. “Why is the nature of an ending story happy?”

“That’s a bit difficult to explain to beings subject to death–”

“Mortal,” I chimed in, knowing it appreciated my attempts at improving its vocabulary.

“Yes, mortal beings. For you, endings are natural, expected, even necessary. We are not mortal, you see. Endings aren’t normal for us. They’re… redundant.”

Its last word choice confused me, but I decided to focus my query on what was really bothering me. “Ah, I see. I understand the unfamiliarity, I suppose. But what makes them happy?”

It thought for an uncomfortable minute, trying to get its thoughts into its own words. And then from its own words into my words. “You see, the lot of us have lived for a very long time. And we have a surprisingly limited vocabulary–only a handful of what you would call ‘words’–and a very rigid set of actions we are capable of performing.” It paused, wondering if I had taken in the significance of the limitations it had just explained. After a moment, my nodding acknowledgement was finally satisfactory, and it continued. “After but an armful–”

“Handful.”

“Yes, handful, thank you. After a handful of your lifetimes–about four hundred billion–we have, each of us individually, lived out all possible combinations of any reasonably finite series of actions that can be taken, and words that can accompany them.”

“Well…” said I.

“We’ve lived them all,” it continued.

This I did require a moment to soak in. It was plenty willing to oblige my contemplative moment. “Wait,” I exclaimed, “you mean that if any of your stories were to have an ending, then you’d have already lived that story?”

“Yes, that is the very reason our stories have no endings. Nor can they.”

“Why couldn’t you have a story that went on just a bit longer than all the stories you’ve experienced, and then end it?” I puzzled.

It looked puzzled at my puzzle and then, quite rightly, replied, “by the time we had heard the recount of a story of such length, we’d have already experienced it ourselves. Nor could the crafting of such a tale be completed before we had fully experienced its contents.”

“Oh, bother,” I reacted. Clearly, I had not fully grasped this notion. I probed further: “So, tell me of the stories you do have.”

“We may have experienced every possible permutation of events, but we still exist in time, and the order in which we experience these events is different for every one of us.” It paused again. “Think of it thus: no matter which book you pick up, no matter the film you view, no matter what event you might hear recounted, everything found therein you have already experienced. But the sequence of when these things were experienced varies from one to another. That’s how we tell each other apart–not by appearance, since we all share the same; not aurally or tactilely, but simply by distinction of the sequence of discrete stories we’ve experienced.”

“Aha! So you need only find an ordering of stories that has never been experienced before to create a new, original tale?”

“That’s hardly practical, since there is no telling if someone else has already experienced your crafted ordering (and yours isn’t the only culture with lawyers). Neither is creating such a lengthy tale practical, nor partaking in the recounting of such a thing, as I’ve already said.”

“But the lack of novelty doesn’t make a story valueless (lawyers aside),” I posited, “many of our stories have elements we’re all familiar with: birth, growing up, death–themes that don’t grow tired in and of themselves.”

“What if each and every one of your stories consisted of ‘Once upon a time.’ That, and only that. Something so familiar and universal that its existence alone without anything accompanying it is so axiomatical as to be meaningless. That is what a finite story is to us. You, being finite, only understand the example if I take an incomplete element of a story, but to us, it’s entire stories that seem this tired and common.”

“I don’t suppose your vocabulary allows for narrative devices that makes the story uniquely inexperiential?”

“You would be right, if I understand your meaning.”

It made its point clearly. “But what are your stories, then? How can they be enough?”

“Simple,” it replied. “Our stories are the lives of those around us. That is enough for us”

Something was still missing, but I think I understood.

of soundtracks and extraterrestrials

Jarrell Waggoner (malloc47@gmail.com) — Mon, 18 Jan 2010 00:00:00 -0800

It has always been the aliens.

Musically speaking, anyway. Fantasy has always had music, and music fantasy. And occasionally aliens.

Whether our lovable characters are humming ditties across the foliage of Middle-earth or our intrepid explorers attempt to decipher the cacophonous intonations of extraterrestrial visitors, music has been accompanying the adventure since before adventures were being put to words.

Maybe we weren’t singing along, but we were being sung to. Maybe we weren’t tapping to the beat, but the beat was there to be tapped.

So where did it go?

It seems that every major fantasy franchise eventually earns a signature tune once it works its way up the ranks and onto the silver screen. From the mysterious celesta stylings of John William’s ‘Hedwig’s Theme’ from Harry Potter to Patrick Doyle’s bombastic ‘Eragon,’ the bigger the budget, the brassier the bravura. Apparently, without the written word, we still need someone to tell us when we should be fearful of the sprawling armies before us or marvelling at our wizard’s mysterious magics.

It might as well be the aliens.

Because who else is going to transport a 70-piece orchestra to the far-off lands of Narnia purely for our listening pleasure? Before, our characters were responsible for the words, our faithful authors responsible for giving them the right meter and rhyme. Our imaginations brought the orchestra. Or the fiddle. When our characters chanted their lines, we didn’t get dramatic montages to keep our attention.

Ah, progress. There are many upsides to having ubiquitous fantasy soundtracks. Fantasy is bigger than life, and some of us like our music imaginative. However, it can also be detached. Trying to sing when the visuals don’t. Or make us laugh when the characters aren’t funny.

But at its best, it’s the Greek chorus of the modern age, without bludgeoning us with every internal dalliance of every character, while preventing us from overlooking subtle emotions or expansive vistas. Like a Greek chorus, our soundtracks even get to poke their head in and be heard by our characters from time to time. But mostly, the aliens play well out of earshot of the action. We wouldn’t want those war drums distracting the archers, after all.

Music just fits better with fantasy. Really, when you’re watching maniacal creatures attack our heroes on the battlefield, who’s going to question whether that string run was played by a wandering minstrel on the battlefield or by aliens watching from the heavens? We’ve suspended quite a bit of disbelief by that point. Which is equally valid for science fiction, where there truly is the distinct possibility that the melody was inspired by extraterrestrial entities.

Say what you will about the quality of fantasy translated to film (and there is much to be said, sadly), but arguing with the bold brass and sweeping melodies of John Williams, James Newton Howard, Harry Gregson-Williams, Howard Shore, and James Horner isn’t nearly as easy a task. It’s music at its most grandiose; it is fantasy. Sure, fantasy may not be its own musical genre, but one need only feel the swelling strings wash over the softly-repeating piano ostinato of Lady in the Water to know that it is unique, it is immaculate, and it is ours.

And whether it be the latest space opera or (yet another) Final Fantasy, there’s likely much more music to come.

Unless, of course, the aliens decide to take some time off.