Duct Tape And Turing Machines

After trying several flavors of David Allen’s Getting Things Done methodology over the last year, I’ve finally managed to develop a personal organization system that works for me. I’m using 37Signals’ wonderful free Backpack service to keep what amounts to a lightweight spiral-bound notebook full of writing, todo lists, reminders and the like. When it comes to personal organization, my main challenge is coralling all my stuff into a single place where I can deal with it and know that I’ve got the whole picture in front of me. Backpack has proven wonderfully effective in that regard in the month since I’ve started using it.

However, I’m also a long-time del.icio.us user and didn’t want to give up my old habits of squirreling away links there to read later at my leisure. To the end of having all my fish in one barrel, I wrote a Greasemonkey script to pull in my most recent del.icio.us posts and display them in the sidebar of my Backpack home page.

You can get the script here. The first time you visit your Backpack home page after installing the script, it’ll prompt you for your del.icio.us username and the number of links to display- you can later change these values through the Greasemonkey User Script Commands menu. I’m releasing this under the MIT license, so feel free to munge, improve and otherwise frobulate it as you see fit. Enjoy!

I recently finished reading Dreaming In Code. Inside was a quote that I thought might be the wisest words I’ve heard about personal projects:

“Nobody should start to undertake a large project. You should start with a small trivial project, and you should never expect it to get large. If you do, you’ll just overdesign and generally think it is more important than it likely is at that stage. Or, worse, you might be scared away by the sheer size of the work you envision. So start small and think about the details. Don’t think about some big picture and fancy design. If it doesn’t solve some fairly immediate need, it’s almost certainly overdesigned.”

– Linus Torvalds

In my copious spare time since I last posted, I’ve managed to get JDE set up and usable. JDE is pretty complex- it essentially turns Emacs into a full Java IDE- so I’m just going to outline the details of how to quickly get set up and exploring the environment. Check the User Guide for the full story.

These procedures were tested on Emacs 22 running on Ubuntu Linux. The next step is to all this set up on the Windows laptop I use at my job.

Installation

Artur Hefczyc has created a shell script that downloads and sets up JDE and all it’s dependencies. If you want it to install the libraries somewhere other than your system default site-lisp directory, define it as a global variable when you run the script.

SITE_LISP_DIR=/.emacs.d/site-lisp/ ./install-jde.sh

I discovered that putting these newly created directories under version control with subversion will break them- you’ll get an error along the lines of “jde-.svn.el does not exist”. In order to keep my emacs environment as mobile as I can, I simply zipped up the libraries and put the zip files under version control instead. Yes, I have to go through the extra step of unzipping the archive in any new environment, but given that these files probably won’t change that much, it’s not a huge disadvantage.

Next, drop the following in your .emacs (assuming you installed the JDE libraries in ~/.emacs.d/site-lisp/):

(load-file “~/.emacs.d/site-lisp/cedet/common/cedet.el”)
(add-to-list ‘load-path “~/.emacs.d/site-lisp/jde/lisp”)
(require ‘jde)

Project Setup

To get the Eclipse-like features of JDE working, you need to setup a project file. Drop a file named prj.el somewhere in the directory tree of the project you’re working on, and JDE will automatically load it when you open a Java file associated with the project. At minimum, you need something like the following:

(jde-project-file-version “1.0″)
(jde-set-variables
‘(jde-sourcepath ‘(”./path/to/source”))
‘(jde-global-classpath (”./relative/entry/on/classpath/”
“./another/relative/entry/on/classpath/”
)))

On the sample project I was testing all this upon, I had all my compiled classes set up in build/classes and a bunch of jars I needed linked in lib. I avoided listing each jar independently by using the directory-files function to expand out a list of all files matching /.*jar$/ in the lib directory.

(jde-set-variables
‘(jde-global-classpath (cons
(expand-file-name “~/magic-project/build/classes/”)
(directory-files “~/magic-project/lib” t “.*jar$”))
))

Unfortunately, I haven’t been able to get this working using relative paths yet.

Nifty Things

So, what about those neat IDE-ish things that were the whole point of this exercise? There’s a metric ton of useful stuff in JDE, but here’s how get to equivalents to the killer features in Eclipse:

• jde-complete: Expands the expression where your cursor lies. If there’s a single completion, it’ll get filled in. Otherwise, you’ll get a menu with all the possible completions. It’s very much like hitting Control-Space in Eclipse.
• jde-open-class-at-point: Jumps to the definition of whatever symbol- class, variable, whatever- that the cursor is on. Note that you have to set jde-sourcepath in prj.el for this work.
• jde-ant-build: Ant build script integration. It’s a simple as (jde-ant-build ”path/to/build.xml” “target-name”)

The default keybindings for most of the jde functions seem to have been created by classical guitarists hell-bent on avoiding conflicts (C-c C-v C-. is mapped to jde-complete, for example). You probably want to rebind them using global-set-key calls in your .emacs.

I’ve been using Emacs for roughly five years now and it’s quite shameful how little I’ve explored of it. I feel like I’ve barely begun to scratch the surface of the power of having a LISP engine underlying one’s day to day work. Having been content to be a journeyman Emacs user for too long, it’s high time I starting investing time in learning more about the program that eats most of my daily keystrokes.

So! Last weekend, I put my .emacs and its associated folder of non-standard modes in a remotely-accessible subversion repository. I use Emacs on four main computers- my Gentoo server, my Ubuntu desktop, my girlfriend’s Powerbook, and my work Windows laptop . Getting all my current customizations and keybindings working sensibly across all these different platforms was a learning experience and reward in and of itself. I’ve been reading up on elisp and starting to write bits and pieces of it; now, if I whip up and save some useful little lambda at work, it’s available wherever I am.

The next major task ahead of me is something of a doozy. At work, I use Emacs for everything- except Java. When I’m working on the Java side of our codebase, I tend to use Eclipse. Shameful, I know. It’s a huge resource hog, entirely too mouse-and-menu-centric, has relatively inefficient default keybindings, and doesn’t have a scripting layer akin to having LISP everywhere. But it’s got some features that make it really indispensable when working on a huge, sprawling Java codebase (and they all are). The ability to explore the APIs you’re working with via introspection and inline autocompletion is insanely useful, as is the ability to jump to where a particular method, class or variable is defined. It’s the size of the language and it’s libraries that makes wrangling a Java project unwieldy- you could probably take out a Buick by dropping Java In A Nutshell from a few stories up.

But I know Emacs can do all this, and I’d really like to be able to work with big Java projects in the world’s best programming environment. It’s just a question of figuring out how. So far, JDEE looks promising. It harnesses, among other things, the Semantic Bovinator to provide the language-specific features I mentioned above. It’s proven to be a real bear to set up, however, and I need to free up a good chunk of time to dig into it.

I’ll try to post any useful tips and “ah-ha!” moments here as I continue my journey towards Emacs zen.

We’ve moved! I felt it was time this blag got it’s own domain. Looks like the move went smoothly, but let me know if you find something not working.

Emacs is my text editor and operating system of choice and I more or less have the basic keybindings burned permanently into my fingertips. This makes editing text in a web browser less than fun- not only do I not have the awesome text slicing-and-dicing power of emacs, but some key sequences have less-than-fun default bindings in other applications. Try hitting C-w to yank text in a browser under Windows…

With the Firefox extension It’s All Text!, you can natively hook up your editor of choice to any plain textarea on a web page. This is not limited to emacs, but extends to anything that can open and save a text file, including shell commands or that other editor. Pretty nifty!

Because clearly I’m just as cool as Eric S. Raymond, I’m been giving Ubuntu a shot.

My not-so-sordid history with Linux

I built my first computer while in high school. Out of both a shoestring budget and a desire to stick it to Bill Gates, I wanted to run Linux on it. I wound up installing Mandrake (now Mandriva), a newbie-oriented distribution that seemed the easiest way to switch over from Windows. And it was indeed pretty easy: there was a nice, safe layer of graphical abstraction over scary thing like, oh, everything related to the command line. Until something broke or your hardware didn’t work quite right. At that point, the five different set of GUI control panels became more an impediment to understanding than a help and the relatively poor documentation and community support finally drove me to dual-boot with Windows and use it for 99% of my day-to-day work for the next few years.

Fast forward to college. I started studying computer science and learning to program on the lab machines running Linux at Oberlin College. With help from some wonderfully patient TAs and labmates, I picked up enough of the fundamentals of bash and *nix-isms to start using Linux with enthusiasm. Once again, I began finding Mandrake horribly limiting. Rather than pick another newbie-friendly distro, I took the advice of more experienced OCCS hackers and took the plunge into Gentoo.

It took me about three days to install. It was about a month before all my hardware was working together harmoniously. I spent more hours than I care to admit prowling the Gentoo forums looking for HOWTOs and bits of arcana needed to get the bleeding edge version of whatever I was trying to install working. And my God, did I learn. Every time I couldn’t figure out how to make something work or something wasn’t quite as slick and cool as I wanted, I dug in to fix it and wound up learning about another facet of *nix system administration. By the end of college, I knew my way around Linux internals pretty well and could fake it on other *nixs like Solaris and the BSDs.

Enter Ubuntu

My creaking, four-year-old computer started to fail last month; programs would segfault mysteriously, compiles would fail in non-reproducible ways, and it would lock up frequently overnight. Rather than replacing what I suspect was either a bad motherboard or some failing RAM, I decided to just bite the bullet and build an entirely new system. After setting up Windows on the first partition (with the host of games that I keep it around for), I sat down to install Gentoo.

For those not familiar with Gentoo, the install process isn’t exactly a point-click-wait affair. Rather, you boot off a LiveCD into a terminal where you partition your disks, download and extract a bare-bones working system, chroot into said system, configure and compile your kernel and bootstrap the thing from the ground up. I’ve done this more than a few times and it’s not really that big a deal.

Well, so long as your hardware is well-supported. Something about my chipset’s SATA controller didn’t play nice with the latest gentoo kernel patchset. Or it’s possible my fstab wasn’t setup correctly. Or maybe grub was improperly configured. In any case, no matter what I tried, I just got kernel panics as soon as it tried to mount the root filesystem.

I’m hardly a novice user and I have no doubt that if I’d kept plugging away and prowling forums and mailing lists I’d have gotten Gentoo running quite happily (after all, the LiveCD booted fine). But this finally got me thinking. Gentoo’s really great if you’re a systems-level hacker. But I’m not. Aside from the usual desktop application suspects (web, IM, and so forth), I do web and AI hacking. High-level stuff. I’m not doing real-time audio, I’m not doing kernel work, I’m not mucking about in the bowels of device drivers. And I’m pretty comfortable with my level of knowledge concerning the under-the-hood workings of Linux. So there’s no real compelling need to be running a distribution that’s as bare-bones-to-the-metal as Gentoo.

So, about two hours after downloading the latest beta of Feisty Fawn, I was up and running with pretty much everything I wanted under Ubuntu.

Musings

I’m just not terribly impressed with the direction, or lack thereof, that Gentoo’s showing as a collective. On one hand, it’s a metadistribution and it’s thus not really fair of me to complain about lack of a cohesive goal. But on the other, it’s been a year since they held something as simple as a site redesign contest and the results of that still haven’t percolated up to production. And that seems to be the state of Gentoo in general: kinda disorganized, no real goal in site beyond loosely holding together a set of packages and a way to tie them together.

And there’s really nothing wrong with any of that. But Ubuntu’s already taken away the single best reason to stick with Gentoo: apt-get is just as good a package manager as portage, has a huge base of software for it and doesn’t have the disadvantage of waiting half a day for something like OpenOffice to compile. Further, Ubuntu’s community and documentation are just as good, if not better, than Gentoo’s- and they’re going somewhere with it. It’s very clear the Ubuntu folks are driving towards clear goals: among them, a cohesive, solid desktop platform that succeeds in not alienating those of us who know what we’re doing.

Thus far, Ubuntu’s done everything I’ve asked of it and otherwise stayed the hell out of my way as I work, hack, surf, and fiddle. Not too bad for an operating system.

I’ve been working with something recently that’s made me reconsider my stance on JavaScript as a sensible language. It’s historically been the red-headed stepchild of dynamic scripting languages and for good reason:

• The standard library is a pitiful thing, indeed.
• Writing in anything resembling an object-oriented style involves some pretty goofy hacks to get inheritance and public/private access working
• Client-side web code has to be riddled with hacks and workarounds to make any non-trivial code work on all major platforms. Hell is other browsers.
• Like PHP, it’s widely used by the cut-and-paste school of people needed to make their website do that shiny thing the boss wanted. There’s a lot of really awful code out there as a result- enough to give the language itself a bad rap.

If you look at JavaScript from a pure language standpoint, it’s actually not too bad. It’s weakly typed and interpreted, which always lend themselves to rapid development. Its object literal notation, JSON, is clean and consise. Functions are first class objects, complete with closures. It supports inline regular expressions, with a reasonably large subset of Perl5’s functionality.

And now, with E4X, it’s actually done one better than Python, Ruby, and their ilk. Well, at least where anything related to XML is concerned.

E4X (ECMAScript for XML) is a language add-on to JavaScript that provides a very clean native syntax for querying and modifying XML. You can create XML objects from strings or inline as a primitive type and do complex operations on them in very few lines of code. At the moment, forgot about using this on your website: the only browser out there supporting E4X is Firefox 2.0, using the partial implementation in SpiderMonkey. However, Mozilla’s Java-based Rhino engine has a full implementation as of release 1.6-r2 and is well worth considering as a server-side or embedded scripting environment.

Code speaks louder than words, so let me present some quick examples to demonstrate how powerful this stuff is:
First, let’s define an XML object. We just drop the sucker inline; they’re now a primitive type, like regular expressions in Perl.

js> var factbook = <ninja>
<fact confirmed="true">Ninjas are mammals</fact>
<fact confirmed="true">Ninjas fight all the time</fact>
<fact><purpose>Flip out and kill people.</purpose>
      <purpose>Make burgers</purpose>
</fact>
<sighting confirmed="true">Royal Place</sighting>
<sighting confirmed="false">Taco Hell</sighting>
</ninja>;

Querying for all direct fact elements is as simple as:

js> factbook.fact
<fact verified="true">Ninjas are mammals</fact>
<fact verified="true">Ninjas fight all the time</fact>
<fact>
  <purpose>Flip out and kill people.</purpose>
  <purpose>Make burgers</purpose>
</fact>

Or even all purpose elements somewhere within the root’s children:

js> factbook..purpose
<purpose>Flip out and kill people.</purpose>
<purpose>Make burgers</purpose>

Adding a new elements into the whole shebang behaves according to the principle of least surprise.

js> factbook.fact[2].purpose += <purpose>Honor!</purpose>
js> factbook
<ninja>
  <fact confirmed="true">Ninjas are mammals</fact>
  <fact confirmed="true">Ninjas fight all the time</fact>
  <fact>
    <purpose>Flip out and kill people.</purpose>
    <purpose>Make burgers</purpose>
    <purpose>Honor!</purpose>
  </fact>
  <sighting confirmed="true">Royal Place</sighting>
  <sighting confirmed="false">Taco Hell</sighting>
</ninja>

Hey, what about all elements that have the confirmed attribute set to true?

js> factbook..*.(@confirmed == 'true')
<fact confirmed="true">Ninjas are mammals</fact>
<fact confirmed="true">Ninjas fight all the time</fact>
<sighting confirmed="true">Royal Place</sighting>

Consider what you’d have to get that done accessing XML at an object level with DOM, and you start to see why this kicks major ass.

var elements = document.getElementsByTagName('*');
var results = [];
for(var i=0;i<elements.length; i++){
    if(elements[i].getAttribute('confirmed') == 'true')
        results.push(elements[i]);
}

I’ve barely scratched the surface of E4X, but this should give you a taste of what’s possible. This stuff is cool enough to make me actually exciting about working with JavaScript and if that’s not a landmark in language development, then I don’t know what is.

My day job has been eating most of my time and mental cycles over the last two months, and so work on the AI project has been somewhat slow. I’m going about this in a sort of Tracer Bullet style of development and have been writing simple, extensible prototypes for each module of code. I’ve created very simple rendering and physics engines; the graphics at this point are more or less drawing lines on the screen and the physics engine still needs collision detection and response to be implemented. Not too bad, considering the amount of free time I’ve had lately.

I think it’s going to be very interesting to see how far I can take a pure-Ruby approach here. Ruby is an interpreted language and isn’t exactly speed king among it’s brethern. I’m working with it because it’s beautifully expressive language and the ability to program in a highly functional style is ideal for AI development. The AI routines, which are really the Neat Thing that makes me interested in this project, will be done with Ruby. However, I’ve also written the physics and graphics modules in Ruby: both traditionally pieces of code written in C for pure speed. Programming in Ruby is far more pleasant than memory-leak-and-error prone C/C++ hacking, and I’m curious to see if the simple graphics and physics simulation I have in mind really requires one to code down to the bare metal. I still have a sneaking suspicion that I’ll wind up rewriting these modules in SWIG-wrapped C at some point…

I upgraded to the development-and-nearing release branch of Emacs about a week ago. It’s quite stable and has a bevy of shiny features: Steve Yegge talks at length about the new ability to embed lisp expressions into replace-regexp commands. The greatest thing for me so far, however, has been ability to use antialiased fonts. Behold!

Emacs 21, default font:

Emacs 23, using XFT and the ProFont font:

That sigh of relief you just heard was my eyeballs.

Installing the development version under Linux is largely dependent on your distribution: under Gentoo, you’ll need to add =app-editors/emacs-cvs-23* to /etc/portage/packages.unmask, then run emerge emacs-cvs. It seems it’s necessary to turn on the xft backend on the command line, so a simple alias in your .bashrc ought to do the trick:

alias emacs='emacs-23.0.0 --enable-font-backend --font "ProFont-11"'

If you’re stuck using Windows (like I am at work), then check out EmacsW32. It’s a nice packaging of Emacs for Windows that’s built against stable cvs snapshots. For you OS X users, I’ve had great success using Emacs.app on the Powerbook I use for road-hacking.