To Lisp or not to Lisp, that is the question.

Musings on Lisp by a routinely Pythonic programmer

Last week I did some maintenance on various Python projects I haven't touched in years (literally), and I was surprised by how easy, almost trivial it was to reorient myself and make the necessary changes.

That observation came at the right time because I've been reading up on Lisp dialects for a while now and questioning whether or not I should be programming in Lisp instead. Lisp enthusiasts (converts?) certainly make persuasive arguments, typically advocating Lisp as the one-true-language with near religious zeal.

As programming languages go, I'm not very loyal. Over the years I've programmed in the usual assortment of languages (e.g., C, C++, Java). Then I discovered how much more productive I could be using a high-level interpreted language and I fell in love. With Perl. The affair lasted a few years until I realized she was a capricious, evil mistress and I left her in disgust for a younger, more elegant programming language: Python. With which I've been living happily ever after. No complaints. I have a happy text life. But sometimes, lying in bed late at night, my mind wanders and I wonder, if there isn't something even better out there for me. Just waiting to be discovered...

Could Lisp be that temptress?

Here's the thing: I've done my homework and it seems to be true that the various Lisp dialects are inherently more powerful programming languages than Python.

The reason for this is that Lisp isn't really a language at all. It's more of a mathematical entity. You might say it wasn't invented but discovered. In fact if there is intelligent life in the universe and they have computers I'll bet they have something equivalent to Lisp too.

You see, other "languages" are compiled/interpreted into a tree of instructions. With Lisp, you write programs directly in a lightweight text representation of the instruction tree. The learning curve is substantial. If you've programmed in other languages many things need to be unlearned. And the result, with all of those parentheses can look intimidating to someone who's mind isn't used to Lisp.

What makes Lisp so powerful is that there is no distinction between data and code. Code is data. Data is code. Lisp macros take advantage of this by allowing you to execute code at compile time that generates your runtime program. Features that need to be built into other languages (e.g., support for object-oriented programming) can be added to Lisp as a standard library of sorts. At its core you can think of Lisp as a sort of meta-programming language.

All of this makes Lisp extremely powerful, but... wait for it... this power comes with a price.

Compared with Python, Lisp is much harder to read and much more verbose than Python. Lisp makes it easier for a programmer to create mind-numbingly complex code. If you think a big ball of mud is bad, imagine a program implemented in a programming language that is itself a big ball of mud. Of course, you can write good and bad programs in any language but Lisp gives you more rope to hang yourself since you can redefine anything and reprogram the language itself.

Since Lisp is so flexible when you try to understand a Lisp program there is a decent chance you are looking at a program written in a domain-specific programming language invented by the programmer that you need to learn and understand first before you can understand the program. It won't necessarily help much that this language superficially shares the parse-tree syntax of Lisp. Two functionally equivalent Lisp programs may be as different under the hood as two programs implemented in Haskell vs Erlang.

Whether it makes sense to pay the price for all this power ultimately depends on the type of programs you are writing and the type of programmers you have in your team.

In my mind just because you can implement your own programming language doesn't mean you should. I think it's only rarely a good idea. Programming languages are hard to design well. Usually, I'd rather let Guido and the other language design experts in the Python community carefully work out what features a language should have rather then hack out my own.

Unless the inherent complexity of a program is very deep (e.g., AI like tasks) and to dig in you actually do need the power to extend your programming language. In that case, Lisp might be a good choice if you have programmers you can trust not to abuse it.

Note that many famous programmers who know both still prefer Python for most programming tasks. Where Lisp is optimized for power, Python is optimized for simplicity and ease of use.