End the War on Freedom

On November 17, I gave a presentation to the Montreal Scheme/Lisp User Group about developing high-performance network servers in Lisp. I looked at the lessons to be learned from existing Common Lisp web servers, notably Antiweb and TPD2, tried out some of the ideas in my own new under-development web server HTTP DOHC along with some new ideas about threading and concurrency, and put the results of my findings into the presentation.If you're interested in the topic, you should also watch John Fremlin's TPD2 presentation to the Shibuya Lisp user's group.I plan on making HTTP DOHC a full-featured Common Lisp web server with an interface that's somewhat compatible with Hunchentoot 1.0.In other news, I now have time available for contract work. Get in touch at vsedach@gmail.com.

Inspired by the previous TC Lispers meeting and spurred on by the probable topic of the next TC Lispers meeting, I have spent the little bit of coding time I’ve had over the past two weeks on making a GUI layer using Sheeple atop CL-OpenGL and employing ZPB-TTF for font-loading. I have dubbed this project Woolly (because it’s made from Sheeple and because Woolly sounds sorta like GUI). I spent about half of my coding time so far getting the basic framework in place (proven through clickable buttons with labels). I am trying to keep it cleanly separated between generic GUI stuff and the CL-OpenGL specifics in the event that someone would like to port it to some other I/O spec. The other half of my coding time was spent getting the font-rendering to be anti-aliased. I promise to write more about how I accomplished the font-rendering in a future post so that if you’re ever stuck rendering fonts in OpenGL, you won’t be stuck with pixelated blockiness or resorting to rendering to a texture-map and letting the mipmapper figure it out. For this post, however, I’ll just show you the code that sets up the interface depicted here. 1234567891011121314151617181920212223242526(require :asdf) (asdf:operate 'asdf:load-op 'woolly-gl :verbose nil) ;; make it easier to change renderer/controller later (rename-package 'woolly-gl 'toolkit) (defun test ()   (let ((font (sheeple:object :parents toolkit:=font=                               :em-size 48                               :pathname "okolaks/okolaksRegular.ttf")))     (let ((app (sheeple:object :parents toolkit:=app=))           (win (sheeple:object :parents toolkit:=window=                                :title "Woolly Window 1"                                :width 640                                :height 480))           (but (sheeple:object :parents toolkit:=button=                                :offset-x 40                                :offset-y 40                                :width 300                                :height 100                                :font font                                :label "Button")))       (woolly:display-window win)       (woolly:add win but)       (woolly:main-loop app)       (woolly:destroy-window win)))) Next up on my agenda is to make the background and button prettier. It should be easy enough to do with GL_LIGHTING and some vertex-coloring for gradations. After that, it’s on to more controls like labels, panels, checkboxes, drop-downs, borders, and (my dread) text input boxes. Then, it’s on to a layout manager.

A while ago now, Nikodemus Siivola moved bug information for SBCL from a flat text file to Launchpad. Historically I have had almost nothing but displeasure working with the "standard" or "industrial" bug trackers; I have found Bugzilla horrible to work with, both as a bug reporter and as an administrator; lighter-weight solutions such as Trac are just about tolerable, but basically anything that requires me to have a Web Browser seems to end up confusing and distracting me. An honourable mention at this point goes to debbugs: being able to report, manipulate, update and close bugs by e-mail is close to my idea of Nirvana. So, Launchpad. Initially, I was dismayed, because there doesn't seem to be a way of getting notifications of bug updates over RSS, which would be a second-best to getting updates by e-mail. I managed to ignore all SBCL bug reports for a while, but eventually I bit the bullet and signed up (having refused to do so a good long while ago, when shortly after I used Ubuntu's bugzilla to report a bug they closed the bugzilla in favour of launchpad without managing to transfer accounts across.) A large motivation for signing up was the discovery that launchpad does, in fact, have an email interface to the bug tracker; as long as you can emit GPG-signed mail (which I can), it seems to have all the required functionality for doing things without needing to go near a web browser; I can now receive bug reports and reply to them, and in at least some cases the References: headers in the mail I receive allows my client to thread the discussion properly (I haven't really stress-tested this yet, but it works at least well enough for now.) If that were all, this would not be news (and not even worthy of a blog post). But now I get to demonstrate my Emacs lisp “scripting” ability, in much the same way as Dan Barlow did for me many years ago: SBCL has a mailing list for reporting bugs, for people who are unsure as to whether their problem is a bug or not, or for people who don't want to go to the trouble to get a launchpad account just to report a bug. When such a report does describe a new bug that we should be tracking, that report needs to make its way to launchpad. Without too much further ado, I present sbcl-bugs-mail-forward, which constructs a message (almost) ready to be sent: (defun sbcl-bugs-mail-forward () (interactive) (let ((message-forward-ignored-headers "") from subject) (gnus-summary-mail-forward 4) (message-goto-to) (insert "new@bugs.launchpad.net") (message-goto-subject) (message-beginning-of-line) (re-search-forward "\\[\\(.*\\)\\].*\\[\\(.*\\)\\] \\(.*\\)$") (setq from (match-string 1) subject (match-string 3)) (message-beginning-of-line) (let ((kill-whole-line nil)) (kill-line)) (insert subject) (message-goto-body) (insert "Report from " from "\n\n") (insert " affects sbcl\n status confirmed\n importance ") (save-excursion (insert "\n tag \n done\n\n") (message-goto-body) (re-search-forward "^\\(-\\)+ Start of forwarded message \\(-\\)+$") (beginning-of-line) (let ((kill-whole-line t)) (kill-line)) (re-search-forward "^\\(-\\)+$") (beginning-of-line) (end-of-buffer) (kill-region (mark) (point))) (mml-secure-message-sign-pgpmime))) You can tell it's scripting, really: it's an odd mixture of plausible and dubious ways of getting things done: regular expressions to extract the original sender of the report, and to remove the forwarded message information (and the dull advert inserted in the footer by SourceForge's mailing list system). On the other hand, that function, coupled with something along the lines of (setq gnus-parameters '(("nnml\\+private:list.sbcl-bugs" (gnus-summary-prepared-hook '(lambda () (local-set-key (kbd "C-c C-f") 'sbcl-bugs-mail-forward) (local-set-key (kbd "S o m") 'sbcl-bugs-mail-forward)))))) gives me exactly what I think I want: a simple way of creating, tagging and classifying an entry in the bug tracker from a mail report. I couldn't find any convenient emacs/launchpad interfaces (or any at all, in fact); I'm not sure this counts as one either, but by all means use, adapt and improve on the above for your purposes – I'll happily take criticism of and improvements to this hack.

Earlier this month Google released Closure Templates, a new templating library intended for generating HTML. Who cares? I personally dislike HTML templating, and avoid it whenever possible in lieu of s-expression based generation tools like CL-WHO. At first look, Closure Templates didn't seem to be anything new or useful.The one thing that makes Closure Templates somewhat interesting is that the library works in both Java and JavaScript, so you get client and server-side templates in one place. I did a similar thing with uri-template by having the URI template expand into code that executed in both Common Lisp and JavaScript via Parenscript (here I have to state that despite being an author of a templating library, I still dislike templating libraries and even find my own creation annoying at times).About a week after Closure Templates was released, Andrey Moskvitin (archimag) wrote an implementation in Common Lisp (interesting note: it took him only five days and 1/15 the number of lines of code as the original). The resulting system, cl-closure-template, similarly generates JavaScript via Parenscript.I still didn't understand the motivation. Yesterday, Andrey was kind enough to explain it. Those pampered by web development in Common Lisp using s-expression HTML generation tools simply don't encounter the problem of trying to fit HTML templating onto the paradigm of incremental page updating via AJAX. So if you want to build an AJAX web application and use HTML templating, give cl-closure-template a try.[Blog metanote: if you're reading this blog through its feed, you will shortly see this post show up in Russian. a CONS is an object which cares is now syndicated on Russian Lisp Planet, and I'm going to be writing Lisp-related posts in Russian as well as English. All such posts will be tagged 'lisp-ru'. Filter out that tag if you don't want the Russian version of the posts to show up in your feed reader.]

As mentioned earlier: the 2010 European Lisp Symposium invites your contributions. Unfortunately, the website for the 2010 event is not set up yet; you can get an impression of what the event is like by looking at last year's website, which in the fullness of time (soon, I hope) will be updated with ELS2010 information. In the meantime, here's the Call for Contributions: we would welcome both papers describing original work, not published elsewhere, and submissions for tutorial sessions. Submission will be through EasyChair's conference management system. 3rd European Lisp Symposium May 6-7, 2010, Fundação Calouste Gulbenkian, Lisbon, Portugal Important Dates

• Submission Deadline: January 29, 2010
• Author Notification: March 1, 2010
• Final Paper Due: March 26, 2010
• Symposium: May 6-7, 2010

Authors of accepted research contributions will be invited to submit an extended version of their papers to a special issue of the Journal of Universal Computer Science (J.UCS). Scope The purpose of the European Lisp Symposium is to provide a forum for the discussion and dissemination of all aspects of design, implementation and application of any of the Lisp dialects. We encourage everyone interested in Lisp to participate. The European Lisp Symposium 2010 invites high quality papers about novel research results, insights and lessons learned from practical applications, and educational perspectives, all involving Lisp dialects, including Common Lisp, Scheme, Emacs Lisp, AutoLisp, ISLISP, Dylan, Clojure, and so on. Topics include, but are not limited to:

• Language design and implementation
• Language integration, interoperation and deployment
• Development methodologies, support and environments
• Reflection, protocols and meta-level architectures
• Lisp in Education
• Parallel, distributed and scientific computing
• Large and ultra-large-scale systems
• Hardware, virtual machine and embedded applications
• Domain-oriented programming
• Lisp pearls
• Experience reports and case studies

We invite submissions in two categories: original contributions and tutorials.

• Original contributions should neither have been published previously nor be under review in any other refereed events or publication. Research papers should describe work that advances the current state of the art, or presents old results from a new perspective. Experience papers should be of broad interest and should describe insights gained from substantive practical applications. The programme committee will evaluate each contributed paper based on its relevance, significance, clarity, and originality.
• Tutorial submissions should be extended abstracts of up to four pages for in-depth presentations about topics of special interest for at least 90 minutes and up to 180 minutes. The programme committee will evaluate tutorial proposals based on the likely interest in the topic matter, the clarity of the presentation in the extended abstract, and the scope for interactive participation.

The tutorials will run during the symposium on May 6, 2010. Programme Chair Christophe Rhodes, Goldsmiths, University of London, UK Local Chair António Leitão, Technical University of Lisbon, Portugal Programme Committee

• Marco Antoniotti, Università Milano Bicocca, Italy
• Giuseppe Attardi, Università di Pisa, Italy
• Pascal Costanza, Vrije Universiteit Brussel, Belgium
• Irène Anne Durand, Université Bordeaux I, France
• Marc Feeley, Université de Montréal, Canada
• Ron Garret, Amalgamated Widgets Unlimited, USA
• Gregor Kiczales, University of British Columbia, Canada
• Nick Levine, Ravenbrook Ltd, UK
• Scott McKay, ITA Software, Inc., USA
• Peter Norvig, Google Inc., USA
• Kent Pitman, PTC, USA
• Christian Queinnec, Université Pierre et Marie Curie, France
• Robert Strandh, Université Bordeaux I, France
• Didier Verna, EPITA Research and Development Laboratory, France
• Barry Wilkes, Citi, UK
• Taiichi Yuasa, Kyoto University, Japan

Today, I submitted a patch (the first free software lisp one in months for me!) to the Hunchentoot project, and it got accepted. Yay! Some backstory: Hunchentoot’s 1.0.0 release dropped a lot of implementation-dependent features, among them functionality to invoke the debugger if an error happens while handling a request. While workarounds exist, none of them were obvious to new users or users who recently upgraded. The patch I sent should fix this, hopefully. It adds a rudimentary error handling protocol to Hunchentoot, and provides two generic functions whose behavior can be adapted to your error handling needs. You can see for yourself in Hunchentoot’s svn repository. If you’re a Hunchentoot user, I urge you to test this (in both development mode using debuggable-acceptor and running with the default settings). The sooner you find bugs, the sooner they can be fixed, the sooner a release can be pushed out. And if you don’t find bugs at all, that’s cool, too (-:

Having struggled with the build system at work lately, I have decided to force myself to learn another build system at home: I am using XCVB for my UNIVAC emulator project. I was intrigued by XCVB when I read about it in ILC'09. ASDF has always felt unfinished, even though it's a vast improvement over loading files manually (or using customized system definitions). However, I've run into problems with ASDF and I'm not against other options. Some thoughts I had during the building, installation and use of XCVB:

• The name needs to change. Is it possible to pick a name that is not some strained acronym? Ant. Maven. Make. These are reasonable (if not goofy) names. XCVB (and for that matter, ASDF) is just alphabet soup. Pick a word and go with it.
• cl-launch feels like a whole lot of magic, but I've seen a whole lot of other magic work before.
• The process of building your own systems is a bit awkward right now. xcvb make-makefile --build foo when in the directory with the build file for foo is excessive specification. Building the system is even more awkward: make -f xcvb.mk obj/foo.image. Something akin to make foo would be much nicer.
• The idea of introducing metadata into a source file to indicate its dependencies doesn't sit well with some people. Personally, I like information about dependencies being close at hand; whether it comes from textual information or system queries doesn't matter. I'm content with textual information at the level of a file for now, but I suspect maintaining that information will become a pain. On the other hand, I found maintaining ASDF system definition files to be a pain (especially with refactoring and reorganization). As much as I would prefer to not use files in a Lisp system, files are a reality. I think starting with file-level meta-information is a good place to start in order find out what works and what doesn't.

Hopefully I get some time to try and help out with the project since I think it's ambitious enough to work.

Inspired by Nathan, I decided to put some of my not-hosted-anywhere things up on Github. This includes raylisp and sb-cga. Neither is released or supported in any shape or form, but since it turns out my yesterday's snapshot was broken on x86, this is easier than making new snapshots and less embarassing than letting people play with known-broken snapshots...

I've opened an account at github and opened repositories for two libraries: Ironclad and binascii. binascii is a library for doing “ASCII-armoring” of data (base64 and similar); I've been working on it for the past two weeks and am pleased with the results so far, both in the elegance of the code and the (untuned) performance. I do plan on making my other libraries available there as well; Ironclad was the most-requested library for DVCS visibility, so I thought I'd start with that.

Update 2009-11-14: since the snapshot linked below turned out to be broken on x86, you may be better off with the git repos instead. My first ever Common Lisp project was a raytracer. Unsurprisingly, it wasn't too great. Over time I started thinking about how I would make one that was more lispy, and during last couple of years I've been hacking on one every once and a while. It's called Raylisp, and while I'm not planning on ever properly releasing it -- this is purely a personal toy project -- I figured I might still make a snapshop available: it may prove useful to someone wanting to do something like this in earnest. The tarball contains:

• Snapshop of another work-in-progress: SB-CGA, a computer graphics algebra library for SBCL -- most suited for x86-64, since there it implements things like matrix multiplication using SIMD instructions. Raylisp uses it, but unlike Raylisp this one may even see a release some day...
• Raylisp itself, including:
  • reasonably efficient KD-tree construction and traversal code
  • very very hacky code for reading subsets of .ply and .obj files
  • basic raytracing stuff
  • a simple CLIM frontend on top of the raytracing library

In addition to the KD-tree code, which may be worth stealing and polishing off, I'm putting this up to demonstrate some techniques which may be non-obvious to new-but-performance-conscious Lisp hackers: stack allocation (all over the place), using CLOS objects for the parts that need flexibility and closures for the parts that need to be fast (rendering protocol), and packing data into specialized vectors to avoid GC overhead (KD tree construction.) (Last time I touched this code was in August, so all the details are not too fresh on my mind -- hence the vague tone in this post. I had inteded to do this then, but didn't get around to it until now.) All the code is under MIT-style license, as usual.

Qualys, Inc. is looking for a Policy Compliance Engineer who can research and write new capabilities in the form of LISP/QScheme functions to support signature development efforts.

Last Tuesday, I gave short talk about a library I have been writing on: Sequence Iterators.The library is perhaps a bit misnamed because at the moment, it's really a convenience layer on top of iterators. It's supposed to provide tools to make writing functions that operate on sequences convenient, and yet not blatantly inefficient.I'd be very much interested in feedback regarding rough corners if you find a case to apply the library. If you want to give it a shot, there's a file "more-sequence-functions.lisp" which contain prototypes of useful sequence function for which I haven't yet had the time to write them.Please notice that the library is still work in progress.

Last spring, the Maxima challenge for ABCL was to get it to complete its test suite. We've mastered that bit of Maxima for some months now. However, as turned out soon after that, Maxima runs rather slow on ABCL. To some extent - being limited by the JVM - that's to be expected. The performance observed was way off base though: much too slow.Through analysis, the cause was established to be the fact that Maxima declares lots of symbols to be special [and that ABCL doesn't offer a way to remove that specialness].Last summer we found that allowing Maxima to undeclare specials increases ABCL performance immensely (roughly 35%). However, the final goal for Maxima is to make more sparingly use of specials and declaring unspecial or undeclaring special isn't defined in the spec. Because of the two reasons it felt not right to implement the solution at the time: it would have been a very Maxima specific one to a problem Maxima intends to fix in the long run.Peter Graves noted that he converted the special bindings storage in XCL to use the same scheme as in SBCL/CCL, using an array with active bindings instead of a linked list of bindings. He observed a performance gain of 10% in his tests.Last weekend, I implemented the same scheme in ABCL and although the general speed up doesn't show 10% in our tests [which may very well differ from Peter's], we observed roughly 40% performance gain on Maxima's test suite!

There has been a great response to my casual mention of the CL-USER map yesterday. Almost 50 new people have been added in the past 24 hours. I think it shows interesting patterns and I hope it can be used for Lispers to find each other for user group meetings, Lisp jobs, etc. A few people have been confused about how to add themselves to the map. There are a few steps:

• Log in to your Google account
• Go to the map and click the "Edit" button:
• Find your location and click on the pin icon to drop a pin and enter your details:

I should also mention that I didn't create this map. Mirko Vukovic set it up initially and all the people who added their info to it have made it interesting. I just hope more people see it and get some value from it, so I've added it more prominently to Planet Lisp. Enjoy!

In the past few days I've made a number of updates and changes to Planet Lisp. If you normally read the Planet via a feed, you will need to visit the site to see some of them.

• Added upcoming Lisp meetings in the sidebar; easily find Lisp meetings!
• Added embedded version of the CL-USER Google map in the sidebar; easily find Lisp users!
• Changed the appearance somewhat to get rid of big horizontal rectangles and lines
• Removed a number of feeds that had not updated in more than a year; that included some important past contributors like Juho Snellman (last updated in December of 2007) and Kevin Rosenberg.

As always, if you have a blog that's at least partly about Common Lisp, please send me a link so I can consider it for Planet Lisp.

On behalf of the developers of ABCL (Armed Bear Common Lisp) I'm glad to be able to announce the 0.17.0 release.This release features - among lots of other things - performance improvements, a fix for unexpected thread termination due to uncaught exceptions and example code for running ABCL on Google App Engine. Please refer to the release notes for the full list.If you have questions regarding use or licensing, or you find issues, please report back to the development list:armedbear-devel at common-lisp dot netSource distribution archives can be downloaded in ZIP (zip signature file) or gzipped tar (tar signature file) format. In addition, a ZIP binary (bin-zip sig file) and gzipped tar binary (bin-tar sig file) are available.

I've been maintaining a Google Calendar for upcoming Lisp-related user meetings. In the past month, several people have said it helped them find meetings, so I'd like to make it a little more visible. To that end, I've added a sidebar to Planet Lisp that shows a list of meetings scheduled in the next 30 days. It's updated daily. I've also created a new twitter account, @lispmeetings, that will post a tweet about a meeting the day before it happens. If you want to see an upcoming Lisp meeting on the calendar, please email me and I'll put it up.

Mikel Evins has released Apis, a sample Common Lisp Cocoa application using CCL's Cocoa bridge. Obligatory screenshot: For what it's worth, Mikel also provides Atta, which is a similar project but based on Gambit Scheme.

You're an intermediate to the Common Lisp language and are desperately looking for a practical project to try your Lisp skills at? You're longing for fame and glory?If that's the case, I have something for you:

1. write a library to access the Web API provided by launchpad,
2. afterwards, using that library, make it possible to conveniently add and update tickets in a (not yet existing) bug tracker for clbuild. The tickets are supposed to add/update repository urls.

Sounds interesting? If so, drop me a mail.

Clozure CL 1.4 is now available. For more information and instructions on how to get it, please see http://trac.clozure.com/ccl.