Thursday 9 April 2009

The Horn Package Manager - Introduction (The developer's cut).

I am writing a series of posts to help me prepare for my talk at Dsl DevCon.

I will be talking about the Dsl that is used to describe package definitions for the horn package manager open source project I have been contributing for.

You can read a series of introductions to the horn package manager here, here and here.

As is customary, I will briefly introduce myself. Compared to most of the speakers at the conference, I can best be classified as the average developer in the field. If any of you are familiar with the "hitch hiker's guide to the galaxy", then I am the developer who would get evaporated by the total perspective vortex if he actually seen himself positioned in his rightful place in the developer food chain. I have a keen interest in the ALT.NET space. I both agree and disagree with many of the exponents of ALT.NET in equal measures. I do find it a vibrant and fun place to share new ideas. Naturally such topics as the merits of DSLs are debated from time to time in this space.

In my day job, I just do not work on the kind of complex application that has deep, hard to fathom business rules that a DSL would help to abstract away the logic. The domain that empowers the development of my current software line is just not rich enough to be expressed in the gushing terms of Eric Evan's famous Domain Driven Design book. Indeed the time spent to create a semi English prose that would help unearth new models in our problem space is just not cost effective. Having just started my own business, the time spent on such a flight of fancy like this would hurt me where it hurts most, in the pocket. That said I am a passionate developer who not only codes for a living but codes in his spare time. More recently the focus of my hobbyist coding exploits has been to help start and contribute to an open source project that hopes to cure a genuine lacking in the .NET space. I can say without fear of recrimination that I have been the main contributor on this project.

As with most open source packages, the goal is to alleviate a genuine point of pain that clogs up a developers progress in his pursuit of delivering the software system. In the .NET space there is a persistent and genuine point of pain that it seems remarkable that nobody has tackled adequately yet.

The horn package manager as we have lovingly christened it sets out to ease the pain of building open source packages or indeed any project and it's dependencies. We are trying to plug a point of extreme pain that has been so effectively plugged in ruby by rubygems, by maven in Java and appget. Our initial goal is to issue an order at the command prompt and watch as our glorious software system takes charge of retrieving the source code of a project like Nhibernate from a source control management system and builds the binary packages as we watch the text of the output stream slide elegantly down the command prompt window. We want to not only build the source of the package but also any dependencies that the package is more than likely to have. Horn predominantly and initially wants to build packages from source code, we believe this will give us the maximum in flexibility and is the only way we can truly cure the pain of upgrading our open source packages. Allow me to give further context to this very real pain that I and countless others experience.



Our target audience is that of developers who like to live on the latest code version "trunk" or as we like to romantically think of it, the bleeding edge. The bleeding edge in this context is what is known as "working off the trunk". My current stack of open source packages that I use to help me with my daily software development tasks include such open source parts as:There is a deep and contrived dependency map between all these packages. A recent happening or event will better explain the treadmill of change that we face as developers if one of these loose house of cards decides to flip or upgrade itself.

If you are familiar with the .NET space then you will surely have noticed the release of the ASP.NET MVC framework. This release caused a lot of ALT.NET developers to quickly grab their svn client of choice and start rebuilding their software stack. Why? Because the upgrading of any dependency causes a domino effect of upgrades. Building this stack is a time consuming irritating, and error prone MANUAL process. Here is a brief synopsis of the build process.
  • MVC Contrib has a dependency on the newly released ASP.NET MVC faramework. In order to use the newly released MVC framework, it will need to be rebuilt against the release candidate binaries.

  • MVC Contrib has a WindsorControllerBuilder piece that gives easy IoC integration with the MVC framework. This is gives us a dependency with the Castle stack.

  • I think I am correct in saying that MVC Contrib is built against the castle trunk version. You will need to rebuild against the Castle trunk verson. This will trigger other rebuilds from other open source projects

  • Rhino has a dependency on the Castle stack.

  • Nhibernate has a dependency on the castle stack.

  • etc., etc.

  • The treadmill of change has been triggered.

And so it goes on. This upgrade is a most time consuming and painful upgrade. This is our intial goal of horn. We want to build our stack whenever we want with the minimum of interruption.

When starting out in such a daunting project such as this, we made the very educated step of copying something existing. Package management in non compiled platforms such as ruby where you are dealing with only code files led us to not follow the excellent ruby gems. Other candidates for blatant plagarism such as maven seemed more to do with binary management than source code binary construction. One of the initial horn developers who namely was not me, suggested that we take a look at the gentoo portage package management system. Portage instantly seemed very interesting and resonated with our ways of thinking.

At the heart of portage is the metaphor of the portage tree. The portage tree can be thought of as a directory tree hive that contains categorised indiviual files of build instructions that are called ebuilds. An ebuild is a specialised bash script that automates the compilation and installation procedures of a software package. A user of portage will instruct portage via the command line as to which software package is required. Portage will then search the directory hive or portage tree for the file that contains the metadata or in portage speak, the ebuild that contains the instructions to get, compile and install the requested software package.

With such a fantastic and intriguing model to copy as portage, we were in business. We would replicate this model package tree database of build scripts in a .NET style. The first step was obviously to define our version of the ebuild. Probably the most obvious choice was Xml but as anyone who is as cool or as trendy as an ALT.NET developer could not take the social stigma off using such an archaic metadata format. In all seriousness having recently migragted my Nant build scripts to ruby's rake, the choice was an obvious one. Programming logic is ill placed in Xml, I think even the creator of Ant has later said that Xml was not a great choice for an otherwise excellent build language.

In my next post I will describe the decision to choose boo as the language to host our internal DSL.

If any of this is of interest to you then please join the Horn user group for updates or check out the source here.

No comments:

Post a Comment