Buildout for Integrators

I'm sold on buildout, for me. It allows me to start working on a project, figure out what products I need and how the whole thing fits together, then scale up to deployment and have a sane environment all the while. It has taken a lot of the annoying, manual work out of replicating our development and production environments at work, and it's making creating a transition from Drupal a little more fun. Right now though, I'm don't think it's ready for most audiences; you need a stomach for configuration files, reading source, and currently a little unix. That does not have to be, though.

Tom Lazar, who has recently been shepherding excellent information and insight into a discussion about what integrators need and what they want, writes "Somebody needs to make sure that the new Zope 3 way of doing things is followed all the way through to the end user. Currently it's stuck somewhere at the developer's feet. This needs to change." So, here's a proposal for that. It certainly won't help everyone, and it will definitely need some work, but I'm willing to give it a go.

First, make buildouts modular in a different way. Currently, you can extend a buildout with another, though something sort of like inheritance or cascade. Unfortunately, this requires quite a bit of familiarity with buildout and, often, with subversion (since that's everyone's favorite way of putting SVN-controlled products and packages into a buildout). If you wanted to put quills into your buildout, you'd need to add its SVN externals into your own (assuming your buildout resides in subversion), then be sure to add that buildout's products, packages, and zcml variables to your [instance] section. I think this may be a tall order for users who are often used to a wizard for configuration.

Instead, I'd like to see a recipe for defining extensions and putting them together. For example, you could have a buildout (called, say, quills-external.cfg) that looks something like the following:



[buildout]
parts =
quills-packages
quills-products
external-def

find-links = https://svn.plone.org/svn/collective/quills.core/trunk#egg=quills.core-dev
...

eggs =

   quills.core
quills.remoteblogging

...

[quills-products]
recipe = infrae.subversion
urls =

    https://svn.plone.org/svn/collective/Quills/trunk Quills
https://svn.plone.org/svn/collective/fatsyndication/trunk fatsyndication
...

[quills-packages]
recipe = z3c.recipe.egg:editable
find-links =

    https://svn.plone.org/svn/collective/quills.trackback/trunk#egg=quills.trackback-dev

eggs =

    quills.trackback

build-directory = src
develop = true

[external-def]
recipe = modulus.recipe.external:define
products =

   ${buildout:directory}/parts/quills-products

eggs =

   ${buildout:eggs}
${quills-packages:eggs}

zcml = quills.core




This could then be included in other buildouts without having to know much about how it works, as follows:

[buildout]
parts =

   plone
externals
instance

find-links =

eggs =

   elementtree

[plone]
recipe = plone.recipe.plone

[externals]
recipe = modulus.recipe.external
profiles =

    ${buildout:directory}/quills-external.cfg
${buildout:directory}/someothercool.cfg

[instance]
recipe = plone.recipe.zope2instance
zope2-location = ${zope2:location}
user = admin:
http-address = 8080
eggs =

   ${buildout:eggs}
${plone:eggs}
${externals:eggs}

zcml =

   ${externals:zcml}

products =

   ${buildout:directory}/products
${externals:products}


I realize that was an absurd about of fake config to put in a blog, but I hope the idea comes across: instead of having to figure out how to integrate plone extensions that might be a mix of released products, svn products, eggs, and ZCML, an integrator need only get the external profile and have a buildout.cfg that has the extra few lines in it (which could actually be done with a new paster template). Developers then would need to write the profiles, but hopefully they could just follow regular buildout cfg syntax (oh, and use findlinks and infrae.subversion instead of SVN externals, because externals are not super-easy to merge from several projects or friendly to non-svn users). Ideally, developers will release their products, so there'll be less subversion involved -- but this would give them the option to not.

Secondly, once the externals recipe is done, adding a nice shiny GUI to buildout that handles grabbing binaries for the windows folks (and maybe getting X Code for the Mac folks) would be wonderful. Ideally, it could query plone.org to get a list of available stand-alone products (that could be installed with productdistros) and external profiles and let integrators select from them. Then, based on the user's configuration (picked from checkboxes and dropdowns) and the user's environment, it could make and run a buildout. Adobe has created a runtime (called AIR) for Windows, Linux, and Mac that allows for GUI apps to be written with HTML, JS, and a little of the API. This seems like an ideal environment for a bunch of web developers. It also would shield windows folks from needing the command line, which, given the powers of Microsoft's shell, I can't blame them for wanting to avoid.

So, now my question is -- are either of these actually good ideas? I certainly like them (or I wouldn't have done so much typing), but I have a feeling that, since I've not seen the externals one yet, there might be a good reason against it. I'd be interested in opinions either way. I also realize that Adobe AIR might not be the ideal choice (I know Ian Bicking wouldn't approve), and I'm totally open to suggestions on a cross-platform GUI runtime that I can code in HTML and JavaScript. I'd be interested to work on both of these projects -- anyone else game? Comments are open.