Buildout for Integrators

I'm sold on buildout, for me. It allows me to start working on a project, figure out what products I need and how the whole thing fits together, then scale up to deployment and have a sane environment all the while. It has taken a lot of the annoying, manual work out of replicating our development and production environments at work, and it's making creating a transition from Drupal a little more fun. Right now though, I'm don't think it's ready for most audiences; you need a stomach for configuration files, reading source, and currently a little unix. That does not have to be, though.

Tom Lazar, who has recently been shepherding excellent information and insight into a discussion about what integrators need and what they want, writes "Somebody needs to make sure that the new Zope 3 way of doing things is followed all the way through to the end user. Currently it's stuck somewhere at the developer's feet. This needs to change." So, here's a proposal for that. It certainly won't help everyone, and it will definitely need some work, but I'm willing to give it a go.

First, make buildouts modular in a different way. Currently, you can extend a buildout with another, though something sort of like inheritance or cascade. Unfortunately, this requires quite a bit of familiarity with buildout and, often, with subversion (since that's everyone's favorite way of putting SVN-controlled products and packages into a buildout). If you wanted to put quills into your buildout, you'd need to add its SVN externals into your own (assuming your buildout resides in subversion), then be sure to add that buildout's products, packages, and zcml variables to your [instance] section. I think this may be a tall order for users who are often used to a wizard for configuration.

Instead, I'd like to see a recipe for defining extensions and putting them together. For example, you could have a buildout (called, say, quills-external.cfg) that looks something like the following:

[buildout]
parts =
quills-packages
quills-products
external-def

find-links = https://svn.plone.org/svn/collective/quills.core/trunk#egg=quills.core-dev
...

eggs =

   quills.core
quills.remoteblogging

...

[quills-products]
recipe = infrae.subversion
urls =

    https://svn.plone.org/svn/collective/Quills/trunk Quills
https://svn.plone.org/svn/collective/fatsyndication/trunk fatsyndication
...

[quills-packages]
recipe = z3c.recipe.egg:editable
find-links =

    https://svn.plone.org/svn/collective/quills.trackback/trunk#egg=quills.trackback-dev

eggs =

    quills.trackback

build-directory = src
develop = true

[external-def]
recipe = modulus.recipe.external:define
products =

   ${buildout:directory}/parts/quills-products

eggs =

   ${buildout:eggs}
${quills-packages:eggs}

zcml = quills.core

---

This could then be included in other buildouts without having to know much about how it works, as follows:

[buildout]
parts =

   plone
externals
instance

find-links =

eggs =

   elementtree

[plone]
recipe = plone.recipe.plone

[externals]
recipe = modulus.recipe.external
profiles =

    ${buildout:directory}/quills-external.cfg
${buildout:directory}/someothercool.cfg

[instance]
recipe = plone.recipe.zope2instance
zope2-location = ${zope2:location}
user = admin:
http-address = 8080
eggs =

   ${buildout:eggs}
${plone:eggs}
${externals:eggs}

zcml =

   ${externals:zcml}

products =

   ${buildout:directory}/products
${externals:products}

I realize that was an absurd about of fake config to put in a blog, but I hope the idea comes across: instead of having to figure out how to integrate plone extensions that might be a mix of released products, svn products, eggs, and ZCML, an integrator need only get the external profile and have a buildout.cfg that has the extra few lines in it (which could actually be done with a new paster template). Developers then would need to write the profiles, but hopefully they could just follow regular buildout cfg syntax (oh, and use findlinks and infrae.subversion instead of SVN externals, because externals are not super-easy to merge from several projects or friendly to non-svn users). Ideally, developers will release their products, so there'll be less subversion involved -- but this would give them the option to not.

Secondly, once the externals recipe is done, adding a nice shiny GUI to buildout that handles grabbing binaries for the windows folks (and maybe getting X Code for the Mac folks) would be wonderful. Ideally, it could query plone.org to get a list of available stand-alone products (that could be installed with productdistros) and external profiles and let integrators select from them. Then, based on the user's configuration (picked from checkboxes and dropdowns) and the user's environment, it could make and run a buildout. Adobe has created a runtime (called AIR) for Windows, Linux, and Mac that allows for GUI apps to be written with HTML, JS, and a little of the API. This seems like an ideal environment for a bunch of web developers. It also would shield windows folks from needing the command line, which, given the powers of Microsoft's shell, I can't blame them for wanting to avoid.

So, now my question is -- are either of these actually good ideas? I certainly like them (or I wouldn't have done so much typing), but I have a feeling that, since I've not seen the externals one yet, there might be a good reason against it. I'd be interested in opinions either way. I also realize that Adobe AIR might not be the ideal choice (I know Ian Bicking wouldn't approve), and I'm totally open to suggestions on a cross-platform GUI runtime that I can code in HTML and JavaScript. I'd be interested to work on both of these projects -- anyone else game? Comments are open.

Comments

[SnowWrite]

Excellent start, MattThe back end code..I'm not sure what that would take and which is the best process, I leave that to those who love buildout, interfaces and adapters :-).

I do agree that some way for TTW customization to move from a one-way tunnel to something the every day "user" can utilize without endangering the file system code would be premium. I actually do all my theming on the file system now.. (after 2 1/2 years of exclusive TTW customization) Partly because I didn't understand how it worked..then I buckled down and learned DIYPloneStyle and I like it much better. Fact of the matter is that there are a lot more "designers" downloading Plone to integrate/implement it for their "boss" at Company X.. and the steep learning curve is a deterrent. OK so maybe that means that Plone is not for them.. but anyway we can make the learning curve.. a little curvier would go a long way towards making Plone much more accessible.

I hereby volunteer to help with the UI (design) aspect of this forthcoming GUI .. even if it's just a matter of creating xhtml wireframes and css to start.. (I realize it could be a long time for this to get done.. but I'm right there with you Matt!)

[Tom Lazar]

Eggs + cheeseshop solve part of the problem already todayGood writeup Matt and thanks for taking the time for doing it. These kind of discussions are precisely what's needed to create a working solution.

I do want to point out, though, that part of the problem you mention (i.e. the explicit dependency on subversion) is already solved: if a developer uploads his/her product as an egg to the cheeseshop, all you need to do is include it with one line (or two, or three lines... *cough*) in your buildout. I'm definitly going to release the next beta of Quills at the cheeseshop, for instance. `svn:externals` is a great concept (git users, feel free to laugh out loud here...) but it is, of course, targeted at developers. buildout is conceptually entirely agnostic in regards to subversion. You are just observing a common developer usage pattern.

It would be great, to see some kind of friendly GUI for handling the other things, particularly the XCode et. al. dependencies on Mac OS X and Windows. I don't know enough about AIR but I have my doubts, whether it would be well-received by the developer community. Perhaps a simply python application would do the trick... seing that we can't do away with the python dependency anway ;-)

[Sasha]

Reading my mind?This is almost the same thing me and Tim (timte) discussed last friday. But we discussed it more in the term of plone.recipe.buildout which would be a recipe to use other buildouts from svn in you own. This would allow you stuff like yours ${externals:eggs} and so on.

I take this as a confirmation of our idea :)

[Anonymous]

Why can't you ...... just use the extends feature?

Like:

[buildout]extends =http://svn.plone.org/svn/collective/quills/buildouts/base-1.0.cfgparts =quillsinstance

[instance]...eggs = ${quills:eggs}

....

That's a partial example anyway. This way, you get the the 'quills' part (and any of its dependencies) automatically via the extends directive, which loads another .cfg file from SVN. I think that's more or less exactly what you're sketching with moduls.recipe.externals, except it avoids the need for another part. Unless I'm missing something?

Also, you shouldn't really be using Subversion externals of dependencies in most cases. Only if you're a developer of that package, or you want to live on the bleeding edge should you track externals in that way - for "normal" people this is far too dangerous.

If quills is an egg, of course, all you say is:

[buildout]parts =...instanceeggs =quills>=0.1

[instance]...eggs = ${buildout:eggs}zcml = quills

...

I wish the "zcml" bit wasn't necessary (it isn't if you're using an somewhere else, like site.zcml or your own package) since then it'd just be one line: dependency (top-levle only: the quills egg should work out what other packages it depends on!) and version (only if you care, but usually you do), but that ain't so bad.

For more information about how to install third party packages, see http://plone.org/documentation/tutorial/buildout

Martin

[Lennart Regebro]

I like the idea of externalI like the idea of external extensions, but the examples you give should probably rather be handled through setup tools and eggs. But the idea of having external base configurations seems more useful for me.

Basic problem: Stability. What happens when the external config is updated?