Additional Shell Resources

tl;dr I've learned about a couple of additional resources since writing Stronger Shell, and I wanted to call them out. I've also updated the original post.

Thanks to comments around the web and a little keener attention elsewhere, I've come across a few other good resources that I think are worth sharing. I'll add these to the original post as well, but if you already read that, I wanted to call them out specifically:

  • Unix for the Beginning Mage (pdf) is a tutorial on the basics of shell work done as a story about learning magic. It's delightful, free, and reminds me of the why's (poignant) guide to ruby (which I credit with getting me started in ruby and subsequently web development), but more practical and less overtly weird. Thanks roneesh for pointing it out.
  • Bash (and other shells) has useful options that may make catching errors easier — for example, set -o errexit will "Exit immediately if a command exits with a non-zero status," which can be helpful in testing (and in real scripts). For the full list of options, see the set help page in bash or see the the options index for zsh. Thanks to bdunbar for the reminder.
  • For contextual help at the commandline itself, there's a neat python app: search cmd. It lets you do things like searchcmd find "sort files by size" and will return relevant examples from places like stack overflow. Being able to search without leaving the terminal (and see the results summarized) nicely helps keep context, which is key for feeling productive. Thanks to Pycoders Weekly, which is a generally useful resource.
  • If you're looking for more examples of interesting people things do in their shell, commandlinefu has tons. Like any large repository, the quality is variable, but there's definitely lots of interesting snippets in there that can help you expand how you think about your shell.

Stronger Shell

tl;dr Shell scripting is strange and somewhat forbidding, but will serve you better than most other frameworks or languages you can learn (after your first). I'll explain how I improved and offer resources to improve yourself.


Part of why I love programming is being able to automate repetitive tasks. That's pretty common for programmers; as the quip goes, I'll happily spend 99 hours to automated a task I could do by manually in 100. Strangely though, until I'd been programming professionally for a more than a few years, I was fine typing out many shell commands in a row every day to do repeated tasks.

As my jobs got more complicated though, my environment also did, and it became infeasible to remember everything that I needed to do in the terminal (so many server names to remember!) and too time consuming to keep searching my history all the time. Since scripting my simpler, day-to-day tasks in python wasn't really attractive, I decided to actively work to improve my shell scripting. This has paid off repeatedly.

Forcing Improvement

The trouble for me with improving at shell was bothering to start; although I knew eventually I'd get the time back, the first several scripts were going to take much longer than they'd ever save me.

Image Credit Randall Munroe /

Image Credit Randall Munroe /

Eventually, I decided to follow a rule that my friend Adam Hutton had mentioned: the second time you type a complex command in your terminal, make an alias. Aliases are quick to create and obviously save typing.

As soon as you start trying to do that though, you realize aliases, while wonderful, are pretty limited, and you stumble into shell functions and small scripts. Then, as a programmer, you are going to want flow control and variables, and that thankfully should start you hurtling through your own obsession to actually learning your shell thoroughly.

As my shell skills have improved, I've steadily widened the bar for what I will turn into a function/script/alias — I still apply the "if I type it twice, automate," but the scope of what "if I type it twice" has expanded considerably, because with scripts and functions, you can have arguments, and so automate nearly everything.

Resources for learning

Actually learning did not follow the path I expected; at first, it was hard to find recent books on bash and zsh at the right level for where I was, and shell scripting has quirks that didn't match my expectations, coming from Python/C++/Javascript. I eventually found a number of websites and books though that helped a ton, and collected a few tips below that would have sped me up considerably.

Online Resources

  • My first, favorite resource is the BashGuide wiki pages on, which is currently being updated to be a fuller tutorial at
  • Joshua Levy's "The Art of Commandline" is a good, quick introduction that will get you moving
  • Explain Shell is a terrific tool for pasting complex commands and seeing what is going on with them
  • Of course, you can read man bash, which, although turgid, is full of useful wisdom. While you're reading the manual, you'll also want to read man test so that you finally understand what [ -f ~/.bashrc ] means.
  • Unix for the Beginning Mage (pdf) is a tutorial on the basics of shell work done as a story about learning magic. It's delightful, free, and reminds me of the why's (poignant) guide to ruby (which I credit with getting me started in ruby and subsequently web development), but more practical and less overtly weird. Thanks roneesh for pointing it out.
  • For contextual help at the commandline itself, there's a neat python app: search cmd. It lets you do things like searchcmd find "sort files by size" and will return relevant examples from places like stack overflow. Being able to search without leaving the terminal (and see the results summarized) nicely helps keep context, which is key for feeling productive. Thanks to Pycoders Weekly, which is a generally useful resource.
  • If you're looking for more examples of interesting people things do in their shell, commandlinefu has tons. Like any large repository, the quality is variable, but there's definitely lots of interesting snippets in there that can help you expand how you think about your shell.
  • For ZSH (which I love), there's a great bunch of examples and capabilities explained in the ZSH Lovers man page. Additionally, Nacho Caballero's Master Your Z Shell with These Outrageously Useful Tips really expanded my mind as to what was possible in ZSH, and I'm grateful that he wrote it. Via Pycoders Weekly.
  • Finally, your nearest linux or mac computer are FULL of shell scripts, and once you get past reading things like [[ -z $(ssh -T host "exit" 2> /dev/null) ]], you can learn a lot from those scripts. A good habit is to read the scripts that libraries tell you to curl and pipe to sh, like any of the scripts on curlpipe (since you don't want to just run some random shell script from the Internet without knowing what it's going to do, right?).

The other thing that may not be immediately obvious is that bash/zsh/whatever-shell-you-like is your language, and the rest of the POSIX tools are your standard library. With that in mind, it'll definitely be worth your time to pick up even a little grep, tar, awk, sed, curl, head, tail, ssh, and friends.

The DigitalOcean community has actually done a really nice job writing tutorials for awk, sed and a a fair number of other tools. For awk specifically, I'm a fan of the Grymoire awk tutorial which substantially demystified it for me.

If style matters to you (PEP8 has made it matter to me everywhere), the bash hackers wiki style guide is a helpful resource so that your scripts look fluent and avoid obvious errors. Similarly, you should setup a linter, like a linter like shellchck.


No Starch Press has several excellent books for the aspiring shell user. My favorite is The Linux Command Line (affiliate link here and below, thanks!), which gives you a gloriously thorough and well-organized overview of many of the relevant shell commands you might need and, once you have those under your belt, shell scripting. If you're a bit farther along, some chapters may be too basic, but if you're like me, there are probably surprising holes in what you know that the book will fill in nicely.

Along similar lines to The Linux Commandline is Linux Command Line and Shell Scripting Bible, which is well-loved and thorough.

If you are just looking for shell scripting specifically, and you also have decided ZSH is awesome, Oliver Kiddle's From Bash to Z Shell is old but still useful; I refer to it from time to time when picking up new bits of ZSH still.

Finally, as you get more into this stuff, I highly recommend Michael W. Lucas's SSH Mastery (and honestly, any of his other sys admin books); if you're like me, you're probably spending a substantial portion of your shell time SSH'd into a remote host, and it's worth your time to really understand how SSH works and is configured.

Things that I wish I'd known

There are a couple things that I did not infer when I was initially trying to learn Bash from reading examples and experimenting, and I hope they speed someone else up in the future. They are also basically impossible to google, since they are mostly punctuation. Most can probably be picked up with a linter like shellchck, but I didn't have that setup at first either, so let my difficulty be your gain:

  • Bash is whitespace sensitive in a way that's... unexpected. You cannot have spaces around the operator in assignments (so export FOO = "bar" won't work, but export FOO="bar" is correct). You must have spaces inside your test expressions (so [[-z $FOO]] won't work; [[ -z $FOO ]] is correct).
  • There are a number of constructions that are nearly functionally equivalent. The ones that most tripped me up are:
    • [ expression ] and [[ expression ]] have the same function, and in general, if you're writing bash scripts, you should just use the [[ expression ]] version; the single [] version is older and doesn't support as many operations. And, to keep things optimally confusing, [] is the same as the test builtin. See the BashFAQ for the gritty details.
    • Single ticks (`command`) and $(command) are functionally equivalent, but in bash prefer $(command)the BashFAQ explains why.
    • Variable substitution can be done as either $VARNAME or ${VARNAME} — generally I do the latter, but you'll see both.
    • source and . do the same thing, which is execute the sourced file in the context of your current shell (so it can set environment variables in your interactive environment and whatnot).
  • Parens aren't exactly obvious either. (command) runs the command in a subshell (for uses, see the bash guide, or if you're impatient, just remember $(command) gets you the output of a command, so the command has to be running in those parens, right?). However, ((1+2)) does math (and $((1+1)) gets you the result of math). Having parens do something other than group expressions/override precedence was not something I expected.
  • Like our dear friend javascript, variables are global by default. In a function, to scope the variable to the function use the local keyword.
  • Bash (and other shells) has useful options that may make catching errors easier — for example, set -o errexit will "Exit immediately if a command exits with a non-zero status," which can be helpful in testing (and in real scripts). For the full list of options, see the set help page in bash or see the the options index for zsh. Thanks to bdunbar for the reminder.
  • If you switch between linux (GNU) and mac (BSD), many of your basic shell commands will differ; you almost certainly cannot copy some awk magic from stack overflow and expect it to work on both.

Where to go Next

The most important thing to do is write more shell code. The best way to learn is to do, and so, do! If you haven't yet, also pickup one of the great editors, either vim or emacs (or, my personal favorite, both via spacemacs) — once you're spending more time in your terminal, you'll likely want to be able to quickly pop into files to edit them, and it's nice to be able to do so right in the terminal.

Finally, make your shell environment nice. Spend some time with awesome shell (if you use bash) or awesome zsh plugins for ZSH users. Get a decent looking theme for vim. Setup nice completions for commands, and care for your dotfiles by versioning them. Spend a few hours making your environment pleasant and you'll be rewarded for years by that work.


Version-controlling my dotfiles

tl;dr: There are lots of neat ways to version-control your dotfiles. I ended up with homeshick and myrepos.


One of my goals for the last year has been to get more competent with my shell. I've used oh-my-zsh for years and found it helpful, but I'd never really bothered to learn shell-scripting or much about how my shell actually works. I've remedied that a bit (which I'll write about separately), and it's meant that my shell dotfiles have become a lot more useful to me.

Once I started to have a real investment of time in my shell and other command-line configs, I started wanting my usual tools -- version control, history, etc. I also wanted to be able to keep various computers in sync; I have a personal macbook where a lot of my configs originate, then my work computer, then various servers I log into to work periodically.

I'd gotten to a similar place a few years ago when I worked at US News, and I'd followed Brandon Rhodes's method for using git to track your dotfiles; however, my needs have changed, so I decided to see what other people do.


Since I'm doing a good bit of development at home and at work, and I want to share some settings but keep others private to their respective environments. Brandon's method of using a single git repo for all my dotfiles has a lot of benefits, but it doesn't lend itself to this particular setup; I want the work stuff in a private account and my personal stuff in my public github repo.

I also wanted to be able to maximize sharing, which means not requiring that a person take my whole setup to use part. For example, I found Byron Peebles's dotvim really helpful in starting to use vim, and I wanted to be able to share my various settings with colleagues and friends. However, I expect that not everyone is going to want to use my many idiosyncratic aliases and utilities. This means making my dotfiles modular, which is a little tricky --- it's not totally weird to want two separate repos that both end up putting files in .vim; however, doing that with git out of the box isn't really obvious.

For myself, I wanted a reasonably braindead way to bootstrap new machines and keep them in sync. It's not too unusual that I'll be doing a lot of work on a server for a few weeks and then stop touching it for months, only to login again and be frustrated that I've evolved how I work on my laptop and can't use similar shortcuts on the new box. Similarly, I tend to "clean home" every few months and then just work with what I've got for a while, touching-up the edges for a year or two before doing some serious remodeling. That means I need to be able to automate a lot of how my setup works so that it "just works" during the time I'm not focused on sharpening my tools.

The tricky part is that adding modularity makes the bootstraping/sync issue harder; I don't want to have to remember which 10 git repos to pulldown or update on a given box. I played a little with using submodules, but as always happens to me, I ended up making a little mess of my submodules within a few days; this reminded me that that tool doesn't mesh well with how I work.

Finally, because where I can, I try to use the most-specific tool to manage dependencies. For example, in a python project, you could use git submodules and shell scripts to manage all your dependencies, but pip has always been a lot less annoying for me. Similarly, vundle in git handles more than just checking-out and updating git plugins.

Where I ended up

It turns out, there are a ton of tools for managing your dotfiles, and github has even curated them. Initially, I was really interested in rcs and the python dotfiles package. But with rcs, my modularity goals were tricky to think through, and with the python package, given that I want all my python packages in virtualenvs, and part of that setup depends on my dotfiles, I decided to try to keep things simpler if I could.

I ended up using a set of shell scripts called homeshick to manage the dotfiles themsleves and myrepos to manage the many git repos. Homeshick handles symlinking files from a gitrepo into your home. It depends on bash, which means I can just clone it and be on my way. Symlinking has its own downsides, but the approach homeshick takes allows you to have a bunch of repos that all link into the same directories in your home, which solves my modularity problems. Then, myrepos handles cloning and bootstrapping the several repos where I keep things. I can have separate configs for myrepos for work and home (and even have one include the other), which is really useful.

I use these utilities to bootstrap tool-specific package managers, like zgen for manage zsh plugins or vundle for vim plugins.

What's cool

Now, I can run mr --trust-all bootstrap in my home and have all my basic tooling setup. Not only does this pull my personal tools, but also external dependencies like homeshick or spacemacs (which I've been playing with). Because myrepos is well-designed, at work I can have a .mrconfig that includes my home one and extends it. This allows for the sort of modularity and layering that I wanted.

I was able to pretty easily separate out my personal base setup from work-specific settings and keep all of this in sync. I can run homeshick pull and mr pull and have my tooling updated.

Also, with automated-setups, I'm starting to think about ways to make my own tooling more useful; for example, since I can create more "layers" of setup, I might want to break my .vimrc into some basic settings I want absolutely everywhere (like, even on my router), and then all the plugins I like using for programming and writing. I've got to think on this more though, since it will add complexity that I need to remember later.

What's uncool

First, I'm using many tools; maybe too many. The domain-specific plugin managers are useful since they usually do more than just manage git repos, but they also mean thinking about a bunch of tools in a specific order if I want to make sure I'm using the same vim plugins everywhere. This means it's not braindead enough yet; for example, if I've added some new zsh plugins, it's not always even obvious to me which tools I should run in which order to make sure they're updated in a given environment. I should be able to solve this with a shell-script or two, but I haven't done so yet (I didn't think of it until writing this, so +1 blogging I guess).

Having so many repos also makes sharing somehow harder; it's hard to point people to any one specific place if they want to use my tools. This blog post is in part to help with that, but I probably need to write another once my setup has settled down a little more to give a better overview.

Also, the homeshick approach of using symbolic links for everything means it can be hard to tell at a glance if I have untracked configs sitting around in my home. This is another thing that I can almost certainly fix with some shell scripting, but I haven't yet, and it's another place I can trip. Then there's all the problems that come with symbolic links generally; part of why I picked homeshick is that they have documented how to work around some of these problems, but just reading the docs should give you an idea of the woes you're headed for with the symlinking approach.

Finally, ironically, homeshick itself makes it slightly harder to share my configs, since homeshick requires that all the configs be in a REPO/home layout, which I don't think other people necessarily want. But, outside of work (where I don't think people will really mind), I don't really expect people to "fork" my dotfiles as much as copy them and use them for their own purposes, in which case, a slightly-weird directory-layout isn't so bad. And, on the positive, it makes it obvious where to put docs, if I ever write any.

What's next

Foremost, I need to just live with this for a few more weeks; I'm pretty happy with how things are working, but only time will tell whether the complexity is really worth it.

In the meantime, I want to smooth out some of the "what's uncool" — the biggest things are

  • making it easy to get an environment totally updated without having to run a ton of commands
  • being able to see which dotfiles aren't currently tracked somewhere
  • coming up with some sort of documentation for the various "castles" (homeshick's name for a set of tracked files) I have tracked

Resources and my repos

As I worked through this, I found the following really helpful:

I've got the following repos available for now:

  • .zsh -- zsh plugins and customization. This deserves its own blog post, because there's a fair bit of work in here.
  • .vim -- a (now pretty distant) fork of Byron Peebles's vim config, with about a million plugins, because I sure do like plugins
  • .myrepos -- the home of everything else
  • .git -- basic git configs and Tim Pope's automation for generating ctags

I still need to get my screen and tmux configs and a few others in, but they'll come before long.

Behind Every Website is a Hypothesis

Behind every website is a hypothesis about how to solve some problem people have. Being explicit about that yields benefits in clarity of thought, insights into process, and easier measurement.

In my last post, I talked a great deal about what an effective process for making a website looks like, describing a useful framework for approaching the complicated problem of making a large content site. I'd like to take a step back today, to thinking about why you might be creating a site in the first place.

Serious websites solve problems. If you're not solving a problem that real people have, you're wasting your time (and likely money). However, in my experience at least, people are not terribly explicit about what problems they are trying to solve. Behind every website, though, is some hypothesis (really a collection of hypotheses) about how the Web could be useful in solving some need real people have, and the site itself is an experiment testing that hypothesis. This applies to both content sites and web applications, although as usual I'll focus on content sites. Being explicit about the fact that you're making a hypothesis and conducting an experiment yields benefits in clarity of thought, insights into process, and easier measurement.

To give credit where it is due, the phrase "Web Design Is a Hypothesis" came from Alistair Croll and Sean Power in Complete Web Monitoring: Watching Your Visitors, Performance, Communities, and Competitors. Croll and Power focus mostly on local hypotheses — the idea that as designers work on specific pages and features, they should keep in mind that each decision they make is, at best, a guess as to what will work effectively for site visitors, and as such designers should plan to test their designs and refine them in the face of real world data. Later, reading Alan Cooper's "If users could lead innovation, they wouldn't be users," I came across his compelling analogy comparing web product designers to doctors, and realized that really, a website is a prescription to cure some diagnosed problem, and that web development is actually hypotheses all the way down, from the high-level product vision to the individual links on a page.

Your Site is an Experiment to Prove a Hypothesis, and That's the Way it Should Be

There are a lot of ways to conceive of a website. Typically, I see people think about it like a gadget — it's a cool thing like a kindle that will have a beautiful look, do some neat things, and reflect well on the people who guided its development. But that's not concrete enough. For content sites, the site is a vehicle for communication — more voice mail message than iPhone in the end.

There's a deeper level than just communication though — there's problem solving. You don't just communicate; that doesn't make any sense. You communicate to change the state of the world in some way. And if you want anyone to hear what you're communicating, you need them to have a reason to listen; you need to be helping them solve some problem, so that they can help you change the world the way you need it changed. So you make a hypothesis: if I develop a site that has « information, then «people will visit it and take « action and provide value « for us, because knowing «and doing « solves problem « for «. Then you can start pulling out things like beautiful design and fancy features, assuming that those things convey your information more effectively and allow the action to be taken more easily. But you need all those variables filled in — you need to know what people you're trying to reach, what information you have to offer them, what action you want them to take, and what problem they need to solve that your information helps them with. The goal here is to connect what real people need with what you have to offer. For web apps, this means some useful task that a computer can help people do. For content-centered sites, this means information that people need that you have a perspective on that you'd like to convey.

The greater the specificity with which you can fill in the blanks, the easier it will be to design an experiment to test your hypothesis, because your hypothesis will be more clear — and by "design an experiment," I mean "build a website." And using this way of thinking about your project will help guide the steps you take in executing it, helping you build better sites.

My Kingdom for an Example

For an example, I'll take my favorite fashion blog, I Found the Perfect (IFTP). IFTP takes on fashion like a design project, bringing to bear a formal creative process with tools like mood boards and careful resource planning to make expressing yourself through fashion easier for people who have an interest but not necessarily an inborn talent. Disclaimer: IFTP Is written by my lovely girlfriend, Laura, and I contribute photography and technical support — but I can talk about it without violating an NDA!

Although we weren't as explicit as I would be now, the site started with a couple hypotheses, one of which was

  • If Laura develops a site that «explains how to approach fashion problems like general design problems», then «women who are interested in fashion»will visit and «use the tools provided»and provide value «by helping us understand what tools are valuable to women who are interested in Fashion»because knowing «how to approach fashion problems like general design problems» and «using the tools provided»solves the problem of «the trouble of planning a personal "look"»for «women who are interested in fashion».

Now there's a higher level goal here — Laura wants to know which tools are actually valuable to women for other reasons, otherwise known as a "business goal." And that is again a hypothesis; Laura sees potential value in knowing which tools are valuable and understanding how to make it easier for other women to approach fashion, and whether she can really turn that knowledge into real value for herself will require other experiments and hypotheses. Because, as I said, it's hypotheses all the way down.

What if People Don't Realize They Have a Problem

It's possible that you want to solve a problem that either isn't top of mind for most people or that people don't even realize they have. I know when I first heard the concept of "wearable computers," I thought, "I don't want a stupid computer with me all the time; that sounds annoying." It took getting an iPod touch, which solved a specific set of computer problems well and was marketed to make clear what it could do for me; that convinced me that having a computer in my pocket was actually a great idea. Now, when my droid runs out of batteries, I feel hobbled without a wearable computer.

This is where tools like marketing and PR come in — sometimes, you need to make a case for the tools and information you're providing to convince people how you can help them. So, you end up making more hypotheses about how to convey the value of your site to people using the outreach you have available.

This is also a good signal to check yourself, though. First, if the problem you're trying to solve isn't obviously a problem or isn't one that many people care about solving, it means you're going to have more variables to understanding whether your site is working; you'll need to figure out how to measure your outreach efforts and then measure the performance of your site taking those outreach efforts into account. More importantly, you need to confirm that you're actually solving a problem and not just dreaming that lots of people have the same goals you do. If you think real people have problems like "they don't use my product" or "they don't visit my website," and that is the whole problem, then you're doing it wrong. Determining whether you're deceiving yourself about things that people need is complicated, though, and will have to wait for another post.

Working from a Hypothesis Motivates a More User-Centered Process

Stating your hypotheses explicitly shows that there are lot of variables in what you're doing. Once you realize you're solving a problem for some audience, you start to wonder "do they even have this problem?" Or, at least you do if you're me. You also might want to know what other, related problems they might have that you can help them solve. And then actually solving the problem well is going to require a good bit of work on its own — you need to be sure your site is actually usable by real people (not just you and your developers), that the information you're conveying is clearly articulated, etc.

This is where process comes in, and what motivates some of the steps in the process for developing a successful website that I described last time. A hypothesis is an educated guess; in this case, it's an educated guess about how I can solve some problem people having by creating a website. So, I want a well educated guess.

This desire to make a better educated guess makes me want to understand my audience better and focus my efforts on solving their problems; I know that's what I'm doing no matter what, so my process for creating the site should give me tools to understand my audience and validate my assumptions about what they need. I want to start learning about my audience so that I can understand how they understand the things my site will focus on. I want to know what tools they're currently using to solve the problems I'm interested in, to know with whom I'm competing, even indirectly. I want to use tools like usability testing to try to eliminate some of the variability in executing my site; I have enough to worry about with whether my my overall idea for the site is a good one.

Remembering my hypothesis also means that I realize, in the end, I am making an educated guess about what people will find useful. This means I'm going to need to (1) find some way to measure to know how good my guess was and (2) make changes to my site as after it's launched, because I'm going to learn things from my site's performance in the wild and therefore make better educated guesses in the future. First, this makes my analytics process a good bit more clear; if I'm thinking about my hypothesis at the outset, I'm less interested in vanity numbers like "visitors" and "pageviews" and more interested in how specific content is performing and how various segments of my audience are hearing about my site at all. This also means I want to find the easiest way to (in)validate my hypothesis that I can as early in the process as I can; tools like prototypes and generative user research become much more attractive when thinking about sites this way.

Realizing that the Hypothesis Matters Shows Process does not Ensure Quality

There's a flip side though. What really matters is the insight of your hypothesis and the strength of your execution. A good process can help you refine your hypothesis, and it can help hone your execution, but it cannot save you from a totally off-base hypothesis. It will, hopefully, help you identify where you're off base earlier and save you from sinking a lot of cost into a misguided "solution," but there's no guarantee. Poor execution can sink a good idea for a site, but great execution cannot save a site that doesn't solve a real problem. No amount of process can change that.

What's next?

Keeping the ideas from this post in mind can be useful for helping bring what you're trying to accomplish into sharper relief. I feel like the hypothesis template above ("If I develop a site that has...") could use some tightening though. There's also still the problem of figuring out how to detect when you're deceiving yourself about the reality or magnitude of the problem you're trying to solve. I'd love any suggestions on either of those topic in the comments.

Making Websites

Starting a website with comps and code is a bad approach for content-centered sites. Instead, start by clarifying your goals and aligning your team around them, working to understand your audience, and honing your messaging to suit your audience and goals. From there, there’s a whole production process that should include content strategy and user experience design before you actually build a site. Read on to learn about the motivations behind this process or skip ahead to see my ideal process outlined.

I have been making websites since 1995 or ‘96. My family got its first computer in 1995, and once I got confident using it, I wanted to make things with it. I scraped together some money from cutting grass and got myself a QBasic book, and a few months later my parents got me an HTML 2 book (no tables for this guy) for Christmas. I’ve made websites ever since; back then, it was for fun; in college, it was for classes and occasionally for a favor; for the last six years, it’s been how I’ve earned my living (with the occasional favor still thrown in).

From the get-go, I’ve approached web projects from the perspective of a developer; the QBasic book did come first. I’ve always wanted to dive in and make something, because I love making things. Because I have brought a programmer’s perspective to projects, I’ve thought in terms of execution; when I hear an idea, it’s hard for me not to imagine how to make it concrete with code. This has typically been what people expect out of talking about a website — except maybe the designers who see images of the site’s interface instead of the logic and software stacks I imagine. But people who want a website are typically excited about it, and they want to see progress quickly, and so my approach has met their hopes well. The Internet still has a hint of glamour; to people I help, getting a new website is a creative expression (even when it’s for their business), and they are excited to see their expression live on the Internet for all to see.

Over the last few years though, an awareness has been growing inside of me about some deficiencies in my process. I’ve always been looking to solve computer problems, but I’m coming to think that what I actually need to solve are people problems. So, here, I’ve taken a closer look at a naive (and, I suspect, commonly used) process for making websites, and then I've outlined the improved process I’ve been refining for the last few years. In this post, I’m going to focus primarily on the problems of content-rich sites without ecommerce or advertising; the method described in this post should also apply to other types of sites, but my experience and passion primarily deals with content-rich sites (so more Wikipedia than Amazon).

A Rough, Suboptimal Process for Making Websites

Boiling it down, I think I can define the process I’ve encountered in the past into nine "easy" steps:

  1. Talk to your client to get a rough idea of what s/he wants; this “talking” could actually be reading an RFP, or it could be a sit-down meeting or two.
  2. Make some drawings of what the site could look like. If you’re feeling ambitious, the first set of these drawings will be called “wireframes.”
  3. Implement in code the drawings that the client liked. If the discussions from step (1) yielded any features that go beyond interface (e.g., some fancy search, a game, etc.), write those as best you understand them.
  4. Post the content that the client has written in MS Word to the website you coded.
  5. Have the client click around the site a little, get his/her blessing, and take it live.
  6. Send some reports to the client about how many people looked at the site.
  7. Post updated content you pull from MS Word files every so often; these might be on a schedule but are more likely driven by outside events and freetime.
  8. One day, field uncomfortable questions about why more people don’t come to the site and why the people who do come don’t stay.
  9. Redesign part or all of the site (you may or may not get to participate in this step). This is step typically not mentioned earlier in the process.

I realize this is a bit of a caricature, but I don’t feel like it’s terribly far off the mark; I've certainly worked on sites with more thoughtful processes, but I've definitely seen sites go live that start at (3) and stop at (5).

What Motivates this Process?

This process has a couple things going for it:

  • For the client, it makes spending feel more predictable — you expect that the web developers can give you an estimate for the cost at (1) and will do work up to that estimate in (2) to (5).
  • It gets the web team making things early; in projects I’ve worked on, there is nearly always substantial deadline pressure, and nothing makes you feel better about your deadline than having the comps and code.
  • For the client, it makes a messy, complicated process feel like buying a normal product, like a car.
  • It minimizes the number of meetings the client has to attend and therefore appears to reduce the work required of the client in producing the site.
  • On sites without a ton of content or very complicated features, it moves quickly enough that there’s good enthusiasm from everyone through launch.
  • For the client, it appears to contain post-launch costs — that is, until you see step 9.

I can see why these features are attractive. No one has unlimited resources, so of course clients want to know what they’re getting for how much. And web teams want satisfied clients and projects that they can deliver on time and under budget, so it makes sense that they want to dive right in and get the project dispatched. It assumes that you can know what will be effective at the outset, which is nice for all parties; the client feels like they've hired experts, and the site creators feel like they are experts.

HOWEVER, and there is always a however, there are some problems here, too.

The Trouble with the Existing Process

The trouble is that the process described above dives in to tackle the technology problems associated with a website without handling the really important part: it ignores the needs of actual people. People are complicated and hard to predict. And although clients are typically excited about their sites once they get them (I’ve been lucky to work with good designers and to be able to provide good technology), eventually they come to realize that they don’t really know whether what they've paid for is actually successful; do people find it useful? You see numbers of visitors, but it’s hard to tell if it’s a lot or a little — and even if you have benchmarks, the numbers still feel meaningless. You see “engagement” numbers about stay times and bounce rates, but they always feel too low and too high (respectively) — not that you can have a great sense of what the ideal would be. It’s hard to know whether the content you’ve provided is useful or whether people are even actually reading it.

The above process elides some useful tools that we have at our disposal if we so choose when creating a content-rich website:

  • Although professional expertise is brought to bear on the design and implementation of the site, the content of the site is left to people who are likely not web content experts.
  • Although the process brings in outsiders (assuming you bring in a design/development firm), no one actually talks to the target audiences for the site to see what they could use.
  • Measurement and analysis only enter the picture as an afterthought.
  • The process assumes that the goals for the site are made explicit upfront and does not offer support for turning desires into measurable outcomes.
  • The process assumes that the site will work correct at launch; there’s no structured plan for determining how the site should be tested and incrementally improved.

My Ideal Process for Making Sites

A picture is worth a thousand words (not that I won’t write another thousand after the picture)

This process breaks into four major phases, and,importantly, it has a loop in it to imply that it is iterative:

  1. In the first phase (Inception, Audience Research, and Message Development), you figure out where you’re going, working to clarify your goals for the site and align your team to those goals (Inception), to figure out to whom you will be talking (Audience Research), and to express what you’re planning on saying (Message Development).
  2. In the second phase (Brainstorm Features, Content Strategy, and Outreach Strategy), you plan how you’re going to get there, putting together a picture of what exactly your new website needs to do to be able to reach your audience (Brainstorm Features), what content you’re planning to put on it and how you’re going to maintain that content over time (Content Strategy), and how your audience will learn about this content and contribute to it (Community and Outreach Strategy).
  3. In the third phase, you develop your first version of your website, starting with the navigation and interactive elements (User Experience Design), moving to the look and feel of the site (Visual Design), and then finally into the actual creation of code and copy in Production. Note that if you're doing things right, you're testing the outputs of this phase with real users to verify that you're making a product they can actually use.
  4. In the last phase, you analyze how you’ve done and look for insights for how to improve the site. Because analysis is an integral step and flows back into the others, you should have been planning for how to analyze your results in all the other phases so that your results here are more than just reports; they are actionable insights about what you need to improve on your site to meet your goals.
  5. From the analysis, you can brainstorm new features to make the site more effective, which will then change your content strategy and outreach strategy, which will then require you to modify your navigation and add or remove elements from the site, which means more design and then development to implement these changes. The process is inherently iterative, acknowledging that any website is at best a hypothesis about what an audience wants that can be improved through further analysis and refinement.

The diagram is of course an abstraction of a real process, and it disguises a lot of real complexity and fluidity in its neat boxes and arrows. For example, new ideas for features will emerge at each stage; content strategy, community strategy, and user experience design will happen almost concurrently and will often be hard to solidly differentiate; once production kicks off, it’ll likely run continuously for the life of the site, creating new features and content, modifying existing code and copy, or performing necessary maintenance. Additionally, it’s not like once you’ve done your audience research you’ll never return to it; however, I expect things will leach back into the audience research and messaging much more slowly than the rapid pace of iterations to improve the site.

That Looks More Complicated, What's It Get Me?

The big thing about this model is that it is goal and audience oriented; you start at inception by clarifying your goals and then dive directly into understanding your audience. The rest of the phases are basically about refining these understandings to reach your audience to realize your goals. These goals might be things like "teach people how to prevent crime" or "help activists promote my new program." The important thing is that you have a goal and then start breaking down the groups like "people" and "activists" into smaller groups that you can actually understand.

Additionally, the process proceeds to clarify the path from goals and audience to a site that you can measure and maintain. By including phases like content strategy, you will actually think in a structured way about what to include on your site and how to maintain it. And by thinking about outreach before the site is deployed, you can design your site to encourage outreach instead of figuring out how to retrofit it.

Finally, the process admits that what you create won't be perfect at launch. As Alistair Croll and Sean Power say, "Web design is a hypothesis." You will learn about your audience (and a bit about yourselves) after the site goes live; it is imperative that this real-world knowledge is analyzed and used to improve your site so that your site improves. It is wonderful to have experts writing your copy and developing your code; it's hard to imagine how you could turn insights into actions without professionals. HOWEVER, these professionals are going to be professionals in creating things for the web, not in your problem space or about your audience; you will learn with them what is actually effective in the real world by being a part of it with a live site that you monitor and analyze.

Ok, That Does Look Better, What’s Next?

Each of those boxes hides its own little process, full of work that needs to be done. Honestly, each box is its own discipline (yes, even inception, at least for some project managers). Additionally, thinking through many of these stages in terms of goals and eventual measurement is not natural for most people I meet; that’s where having people with experience running the show is really helpful. From these broad outlines, I’ve been working to develop a more detailed process for working your way from “I need a new site or to fix a really confusingly broken one” to “I have a vibrant, effective online communications program.” I look forward to sharing my approach as I refine it.