Stronger Shell

tl;dr Shell scripting is strange and somewhat forbidding, but will serve you better than most other frameworks or languages you can learn (after your first). I'll explain how I improved and offer resources to improve yourself.

Motivation

Part of why I love programming is being able to automate repetitive tasks. That's pretty common for programmers; as the quip goes, I'll happily spend 99 hours to automated a task I could do by manually in 100. Strangely though, until I'd been programming professionally for a more than a few years, I was fine typing out many shell commands in a row every day to do repeated tasks.

As my jobs got more complicated though, my environment also did, and it became infeasible to remember everything that I needed to do in the terminal (so many server names to remember!) and too time consuming to keep searching my history all the time. Since scripting my simpler, day-to-day tasks in python wasn't really attractive, I decided to actively work to improve my shell scripting. This has paid off repeatedly.

Forcing Improvement

The trouble for me with improving at shell was bothering to start; although I knew eventually I'd get the time back, the first several scripts were going to take much longer than they'd ever save me.

Image Credit Randall Munroe / xkcd.com

Image Credit Randall Munroe / xkcd.com

Eventually, I decided to follow a rule that my friend Adam Hutton had mentioned: the second time you type a complex command in your terminal, make an alias. Aliases are quick to create and obviously save typing.

As soon as you start trying to do that though, you realize aliases, while wonderful, are pretty limited, and you stumble into shell functions and small scripts. Then, as a programmer, you are going to want flow control and variables, and that thankfully should start you hurtling through your own obsession to actually learning your shell thoroughly.

As my shell skills have improved, I've steadily widened the bar for what I will turn into a function/script/alias — I still apply the "if I type it twice, automate," but the scope of what "if I type it twice" has expanded considerably, because with scripts and functions, you can have arguments, and so automate nearly everything.

Resources for learning

Actually learning did not follow the path I expected; at first, it was hard to find recent books on bash and zsh at the right level for where I was, and shell scripting has quirks that didn't match my expectations, coming from Python/C++/Javascript. I eventually found a number of websites and books though that helped a ton, and collected a few tips below that would have sped me up considerably.

Online Resources

  • My first, favorite resource is the BashGuide wiki pages on wooledge.org, which is currently being updated to be a fuller tutorial at guide.bash.academy.
  • Joshua Levy's "The Art of Commandline" is a good, quick introduction that will get you moving
  • Explain Shell is a terrific tool for pasting complex commands and seeing what is going on with them
  • Of course, you can read man bash, which, although turgid, is full of useful wisdom. While you're reading the manual, you'll also want to read man test so that you finally understand what [ -f ~/.bashrc ] means.
  • Unix for the Beginning Mage (pdf) is a tutorial on the basics of shell work done as a story about learning magic. It's delightful, free, and reminds me of the why's (poignant) guide to ruby (which I credit with getting me started in ruby and subsequently web development), but more practical and less overtly weird. Thanks roneesh for pointing it out.
  • For contextual help at the commandline itself, there's a neat python app: search cmd. It lets you do things like searchcmd find "sort files by size" and will return relevant examples from places like stack overflow. Being able to search without leaving the terminal (and see the results summarized) nicely helps keep context, which is key for feeling productive. Thanks to Pycoders Weekly, which is a generally useful resource.
  • If you're looking for more examples of interesting people things do in their shell, commandlinefu has tons. Like any large repository, the quality is variable, but there's definitely lots of interesting snippets in there that can help you expand how you think about your shell.
  • For ZSH (which I love), there's a great bunch of examples and capabilities explained in the ZSH Lovers man page. Additionally, Nacho Caballero's Master Your Z Shell with These Outrageously Useful Tips really expanded my mind as to what was possible in ZSH, and I'm grateful that he wrote it. Via Pycoders Weekly.
  • Finally, your nearest linux or mac computer are FULL of shell scripts, and once you get past reading things like [[ -z $(ssh -T host "exit" 2> /dev/null) ]], you can learn a lot from those scripts. A good habit is to read the scripts that libraries tell you to curl and pipe to sh, like any of the scripts on curlpipe (since you don't want to just run some random shell script from the Internet without knowing what it's going to do, right?).

The other thing that may not be immediately obvious is that bash/zsh/whatever-shell-you-like is your language, and the rest of the POSIX tools are your standard library. With that in mind, it'll definitely be worth your time to pick up even a little grep, tar, awk, sed, curl, head, tail, ssh, and friends.

The DigitalOcean community has actually done a really nice job writing tutorials for awk, sed and a a fair number of other tools. For awk specifically, I'm a fan of the Grymoire awk tutorial which substantially demystified it for me.

If style matters to you (PEP8 has made it matter to me everywhere), the bash hackers wiki style guide is a helpful resource so that your scripts look fluent and avoid obvious errors. Similarly, you should setup a linter, like a linter like shellchck.

Books

No Starch Press has several excellent books for the aspiring shell user. My favorite is The Linux Command Line (affiliate link here and below, thanks!), which gives you a gloriously thorough and well-organized overview of many of the relevant shell commands you might need and, once you have those under your belt, shell scripting. If you're a bit farther along, some chapters may be too basic, but if you're like me, there are probably surprising holes in what you know that the book will fill in nicely.

Along similar lines to The Linux Commandline is Linux Command Line and Shell Scripting Bible, which is well-loved and thorough.

If you are just looking for shell scripting specifically, and you also have decided ZSH is awesome, Oliver Kiddle's From Bash to Z Shell is old but still useful; I refer to it from time to time when picking up new bits of ZSH still.

Finally, as you get more into this stuff, I highly recommend Michael W. Lucas's SSH Mastery (and honestly, any of his other sys admin books); if you're like me, you're probably spending a substantial portion of your shell time SSH'd into a remote host, and it's worth your time to really understand how SSH works and is configured.

Things that I wish I'd known

There are a couple things that I did not infer when I was initially trying to learn Bash from reading examples and experimenting, and I hope they speed someone else up in the future. They are also basically impossible to google, since they are mostly punctuation. Most can probably be picked up with a linter like shellchck, but I didn't have that setup at first either, so let my difficulty be your gain:

  • Bash is whitespace sensitive in a way that's... unexpected. You cannot have spaces around the operator in assignments (so export FOO = "bar" won't work, but export FOO="bar" is correct). You must have spaces inside your test expressions (so [[-z $FOO]] won't work; [[ -z $FOO ]] is correct).
  • There are a number of constructions that are nearly functionally equivalent. The ones that most tripped me up are:
    • [ expression ] and [[ expression ]] have the same function, and in general, if you're writing bash scripts, you should just use the [[ expression ]] version; the single [] version is older and doesn't support as many operations. And, to keep things optimally confusing, [] is the same as the test builtin. See the BashFAQ for the gritty details.
    • Single ticks (`command`) and $(command) are functionally equivalent, but in bash prefer $(command)the BashFAQ explains why.
    • Variable substitution can be done as either $VARNAME or ${VARNAME} — generally I do the latter, but you'll see both.
    • source and . do the same thing, which is execute the sourced file in the context of your current shell (so it can set environment variables in your interactive environment and whatnot).
  • Parens aren't exactly obvious either. (command) runs the command in a subshell (for uses, see the bash guide, or if you're impatient, just remember $(command) gets you the output of a command, so the command has to be running in those parens, right?). However, ((1+2)) does math (and $((1+1)) gets you the result of math). Having parens do something other than group expressions/override precedence was not something I expected.
  • Like our dear friend javascript, variables are global by default. In a function, to scope the variable to the function use the local keyword.
  • Bash (and other shells) has useful options that may make catching errors easier — for example, set -o errexit will "Exit immediately if a command exits with a non-zero status," which can be helpful in testing (and in real scripts). For the full list of options, see the set help page in bash or see the the options index for zsh. Thanks to bdunbar for the reminder.
  • If you switch between linux (GNU) and mac (BSD), many of your basic shell commands will differ; you almost certainly cannot copy some awk magic from stack overflow and expect it to work on both.

Where to go Next

The most important thing to do is write more shell code. The best way to learn is to do, and so, do! If you haven't yet, also pickup one of the great editors, either vim or emacs (or, my personal favorite, both via spacemacs) — once you're spending more time in your terminal, you'll likely want to be able to quickly pop into files to edit them, and it's nice to be able to do so right in the terminal.

Finally, make your shell environment nice. Spend some time with awesome shell (if you use bash) or awesome zsh plugins for ZSH users. Get a decent looking theme for vim. Setup nice completions for commands, and care for your dotfiles by versioning them. Spend a few hours making your environment pleasant and you'll be rewarded for years by that work.

Updates: