Non-obvious behaviors of shell syntax

When it comes to shell scripting there are some odd quirks and behaviours to the syntax that may not be immediately obvious. This post aims to collect my learnings of general shell weirdness, and maybe it will help you as well!

Escaping

When using a variable, you will always want to use double quotes when getting it's value. This is because otherwise your terminal of choice might decide to interpret a number of characters in it's own, delightfully destructive way.

An easy example can be given with looping over a variable's contents without using quotes.

list="one two three"

for word in $list
do
    echo "$word"
done

# Contrary to what you might expect this will output
# three variables in sequence:

## => one
## => two
## => three

Woops! Looks like your terminal just turned your nice string into a whitespace-delimited array. Which is clearly what you wanted all along, no?

To prevent this, you need only wrap the $list variable in quotes, like so

list="one two three"

for word in "$list"
do
    echo "$word"
done

# This will now output the expected string

## => one two three

So as an easy rule to remember: Unless you're on a treasure hunt for pain, just always wrap your variables in quotes.

Conditionals

Conditionals in shell scripting is super fun, because wrapping things in various bracket formats is totally what you expect to do. But those brackets hold deep dark mysteries that can easily trip you up when you want to do simple conditionals.

Let's do a simple number comparison. For our intents and purposes let's agree that 10 is bigger than 5. Now consider the following example

# conditionals.sh

ten=10
five=5

if [ "$ten" > "$five" ]
then
        echo "ten is bigger than five"
else
        echo "ten is smaller than five"
fi

The outcome should be quite obvious, no? Run the script and we should see ten is bigger than five. Easy!

And that's where you're wrong. Here's what the actual output is.

./conditionals.sh: line 6: 10: No such file or directory
ten is smaller than five

Wait, what? We're getting a file not found error and according to the shell 10 is smaller than 5. I don't know what math class you took, but in my world that's just all kinds of wrong.

What happened?

The first thing that went wrong is that we used a single pair of brackets. When we do that, the shell will automatically try to execute and compare to a file glob pattern and trips up because it can't find a directory named >. The easy way to fix this is either to use double braces like so: [[ "$ten" > "$five" ]] or just escape the greater-than symbol with a backward slash like so \>.

This will get rid of the first error, however you will still receive output telling you that ten is smaller than five. The problem here is that by using square brackets, the shell will automatically do a lexical comparison. And in the alphabet one and zero generally come before five.

Now there's two ways we can correct this, and I'll show you both.

# conditionals.sh

ten=10
five=5

# Solution 1
# This version uses the -gt switch to force
# arithmetic comparison.
if [ "$ten" -gt "$five" ]
then
        echo "ten is bigger than five"
else
        echo "ten is smaller than five"
fi

# Solution 2
# This version uses a pair of parentheses (( )) to force 
# arithmetic comparison.
if (( "$ten" > "$five" ))
then
        echo "ten is bigger than five"
else
        echo "ten is smaller than five"
fi

There are lots of other different comparison operators like -gt and you can find an extensive guide over at tldp.org along with the exact comparison syntax.

Functions

For the last chapter of this wild joy ride through shell behavior, we will take a look at functions. They too behave in ways that might be confusing if you come from other programming languages.

Calling a function

Contrary to basically every other programming language out there (save for some exceptions), bash does not use parentheses for a function call. You simply write the name. If you want to pass in some arguments, just add them with a space after the function call. It looks like this:

function foobar {
    echo "Foo to the bar"
}

foobar
foobar "argument1" "argument2"

Both calls will do the exact same thing. One thing to note is that you can pass any number of arguments to a shell function and it will not care. If you don't use arguments, they will simply be ignored. Another thing to be aware of is that you do not need to write parentheses after the function name. Your terminal simply doesn't care.

Using arguments

Unlike other programming languages, arguments in shell scripting are not defined in the function signature, but are rather used by using special $ variables, like so.

function foobar {
        argument_1=$1
        argument_2=$2
        argument_list=$@

        echo "First argument: $argument_1"
        echo "Second argument $argument_2"
        echo "List of arguments: $argument_list"
}

foobar "argument1" "argument2"

# Generated output
## => First argument: argument1
## => Second argument argument2
## => List of arguments: argument1 argument2

This also means that it's completely up to you to check whether an argument is actually set and throw an error if that is not the case. If you want to output nice error messages I highly recommend my post on coloring your terminal output. Ahem.

Variable scope

Another fun thing about shell scripts. If you define variables in your function like I did above, they are by default in global scope and can be accessed by anyone after the first time the function was called. Check this out

function foobar {
        argument_1=$1
        argument_2=$2
        argument_list=$@

        echo "First argument: $argument_1"
        echo "Second argument $argument_2"
        echo "List of arguments: $argument_list"
}

foobar "argument1" "argument2"
echo "Outside the function: $argument_1"

# Generated output
## => First argument: argument1
## => Second argument argument2
## => List of arguments: argument1 argument2
## => Outside the function: argument1

To prevent this behavior, you simply need to prefix your variables with the local keyword to prevent the function scope from spilling into the outside world. It looks like so

function foobar {
        local argument_1=$1
        local argument_2=$2
        local argument_list=$@

        echo "First argument: $argument_1"
        echo "Second argument $argument_2"
        echo "List of arguments: $argument_list"
}

foobar "argument1" "argument2"
echo "Outside the function: $argument_1"

# Generated output
## => First argument: argument1
## => Second argument argument2
## => List of arguments: argument1 argument2
## => Outside the function: <nothing>

And as grand finale, let's talk about return values!

Returning values

You'd think that using a simple return statement would be enough, yes? I hate to break it to you, but no. No, of course not. Because doing so would be entirely too easy. Instead what you do is to echo your return value and then capture the output into a new variable. This looks somewhat like this

function foobar {
        local argument_1=$1
        local argument_2=$2
        local argument_3=$3

        echo "$argument_1 $argument_2 $argument_3"
}

foobar_returned=$(foobar "foo" "bar" "baz")
echo "Foobar return value: $foobar_returned"

# Generated output
## => Foobar return value: foo bar baz

Phew. Makes you feel like somebody repairing their boat by putting duct tape over all the leaky holes.

Final thoughts

As you can see the world of shell scripting has lots of traps for unsuspecting people strolling past. I hope this post helps you avoid some of these traps in your own scripts. For a concise reference on most shell syntax you can check out learnxinyminutes.com. It's my go-to reference for odd quirks and things I just cannot seem to remember.

Enjoy, and good luck out there!~