Answerman - Help, I've Fallen

by Gary Kline, David Leonard and Dirk Myers

This time we tackle some of the more basic problems that most newcomers will find helpful in learning some basic shell programming. This is an art which is simple enough for anyone to do, yet subtle enough to take years to fully master. We won't be able to make you a master, but we can get you started along the road to mastery. And, yes, by studying the examples and comments here, now you, too, will be able to claim : "I am a computer programmer!"

Before we begin, we'd like to offer some thoughts to help you approach shell programming with the proper frame of mind.

There are (at least!) two schools of thought on the way a computer ought to work. One school maintains that a computer ought to be a simple device, something like a toaster or blender -- easy to use for a specific function. The other school of thought maintains that a computer ought to be more like a box full of gears, pulleys, heating elements, wires, screws, brackets, strong cable, and what have you, so that, should you need a toaster, a blender, or an electric screwdriver, you already have all the parts you might need to build one. As you can see, this might make it quite a bit tougher to get your bread toasted the first time out, but in return for that, you can make an electric screwdriver... or a waffle iron, or a coffeepot, or whatever else you might need.

We lean toward the second view, though we still support the thought that computer systems are a bit hard to use.

(In all fairness, today's freely available Unix systems most often try to combine these two approaches and offer you a box of parts as well as a working toaster, blender, coffee maker, answering machine, and electric screwdriver. Nevertheless, it's good to know that if you suddenly need a garage door opener, the parts are still there. Assuming you are willing to take the time to put them together.)

Okay, to business. For this column, we'll assume that you know little if anything about programming logic, and that you are ready to learn some basics. As you read through the column, though, remember that all we're really doing here is putting together a set of parts that come with your BSD system.

One last thing: we're going to cover bourne-style (/bin/sh) syntax in this column rather than c-shell (csh) syntax. Which style to use has been a matter of debate for some people, but we (the authors) all use /bin/sh-style shells, so we're covering what we're familiar with.

So, in answer to the question:

How do I write a shell script?

we present the following tutorial discussion.

What's a Shell Script?

A shell script in its simplest form is just a set of shell commands saved in a file. When the script is run, the commands are executed just as though you'd typed them in from the command line.

(In fact, even in its most complex form, a shell script is just a saved set of commands. More complicated scripts, though, tend to contain a series of commands which would be time consuming to type at the command line, not to mention difficult to type correctly, in sequence, every time you wanted to have them run.)

The basic principle is that any set of commands you can type on the command line can be made into a shell script.

Script files

To write a shell script you need only two things:
     - ability to use the command line
     - ability to use an editor

Apply the basic principle: If you start editing an empty file and then type in commands just as if you were typing at a shell prompt, you'll end up with a script file!

Other operating systems call script files "batch files" or "command files".

For example, in the file named "printdate.sh" you might put:

	echo "Hello.  The current date and directory listing follow";
	date;
	ls -ls;
	echo "This directory is";
	pwd;

One command you might not be familiar with is 'echo'. This command is simple: it just copies its arguments to the terminal. This explanation might not be immediately clear to you, but running the script should make what 'echo' does obvious.

To run this script, just type:

% /bin/sh printdate.sh

The script should produce output something like:

harpo 18% sh printdate.sh
Hello.  The current date and directory listing follow
Thu Sep 21 19:10:46 PDT 2000
total 8
2 -rw-r--r--  1 dirkm  wheel  541 Sep 21 19:10 age
2 -rw-r--r--  1 dirkm  wheel   83 Sep 21 19:10 foo.txt
2 -rw-r--r--  1 dirkm  wheel   54 Sep 21 19:09 htm2html
2 -rw-r--r--  1 dirkm  wheel  110 Sep 21 19:02 printdate.sh
This directory is
/home/dirkm/octtest
harpo 19%

Congratulations. You have just created and run your first shell script.

More Power

In the script above, we used a completely 'canned' set of commands. No matter where you run the script, the same thing will happen. Worse, there's not much advantage to that script over just typing the commands. The polite banners are certainly nice, but we haven't saved much work over just typing 'date; ls -ls ; pwd;'.

Fortunately, there's more power than that lurking in the shell. You can use variables in your shell commands to do more complex and powerful automation. If you're not familiar with the concept of variables, just think of them as place-holders that get filled in by the shell. Take a deep breath, because we're going to cover a lot of territory in this next script.

Here's an example of a simple shell script that renames all files in the current directory that end with .htm so that the filenames end with .html instead:

	for f in *.htm
	do
		mv ${f} ${f}l
	done

In this case, f is the variable, and most of the magic happens in the line that begins with the word "for". To understand what happens on the "for" line, it helps to start at the right side.

The rightmost element on the "for" line is the expression *.htm. The shell expands this into a list of every file in the current directory with a filename that ends in '.htm'. The word "in" is a marker that tells the shell where the list begins (and also makes the line easier to read). The "f" is the name of the variable -- we chose "f" to stand for file.

Now, we've backed up to the "for", so let's read the statement "for"-wards [1]. Translated from shell to English, the statement reads:

Use f as a place-holder for every word in the list (which, in this case, is the list of files ending with *.htm).

f will be made a place-holder for each word in the list, one after the other, until the shell runs out of words. This isn't very useful in itself. Fortunately, we can also tell the shell to do something with the list. The shell will execute every command between the word "do" and the word "done" for each word in the list. Each time the shell executes the commands, the variable f will be a place-holder for a different value.

In the script above, we want to move files which end in .htm to files which end in .html. So, we construct the following ccommand:

    mv ${f} ${f}l

The ${f} tells the shell "substitute the value of f here". In other words, each time f is a place-holder for a new word, the shell runs a different command. Notice the final 'l'. If, at a given moment, f stands for 'mypage.htm', the shell will run the command:

    mv mypage.htm mypage.html

So, there's the magic revealed. Every file ending in .htm gets a new command, and with these few lines of script, you've saved yourself from typing one command for each file.

Now, that's power!

A Better Way To Run

If the script above were in a file were called "htm2html", it might look like this:

	-rw-r--r--  1 user  group  38 Sep 19 13:41 htm2html

And, as before, you could run the script like this:

	$ /bin/sh htm2html

However, it is more convenient to set up the script so that you don't have to tell /bin/sh to run every time you want to run the script -- remember, scripts are all about saving work.

It's easy to set this up. First, the file is edited so that it starts with this line:

	#! /bin/sh

This line specifies that you want this file to be a shell script run with /bin/sh. Next, you need to set "execute permission" on the file to tell the system that you want to be able to run the file directly. You do this by typing:

    
    % chmod +x htm2html

(If you're not familiar with the chmod command, don't worry about it now -- but take some time to read up on the chmod man page. It's an important command.)

Afterwards, the file looks like this:

	-rwxr-xr-x  1 user  group  49 Sep 19 13:42 htm2html

And can be now run like any other program:

	$ ./htm2html

You could omit the leading ./ if the script is placed in a directory named in $PATH. The first two characters of any standalone script file must be `#!' so that the kernel knows that this is a script file, and that the rest of the line is the name of the program to run to interpret the script.

Script Arguments

Its all very nice to run a file of canned commands, and it's even better to run a script that looks at what's going on around it and tears through a repetitive task in seconds, but what if you only wanted to rename a few particular .htm files to end in .html? Or, what if you wanted to rename .htm files in more than one directory?

You can pass command line arguments to shell scripts: they appear as the variables $1, $2, ... $9, and the variable $* represents all the command line arguments together.

For the htm2html script, it might be better to create the list of files from the command line argument. That way, whenever you run the script, you can decide which files you want to rename. So, we'll take out the *.htm on the "for" line, and replace it with $*.

The htm2html script now looks like this:

	#!/bin/sh
	for f in $*
	do
		mv ${f} ${f}l
	done

It can be invoked on some htm files like this:

	$ htm2html somedirectory/*.htm anotherdirectory/*.htm

Or, we can get back the "classic" behavior just by using:

    $ htm2html *.htm

Prompting

Arguments aren't the end of the story, though. Another way that scripts can get user arguments is by printing a prompt and expecting a reply. The shell's "read" command does exactly this.

Here's a script that finds out how old you are in days:

	#! /bin/ksh
	y=`date +%Y`		# today's date
	m=`date +%m`
	d=`date +%d`
	echo "Today is $d, month $m, year $y"
	read By?"What year were you born? "
	read Bm?"What month number were you born? "
	read Bd?"What day of that month? "
	age=$(( ($y-$By)*365 + ($m-$Bm)*30 + ($d-$Bd) ))
	echo "You are approximately $age days old"

Here, the read command asks questions, and assigns the answers to the variables By, Bm, and Bd. Notice, though, that we're using ksh instead of sh. We shifted gears (or shells, anyway) because the Korn Shell (ksh) offers support for mathematical expressions like `$((1+2))' -- or, in the script above, 'age=$(( ($y-$By)*365 + ($m-$Bm)*30 + ($d-$Bd) ) )'. The Korn Shell (ksh) will do the math on these variables. In the case of $((1+2)), the shell will convert the expression into '3'. For the age line, the shell will take 365 times your age in years, add that to 30 times your age in months, and add that to your age in days. (Ok, so it's not a perfect guess. We still like it as a fun little script.)

If you're stuck using plain 'sh', you'd have to use the expr(1) program to do the calculations.

The Shell Game

The fact that there are lots of different shells is one of the subtleties we mentioned at the beginning of the column. Each shell does things a little -- or a lot -- differently. To make matters more confusing, different versions of a shell may behave slightly differently. Some things that work for 'sh' on a BSD system may not work quite the same way on, for example, a Solaris system.

Traditionally, the solution for many of the problems was for a shell programmer to keep track of the differences and write scripts that used commands like uname(1) to figure out where they were running and react accordingly. These days, though, many systems come with a higher-level scripting language such as perl preinstalled, and even more systems have a scripting language installed as an option. These days, most people who have to write scripts that work on different types of systems prefer to use one of these scripting languages rather than shell scripts.

So, be aware that there are some subtleties, but don't worry about them unless it actually becomes a problem for you. When it becomes a problem, consider whether or not it's time to learn perl, or python, or tcl, or ruby, or... you get the picture. You'll find that the time you spent learning shell scripting will make learning another language easier [2].

Why not always use a scripting language? Because, for everyday tasks, it's often simpler to string together the commands you're already using rather than learning a new language. Even after you learn a scripting language, shell scripts often run more quickly -- and you can sometimes just paste your command history into a file to make a shell script, which makes the process of writing a script easy, too.

Don't underestimate the power of the shell.

Taking Control

You've already seen the "for" control statement. Three other control statements are "if", "while" and "case". These are not really shell commands, but rather statements that control the order in which the shell executes commands. The order in which commands execute is called the 'flow of control'. These statements are used to change the flow of control within the shell script.

For example, the "if" statement looks like this:

	if cp file1 file2; then
		echo "Copy was successful"
	else
		echo "Copy failed!"
	fi

If the command "cp file1 file2" exited with a success code (of zero), then the above script would write "Copy was successful" to the output. Notice that the 'if' statement ends with the word 'fi' ('if' spelled backwards). Like the 'done' statement that we saw as the ending marker for 'do', the 'fi' statement tells the shell where the 'if' statement ends. Don't leave it out, or your script won't run.

The while statement looks like this:

	while w | grep -q joe; do
		echo "Joe is still logged in..."
		sleep 60
	done
	echo "Joe logged out!"

Here, the pipeline "w | grep -q joe" is run, and the exit code of the last part of the pipeline (the grep) is examined. Grep exits with success if it finds "joe" on its input. (The "-q" is GNU for quiet -- this keeps grep from writing to the terminal.)

'sleep' is another command which you may not have seen if you haven't done scripting before. It simply waits for the number of seconds you specify.

The case statement is used to select one particular action based on how a string is matched:

	case "$USER" in
		fred)
			echo "Hello, fred"
			;;
		pierre)
			echo "Bonjour, pierre"
			;;
		bruce | sheila)
			echo "G'day, $USER"
			;;
		*z)
			echo "Hmmm. Your login name ends with a Z!"
			;;
	esac

The shell tries each possibility -- first 'fred', then 'pierre', then either 'bruce' or 'sheila', and last "anything that ends with z". When if finds a string that matches, it executes the commands that follow the string, then moves on past the 'esac' ('case' spelled backwards) which ends the statement.

Tests

The (often) built-in command called "test" is useful in control statements because it directly tests something you want and exits with success if it is true. For example:

	if test -e /etc/nologin; then
		echo "Logins are not allowed at the moment"
	fi

	if test "$USER" = "joe"; then
		echo "What are you doing here, Joe?"
	fi

	if test 123 -gt 456; then
		echo "Problem with reality: 123 > 456"
	fi

Two other commands are "true" and "false" which exit as you would expect. There are useful when used like this:

	CRITICAL_SITUATION=true

	# ...

	if $CRITICAL_SITUATION; then
		echo "Arghhh!!!!"
	fi

Here, the variable is set up in advance, (maybe changed later to false) and then expanded in the if statement.

Where Next?

The final step in shell scripting is to become as familiar as possible with the utilities on the system. Most of the power of shell scripting doesn't come from the shell itself; rather, the shell is just a way to connect up the huge variety of useful commands provided on your system. Anything in section 1 of the man pages is potentially useful -- to become an expert at shell scripting, knowing these commands is a must.

It's also worthwhile to read scripts on your system and figure out how they work. Most startup scripts will be shell scripts, and you might be surprised at the proportion of utilities on your system are actually simple shell scripts which call other programs (then again, depending on what you have installed, the proportion might be small).

http://www.oase-shareware.org/shell/ is a good reference for further information.

Our final advice is to enjoy yourself. If you've stayed with us this far, you're well on your way to building that electric hedge trimmer you've always wanted.

[1] Ok, we apologize for that one.

[2] On the other hand, if you don't want to learn more than one scripting language, or you spend a lot of time on non-Unix systems, learning a language which is the same across platforms can make a lot of sense. For example, of the three of us writing this column, Dirk tends to write more perl than shell even on Unix systems, because he also spends time on machines which have perl (he makes sure of that!) but which often don't have a shell (think Macs, here).

Even so, Dirk still writes shell, for the reasons mentioned above. And so do David and Gary, for that matter.

And--uh-oh, here's another AnswerMan news flash: Gary is finally learning perl. Don't quote this column, but the sky may be falling....

About the Authors

Gary Kline has been porting code since the late 1970's. When he isn't hacking code, he's hacking prose or quasi-poetry, or listening to jazz radio and slurping down espresso.

For going on four years he has been writing the software equivalent of a mind-machine, dubbed Muuz, and has already released some alpha code for FreeBSD. Check the FreeBSD ports tree if you are interested.

http://muuz.deadbbs.com will have the documentation and source under CVS.

[home| mail]

David Leonard is a PhD student in the Department of Computer Science and Electrical Engineering at the University of Queensland, Brisbane, Australia.

His area of research is QoS-adaptive component software architectures, and in his spare time is a developer for the OpenBSD project. That said, David enjoys living the quiet life with his wife, Kylie and cat, Mu. He especially enjoys frequenting Moreton Bay's many fabulous places to eat. Mmmmm!

[home| mail]

Dirk Myers works for a company that builds bar code scanners, making Unix systems at the company do things which are useful for document control. Much of his spare time is spent losing at Go, enjoying the central Oregon outdoors, and making Unix systems do things which are not useful for document control. He is married to Shanie, a scientist, and spends the rest of his spare time trying to understand the things his wife does with her time. Together, Dirk and his wife own large quantities of domestic animals, and they have recently acquired a house to put the animals in. Dirk is still looking for the man pages for the house.

[mail]