6.1. Binary files and the repository; possible problems

You have probably started to think about how you are going to store our binary files (the JPEG images) in your CVS repository. This is a tricky subject, and one worth spending some time on.

Obviously, binary files are different from ordinary text files. For one thing, you cannot perform meaningful diffs on them. You could not look at the binary differences between two JPEG files, and tell what the actual differences to the image were. This slightly reduces CVS' usefulness.

It is also not possible to use CVS revision strings in binary files. I haven't mentioned these yet, but suffice to say that if CVS encounters special keywords in a file it is checking out, such as $Header: /usr/local/CVSROOT/ezine/articles/200004/ready/managing,v 1.9 2000/04/01 19:50:47 gsutter Exp $, or $Id: managing,v 1.9 2000/04/01 19:50:47 gsutter Exp $, it replaces the keyword with information about the file, such as its path, or the date and time it was checked out. This is useful information to have, particularly when you want to quickly see what revisions of files you have checked out.

Despite this, storing binary files directly in the repository does have one big advantage. It's very easy to do, and you don't need to change any of your Makefiles to support it.

So, knowing that you should definitely store either the binary files, or a mechanism for regenerating the binary files, in the repository, how do you go about doing that?

---

6.2. The simple way, -kb

The easiest way to bring binary files in to your CVS repository is to use -kb when you cvs add the file. This tells CVS that this is a binary file, and it turns off a few pieces of CVS functionality for this file in the process. For example, if CVS knows that a file is a binary file, it will not try and merge in any changes that have been made to that file if you do a cvs update.

To add nik.jpeg to your repository, as a binary file (should you decide to do such a thing) you would run;

    % cvs add -kb nik.jpeg

As I say, this is simple, easy, and does not give me the chance to write any more about suffix rules. So I will ignore it for the time being. Don't run that example.

---

6.3. The more interesting way, with uuencode(1)/uudecode(1)

Suppose, however, that you prefer all the files in the repository to be text files (even if they should contain binary data). What do you do?

This is already a solved problem, thanks to the pervasiveness of e-mail. E-mail over the Internet is not safe for binary files. There are too many systems out there with differing ideas of how many bits in each character are significant to make it feasible to safely e-mail binary files without first encoding them. This encoding process converts the binary files in to files that only use alphanumeric characters, making them safe for e-mail. Of course the size of a file that has been encoded in this way can be considerably larger than the original binary file, but that's the tradeoff you have to make.

There are several possible methods you can use to encode files, but the grand-daddy of programs to do this is uuencode(1) (like uucp(1), I think the uu means "Unix to Unix", but I could be wrong). You use uuencode(1) to convert binary files to their text only equivalent, and you use uudecode(1) to convert the encoded text file back to the binary file. These programs are part of FreeBSD, and are almost certainly a part of whatever other Unix-like OS you are using.

Using uuencode(1) is quite simple. Suppose you want to encode nik.jpeg so that it becomes a text file, and store the results in nik.uue.

    % uuencode nik.jpeg nik.jpeg > nik.uue

If you investigate the nik.uue file, and work through the uuencode(1) manual page you will see how the conversion is done.

The command line above is relatively simple. However, you might be wondering why nik.jpeg had to specified twice on the command line.

The first occurence tells uuencode(1) the name of the file to convert. The second occurence tells uuencode(1) what this file should be called inside the encoded file. You can use this parameter to encode a file such that when it is decoded a different file is generated.

If you were to run;

    %
    % uuencode nik.jpeg foo.jpeg > nik.uue
    % uudecode nik.uue

You would see that the decoded file has been called foo.jpeg, because that is what was specified when encoding the file.

To tell you the truth, I've never had a need for this feature, and when running uuencode(1) you just get in to the habit of typing the filename you are encoding twice.

This is all very well, but how do we use it with make(1)?

If you are thinking ahead, you should be able to see that if we have a .uue file, we can now construct a series of commands to convert that file first to its original JPEG format, and then use the rules we have already created to convert the JPEG file to a GIF file.

A simple approach might look like this.

    #
    # Simple use of uudecode
    #
    
    # This is the default target
    nik.gif: nik.jpeg
            djpeg -pnm ${.ALLSRC} | pnmscale .25 | \
                        ppmquant -fs 64 | ppmtogif > ${.TARGET}
    
    nik.jpeg: nik.uue
            uudecode nik.uue

Put that code in a file called uue.mk (don't forgot about using TABs and not spaces in the rule body).

Generate the nik.uue file if you have not already done so, then make sure that you have deleted nik.jpeg and nik.gif. Then run make(1) on uue.mk.

    % uuencode nik.jpeg nik.jpeg > nik.uue
    % rm nik.jpeg nik.gif
    % make -f uue.mk
    uudecode nik.uue
    djpeg -pnm nik.jpeg | pnmscale .25 |  ppmquant -fs 64 | ppmtogif > nik.gif
    ppmquant: making histogram...
    ppmquant: 7454 colors found
    ppmquant: choosing 64 colors...
    ppmquant: mapping image to new colors...
    ppmtogif: computing colormap...
    ppmtogif: 64 colors found

make(1) has had to do a little more work this time. nik.gif is the default target, and that file doesn't exist. make(1) can see that nik.jpeg must exist first, in order to create nik.gif, but that file does not exist either. But make(1) can see that it can generate nik.jpeg from nik.uue. which does exist. So make(1) first runs the body of nik.jpeg target, which generates nik.jpeg, which can then be used to generate nik.gif. You did not have to specify the direct relationship between nik.gif and nik.uue, make(1) inferred that from the rules you gave it. If you recall, we covered this back in the second article.

We can use this information to update web.mk to teach it about .uue files, and create a suffix rule that will try and generate a .jpeg file from a .uue file as necessary.

The changes are very simple. First, you have to replace the SUFFIXES line with this one.

    SUFFIXES: .uue .jpeg .gif

As you can see, all this does is tell make(1) about a new suffix it can use. Then you have to add a suffix rule that tells make(1) how to go from .uue to .jpeg.

    .uue.jpeg:
            uudecode ${.ALLSRC}

Finally, you have to update the dependency line on the install target in web.mk.

    install: ${GIF} ${JPEG} _SUBDIRUSE

You might be wondering why you now need to explicitly list ${JPEG}.

Previously, you didn't, because it was assumed that all the JPEG files were part of the repository, and would therefore exist when you checked out a copy. However, now, that's not the case, as the JPEG files have to be created from the uuencoded files.

If (and only if) every JPEG file is also used to generate a GIF file then there's no problem, and you would not need to add ${JPEG} as a dependency. This is because if all the GIF files are dependencies, then the build process for the GIF files will ensure that the JPEG files will exist as well--they have to, otherwise you can not create the GIF files from them.

However, if you have some JPEG files that are only used as JPEG files, and are not listed as dependencies for any GIF files (for example, a foo.jpeg that was not used to generate foo.gif, but that still had to be installed) the JPEG file would not be created.

By listing ${JPEG} as an explicit dependecy, make(1) will ensure that all the JPEG files exist before attempting to install them, rather than just the ones used to create GIF files.

Were you expecting to add another variable to Makefile and web.mk, perhaps something like this?

    UUE=    nik.uue

There's absolutely no need. At the moment, the variables we have defined are used to determine which files should be installed. Because the .uue files are never installed, and because make(1) can work out the name of a .uue file based on the name of the .jpeg file it is required for, you do not need to explicitly list the UUE files.

---

6.4. A possible problem, and how to work around it

Suppose you have one or more GIF files that are not generated from JPEG files, but that have to be stored in the repository. You might think that you can uuencode(1) these files, and create another suffix rule that describes how to create a GIF file from a UUE file. Something like this perhaps?

If you try that, you're in for a shock, because it doesn't work. You will see output like this.

    % make
    uudecode nik.uue
    uudecode foo.uue
    [ -d ../mystage ] || /bin/mkdir -p ../mystage
    cp: nik.gif: No such file or directory
    *** Error code 1
    
    Stop.

You've been a victim of your own cleverness[4]. What's happened is that make(1) has tried to create nik.gif. Prior to adding the GIF to UUE suffix rule, the only way make(1) could do this was by first creating the JPEG file from the uuencoded file, and then converting the JPEG file to a GIF file - everything worked.

However, now make(1) has a new rule that describes how GIF files can be generated directly from uuencoded files. make(1) applies this rule, and has assumed that nik.gif can be generated directly from nik.uue, without having to go through the intervening JPEG stage.

This is obviously wrong, but make(1) doesn't notice until it tries to install nik.gif, which doesn't exist, because decoding nik.uue gives nik.jpeg instead.

You can work around this problem, but it requires a bit of a kludge.

You can then remove the generated files, and re-run make(1).

    # rm *.gif *.jpeg
    # make
    uudecode nik.jpeg-uue
    djpeg -pnm nik.jpeg | pnmscale .25 |  ppmquant -fs 64 | ppmtogif > nik.gif
    ppmquant: making histogram...
    ppmquant: 7454 colors found
    ppmquant: choosing 64 colors...
    ppmquant: mapping image to new colors...
    ppmtogif: computing colormap...
    ppmtogif: 64 colors found
    uudecode foo.gif-uue
    [ -d ../mystage ] || /bin/mkdir -p ../mystage

So first of all nik.jpeg-uue is decoded, and then converted to nik.gif, and then foo.gif-uue is just decoded, to produce foo.gif.

As I say, it's a bit of a kludge, but it works.

---

6.5. Extending this to other file formats

You can use this approach for any binary data you might want to sore in the repository, regardless of the format. The only steps you have to follow each time you add a new format are:

1. Choose a new suffix for the encoded format. These last examples have used dest-format-uue, but you are free to choose your own.

2. Update the .SUFFIXES: line, so that make(1) knows about the format.

3. Create a new suffix rule that describes the steps necessary to decode the format.

   There is a risk you might end up with a lot of suffix rules all with the same body. If this happens, you can always use the .USE directive to keep the body of the rule somewhere central. For example:

       .SUFFIXES: .jpeg-uue .gif-uue .au-uue .wav-uue
       
       .jpeg-uue: _UUEUSE
       
       .gif-uue: _UUEUSE
       
       .au-uue: _UUEUSE
       
       .wav-uue: _UUEUSE
       
       ...
       
       _UUEUSE: .USE
               uudecode ${.ALLSRC}