Cross-Development with NetBSD

Hubert Feyrer <hubert@feyrer.de>

Introduction

When targeting a product for an embedded platform, it's not feasible to have all the development tools available on that same platform. Instead, some method of cross-compiling is usually used today. NetBSD 1.6 will contain (and NetBSD-current has today) a new framework to build both the operating system's kernel and the whole userland for either the same platform that the compiler runs on, or for a different platform, using cross-compiling. Cross-compiling requires assembler, linker, compiler, etc., to be available and built for the target platform. The new build scheme will take care of creating these tools for a given platform, and make them available and ready to use to do development work.

Grabbing sources

As an example, we will examine how to make a kernel for a Shark (StrongARM based system) from a machine running NetBSD/i386 in this article. The only requirement is that there is a fresh source tree of NetBSD-current checked out from CVS. "Fresh" here means that there should be no stale object files and such in the tree - these can really get in the way here! The following commands can be used to get a copy of the -current branch checked out in /usr/cvs/src-current:

	# mkdir /usr/cvs
	# cd /usr/cvs
	# env CVS_RSH=ssh cvs -d anoncvs@anoncvs.netbsd.org:/cvsroot co src
	# mv src src-current

The process of cross-compiling a kernel consists of three steps, which we will describe in more detail below:

1. Create the toolchain for crosscompiling (compiler, assembler, linker, ...)
2. Configure kernel
3. Build

Creating the Cross-compiler

The first step to do cross-development is to get all the necessary tools available. In NetBSD terminology, this is called the "toolchain", and it includes BSD-compatible make(1), C/C++ compilers, linker, assembler, config(8), as well as a fair number of tools that are only required when cross-compiling a full NetBSD release, which we won't cover here.

The command to create the toolchain is quite simple, using NetBSD's new src/build.sh script:

	# cd /usr/cvs/src-current
	# ./build.sh -m shark -u -t

This will build all the tools for the named target platform. Arguments used here are:

• -m: define the machine, MACHINE
• -u: update - only build files that have changed since last build
• -t: only build the toolchain (i.e. the cross-compiler!)

During the build, object directories are used consistently, i.e., special directories are kept that keep the platform-specific object files and compile results. In our example, they will be kept in directories named "obj.shark" as we build for a Shark as target platform.

The toolchain itself is part of this, but as it's hosted and compiled for a i386 system, it will get placed in its own directory indicating where to cross-build from. Here's where our cross-compiler tools are located:

	miyu# pwd
	/usr/cvs/src-current
	miyu# ls tools/obj.shark/
	tools.NetBSD-1.6-i386

So the general rule of thumb is for a given "host" and "target" system combination, the crosscompiler will be placed in the "src/tools/obj.target/tools.host" directory by default. A full list of all tools created for crosscompiling the whole NetBSD operating system includes:

	miyu# ls tools/obj.shark/tools.NetBSD-1.6-i386/bin/
	arm--netbsdelf-addr2line  arm--netbsdelf-strings    nbmakefs
	arm--netbsdelf-ar         arm--netbsdelf-strip      nbmakeinfo
	arm--netbsdelf-as         nbasn1_compile            nbmakewhatis
	arm--netbsdelf-c++        nbcap_mkdb                nbmenuc
	arm--netbsdelf-c++filt    nbcompile_et              nbmkdep
	arm--netbsdelf-cpp        nbconfig                  nbmklocale
	arm--netbsdelf-dbsym      nbcrunchgen               nbmsgc
	arm--netbsdelf-g++        nbctags                   nbmtree
	arm--netbsdelf-g77        nbeqn                     nbpax
	arm--netbsdelf-gasp       nbgencat                  nbpic
	arm--netbsdelf-gcc        nbgroff                   nbpwd_mkdb
	arm--netbsdelf-gcov       nbhost-mkdep              nbrefer
	arm--netbsdelf-ld         nbindxbib                 nbrpcgen
	arm--netbsdelf-lint       nbinfo                    nbsoelim
	arm--netbsdelf-mdsetimage nbinstall                 nbtbl
	arm--netbsdelf-nm         nbinstall-info            nbtexi2dvi
	arm--netbsdelf-objcopy    nblex                     nbtexindex
	arm--netbsdelf-objdump    nblorder                  nbtsort
	arm--netbsdelf-ranlib     nbm4                      nbuudecode
	arm--netbsdelf-readelf    nbmake                    nbyacc
	arm--netbsdelf-size       nbmake-shark              nbzic

As you can see, most of the tools that are available natively on NetBSD are also available, with some program prefix to identify the target platform. (The naming here is a bit redundant due to the directory structure containing all the information, but the program names are created by the GNU based toolchain and were chosen not to be changed).

One important tool that should be pointed out here is "nbmake-shark". This is a shell wrapper for a BSD compatible make(1) command that's setup to use all the right commands from the cross-compiler toolchain. Using this wrapper instead of /usr/bin/make allows cross-compiling programs that were written using the NetBSD Makefile infrastructure (see src/share/mk). We will use this make(1) wrapper in a second!

Configuring the kernel

Now that we have a working crosscompiler available, the "usual" steps for building a kernel are needed - config, then build. As the config(8) program used to create header files and Makefile for a kernel build is platform specific, we need to use the "nbconfig" program that's part of our new toolchain. That aside, the procedure is just as like compiling a "native" NetBSD kernel. Commands involved here are:

	# cd /usr/cvs/src-current/sys/arch/shark/conf/
	# /usr/cvs/src-current/tools/obj.shark/tools.NetBSD-1.6-i386/bin/nbconfig GENERIC

That's all. This command has created a directory "../compile/GENERIC" with a number of header files defining information about devices to compile into the kernel, a Makefile that is setup to build all the needed files for the kernel, and link them together. As the Shark port uses ELF as execution format but the Shark's OpenFirmware can only load a.out kernels, that Makefile will also convert the kernel from ELF to a.out once it's built.

More information about building NetBSD kernels can be found at http://www.netbsd.org/Documentation/kernel/.

Cross-compiling the kernel

We have all the files and tools available to cross-compile our ARM-based kernel from our Intel-based host system, so let's get to it! After changing to the directory created in the previous step, we need to use the cross-compiler toolchain's "nbmake-shark" shell wrapper, which just calls make(1) with all the necessary settings for cross-compiling for a shark:

	# cd ../compile/GENERIC/
	# /usr/cvs/src-current/tools/obj.shark/tools.NetBSD-1.6-i386/bin/nbmake-shark

This will churn away a bit, then spit out a kernel:

	...
	text    data    bss     dec     hex     filename
	1687520 69632   184576  1941728 1da0e0  netbsd.aout
	miyu# ls -la netbsd.aout
	-rwxr-xr-x  1 root  wheel  1757216 Mar 27 02:55 netbsd.aout
	miyu# file netbsd.aout
	netbsd.aout:    NetBSD/arm32 demand paged executable

Now the a.out(!) kernel can either be transferred to a shark (via NFS, FTP, scp, etc.) and booted from a possible hard disk, or directly from our cross-development machine using NFS. Be sure to actually use the a.out kernel, as the Shark's firmware cannot use the one in ELF format.

Cross-compiling the whole Operating System

Of course you can not only cross-compile the NetBSD kernel, but the whole system. This is as easy as:

	$ ./build.sh -m shark -d \
	                -D /usr/tmp/shark-root \
	                -R /usr/tmp/shark-release

This command will first build a cross-compiler, as described above. After that it will cross-compile the whole operating system including all libraries, binaries, etc. To make things complete, kernels for installing and running the systems will be built, and everything will be packed into distribution sets & install media, so a full NetBSD release is available in /usr/tmp/shark-release after that command!

Too easy? Sorry, but that's NetBSD! :-)

Summary

So much for our small example on how to use the new build framework to do cross-development of a kernel for and with NetBSD. Let me re-emphasize at this point again that the toolchain produced in the first step is capable of (re)building the complete NetBSD operating system including libraries and programs, not only the kernel. More information on building the whole operating system can be found in src/BUILDING. More documentation is also available as src/tools/compat/README, which has special emphasize on setting up cross-compiling from various host operating systems other than NetBSD, e.g., Solaris and Linux.

(c) Copyright 20020110 Hubert Feyrer <hubert@feyrer.de>
$Id: xdevnetbsd,v 1.5 2002/10/30 02:41:14 jwarner Exp $