How to Boot Unix from a Network Server

Introduction
This text deals with the issue of setting up an assortment of Unices to boot from a server located on the same LAN. Many people consider diskless machines an anachronism, some marketing droids call them the ultimate solution to all administration problems and finally some people like me just prefer to stuff their disk server in a silenced room all the way down the hall.

This text starts with an introduction about the protocols used for network booting issues on a variety of platforms. Following it is another section organized by Unix variants. For each variant we first describe its net boot mechanism and its particularities, then continue with the client side and finish with particularities if you want to run a server on that platform. This ordering should provide the most useful if you're setting up some clients together with an existing server. If you're starting from scratch I'd recommend to start with the server.

For purely educational purposes this text may be somewhat inadequate, but if you're looking for that sort of information you might prefer W. Richard Stevens, TCP/IP Illustrated, Vol. 1 and the assorted man pages of the systems in question.

Protocols
First we list the protocols used in conjunction with network-booted machines. We don't describe the protocols but just supply two lists for each, one of the platforms serving that protocol and another listing the platforms using them on the client side.

The fields holding a "S" indicate that the platform provides a server for the given protocol. A "C" marks the services needed by a client to boot across the network. Finally, a "(C)" entry indicates that the client can optionally use the protocol.

Platform RARP BOOTP BOOTPGW BOOTPARAMD TFTP NFS
FreeBSD S S, C S, (C) S S, C S, C
FreeBSD 2.2.6
Running a FreeBSD Client
How Things Work
The boot process with FreeBSD looks like this:

* Start the boot ROM code for your network card.
* Use BOOTP to find out the IP address and netmask.
* Use TFTP to learn the root and swap file system location.
* Read the kernel from the root file system via NFS.
* Mount root and swap(?) file system via NFS.
* Continue the boot process as usual.

As a consequence you need a BOOTP, TFTP and NFS server for a network boot.

Hardware Preparations: Making Your Network Card Boot
The first thing you have to do is to make your machine capable to boot from the net. There are two possibilities: You may equip your network card with a boot EEPROM or you may stuff that EEPROM data on a boot floppy. At the time of this writing supported network cards are 3c509 and (with some minor makefile tweaking) 3c503 cards and the NE2000 clones family. See the netboot(8) man page and /usr/src/sys/i386/boot/netboot/Makefile for details.

You find pre-built ROM images in /usr/mdec/nb{8390,3c509}.rom in case you don't want to roll your own in /usr/src/sys/i386/boot/netboot. Just put them in an EEPROM, stuff it in your network card and be happy (hopefully, I haven't tried this myself).

If you rather want to use a boot floppy you can use DOS-style .COM files. You'll find them in the same place as the EEPROM images but their filename suffix is .com instead of .rom. You can either place them on a DOS floppy and start them from DOS or use a specialized .COM file loader available from the Linux folks. Make your favorite search engine look for the comboot package.

Server Configurations
You'll find some additional information about the boot setup in the FreeBSD Handbook. Here's a all you need for a more minimal setup, though --- no need to try to be fancy until things basically work.

BOOTP
You need an entry in the BOOTPD config that looks at least something like this:
snafu.devnull.ruhr.de:\ :ha=00.00.1c.1e.0c.9b:\ :ip=192.168.47.11:
They just match the hardware address with the IP address. Note that the hardware address must not be written with colons separating the bytes --- that's one of the problems using a cap-style data base file.

Additionally it may be necessary to add some of these tags:
:hn:\ :sm=255.255.255.0:\ :ds=192.168.47.:\ :ht=ether:\ :vm=rfc1048:The hn field makes the BOOTP server transmit the host name in its reply. This is pretty handy because the /etc/rc.network script uses this as the host name no matter what you put into /etc/rc.conf. As a consequence this allows you to use the same set of config files for a whole bunch of netboot clients.

The sm field sets the netmask for the IP address given. You may need this if you use subnets. The ds field defines the IP address of a name server. I haven't tried to use FQDNs instead of IP addresses in the assorted config files. Even if that worked out I'd feel pretty uncomfortable about it. The ht field defines the type of network hardware in use. Apparently 10 Mbit Ethernet is the default value, if you use anything else please see the bootptab(5) man page or equivalent or the latest Assigned Numbers RFC.

Finally, there are several BOOTP reply message formats. FreeBSD used the RFC1048-style one, so if your BOOTP server uses a different one by default you may have to set this explicitly.

TFTP
Next you need a file that holds additional configuration data. It is read via TFTP, so this file is usually placed in the /tftpboot directory by convention. Different to the notes in the FreeBSD Handbook a variety of names are checked in sequence, partly prefixed with a /tftpboot path, so you better stick with this convention. Just to be sure we go for the most specific one, which is freebsd.IP_ADDRESS_IN_DOTTED_QUAD in the /tftpboot directory, e.g. /tftpboot/freebsd.192.168.47.11 for the example above.

This file contains any of the assorted netboot commands necessary to make the machine boot. This is about all you need:
rootfs 192.168.47.1:/netboot/rootfs/snafu swapfs 192.168.47.1:/netboot/swapfs
The meaning should be obvious, except maybe for the swap file entry, which is explained below.

Additionally it may be helpful to specify the host name of the client here:
hostname snafu.devnull.ruhr.de
This may help to recycle the same root file system for multiple machines. Then again, it may not.

WARNING: The documentation in netboot(8) suggests that these parameters can be passed with the BOOTP reply. This doesn't work, you do need the TFTP file.

NFS
Next you need to make the root file system for the client accessible via NFS. How to to so depends on the platform your server is running on, so take a look at the related section for this issue. Just one note, though: It is probably necessary to disable root UID mapping (aka. root squashing).

Building the Kernel
Now you'll need to build a kernel for a diskless configuration. Make sure it supports NFS file systems. And what's worse, don't forget to configure its hardware (IRQs, DMA and I/O addresses) properly.

There are a couple BOOTP-related kernel configuration options available. I haven't figured out what exactly they're good for, but here's the relevant excerpt from /usr/src/sys/i386/conf/LING:
# Kernel BOOTP support options BOOTP # Use BOOTP to obtain IP address/hostname options BOOTP_NFSROOT # NFS mount root filesystem using BOOTP info options "BOOTP_NFSV3" # Use NFS v3 to NFS mount root options BOOTP_COMPAT # Workaround for broken bootp daemons.
Maybe they're some sort of help. Then again, maybe they're not.

Populating the Boot File System
Now you need to build a root file system for the machine. Take it from a "disky" installation, fix the config files in /etc and that should do. If you want to use the same root file system for both the server and client(s) you're best off making the hostname made available via the BOOTP or TFTP entries shown above. Then the hostname will be known to the machine before the /etc/rc* scripts are run. The /etc/rc.network script is smart enough to use that name instead of the one you've specified in /etc/rc.conf. Now if you're smart enough to use some `hostname -s` instead of an actual host name in the remainder of /etc/rc.conf you might well get away with a single set of config files.

Finally, you need to set up a swap file. This swap file is put in the directory specified in the TFTP-loaded file (/netboot/swapfs in our example above) and named swap.IP_ADDRESS_IN_DOTTED_QUAD, so in our example it would be /netboot/swapfs/swap.192.168.47.11. You need to create it manually using dd. Make sure it has the size you want to allocate for swap.

Open Questions
The netboot(8) documentation suggests that it is possible to boot a FreeBSD client machine solely via BOOTP and NFS, without actually using any TFTP. I assume that this would require some modifications (or just compile-time options I've missed?) to the netboot(8) code so the BOOTP values for the root and swap file system locations are used instead of the TFTP file contents.
Running a FreeBSD Server
rarpd
WARNING: I haven't tested the FreeBSD rarpd yet. I don't need it at this time, so I've just read the man page. Watch your butt here.

The FreeBSD RARP daemon is a stand-alone server and not run from inetd(8). It isn't configurable in /etc/rc.conf, you'll have to start it yourself somewhere.

It only answers requests for IP addresses that have a related file in /tftpboot unless you give it a -s flag. Unfortunately the file names searched don't match the naming convention used with the FreeBSD netboot(8) boot EEPROM code.

Furthermore it only answers on a single interface as specified as argument unless you give it a -a flag.

It needs a BPF device, so make sure your kernel is configured for this.

According to the man page the FreeBSD rarpd reads all its information from the /etc/hosts and /etc/ethers files. While the Ethernet configs support NIS the IP configs don't. I haven't checked this out (yet), but this seems a good deal of potential trouble. The /etc/ethers file just contains lines starting with the MAC address in hex with colons separating the octets, some whitespace and an FQDN (why on earth not an IP address?).
bootpd
The FreeBSD bootpd can be run either via inetd(8) and stand-alone. It detects this automatically, no options are needed (but available if you want to be sure). The configs are put in /etc/bootptab which is in a termcap(5)-style format. For a complete list of all available tags see the bootptab(5) man page. The most important are (from the man page):
ha Host hardware address ip Host IP address sm Host subnet mask sa TFTP server address client should use hn Send client's hostname to client tc Table continuation (points to similar "template" host entry)
Only the first two fields are mandatory, the hardware address (don't use colons between octets!) and the IP address. You may also need to set the subnet mask, too. Sending the hostname to the client may be extremely helpful if you want to use the same configs for a set of machines. And of course the tc tag may make things somewhat easier, too.

bootpgw
FreeBSD supports a gateway server for the BOOTP protocol. You run it with a single argument specifying the actual BOOTP server. For details, see the man page.

bootparamd
Yes, FreeBSD apparently has a SUN Solaris® style bootparamd. I haven't had a closer look at it, yet. The man page however is deceptively short.

tftpd
The FreeBSD tftpd is started by inetd. You'll find a (by default disabled) entry in /etc/inetd.conf. It specifies the /tftpboot directory as the base directory for all TFTP access. If you're using the FreeBSD netboot code you better stick with this convention---that's where the boot config files are searched for. Files are only served from this directory and its subdirectories. There are no config files except for that entry in /etc/inetd.conf