I initially tried to make the node filesystems essentially embedded systems, using Gentoo’s baselayout-lite, busybox, dropbear for SSH, etc. But baselayout-lite is really in its infancy — it’s hard-masked to reflect that, but I was hoping to ignore that problem. Brian Jackson wrote a brief howto on this setup. It was working reasonably, after gratuitous use of MAKEDEV to create miscellaneous devices, but the dependency resolution of the init scripts is way off and they can’t really restart or stop, and I’d rather be working on the cluster.
I spent a couple of days tracking down the last problem: an NFS filesystem that was mounted read-only just before chrooting from the initrd into the node root turned into read-write later in the process. But this only happened when it was mentioned in /etc/fstab. It ended up being in a C-based version of /sbin/rc that remounted root without the ‘ro’ option, and remount apparently mounts rw by default rather than respecting /etc/fstab or something crazy like that.
After about a week of digging through problems with this setup, I’ve decided to (at least temporarily) drop it for a uclibc-based setup that’s otherwise standard Gentoo.
First, I decided to make the node root switchable by having multiple roots in /diskless/nfsroot/<subdir>, with a symlink called ‘current’ pointing to the one I’m using. This eliminates the need to change multiple files (linuxrc and /etc/fstab) when I change the root — just one symlink.
Here’s what I’ve done so far in creating the uclibc-based root. Note that this is installed from a uclibc chroot within the master node’s glibc-based installation. The uclibc chroot has /usr/portage, /usr/local/portage and /diskless bind-mounted.
mkdir /diskless/nfsroot/uclibc
ROOT=/diskless/nfsroot/uclibc emerge baselayout bash openssh \
coreutils syslog-ng vixie-cron debianutils udev tar procps kbd \
findutils grep sed net-tools psmisc shadow -vab
chroot /diskless/nfsroot/uclibc /bin/bash
. /etc/profile
rc-update add sshd default
rc-update add syslog-ng default
rc-update add vixie-cron default
cd /dev
tar jxvf /lib/udev-state/devices.tar.bz2
MAKEDEV console
MAKEDEV std
MAKEDEV fd
Things are working reasonably well, but an awk call in an early init script can’t find stdout, and an attempt to create a device tarball at /lib/udev-state/devices.tar.bz2 at shutdown obviously fails because / is read-only.
I’m not sure how to fix the first problem, but for the second one I’ll probably either prevent the tarball creation or mount /lib/udev-state as tmpfs.
Striving for greatness 