[Gentoo] Setting up a diskless cluster, volume 1

I’ll be putting together a small, diskless, mixed-node Gentoo cluster this week and hopefully detailing my progress on here. I recently got some interesting ideas from a fellow dev, Christian Zoffoli, on the gentoo-cluster mailing list. He had a setup something like this:

What it should do:

  • Power on.
  • Start PXE — look for DHCP, TFTP servers.
  • Download the kernel and a uclibc initrd.
  • Mount an NFS shared image (one per cluster).
  • Autoconfigure.
  • Start a full Gentoo system.

Technical details:

  • Create a diskless boot setup. Set up DHCP, TFTP and NFS servers. The DHCP server tells nodes their hostnames and where to go next to get a kernel. (Remember to use -H to dhcpcd on the client or else it ignores the hostname.) The TFTP server provides the kernel and initrd, and the NFS server provides a full root filesystem.
  • Create a layer using an embedded environment so we can have a diskless node up and running quickly. ROOT support in Gentoo’s Portage should allow us to do this fairly easily.
  • The linuxrc within the initrd does much of the real work.
    • Mount /proc.
    • Read the hostname from /proc/sys/kernel/hostname.
    • Parse /proc/net/pnp to find the boot server.
    • Download a more complex linuxrc (linuxrc.real) from a TFTP server, so there can be a simple, statically generated initrd.
    • Execute linuxrc.real
      • Mount the NFS root with a full Gentoo installation in a common path [COM_P1].
      • Mount a tmpfs in another common path within the prior one [COM_P2].
      • Generate resolv.conf, hostname, dnsdomainname, etc. in $COM_P2.
      • Download SSH keys (common to all nodes) to $COM_P2.
      • Mount a tmpfs in /tmp, /var/lock, /var/run, /var/lib/init.d and /var/log. The logger sends messages to a primary logger on the master.
      • Link $COM_P2 to the real location (in $COM_P1).
      • Link and download wathever you need.
      • Switch roots to the NFS share using pivot_root.
      • Execute /sbin/init in $COM_P1 to start the normal Gentoo initialization.