Linux containers & PXE boot

Create the LXC container

LXC install

To install LXC support:

sudo apt-get install lxc uidmap debootstrap

# To check config is fine:
lxc-checkconfig

LXC user specific configuration
mkdir ~/.config/lxc
chmod a+rx ~/.local ~/.local/share

User permission mapping:

Get system-wide LXC user uid/gid mapping
cat /etc/subuid /etc/subgid

# Set user/group mapping (and default network), change uid/gid offset/range as needed
cat > ~/.config/lxc/default.conf << "EOF"
lxc.idmap = u 0 100000 65536
lxc.idmap = g 0 100000 65536
EOF

Create the container:

lxc-create -n netboot -t download
Distribution: mint
Release: tina
Architecture: amd64

Note: lxc will be located at: ~/.local/share/lxc/

Container customization

LXC is configured via:

We do need 2 network interfaces bridged (non NATed)

# Add quotas for br0/br1:
echo "
$USER veth lxcbr0 2
$USER veth br1 2" | sudo tee -a /etc/lxc/lxc-usernet

cat > ~/.local/share/lxc/netboot/config << "EOF"
# Network configuration
lxc.net.0.type = veth
lxc.net.0.link = lxcbr0
lxc.net.1.type = veth
lxc.net.1.link = br1

# We may also need to set:
#lxc.net.1.flags = up
#lxc.net.1.hwaddr = 00:16:3e:xx:xx:xx

# To also share host directory:
lxc.mount.entry = /media media none rbind 0 0
EOF

By default, LXC use lxcbr0 private bridge that is NATed to the host network
This allows to use any availble host interface.
Direct bridge could be use (to remove one NAT), but WLAN often do not support bridge.

Since we do use direct connection for eth1, we need to create the bridge:

# 2nd bridge is created manually
sudo brctl addbr br1
sudo brctl addif br1 eth1
sudo ifconfig br1 up

Note: Alternative would be to make the config persistant...

/etc/network/interfaces
auto lxcbr0
iface br1 inet dhcp
    bridge_ports eth1
    bridge_fd 0
    bridge_maxwait 0

Start the container

lxc-start -n netboot -d

# or, to run in foreground
# lxc-start -n netboot -F

Warning: Mint/Ubuntu images come with default user disabled by default!

We need to configure user / password before we can login

lxc-attach -n netboot
vim /etc/passwd
# Rename the default user
vim /etc/group
# change user group and sudo members
passwd user
# set a new password
# .. or rename and remove password (or '!') for the default user in /etc/shadow

# Rename user home dir:
sudo mv /home/ubuntu /home/user

# or directly apply changes to ~/.local/share/lxc/netboot/rootfs/*, e.g.:
# sudo vim ~/.local/share/lxc/netboot/rootfs/etc/passwd

Connect to container:

lxc-console -n netboot

Container network settings:

In recent Ubuntu, ifup/down has been replaced by netplan, switching back

More info | html

sudo apt install ifupdown
sudo apt remove nplan netplan.io
sudo vim /etc/network/interfaces
auto lo
iface lo inet loopback

auto eth0
iface eth0 inet dhcp

auto eth1
iface eth1 inet static
    address 10.0.0.1
    netmask 255.255.255.0
    network 10.0.0.0
    broadcast 10.0.0.255
#    dns-nameservers 192.168.8.1
# Restart networking
sudo service networking restart

# Start at boot, remove old stuff
sudo systemctl unmask networking
sudo systemctl enable networking
sudo systemctl disable systemd-networkd.socket systemd-networkd networkd-dispatcher systemd-networkd-wait-online
sudo systemctl mask systemd-networkd.socket systemd-networkd networkd-dispatcher systemd-networkd-wait-online

Container management

# to list containers
lxc-ls -f

# to stop the container
lxc-stop -n netboot

# to get container info
lxc-info -n netboot

# to delet a container
lxc-destroy -n netboot

Install boot servers

TFTP Server (tftpd-hpa)

PXE images are loaded via TFTP

sudo apt install tftpd-hpa
service tftpd-hpa status

TFTP is located at: /var/lib/tftpboot/

DHCP Server (isc-dhcp-server)

sudo apt install isc-dhcp-server
sudo vim /etc/default/isc-dhcp-server
INTERFACESv4=""
INTERFACESv4="eth1"
sudo vim /etc/dhcp/dhcpd.conf
allow booting;
allow bootp;
option client-system-architecture code 93 = unsigned integer 16;

class "PXE" {
    match if substring (option vendor-class-identifier, 0, 9) = "PXEClient";

    # For PXE requests, we provide the correct TFTP bootloader file to the
    # client (UEFI or legacy BIOS), based on request architecture
    if option client-system-architecture = 00:06 { # UEFI ia32 system
        filename "bootia32.efi";
    } else if option client-system-architecture = 00:07 { # UEFI x64 system
        filename "bootx64.efi";
    } else { # Nothing set, we assume legacy BIOS
        filename "pxelinux.0";
    }
}

subnet 10.0.0.0 netmask 255.255.255.0 {
    range 10.0.0.2 10.0.0.254;
    option subnet-mask 255.255.255.0;
    option domain-name-servers 192.168.8.1;
    option routers 10.0.0.1;
    option broadcast-address 10.0.0.255;
}

Restart DHCP:

sudo service isc-dhcp-server restart
service isc-dhcp-server status

# Now br1 can get an IP Address (on host!)
sudo dhclient -v br1

NFS server

Linux NFS server is in Kernel space. Since Kernel is shared, using NFS from LXC is complicated, so we are sharing NFS on Host

To install NFS server (on host):

sudo apt install nfs-kernel-server

Share directory

sudo vim /etc/exports
/media/iso 10.0.0.0/255.255.0.0(ro,sync,no_subtree_check)

Restart server and test it

sudo service nfs-kernel-server restart
service nfs-kernel-server status

sudo mount.nfs 10.0.0.1:/var/lib/tftpboot/iso /mnt/nfs
sudo umount /mnt/nfs

Note for LXC: html

Network routing

Router provides WAN access (and eventually NAT) to the 2nd interface:

echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward

# NATing between eth0 and eth1
sudo apt install iptables
sudo iptables -A FORWARD -i eth1 -j ACCEPT
sudo iptables -t nat -A POSTROUTING -o eth0 -j MASQUERADE

# Todo: NAT is not required, route on host should be sufficient.

Linux Live

Linux ISO booting

Note for netboot: html

Files required for network boot are

Syslinux supports (unsecured) UEFI boot, however, secureboot enabled hosts mandate a signed bootloader.
Ubuntu provides signed x64 UEFI Grub image able to boot Linux, but Grub does not support Legacy BIOS PXE boot

PXE syslinux (Legacy BIOS)

Ubuntu provide a netboot installer: A minimal system (kernel + init ramdisk) is booted, and package are downloaded from internet.

cd /var/lib/tftpboot/
sudo su
apt install wget
mkdir ubuntu-netboot pxelinux.cfg
wget http://archive.ubuntu.com/ubuntu/dists/devel/main/installer-amd64/current/images/netboot/pxelinux.0
wget http://archive.ubuntu.com/ubuntu/dists/devel/main/installer-amd64/current/images/netboot/ubuntu-installer/amd64/boot-screens/ldlinux.c32
cd ubuntu-netboot
wget http://archive.ubuntu.com/ubuntu/dists/devel/main/installer-amd64/current/images/netboot/ubuntu-installer/amd64/linux
wget http://archive.ubuntu.com/ubuntu/dists/devel/main/installer-amd64/current/images/netboot/ubuntu-installer/amd64/initrd.gz

vim pxelinux.cfg/default
pxelinux.cfg/default
default mint

label mint
  menu label Live Linux ^Mint
  menu default
  kernel linuxmint/vmlinuz
  append initrd=linuxmint/initrd.lz boot=casper netboot=nfs nfsroot=10.0.3.1:/media/iso/linuxmint net.ifnames=0 biosdevname=0 --

label install
  menu label ^Install Ubuntu from Internet
  menu default
  kernel ubuntu-netboot/linux
  append vga=788 initrd=ubuntu-netboot/initrd.gz ---

label GParted Live
  menu label ^GParted Live
  kernel gparted/vmlinuz
  append initrd=gparted/initrd.img boot=live config components union=overlay username=user noswap noeject netboot=nfs nfsroot=10.0.3.1:/srv/nfs/gparted toram

label GParted Live (http)
  menu label GParted Live
  kernel gparted/vmlinuz
  append initrd=gparted/initrd.img boot=live config components union=overlay username=user noswap noeject ip= vga=788 fetch=http://192.168.8.253/gparted/filesystem.squashfs

Alternative: Use full netboot package (install in TFTP root)...

wget http://archive.ubuntu.com/ubuntu/dists/eoan/main/installer-amd64/current/images/netboot/netboot.tar.gz

Grub (UEFI)

Let's boot the same kernel / initrd from UEFI system (using GRUB)

cd /var/lib/tftpboot/
sudo su
wget http://archive.ubuntu.com/ubuntu/dists/trusty/main/uefi/grub2-amd64/current/grubnetx64.efi.signed
ln -s grubnetx64.efi.signed bootx64.efi
mkdir grub
vim grub/grub.cfg
grub/grub.cfg
menuentry "Live Mint" {
  linux  linuxmint/vmlinuz boot=casper netboot=nfs nfsroot=10.0.3.1:/media/iso/linuxmint net.ifnames=0 biosdevname=0 --
  initrd linuxmint/initrd.lz
}

menuentry "Live Ubuntu" {
  linux  ubuntu-live/vmlinuz ip=dhcp boot=casper netboot=nfs nfsroot=10.0.3.1:/media/iso/ubuntu nosplash --
  initrd ubuntu-live/initrd
}

menuentry "Install Ubuntu from Internet" {
  linux  ubuntu-netboot/linux
  initrd ubuntu-netboot/initrd.gz
}

menuentry "Live GParted" {
  linux  gparted/linux
  initrd gparted/initrd.gz boot=live config components union=overlay username=user noswap noeject netboot=nfs nfsroot=10.0.3.1:/media/iso/gparted toram
}

Content shares (tftp and NFS)

The content can be shared from ISO image:

wget https://downloads.sourceforge.net/gparted/gparted-live-1.0.0-5-amd64.iso

# @ host:
sudo mkdir -p /media/iso/ubuntu /media/iso/linuxmint /media/iso/gparted
for dist in `ls /media/iso`; do
  sudo mount -o loop,ro ~/Downloads/iso/$dist*.iso /media/iso/$dist
  sudo exportfs -o rw,sync,no_subtree_check,insecure,no_root_squash 10.0.0.0/255.255.0.0:/media/iso/$dist
done

# @ container (copy kernel/initrd to tftp server)
sudo su
mkdir /var/lib/tftpboot/{ubuntu-live,linuxmint,gparted}
cp /media/ubuntu/casper/{vmlinuz,initrd}  /var/lib/tftpboot/ubuntu-live
cp /media/linuxmint/casper/{vmlinuz,initrd.lz} /var/lib/tftpboot/linuxmint
cp /media/gparted/live/{vmlinuz,initrd.img} /var/lib/tftpboot/gparted/

To stop sharing:

for dist in `ls /media/iso`; do
  sudo exportfs -u 10.0.0.0/255.255.0.0:/media/iso/$dist
  sudo umount /media/iso/$dist
done

Recap

Once installed:

# Start bridge
sudo brctl addbr br1
sudo brctl addif br1 eth1
sudo ifconfig br1 up

# Mount/share ISO images
for dist in `ls /media/iso`; do
  sudo mount -o loop,ro ~/Downloads/iso/$dist*.iso /media/iso/$dist
  sudo exportfs -o rw,sync,no_subtree_check,insecure,no_root_squash 10.0.0.0/255.255.0.0:/media/iso/$dist
done

# Start container
lxc-start -n netboot -F

# From container, start NAT
sudo iptables -A FORWARD -i eth1 -j ACCEPT
sudo iptables -t nat -A POSTROUTING -o eth0 -j MASQUERADE

# Once done ...
sudo shutdown now

for dist in `ls /media/iso`; do
  sudo exportfs -u 10.0.0.0/255.255.0.0:/media/iso/$dist
  sudo umount /media/iso/$dist
done

Tips

# check distribution
cat /etc/os-release

# check memory:
sudo dmidecode --type 17

# restore grub from Live CD
sudo mount /dev/sda5 /mnt
sudo grub-install --recheck --root-directory=/mnt /dev/sda
06-Dec-2019