Project NEALE

Finnix, the LiveCD for system administrators

Project NEALE (Normalized Extraction and Assembly of LiveCD Environments) is a set of procedures to build Finnix from a low level in a consistent manner. It starts with a minimal Debian bootstrap, adds Finnix packages and metapackages, and then assembles the environment into a final ISO.

Project NEALE has several advantages over the old, manual method of building Finnix releases, including:

  • Consistent builds are made every time. There is no risk of outdated files left from a previous release being carried over to a new release.
  • The risk of major differences between architectures is greatly reduced.
  • New architectures may be developed more easily.
  • Automated build tests of Finnix may be performed, detecting failures related to changes in upstream Debian testing over time.

Background

Throughout its history, most Finnix releases were assembled by hand to varying degrees. Finnix 0.03 was created by modifying an existing Red Hat Linux installation manually, though this was at a time when Finnix was one of the first LiveCDs to be produced, so there was little precedent.

Finnix 86.0 was based on a Debian bootstrap, and all Debian-based versions of Finnix relied on project management using Debian's own tools (apt repositories, etc). However, releases were made by taking the previous release, bringing upstream software up to date, and adding Finnix-specific updates. Essentially, each Finnix release was a remaster of the previous release.

Finnix 105 was the first Finnix release to be built using NEALE; no previous Finnix release was used as a base. All software within the release was downloaded and installed via apt repositories.

Procedure

Most of the work is handled by a script, build-neale, which does the following tasks:

  • Downloads and extracts Finnix-specific base components (kernels, initial ramdisk layout, general CD layout, miscellaneous files ("stuff")).
  • Installs a Debian bootstrap.
  • Adds temporary apt repositories for Debian and Finnix (most likely using a cached proxy, see below).
  • Installs the package finnix-base, which is a metapackage for the Finnix GPG keyring and a few base packages.
  • Installs the package finnix-standard, which is a metapackage for all desired software in the Finnix release, including Finnix-specific packages.
  • Replaces the temporary apt repositories with the permanent equivalents.
  • Moves the kernels into place.
  • Hands off to finnix-build-stage1 and finnix-build-stage2, which are responsible for building the compressed root filesystem and final CD, respectively.

Installation

The bulk of build-neale is available at http://archive.finnix.org/finnix/project/neale/build-neale.remote . This is a special shell script which is inline GPG signed by the archive key. To properly use this script, you will need to write a bootstrap script which will check for and optionally download updates for it, verify the signature is correct, and run it.

First, install build dependencies. Finnix does not require much software to build, most of which will be installed on any system. However, a few non-standard packages will be needed, particularly mksquashfs, which requires at least version 4.2. If any required software is missing, build-neale will tell you. Here are the packages you will most likely need beyond what is on a standard system:

root@host:~# apt-get install debootstrap genisoimage squashfs-tools xz-utils uuid-runtime binutils

In addition, there are a few optional pieces of software which can be used during a build:

  • isohybrid (Debian syslinux package, will be used automatically if present, x86 only)
  • implantisomd5 (Debian isomd5sum package, will be used automatically if present)
  • vanityhash (not packaged in Debian but provided in Finnix, will be used if "FINNIX_VANITYHASH=abc123" is exported to stage2)

Next, grab the Finnix GPG keyring and import it. You will want to add this to the root user's keyring, as build-neale requires root privileges to run (since it relies heavily on chroots).

root@host:~# wget http://archive.finnix.org/finnix/project/finnix-keyring.gpg
root@host:~# gpg --import finnix-keyring.gpg 
root@host:~# gpg --fingerprint A89BA58D
pub   4096R/A89BA58D 2012-05-14
      Key fingerprint = 3954 B91A 987B 0B68 17E3  FBFF 8087 F4ED A89B A58D
uid                  Finnix Archive Signing Key <keymaster@finnix.org>

Next, create a bootstrap script. It is suggested this be named build-neale:

#!/bin/sh

# Export NEALE/Finnix specific environment variables here, if needed

set -e
DESTDIR="$(dirname $0)"
wget -N -P "$DESTDIR" http://archive.finnix.org/finnix/project/neale/build-neale.remote
gpg --verify "$DESTDIR/build-neale.remote"
chmod 0755 "$DESTDIR/build-neale.remote"
exec "$DESTDIR/build-neale.remote" "$@"

Then simply run it!

root@host:~# build-neale

By default, build-neale will work with its files within the directory /srv/finnix/neale. build-neale will take a long time to run, expect it to take between 20 minutes and an hour. When it is done, assuming everything went well, you should have an ISO at /srv/finnix/neale/finnix-neale.iso on x86 systems, or /srv/finnix/neale/finnix-neale-ppc.iso on PPC systems.

Caching proxy

build-neale downloads a lot of data, at least 300MB of debs. NEALE-specific downloads are handled in an efficient manner; it uses "wget -N" to check the remote timestamps against the local copies, and will only download if the remote archives have changed. However, the majority of the downloads come from apt downloads. It is strongly recommended you install a caching proxy server on your local network.

Here is a working example for Squid 3, which can also be trivially modified to work with Squid 2.7. It is based loosely on Ubuntu's squid-deb-proxy package:

visible_hostname squid-deb-proxy
maximum_object_size 16 MB
cache_mem 96 MB
maximum_object_size_in_memory 2 MB

# refresh pattern for debs and udebs
refresh_pattern deb$   129600 100% 129600
refresh_pattern udeb$   129600 100% 129600
refresh_pattern tar.gz$  129600 100% 129600
refresh_pattern tar.bz2$  129600 100% 129600
refresh_pattern tar.xz$  129600 100% 129600

# always refresh other files
refresh_pattern . 0 0% 0

acl all src 0.0.0.0/0.0.0.0
acl sdp dstdomain squid-deb-proxy
acl debian urlpath_regex ^/debian
acl finnix urlpath_regex ^/finnix

http_port 8090 accel defaultsite=squid-deb-proxy

cache_peer mirrors.kernel.org parent 80 0 name=debian no-query no-digest originserver forceddomain=mirrors.kernel.org
cache_peer_access debian allow debian
cache_peer_access debian deny all

cache_peer archive.finnix.org parent 80 0 name=finnix no-query no-digest originserver forceddomain=archive.finnix.org
cache_peer_access finnix allow finnix
cache_peer_access finnix deny all

http_access allow sdp
http_access deny all
icp_access deny all

cache_dir aufs /var/spool/squid3 500 16 256

Once this is installed, tell build-neale to use this caching proxy for the initial downloads, by adding exports to your build-neale bootstrap script:

export NEALE_DEBIAN_INIT_MIRROR="http://$PROXY_IP:8090/debian"
export NEALE_FINNIX_INIT_MIRROR="http://$PROXY_IP:8090/finnix"

Replace $PROXY_IP with the IP of your caching proxy. These variables are used by build-neale to do the actual deb downloads. Once the downloads are complete, the "real" public Debian and Finnix mirrors are put in the build's /etc/apt/sources.list, so the final build does not contain mention of the local caching proxy.

One-off builds

Due to the length of time needed to do a proper NEALE build, it is often desirable to quickly test changes by manipulating a previous build. build-neale does not clean up the build area after its run (but does before), so you can make changes and rebuild a previous build made with NEALE. Since NEALE uses different directory paths than the normal remastering environment, the instructions in that guide are not directly applicable. These wrapper scripts help with a comparable environment.

neale-oneoff-finnix - chroot into the NEALE root filesystem, with development options enabled (start-stop-daemon protection, etc):

#!/bin/sh

export FINNIX_BASE="/srv/finnix/neale/build"
export FINNIX_SOURCE="$FINNIX_BASE/source/FINNIX"
export FINNIXDEV=1

exec /usr/sbin/chroot "$FINNIX_SOURCE" /bin/bash -l

neale-oneoff-stage1:

#!/bin/sh

export FINNIX_BASE="/srv/finnix/neale/build"
export FINNIX_SOURCE="$FINNIX_BASE/source/FINNIX"

exec "$FINNIX_SOURCE/usr/sbin/finnix-build-stage1"

neale-oneoff-stage2:

#!/bin/sh

export FINNIX_BASE="/srv/finnix/neale/build"
export FINNIX_SOURCE="$FINNIX_BASE/source/FINNIX"
export FINNIX_ISO="/srv/finnix/neale/finnix-neale-oneoff.iso"
export FINNIX_VANITYHASH
export FINNIX_GPGUSER

exec "$FINNIX_SOURCE/usr/sbin/finnix-build-stage2"

AMD64 64-bit userland Finnix builds

Finnix for x86 includes a 32-bit userland, as well as both 32-bit and 64-bit kernels. A 64-bit kernel can use a 32-bit userland, and still take advantage of 64-bit specific features this way (increased total memory support, running 64-bit static binaries, chrooting into 64-bit userland trees, etc), so there is no logistical reason to distribute a version of Finnix with a 64-bit userland.

However, NEALE does have the ability to build a pure 64-bit userland build of Finnix for x86. On a 64-bit x86 build host, export these variables in your build-neale bootstrap script:

export FINNIX_DISTARCH=amd64
export FINNIX_ISO="/srv/finnix/neale/finnix-neale-amd64.iso"

Finnix does not officially support a 64-bit userland x86 build, but due to the abstraction of Project NEALE, this is indeed possible. Note that the same is not true of PowerPC, as Debian does not distribute a 64-bit PowerPC userland.

Minimal builds

By default, Finnix includes a large number of administration tools. As of Finnix 109, a minimal environment can be built by NEALE, which installs the finnix-minimal metapackage instead of the finnix-standard metapackage. The minimal build includes just enough packages to go through standard boot and shutdown. Export the following variable to do a minimal build:

export NEALE_BUILD_TYPE=minimal

Supported build hosts

Officially, the only supported build host is a Finnix instance itself, or a Debian testing host running up-to-date testing sources. That is, an environment as close to the target Finnix environment as possible.

However, the build process is very generalized, and does not use any Debian-specific methods to build Finnix. For example, NEALE base components are distributed as .udebs, however build-neale uses dpkg-deb to extract them only if found, otherwise it will use ar and tar to extract them. build-neale has been tested in the following host configurations:

  • Finnix (official)
  • Debian testing (official)
  • Scientific Linux 6, using backported packages debootstrap and squashfs-tools from Fedora 18. All other required software was installed from the SL 6 repository itself. Likewise, a Fedora 18 build host with the same software should work as well. debootstrap 1.0.48 or later is required, due to a fixed bug.
  • Gentoo Linux (as of 2013-06-27), with the following extra installed software:
    • app-cdr/cdrkit
    • sys-fs/squashfs-tools
    • app-crypt/gnupg
    • dev-util/debootstrap
Personal tools