Project NEALE

Finnix, the LiveCD for system administrators
Revision as of 05:59, 22 October 2012 by Ryan Finnie (talk | contribs)
Jump to navigation Jump to search

Template:Dev feature

Project NEALE (Normalized Extraction and Assembly of LiveCD Environments) is a set of procedures to build Finnix from a low level in a consistent manner. It starts with a minimal Debian bootstrap, adds Finnix packages and metapackages, and then assembles the environment into a final ISO.

Background

Throughout its history, most Finnix releases were assembled by hand to varying degrees. Finnix 0.03 was created by modifying an existing Red Hat Linux installation manually, though this was at a time when Finnix was one of the first LiveCDs to be produced, so there was little precedent.

Finnix 86.0 was based on a Debian bootstrap, and all Debian-based versions of Finnix relied on project management using Debian's own tools (apt repositories, etc). However, releases were made by taking the previous release, bringing upstream software up to date, and adding Finnix-specific updates. Essentially, each Finnix release was a remaster of the previous release.

Finnix 105 was the first Finnix release to be build using NEALE; no previous Finnix release was used as a base. All software within the release was downloaded and installed via apt repositories.

Procedure

Most of the work is handled by a script, build-neale, which does the following tasks:

  • Downloads and extracts Finnix-specific base components (kernels, initial ramdisk layout, general CD layout, miscellaneous files ("stuff")).
  • Installs a Debian bootstrap.
  • Adds temporary apt repositories for Debian and Finnix (most likely using a cached proxy, see below).
  • Installs the package finnix-base, which is a metapackage for the Finnix GPG keyring and a few base packages.
  • Installs the package finnix-standard, which is a metapackage for all desired software in the Finnix release, including Finnix-specific packages.
  • Replaces the temporary apt repositories with the permanent equivalents.
  • Moves the kernels into place.
  • Hands off to finnix-build-stage1 and finnix-build-stage2, which are responsible for building the compressed root filesystem and final CD, respectively.

Installation

The bulk of build-neale is available at http://archive.finnix.org/finnix/project/neale/build-neale.remote . This is a special shell script which is inline GPG signed by the archive key. To properly use this script, you will need to write a bootstrap script which will check for an optionally download updates for it, verify the signature is correct, and run it.

First, install build dependencies. Finnix does not require much software to build, most of which will be installed on any system. However, a few non-standard packages will be needed, particularly mksquashfs, which requires at least version 4.2. If any required software is missing, build-neale will tell you. Here are the packages you will most likely need beyond what is on a standard system:

root@host:~# apt-get install debootstrap genisoimage squashfs-tools xz-utils uuid-runtime

Next, grab the Finnix GPG keyring and import it. You will want to add this to the root user's keyring, as build-neale requires root privileges to run (since it relies heavily on chroots).

root@host:~# wget http://archive.finnix.org/finnix/project/finnix-keyring.gpg
root@host:~# gpg --import finnix-keyring.gpg 
root@host:~# gpg --fingerprint A89BA58D
pub   4096R/A89BA58D 2012-05-14
      Key fingerprint = 3954 B91A 987B 0B68 17E3  FBFF 8087 F4ED A89B A58D
uid                  Finnix Archive Signing Key <keymaster@finnix.org>

Next, create a bootstrap script. It is suggested this be named build-neale:

#!/bin/sh

# Export NEALE/Finnix specific environment variables here

DESTDIR="$(dirname $0)"
wget -N -P "$DESTDIR" http://archive.finnix.org/finnix/project/neale/build-neale.remote
gpg --verify "$DESTDIR/build-neale.remote"
chmod 0755 "$DESTDIR/build-neale.remote"
exec "$DESTDIR/build-neale.remote" "$@"

Then simply run it!

root@host:~# build-neale

By default, build-neale will work with its files within the directory /finnix/neale. build-neale will take a long time to run, expect it to take between 20 minutes and an hour. When it is done, assuming everything went well, you should have an ISO at /finnix/neale/finnix-neale.iso on x86 systems, or /finnix/neale/finnix-neale-ppc.iso on PPC systems.

Caching proxy

build-neale downloads a lot of data, at least 300MB of debs. NEALE-specific downloads are handled in an efficient manner; it uses "wget -N" to check the remote timestamps against the local copies, and will only download if the remote archives have changed. However, the majority of the downloads come from apt downloads. It is strongly recommended you install a caching proxy server on your local network.

Here is a working example for Squid 3, which can also be trivially modified to work with Squid 2.7. It is based loosely on Ubuntu's squid-deb-proxy package:

visible_hostname squid-deb-proxy
maximum_object_size 16 MB
cache_mem 96 MB
maximum_object_size_in_memory 2 MB

# refresh pattern for debs and udebs
refresh_pattern deb$   129600 100% 129600
refresh_pattern udeb$   129600 100% 129600
refresh_pattern tar.gz$  129600 100% 129600
refresh_pattern tar.bz2$  129600 100% 129600
refresh_pattern tar.xz$  129600 100% 129600

# always refresh other files
refresh_pattern . 0 0% 0

acl all src 0.0.0.0/0.0.0.0
acl sdp dstdomain squid-deb-proxy
acl debian urlpath_regex ^/debian
acl finnix urlpath_regex ^/finnix

http_port 8090 accel defaultsite=squid-deb-proxy

cache_peer mirrors.kernel.org parent 80 0 name=debian no-query no-digest originserver forceddomain=mirrors.kernel.org
cache_peer_access debian allow debian
cache_peer_access debian deny all

cache_peer archive.finnix.org parent 80 0 name=finnix no-query no-digest originserver forceddomain=archive.finnix.org
cache_peer_access finnix allow finnix
cache_peer_access finnix deny all

http_access allow sdp
http_access deny all
icp_access deny all

cache_dir aufs /var/spool/squid3 500 16 256

Once this is installed, tell build-neale to use this caching proxy for the initial downloads, by adding exports to your build-neale bootstrap script:

export NEALE_DEBIAN_INIT_MIRROR="http://$PROXY_IP:8090/debian"
export NEALE_FINNIX_INIT_MIRROR="http://$PROXY_IP:8090/finnix"

Replace $PROXY_IP with the IP of your caching proxy. These variables are used by build-neale to do the actual deb downloads. Once the downloads are complete, the "real" public Debian and Finnix mirrors are put in the build's /etc/apt/sources.list, so the final build does not contain mention of the local caching proxy.

One-off builds

Due to the length of time needed to do a proper NEALE build, it is often desirable to quickly test changes by manipulating a previous build. build-neale does not clean up the build area after its run (but does before), so you can make changes and rebuild a previous build made with NEALE. Since NEALE uses different directory paths than the normal remastering environment, the instructions in that guide are not directly applicable. These wrapper scripts help with a comparable environment.

neale-oneoff-finnix - chroot into the NEALE root filesystem, with development options enabled (start-stop-daemon protection, etc):

#!/bin/sh

export FINNIXDEV=1
exec /usr/sbin/chroot /finnix/neale/build/source/FINNIX /bin/bash -l

neale-oneoff-stage1:

#!/bin/sh

export FINNIX_BASE="/finnix/neale/build"

exec "$FINNIX_SOURCE/usr/sbin/finnix-build-stage1"

neale-oneoff-stage2:

#!/bin/sh

export FINNIX_BASE="/finnix/neale/build"
export FINNIX_SOURCE="$FINNIX_BASE/source/FINNIX"
export FINNIX_ISO="/finnix/neale/finnix-neale-oneoff.iso"
export VANITYHASH
export GPGUSER

exec "$FINNIX_SOURCE/usr/sbin/finnix-build-stage2"

Supported build hosts

Officially, the only supported build host is a Finnix instance itself, or a Debian testing host running up-to-date testing sources (that is, an environment as close to the target Finnix environment as possible).

However, the build process is very generalized, and does not use any Debian-specific methods to build Finnix. NEALE base components are distributed as .udebs, however build-neale uses dpkg-deb to extract them only if found, otherwise it will use ar and tar to extract them. build-neale has been tested in the following host configurations:

  • Finnix (official)
  • Debian testing (official)
  • Scientific Linux 6, using backported packages debootstrap and squashfs-tools from Fedora 17. All other required software was installed from the SL 6 repository itself.