Building Smoothwall Express 3

There are many ways to build a Smoothwall Express 3 ISO, but the recommended method is to use a Smoothwall Express 3 “devel” ISO system and perform the build on that machine. This system not only has the exact needed tools pre-installed but it also lets you test out your code changes on a live system, namely the system you are building on. It also operates as a normal Express system.

When building on a 64bit system (debian is the principle tested environment), a 64bit ISO will result. When building on a 32bit platform a 32bit ISO will result. It is currently not possible to cross-compile.

Building Express on Express

Begin by installing the Express 3 devel ISO, just as if it was any other Smoothwall install. There are several differences between the devel build and a normal build:

  • It contains the core build tools, such as make and gcc.
  • It has un stripped packages. This means the binaries are not run through “strip” and also that the include files and libraries are left intact.
  • It contains a subversion client for checking out the code directly from sourceforge, as well as a copy of the program “screen”, which is handy when running long tasks such as OS builds.

Begin by configuring networking as per an ordinary Smoothie, if you haven’t already done so. Login as root, and run the checkout script:

# /usr/bin/smoothwall/expresscheckout

This is a small shell script that simple runs SVN to download the source. You will have to accept the https certificate offered by sourceforge by typing “p”. The sources will then be checked out. This takes only a matter of minutes as the actual sources for Express 3 are quite small.

Take a look around. The sources will all be under /usr/src/expresscore/. This will take several hours even on very fast machines (datapoints: P3 550 takes 11 hours; a 2ghz Athlon will take about 4 hours, disk bandwidth is a large factor). Again there is a script:

# /usr/bin/smoothwall/expressbuilder

Before running the script it is probably a good idea to use the program “screen” so you can detach the session and logout while leaving the build continuing. For some information on screen, have a look at this tutorial.

Once the build is complete, an ISO will appear in the directory /usr/src/exprersscore/distrib/build/target/isos.

If you are interested in joining our Express team, why not check out the forums?

Building Express on ordinary Linux systems

First of all, view the notes about tested systems at the bottom of this document. As well as a working compiler and GNU tool-chain, you will also need to be able to checkout the SVN tree from sourceforge with the svn program, plus bison and wget. A good starting point on debian-based systems is to install “build-essential” which is a meta-package for doing general development. Other distros vary.

Currently, all work must be done as the root user.

Sources live in /usr/src, so checkout the tree:

cd /usr/src

svn co https://smoothwall.svn.sourceforge.net/svnroot/smoothwall/trunk expresscore

Next, download the sources to each of the packages used. While this can be done as the build progresses, it is usually easier to do it at the beginning:

cd expresscore/distrib/build

make predownload

There is, at time of writing, 318MBytes of code to download, so this is not something you can do without bandwidth. By default, the build system is set to download code from a mirror that the project maintains. This mirror is located in London, UK. The reason for a mirror is that software projects come and go, and sometimes not all the particular versions of each and every package are available from the main source. The mirror URL is configured in the script build/dlverify, so once you have downloaded all of the sources you can make your own mirror. Simply copy all of the sources from build/downloads into your mirror URL, which could be a web server running locally or another machine on your LAN.

It is recommended that you run the build within the “screen” program:

screen

make build

Once the build is complete, an ISO will appear in the directory /usr/src/exprersscore/distrib/build/target/isos.

Build process in detail

The build processes consists of 3 main steps:

1. Construct a temporary “tools” environment using the hosts toolchain. A symlink in the host (/tools) ensures that all programs, when a “make install” is done on them, end up in the temporary tree and not in the host system. This symlink is created automatically.

2. Compile all packages that will end up in the final installation smoothwall.tgz, plus any programs and libraries that are required to compile those programs. This stage is done using the temporary tools environment and is chrooted into the working build environment. Before building the “core” packages, such as the kernel etc, the build toolchain that will be used in the final build is compiled up with the /tools toolchain, thus the /tools structure is no longer needed. Packages that are compiled and installed are placed into the build environment (i.e.. under distrib/). Also, each compiled package is tarred up into a tarball that is used in step 3.

3. Create the last few tools needed to create the final bootable ISO (such as mkisofs, and syslinux). Then build the smoothwall.tgz, by untarring the packages compiled in step 2 into a new directory. Smoothwall specific files are added from the “coretree” package. Finally build the bootdisk images and the ISO itself.

Feedback

The development team would love to hear any feedback, observations, patches, improvements or ideas relating to the Build Environment as well as Smoothwall Express. Please direct such feedback to us either using the forums (community.smoothwall.org/forum) or, if you prefer via email to feedback (at) smoothwall.org

Layout

The main working directory is expresscore/distrib. Once an initial build has been completed, this dir can be chrooted into and it will be possible to work on the tree, add new packages, etc, etc. expresscore itself can be used to hold documentation and other files that wont be used in the build environment itself.

expresscore/distrib/tools is the location of the temporary build environment. Even after a completed build, you should leave this dir alone because some build scripts have to reference files in it because they are normally called before step 2 is complete. It shouldn’t ever be necessary to change how this tools dir works, so just ignore it unless you are REALLY interested.

expresscore/distrib/build is where the “action” is. This dir contains the build top level makefile, scripts to build stage 1 and stage 2, and a couple of helper scripts.

build/sources is the real heart of the build process. Each dir is a package, including some smoothwall-specific ones, like libsmooth+setup for the setup program, setuids for the little C setuids, etc. Each dir contains a Makefile that must have specific targets, such “download”. The clever part is it is possible to use ready made targets, by including the Makefile.rules in the dir above. This has targets for configuring, compiling, installing and creating the package archive. For gnu-compliant, ./configure compatible packages it is possible to make a makefile to build the package in only a few lines, including the download URL. See the sources/sed/Makefile for a good example.

Two packages are worth extra attention: coretree is a package that builds a tarfile tree of all the extra files needed by smoothie, such as CGIs etc. It does this by copying them out of sources/coretree/tree into a temp dir, before taring up the temp dir. So if you want to add static files to the distrib, add them to this tree. Symlinks and permissions are not directly set in this tree – they are created and set by the Makefile, scripted fashion.

Finally, sources/media is where the bootdisks and ISO are created. The Makefile loops copying compiled packages from the target dir. Add additional packages to the list as needed. This makefile also removes unneeded stuff, such as /usr/include, from the final smoothwall.tgz, as well as running strip on directories expected to hold binaries. The final ISO will end up in this build/target/iso dir.

The build dir

Makefile – Top level makefile. Really just a wrapper for makebuild1. Modify this file to hold DNS server IP addresses.

makebuild1 – Inital build script. Makes the temporary build env (under /tools) and then fires off makebuild2 chrooted.

makebuild2 – Main build script, which runs chrooted in the directory above this one. Uses the makefiles in sources dir to compile up main distro components into compiled packages living under targets. Also compiles up smoothwall-specific code. Finally creates boot disks using sources/media makefiles, overlays static parts that are packaged up in sources/coretree, and builds the final ISO.

prebuild2 – Little script that creates files needed to compile up the second level build, such as a working /dev/null. Needed because build2 happens chrooted.

build1done – This is a touch file created when stage 1 has been completed successfully. The mailebuild1 script checks for the existence of this file before starting to build stage 1. Therefore it is possible to restart “make build” without it going though stage 1 all over again.

dlverify – “Download and verify”. This is a little helper script that wraps wget. The purpose of the wrapper is two-fold: to stop wget being called on a completed download, thus causing unnecessary network traffic, and also to verify a completed download by MD5. The arguments are: [md5]. Directory is usually the standard d/l area. The MD5 can be left off, in which case no download check is done. To cope with partial downloads, once a d/l has completed, an additional touch file will be created with the same name as the download file, plus “.done”. Only if this file is present will the download be skipped over. Note that the Makefile.conf (in sources) defines DL_CMD for you, and it includes the download destination dir. So to do a download from within a makefile, you should usually just do: $(DL_CMD) http://… This script also has the ability to use a local mirror. The URL root for the mirror is set at the top of the script. To create a local mirror for yourself, after doing a full build, copy the files in build/download (except the .done files) into your mirror.

sources – “Packages” are assigned a directory in here, with a Makefile for building.

Using the build scripts

Currently the environment is only tested under the root user, and when it is installed in /usr/src/expresscore. More testing will be done, and perhaps eventually it will be possible to compile an ISO under a different user. But not yet

Note that before starting the build process, the symlink /tools will be first deleted and then created like this, automatically:

ln -s /usr/src/expresscore/distrib/tools

To start the build, do “make build”. Do “make” with no target to get a list of available targets. Interesting ones are:

predownload – Downloads all needed sources.

packageinfo.html – Spits out some crudely formatted HTML containing packages and versions, download URLs and patch URLs. Call with make -s to hide the make output.

Restarting the build

Currently it is not easily possible to restart the build and have it continue where it left off. This is a hard thing to implement, but we are interested in doing so.

Tested host environments

  • Express 3 “Koala”, “Degu”, “Sammy” and “Polar”
  • Red Hat 8.0-9.0
  • Fedora Core 1-3, 6-7
  • SuSE Linux (9.0, Enterprise Desktop)
  • Debian Sarge (3.1)
  • Debian Etch (4)
  • Ubuntu 7.04

Generally, the build should work on pretty much any modern Linux system.

Thanks to

Loosely based on Linux From Scratch ideas.

Credits

The build system was written by Lawrence Manning (smoothwall@aslak.net). Originally written for Smoothwall Ltd commercial products, it was later modified for building Express and released under the GPL v2.

Copyright and license

The Smoothwall build system and associated scripts are (c) Smoothwall Ltd, 2005-2007 and licensed under the GNU GPL v2.