debian-installer/build

The files in this directory are used to build the Debian installer.
Basically it consists of downloading udebs, unpacking them, applying some
magic (library reduction etc.) and building an image.

Warning: The build system for the installer is often tightly bound to the
version of Debian for which it is targeted. If you are using stable, use
"apt-get source debian-installer" to get a version of the installer that
will build on your system. If you are using testing or unstable, check out
a copy of the installer using the command 
"git clone git://git.debian.org/d-i/debian-installer.git"

Recipe:
 - Install the build-dependencies on the host system
   (run dpkg-checkbuilddeps in the parent installer/ directory).
 - Create your own sources.list.udeb.local, otherwise the build host's
   sources.list is taken as a template for sources.list.udeb.
 - Run "make" to get a list of available targets.
 - Build an image using one of the build_ targets (build_netboot,
   all_build, etc). You may want to set the USE_UDEBS_FROM variable,
   documented below.
 - Look in dest/ for the completed images.
 
Note that this does not create full debian ISO images; that is left to the
debian-cd package. As a shortcut, you can create a mini-ISO image, with
only the netboot initrd on it. make build_netboot will create a
dest/netboot/mini.iso, using isolinux. Any size initrd can be placed on
this ISO, which may be useful for testing.
 
A more detailed overview of how the installer is built:

* 'sources.list.udeb' is used to configure apt to download udebs from a
  mirror. It is autogenerated based on '/etc/apt/sources.list', by the
  Makefile's 'sources.list.udeb' target. Or you can provide your own
  locally modified 'sources.list.udeb.local'. You can also provide a
  'preferences.udeb.local' to select specific versions of the packages
  (see apt_preferences(5) for its syntax).
* The Makefile is configured via the make fragments in the config
  directory. These are organized in a subarch/medium/flavour hierarchy and
  get included by the main Makefile.
* config/local can be added to override anything set in other fragments
  which are local to your system which you want to avoid accidentially
  committing.
* 'pkg-lists' has subdirectories for the different image types that list
  udebs that are put on each image. The pkg-lists/*/common files list
  udebs common to all architectures, and the files named by architecture
  (<arch>.cfg) list udebs specific to an architecture. It is also possible
  to include udebs only on a specific subarchitecture by creating a
  directory for the architecture and putting config files for the
  subarchitecture there (<arch>/<subarch>.cfg).  All of these files can
  have #include lines to include files from pkg-lists. Also,
  ${kernel:Version} in these files is replaced with the kernel version,
  as set in the KERNELVERSION variable (plus the KERNEL_FLAVOUR variable).
* Anything listed in pkg-lists/local and pkg-lists/*/local is included in
  the image that is built. This is to provide an easy way to add extra
  udebs to a test image.
* All dependencies of packages in pkg-lists are automatically included
  in an image by default. If you know that some of the dependencies are
  wrong, or do not apply to an image, you can remove dependencies from the
  pkg-lists by listing them and putting a space and minus sign after them.
* If you need to include a package for one major version of a kernel, but
  not for another, put the kernel major version(s) that should have the
  package in brackets at the end of the line.
  For example: discover1 [2.2 2.4]
* Similarly, if you need to include a package for only one type of kernel,
  put the name of the kernel(s) that should have the package in brackets at
  the end of the line.
  For example: gnumach-udeb [hurd]
* If you'd like to include a given package only if it's available, and not
  fail otherwise, put a question mark at the end of the line.
  For example: pcmcia-modules-${kernel:Version} ?
  It's recommended to use this option seldom and with care as it can make the
  build less robust if packages can be dropped from it without warning.
* Apt is used to download the required udebs. This does *not* include
  libraries; libraries used by udebs must be installed on the build system,
  and so are build-depended on.
* If you have some udebs that are not available on your mirror yet,
  you can drop them in 'localudebs/' and they will be used. 
* dpkg is used to unpack the udebs into the build directory.
* A customized set of reduced libraries is generated to correspond to the
  udebs that were installed.
* Some boot images have associated "driver" disks. These disks just get
  udebs put on them.
* The final images are put in the dest/ directory.

------------


The debian-installer build system
---------------------------------

The build system is designed to allow the addition of configurations
with minimal effort. It achieves this by evaluating some magic
variables from the configuration snippet and feeding some generic
rules you hopefully never need to know about. The configuration
snippets live in the config/ directory and are organized in a
hierarchical tree structure, the directory names therein have also a
specific meaning. It follows a config/$ARCH/$SUBARCH/$MEDIUM/$FLAVOUR
scheme, where $ARCH is mandatorily the debian architecture name. For
the other three, the requirement is you need to use at least one.

$SUBARCH is supposed to be the subarchitecture. For e.g. m68k this
would be something like "amiga" or "atari", i386 doesn't use it.

$MEDIUM is the boot medium, like "cdrom", "floppy", "netboot".

$FLAVOUR distinguishes between other differences, like 2.4 vs. 2.6
kernels, or several floppy images.

Note: You don't have to follow this usage strictly, the variable naming
is merely a hint. It could also have been named as $LEVEL1/$LEVEL2/LEVEL3.

Note2: The resulting file tree in the dest/ directory has a similiar
$SUBARCH/$MEDIUM/$FLAVOUR layout. It can be slightly adapted with the
EXTRANAME and SOME_DEST variables.

An example tree layout looks like this:

config
|-- i386
|   |-- cdrom
|   |   |-- el-torito-2.6.cfg
|   |   |-- el-torito.cfg
|   |   |-- isolinux-2.6.cfg
|   |   `-- isolinux.cfg
|   |-- cdrom.cfg
|   |-- floppy
|   |   |-- boot.cfg
|   |   |-- cd-drivers.cfg
|   |   `-- root.cfg
|   |-- floppy.cfg
|   |-- hd-media
|   |   `-- 2.6.cfg
|   |-- hd-media.cfg
|   |-- monolithic
|   |   `-- 2.6.cfg
|   |-- monolithic.cfg
|   |-- netboot
|   |   `-- 2.6.cfg
|   `-- netboot.cfg
`-- i386.cfg

In the first level, you have an $ARCH directory, and an $ARCH.cfg
configuration snippet, which needs to define what to use from the tree
below, and can also define commonly used variables for all
configurations of this architecture. An example $ARCH.cfg would look
like this:

SUBARCH_SUPPORTED = r4k-ip22 r5k-ip22 sb1-bcm91250a miniiso

KERNELMAJOR = 2.4
KERNELMINOR = 25
KERNEL_FLAVOUR = di
KERNELNAME = $(foreach ver,$(KERNELVERSION),vmlinux-$(ver))

VERSIONED_SYSTEM_MAP = t

arch_boot_screens:

The SUBARCH_SUPPORTED line defines what to look up in the next
directory level. In this case, these are the configuration snippets
r4k-ip22.cfg, r5k-ip22.cfg, sb1-bcm91250a.cfg and miniiso.cfg. Each of
those snippets can define a MEDIUM_SUPPORTED variable, which
opens the next subdirectory level. Those subdirectories have to have
the name of their subarchitecture, for the example above e.g. r4k-ip22
or sb1-bcm91250a.

The various KERNEL* variables are common for all configurations of this
architecture, so they were moved upwards from the leaf configuration
snippets. The same can be done for make rules.

Caveats:
- You can re-/undefine variables defined in a higher level, but you
  can't do that for rules. Make will complain about it.
- Use always recursively expanded variables (that is '=', not ':=').
  Make works though the tree by recursively calling itself. If you
  use simply expanded variables, the variables of higher levels (like
  *_SUPPORTED) won't get re-evaluated, and things will break.


Interesting Variables in configuration snippets
-----------------------------------------------

First, mandatory variables:

SUBARCH_SUPPORTED
MEDIUM_SUPPORTED
FLAVOUR_SUPPORTED
  As already explained, they define what configurations the build
  system should look for. They cause the definition of SUBARCH,
  MEDIUM and FLAVOUR for every snippet they invoke that way.
  At least one of them must be defined.

KERNELMAJOR
  The high part of the kernel version, like "2.6". Unluckily, this is
  what the kernel people call MAJOR.MINOR.

KERNELVERSION
  The version of the kernel .udeb package, like "2.4.26-r4k-ip22"

KERNELNAME
  The full name of the kernel image. If you build an EXTRA target (e.g.
  for a driver floppy), this variable has to be empty.

TYPE
  This variable points to the subdirectory in pkg-lists/ which should
  be used for this config. E.g. $TYPE=cdrom means that
  pkg-lists/cdrom/local, pkg-lists/cdrom/common and
  pkg-lists/cdrom/<arch> are used. Defaults to $(MEDIUM)/.


Targets:

TARGET
  Define this variable in the leaf configuration snippet to one or more
  of the following generic targets, and any local target you want to
  run.

INITRD
  Create an gzipped initrd.gz.

KERNEL
  Create the kernel image.

BOOT
  Create an optionally gzipped bootable image. This creates kernel,
  initrd and boot screens, and depends on the arch_boot rule, where the
  image creation is handled.

ROOT
  Create an optionally gzipped root image. This creates an initrd and
  depends on the arch_root rule, where the image creation is handled.

EXTRA
  This creates a file system image with driver .udebs in it.

MINIISO
  This creates a mini ISO image, by creating the boot screens and
  depending on arch_miniiso rule, where the image creation is handled.

DEBIAN_CD_INFO
  This provides the boot screens in a gzipped tarball.

MISC
  This copies files to the destination directory of your configuration.

MANIFEST-INITRD
MANIFEST-KERNEL
MANIFEST-BOOT
MANIFEST-ROOT
MANIFEST-EXTRA
MANIFEST-MINIISO
MANIFEST-DEBIAN_CD_INFO
MANIFEST-MISC
  These variables correspond to the generic targets and hold their
  description which is added to the MANIFEST file in the dest/
  directory. 

TEMP_INITRD
TEMP_UDEB_LIST
TEMP_KERNEL
TEMP_BOOT
TEMP_ROOT
TEMP_MINIISO
TEMP_EXTRA
TEMP_BOOT_SCREENS
TEMP_CD_TREE
TEMP_POWERPC_INITRD
  Those are intermediate targets. Everything is built for them, the
  generic targets copy the stuff they are interested in over to the
  dest/ and add the MANIFEST entry. They are useful as dependencies for
  things which needs to be built but shouldn't show up in the
  destination directory. You should never need to redefine them.


Other useful variables for configuration snippets:

LINUX_KERNEL_ABI
  The first part of the kernel version, e.g. 3.14-2. Used as the
  default base/suffix for KERNELVERSION (i.e. $(LINUX_KERNEL_ABI)-amd64).

KERNEL_FLAVOUR
  The flavour of the kernel .udeb package. This has nothing to do with
  FLAVOUR. It has always a value of "di" these days.

SUBARCH_SUPPORTED_EXTRA
MEDIUM_SUPPORTED_EXTRA
FLAVOUR_SUPPORTED_EXTRA
  These are like SUBARCH_SUPPORTED (etc.), except that they are not built by
  default with 'make all_build' but are only used when selected manually,
  either by passing an explicit target to make or by defining RECURSE_EXTRA
  to a non-empty value.

EXTRANAME
  This allows to vary the name of the dest/ target a bit. Example:
  EXTRANAME = $(MEDIUM)/    leads to: floppy/netboot.img
  EXTRANAME = $(MEDIUM)-    leads to: floppy-netboot.img
  Nice to avoid subdirectories with only a single file in it.

DEST
  The destination directory of your configuration.
  Caveat: Always use $(DEST), not ./dest/foo/...  The latter will only
  work for one specific location, and fails if you relocate the
  configuration snippet in the tree. DEST changes its definition for
  every leaf configuration appropriately.

SOME_DEST
  Like DEST, but with the last path element removed. Mostly useful in
  combination with EXTRANAME.

TEMP
  The intermediate directory, where everything is built.
  Caveat: Always build stuff in TEMP, and then install to DEST from
  there. Doing this differently tends to turn into a mess.
  Caveat2: Always use $(TEMP), not ./tmp/foo/...  The latter will only
  work for one specific leaf configuration. TEMP changes its definition
  for every leaf configuration appropriately.

DROP_LANG
  Purge unwanted languages. Mostly useful for space constrained media.

KEEP_GI_LANGS
  If set the languages in $(GI_LANGS) will NOT be dropped. Should only be
  set in targets building graphical installer images.

GI_LANGS
  List of languages only supported with the graphical cdebconf frontend.

VERSIONED_SYSTEM_MAP
  If set, the kernel's System.map is expected to have an unique
  filename for each kernel of this architecture.

FLOPPY_SIZE
  The size of a floppy image, in kilobytes.
IMAGE_SIZE
  The size of a an image, in kilobytes. Only needed and supported for
  some types of images.

DISK_LABEL
  Disk label to use when formating disk images. Allowed content varies
  by disk type.

MEDIA_TYPE
  The type of media produced, currently used to make qemu boot the image
  properly. Suggested values: "CD-ROM", "netboot image", "floppy", etc.

DEBIAN_RELEASE
  The codename of the debian release to be installed by default.
  Included in /etc/default-release in most initrds.

DEBIAN_VERSION
  The version identification (number + codename) of the Debian release.

USE_UDEBS_FROM
  Normally the codename of the release to use for the build. Defaults to
  unstable for daily builds. Set from debian/rules when building through
  dpkg-buildpackage.
  Included in /etc/udebs-source in most initrds.

USE_PROPOSED_UPDATES
  Set to 1 in debian/rules to have the gen-sources.list.udeb script include
  udebs from $USE_UDEBS_FROM-proposed-updates (for official builds of e.g.
  stable updates). For manual builds use for example:
     make USE_PROPOSED_UPDATES=1 all_build
  If no valid source for proposed updates is found, the build will fail.

OMIT_RELEASE_INFO
  Can be used to suppress the inclusion of the files /etc/default-release and
  /etc/udebs-source in some targets (like boot floppies).

DOS_VOLUME_ID
DOS_VOLUME_LABEL
  Volume ID and label for DOS filesystems.

SYSLINUX_CFG (x86 only)
  Determines how the syslinux configuration is created. Options are:
  - prompt: non-graphical syslinux configuration allowing installation of
    only a default desktop environment
  - standard: as prompt, but graphical menu (using vesamenu)
  - template: the available files are merely copied; the actual configuration
    is created by debian-cd

BOOTMENU_BEEP
  Set to y to produce a beep when the boot menu is ready to accept commands
  (e.g. extra kernel parameters, etc.)
  This improves accessibility of the image for blind people.

GZIPPED
  Create gzipped images. This works only for the BOOT, ROOT and EXTRA
  targets.

EXTRADRIVERS
  This variable points to the directories where additional driver
  .udebs reside. This causes these .udebs to be considered in the font
  and library reduction steps, so the characters and library symbols needed
  for them are kept.

EXTRATARGETS
  Additional targets which are run before the ones in TARGET. This is
  e.g. useful to build the additional driver images first, and then
  use their intermediate tree for EXTRADRIVERS.

EXTRAUDEBS
  This variable points to additional udebs that should be considered by the
  library reduction step. cdebconf plugins should be listed here so the
  symbols they will need are available on the initrd (since cdebconf runs
  using the initrd libc, its plugins do too)

DRIVER_FOR
  The variable indicates that the image is just a driver image for some
  other image. The value points to the directory in pkg-lists for the image
  that it's a driver for.

UPX
  If this variable is set, it should point to a usable UPX compressor.
  This is then used to create UPX compressed kernels.

SYSLINUX_OPTS
  Additional syslinux options.

SPLASH_RLE
SPLASH_PNG
  The syslinux splash screen file.

INITRD_FS
  The filesystem to use for initrds, needs to be built into the kernel.
  ext2, initramfs, or jffs2.

MKFS_JFFS2
  If using jffs2 for the INITRD_FS, you will need to set this variable to
  a command to use to create the image.
  For example: MKFS_JFFS2 = mkfs.jffs2 -f -p -b -e 0x20000

WRITE_MEDIA
  Add you configuration's name if it can be written to a raw device.
  Mostly useful for floppies.

UDEB_LISTS
  List of files listing udebs that were used to build the image.

TRANSSTATUS
  Location of file with translation statistics. This file is intended to
  be included only for official releases and thus this variable is set in
  debian/rules.
  The file is only included if it exists. A warning will be displayed if
  the file is older than 2 weeks than the time of the build.
  If needed, inclusion of the file can be prevented by setting the variable
  to empty for specific targets (e.g. most floppy images) in config or
  globally in config/common.


Variables useful for testing:

FLOPPYDEV
  The device name to use for writing a image to a raw device.

QEMU
  The name of the qemu emulator binary.

EXTRAS
  Add additional .udebs to the target image.

EXTRAFILES
  Add additional files to the target image.

SSL_CERTS
  Add SSL certificates to the target image. For this to be useful, you will
  also need to supply d-i with GNU wget.

MIRROR
  The mirror to get .deb and .udeb packages from. Defaults to the same
  as the host machine uses.

UDEB_COMPONENTS
  Archive components from which to fetch .udeb packages. Defaults to
  main/debian-installer.

PRESEED
  An initrd preseed file to put in the initrd.


Rules invoked from the various targets:

arch_boot:
  The bootable image TEMP_BOOT is created in this rule, usually from
  TEMP_KERNEL and TEMP_INITRD, and probably from TEMP_BOOT_SCREENS and
  other files.

arch_root:
  The root image TEMP_ROOT is created in this rule, usually from
  TEMP_INITRD, and probably some other files.

arch_boot_screens:
  The boot screens in TEMP_BOOT_SCREENS are created here.

arch_miniiso:
  The mini ISO image TEMP_MINIISO is built here, from whatever the
  particular (sub-)architecture needs for it.