Bilder

# Adding support for a new package

Bilder supports two types of packages: tarballs and code repos, with the latter assumed at present to be one of subversion, git, or mercurial.  In either case the build of a package is governed by its Bilder package file and possibly a patch file to fix build or other problems in that package.  For tarballs one must go further add the tarball to some accessible location.  In this discussion of a generic package, we use *pkg* to denote its name in lower case, PKG is the all caps form of the name, and Pkg is the (camel-) capitalized form of the name.

For repo packages or tarball packages, one must generate a file, pkg.sh, which contains the build information.  Bilder keys off of changes of this file to build the package. In addition, there should be an auxiliary package file, pkg_aux.sh, containing information on versions and how to find the pkg.  Bilder will not key off of changes of the auxiliary file, but of course Bilder might key off of the change of information inside the auxiliary file.

Repo packages, whether manually, as an external, or as a git or mercurial repo, should be checked out as pkglc, and anypatch should be pkglc.patch.

Tarball packages come in the form of a file, __Pkg-<version>.tar.{gz,bz2,xz}__ (or .tgz) which unpacks into a directory, __Pkg-<version>__.  The patch file, pkg-<version>.patch, contains the version in its name.  If the package is not a python package, then the preference is for this to be lower case, i.e., pkg-<version>.tar.gz, with patch, pkg-<version>.patch.  This may require repacking to replace "_" with "-", to remove "-Source" from the name, and/or to change to lower case.

For Python packages, the preference is for all names to be consistent with what the import statement uses.  E.g., if one does "import foo", then the tarball should be foo-<version>.tar.{gz,bz2,xz}, the Bilder package file should be foo.sh, and the patch file should be foo-<version>.patch.  If the import statement is "import Foo", then one has Foo-<version>.tar.{gz,bz2,xz}, Foo.sh, and Foo-<version>.patch.  If a case transformation is not possible (Imaging uses "import PIL"), then one should try to maximize consistency while minimizing repacking.

## Creating the Bilder package file 

There are five sections to the file: (1) auxiliary file sourcing section, (2) the non-trigger variables section, (3) the build section, (4) the test section, and (5) the installation section.

### Auxiliary file sourcing section section 

This sources the auxiliary file, which will set any trigger variables, variables whose change alone will trigger a build.  Examples include the version and the builds.  More on this later when we discuss the auxiliary file.

An example for cmake is

~~~~~~

mydir=`dirname $BASH_SOURCE`

source $mydir/cmake_aux.sh

~~~~~~

This form allows a user to set their own values for these variables in, e.g., a machine file.

### Non-trigger variables section 

The example for cmake is

~~~~~~

setCmakeNonTriggerVars() {

  CMAKE_UMASK=002

}

setCmakeNonTriggerVars

~~~~~~

### The build section 

The build section has a method, buildPkg, that either unpacks (tarball) or preconfigures (repo) the package, after checking whether there is any reason to build the package.  The associated functions are bilderUnpack and bilderPreconfig.  If there is no reason to build the package, then this method should return immediately.

Next this method should configure and build each build for this package using the methods, bilderConfig and bilderBuild.  Those methods will ignore builds that are not in the PKG_BUILDS list, and each build will be launched in background, with its process ID stored for later collection.

### The test section 

### The installation section 

## Example package files 

When you are considering adding a package file, it is easiest to start by copying a file close to what you will need.  Here are some descriptions to help in that selection.

### Simple 

* autoconf.sh shows a vanilla, serial only build.

* toric.sh shows a vanilla, serial and parallel build.

### Easy (one variation) 

* atlas.sh shows how to fix a Makefile bad for some platforms before building.

* facets.sh shows a 3-build case with the builds depending on the other args, and post configuration of the tests directory.

* metatau.sh shows how to add a needed runpath to a build that does not really use configure.

* libtool.sh shows how to modify the environment for installation.

* matplotlib.sh deals with the ugliness of setting arch flags for Darwin

* netcdf.sh add extra, compiler dependent flags.

* netlib_lite.sh shows how to add in another backend node build when the backend nodes use a different compiler from the frontend nodes.

* numpy.sh shows distutils build with patching.

* trilinos.sh shows how to handle a build that changed from autotools to cmake at later versions.

* txbase.sh shows how to build a package that might be from either a tarball or an svn repo.

### Medium (2 variations or interpackage build parameter dependencies) 

* autotools.sh shows how to build a group of packages when one must build one package (m4) and reset one's path before building others (autoconf, automake, libtool).  It also shows how to group the installation directory according to the version of one of the packages (libtool).

### Complex 

* lapack.sh shows how to add another build when the compiler is not gcc, how to apply a a patch, and how to do a completely custom (neither autotools nor cmake nor distutils) configuration, build, and installation.  The gcc build is needed when the package is being used for building a python package.

* hdf5.sh has to deal with the fact that some versions of gfortran will not work, that on Darwin, shared and fortran are incompatible with the configure system, so one needs two builds, that the source must be modified for version 1.8.2, and to create the shared fortran library on Linux and Darwin, where one actually can, even though the configure system will not allow it.

## Creating the Bilder package auxiliary file 

There are five sections to the file: (1) the trigger variables section and (2) the find section.

### Trigger variables section 

Here one should define any variables that will trigger a build if they are changed

or should not trigger a build if they are not changed..

* PKG_BUILDS gives the names of the builds that are desired.

* PKG_DEPS gives the (all lower case) names of the packages that this package depends on

In all cases, the setting of the above variables should respect the values that might be previously set.  I.e., if a variable above is already set, the Bilder package file should not change it.

Package files should avoid introducing other variables, as Bash namespace is global by default.  If other global variables are introduced, they should be prefixed by the all-caps package name, e.g., PKG in the above example.

In some cases, the particular settings above may depend on the operating system.  E.g., for hdf5 on Darwin, one does not have the serial-shared or parallel-shared builds, as those do not work with Fortran.  Instead, there is a special build, cc4py, that does not support Fortran, but does build dylibs. In such cases one may define auxiliary functions that should go in this section.

Examples:

~~~~~~

setCmakeTriggerVars() {

  CMAKE_BLDRVERSION_STD=${CMAKE_BLDRVERSION_STD:-"2.8.12.1"}

  CMAKE_BLDRVERSION_EXP=${CMAKE_BLDRVERSION_EXP:-"2.8.12.1"}

  CMAKE_BUILDS=${CMAKE_BUILDS:-"ser"}

  CMAKE_DEPS=

}

setCmakeTriggerVars

~~~~~~

### Find section 

Bilder follows the model of *each installation in its own subdirectory*, as opposed to the GNU model of *all libraries in /usr/local/lib*.  The Bilder model is more similar to the Windows model for installing in the *Program Files* directory.  It has the advantage of easier removal (just remove the directory) with the disadvantage of having to know installation conventions to find the package.

This provides a function to add to any paths or to define variables that help configure other packages. Here also one would add to LD_LIBRARY_PATH.  

Examples:

~~~~~~

findCmake() {

  addtopathvar PATH $CONTRIB_DIR/cmake/bin

  CMAKE=`which cmake`

}

~~~~~~

addtopathvar adds the path with the correct separator for Windows versus Unix.

Bilder keeps track of all path additions and provides, at build conclusion, a file for sourcing to modify one's paths for the new build.

## Creating a patch file for a tarball package 

To get some packages to build on new platforms, one may need to patch the packages.  We do not modify the tarball, as then there would be two tarballs, the original and our patched version, that would have the same name but different contents.  Instead, we keep patch files, which we install with the package.  Bilder checks for a change in the patch, and if a change is found, it rebuilds the package.

Here are the steps, using hdf5-1.8.7 as an example, for generating a patch:

1. Unpack the tarball and in the resulting directory apply any existing patch

~~~~~~

    patch -p1 <../../bilder/patches/hdf5-1.8.7.patch

~~~~~~

2. Resolve any rejected hunks, then remove and patch detritus

~~~~~~

    find . -name '*.rej' -delete

    find . -name '*.orig' -delete

~~~~~~

3. Modify the package, editing any files as necessary.  Ensure that the patched package builds on all platforms.  Remove any build files.

4. Move the patched version aside and unpack the tarball again.

~~~~~~

    mv hdf5-1.8.7 hdf5-1.8.7-new

    tar xzf ../../svnpkgs/hdf5-1.8.7.tar.gz

~~~~~~

5. Just under the builds directory recursively diff to create the new patch and commit it.

~~~~~~

    diff -ruN hdf5-1.8.7 hdf5-1.8.7-new >../bilder/patches/hdf5-1.8.7.patch

    svn ci ../bilder/patches/hdf5-1.8.7.patch -m "<commit msg: what was fixed...>"

~~~~~~

6. The next time bilder is run for a project needing hdf5, hdf5 will be built after applying the new patch.