Package Building Procedures
Prev |  | Next

# 4 Starting the Build

Several separate builds for each architecture - branch combination are supported. All data private to a build (ports tree, src tree, packages, distfiles, log files, bindist, Makefile, etc) are located under ``${arch}`/`${branch}`/builds/`${buildid}``. The last created build can be alternatively referenced under buildid `latest`, the one before is called `previous`.

New builds are cloned from the `latest`, which is fast since it uses ZFS.

## 4.1 `dopackages` scripts

The `scripts/dopackages` scripts are used to perform the builds.

### 4.1.1 Old codebase

For the old codebase: the most useful are:

- `dopackages.7` - Perform a 7.X build

- `dopackages.7-exp` - Perform a 7.X build with experimental patches (7-exp branch)

- `dopackages.8` - Perform a 8.X build

- `dopackages.8-exp` - Perform a 8.X build with experimental patches (8-exp branch)

- `dopackages.9` - Perform a 9.X build

- `dopackages.9-exp` - Perform a 9.X build with experimental patches (9-exp branch)

- `dopackages.10` - Perform a 10.X build

- `dopackages.10-exp` - Perform a 10.X build with experimental patches (10-exp branch)

These are wrappers around `dopackages`, and are all symlinked to `dopackages.wrapper`. New branch wrapper scripts can be created by symlinking `dopackages.${branch}` to `dopackages.wrapper`. These scripts take a number of arguments. For example:

    dopackages.7 ${arch} ${buildid} [-options]

### 4.1.2 New codebase

The symlinks go away, and you just use `dopackages.wrapper` directly. For example:

    dopackages.wrapper ${arch} ${branch} ${buildid} [-options]

### 4.1.3 Either codebase

Most often, you will be using `latest` for the value of `buildid`.

`[-options]` may be zero or more of the following:

- `-keep` - Do not delete this build in the future, when it would be normally deleted as part of the `latest` - `previous` cycle. Don't forget to clean it up manually when you no longer need it.

- `-nofinish` - Do not perform post-processing once the build is complete. Useful if you expect that the build will need to be restarted once it finishes. If you use this option, don't forget to cleanup the clients when you don't need the build anymore.

- `-finish` - Perform post-processing only.

- `-nocleanup` - By default, when the `-finish` stage of the build is complete, the build data will be deleted from the clients. This option will prevent that.

- `-restart` - Restart an interrupted (or non-`finish`ed) build from the beginning. Ports that failed on the previous build will be rebuilt.

- `-continue` - Restart an interrupted (or non-`finish`ed) build. Will not rebuild ports that failed on the previous build.

- `-incremental` - Compare the interesting fields of the new `INDEX` with the previous one, remove packages and log files for the old ports that have changed, and rebuild the rest. This cuts down on build times substantially since unchanged ports do not get rebuilt every time.

- `-cdrom` - This package build is intended to end up on a CD-ROM, so `NO_CDROM` packages and distfiles should be deleted in post-processing.

- `-nobuild` - Perform all the preprocessing steps, but do not actually do the package build.

- `-noindex` - Do not rebuild `INDEX` during preprocessing.

- `-noduds` - Do not rebuild the `duds` file (ports that are never built, e.g. those marked `IGNORE`, `NO_PACKAGE`, etc.) during preprocessing.

- `-nochecksubdirs` - Do not check the `SUBDIRS` for ports that are not connected to the build. (New codebase only).

- `-trybroken` - Try to build `BROKEN` ports (off by default because the amd64/i386" clusters are fast enough now that when doing incremental builds, more time was spent rebuilding things that were going to fail anyway. Conversely, the other clusters are slow enough that it would be a waste of time to try and build `BROKEN` ports).

  > Note: With `-trybroken`, you probably also want to use `-fetch-original` (and, on the new codebase, `-unlimited-errors`).

- `-nosrc` - Do not update the `src` tree from the ZFS snapshot, keep the tree from previous build instead.

- `-srccvs` - Do not update the `src` tree from the ZFS snapshot, update it with `cvs update` instead.

- `-noports` - Do not update the `ports` tree from the ZFS snapshot, keep the tree from previous build instead.

- `-portscvs` - Do not update the `ports` tree from the ZFS snapshot, update it with `cvs update` instead.

- `-norestr` - Do not attempt to build `RESTRICTED` ports.

- `-noplistcheck` - Do not make it fatal for ports to leave behind files after deinstallation.

- `-nodistfiles` - Do not collect distfiles that pass `make checksum` for later uploading to `ftp-master`.

- `-fetch-original` - Fetch the distfile from the original `MASTER_SITES` rather than `ftp-master`.

- `-unlimited-errors` (new codebase only) - defeat the "qmanager threshhold" check for runaway builds. You want this primarily when doing a `-restart` of a build that you expect to mostly fail, or perhaps a `-trybroken` run. By default, the threshhold check is done.

Unless you specify `-restart`, `-continue`, or `-finish`, the symlinks for the existing builds will be rotated. i.e, the existing symlink for `previous` will be deleted; the most recent build will have its symlink changed to `previous/`; and a new build will be created and symlinked into `latest/`.

If the last build finished cleanly you do not need to delete anything. If it was interrupted, or you selected `-nocleanup`, you need to clean up clients by running

`build cleanup `${arch}` `${branch}` `${buildid}` -full`

`errors/`, `logs/`, `packages/`, and so forth, are cleaned by the scripts. If you are short of space, you can also clean out `ports/distfiles/`. Leave the `latest/` directory alone; it is a symlink for the webserver.

> Note: `dosetupnodes` is supposed to be run from the `dopackages` script in the `-restart` case, but it can be a good idea to run it by hand and then verify that the clients all have the expected job load. Sometimes, `dosetupnode` cannot clean up a build and you need to do it by hand. (This is a bug.)

Make sure the `${arch}` build is run as the ports-`${arch}` user or it will complain loudly.

> Note: The actual package build itself occurs in two identical phases. The reason for this is that sometimes transient problems (e.g. NFS failures, FTP sites being unreachable, etc.) may halt a build. Doing things in two phases is a workaround for these types of problems.

Be careful that `ports/Makefile` does not specify any empty subdirectories. This is especially important if you are doing an -exp build. If the build process encounters an empty subdirectory, both package build phases will stop short, and an error similar to the following will be written to ``${arch}`/`${branch}`/make.[0|1]`:

    don't know how to make dns-all(continuing)

To correct this problem, simply comment out or remove the `SUBDIR` entries that point to empty subdirectories. After doing this, you can restart the build by running the proper `dopackages` command with the `-restart` option.

> Note: This problem also appears if you create a new category `Makefile` with no `SUBDIR`s in it. This is probably a bug.

Example 1. Update the i386-7 tree and do a complete build

`dopackages.7 i386 -nosrc -norestr -nofinish`

`dopackages.wrapper i386 7 -nosrc -norestr -nofinish`

Example 2. Restart an interrupted amd64-8 build without updating

`dopackages.8 amd64 -nosrc -noports -norestr -continue -noindex -noduds -nofinish`

`dopackages.wrapper amd64 8 -nosrc -noports -norestr -continue -noindex -noduds -nofinish`

Example 3. Post-process a completed sparc64-7 tree

`dopackages.7 sparc64 -finish`

`dopackages.wrapper sparc64 7 -finish`

Hint: it us usually best to run the `dopackages` command inside of `screen(1)`.

## 4.2 `build` command

You may need to manipulate the build data before starting it, especially for experimental builds. This is done with the `build` command. Here are the useful options for creation:

- `build create `arch` `branch` [`newid`]` - Creates `newid` (or a datestamp if not specified). Only needed when bringing up a new branch or a new architecture. (TODO: document whether newid must be specified as `latest` in the new codebase.)

- `build clone `arch` `branch` `oldid` [`newid`]` - Clones `oldid` to `newid` (or a datestamp if not specified).

- `build srcupdate `arch` `branch` `buildid`` - Replaces the src tree with a new ZFS snapshot. Don't forget to use `-nosrc` flag to `dopackages` later!

- `build portsupdate `arch` `branch` `buildid`` - Replaces the ports tree with a new ZFS snapshot. Don't forget to use `-noports` flag to `dopackages` later!

## 4.3 Building a single package

Sometimes there is a need to rebuild a single package from the package set. This can be accomplished with the following invocation:

`/var/portbuild/evil/qmanager/packagebuild `amd64` `7-exp` `20080904212103` `aclock-0.2.3_2.tbz``

Prev | Home | Next
Chroot Build Environment Setup |  | Anatomy of a Build