Another year of pkgsrc

News since last pkgsrcCon

Joerg Sonnenberger <joerg@NetBSD.org>

The NetBSD Foundation

Preamble

Focus on infrastructure changes since last pkgsrcCon
No credits — I'm not responsible for many of the changes
No specific order — I tried to group them semantically
Contact me if I missed something important

Performance improvements

Ever ktrace(1)'d bmake in pkgsrc?
bsd.prefs.mk gets included all over the place
Include guards, but still 16 KB to parse by bmake
Idea: include guard + include of real file == fast
bsd.fast.prefs.mk: 10% time saving for kdebase3

Apache tribes

Apache version 2 as new default
More consistent naming for Apache and PHP/PEAR modules
- Inspired by Python handling
- Apache prefix is ap13 vs. ap2 vs. ap22
- PHP packages prefixed with php4/php5
- ap-php dependency selects both Apache and PHP version
Apache Portable Runtime (APR) still a mess

Splitting BUILD_DEFS

List of settings affecting the build
Problem: some are used early by the infrastructure
Split into package settable files and infrastructure parameters
LOCALBASE, PKGINFODIR, PKGMANDIR are the latter
Added BUILD_DEFS_EFFECTS to describe the resulting changes

Unphase build

Concept of barriers replaces concept of build phases
Simplifies handling of su targets
Eliminates most recursive makes
Bug: NO_CONFIGURE killed the wrappers
Drop .ORDER, pkgsrc does not support -j
RECURSIVE_MAKE vs. MAKE, BUILD_ENV vs. PKGSRC_MAKE_ENV
Resort processing, depends -> tools -> extract -> wrapper

Output consistency

Before:
- Echos for warnings
- More echos for errors
- ...and mixed between build output
After:
- STEP_MSG, PHASE_MSG, WARNING_MSG, ERROR_MSG
- DELAYED_*_MSG for later inspection
Improve the config.h parsing to make debug more useful
Print the original arguments after failing wrapped commands

CHECK_FILES improvements

Fixed with PREFIX as symlink
Default for developers (and most bulk builds)
Make it optional for platforms that can't support it
Sort file lists first, avoids false positive
Skip PACKAGES, DISTDIR, fontconfig cache
Deobscure output for overwritten files

Package consistency (1)

Stricter PLIST entries for font packages:
- fonts.alias is static content, must be in PLIST
- fonts.encoding is handled by the infrastructure
- To Do: fontconfig handling
Pass MAKECONF down, su-targets didn't get it
Fixes to run with set -u and set -e

Package consistency (2)

Optional check for TOOLS_DIR references
Permission check added, not active by default (yet)
SHLIB_HANDLING -> CHECK_SHLIBS_SUPPORTED, controlled by CHECK_SHLIBS, active for developers
Check for unexpanded ${prefix} and ${exec_prefix} in headers

mtree

MTREE_FILE and MTREE_ARGS are internal now
NO_MTREE as default post-2007Q1
to be gone after 2007Q2

DESTDIR support (1)

Don't use DESTDIR internally
Removal of MAKE_PACKAGE_AS_ROOT, DESTDIR replaces this mostly
_INSTALL_ROOT_USER -> REAL_ROOT_USER, used e.g. for SETUID_ROOT_PERMS
Added _SU_ROOT_USER to allow unprivileged builds mixed with su-targets
Honour it in warning/error messages
Fixed check-shlibs to deal with relative paths

DESTDIR support (2)

Install to area in WRKDIR
Conditionally run install target via su-target
For USE_DESTDIR=yes run make clean as root
package-install target
Use @owner / @group to set ownership of files with USE_DESTDIR=full
Problem with tar and intermediate directories -> pkg_create changes pending to fix issues with this

Modular Xorg support (1)

kill commands to make X11BASE if it didn't exist, it was never used anyway
Add include guards in mk/x11.b3.mk as this is now included quite often
Fail properly if no valid X11_TYPE is set
No ${WRKDIR}/.x11-buildlink for modular Xorg
No magic mk/x11.b3.mk include for USE_X11BASE

Modular Xorg support (2)

No xpkgwedge
USE_XPKGWEDGE moved to bsd.prefs.mk -> overriding of X11_TYPE per platform
Use xdg-dirs directly for modular Xorg
Fix bugs with Xaw3d and friends: don't use Xaw headers and possibly even wrong lib
Handle LOCALBASE before X11BASE – more specific first and X11_TYPE=xorg is going away

Modular Xorg support (3)

For mk/motif.b3.mk, move mk/x11.b3.mk include into non-openmotif, non-lesstif case
mkfontscale vs type1inst
Include modular Xorg b3.mk files, redirect to mk/x11.b3.mk for non-modular builds

Bulk build system (1)

Remove old .bulk-not_available
BULK_BUILD_ID_FILE is using BULK_ID now
Avoid zombies in printindex on NetBSD
Hack sort-packages to speed-up vulnerable search, 60% improvement
Don't try to pkg_delete -f
Remove packages with pkg_delete -r "*", no iteration

Bulk build system (2)

Cleanup by using awk in some places instead of sed
Optionally preserve WRKDIR on failure, print a message in that case
Call make once in pkgtools/pkglint to extract variables in one go
Optionally run only post-build steps
Replace wc with awk, saving two forks/package

Bulk build system (3)

Include PKGSRC_COMPILER in the report
Better handle *SPECIFIC_PKGS, e.g. non-existing directories
Add lang/nawk and sysutils/install-sh to the precious package list (about to change)
Refix the upload script to never upload restricted packages
Create the pkg_summary file
With current pkg_install, record all packages used as dependency

Refactoring of the infrastructure

Implementation of most parts split into separate files and direcorties
Documentation of public and private variables and targets
More code moved from .mk files to separate shell/awk scripts
Abstraction of low-level pkg_install access to allow different backends
Overall aim: smaller, better manageable pieces

Refactoring examples

show-all-depends-dirs and show-all-depends-dirs-excl replaced by awk scripts
Fetch code moved into a shell script
Checksum handling moved into a shell script
Locking uses make macros, split lock into WRKDIR and PKGDB lock
Add ${RUN} to replace ${_PKG_SILENT}${_PKG_DEBUG}

Fetching

FETCH_MESSAGE is public now
Renamed DYNAMIC_SITE_* to DYNAMIC_SITES_*
Prefer ${FILESDIR}/getsite.sh over SITE_<file> over MASTER_SITES
Randomize MASTER_SITES for normal users

Extracting

Handle new cases: .gem, .cpio, -tar.gz
EXTRACT_DIR when WRKDIR is not the right target directory
Replace gzcat with gzip -cd, latter more portable

Package changes

Added PostgreSQL 8.2, removed 7.3 and 7.4
Removed teTeX 2
SuSE 7.3 gone, SuSE 9.1 legacy
NetBSD before 2.x requires bootstrap
XFree86 removed

Compiler changes (1)

Enumerate languages explicitly, don't add to USE_LANGUAGES (nice breakage!)
Generic C99 support
- Annotate it as language
- For GCC, require 3.x or newer
- Solaris difficult due to broken system headers

Compiler changes (2)

Missing languages get ${FALSE} wrappers
f2c-f77 aliased as g77 and f77 for libtool
f2c as Fortran backend adds C requirement
Allow full paths for CPP, e.g. Solaris and IRIX need it

Compiler changes (3)

More GCC options to filter out / translate for other compilers
Many SunPro fixes, e.g. -O0/-O1 -> -xO0/-xO1
Allow ccache to use cache outside of ${WRKDIR}

Tools changes

intltool and itools as full tools, reduces hackery
Many tools get a broken ${FALSE} version by default
- Better catches missing dependencies in normal environments
- Some fallout, e.g. pkg-config breaks GTK detection

Wrapper changes

Turn off WRAPPER_DEBUG by default, work.log could grow to over 100 MB
Make the wrapper cache parallel build safe by using copy+rename
imake-check uses real imake, less issues with the wrappers
Reordered to be back between patch and configure

Portability fixes (1)

Normalize ${TEST} vs [ ], use the later
Use test -f instead of ls to check for libtool, config.*, detects broken symlinks
CHECK_PORTABILITY
- Warn about $RANDOM
- Error out on test == and [ == ] (bash vs coreutils)

Portability fixes (1)

Many sh implementations are reading byte-wise, avoid it in some cases
/dev/stderr in awk scripts is bad, e.g. IRIX doesn't have it
MAKEFILE -> MAKE_FILE, not useful otherwise
Use find -o, not find -or

MAKE_JOBS

Allows parallel building within a package
Works by passing down -jX to make
Dangerous and prone to failures

pkgsrc documentation

Refactored help subsystem: make help topic=pkgname
Now recognises "# variable" comments e.g. in mk.conf
make show-all
Split many docs into user and developer parts
When printing supported options, also show the current package
Explain errors from check framework

SUBST framework

SUBST_VARS.* for replacement of @FOO@ by ${FOO}
SUBST_SKIP_TEXT_CHECK.* disables file(1) check
Warn about unmodified files
Even simpler for awk, bash and sh: REPLACE_*

Buildlink changes (1)

BUILDLINK_ORDER changes to allows tree-like output
BUILDLINK_RECOMMENDED.foo renamed to BUILDLINK_ABI_DEPENDS.foo
BUILDLINK_DEPENDS.foo renamed to BUILDLINK_API_DEPENDS.foo
IGNORE_RECOMMENDED renamed to USE_ABI_DEPENDS with inverted meaning

Buildlink changes (2)

BUILDLINK_TRANSFORM.* renamed to BUILDLINK_FNAME_TRANSFORM.*
Common handling for build options

Other cleanup (1)

If running "make replace", it replaces install
Silence rm of the work symlink
Better test infrastructure, similiar to normal build variables
Deprecate PKG_SKIP_REASON
Override public targets if PKG_FAIL_REASON is set
Kill trailing spaces
Build dependencies of build dependencies no longer work

Other cleanup (2)

Bootstrap dependency phase
OVERRIDE_DEPTH for highly nested autoconf users
NOT_FOR_UNPRIVILEGED / ONLY_FOR_UNPRIVILEGED
BSD_MAKE_ENV to pass down BINOWN, BINGRP, ...
No interpreter check in share/doc and share/examples
More quoting fixes
NO_DEPENDS renamed to SKIP_DEPENDS

The end

Questions?