pkgsrc/ pkgdb-change

NetBSD package database directory change

The pkgsrc default database location on NetBSD has changed from /var/db/pkg to /usr/pkg/pkgdb (${PREFIX}/pkgdb).

This does not affect other operating systems that already were set up using the bootstrap script. This script already defaulted to ${PREFIX}/pkgdb.

If you are running NetBSD, please read this page carefully all the way through and including the "Known Issues" section.

High-level summary

On NetBSD, pkgsrc's default location for the package database has changed location. New versions of tools were released in early December of 2020 that used the new locations, along with migration instructions. In mid-December, another tool version was released that will quietly use the old location on not-yet-migrated systems. pkgsrc now has a script that can check a system, fix common issues, and do the migration.

The database location has a compiled-in default, and also is set in config files. These instructions say to ALWAYS set the config files, for robustness, even though it is often not necessary. Basically, once PKG_DBDIR is set in config files, old or new tools will use your chosen location.

pkgsrc users who have not run the early December tools can likely simply ignore all of this and continue to use the old database location. The old tools will use the old location, and new tools will notice the old location and silently use it. However, those who have used the new tools should check their situation carefully.

All users should consider running the check/fix script, which will set the database location in use on the system in config files. Alternatively, do this manually - it is very simple.

The rest of this page has instructions, using a migration script, a "Concepts" section, for a fuller discussion of the situation, and then a "manual instructions" section.

On the necessity of setting PKG_DBDIR {#set-PKG_DBDIR}

While in theory it is not necessary to set PKG_DBDIR in all cases, enough people have had problems that the standard advice is:

  • Set PKG_DBDIR in /etc/pkg_install.conf.
  • Set PKG_DBDIR in /usr/pkg/etc/pkg_install.conf regardless of whether your system currently has pkgtools from pkgsrc installed.
  • Set PKG_DBDIR in /etc/mk.conf if you build from source.

If your database is in /var/db/pkg then deviating from the above 3 guidelines will almost certainly cause trouble.

Note that pkg_add etc. may refuse to run if /var/db/pkg exists and PKG_DBDIR is not explicitly set.

In practice, the following deviations can be considered if your database is in the standard location of /usr/pkg/pkgdb and /var/db/pkg does not exist:

  • If you have confirmed that PKG_DBDIR is set properly without an override in mk.conf, that can be omitted from mk.conf.
  • If you do not use binary packages built by others (including official ones), then you can likely drop setting PKG_DBDIR in both pkg_install.conf files
  • If you use binary packages, but you check them to be sure their PKG_DBDIR definition matches yours, you can probably drop setting PKG_DBDIR. However, consider if the benefit of removing two config lines outweighs the risk of trouble.

Instructions in all cases

First, make a backup.

If you have pkgsrc on your system, ensure that either you are on 2020Q3, 2020Q4, some later branch, or that you are at pkgsrc head. Do not use pkgsrc from earlier in December. If you have no sources from pkgsrc, that's fine.

Check that /usr/sbin/pkg_add -V and /usr/pkg/sbin/pkg_add -V (if it exists) are either <= 20200701 or >= 20201218. If you have a version that is early December, then:

  • If it is the base system, update your base system along the branch again.

  • If it is in pkgsrc, remove it with pkg_delete -f -f pkg_install. If pkgin is installed, remove that too, and re-add it with pkg_add from a binary package set of 2020Q3 or 2020Q4, or build it from source after following the rest of the instructions.

These instructions are for a LOCALBASE of /usr/pkg. This is the LOCALBASE for the versions of package tools provided by the base system. If you have configured a different one, read the script and either adapt it, or figure out what to do manually. Besides the backup mentioned in the instructions, make an extra backup. Remember that if the script breaks you get to keep both pieces!

Instructions for remaining with /var/db/pkg via the migration script

This section provides instructions on how to stay with the old database location. This is the simplest and therefore arguably the safest approach for most users for now.

Obtain _NetBSD-pkgdb from the top level of pkgsrc (HEAD or 2020Q4). Alternatively get it from http://ftp.netbsd.org/pub/pkgsrc/packages/NetBSD/_NetBSD-pkgdb.

Run the script with the argument check. This does not need root.

One approach is to manually fix issues reported by check.

The other is to run with the argument fix, as root. The script will add PKG_DBDIR= lines to pkg_install.conf (base and pkgsrc) and also to mk.conf. (The script will also patch up paths in /var/db/pkg.refcount if they point to the other location.)

Instructions for migrating to /usr/pkg/pkgdb via the migration script

Then, run the script with check. Then run it with migrate. It will move the two database directories, and then point the config files at the new location. Almost certainly it will report issues with the refcount database and fix them. After that run check to see if all issues were resolved.

Split-brain recovery

If you have both /var/db/pkg and /usr/pkg/pkgdb, then see this pkgsrc-users message about how to recover.

NB: There is an error in the message. In step 7, it says /usr/pkg/pkgdb and it should say /usr/pkg/pkgdb.refcount.

Known issues with the migration script

/var/db/pkg and /var/db/pkg.refcount mismatch

The script assumes that there is always a refcount directory along with the main database directory. As part of trying to check that all is OK before migrating, the script insisted on this correspondence and errored out if one existed without the other. It turns out that the refcount directory is created as necessary, so it's not present in new installs and is not present with only a few packages, until a package is installed that needs a refcount entry.

If you get a message about this, and you look and see that you have /var/db/pkg but not /var/db/pkg.refcount, simply (as root, so it has the same uid as the db dir) mkdir /var/db/pkg.refcount.

The same applies for /usr/pkg/pkgdb and /usr/pkg/pkgdb.refcount.

Concepts

pkgsrc stores information about installed packages, and this is called the package database or pkgdb for short. This is in two parts: pkgdb, which has per-package information like lists of installed files and records of other packages that depend on the package, and pkgdb.refcount, which keeps track of how many packages use things like userids.

The default value of PKG_DBDIR in the make world changed in December 2020. The old locations were /var/db/pkg and /var/db/pkg.refcount. The new standard location is /usr/pkg/pkgdb and /usr/pkg/pkgdb.refcount.

The refcount directory is always next to the main directory, and therefore we will often omit mentioning it.

There are tools used to install packages, such as pkg_add, pkg_delete, pkg_info. Together these are referred to by the name of the package containing the tools: pkg_install.

Note that one could before and still can bootstrap on NetBSD and have a separate pkgdb for that separate prefix.

The location of the pkgdb is configured in two main places, and it is important for all the pieces of pkgsrc to have a consistent view of the location.

The primary place is the variable PKG_DBDIR within the make (/etc/mk.conf) world. That value used to default to /var/db/pkg, and now defaults to /usr/pkg/pkgdb (really ${PREFIX}/pkgdb). If set explicitly, that explicitly-set value is used. The default or configured value becomes the default value in pkg_install when those tools are compiled.

The base system copy of pkg_install (/usr/sbin/pkg_add and so on) has a compiled-in default. Before December 2020, that was /var/db/pkg, and now it is /usr/pkg/pkgdb.

The second and third places are the configuration variable PKG_DBDIR in /etc/pkg_install.conf (for the base tools) and /usr/pkg/etc/pkg_install.conf (for the /usr/pkg/sbin version of the tools).

While the actual requirement for correct operation is that the runtime value of the variable in the make world match the path that is chosen by pkg_install, it is safer to set this explicitly in all three places if there is any chance that either pkgsrc or the base system tools will change. Essentially, the argument is that it is on balance less time to set the variable in three places (one line each) than to debug problems arising from changes and incorrect assumptions.

Up-to-date versions of pkg tools, while defaulting to /usr/pkg/pkgdb, will detect if the database is in /var/db/pkg, and if so use it there.

refcount complexity

The refcount part of the database contains paths relative to the database. While arguably this is a bug, it means that additional care is needed when migrating.

Strategies

There are basically two strategies for users: leave the database where it is and set the variables to point to it, and move the database and set variables to point to it.

Manual instructions

See the instructions above about being clear that you have either old tools or new enough tools.

Staying with /var/db/pkg

  1. Put PKG_DBDIR=/var/db/pkg in /etc/mk.conf.
  2. Put PKG_DBDIR=/var/db/pkg in /etc/pkg_install.conf.
  3. Put PKG_DBDIR=/var/db/pkg in /usr/pkg/etc/pkg_install.conf.

Moving to /usr/pkg/pkgdb

Do these steps all at once, with no pkg_install operations in the middle.

  1. Put PKG_DBDIR=/usr/pkg/pkgdb in /etc/mk.conf.
  2. Put PKG_DBDIR=/usr/pkg/pkgdb in /etc/pkg_install.conf.
  3. Put PKG_DBDIR=/usr/pkg/pkgdb in /usr/pkg/etc/pkg_install.conf.
  4. mv /var/db/pkg /usr/pkg/pkgdb
  5. mv /var/db/pkg.refcount /usr/pkg/pkgdb.refcount

In every file in the refcount directory, change the old path to the new path. (See the script section above.)

pkgin

If you use pkgin, be sure to have set the PKG_DBDIR variables in all three places before you run it.

When you use pkgin and upgrade your NetBSD base system first make sure to manually update the installed pkg_install package as well since pkgin uses that package's PKG_DBDIR information for finding the database.

pbulk

pbulk has the path to the pkgdb in its pbulk.conf file. Update it:

pkgdb=/usr/pkg/pkgdb

pbulk.conf is often located at /usr/local/pbulk/usr/pbulk/etc/pbulk.conf.

Make sure to set PKG_DBDIR for mk.conf, /etc/pkg_install.conf, and /usr/pkg/etc/pkg_install.conf.