FreeBSD CURRENT and STABLE Binary Upgrades

"Because tracking CURRENT shouldn't be a rite of passage"

Disclaimer

This proof-of-concept is not a publication of FreeBSD.

Description

up.bsd.lv is a proof-of-concept of binary updates using freebsd-update(8) for FreeBSD 13.0-CURRENT and 12-STABLE to facilitate the exhaustive testing of FreeBSD and the bhyve hypervisor and OpenZFS 2.0, and to help elevate ARM64 to Tier 1 status. Updates are based on the SVN revisions of official FreeBSD Release Engineering weekly snapshots.

Requirements

Upgrades use SSL to the extent possible and thus require the ca_root_nss package, certctl(8), or another certificate bundle to support the retrieval of two .ssl files. Note that an accurate system time and date may be required for proper SSL handling.

Upgrades can be performed on 12.1 RELEASE or updated systems, or with a provided disc1.iso or memstick image.

Upgrading

To upgrade from Official 13.0-ALPHA1 or later to Official 13.0-BETA* or up.bsd.lv 13.0-BETA*:

freebsd-update -r 13.0-BETA3 -s up.bsd.lv -k b82bddb79643d1fe02685dc81ef00c1182a93ea8fb319bf3cd7fb9b7eee4ce4e --currently-running 13.0-CURRENT-ALPHA1 upgrade

To upgrade to 12-STABLE r363433 from a 12.1 RELEASE or patch level system, specify the up.bsd.lv server and signature:

freebsd-update upgrade -r 12.1-STABLE-364849 -s up.bsd.lv -k b82bddb79643d1fe02685dc81ef00c1182a93ea8fb319bf3cd7fb9b7eee4ce4e
freebsd-update install
reboot
freebsd-update install

To upgrade to 13-CURRENT r363439 from a 12.1 system:

freebsd-update upgrade -r 13.0-CURRENT-364846 -s up.bsd.lv -k b82bddb79643d1fe02685dc81ef00c1182a93ea8fb319bf3cd7fb9b7eee4ce4e
(install, reboot, install)

Once on an up.bsd.lv train, the available upgrades can be listed with:

freebsd-update listupdates

To upgrade to a listed upgrade: freebsd-update upgrade -r 13.0-CURRENT-363439
(install, reboot, install)

To upgrade to an up.bsd.lv release from an official FreeBSD RE snapshot:

Identify a destination snapshot on up.bsd.lv
Identify the current running version with uname -r
Run the following upgrade command with the source and destionation versions:

freebsd-update upgrade -r <destination revision> -s up.bsd.lv -k b82bddb79643d1fe02685dc81ef00c1182a93ea8fb319bf3cd7fb9b7eee4ce4e --currently-running <current revision>

Install, reboot, install as normal.

Note however that /etc/freebsd-update.conf and /usr/sbin/freebsd-update will NOT be updated during this process. Manually replace them with the up.bsd.lv freebsd-update.conf and freebsd-update.

Packages and ports should work no differently than on the official snapshots but freebsd-update fetch is not supported due to the nature of freebsd-update(8) patch levels which are tied to security advisories.

Differences

up.bsd.lv releases are different from the official FreeBSD CURRENT snapshots in the following ways:

Naming convention: FreeBSD-13.0-HEAD-######-amd64-disc1.iso
Kernels are built with the GENERIC-NODEBUG configuration file
/etc/freebsd-update.conf is modified to use up.bsd.lv
/usr/sbin/freebsd-update is modified to "xargs" parallel phttpget (Thank you Allan Jude)
up.bsd.lv uses https to the extent possible
freebsd-update(8) does not output to the $PAGER
Update payloads are build from disc1 rather than the DVD image when available, and from distribution sets if not available (i.e. on ARM64)

Available Updates

freebsd-update listupdates retrieves the list of available-updates.txt

Installer Downloads

Changelists for Available Upgrades

News

2021-07-26 - This Proof of Concept was extremely informative and is taking a break to build a Proof of Concept replacement for freebsd-update. Thank you everyone who helped out with it!
2021-07-17 - 12-STABLE, 13-STABLE, and 14-CURRENT amd64 builds are up, plus riscv on 14-CURRENT - Build on 13.0R
2021-07-09 - 12-STABLE, 13-STABLE, and 14-CURRENT amd64 builds are up, plus riscv on 14-CURRENT - RIP 11-STABLE snapshots
2021-07-02 - 12-STABLE, 13-STABLE, and 14-CURRENT amd64 builds are up, plus riscv on 14-CURRENT delayed by an emergency room visit
2021-06-26 - 12-STABLE, 13-STABLE, and 14-CURRENT amd64 builds are up, plus riscv on 14-CURRENT
2021-06-22 - June 17th builds are up for 12-STABLE, 13-STABLE, and 14-CURRENT amd64, plus riscv on 14-CURRENT
2021-06-12 - 12-STABLE, 13-STABLE, and 14-CURRENT amd64 builds are up, plus riscv on 14-CURRENT
2021-06-03 - 13-STABLE, and 14-CURRENT amd64 builds are up, plus riscv on 14-CURRENT
2021-06-02 - 12-STABLE, 13-STABLE, and 14-CURRENT amd64 builds are up, plus riscv on 14-CURRENT
2021-05-24 - 12-STABLE, 13-STABLE, and 14-CURRENT amd64 builds are up, plus riscv on 14-CURRENT
2021-05-13 - 12-STABLE, 13-STABLE, and 14-CURRENT amd64 builds are up, plus riscv on 14-CURRENT
2021-05-07 - 12-STABLE, 13-STABLE, and 14-CURRENT builds are up - fewer plaforms, more branches - feeback welcome
2021-05-03 - Two weeks of CURRENT and STABLE builds are up, including the first 13-STABLE build
2021-04-19 - 12-STABLE amd64 and i386 snapshots are up
2021-04-18 - 14-CURRENT amd64, i386, and riscv are up but amd64 needs to be reworked to follow the new, official conventions. Pulling them for now.
2021-04-09 - 12-STABLE amd64, i386, and arm64 builds are up
2021-04-08 - 14-CURRENT amd64, i386, arm64, and riscv are up, with 12-STABLE in the oven.
4321 - 13.0RC5 for all architectures is up for the up.bsd.lv train
2021-04-02 - 12-STABLE amd64, i386, and arm64 builds are up
2021-04-01 - 14-CURRENT amd64, i386, arm64, and riscv are up, with 12-STABLE in the oven. No joke!
2021-03-30 - 13.0RC4 for all architectures is up for up.bsd.lv and official trains
2021-03-22 - 13.0RC3 for all architectures is up for up.bsd.lv and official trains
2021-03-18 8 14-CURRENT amd64, i386, arm64, and riscv are up, along with 12-STABLE arm64, i386, and arm64
2021-03-16 - 13.0RC2 for all architectures is up for up.bsd.lv and official trains
2021-03-11 - 14-CURRENT amd64, i386, arm64, and riscv are up, along with 12-STABLE, minus risc for the same reason as before
2021-03-06 - 13.0RC1 for all architectures is up for up.bsd.lv and official trains
2021-03-04 - 14-CURRENT and 12-STABLE are up but it looks like the riscv VM image build needs to be MFC'd
2021-02-27 - 13.0-BETA4 is up for the up.bsd.lv and official trains
2021-02-25 - 14-CURRENT and 12-STABLE are up
2021-02-25 - amd64 checkout, build, and upload takes about 75 minutes
2021-02-20 - 13.0-BETA3 is up for the up.bsd.lv and official trains
2021-02-19 - 12-STABLE is up, but RISC-V VM image generation failed with "vmimage.subr: unsupported target 'riscv:riscv64'"
2021-02-18 - 14.0 is up and 12-STABLE is in the oven
2021-02-15 - 13.0-BETA2 is up on the official train
2021-02-13 - 13.0-BETA2 is up for the up.bsd.lv train
2021-02-11 - The first 14-CURRENT builds are up! 12-STABLE is up minus RISCV, which appears to not have VM image generation MFC'd
2021-02-10 - A build of D26035, the "bhyve configuration file support" review is up
2021-02-09 - Official amd64 13.0-ALPHAs and BETA1 are up to allow upgrading interchangeably
2021-02-06 - 13.0-BETA1 is up, minus i386 which is offline upstream
2021-02-05 - 12-STABLE is up and we have archived many older updates. If you need to update from an older snapshot, we may still have it around and can share it for you
2021-01-29 - 12-STABLE and 13.0-ALPHA3 are up
2021-01-22 - 13.0-ALPHA2 is up, built with headbanger. arm64 and riscv are having ISO issues
2021-01-21 - No 13.0-CURRENT snapshot, see 13.0-ALPHA1
2021-01-16 - The 2021-01-14 snapshots are rebuilt along with support for upgrading from recent 12.2 patch levels
2021-01-15 - An upstream change to freebsd-update.sh broke ours and we are re-rolling the build. A working freebsd-update for use in place of /usr/sbin/freebsd-update is available in the interim
2021-01-14 - 12-STABLE is now syncing from Git, but RISC-V failed on CURRENT
2021-01-08 - The Git to SVN bridge has been offline and the last 12-STABLE official builds were ambiguously-named. The is fixed and the next 12-STABLE builds should be usable
2020-12-24 - CURRENT builds now pull from git.freebsd.org
2020-12-11 - By popular demand, CURRENT/HEAD upgrades from 13.0-HEAD-255032 onward are named using the incrementing Git commit count, rather than hash, to provide more user-friendly naming. The hash can be seen in the uname -a output. Also added first riscv64 upgrades
2020-11-11 - If you have issues with recent updates, use this freebsd-update script in /usr/sbin - make sure it is executable
2020-11-01 - Builds are mainly on EPYC and NFS. 12-STABLE snapshots are back now that 12.2R has arrived
2020-10-16 - Published the up.bsd.lv Report for interested parties
CURRENT builds are from cgit-beta.freebsd.org now that official snapshots are from Git
The naming is changed back from "CURRENT" to "HEAD" to accomodate an ISO naming issue
The first build from cgit-beta.freebsd.org is up with FreeBSD-13.0-HEAD-c122cf32f2a-amd64-disc1.iso naming to fit in the ISO disk label
The first builds of code under review are available to facilitate testing
FreeBSD 12.1 RELEASE ARM64 is now a supported starting point, allowing for 12.1 RELEASE ARM64 systems to upgrade to 12-STABLE or 13-CURRENT. This has been tested on a Cavium ThunderX system and a Raspberry Pi 3
2020-06-16 - up.bsd.lv was featured on BSD Now
freebsd-update listupdates will list the available upgrades on up.bsd.lv
The published snapshot VM images are all tested for networking, and booting with 1 and 16 vCPUs under bhyve
The build system now supports multiple architectures rather than only amd64
The naming is changed from "HEAD" to "CURRENT" to reduce our diff
2020-06-08 - up.bsd.lv was featured on Valuable News
2020-05-29 - up.bsd.lv launch

Questions and Answers

Q: What is the long-term goal of this effort?
A: To be replaced by official FreeBSD binary updates for FreeBSD CURRENT and STABLE if the proof-of-concept proves successful, moving on to builds of bhyve-related reviews from reviews.freebsd.org.

Q: How do I validate the thumbprint of the update server?
A: Verify that /etc/freebsd-update.conf includes KeyPrint b82bddb79643d1fe02685dc81ef00c1182a93ea8fb319bf3cd7fb9b7eee4ce4e

Q: How do I enable verbose output?
A: freebsd-update ... --debug

Q: Can these releases be used in production?
A: Only you can determine that but some official snapshots have failed to boot.

Q: What do I do if I get kernel mismatch and missing dependency errors with packages?
A: You will probably get the exact same errors with the official snapshots due to the dynamic nature of the CURRENT branch.

Q: Are these patch levels or releases?
A: They are releases because patch levels are tied to security advisories. We are investigating if the patch level facility would be usable.

Q: What platforms have the ARM64 snapshots been tested on?
A: On an OverDrive 1000. Note that the installer may have an issue with the vfs.zfs.min_auto_ashift sysctl.

Q: What if I receive No mirrors remaining, giving up. errors?
A: This indicates that we need to rebuild an upgrade payload after a storage upgrade and path change. Please report these by email if you encounter them.

Q: What if I receive Cannot upgrade from a version that is not a release if using an early "HEAD" build?
A: Add "HEAD" to the whitelist in the /usr/sbin/freebsd-update shell script. i.e. *-RELEASE | *-ALPHA* | *-BETA* | *-RC* | *-STABLE* | *-HEAD* | *-CURRENT*)

Q: How can I help?
A: 1. Test FreeBSD 2. Test this testing resource. 3. All hardware and network resources are provided out-of-pocket and you are welcome to contribute.

Discoveries

freebsd-update(8) will not handle an update server name specified with a protocol such as https://up.bsd.lv
freebsd-update(8) depends on phttpget(8) which does not support https. Presumably the internal checksum validation is adequate to mitigate most threats and the use of http would allow significantly older releases to update to newer ones, provided that the upgrade traversal is supported. up.bsd.lv uses https to the extent that it is supported, specifically for the latest.ssl and pub.ssl files.
freebsd-update does not appear to have a notion of target architecture.
freebsd-update is oriented to DVD images but will work with substituted disc1.iso images.
ARM64 snapshots do not include ISO or DVD images and we are experimenting with using distribution sets directly. Moving entirely to distribution sets might be wise.
freebsd-update.conf moved in 12-STABLE as part of the packaged base effort.
11-STABLE does not appear to distinguish architectures in the object directory, making it difficult to support.
freebsd-update does not check the architecture for "to-" incremental upgrades built with build-upgrade-patches.sh and does not fall back to the nonr- "to-" upgrade.
The 362037 official and 362074 unofficial snapshots fail with "ip length 328 disagrees with bytes received 326" when a DHCP request is made, and fail on mountroot upon second boot, both of which appear to be resolved.

Acknowledgments

Thank you Conor Beh for co-developing this service and Colin Percival for developing freebsd-update(8). Thank you FreeBSD Release Engineering Team for maintaining the FreeBSD build(7) infrastructure. Thank you ScaleEngine for providing the bandwidth for this service.

up.bsd.lv is developed by Michael Dexter