FreeBSD 13.2 upgrade to 14.0 – properly detailed and (hopefully) correct way

---

Hey there. So now, FreeBSD 14.0 is released! It introduces several notable features and enhancements that contribute to its performance, scalability, and virtualization capabilities.

Here are few of the new features/changes I really loved;

Default TCP Congestion Control Mechanism – CUBIC:
FreeBSD 14.0 adopts CUBIC as the default congestion control mechanism for TCP. CUBIC is known for its efficiency in managing network congestion, providing improved performance and stability for network communications.

Increased Core Support on amd64 and arm64 Platforms:
This release extends core support, now accommodating up to 1024 cores on both the amd64 and arm64 platforms. This enhancement is particularly significant for high-performance computing and large-scale server deployments, allowing FreeBSD to harness the power of modern multi-core systems.

The bhyve Hypervisor Enhancements:
The bhyve hypervisor receives notable upgrades in FreeBSD 14.0, including support for TPM (Trusted Platform Module) and GPU passthrough. These additions enhance virtualization capabilities, enabling users to leverage advanced security features and dedicate GPU resources to virtual machines for improved graphics performance.

ZFS Upgrade to OpenZFS Release 2.2:
The ZFS filesystem has been upgraded to OpenZFS release 2.2. This brings significant performance improvements.

KTLS (kernel TLS implementation):
The addition of KTLS with offload support for TLS 1.3 enhances the performance, security and efficiency of encrypted communication within the file system. Actually it was possible to activate kernel-level TLS implementation in 13.2 by adding kld_list=”ktls_ocf” in rc.conf file, this is not needed now, as it comes already loaded with FreeBSD 14.

So how to upgrade from 13.2 to 14.0?
It’s important to fetch and install the latest updates and patches of your current system, before proceeding to the upgrade. And of course, you must always do a backup of your current system before you start an upgrade procedure.

Upgrading FreeBSD is possible either by compiling from source or using freebsd-update tool (binary upgrade). I follow the latter option for its efficiency and practicality, so here are the whole set of the commands – fetching&installing latest pending updates and patches, updating the installed packages, upgrading to 14.0-RELEASE and some clean-ups;

[root@bsdb0x:~] freebsd-update fetch
#Then, proceed with 'install' if (and ONLY if!) fetching the packages is successfully done
[root@bsdb0x:~] freebsd-update install
[root@bsdb0x:~] pkg update && pkg upgrade
[root@bsdb0x:~] freebsd-update -r 14.0-RELEASE upgrade
[root@bsdb0x:~] freebsd-update install
[root@bsdb0x:~] reboot

#This might take time - so just wait - fetching and installing of patches took -for me- ~2 hours
[root@bsdb0x:~] freebsd-update install
...
[root@bsdb0x:~] pkg bootstrap -f
[root@bsdb0x:~] pkg-static install -f pkg
[root@bsdb0x:~] pkg update
[root@bsdb0x:~] pkg upgrade
[root@bsdb0x:~] freebsd-update install
[root@bsdb0x:~] reboot

#Cleaning up temporary pkg oldies and now-unused orphan dependencies:
[root@bsdb0x:~] pkg clean
[root@bsdb0x:~] pkg autoremove

By the way, all ZFS users should do:

sudo sysctl vfs.zfs.dmu_offset_next_sync=0

also add:

vfs.zfs.dmu_offset_next_sync=0

in sysctl.conf file, as it has been reported that there’s a silent data corruption bug in ZFS, here.
As of 2023-12-01, the issue above is solved by the FreeBSD Errata: FreeBSD-EN-23:16.openzfs.

---

Now, due to a newer version of OpenZFS, it’s time to do a Bootcode Update. This is needed only when you’ve a system booting from ZFS, if your system is running on UFS, then you don’t need the steps below.

First, check if your boot method is traditional CSM/BIOS or UEFI;

[root@bsdb0x:~] sysctl machdep.bootmethod
machdep.bootmethod: BIOS

So we have traditional BIOS boot there. If the answer is UEFI, skip to “UEFI Bootcode – Upgrade” section below.

BIOS/CSM Bootcode – Upgrade:

“gpart show” will show you the list of partitions and their indexes. Look for the “freebsd-boot” partition. That’s the index you need to use. And update the bootcode on every disk that has a “freebsd-boot” partition.

[root@bsdb0x:~] gpart show
=>      40  80002992  da0  GPT  (38G)
        40      1024    1  freebsd-boot  (512K)
      1064       984       free  (492K)
      2048   4194304    2  freebsd-swap  (2.0G)
   4196352  75804672    3  freebsd-zfs  (36G)
  80001024      2008       free  (1.0M)

Definitely do verify what index your system uses for the ‘freebsd-boot’ partition because it doesn’t necessarily have to be index 1. In some cases there’s an ‘efi’ and a ‘freebsd-boot’, so the index for ‘freebsd-boot’ becomes: 2.

Indexes above are;
1 freebsd-boot
2 freebsd-swap
3 freebsd-zfs

So, shortly; disk: da0 – index: 1 – thus we proceed with updating the boot code(s).

I strongly suggest to update boot code(s) BEFORE upgrading the ZFS pool.

[root@bsdb0x:~] gpart bootcode -b /boot/pmbr -p /boot/gptzfsboot -i 1 da0
partcode written to da0p1
bootcode written to da0

If there would be one more disk, like “da1”, (and in case if it has ‘freebsd-boot’ partition, too) we’d update it too, with;

gpart bootcode -b /boot/pmbr -p /boot/gptzfsboot -i 1 da1

And now upgrading the ZFS pool (assuming the pool is named as ‘zroot’ – the default one, or you can learn it with: ‘zpool list‘ command);

[root@bsdb0x:~] zpool upgrade zroot

This system supports ZFS pool feature flags.

Enabled the following features on 'zroot':
  edonr
  zilsaxattr
  head_errlog
  blake3
  block_cloning
  vdev_zaps_v2

We’re done. One more reboot.

So when the system is back, if everything seems fine and whole the system is set&correctly working, you might wish –IF YOU DON’T NEED THEM– to delete automatically-saved boot environments on ZFS in your system, because ‘freebsd-update‘ tool creates ZFS snapshots of your boot env. with each system upgrade.

You can disable this behavior by setting CreateBootEnv directive NO in /etc/freebsd-update.conf.

Saved/older Boot Environments might be useful for some, in case of an upgrade failure for instance, however these snapshots grow rapidly, and they will consume too much disk space..

Once, I was in such a situation and the services running on the server were alarming “out of disk space!“. I barely found what was causing that.. Since then, after a system upgrade, I test the system for few days, and if everything is good, I usually eliminate older boot environments and saving –sometimes lots of– disk space like this;

Get the list of the older and current boot env. first:

bectl list

and remove them one by one with this command: (KEEP the one named ‘default’, under ‘BE’, which is the active boot environment of your current system)

bectl destroy -Fo name_of_your_listed_boot_env_under_BE_Column

Example:

bectl destroy -Fo 13.2-RELEASE-p4_2023-11-21_234504

bectl destroy -Fo 13.2-RELEASE-p5_2023-11-22_000013

Now, time to move to the UEFI part.

EFI Bootcode – Upgrade:

So far, I explained updating the bootcode(s) for traditional CSM/BIOS systems. However, in case of having UEFI system instead; in order to update EFI system partitions, we don’t proceed with “gpart bootcode” but instead we simply copy the updated loader.efi file to EFI partitions on our disk(s).

First, check for the disk names&partition numbers with “gpart show“;

[root@bsdb0x:~] gpart show
=>      40  33554352  ada0  GPT  (16G)
        40    532480     1  efi  (260M)
    532520      1024     2  freebsd-boot  (512K)
    533544       984        - free -  (492K)
    534528   4194304     3  freebsd-swap  (2.0G)
   4728832  28823552     4  freebsd-zfs  (14G)
  33552384      2008        - free -  (1.0M)

=>      40  33554352  ada1  GPT  (16G)
        40    532480     1  efi  (260M)
    532520      1024     2  freebsd-boot  (512K)
    533544       984        - free -  (492K)
    534528   4194304     3  freebsd-swap  (2.0G)
   4728832  28823552     4  freebsd-zfs  (14G)
  33552384      2008        - free -  (1.0M)

So we got; ada0 disk with ‘1‘ efi partition, and ada1 disk with ‘1‘ efi partition.

Proceeding with every disk that has an “efi” partition:

First, the “ada0” disk:

[root@bsdb0x:~] mount_msdosfs /dev/ada0p1 /boot/efi
cp /boot/loader.efi /boot/efi/EFI/BOOT/Bootx64.efi
cp /boot/loader.efi /boot/efi/EFI/FreeBSD/loader.efi (please see the Update below)
umount /boot/efi

Now it’s time to go with “ada1” disk:

[root@bsdb0x:~] mount_msdosfs /dev/ada1p1 /boot/efi
cp /boot/loader.efi /boot/efi/EFI/BOOT/Bootx64.efi
cp /boot/loader.efi /boot/efi/EFI/FreeBSD/loader.efi (please see the Update below)

I think that’s all. In case of anything else, keep the comments up please!

Update: Certain setups MAY have (or not at all) created a “EFI/FreeBSD/loader.efi” file. So IF you have it there, copy your loader.efi file to that path too, just like its shown above, replacing the current one, however, if you don’t have it at all, you’re good to go without.

Özgür Konstantin

https://x.com/ozgurkazancci

---