How to install Fedora Minimal with Snapper

How to install Fedora Minimal with Snapper

2022-11-16

The goal: Get a “riceable” Linux setup that can be rolled back in a flexible manner, by running a clever OpenSUSE tool on a non-OpenSUSE distro. Make a system that is difficult to break in a way that will waste a whole day getting it working again. All the fun of Linux desktop ricing, but none of the risk.

Skip introduction

Snapper? But what about Silverblue? #

Silverblue is a fantastic project that intends to introduce the concept of immutable operating systems to the Linux desktop and server. It does this by packaging an entire set of operating system files together, updating it all at once when you reboot the system. Currently, there are four OSTree-based projects, one that includes system files for GNOME desktop (Silverblue), another that provides system files for a KDE desktop (Kinoite), as well as an IoT platform (Fedora IoT) and a headless container platform (CoreOS).

This system is broadly similar to how Windows, Android, macOS, and iOS work.

Analogy: Windows Update

Windows manages the files under C:\Windows\System32 and prevents the user from modifying them with loose technical restrictions imposed by Windows Explorer. Applications like notepad.exe and all the various dynamically-linked libraries Windows comes with are not meant to be edited. This provides a stable and predictable system, carefully managed by Windows Update.

On Windows, anything outside of this can be installed to various user- and administrator-controlled folders like C:\Program Files or %appdata%. The analogue to this in Silverblue are the various mutable directories that are attached to /var, such as /home/, /home/foo/.local/bin/, /var/lib/flatpak/, /etc/

It should be noted that Windows Update is not a true immutable system, and has numerous disadvantages compared to a true immutable system, which are discussed below. But like Windows Update, Silverblue provides the benefit of being able to rollback to the last working system files if something breaks.

rpm-ostree? #

Continuing with the Windows analogy, sometimes a program wants to make deeper changes to the system. It can screw with .dll files and add .sys driver files to the Windows folder, which is a clunky but ultimately effective way to modify the Windows system. But it’s not great, on Windows you don’t have a vetted distribution that is designed to work with the base files, rather you have some possibly-broken drivers distributed by a vendor that doesn’t care if they break your system.

rpm-ostree is the analogue to this in Fedora Silverblue. It uses some special Linux kernel filesystem features to overlay extra files on top of the immutable base system. It can even be used to install out-of-tree drivers such as the NVidia drivers distributed from RPMFusion.

Generally rpm-ostree works pretty well. The reason rpm-ostree works so well is that the original system snapshot is constructed from an amalgamation of rpms from the Fedora distribution, and rpm-ostree is designed to be aware of how the original snapshot was constructed. It’s quite an elegant solution!

Flatpak? #

Flatpak is an incredible innovation in the Linux desktop space — it is an alternative way to install Linux desktop software that serves to alleviate deficiencies in the usual Linux distro model. In the Windows analogy, it is C:\Program Files, but much more sophisticated.

There are numerous benefits of Flatpak:

  • It provides numerous security guarantees and an Android-like permission system. Flatpak (or another isolation technology) is absolutely essential for desktop Linux to be secure.
  • It provides developer-to-user distribution. This is not always what we want — Linux distros maintain an adversarial relationship with upstream developers, and this is by design — it is a check on developers. Distros can patch stuff and make packages more compatible and cohesive with each other. But most of the time, the end user just wants fast updates, which Flatpak can do because software is published directly by the author.
  • It provides consistent runtime libraries, meaning less DLL hell.
  • It works well with immutable systems like Silverblue and Valve SteamOS, because it doesn’t modify the base system like rpm-ostree does, and can carry libraries that are separate from the base system, while still having great integration with the platform.

So given the first three benefits, we should probably be using Flatpak to install certain applications, regardless of whether we’re using Silverblue or normal Fedora. Later in the tutorial we will be installing Firefox using Flatpak.

Now that we’re primed on the solution that OSTree, Silverblue, and Flatpak in combination presents, we’ll explore the Linux Ricing subculture and their requirements, which is the topic of this tutorial.

Linux Ricers, and ridding your system of bloat #

There is a subculture of the Linux community that wishes to completely customize their Linux system from the ground up. They are terrified of bloat & extra software that is not needed. So they have specific requirements and either run into issues with Silverblue, or more likely immutable Linux systems just weren’t marketed to them in the first place.

Silverblue Problem #1: Limited to GNOME and KDE #

The Silverblue OSTree distributions are only available in two flavors, GNOME and KDE. Don’t want GNOME’s gedit text editor? You could use rpm-ostree to remove it — this will mask the files from appearing, but they are still stored. But if you wanted to remove hundreds of GNOME packages, rpm-ostree will have to maintain a list of thousands of files to mask.

Linux ricers are often not using GNOME or KDE, they usually want to start from scratch using basic Xorg or Wayland wlroots-based software. They want to be able to directly manipulate a .xinitrc script. After freeing themselves of bloat, they want to install an assortment of “unix philosophy” system-level software that doesn’t easily fit into the Flatpak way of doing things.

This is why they are often using Archlinux, Gentoo, NixOS, Debian, and Voidlinux. These OSes plunge you into a command line so you can edit some text files to start a computing setup that you will hopefully be using for years. These are great operating systems, but they are not Fedora — they aren’t keeping up with the innovation in the Linux desktop space (except for NixOS perhaps), and at the same time they aren’t always so stable.

Silverblue Problem #2: Rollbacks & updates only somewhat help you fix /etc/ — its handling of /etc is not time machine snapshots #

Each time you switch to a different snapshot, OSTree will do a three-way diff based merge on /etc/ to update its contents. If this three-way diff fails, you could get an /etc/ that is broken. A Linux ricer may wish to have their system packages and their /etc/ folder always in sync with the backup snapshots.

No, Silverblue won't give you this.

Silverblue Problem #3: Reboots required #

It’s not rational of course, but Linux ricers care about uptime. They know when a reboot is actually required vs. when the software is just being cautious. Fedora Silverblue requires reboots to make changes to rpm-ostree. Thankfully it does not have a Windows Vista style loading screen (it’s very quick to switch out the old snapshot for the new one) but still, it would be nice to only need to do this for rollbacks and new kernels.

Snapper #

Snapper is an OpenSUSE technology. OpenSUSE uses the Btrfs filesystem by default, which has copy-on-write and snapshots. Filesystem-level snapshots give you the capability to take a set of files and create a copy of it that doesn’t actually take up any extra space, until one version of the files is modified and diverges from the original.

Snapper puts the entire system into a Btrfs subvolume, and has hooks in tools like the package manager and YaST to automatically create a Btrfs snapshot every time you install a package or update the system.

For Linux ricers, their entire root filesystem is not just a set of changes from a base image (ala Silverblue), but rather is a constantly evolving system as they change customizations over the years. So Snapper is the perfect solution to have whole-system rollback capabilities on a heavily customized Linux desktop.

Tutorial: Set up a Fedora Minimal installation with Snapper #

This is a tutorial for those who wish to start with a minimal Fedora installation and set up minimalistic X11 window managers, all while having Snapper for whole-system rollback capabilities.

Download the “Everything” installation medium from the alternate downloads page for Fedora Linux. If you are not looking to create a minimal installation, you can just get Workstation or any of the other spins.

Boot the medium, select a language. Click on Software Selection.

If you are not interested in setting up a minimal installation, you can skip this step.

Select Minimal Install and add Common NetworkManager Submodules (will help you get Wifi working with nmcli or nmtui later on) and Standard.

Click Installation Destination.

Click Advanced Custom (Blivet-GUI) and press Done.

Verify that you are installing to the correct disk and click the plus icon.

Add an EFI system partition, or a boot partition. If using UEFI, use the following configuration:

Size: 512 MiB
Filesystem: EFI System Partition
Mountpoint: /boot/efi

If using legacy boot, consider upping the size to 1 to 2 GiB, because Snapper will put multiple different kernel images on this partition. Note that for legacy boot you will also need a 1MiB BIOS boot partition as the first partition on the drive.

[If using legacy boot]
First partition:
    Size: 1 MiB 
    Filesystem: BIOS Boot

Second partition:
    Size: 1536 MiB 
    Filesystem: ext4
    Mountpoint: /boot

Add another partition.

Using the following configuration, fill the rest of the available space:

Device Type: Btrfs Volume
Mountpoint: /

On the left column, click on the Btrfs subvolume you just created.

Create a subvolume to store Snapper snapshots. Name it snapper and let its mountpoint be /.snapshots

After this, you should create four other subvolumes, so there should be:

(This is the one you just created)
Name: snapshots
Mountpoint: /.snapshots
Name: home
Mountpoint: /home
Name: var-cache
Mountpoint: /var/cache
Name: var-tmp
Mountpoint: /var/tmp
Name: var-log
Mountpoint: /var/log
Name: plocate-data
Mountpoint: /var/lib/plocate

Optional: add these subvolumes for developer tools that store large amounts of frequently-updating data:

If you want to use libvirt and create virtual machine disk images

Name: libvirt-data
Mountpoint: /var/lib/libvirt

If you use the MySQL database software

Name: mysql-data
Mountpoint: /var/lib/mysql

If you use the MariaDB database sotware

Name: mariadb-data
Mountpoint: /var/lib/mariadb

If you use the PostgreSQL database software

Name: pgsql-data
Mountpoint: /var/lib/pgsql

If you use Redis key-value store software

Name: redis-data
Mountpoint: /var/lib/redis

If you use Docker containerization software

Name: docker-data
Mountpoint: /var/lib/docker

If you use Docker containerization software, or anything else that uses containerd

Name: containerd-data
Mountpoint: /var/lib/containerd

If you use lxd containerization software

Name: lxd-data
Mountpoint: /var/lib/lxd

If you use Toolbox or Podman

Name: containers-data
Mountpoint: /var/lib/containers

If you use the /srv directory

Name: srv
Mountpoint: /srv

You can always create these subvolumes later.

Alternatively, you might consider putting the entire /var folder on a separate subvolume, but by doing this you won’t include the Flatpak folder /var/lib/flatpak in the Snapper rollbacks.

The reason for all these subvolumes is to ensure Snapper is ignoring temporary files and other stuff that is changing constantly. We don’t need these to be considered part of the system state that Snapper needs to preserve.

Here is what the final result should look like:

Click Done after you have verified everything is correct. Click Accept Changes to continue.

Create any accounts you need, and configure anything else you might need to change in the installation.

Begin the installation.

You have now installed a minimal Fedora setup!

SSH daemon [OPTIONAL] #

When starting out on a newly set up minimal installation, you should probably ssh in from another PC rather than using the clunky Linux console. You can disable this later when you are done.

Assign a hostname:

# sudo hostnamectl set-hostname foobar

Start sshd:

# sudo systemctl start sshd
# sudo systemctl enable sshd

To show LAN IP:

# ip addr

Shell ricing [OPTIONAL] #

Before we start, lets install some dependencies to get a nice comfortable zsh-based shell:

# sudo dnf install zsh fzf PackageKit-command-not-found util-linux-user sqlite git

Install zplug, an easy scripts manager for zsh:

# curl -sL --proto-redir -all,https https://raw.githubusercontent.com/zplug/installer/master/installer.zsh | zsh

Run chsh and set the shell to /usr/bin/zsh. Log out and log back in.

Save the following to ~/.zshrc:

[[ -f ~/.profile ]] && source ~/.profile
source ~/.zplug/init.zsh

zplug themes/dpoggi, from:oh-my-zsh

if ! zplug check --verbose; then
    printf "Install? [y/N]: "
    if read -q; then
        echo; zplug install
    fi
fi

zplug load

source /usr/share/fzf/shell/key-bindings.zsh

HISTFILE=~/.zsh_history
HISTSIZE=100000000
SAVEHIST=100000000

bindkey -e
bindkey "^[[1;5C" forward-word
bindkey "^[[1;5D" backward-word
bindkey "^H" backward-kill-word
bindkey "^[[3;5~" kill-word
setopt BANG_HIST
setopt EXTENDED_HISTORY
setopt INC_APPEND_HISTORY
setopt SHARE_HISTORY
setopt HIST_EXPIRE_DUPS_FIRST
setopt HIST_IGNORE_DUPS
setopt HIST_IGNORE_ALL_DUPS
setopt HIST_FIND_NO_DUPS
setopt HIST_IGNORE_SPACE
setopt HIST_SAVE_NO_DUPS
setopt HIST_REDUCE_BLANKS
setopt HIST_VERIFY
setopt HIST_BEEP

autoload -Uz compinit
compinit
autoload -U bashcompinit
bashcompinit

We have a nice shell now, much more pleasant than the bash defaults. Feel free to add on to the zplug config.

Installing Snapper and setting up Grub integration #

We can inspect how the subvolumes are set up:

# sudo btrfs subvolume list /
# sudo btrfs filesystem show /
# sudo lsblk -p
# cat /etc/fstab

/etc/fstab contents example:

UUID=FD87-791A          /boot/efi               vfat    umask=0077,shortname=winnt 0 2
UUID=9f2ebb76-0cc0-4473-9c82-2856500e1d22 /                       btrfs   defaults        0 0
UUID=9f2ebb76-0cc0-4473-9c82-2856500e1d22 /.snapshots             btrfs   subvol=snapshots,compress=zstd:1 0 0
UUID=9f2ebb76-0cc0-4473-9c82-2856500e1d22 /home                   btrfs   subvol=home,compress=zstd:1 0 0
UUID=9f2ebb76-0cc0-4473-9c82-2856500e1d22 /srv                    btrfs   subvol=srv,compress=zstd:1 0 0
UUID=9f2ebb76-0cc0-4473-9c82-2856500e1d22 /var/cache              btrfs   subvol=var-cache,compress=zstd:1 0 0
UUID=9f2ebb76-0cc0-4473-9c82-2856500e1d22 /var/lib/containerd     btrfs   subvol=containerd-data,compress=zstd:1 0 0
UUID=9f2ebb76-0cc0-4473-9c82-2856500e1d22 /var/lib/containers     btrfs   subvol=containers-data,compress=zstd:1 0 0
UUID=9f2ebb76-0cc0-4473-9c82-2856500e1d22 /var/lib/docker         btrfs   subvol=docker-data,compress=zstd:1 0 0
UUID=9f2ebb76-0cc0-4473-9c82-2856500e1d22 /var/lib/libvirt        btrfs   subvol=libvirt-data,compress=zstd:1 0 0
UUID=9f2ebb76-0cc0-4473-9c82-2856500e1d22 /var/lib/lxd            btrfs   subvol=lxd-data,compress=zstd:1 0 0
UUID=9f2ebb76-0cc0-4473-9c82-2856500e1d22 /var/lib/mariadb        btrfs   subvol=mariadb-data,compress=zstd:1 0 0
UUID=9f2ebb76-0cc0-4473-9c82-2856500e1d22 /var/lib/mysql          btrfs   subvol=mysql-data,compress=zstd:1 0 0
UUID=9f2ebb76-0cc0-4473-9c82-2856500e1d22 /var/lib/pgsql          btrfs   subvol=pgsql-data,compress=zstd:1 0 0
UUID=9f2ebb76-0cc0-4473-9c82-2856500e1d22 /var/lib/plocate        btrfs   subvol=plocate-data,compress=zstd:1 0 0
UUID=9f2ebb76-0cc0-4473-9c82-2856500e1d22 /var/log                btrfs   subvol=var-log,compress=zstd:1 0 0
UUID=9f2ebb76-0cc0-4473-9c82-2856500e1d22 /var/tmp                btrfs   subvol=var-tmp,compress=zstd:1 0 0

If your setup looks roughly like this, you can continue.

First, if you created these, you may wish to configure some of the heaviest /var/ subvolumes to disable copy-on-write. Some database software doesn’t play well with it and causes performance issues.

# sudo chattr -R +C /var/lib/libvirt
# sudo chattr -R +C /var/lib/mysql
# sudo chattr -R +C /var/lib/mariadb
# sudo chattr -R +C /var/lib/pgsql

Allow for snapshot booting by editing /etc/default/grub. Add the following line:

[...]
SUSE_BTRFS_SNAPSHOT_BOOTING="true"

Update grub configuration to reflect this change:

# sudo grub2-mkconfig -o /boot/grub2/grub.cfg

Reboot:

# sudo reboot

Install Snapper, and the dnf hooks for automatically creating before/after snapshots every time packages are changed:

# sudo dnf install snapper python3-dnf-plugin-snapper inotify-tools

Delete and recreate the /.snapper subvolume to get snapper to take over it.

# sudo umount /.snapshots
# sudo rmdir /.snapshots
# sudo snapper -c root create-config /
# sudo btrfs subvolume delete /.snapshots
# sudo mkdir /.snapshots

Reload fstab:

# sudo systemctl daemon-reload
# sudo mount -a

You can verify that the subvolume is mounted correctly:

# lsblk -p

Configure Snapper to allow your account to administrate it:

# sudo snapper -c root set-config ALLOW_USERS=$USER SYNC_ACL=yes
# sudo chown -R :$USER /.snapshots

Unhide the grub menu:

# sudo grub2-editenv - unset menu_auto_hide

Install grub-btrfs, a tool to automatically generate grub listings for Snapper snapshots. This is a script that works transparently to update grub every time /.snapshots is modified, similar to how OpenSUSE works.

# sudo dnf install make
# git clone https://github.com/Antynea/grub-btrfs.git /tmp/grub-btrfs
# cd /tmp/grub-btrfs
# sudo make install

Edit the config file for grub-btrfs at /etc/default/grub-btrfs/config.

Edit #1: Make sure GRUB_BTRFS_MKCONFIG is uncommented and set to /usr/sbin/grub2-mkconfig

GRUB_BTRFS_MKCONFIG="/usr/sbin/grub2-mkconfig"

Edit #2: Make sure GRUB_BTRFS_SCRIPT_CHECK is uncommented and set to grub2-script-check

GRUB_BTRFS_SCRIPT_CHECK="grub2-script-check"

Edit #3: Make sure GRUB_BTRFS_GRUB_DIRNAME is uncommented and set to /boot/grub2. Do not follow the Fedora advice in the commented out section, it is meant for Fedora 33 and earlier.

GRUB_BTRFS_GRUB_DIRNAME="/boot/grub2"

In patch format:

--- /tmp/a
+++ /tmp/b
@@ -90,7 +90,7 @@
 # Might be grub2 on some systems.
 # For example, on Fedora with EFI : "/boot/efi/EFI/fedora"
 # Default: "/boot/grub"
-#GRUB_BTRFS_GRUB_DIRNAME="/boot/grub2"
+GRUB_BTRFS_GRUB_DIRNAME="/boot/grub2"
 
 # Location of kernels/initramfs/microcode.
 # Use by "grub-btrfs" to detect the boot partition and the location of kernels/initrafms/microcodes.
@@ -104,13 +104,13 @@
 # For example, on Fedora : "/sbin/grub2-mkconfig"
 # You can use only name or full path.
 # Default: grub-mkconfig
-#GRUB_BTRFS_MKCONFIG=/usr/bin/grub2-mkconfig
+GRUB_BTRFS_MKCONFIG="/usr/sbin/grub2-mkconfig"
 
 # Name of grub-script-check command, use by "grub-btrfs"
 # Might be 'grub2-script-check' on some systems (Fedora ...)
 # For example, on Fedora : "grub2-script-check"
 # Default: grub-script-check
-#GRUB_BTRFS_SCRIPT_CHECK=grub2-script-check
+GRUB_BTRFS_SCRIPT_CHECK="grub2-script-check"
 
 # Path of grub-mkconfig_lib file, use by "grub-btrfs"
 # Might be '/usr/share/grub2/grub-mkconfig_lib' on some systems (Opensuse ...)

Once you are done, regenerate the grub config and enable the systemd unit for watching /.snapshots in real time:

# sudo grub2-mkconfig -o /boot/grub2/grub.cfg
# sudo systemctl enable --now grub-btrfsd.service

Using Snapper #

Success, you have now installed Snapper! You can use snapper ls to see the snapshots it has created:

A snapshot will be generated every time you run a dnf command that changes the system. You can access these snapshots from the grub menu.

By selecting this, you will boot into a read-only system where you can verify your system is in a working state before switching to it. As an example, we rollback to a snapshot before htop was installed:

htop is gone:

To rollback to it, you can run this command, where <NUM> is the id number of the snapshot you wish to return to. You can use --ambit classic to switch to a read-write copy of the chosen snapshot. --ambit transactional is equivalent to selecting the read-only grub entry.

# snapper --ambit classic rollback <NUM>

Refer to the official Snapper Tutorial for advanced usage.

Ricing an X11 desktop [OPTIONAL] #

Now that we have a minimal Fedora setup, we can start installing things. In this example we’ll install SDDM with Openbox and Polybar. First we’ll be using some package groups to make the basics easy:

# sudo dnf install @base-x "@Hardware Support" "@Fonts" "@Input Methods" "@Multimedia" "@Printing Support"

Then we install the meat of what we want:

# sudo dnf install sddm polybar openbox lxappearance lxappearance-obconf obconf picom

We’ll grab some themes:

# sudo dnf install sddm-themes arc-theme arc-kde lxappearance feh

Install some applications:

# sudo flatpak remote-add --if-not-exists flathub https://flathub.org/repo/flathub.flatpakrepo
# sudo flatpak install org.mozilla.firefox
# sudo dnf install kitty neofetch

Add autostart script for Openbox at ~/.config/openbox/autostart

#!/bin/bash
picom &
polybar &
feh --bg-scale /usr/share/backgrounds/f37/default/f37-02-night.png &

Start sddm on boot:

sudo systemctl enable sddm
sudo systemctl start sddm
sudo systemctl set-default graphical.target

© lordpipe

Licensed CC BY — copy this document for your own use.