The other day, I was chilling on my computer, reading the Arch boot process on the Arch wiki, like one does. I’m familiar with the high-level boot sequence but the exact details were blurry in my head so I was due for a refresher. Maybe around half-way through the article, I remembered my recent troubles with EFI variables1 and wondered if my current bootloader was somehow faulty after all. Would simply moving to a different bootloader resolve the issue2? I heard positive things about Systemd-boot3 so I figured it’s a good opportunity to give it a try. This has been on my mind for a while now, since I’m constantly striving for a lean Linux installation. With that decision made, I set out to go over the Systemd-boot Arch wiki article right after I’m done with the boot process one.

Info

It’s generally recommended to use Systemd-boot if you’re using Systemd on your system and don’t need advanced boot loading customization.

Configuring Systemd-boot

Given the sensitive nature of the boot process, I took some time to prepare a bootable USB drive with the Arch Linux ISO4, before changing anything. I picked Ventoy5 for the job. It’s a nice piece of software that I’ve been wanting to try out anyway.

On the surface, setting up Systemd-boot is fairly simple. All you have to do is run the bootctl install command and that should take care of everything. Unfortunately, that didn’t go so smoothly for me. The command resulted in an EFI vars input/output error - an error I’m starting to get annoyed by familiar with at this point. I ensured that my system was booted through UEFI6, and that EFI vars were all loaded as they should. I went through bootctl’s man page and ran the command a bunch of time with different flags, but it kept falling flat. It copied some files into the boot location but always ended in that EFI error. Despite my enthusiasm to get Systemd-boot working, I had to shift gears and focus on my EFI situation before I can make any meaningful progress.

Stubborn EFI

I started by resetting the BIOS to its default configuration. That only made matters worse because it enabled Secure Boot which blocked the bootloader, so I had to turn it off7. After that, I cleared the NVRAM8 state that holds EFI vars. It didn’t seem to make a difference, except that it nuked the GRUB loader. Now, I’m getting a Systemd-boot loader instead with a single entry to reboot into BIOS. I’m not sure if this was the result of wiping the NVRAM or the bootctl install command. Perhaps the command did actually work halfway despite the EFI input/output error? Either way, now that the bootloader is corrupted, there’s no giving up. I must sort this out if I want to get my system back up and running.

Next, I flushed the BIOS with the latest firmware. This part took way more time than I’m willing to admit. I didn’t have a spare USB stick on hand to use for the firmware update. So I needed to find a way to use my Ventoy USB drive without having to remove Ventoy, because I might need it to boot into a live ISO for debugging. By default, Ventoy creates 2 partitions on the USB drive: One bootable partition for Ventoy itself, and another exFAT9 partition to host ISO files. I figured I could just drop the firmware file in the exFAT partition and load it from the BIOS. Not so fast there, Stacy! The BIOS native file browser couldn’t find the Ventoy partition for some reason. I kept looking and browsing other disks from the BIOS in search for the firmware file, but to no avail.

Well, it turns out my BIOS doesn’t support exFAT partitions. Or so it appeared - all I know for sure it that I wasn’t able to get to the Ventoy partition from the BIOS. Next step was to format that partition using a more widely supported filesystem, like FAT32. Luckily, that did the trick. I was able to read the now-FAT32-formatted Ventoy partition, find the firmware file, and flush it to BIOS. Once this side quest completed, I could go back to the main task: figuring out the EFI vars IO error and installing Systemd-boot.

Note

Keep in mind that I’m doing all this without my usual tools. My main computer is the very machine I’m troubleshooting, since my bootloader is gone. So I’m making do with whatever old, half-working, potato machine I can get my hands on, hoping it, too, doesn’t break on me. The seemingly simple maneuver of downloading the latest firmware from the motherboard’s vendor website, throwing it in a flash drive, and loading it in the BIOS for a firmware update ended up being quite challenging.

Now that the BIOS is running a shiny new firmware version, I can finally get down to business. I grabbed a copy of the Arch Linux ISO and placed it in the Ventoy partition. Fortunately, formatting the Ventoy USB drive only formatted the partition where ISO files are stored and didn’t touch the Ventoy bootable partition, so Ventoy was still intact. On top of that, Ventoy supports FAT32 partitions which means I didn’t have to reformat that ISO storage partition. I’m telling you, Ventoy is piece of art!

I’m now at a point where I can simply boot into the Arch Linux ISO, chroot into my system, and install Systemd-boot. If you’re thinking “yeah, right. Easier said than done.” you’d be totally right. But this article is getting long already, so I’m going to have to cut it here. Stay tuned for what happens next.

Wrap Up

You saw how I managed to totally obliterate my boot loader thereby rendering the whole system unbootable. I’ve gone through the steps I took to troubleshoot the problem, and touched on some findings like the lack of exFAT support in my BIOS for firmware updates, and the discovery of the amazing bootable USB drive tool called Ventoy. How about you? When was the last time you broke your Linux system and what was the issue? Also, if you have any ideas on what could be wrong with my EFI vars, I’m all ears! Feel free to reach out on Mastodon or via email. Until next time, take care.

  1. EFI variables are key/value pairs used in the UEFI system to store non-volatile data, such as boot configuration settings. ↩︎

  2. The problem is an input/output error I get whenever I try to change the boot order. More on this in my Dual Boot Struggles article. ↩︎

  3. Systemd-boot is a simple boot manager that comes pre-installed with Systemd. ↩︎

  4. An ISO file, or live image, is a bootable version of an operating system that can run directly from a removable medium like a USB stick without needing to be installed on a hard drive. ↩︎

  5. Ventoy is an open-source tool that allows you to create bootable USB drives for various operating systems using ISO files without needing to format the drive each time. ↩︎

  6. Unified Extensible Firmware Interface. ↩︎

  7. It is possible to boot Arch Linux with Secure Boot enabled but I’ll park this for another day. ↩︎

  8. Non-Volatile Random Access memory. It’s a type of memory that retains data even when the power is turned off. ↩︎

  9. exFAT is a file system optimized for flash memory such as USB flash drives and SD cards. ↩︎