Folder structure and rootfs

Folder structure is an important part of Unix and Unix-like operating systems (like Linux). The folder structure starts at a single root / and when you check the content of that root (ls /), you may realise that different distributions often have similar or overlapping folders. The folder structure of the root is actually defined by the Filesystem Hierarchy Standard. Basically some folders are always expected by the Linux kernel to be present.

In a Linux distribution, we sometimes talk about a "rootfs", short for "root file system". This is basically the folder and file structure starting at /. It's possible to mount images, folders, and storage devices to specific places in the folder hierarchy. Generally the rootfs is the structure starting at / without these extra mount points.

A nice way to start playing and understanding this rootfs thing more, is the so-called chroot command (short for "change root", so the "ch" can be pronounced as the "ch" in "change"). With it you can basically say "pretend this folder is the root and then run this comand". Let's try it!

Chroot jail

For the following I installed Alpine 3.16 on some old hardware. You should be able to do similar things with other Linux distros as well. Note that you need elevated privileges, so make sure you run the following commands as the root user (not to be confused with the root folder, they are two different things).

Let's say we are in our home folder ~. Let's make a folder to play in

mkdir my_chroot

Now let's try to start a shell who believes this folder is actually the root.

chroot my_chroot sh

What we see is that we get an error chroot: can't execute 'sh': No such file or directory, so what happened?

We started a chroot in the folder my_chroot and asked to execute the command sh. When you ask to execute a command without specifying it's location, your shell will try to find the command in the so called PATH. The PATH is nothing more than a variable holding a colon separated list of folders where we can store the executables we want to be able to call without needing to specify the specific location. To see the list, you can run echo $PATH. For me it looks like /usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin

So what happened now is that we try to locate sh in these folders, but these folders don't exist. After all, when we try to locate /bin inside the chroot, what we really look at is the ~/my_chroot/bin folder when looked at from the host system. So even if the command exists on our host, we can't access it from inside the chroot. To make sure we can run a command, we need to make it accesable inside the chroot folder structure. To do that we can mount the folder containing the command, or we can place the command there manually. Let's do the seconds one!

First we'll check where the correct sh binary is. Then we can copy it over to the location we need it. There's a good chance this binary relies on some libraries to be present, so we check for that with the ldd command and copy those over as well.

which sh
# I see the binary is `/bin/sh`
mkdir my_chroot/bin
cp /bin/sh my_chroot/bin/sh

ldd /bin/sh
# I see there's one lib `/lib/ld-musl-x86_64.so.1`
mkdir my_chroot/lib
cp /lib/ld-musl-x86_64.so.1 my_chroot/lib/ld-musl-x86_64.so.1

Now we can try to run this shell in the root container again chroot my_chroot sh, and if it works, you should now have a shell! If you still get the chroot: failed to run command ‘sh’: No such file or directory error, you should double check that both the binary and the libs are indeed there and with the right permissions. The difference between the shell you had and the one you have now may not be immediately visible, but you should be at root now and a lot of commands wont work who otherwise did. Some may work because they are part of the shell, but others may not be and you need the binaries for them. In my case ls didn't work, pwd did. To exit the shell, you can do exit. If you want to add more commands, you can do that in a similar way. Btw, if you ever hear talk about a "chroot jail", this is basically what they are talking about; A piece of software that's run with chroot. The chroot environment (i.e. the environment that the software runs in after starting it with the chroot command) is the "chroot jail".

It's also possible that some commands need certain configuration files. When I copy apk in the same way and then do apk --help from inside the chroot, it works. But when I do apk update it gives errors about not being able to lock and open a database. After a lot of searching and trial and error I eventually did

# First move apk and the libraries
which apk
mkdir my_chroot/sbin
cp /sbin/apk my_chroot/sbin/apk

ldd /sbin/apk my_chroot/lib/
cp /lib/libcrypto.so.1.1 my_chroot/lib/libcrypto.so.1.1
cp /lib/libz.so.1 my_chroot/lib/libz.so.1
cp /lib/libapk.so.3.12.0 my_chroot/lib/libapk.so.3.12.0
cp /lib/ld-musl-x86_64.so.1 my_chroot/lib/ld-musl-x86_64.so.1
cp /lib/libssl.so.1.1  my_chroot/lib/libssl.so.1.1

# Now the following already works
chroot my_chroot apk --help

# But we need a bunch of other stuff
# if we want to `chroot my_chroot apk update`
mkdir my_chroot/lib/apk/
cp -r /lib/apk/db my_chroot/lib/apk/db
mkdir -p my_chroot/etc/apk
cp /etc/apk/repositories my_chroot/etc/apk/repositories
touch my_chroot/etc/apk/world
cp -r /lib/apk my_chroot/lib/apk
mkdir my_chroot/var
cp /etc/resolv.conf my_chroot/etc/resolv.conf
cp /bin/busybox my_chroot/bin/busybox

# Now update almost works
# It still gives an error about an untrusted key
# We can probably fix it by copying the correct files
# Or we can work around it by using the ` --allow-untrusted` parameter
chroot my_chroot apk update --allow-untrusted

Now we can actually install packages inside our chroot environment. That's pretty sweet huh!

# With this we can install all basic tools you get in Alpine
chroot my_chroot sh
apk add alpine-base --allow-untrusted

Everything is a file

When we now check the content of the chroot folder ls my_chroot, you may notice a new folder proc. This was added when we did apk update. You may have heard that in Unix "everything is a file". What people mean by this is that you can access everything, even processes and devices, from the filesystem. The dev, proc and sys folders are originally empty, but when you start up your operating system, the Linux kernel will populate them. The dev folder will show devices while proc will show processes. Note that while we see a proc folder here, it's not actually mounted from the kernel. I assume apk needed it for some reason and then created it because it wasn't there.

When you need these folders in the chroot, one way to do it, is to mount them. When looking online, it seems that the following is often used for this.

mount -t proc none /mnt/proc
mount --rbind /sys /mnt/sys
mount --rbind /dev /mnt/dev

Limitations

With chroot we can limit some access a program has to the system. Not only can't it access certain parts of the file system, it doesn't even know about those parts. And that's pretty neat I think. But there's more than just the file system. It's possible that an attacker may still get access to the host system through other means. First of all there's the fact that we can mount or link things into the chroot, meaning that those parts of our system are now accessable. If the sh binary isn't a separate binary, but linked to the sh of the host system, then making changes to the binary means we make the changes to the actual sh binary on our host system! But even besides that, the file system isn't everything. The proc folder already shows us that there's a thing like processes, and there's other things as well. If you want to go a step further than only chroot, there's containers. Containers use chroot, but also similar techniques to limit other parts of the system. You can check out LXC containers for example who can be easily managed using LXD. What you end up with is a system that uses the same kernel as the host, but can start it's own operating system from the rootfs in the containerised environment. One step further than this is virtual machines, where you even run your own separate boot process and kernel.

Round-up

Even with all the caveats, chroot already gives us a first insight of what's needed for some software to work and gives us a playground to get some first insights into a rootfs a bit. Try it!

Now that we've played with rootfs in chroot a bit, let's see if we can actually boot something. When I install Alpine 3.16, I see three partitions; A boot partition /dev/sda1, a swap partition /dev/sda2, and the rootfs /dev/sda3. When we start Alpine, the boot partition is mounted on /boot. Here we find a couple of files, including vmlinuz-lts. This file is our Linux kernel. Yup, if you never knew where that things was to be found, there it is! This can be a file or it can be a symbolic link to the actual kernel. On my Kubuntu for example, the kernel is called /boot/vmlinuz-5.15.0-52-generic, but linked to from /boot/vmlinuz. Note that on Kubuntu the file is vmlinuz while on Alpine it's vmlinuz-lts. It seems the 'vmlinuz' name grew historically. Originally, Unix kernels were typically called 'unix', later it was possible to add an option called 'Virtual Memory' and the name of kernels compiled with that option were prefixed with 'vm'. Later it got possible to compress the kernel file to save storage and the 'x' was changed to 'z' to show it's a compressed kernel. Linux distributions just took over the same conventions. Anyhow, what we want to do now, is play with the rootfs and boot something up!

For the following we need an extra partition. You can use an extra drive, or add an extra partition with filesystem on the current drive. I don't want to go into much detail on how you can do this stuff, you should find info online if needed. If you want to add a new formatted partition to the current drive, it's probably easiest to do it from a live distro with a graphical DE (e.g. Trisquel) and do it with a graphical tool like gparted. In my case I have /dev/sda3 which holds my Alpine installation and /dev/sda4 which is the new partition.

When the kernel boots, it does some things and eventually starts /sbin/init. This is very similar to what we did with chroot. The big difference is that there we said what to start, here Linux decides to start /sbin/init. I'm unsure if this is hard coded or can be changed with config files somewhere, but from what I see online, it at least seems to be the norm to start this.

What we'll try, is to use Alpine's boot partition and boot something simple that we create on our own rootfs.

Just to see things working, I first booted into a live iso and copied all files from sda3 to sda4.

mkdir /mnt/sda1 /mnt/sda3 /mnt/sda4
mount /dev/sda1 /mnt/sda1
mount /dev/sda3 /mnt/sda3
mount /dev/sda4 /mnt/sda4

cp -r /mnt/sda3 /mnt/sda4

In the boot folder (/mnt/sda1) I notice a file extlinux.conf. This is from the part of our boot system where I don't want to delve too much in yet, but one important line is APPEND root=UUID=91c0e3ea-fca1-4c7c-96ed-ebc551066204 modules=sd-mod,usb-storage,ext4 quiet rootfstype=ext4. This tells us what partition should be considered as the one with the rootfs. I take the guess that this will probably also work with the name instead of UUID, so I changed this to APPEND root=/dev/sda4 modules=sd-mod,usb-storage,ext4 quiet rootfstype=ext4. Just to be sure you can add a file or something to sda3 and sda4 just so you can be sure which of the two is used as rootfs, and then reboot the machine (without the live iso). The result was that we are now booted from /dev/sda4. That's pretty sweet! Not only did we boot from another partition, the partition was filled up by simply copying files over. That implies to me that we don't need certain bits to be set at a certain place on the drive or anything. Linux understands what's on this drive and can execute things.

Let's see if we can boot something up ourselves. First I change extlinux.conf back so it boots from /dev/sda3. First we make sure we can access the filesystem on our partitions, then we remove everything from sda3 and create the folders Linux needs. If something goes wrong, we can always boot again from a live iso, mount the drives where we think something went wrong, and fix things.

mkdir /mnt/sda3 /mnt/sda4
mount /dev/sda3 /mnt/sda3
mount /dev/sda4 /mnt/sda4
cd /mnt/sda3

rm -r ./
mkdir sys dev proc

If we reboot the machine now, we'll get a kernel panic. Hooray! That means the kernel is booting, it just can't continue which is normal since the partition it tries to boot from is empty, and therefor there's no /sbin/init to start. Let's try to start a simple shell on boot. This is very similar to how we set things up with chroot. To set things up, we'll boot from a live iso again.

Alpine uses something called Busybox. This is a single binary that contains a whole set of tools. For example, if you have Busybox installed, instead of doing ls, you can also do busybox ls. So what we'll do is, we'll copy busybox and the needed libraries to the rootfs we're creating and then call it from an init script.

mkdir /mnt/sda3 /mnt/sda4
mount /dev/sda3 /mnt/sda3
mount /dev/sda4 /mnt/sda4

# We copy the binaries and libraries just like we had to do when playing with chroot
mkdir /mnt/sda3/bin
cp /mnt/sda4/bin/busybox /mnt/sda3/bin/busybox
mkdir /mnt/sda3/lib
cp /mnt/sda4/lib/ld-musl-x86_64.so.1 /mnt/sda3/lib/ld-musl-x86_64.so.1

Then we'll create our init script and make it executable

cd /mnt/sda3
mkdir sbin
touch sbin/init
chmod +x sbin/init

echo '#!/bin/busybox sh' >> sbin/init
echo "echo 'Welcome to your very own Linux Distro!'" >> sbin/init
echo '/bin/busybox sh' >> sbin/init

If you made sure we're using /dev/sda3 as root partition again on boot, you can now reboot and should see something like

Welcome to your very own Linux Distro!
sh: can't access tty: job control turned off
/ #

Awesome! Since busybox has a whole bunch of tools, we can already use these. If ls doesn't work, try busybox ls. You can also try to add some folders or files (e.g. busybox touch catgirl_titties), but you'll quickly notice the filesystem is read-only. We can fix that with busybox mount -o remount,rw /. There's also the error "sh: can't access tty: job control turned off", but I conveniently decided to consider this one out of scope for this article 😏

When we want to exit our shell, we can do exit, which will now give us a kernel panic. If you don't want a kernel panic, you can call busybox poweroff -f, which will immediately poweroff the machine.

When you call a program, you can give it a bunch of parameters. In the case of busybox ls, we provide "ls" as a parameter. But one parameter that's always provided is how the command was called. If we call busybox, then busybox will know that this is how it was called. If we link a file called "ls" to busybox and then call that "ls" file, busybox will know that it was called with the command "ls". Busybox is programmed in such a way that it will use this information to conclude what command you wanted to run and then run that. If you want a list of the commands busybox has, you can run busybox --list. To see what location busybox expect these tools to be placed, you can do busybox --list-full.

Knowing these things, there's some optimalisations we can add.

First we'll make it easier to call the commands by making system links to busybox in the correct locations.

busybox mount -o remount,rw /

# Maybe there's a built in way to do this, idk, this works :)
for link in $(busybox --list-full)
do
folder=$(busybox echo "$link" | busybox sed -E 's@(.*/)[^/]*$@\1@')
busybox mkdir -p "$folder"
busybox ln -s /bin/busybox "$link"
done

Another thing we can do, is change the init file to have the following content. That way the file system will be mounted as read-write, and when we do exit, the machine will power off instead of throwing a kernel panic.

\#!/bin/busybox sh
/bin/busybox mount -o remount,rw /
echo 'Welcome to your very own Linux Distro!'
/bin/busybox sh
/bin/busybox poweroff -f

Now we can reboot and we have our own very very minimal distro 😁

Distro's typically do more than just start a shell. They also have an init system to start up several services for example. Afaict busybox can do all that, but it requires more looking into. I'm not gonna do that now. For now we already have a neat thing to brag to our friends about, so let's enjoy that first ;)