Yesterday I gave a presentation about DragonFlyBSD’s HAMMER filesystem at the end of the system architecture lecture here at Karlsruhe Institute of Technology (also well known as University of Karlsruhe). It was quite interesting and exciting to stand in our hugest lecture hall and talk to maybe 150 students about an highly “innovative” topic, at least that’s how our Professor announced my presentation.

Automatic PFS creation for DragonFly’s HAMMER file system has now been committed (read here).

This article briefly describes how to install DragonFlyBSD on a Hammer filesystem, i.e. using Hammer for the root filesystem. Note that I make the assumption that the DragonFly installation will use the entire disk (in my case this is disk ad4). Also make sure that the DragonFly version you are using includes my RootOnHammer patches which should be the case for any release after 2.0 and every snapshot ISO image from now on. Btw, the whole approach is similar to ZFSOnRoot, except that I will not use the installer.

Start the live-cd installer and enter the root prompt.

Partition ad4 creating one slice covering the entire disk.

fdisk -B -I ad4

Generate a disklabel in your favourite editor that looks like the following one and save it under /tmp/disklabel.ad4s1.

#          size     offset    fstype
  a:     262144          0    4.2BSD	#     128.000MB
  b:    6291456     262144      swap	#    3072.000MB
  c:  234441585          0    unused	#  114473.430MB
  d:  227887984    6553600    4.2BSD	#  111273.430MB

I’m using 128 MB for the boot partition (a), which should be enough to hold several kernels and it’s modules. Note that I am using 3 GB for the swap partition (b). The remaining space is left for the Hammer partition (d).

Then store the disklabel onto your disk:

disklabel -R -r ad4s1 /tmp/disklabel.ad4s1

Create the boot filesystem (UFS) and the Hammer filesystem:

newfs /dev/ad4s1a
newfs_hammer -L HammerGeil /dev/ad4s1d 

At first we set up the boot filesystem by mounting it and copying the /boot directory, the kernel and it’s modules onto it:

mount /dev/ad4s1a /mnt
cpdup -vvv /boot /mnt/boot
cpdup -vvv /kernel /mnt/kernel
cpdup -vvv /modules /mnt/modules

Then we adjust /mnt/boot/loader.conf to use ad4s1d (our Hammer partition) as root filesystem:

touch /mnt/boot/loader.conf
echo 'vfs.root.mountfrom="hammer:ad4s1d"' >> /mnt/boot/loader.conf

That’s all we need to do for the boot partition, so we unmount it.

umount /mnt

Next we continue with the Hammer partition:

mount_hammer /dev/ad4s1d /mnt

onto which we will install all remaining parts of DragonFlyBSD. It’s as easy as copying the files from the live-cd to the Hammer partition:

cpdup -vvv /bin /mnt/bin
cpdup -vvv /sbin /mnt/sbin
cpdup -vvv /root /mnt/root
cpdup -vvv /usr /mnt/usr
cpdup -vvv /var /mnt/var
cpdup -vvv /dev /mnt/dev
cpdup -vvv /etc.hdd /mnt/etc

cp /.cshrc /.profile /mnt
mkdir /mnt/mnt /mnt/proc /mnt/tmp

We further want later on to mount the boot partition into /bootdir, and use /boot as short-cut (symlink) to /bootdir/boot:

mkdir /mnt/bootdir
cd mnt
ln -s bootdir/boot boot
cd ..

We also want to edit /mnt/etc/fstab to contain the following:

# Device            Mountpoint      FStype  Options         Dump    Pass#
/dev/ad4s1a         /bootdir        ufs     rw              1       1
/dev/ad4s1b         none            swap    sw              0       0
/dev/ad4s1d         /               hammer  rw              1       1
/dev/cd0            /cdrom          cd9660  ro,noauto       0       0
proc                /proc           procfs  rw              0       0

You might want to edit /mnt/etc/rc.conf now or later, after we have booted into the system.

That’s all, so we unmount the Hammer root partition and reboot (don’t forget to remove the live-cd):

umount /mnt
reboot

Once we are up and running in our new Hammer-powered DragonFly system, we want to set up the timezone edit /etc/rc.conf and set the root password. You can easily set your timezone via:

tzsetup

You also might want to make /usr/obj non-history aware:

chflags nohistory /usr/obj
chflags noshistory /usr/obj   # do we need this?

Checkout the pkgsrc sources:

cd /usr
make pkgsrc-checkout

What I also found to be very useful is to extend /usr/pkg/etc/mk.conf for the following two lines:

WRKOBJDIR=/usr/obj
CREATE_WRKDIR_SYMLINK=no

This tells pkgsrc to not create a work directory within each pkgsrc package where it extracts the sources and compiles the application. Instead it will use /usr/obj.

As we don’t want to backup and mirror all the contents of /usr/obj (which should be considered temporary data of no value), we can create it’s own Hammer pseudo-filesystem for it, which allows to treat it separately for pruning and mirroring operations.

rmdir /usr/obj
cd /usr
hammer pfs-master obj 

As Hammer’s pseudo-filesystems are represented by symlinks, it isn’t possible to assign access rights to them or flags like nohistory, but there should be a workaround using null mounts:

# We rename the PFS to obj.pfs
mv /usr/obj /usr/obj.pfs

# We create a directory /usr/obj, which holds the permissions 
mkdir /usr/obj

# Finally we null mount /usr/obj.pfs on top of /usr/obj
mount_null /usr/obj.pfs /usr/obj

We’d probably like to do the same for /tmp:

rm -rf /tmp
mkdir /tmp
chmod 1777 /tmp # sticky flags for /tmp

cd /
hammer pfs-master tmp.pfs
mount_null /tmp.pfs /tmp

Enjoy DragonFly and Hammer!

Last week I installed FreeBSD 7.0-RC2. Now that FreeBSD 7.0-RELEASE is available, I tried to update using freebsd-update:

freebsd-update -r 7.0-RELEASE upgrade
freebsd-update install
reboot

But I always got error message during “install” like:

chflags: ///etc/mail/freebsd.cf: Operation not supported

So maybe it only occurs if you’ve installed FreeBSD on a ZFS filesystem, which doesn’t possibly support those file flags like noschg. The simple solution to overcome this was:

mv /bin/chflags /bin/chflags.old
cat > /bin/chflags
#!/bin/sh
/bin/chflags.old $@
exit 0
^D
chmod +x /bin/chflags

After that I retried:

freebsd-update install
reboot

And it works!

A few seconds ago I tried out DragonFly’s HAMMER filesystem myself. All you need is to compile a new kernel with “options HAMMER” and the userland programs hammer, newfs_hammer and mount_hammer, plus a free partition (/dev/ad0s1d in my case). Here we go:

# newfs_hammer -L Home /dev/ad0s1d
Volume 0 DEVICE /dev/ad0s1d     size   4.00GB
---------------------------------------------
1 volume total size   4.00GB
cluster-size:         16.00MB
max-volume-size:       0.50TB
max-filesystem-size: 16384.00TB
boot-area-size:       16.00MB
memory-log-size:      16.00MB

allocate cluster id=10a9b3f3f83e5fd0 0@02040000
cluster 0 has 1024 buffers
# mount_hammer /dev/ad0s1d /mnt
# echo "hallo" > /mnt/abc
# hammer now
0x47910e84
# echo "hallo leute" > /mnt/abc
# cat /mnt/abc@@0x47910e73
hallo
# cat /mnt/abc
hallo leute

As you can see, at first I create a new HAMMER filesystem on partition ad0s1d using newfs_hammer. In the next step I mount that filesystem onto /mnt using mount_hammer. Then I create a file on it which contains “hallo”. After I acquire the current timestamp (hammer now) I overwrite the file with content “hallo leute”. But the old content is not gone. I can still access it using the @@timestamp notation. Nice ;-)

ZFS (Zeta filesystem), according to Sun the "last word in filesystems", is definitively a great piece of software. It frees us sysadmins from the burden to plan upfront how to partition a disk, makes backups so much easier due to snapshots, allows space efficient storage thanks to optional compression support and is very safe in terms of data corruption (at least it detects corruption) and it’s quickly online again after a crash. But is it really the "last word in filesystems"?

Well, it will be successful, no question. FreeBSD has it, Solaris of course, Apple too, and it’s planned for NetBSD. Don’t know about Linux (licensing issues) and Windows.

DragonFly’s HammerFS is progressing well and will be available in an alpha version with the next release of DragonFlyBSD which is planned to be out around by mid of February. First interest was shown to port it over to NetBSD (in userland using puffs), which is great news because NetBSD is still more widely used than DragonFly and just runs on more hardware (at least DragonFly still doesn’t run on my new laptop).

So what’s so great about HammerFS?

• Infinite snapshots.

  You don’t need to manually take snapshots. The system is able keep a history of every change according to a retention policy that you can specify.

• As-of mounts

  You can mount a filesystem or access a file as-of a specific point in time (in the past). What was the content of this file yesterday? Last week? Even atime and mtime is tracked.

• Clustered

  The major goal of HammerFS is clustering. A HammerFS filesystem can be shared by a bunch of machines. No need of a single master. Based on a Chorum protocol. (this is currently not implemented)

• Backups made easy

  Three things make backups easy. First, the as-of feature (eases taking tarballs). Second, journaling. And third, clustering. Do you need to backup at all if three or more machines have an identical copy of your filesystem including history? Think about the hurdles when restoring a traditional backup. With HammerFS you simply replace the harddisk and reconnect the machine to the network. Done! Think about a malicious or stupid (like me sometimes) user deleting files accidentially, right before the next backup is taken. You will loose your work, unless you use something like HammerFS. ZFS doesn’t help here because it doesn’t has the infinite snapshot feature. While I trust harddisks and RAIDs, I don’t trust users!

• File database

  HammerFS will allow you to open a file in DB mode. I don’t know how exactly this works out, but you will be able to use a file as a simple key/value database. No need for special libraries and probably quite fast.

The last couple of days I played with DragonFlyBSD. Except some minor issues (like fdisk completely hangs my BIOS :) it works pretty well. I was mostly interested in it's journaling provided by jscan. You can read more here (or follow the direct link to my mail).

I like DragonFly due to it's simplicity. Currently I am running FreeBSD 7.0-CURRENT. It has tons of features, but many of them are not well thought out. For example disk or file-system labeling through glabel confuses me a lot and doesn't work as expected. Normally it should show up the labeled disks/file systems in /dev/ufs/name or /dev/label/name, so that you can easily mount them without knowing the exact device number or slice. The concept is nice, but I think the implementation is way too complex (or too generic). Or I am just to stupid to understand. But I think the whole geom stuff (this framework is called geom) is not yet 100% stable.

Matthew Dillon (DragonFly) in now working on getting user land file systems working. Well, as I know him, this will be a comparatively easy task, as all the former work he did on DragonFly will help him a lot! So for example, the whole kernel can already run in user land. Or the work he did on syslink, which is a message passing interface.

Most of the features FreeBSD provides I don't need at all. What I really like is if the basic kernel infrastructure is very clean and understandable. This leads to a stable kernel and will make it easy to extend it.