Honestly, did you ever actually make use of the swap partition on your operating system? I don’t remember that my server ever made use of it and that with as little as 256 MB main memory. Nowadays, memory is very cheap and if your system starts to use the swap, something is usually going wrong. Most of the time it simply means that too many processes are running or that some processes use up too much memory or that they are configured to use too much memory, because they are optimized for more recent hardware.

Matthew Dillon, the leader of DragonFly and well-respected “guru” in the community, revived the swap as a system-wide fast filesystem cache when using SSDs. He extended DragonFly so that not only anonymous memory (i.e. memory not backed by a file) is written to the swap in case of a low physical memory situation, but also other types of memory, especially memory used up by the buffer cache. The buffer cache caches data read from a device (e.g. from a hard disk). By default, all unused physical memory is used for the buffer cache, greatly speeding up reads from that device in case of a buffer cache hit. But memory is limited. So the idea is to use SSDs as 2nd level buffer cache, i.e. when data is eleminated from the in-memory buffer cache because new data is read and the cache is full, it is written to the 2nd level SSD buffer cache, which usually is much larger in size (e.g. 40-120 GB). SSDs are much faster when it comes to random accesses (a hard disk can’t do more than 10 MB/sec) and also linear reads (you can get about 200 MB/sec with cheap SSDs) are almost twice as fast as regular SATA disks.

As SSDs have limited per-cell write cycles (1000 to 10000 depending on the technology), it is also important to limit writes to the SSDs depending on how long you want to use the SSD.

Read more in the thread describing various interesting issues and take a look at the swapcache manpage of DragonFly.

Yesterday I gave a presentation about DragonFlyBSD’s HAMMER filesystem at the end of the system architecture lecture here at Karlsruhe Institute of Technology (also well known as University of Karlsruhe). It was quite interesting and exciting to stand in our hugest lecture hall and talk to maybe 150 students about an highly “innovative” topic, at least that’s how our Professor announced my presentation.

This article briefly describes how to install DragonFlyBSD on a Hammer filesystem, i.e. using Hammer for the root filesystem. Note that I make the assumption that the DragonFly installation will use the entire disk (in my case this is disk ad4). Also make sure that the DragonFly version you are using includes my RootOnHammer patches which should be the case for any release after 2.0 and every snapshot ISO image from now on. Btw, the whole approach is similar to ZFSOnRoot, except that I will not use the installer.

Start the live-cd installer and enter the root prompt.

Partition ad4 creating one slice covering the entire disk.

fdisk -B -I ad4

Generate a disklabel in your favourite editor that looks like the following one and save it under /tmp/disklabel.ad4s1.

#          size     offset    fstype
  a:     262144          0    4.2BSD	#     128.000MB
  b:    6291456     262144      swap	#    3072.000MB
  c:  234441585          0    unused	#  114473.430MB
  d:  227887984    6553600    4.2BSD	#  111273.430MB

I’m using 128 MB for the boot partition (a), which should be enough to hold several kernels and it’s modules. Note that I am using 3 GB for the swap partition (b). The remaining space is left for the Hammer partition (d).

Then store the disklabel onto your disk:

disklabel -R -r ad4s1 /tmp/disklabel.ad4s1

Create the boot filesystem (UFS) and the Hammer filesystem:

newfs /dev/ad4s1a
newfs_hammer -L HammerGeil /dev/ad4s1d 

At first we set up the boot filesystem by mounting it and copying the /boot directory, the kernel and it’s modules onto it:

mount /dev/ad4s1a /mnt
cpdup -vvv /boot /mnt/boot
cpdup -vvv /kernel /mnt/kernel
cpdup -vvv /modules /mnt/modules

Then we adjust /mnt/boot/loader.conf to use ad4s1d (our Hammer partition) as root filesystem:

touch /mnt/boot/loader.conf
echo 'vfs.root.mountfrom="hammer:ad4s1d"' >> /mnt/boot/loader.conf

That’s all we need to do for the boot partition, so we unmount it.

umount /mnt

Next we continue with the Hammer partition:

mount_hammer /dev/ad4s1d /mnt

onto which we will install all remaining parts of DragonFlyBSD. It’s as easy as copying the files from the live-cd to the Hammer partition:

cpdup -vvv /bin /mnt/bin
cpdup -vvv /sbin /mnt/sbin
cpdup -vvv /root /mnt/root
cpdup -vvv /usr /mnt/usr
cpdup -vvv /var /mnt/var
cpdup -vvv /dev /mnt/dev
cpdup -vvv /etc.hdd /mnt/etc

cp /.cshrc /.profile /mnt
mkdir /mnt/mnt /mnt/proc /mnt/tmp

We further want later on to mount the boot partition into /bootdir, and use /boot as short-cut (symlink) to /bootdir/boot:

mkdir /mnt/bootdir
cd mnt
ln -s bootdir/boot boot
cd ..

We also want to edit /mnt/etc/fstab to contain the following:

# Device            Mountpoint      FStype  Options         Dump    Pass#
/dev/ad4s1a         /bootdir        ufs     rw              1       1
/dev/ad4s1b         none            swap    sw              0       0
/dev/ad4s1d         /               hammer  rw              1       1
/dev/cd0            /cdrom          cd9660  ro,noauto       0       0
proc                /proc           procfs  rw              0       0

You might want to edit /mnt/etc/rc.conf now or later, after we have booted into the system.

That’s all, so we unmount the Hammer root partition and reboot (don’t forget to remove the live-cd):

umount /mnt
reboot

Once we are up and running in our new Hammer-powered DragonFly system, we want to set up the timezone edit /etc/rc.conf and set the root password. You can easily set your timezone via:

tzsetup

You also might want to make /usr/obj non-history aware:

chflags nohistory /usr/obj
chflags noshistory /usr/obj   # do we need this?

Checkout the pkgsrc sources:

cd /usr
make pkgsrc-checkout

What I also found to be very useful is to extend /usr/pkg/etc/mk.conf for the following two lines:

WRKOBJDIR=/usr/obj
CREATE_WRKDIR_SYMLINK=no

This tells pkgsrc to not create a work directory within each pkgsrc package where it extracts the sources and compiles the application. Instead it will use /usr/obj.

As we don’t want to backup and mirror all the contents of /usr/obj (which should be considered temporary data of no value), we can create it’s own Hammer pseudo-filesystem for it, which allows to treat it separately for pruning and mirroring operations.

rmdir /usr/obj
cd /usr
hammer pfs-master obj 

As Hammer’s pseudo-filesystems are represented by symlinks, it isn’t possible to assign access rights to them or flags like nohistory, but there should be a workaround using null mounts:

# We rename the PFS to obj.pfs
mv /usr/obj /usr/obj.pfs

# We create a directory /usr/obj, which holds the permissions 
mkdir /usr/obj

# Finally we null mount /usr/obj.pfs on top of /usr/obj
mount_null /usr/obj.pfs /usr/obj

We’d probably like to do the same for /tmp:

rm -rf /tmp
mkdir /tmp
chmod 1777 /tmp # sticky flags for /tmp

cd /
hammer pfs-master tmp.pfs
mount_null /tmp.pfs /tmp

Enjoy DragonFly and Hammer!

On Linux with the ext2 or ext3 file system you better not store more than 31998 directories in a directory. The same probably applies to files as well. Try this:

Dir.mkdir("dir")
33000.times {|i| Dir.mkdir("dir/#{i}") }

What I get on a Linux box is:

dir.rb:3:in `mkdir': Too many links - dir/31998 (Errno::EMLINK)

On FreeBSD using the default ufs2 there seems to be no limit. I tried 200_000 directories and it worked like a charm. Just don’t try to ls it :).

Why is this important? Because the crawler I am working on has crawled a little more than 33000 domains and now gets in trouble! I want a FreeBSD box!

Why the lucky stiff, a great Rubyist and Artist, has extended my DragonFly checkpointing interface for Ruby as you can read here. He also talks about the new 1.2 release of the great DragonFlyBSD.

This patch was my first attempt in kernel hacking. The “challenge” was to implement a function lwkt_token_is_stale that determines whether a serializing token got stale while a light weight kernel thread (LWKT) is sleeping. Serializing tokens in DragonFlyBSD are a very nice concept for synchronizing data access. It’s somewhat integrated with the LWKT scheduler so that I had to understand that code prior of being able to make any modifications.

A few seconds ago I tried out DragonFly’s HAMMER filesystem myself. All you need is to compile a new kernel with “options HAMMER” and the userland programs hammer, newfs_hammer and mount_hammer, plus a free partition (/dev/ad0s1d in my case). Here we go:

# newfs_hammer -L Home /dev/ad0s1d
Volume 0 DEVICE /dev/ad0s1d     size   4.00GB
---------------------------------------------
1 volume total size   4.00GB
cluster-size:         16.00MB
max-volume-size:       0.50TB
max-filesystem-size: 16384.00TB
boot-area-size:       16.00MB
memory-log-size:      16.00MB

allocate cluster id=10a9b3f3f83e5fd0 0@02040000
cluster 0 has 1024 buffers
# mount_hammer /dev/ad0s1d /mnt
# echo "hallo" > /mnt/abc
# hammer now
0x47910e84
# echo "hallo leute" > /mnt/abc
# cat /mnt/abc@@0x47910e73
hallo
# cat /mnt/abc
hallo leute

As you can see, at first I create a new HAMMER filesystem on partition ad0s1d using newfs_hammer. In the next step I mount that filesystem onto /mnt using mount_hammer. Then I create a file on it which contains “hallo”. After I acquire the current timestamp (hammer now) I overwrite the file with content “hallo leute”. But the old content is not gone. I can still access it using the @@timestamp notation. Nice ;-)

ZFS (Zeta filesystem), according to Sun the "last word in filesystems", is definitively a great piece of software. It frees us sysadmins from the burden to plan upfront how to partition a disk, makes backups so much easier due to snapshots, allows space efficient storage thanks to optional compression support and is very safe in terms of data corruption (at least it detects corruption) and it’s quickly online again after a crash. But is it really the "last word in filesystems"?

Well, it will be successful, no question. FreeBSD has it, Solaris of course, Apple too, and it’s planned for NetBSD. Don’t know about Linux (licensing issues) and Windows.

DragonFly’s HammerFS is progressing well and will be available in an alpha version with the next release of DragonFlyBSD which is planned to be out around by mid of February. First interest was shown to port it over to NetBSD (in userland using puffs), which is great news because NetBSD is still more widely used than DragonFly and just runs on more hardware (at least DragonFly still doesn’t run on my new laptop).

So what’s so great about HammerFS?

• Infinite snapshots.

  You don’t need to manually take snapshots. The system is able keep a history of every change according to a retention policy that you can specify.

• As-of mounts

  You can mount a filesystem or access a file as-of a specific point in time (in the past). What was the content of this file yesterday? Last week? Even atime and mtime is tracked.

• Clustered

  The major goal of HammerFS is clustering. A HammerFS filesystem can be shared by a bunch of machines. No need of a single master. Based on a Chorum protocol. (this is currently not implemented)

• Backups made easy

  Three things make backups easy. First, the as-of feature (eases taking tarballs). Second, journaling. And third, clustering. Do you need to backup at all if three or more machines have an identical copy of your filesystem including history? Think about the hurdles when restoring a traditional backup. With HammerFS you simply replace the harddisk and reconnect the machine to the network. Done! Think about a malicious or stupid (like me sometimes) user deleting files accidentially, right before the next backup is taken. You will loose your work, unless you use something like HammerFS. ZFS doesn’t help here because it doesn’t has the infinite snapshot feature. While I trust harddisks and RAIDs, I don’t trust users!

• File database

  HammerFS will allow you to open a file in DB mode. I don’t know how exactly this works out, but you will be able to use a file as a simple key/value database. No need for special libraries and probably quite fast.

Last week my new laptop finally arrived! Juhuu! After thinking quite a long time about different vendors and models I decided to buy a HP Compaq 6710b. Well, it was definitively time to buy a new one. The old one couldn’t run at 1300 MHz anymore, only at 600 Mhz because it produced too much heat. Furthermore it couldn’t even start with power supply plugged in. But even worse was the cut in the display from top to bottom which made it hard to read a text on it.

At first I had to finalize the Windoof Visda installation and had to take backups, which took hours. And it was so slow. Probably due to having only 1 GB of RAM. By accident, or by pure interest, I choose the 64-bit version of Windoof, which was the second fault ;-). It worked, but very slow.

After spending hours of backing up Windoof, I installed NetBSD. Wow! It worked like a charm. I used the amd64 version of NetBSD-4.0-RC4. Everything worked out of the box. I installed it within half an hour. Then I noticed that, while X was running fine, my video card was at that time not supported. As such, the resolution was not optimal (200 pixels missing left and right). I couldn’t manage it to install a specific branch (jmcneill-pm) of NetBSD-current, which promised support for the Intel GM965 chipset.

Then I tried FreeBSD 7.0-BETA3. Peng! It couldn’t even boot. After researching on the mailing lists, it seemed to be a problem with the bge network card driver. I can’t remember how I got it installed exactly. I think disabling most of the devices in the BIOS got me to the point where I could install it on the harddrive. Then, I checked out the 8.0-current version of FreeBSD and compiled the kernel myself. This then worked with the bge driver. So, now I’m running FreeBSD 8.0-CURRENT.

The cool thing, everything is working! Bluetooth, USB, 11g WLAN, Gbit LAN, graphic card and the two cores. And it’s pretty stable, despite the CURRENT version. Well, software-wise 7.0 or 8.0 has some problems, mainly due to some problems with Linux emulation (Flash), which don’t work yet with symbol versioning.

Last night I thought, cool lets use the remaining 80 GB of my harddisk and install NetBSD on it again and then use it with Xen. But after installing the GRUB boot loader I couldn’t boot FreeBSD anymore. I think either GRUB or NetBSD just wrote some blocks too much into the FreeBSD partition and as such destroyed the disklabel. After spending a few hours in the night without success, this morning I finally had a success. I remembered the size of the BSD partitions (I only had two, one root and one swap) and installed a new disklabel into that partition. Yeah, after running fsck, the file system was working again. But by accident, I installed FreeBSD 6.2 over my 7.0/8.0 installation, because I used a 6.2 installer CD do write the disklabel. Now I couldn’t boot into my system anymore. So I booted the installed NetBSD, fetched the FreeBSD 7.0 base system, booted via CD and extracted those files over the 6.2 installation :). After some further magic here and there it was booting again. Puh!

I really have to say that I am impressed by NetBSD. It is rock solid and has really good hardware support (actually better than FreeBSD in my case). And it’s the only operating system other than Linux that is capable of running Xen as Dom0. Once this graphics driver problem finds it’s way into NetBSD current (hopefully it will merged into the 4.0 release), it’ll undertake a new step and get rid of my native FreeBSD install, and try to run it within Xen.

Dreister geht es kaum noch als diese Linux jungs hier. Können die kein Englisch, denn da steht doch klar und deutlich:

Permission to use, copy, modify, and distribute this software for any purpose with or without fee is hereby granted, provided that the above copyright notice and this permission notice appear in all copies.

Zu dumm nur wenn man sich erwischen lässt :)

This article shows some interesting results on how good PostgreSQL scales on FreeBSD 7.0 on a 8-core machine (compared to FreeBSD 6.2 and 2-core).

The last couple of days I played with DragonFlyBSD. Except some minor issues (like fdisk completely hangs my BIOS :) it works pretty well. I was mostly interested in it's journaling provided by jscan. You can read more here (or follow the direct link to my mail).

I like DragonFly due to it's simplicity. Currently I am running FreeBSD 7.0-CURRENT. It has tons of features, but many of them are not well thought out. For example disk or file-system labeling through glabel confuses me a lot and doesn't work as expected. Normally it should show up the labeled disks/file systems in /dev/ufs/name or /dev/label/name, so that you can easily mount them without knowing the exact device number or slice. The concept is nice, but I think the implementation is way too complex (or too generic). Or I am just to stupid to understand. But I think the whole geom stuff (this framework is called geom) is not yet 100% stable.

Matthew Dillon (DragonFly) in now working on getting user land file systems working. Well, as I know him, this will be a comparatively easy task, as all the former work he did on DragonFly will help him a lot! So for example, the whole kernel can already run in user land. Or the work he did on syslink, which is a message passing interface.

Most of the features FreeBSD provides I don't need at all. What I really like is if the basic kernel infrastructure is very clean and understandable. This leads to a stable kernel and will make it easy to extend it.

FreeBSD 7.0 will contain lot of new stuff, as you can read here.

Freshbsd collects commit messages from all BSDs in one place. Great to keep informed what's going on at the BSD front.

Feed for a BSD daemon. The first page of this lecture about logics contains a cartoon I really like. A penguin says

Penguins are black and white.

Some old TV shows are black and white.

Therefore some penguins are old TV shows.

followed by a statement

Logic: Another thing that penguins aren't very good at.

Bsdtalk 119 rambles about experiments with IPv6. There is also an associated homepage An IPv6 Test Lab which contains useful information. It also considers Windows Vista, which is quite uncommon for a bsdtalk :). Running IPv6 is also a very valuable source of information about IPv6.

A guy named Noah is starting a AMD 64-bit port of DragonFlyBSD. That's very cool!

DragonFly is getting support for running a virtual kernel multi-threaded, i.e. a SMP virtual kernel. A virtual kernel is a userland process (like running any other application) that runs a whole kernel. It's especially suited to make kernel programming life much easier, because you can now test a new kernel within seconds while you don't have to reboot your machine where you are working on.

I think this is required to ease Matt's work on the new HammerFS clustered file system.

I want to quote this article which quotes another article whose author concludes that "The Free Software Foundation has dumped a load of restrictions on us with GPLv3 and told us that restrictions lead to freedom and that it is good for us. That's a little too Bush administration-like for me [...]".

I bought a very small microphone for my old laptop, which can be plugged directly into the microphone port. Now how do I record audio on FreeBSD? It's very simple! At first make sure that your mixer settings are correct.

mixer rec 100:100
mixer mic 100:100
mixer =rec mic

Here we set the volume for the record and the microphone to 100% and as recording device we use the microphone. After that you can record sound with:

cat /dev/dsp > /tmp/sound.raw

And play it back with:

cat /tmp/sound.raw > /dev/dsp

Easy, uh?

OpenKeta is a webserver running inside the kernel. As such it saves some time required for switching context between kernel and user-mode. It's available for Linux and FreeBSD. I tried to compile the kernel module for FreeBSD. It compiles easily, but it doesn't run. I tried to find the reason, hacked around in the sources, but in the end give up. Luckily it didn't let my laptop crash :)

It's fantastic that people like Matthew Dillon -- the founder of DragonFlyBSD --- exist. He's doing really exciting work. Not only this, he also writes great commit messages, explaining what he is doing and why he is doing it this way. So if you want to learn something about operating systems, databases or distributed systems, the best is to subscribe to dragonfly.commits and dragonfly.kernel mailinglists and carefully read what he is writing. I do that!

Pkgsrc is the package management system (or ports system how it's called in FreeBSD) which was originally developed for NetBSD, but is also available for a lot of other systems (Solaris, Windows, Linux etc.). It is also the main package system for DragonFlyBSD.

Pkgjam is a "spin-off" from Pkgsrc with lots of new good ideas, using a database (SQLite) for storing the dependencies and using hard-links between all dependencies, so that each package is self-contained. You can simply remove a package without affecting any other package. I hope someday this will become the standard, as it'd make administration so much easier and less error-prone.

Dragonfly is a spin-off from the FreeBSD 4.x line of development and is mainly oriented towards cluster-computing, which is getting more important every day. But that’s not it’s primary goal, it runs as well on a simple desktop machine (I tried that in the past, and it runs as good as FreeBSD). What’s new in 1.8?

• Virtual Kernel: Runs a kernel in user-space, similar to User Mode Linux. This work is very important for future work, mainly for testing the cache coherency layer and clustering, which will be implemented as part of the kernel.

• nullfs: arbitrary stacking

• Much more

What I’d really like to see for BSD (FreeBSD or DragonFly) is something like upstart for Linux or launchd for OSX. I like upstart a lot more and dislike launchd’s XML config files. Such a daemon should make crond and inetd superfluous. And it monitors processes, so in case a process abnormally exits it would get restartet. This makes the whole system much more stable (imagine accidentally shutting down ssh).

By Jeremy C. Reed, a guy well known to at least NetBSD and DragonFly people. Here. Looks good! Buy it!

It’s still work in progress. But it’s great news. Once this is in a stable release, it gives you (as an admin) huge advantages! You don’t need to create fixed partitions anymore. BSD is growing extremely these days, that’s wonderful.

Hakin9 is all about security.

In my /var/log/auth.log file, I regularily see hundrends of lines like this:

Aug  1 15:03:20 core sshd[18272]: Invalid user test from 85.42.62.82

Some script-kiddies try to get into. Of course they cannot as we’ve choosen very good passwords or use PKI, but anyway it’s annoying. So I decided to block those IPs. The best way would be to do this inside of sshd. So I looked at the sourcecode, but quickly realized that it’s not easily possible to achive. No, the sourcecode is very easy to read, that’s not the problem. But it doesn’t work easily because there is one process listening for connections which then spawns off a child process for each connection. So there is no common information available about which IPs to block. But wait: I’ve had an idea how it could work! Upon authentification failure, the child terminates itself and returns a special AUTH_FAILED exitcode. The parent registers a SIGCHLD signalhandler and can this way block the IP. Now lets fetch my FreeBSD laptop and implement it!

Now I realized that it isn’t easily possible this way. Because sshd loops and asks the user for a password even if that user does not exist in the system.

But I’ve now wrote a little Ruby script that is called from swatch (a system logging daemon). It locks you out if you supply three times a wrong password or a wrong username.

Read this mail.

Of course, FreeBSD has thousands more users behind and a lot more developers than DragonFlyBSD has. But I’m more interested in a feature-wise comparison, and whether these features are really useful (for me). Personally I feel more attracted to DragonFly (especially as a server platform) because of:

• pkgsrc as packaging framework, which is also used by NetBSD.
• They tend to do it right and not add thousands of features.
• It has ever been pretty stable. If you ever ran FreeBSD 5.x, you know what instability means (okay, that were probably issues with unstable drivers, not the OS per se).
• I have realized that FreeBSD’s features like ACL (Access Control Lists) and MAC (Mandantory Access Control) are insufficient for my purposes. The latter is only able to deny access, but not to allow. NetBSD’s approach (taken from Apple) is much more generic here and I hope it will someday find it’s way into DragonFly.
• DF follows a “kind-of” MicroKernel approach with message passing.
• I think the approach taken by DF will lead to pretty good scalability. Locality is the keyword here. Trying to avoid locks or communication between multiple processors as far as possible and use advanced algorithms.
• A journaling filesystem is going to be implemented (it’s mostly there), where you can send the journal to a remote server and this way keep a live-backup!
• The plan to use ZFS as filesystem (no more need for vinum or geom). This leads to a great simplification of administration (you can grow/shrink partitions dynamically).

For Desktop use, DragonFly must support: USB 2.0, WPA and ACPI. The last time I tried it, USB 2.0 worked as well as ACPI. It seems that WPA is now there, but even on FreeBSD I didn’t have gotten it to work. I also tried PC-BSD as a desktop platform, but I’m not really happy with it as some ports simply don’t work (vim for example).

You can now get the 1.5 Java binaries from here.