I moved my personal blogspot blog to Posterous which I like much more for a number of reasons. For example, to include pictures from flickr, I just have to put the URL and everything else is done by posterous. So from now on my new personal blog is this one.

I started a new blog about personal stuff.

Honestly, did you ever actually make use of the swap partition on your operating system? I don’t remember that my server ever made use of it and that with as little as 256 MB main memory. Nowadays, memory is very cheap and if your system starts to use the swap, something is usually going wrong. Most of the time it simply means that too many processes are running or that some processes use up too much memory or that they are configured to use too much memory, because they are optimized for more recent hardware.

Matthew Dillon, the leader of DragonFly and well-respected “guru” in the community, revived the swap as a system-wide fast filesystem cache when using SSDs. He extended DragonFly so that not only anonymous memory (i.e. memory not backed by a file) is written to the swap in case of a low physical memory situation, but also other types of memory, especially memory used up by the buffer cache. The buffer cache caches data read from a device (e.g. from a hard disk). By default, all unused physical memory is used for the buffer cache, greatly speeding up reads from that device in case of a buffer cache hit. But memory is limited. So the idea is to use SSDs as 2nd level buffer cache, i.e. when data is eleminated from the in-memory buffer cache because new data is read and the cache is full, it is written to the 2nd level SSD buffer cache, which usually is much larger in size (e.g. 40-120 GB). SSDs are much faster when it comes to random accesses (a hard disk can’t do more than 10 MB/sec) and also linear reads (you can get about 200 MB/sec with cheap SSDs) are almost twice as fast as regular SATA disks.

As SSDs have limited per-cell write cycles (1000 to 10000 depending on the technology), it is also important to limit writes to the SSDs depending on how long you want to use the SSD.

Read more in the thread describing various interesting issues and take a look at the swapcache manpage of DragonFly.

Yesterday I gave a presentation about DragonFlyBSD’s HAMMER filesystem at the end of the system architecture lecture here at Karlsruhe Institute of Technology (also well known as University of Karlsruhe). It was quite interesting and exciting to stand in our hugest lecture hall and talk to maybe 150 students about an highly “innovative” topic, at least that’s how our Professor announced my presentation.

Here is the first round of panorama pictures taken on the Pamir highway from Osh (Kyrgyzstan) to Murgab (Tajikistan). It was the 30th October 2009.

Sunrise in the kyrgyz mountains, early in the morning.

The first high pass. We are still in Kyrgyzstan. There we saw a crashed truck fallen down the twisting road, which looked like a scrapped wrack.

Great panoramic view of 7000m mountains after leaving Sary-Tash, the last kyrgyz village before Tajikistan. We are close to Pik Lenin.

The same view as before but a bit darker/higher constrast.

Closer to the mountains.

Even more close to the mountains.

Heading towards Pik Lenin, the white mountain in the background. Still on kyrgyz territory.

In the Pamir highland right after crossing the Tajik border. From now on we are above 4000m for the next couple of hours. On the left you see the border fence to China.

Great view towards Karakul Lake

Stone circles at Karakul Lake. I don’t remember what my russian driver told me about those circles as my conciousness was only half left (due to stomach problems, hydrogen insufficiency and altitude sickness).

The same stone circles, but from another perspective.

After driving for hours through the desert without any water except the water for cooling the machine (which clearly wasn’t drinkable), this is the place where we recharge our driking water. From here it takes us hours to get to Murgab.

When we arrive Murgab at around 8pm we have over 14 hours behind us. The next morning I wake up with heavy headache, the result of altitude sickness, spending half a day at above 4000m and then the whole night at 3500m.

Stay tuned for Day 2, where I am on the way to the Wakhan Corridor which is only a stone’s throw away from the Afghan border.

I made some really delicious meals during the last couple of weeks using my new cookbook Die persische Küche. I started with Abdugh Xiar, a dessert with yoghurt, cucumber, raisins and walnuts and for the main meal I made Adas Polou, which consists of Basmati rice with dates and black lentils.

Last friday I made Kabab Tabei, which basically is minced meat made in a pan. Also very delicious.

But the highlight was clearly my cooking session on saturday, which resulted in Borani Badenjan, eggplant yoghurt (with a bit too much garlic), Shirin Polou, a feastful meal with stewed chicken and sweet orange rice and for dessert Sholeh Zard, golden rice pudding.

Around 4 years have passed since the last release of Wee.

What is Wee?

Wee is a Seaside-inspired 1 web-framework for building highly-dynamic component-based web-applications.

What’s new?

• 100% Rack based
• Support for continuations
• JQuery AJAX support
• Ruby 1.9 ready

Distinctive Features

Wee is not just another web framework. It is completely different.

Continuations

You can use continuations to model the page flow of the your application, in the same way as you’d call a method in a GUI application. Continuations are optional; nothing in Wee depends on them. Example:

# you can write code like this:

if callcc YesNoMessageBox.new('Really?')
  # do something
else
  # something else
end

Without continuations one has to use (ugly) Continuation Passing Style (CPS):

call YesNoMessageBox.new('Really?') do |res|
  if res
    # do something
  else
    # something else
  end
end

Backtracking

Backtracking means that the user can naturally use the browsers back button despite Wee’s statefulness. This is not taken for granted for stateful web frameworks. Behind the scenes, Wee keeps multiple states of the application around with only little help by the programmer.

Components

Contrary to the widely used model/view/controller (MVC) paradigm, Wee tighly couples the controller and the view within a component. Components itself are highly decoupled from the rest of the application and can be reused to construct more complex applications. The powerful programmatic HTML rendering approach further reduces wasting the programmers mind by avoiding switching files (controller/view) or languages (Ruby/HTML). Therefore in Wee everything is written in Ruby. The programmatic HTML renderer is not just a simple HTML builder, it provides very powerful methods for easily generating HTML constructs and registering callbacks. Example:

#
# Generating a <select> tag
#

# select an object from these items
items = [1, 2, 3, 4]

# the labels shown to the user
labels = items.map {|i| i.to_s}

# render it
r.select_list(items).labels(labels).callback {|choosen| p choosen}

# render a multi-select list, with objects 2 and 4 selected
r.select_list(items).multi.labels(labels).selected([2,4])

Call/Answer mechanism

From callback handlers you can call other components which in turn replace the current components view. The called component can later return (or answer) back to the calling componenent. Example:

class YesNoMessageBox < Wee::Component
  def initialize(msg)
    super()
    @msg = msg
  end

  def render(r)
    r.bold(@msg)
    r.form do
      r.submit_button.value('YES').callback { answer true }
      r.space
      r.submit_button.value('No').callback { answer false }
    end
  end
end

# Use call (or callcc) as in the Continuation section above
# to call a component:
call YesNoMessageBox.new('Really?')

The classical Hello World Example

require 'rubygems' if RUBY_VERSION < "1.9"
require 'wee'

class HelloWorld < Wee::Component
 def initialize
   super
   add_decoration Wee::PageDecoration.new(title="Hello World")
 end

 def render(r)
   r.h1 "Hello World from Wee!"
   r.div.onclick_callback { p "clicked" }.with("click here")
 end
end

Wee.run(HelloWorld) if __FILE__ == $0
# Now point your browser to http://localhost:2000/

Installation

gem install wee

More Resources

• http://rubyforge.org/projects/wee

• http://www.ntecs.de/projects/wee/doc/rdoc/

• http://github.com/mneumann/wee

After Sun (Java: JRuby), Microsoft (.NET: IronRuby), Apple (ObjectiveC: MacRuby) and GemStone (Smalltalk: Maglev) showed their interest in Ruby, another big player – no less than SAP – is working on support for Ruby: BlueRuby – a RubyVM that integrates seamlessly with the ABAP environment.

The last couple of days I spent refactoring my web application framework Wee, a Seaside-like framework for Ruby that I started back in 2004 with a lot of mental help from Avi Bryant (one of the main guys behind Seaside). It’s now approaching a 2.0 release. Wee is now fully Rack based (Rack is a commonly used Ruby Webserver Interface) and the code is in general even cleaner than it was before (huge parts were rewritten). Plus one cool feature: Continuations.

Continuations

Yesterday I finally thought it’s time to put back in continuations mainly because of some very interesting developments (i.e. patches) for Ruby 1.8.x, which seems to fix memory leaks that occured when using continuations.

Continuations in Wee were never used as extensively as they were used in Seaside. And it’s interesting to read that Seaside 2.8 reduced heavy usage of continuations – something that Wee did from the beginning :)

An example says more than thousands words, so here we go:

require 'wee'

class Page < Wee::Component
  def initialize
    add_decoration Wee::PageDecoration.new('Title')
    super
  end
  def render(r)
    r.anchor.callback {
      if callcc YesNoMessageBox.new('Really delete?')
        callcc InfoMessageBox.new('Deleted!')
      else
        callcc InfoMessageBox.new('Deleted action aborted')
      end
    }.with("delete?")
  end
end

class InfoMessageBox < Wee::Component
  def initialize(msg)
    @msg = msg
    super()
  end

  def render(r)
    r.h1(@msg)
    r.anchor.callback { answer }.with('OK')
  end
end

class YesNoMessageBox < InfoMessageBox
  def render(r)
    r.h1(@msg)
    r.anchor.callback { answer true }.with('YES')
    r.space(1)
    r.anchor.callback { answer false }.with('NO')
  end
end

Wee.runcc(Page)

The interesting part of this example is the anchor tag callback handler of component Page. When clicked it will display a YES/NO message box which will return true if you clicked on YES, otherwise false. When it returns, execution resumes exactly at the point after the callcc call. In fact, callcc returns the return value of the called component. A called component returns by calling the answer method (it behaves mostly like the regular return statement). The concrete mechanism how this all works out is a bit more complicated as it involves installing and removing several decorations to delegate rendering and catch exceptions upon answer, but this is totally unrelated to continuations.

So how would it look without the use of continuations? Well, the callback handler would turn from:

if callcc YesNoMessageBox.new('Really delete?')
  callcc InfoMessageBox.new('Deleted!')
else
  callcc InfoMessageBox.new('Deleted action aborted')
end

into:

call YesNoMessageBox.new('Really delete?') do |res|
  if res
    call InfoMessageBox.new('Deleted!')
  else
    call InfoMessageBox.new('Deleted action aborted')
  end
end

Actually in this concrete example it’s not hard to rewrite the code without using continuations. But for more complex examples it would get much worse. For example, using continations, a simple sequential flow like the following one:

callcc page1
callcc page2
callcc page3

would turn into the much less readable equivalent using CPS (“continuation passing style”):

call page1 do
  call page2 do
    call page3
  end
end

Which one would you prefer? The good thing: In Wee you can use both, that’s why I have the two methods call and callcc, the latter not to mix up with Kernel.callcc.

Performance and Memory Usage

The reason why continuations were basically unsupported in Wee for a very long time was that they leaked memory. Since a few months there circulated patches on the ruby-core mailing list that seem to fix those leaks. Actually the reason seams to be that some parts of the stack are not overwritten when calling a function and as such old values keep referenced: a leak is born! I haven’t tried those patches, but I think they work. Instead I tried Ruby 1.9.1. And wow! It’s incredible! Memory usage stays constant at 12 MB, regardless of the number of requests. And performance scales nearly linearily as I increase the number of threads.

For 10000 requests (with one thread) Ruby 1.9.1 takes 16 seconds and requires 12 MB of memory. The same example with Ruby 1.8.7 grows to 329 MB of memory and takes 56 seconds. That’s an increase in performance of factor 3.5 and a 27-fold reduction of memory.

The example I’m using for the benchmark is the following 2-level nested callcc component call:

class Benchmark < Wee::Component

  #
  # calls Called2 then returns
  #
  class Called1 < Wee::Component
    def render(r)
      r.anchor.callback { callcc Called2.new; answer }.with('back')
    end
  end

  class Called2 < Wee::Component
    def render(r)
      r.anchor.callback { answer }.with('back')
    end
  end

  def render(r)
    r.anchor.callback { callcc Called1.new }.with("click")
  end
end

Conclusion

Continuations seem to be stable enough and not too expensive in terms of memory and performance in Ruby 1.9 so that there is nothing against using them (wisely) within Wee. This makes Wee the only web framework for Ruby to my knowledge that uses continuations. Coupled with other great features provided by Wee, for example reusable components, backtracking or the programmatic HTML generation, this undoubtly allows Wee to be called one of the most advanced web application frameworks for Ruby. Worth to note is that Wee does not focus on RESTful multi-million page-view serving, scaling applications. Instead Wee focuses on very complex applications, similar as found in traditional GUIs, and to get the job done quick and beautiful.

As you can read here, PostgreSQL will contain so called window functions which is part of the SQL 2008 specification. Now that I know how this feature is named, I know what I was missing for the past 10 years. Window functions allow you to produce aggregated values from each row over a specific set of rows (a partition).

Assuming a table staff defined as

create table staff (
    name    varchar(30),
    dept    varchar(30),
    salaray int
);

we can now show each staff person together with the average salary of the department he/she belongs to using a very simple SQL statement like

select name, dept, salary,
       avg(salary) over (partition by dept)
from staff;

Without using window functions the necessary SQL statement is considerably more complex and requires ugly subselects and joins:

select s.name, s.dept, s.salary,
       a.avg_salary
from staff s,
     (select dept, avg(salary) as avg_salary
      from staff group by dept) as a
where s.dept = a.dept

Window functions can do a lot more, so for example can produce cummulative sums easily.

VirtualBox is a nice piece of software that lets you easily run another operating system (e.g. Windows) from your host system which in my case is Linux at the moment. But for some obscure timer-related reasons, VirtualBox cannot run DragonFlyBSD, the system of my desire, whereas all the other “emulators” like VMware, Qemu or KVM have no problems running DragonFly.

I tried to patch DragonFly to be able to run within VirtualBox, inspired by an idea of someone else. It’s not perfect, but it works. Nevertheless, the rest of this article is about how to migrate from VirtualBox to Kernel Virtual Machine (KVM5).

O’Browser is an Objective Caml bytecode interpreter written in Javascript. It is capable of running a stripped down version of OCaml’s standard library. Some example applications can be seen here.

The implementation of the runtime counts around 6000 lines of Javascript code, plus another 7000 lines of OCaml code for the standard library. That’s quite a lot when compared against the approx. 4000 lines of Ruby code, including inline Javascript code of the Ruby core classes, of RubyJS. But it’s definitively a great way to run little OCaml applications within the browser, for those that use OCaml. Unfortunately I never made it to write any larger application in OCaml, I only ever used SML (MLTon) in a more extensive way.

Before I started to work on RubyJS (that’s now around 2 years ago), I also thought about compiling OCaml down to Javascript. My idea was to emit Javascript code instead of running OCaml bytecode within Javascript, and I think together with OCaml’s object model, this approach could be a viable alternative to Google’s Java to Javascript compiler when it comes to performance and static typing. RubyJS can’t simply give you all those static type guarantees and due to it’s dynamic typing system of Ruby where everything is a method call, it can’t be all that fast. But then, we all use Ruby or a similar dynamic language and it is just fast enough, so why stick with Java?

Christopher Nelson talked about Ruby in the Browser at this years’ RubyConf 2008. He mentioned RubyJS, my Ruby to Javascript compiler, as well as a new library called Red, which also translates Ruby to Javascript, but in a slightly different way.

There are lots of interesting articles about RubyJS and rubyjs_on_rails on his blog.

Automatic PFS creation for DragonFly’s HAMMER file system has now been committed (read here).

This picture is from last years RubyConf (2007) and shows Matz (the creator of Ruby ;-) in the middle surrounded by Nick Sutterer on the right and myself on the left.

I guess our friend Helder from Brasil took the photo. Both Nick and me nearly waited one year to get this picture. Thanks Helder!

This article describes how to get Qemu running on FreeBSD 7.0 using network address translation (NAT). I’ll assume bge0 as external network interface; replace it with your interface (e.g. re0 or wpi0). Using NAT is actually the only way to get internet connectivity via a wireless network from the guests. I’ll use 192.168.3.0/24 as a local network for all qemu instances and run the NAT gateway and DNS server on 192.168.3.1.

Installing Qemu

Install the ports qemu and kqemu-kmod from /usr/ports/emulators. Don’t forget to configure the qemu port with the kqemu kernel module accelerator, otherwise performance will not be optimal.

Host Configuration

To /boot/loader.conf add:

aio_load="YES"
kqemu_load="YES"

To /etc/rc.conf add:

cloned_interfaces="tap0"
ifconfig_tap0="inet 192.168.3.1 netmask 255.255.255.0 up"
gateway_enable="YES"
firewall_enable="YES"
firewall_type="OPEN"
natd_enable="YES"
natd_interface="bge0"
natd_flags="-same_ports"
named_enable="YES"
pf_enable="NO"

To /etc/sysctl.conf add:

net.link.tap.user_open=1

To /etc/devfs.rules add (also make sure that the user running qemu is part of group wheel):

add path 'tap*' mode 660

To /etc/namedb/named.conf add:

listen-on { 192.168.3.1; };

Now reboot.

Starting Qemu

Take a look at the man page for qemu for more options. For example to boot a DragonFly ISO image use:

qemu -m 256 -localtime -cdrom LATEST-Devel.iso \
     -net nic -net tap,ifname=tap0,script=no

For Windows XP I prefer:

qemu -m 512 -localtime -usb -std-vga -hda $IMG \
     -soundhw es1370 \
     -name WindowsXP \
     -net nic -net tap,ifname=tap0,script=no

The -snapshot option is also very useful, especially for Windows.

Configuring the Guest

Once your guest system is up and running you have to configure it’s network settings, so that you can connect to the internet. In case of BSD add the following two lines to /etc/rc.conf:

defaultrouter="192.168.3.1"
ifconfig_ed0="inet 192.168.3.2"

And for /etc/resolv.conf use:

nameserver 192.168.3.1

Voila!

Today the new and redesigned homepage of Hello2morrow was launched, which was primarily developed by myself (except for the graphics and the photoshop template which came from a design agency). That’s a good feeling when something gets finished!

It seems to be common pratice to separate the unit test code from the classes under test. In Ruby for example, the convention is to have a test/test_my_class.rb file which contains the test code for class MyClass (itself stored in lib/my_class.rb). But is it actually good pratice to move the test code out into another file?

What we have already learned in the past is that inline documentation – i.e. keeping the documentation next to the code – is in many cases superior to external documentation, for the following reasons:

• It lowers the barriers to write documentation at all.
• It is more likely that the documentation stays up-to-date (in sync with the code).
• It eases the understanding of the code.

Unit tests are not so different. They too document behaviour, just in a different language, one that is machine readable and executable. Having the unit tests within the same class (and file) could lead to some interesting advantages:

• It lowers the barriers to write unit-tests at all.
• It is more likely that the unit-tests stay up-to-date when new functionality is added.
• It eases the understanding of the code as the unit-tests can serve as additional documentation.

You mostly see the same advantages as for inline documentation. This is because unit tests exhibit exactly the same problems as documentation: Developers don’t like to write or keep them up-to-date. With inline unit testing this situation could be improved.

This article briefly describes how to install DragonFlyBSD on a Hammer filesystem, i.e. using Hammer for the root filesystem. Note that I make the assumption that the DragonFly installation will use the entire disk (in my case this is disk ad4). Also make sure that the DragonFly version you are using includes my RootOnHammer patches which should be the case for any release after 2.0 and every snapshot ISO image from now on. Btw, the whole approach is similar to ZFSOnRoot, except that I will not use the installer.

Start the live-cd installer and enter the root prompt.

Partition ad4 creating one slice covering the entire disk.

fdisk -B -I ad4

Generate a disklabel in your favourite editor that looks like the following one and save it under /tmp/disklabel.ad4s1.

#          size     offset    fstype
  a:     262144          0    4.2BSD	#     128.000MB
  b:    6291456     262144      swap	#    3072.000MB
  c:  234441585          0    unused	#  114473.430MB
  d:  227887984    6553600    4.2BSD	#  111273.430MB

I’m using 128 MB for the boot partition (a), which should be enough to hold several kernels and it’s modules. Note that I am using 3 GB for the swap partition (b). The remaining space is left for the Hammer partition (d).

Then store the disklabel onto your disk:

disklabel -R -r ad4s1 /tmp/disklabel.ad4s1

Create the boot filesystem (UFS) and the Hammer filesystem:

newfs /dev/ad4s1a
newfs_hammer -L HammerGeil /dev/ad4s1d 

At first we set up the boot filesystem by mounting it and copying the /boot directory, the kernel and it’s modules onto it:

mount /dev/ad4s1a /mnt
cpdup -vvv /boot /mnt/boot
cpdup -vvv /kernel /mnt/kernel
cpdup -vvv /modules /mnt/modules

Then we adjust /mnt/boot/loader.conf to use ad4s1d (our Hammer partition) as root filesystem:

touch /mnt/boot/loader.conf
echo 'vfs.root.mountfrom="hammer:ad4s1d"' >> /mnt/boot/loader.conf

That’s all we need to do for the boot partition, so we unmount it.

umount /mnt

Next we continue with the Hammer partition:

mount_hammer /dev/ad4s1d /mnt

onto which we will install all remaining parts of DragonFlyBSD. It’s as easy as copying the files from the live-cd to the Hammer partition:

cpdup -vvv /bin /mnt/bin
cpdup -vvv /sbin /mnt/sbin
cpdup -vvv /root /mnt/root
cpdup -vvv /usr /mnt/usr
cpdup -vvv /var /mnt/var
cpdup -vvv /dev /mnt/dev
cpdup -vvv /etc.hdd /mnt/etc

cp /.cshrc /.profile /mnt
mkdir /mnt/mnt /mnt/proc /mnt/tmp

We further want later on to mount the boot partition into /bootdir, and use /boot as short-cut (symlink) to /bootdir/boot:

mkdir /mnt/bootdir
cd mnt
ln -s bootdir/boot boot
cd ..

We also want to edit /mnt/etc/fstab to contain the following:

# Device            Mountpoint      FStype  Options         Dump    Pass#
/dev/ad4s1a         /bootdir        ufs     rw              1       1
/dev/ad4s1b         none            swap    sw              0       0
/dev/ad4s1d         /               hammer  rw              1       1
/dev/cd0            /cdrom          cd9660  ro,noauto       0       0
proc                /proc           procfs  rw              0       0

You might want to edit /mnt/etc/rc.conf now or later, after we have booted into the system.

That’s all, so we unmount the Hammer root partition and reboot (don’t forget to remove the live-cd):

umount /mnt
reboot

Once we are up and running in our new Hammer-powered DragonFly system, we want to set up the timezone edit /etc/rc.conf and set the root password. You can easily set your timezone via:

tzsetup

You also might want to make /usr/obj non-history aware:

chflags nohistory /usr/obj
chflags noshistory /usr/obj   # do we need this?

Checkout the pkgsrc sources:

cd /usr
make pkgsrc-checkout

What I also found to be very useful is to extend /usr/pkg/etc/mk.conf for the following two lines:

WRKOBJDIR=/usr/obj
CREATE_WRKDIR_SYMLINK=no

This tells pkgsrc to not create a work directory within each pkgsrc package where it extracts the sources and compiles the application. Instead it will use /usr/obj.

As we don’t want to backup and mirror all the contents of /usr/obj (which should be considered temporary data of no value), we can create it’s own Hammer pseudo-filesystem for it, which allows to treat it separately for pruning and mirroring operations.

rmdir /usr/obj
cd /usr
hammer pfs-master obj 

As Hammer’s pseudo-filesystems are represented by symlinks, it isn’t possible to assign access rights to them or flags like nohistory, but there should be a workaround using null mounts:

# We rename the PFS to obj.pfs
mv /usr/obj /usr/obj.pfs

# We create a directory /usr/obj, which holds the permissions 
mkdir /usr/obj

# Finally we null mount /usr/obj.pfs on top of /usr/obj
mount_null /usr/obj.pfs /usr/obj

We’d probably like to do the same for /tmp:

rm -rf /tmp
mkdir /tmp
chmod 1777 /tmp # sticky flags for /tmp

cd /
hammer pfs-master tmp.pfs
mount_null /tmp.pfs /tmp

Enjoy DragonFly and Hammer!

The presentation I gave in the context of my study thesis is now available here in german language. The thesis itself, written in English, is available here.

After working very hard during the last couple of weeks on my study thesis, I finished the last chapters today. In short what I did was to implement a performance efficient spiking neural net simulator from scratch in C++, including interfaces for Octave/Matlab (oh, how ugly can an interface be!) and Ruby.

The time writing reminded me at the time when I was writing the Ruby Developers Guide book back in 2001. At the beginning, you need to force yourself to get some sentences written. But after a while they flow more and more fluent out of your mouth (hands).

Ah, should I mention that I wrote my study thesis in English? I did it mainly to get a wider audience, but also because sometimes it is easier to describe technical aspects in English than it is in German, because a lot of words in computer sciene are already in English. But I have still problems with commats and English grammar :)

As you can see on the following pictures, RubyJS runs fine even within a browser on a mobile phone. It’s a bit slower of course, but it’s funny to follow the progress of the tests which you can’t see in a browser on a PC, because it’s way too fast. The mobile is a Sony Ericsson P1i.

I am pleased to announce the first public release of RubyJS, a Ruby to Javascript compiler. It is now available as a Rubygem, so that you can install it with a single command:

gem install rubyjs

To see what it compiles, and whether the generated Javascript code runs in your browser, take a look at it’s test suite. Watch out for the color red! If everything is “green”, it runs fine within your browser.

Right now there are only two examples, the obvious Hello World example (Ruby code here) and another example (Ruby code here) that demonstrates some preliminary support for Ruby Web Toolkit (an unfinished port of Google Web Toolkit), which btw will hopefully be completed by a this years Google Summer of Code project.

In this morning I made a benchmark that compares an example written in Javascript using Prototype against the equivalent version in RubyJS (which uses plain Ruby code).

The RubyJS version:

sum = 0
500_000.times {|i| sum += i}

The Javascript/Prototype version:

var sum = 0;
(500000).times(function(i) {sum += i;});

Not only is the Ruby(JS) version more readable (IMHO), but it is also faster.

Great news! Yesterday I tried to compile a C extension on Windows for Ruby and figured out the following problem myself (as described here):

The problem with this flexibility is that programs compiled and linked against version 6.0 cannot be called or used in programs linked with version 7.1 or 8.0, and vice versa.

So, if you’re not the owner of an old VC++ 6.0 compiler you will not succeed (the C++ compiler is free, but you just can’t find a download for it anymore).

I’ve learned a lot of languages over the past. But only a handful of those are still of interest to me. What those languages “left” have in common is that they allow to divide groups of digits in numbers, like in:

1_000_000

This rules out Python, Erlang, C, C++, Java, while leaves in Ruby, Ocaml and D. The latter three are those languages that I like most. Just a coincidence?

On Linux with the ext2 or ext3 file system you better not store more than 31998 directories in a directory. The same probably applies to files as well. Try this:

Dir.mkdir("dir")
33000.times {|i| Dir.mkdir("dir/#{i}") }

What I get on a Linux box is:

dir.rb:3:in `mkdir': Too many links - dir/31998 (Errno::EMLINK)

On FreeBSD using the default ufs2 there seems to be no limit. I tried 200_000 directories and it worked like a charm. Just don’t try to ls it :).

Why is this important? Because the crawler I am working on has crawled a little more than 33000 domains and now gets in trouble! I want a FreeBSD box!

This morning I came across HotRuby, a virtual machine for YARV (Ruby 1.9) bytecode for Javascript and Flash. When I started implementing RubyJS I also considered this approach (also using Ocaml bytecode :), but then abandoned it due to performance issues.

I quickly implemented the benchmark they give using RubyJS. It takes 11.57 seconds in HotRuby but only 4 seconds in RubyJS. The same benchmark executed in Ruby 1.8 takes around 24 seconds!!! But only 0.12 seconds if you replace the “+=” by ”«”, so I consider this a very bad benchmark because it only benchmarks the performance of String allocation in the underlying VM/interpreter and not that of RubyJS/HotRuby. A benchmark with a lot of method calls, as is usual in a regular Ruby application, would be a much better benchmark. I think this would show even better results for RubyJS.

HP announced an 8-socket (32-cores) AMD server as you can read on heise online. Another very interesting platform is the Tyan Transport GT28, which is a 1U (!) server with two dual-socket mainboard, i.e. up to 16-cores (using AMD quad core processors) in 1U. 32 memory slots make room for a lot of memory and 8 gigabit ethernet ports and optional Infiniband provide enough network bandwidth. The interesting thing is that it’s two computers in one, so you can run two completely independent operating systems on it (without the need of a hypervisor like Xen). And the price is lower, because AMD processors that can be used in a 4-socket setting are more expensive than two-socket ones. Perfect solution, especially as the bare-bone is only around $1900.

A few minutes ago I announced version 1.0.0 of Cplus2Ruby. I use it extensively in the pulsed neural net simulator called Yinspire that I am currently developing.

Damien Katz, the creator of CouchDB, writes about what sucks about Erlang.

Last week I installed FreeBSD 7.0-RC2. Now that FreeBSD 7.0-RELEASE is available, I tried to update using freebsd-update:

freebsd-update -r 7.0-RELEASE upgrade
freebsd-update install
reboot

But I always got error message during “install” like:

chflags: ///etc/mail/freebsd.cf: Operation not supported

So maybe it only occurs if you’ve installed FreeBSD on a ZFS filesystem, which doesn’t possibly support those file flags like noschg. The simple solution to overcome this was:

mv /bin/chflags /bin/chflags.old
cat > /bin/chflags
#!/bin/sh
/bin/chflags.old $@
exit 0
^D
chmod +x /bin/chflags

After that I retried:

freebsd-update install
reboot

And it works!

Why the lucky stiff, a great Rubyist and Artist, has extended my DragonFly checkpointing interface for Ruby as you can read here. He also talks about the new 1.2 release of the great DragonFlyBSD.

I liked Mercurial a lot but got disappointed in the past by it’s inability to store files larger than a few megabytes. Furthermore, I prefer the storage model of git, which is based on cryptographic hashes, which I have first seen in OpenCM many years ago and which was first popularized (IMHO) by Monotone.

So, yesterday I decided to get rid of Mercurial in favor of using git. This will allow me to store huge videos in a repository. I have also converted some of my subversion repositories which wasn’t all that easy because they got somehow corrupted so that I had to use a two year old backup (I haven’t used svn since two years :). It seems that I destroyed the original repository simply by using a newer version of the subversion client svn. Luckily, I haven’t made any change to the repository for more than two years, so the backup contained exactly the same data. In this point I fully agree with Linus that “Subversion is the most pointless project ever started”.

The last day I hacked on another piece of Erlang code. I gave it the name http_hub. It’s not that easy to explain what it actually does. Clients can connect to it, as it behaves for them like a HTTP server. The client’s request is forwarded to a backend server. But no answer is sent back to the client. Instead http_hub waits on another port (e.g. 14000) for another HTTP request from the backend server (usually a single worker node in a cluster will answer), which includes the response that should be sent back to the client. Once this arrives, the client is answered with the included response. If no request on port 14000 arrives within a specified time, an error response is sent back to the client.

The code is here.

A few months ago I implemented CplusRuby, a library for gluing C and Ruby together. Cplus2Ruby (note the "2" in it's name) is very similar except that it glues C++ instead of C with Ruby. One new feature is that mixing in modules now basically works, except for nested modules and open modules.

During the last two hours I implemented a hyper-fast (4000 files with ~110 MB in total in around 4 seconds) link extractor. It reads file names of HTML files from stdout, extracts all anchor href attributes and writes them line-by-line in the file with the same name + “.links”. The sources are here.

I read the term “Monkey Patching” a few days ago for the first time. But I didn’t knew what it means. Then, I heard it again in the chaosradio podcast about Moderne Webentwicklung (modern web development). So Monkey Patching is when you extend an existing class and add or change a method, for example to fix a bug, without needing to modify the source code itself (which you might not have access to). In Ruby this is easy, thanks to it’s open classes.

It’s nice to hear a new name for a concept that you are using since 9 years :)

I finally made it to put the slides of my RubyConf 07 talk RubyJS - Efficient Ruby to Javascript Compilation online.

I am looking for someone to sponsor me, so that I can continue work on RubyJS, my Ruby to Javascript compiler and finish the Google Web Toolkit port, called RWT. Interested? Then read my proposal and contact me.

Dieses Wochenende war ich mit meinem Cousin in dem Film I am Legend, mit Will Smith. Leider eher ein enttäuschender Film. Die paar Monster, vor denen man sich übrigends nicht mal richtig gruseln brauchte, habens dann auch nicht rausgerissen. Die Story war zwar ganz nett, aber irgendwie kam nichts dabei rüber. Man hätte viel mehr draus machen können. Also entweder total krasser Monsterfilm, oder lustig, aber die Kombination hat hier nicht geklappt, zumal die paar Gags nicht wirklich lustig waren.

ruby -rsocket -e's=TCPServer.new(5**5);loop{_=s.accept;_<<"HTTP/1.0 \
200 OK\r\n\r\n#{File.read(_.gets.split[1])rescue nil}";_.close}'

Then, to get the message of the day, point your browser to http://localhost:3125/etc/motd.

Today I gave a presentation within a small circle of students at my university. Guess about what. Ruby, of course! The slides and everything else can be found here. Don’t ask me why the PDF looks ugly. It’s converted from HTML :)

Started by Dave, replied by Matz, it ends lovely:

Oh, I’m having a bad day. More medicine is called for. Sorry.

Get well soon. Keep yourself warm, take nutritious food, and rest, including staying away from your computer ;-)

I wrote an implementation of the Arc Challenge in Ruby using my web-framework Wee. The sourcecode is available here. I haven’t used continuations, because I think I broke continuation support in some of the later versions of Wee (just because I haven’t used continuations in my Wee apps). Once Rubinius gets (even) more mature, and my spare time allows, I’d like to get back to some ideas of Wee and make use of continuations.

This article took the words right out of my mouth. I am a happy Mercurial user and git always confused me too much (besides that you shouldn’t take a look at it’s source code ;-). While the concepts behind git are more advanced, the UI is a lot worse IMHO.

Ein Ruby Programmierer:

cat *.rb | wc -l
> 33456
grep '$;' *.rb | wc -l
> 2

Einen (ehemaligen) Java Programmierer:

cat *.rb | wc -l
> 33456
grep '$;' *.rb | wc -l
> 10234

This patch was my first attempt in kernel hacking. The “challenge” was to implement a function lwkt_token_is_stale that determines whether a serializing token got stale while a light weight kernel thread (LWKT) is sleeping. Serializing tokens in DragonFlyBSD are a very nice concept for synchronizing data access. It’s somewhat integrated with the LWKT scheduler so that I had to understand that code prior of being able to make any modifications.

Have you ever wondered why operating system kernels are compiled without full compiler optimizations turned on? The reason is that it’s impossible to implement proper threading in C/C++ in a library, because the execution model of C/C++ is single-threaded. If the compiler doesn’t know about threads, it might optimize a global variable access into a register access and if now another thread writes to that global variable we have a race. That’s just one example what can happen. A good introduction into the problem is given here (in German) or in the video Getting C++ Threads Right by Hans Boehm.

The next version of C++ called C++0x will include approaches to overcome this limitation. That would be a nice addition to the C standard as well, as C is used quite extensivly in systems programming where this matters even more.

If you are into high-performance number crunching then you should consider Unified Parallel C (UPC). It unifies both message passing (as used by MPI) and the shared memory model (as used by OpenMP).

What I am dreaming of is a combination of features from UPC (unified message passing/shared memory), Cyclone (abstract data types, region analysis, fat pointers), C (performance) and some concepts from D (modules, templates, closures). I’d call this language ASYL - Advanced Systems Language.