ntecs.de

Swap is Dead - Long Live (SSD) Swap

2010-02-08T12:36:36.295241+01:00

Honestly, did you ever actually make use of the swap partition on your operating system? I don’t remember that my server ever made use of it and that with as little as 256 MB main memory. Nowadays, memory is very cheap and if your system starts to use the swap, something is usually going wrong. Most of the time it simply means that too many processes are running or that some processes use up too much memory or that they are configured to use too much memory, because they are optimized for more recent hardware.

Matthew Dillon, the leader of DragonFly and well-respected “guru” in the community, revived the swap as a system-wide fast filesystem cache when using SSDs. He extended DragonFly so that not only anonymous memory (i.e. memory not backed by a file) is written to the swap in case of a low physical memory situation, but also other types of memory, especially memory used up by the buffer cache. The buffer cache caches data read from a device (e.g. from a hard disk). By default, all unused physical memory is used for the buffer cache, greatly speeding up reads from that device in case of a buffer cache hit. But memory is limited. So the idea is to use SSDs as 2nd level buffer cache, i.e. when data is eleminated from the in-memory buffer cache because new data is read and the cache is full, it is written to the 2nd level SSD buffer cache, which usually is much larger in size (e.g. 40-120 GB). SSDs are much faster when it comes to random accesses (a hard disk can’t do more than 10 MB/sec) and also linear reads (you can get about 200 MB/sec with cheap SSDs) are almost twice as fast as regular SATA disks.

As SSDs have limited per-cell write cycles (1000 to 10000 depending on the technology), it is also important to limit writes to the SSDs depending on how long you want to use the SSD.

Read more in the thread describing various interesting issues and take a look at the swapcache manpage of DragonFly.

HAMMER Presentation at KIT

2010-01-28T16:47:01.384695+01:00

Yesterday I gave a presentation about DragonFlyBSD’s HAMMER filesystem at the end of the system architecture lecture here at Karlsruhe Institute of Technology (also well known as University of Karlsruhe). It was quite interesting and exciting to stand in our hugest lecture hall and talk to maybe 150 students about an highly “innovative” topic, at least that’s how our Professor announced my presentation.

Panoramas from Tajikistan Day 1

2010-01-11T17:00:26.847067+01:00

Here is the first round of panorama pictures taken on the Pamir highway from Osh (Kyrgyzstan) to Murgab (Tajikistan). It was the 30th October 2009.

Sunrise in the kyrgyz mountains, early in the morning.

The first high pass. We are still in Kyrgyzstan. There we saw a crashed truck fallen down the twisting road, which looked like a scrapped wrack.

Great panoramic view of 7000m mountains after leaving Sary-Tash, the last kyrgyz village before Tajikistan. We are close to Pik Lenin.

The same view as before but a bit darker/higher constrast.

Closer to the mountains.

Even more close to the mountains.

Heading towards Pik Lenin, the white mountain in the background. Still on kyrgyz territory.

In the Pamir highland right after crossing the Tajik border. From now on we are above 4000m for the next couple of hours. On the left you see the border fence to China.

Great view towards Karakul Lake

Stone circles at Karakul Lake. I don’t remember what my russian driver told me about those circles as my conciousness was only half left (due to stomach problems, hydrogen insufficiency and altitude sickness).

The same stone circles, but from another perspective.

After driving for hours through the desert without any water except the water for cooling the machine (which clearly wasn’t drinkable), this is the place where we recharge our driking water. From here it takes us hours to get to Murgab.

When we arrive Murgab at around 8pm we have over 14 hours behind us. The next morning I wake up with heavy headache, the result of altitude sickness, spending half a day at above 4000m and then the whole night at 3500m.

Stay tuned for Day 2, where I am on the way to the Wakhan Corridor which is only a stone’s throw away from the Afghan border.

Delicious Persian Food

2010-01-11T14:55:36.384602+01:00

I made some really delicious meals during the last couple of weeks using my new cookbook Die persische Küche. I started with Abdugh Xiar, a dessert with yoghurt, cucumber, raisins and walnuts and for the main meal I made Adas Polou, which consists of Basmati rice with dates and black lentils.

Last friday I made Kabab Tabei, which basically is minced meat made in a pan. Also very delicious.

But the highlight was clearly my cooking session on saturday, which resulted in Borani Badenjan, eggplant yoghurt (with a bit too much garlic), Shirin Polou, a feastful meal with stewed chicken and sweet orange rice and for dessert Sholeh Zard, golden rice pudding.

Wee 2.0.0 Released

2009-04-07T12:49:43.163656+02:00

Around 4 years have passed since the last release of Wee.

What is Wee?

Wee is a Seaside-inspired 1 web-framework for building highly-dynamic component-based web-applications.

What’s new?

100% Rack based
Support for continuations
JQuery AJAX support
Ruby 1.9 ready

Distinctive Features

Wee is not just another web framework. It is completely different.

Continuations

You can use continuations to model the page flow of the your application, in the same way as you’d call a method in a GUI application. Continuations are optional; nothing in Wee depends on them. Example:

# you can write code like this:

if callcc YesNoMessageBox.new('Really?')
  # do something
else
  # something else
end

Without continuations one has to use (ugly) Continuation Passing Style (CPS):

call YesNoMessageBox.new('Really?') do |res|
  if res
    # do something
  else
    # something else
  end
end

Backtracking

Backtracking means that the user can naturally use the browsers back button despite Wee’s statefulness. This is not taken for granted for stateful web frameworks. Behind the scenes, Wee keeps multiple states of the application around with only little help by the programmer.

Components

Contrary to the widely used model/view/controller (MVC) paradigm, Wee tighly couples the controller and the view within a component. Components itself are highly decoupled from the rest of the application and can be reused to construct more complex applications. The powerful programmatic HTML rendering approach further reduces wasting the programmers mind by avoiding switching files (controller/view) or languages (Ruby/HTML). Therefore in Wee everything is written in Ruby. The programmatic HTML renderer is not just a simple HTML builder, it provides very powerful methods for easily generating HTML constructs and registering callbacks. Example:

#
# Generating a <select> tag
#

# select an object from these items
items = [1, 2, 3, 4]

# the labels shown to the user
labels = items.map {|i| i.to_s}

# render it
r.select_list(items).labels(labels).callback {|choosen| p choosen}

# render a multi-select list, with objects 2 and 4 selected
r.select_list(items).multi.labels(labels).selected([2,4])

Call/Answer mechanism

From callback handlers you can call other components which in turn replace the current components view. The called component can later return (or answer) back to the calling componenent. Example:

class YesNoMessageBox < Wee::Component
  def initialize(msg)
    super()
    @msg = msg
  end

  def render(r)
    r.bold(@msg)
    r.form do
      r.submit_button.value('YES').callback { answer true }
      r.space
      r.submit_button.value('No').callback { answer false }
    end
  end
end

# Use call (or callcc) as in the Continuation section above
# to call a component:
call YesNoMessageBox.new('Really?')

The classical Hello World Example

require 'rubygems' if RUBY_VERSION < "1.9"
require 'wee'

class HelloWorld < Wee::Component
 def initialize
   super
   add_decoration Wee::PageDecoration.new(title="Hello World")
 end

 def render(r)
   r.h1 "Hello World from Wee!"
   r.div.onclick_callback { p "clicked" }.with("click here")
 end
end

Wee.run(HelloWorld) if __FILE__ == $0
# Now point your browser to http://localhost:2000/

Installation

gem install wee

More Resources

http://rubyforge.org/projects/wee
http://www.ntecs.de/projects/wee/doc/rdoc/
http://github.com/mneumann/wee

SAP goes Ruby? (Enterprise Ruby)

2009-04-01T11:01:41.665476+02:00

After Sun (Java: JRuby), Microsoft (.NET: IronRuby), Apple (ObjectiveC: MacRuby) and GemStone (Smalltalk: Maglev) showed their interest in Ruby, another big player – no less than SAP – is working on support for Ruby: BlueRuby – a RubyVM that integrates seamlessly with the ABAP environment.

Wee, Ruby 1.9 and Continuations

2009-02-25T17:33:33.197308+01:00

The last couple of days I spent refactoring my web application framework Wee, a Seaside-like framework for Ruby that I started back in 2004 with a lot of mental help from Avi Bryant (one of the main guys behind Seaside). It’s now approaching a 2.0 release. Wee is now fully Rack based (Rack is a commonly used Ruby Webserver Interface) and the code is in general even cleaner than it was before (huge parts were rewritten). Plus one cool feature: Continuations.

Continuations

Yesterday I finally thought it’s time to put back in continuations mainly because of some very interesting developments (i.e. patches) for Ruby 1.8.x, which seems to fix memory leaks that occured when using continuations.

Continuations in Wee were never used as extensively as they were used in Seaside. And it’s interesting to read that Seaside 2.8 reduced heavy usage of continuations – something that Wee did from the beginning :)

An example says more than thousands words, so here we go:

require 'wee'

class Page < Wee::Component
  def initialize
    add_decoration Wee::PageDecoration.new('Title')
    super
  end
  def render(r)
    r.anchor.callback {
      if callcc YesNoMessageBox.new('Really delete?')
        callcc InfoMessageBox.new('Deleted!')
      else
        callcc InfoMessageBox.new('Deleted action aborted')
      end
    }.with("delete?")
  end
end

class InfoMessageBox < Wee::Component
  def initialize(msg)
    @msg = msg
    super()
  end

  def render(r)
    r.h1(@msg)
    r.anchor.callback { answer }.with('OK')
  end
end

class YesNoMessageBox < InfoMessageBox
  def render(r)
    r.h1(@msg)
    r.anchor.callback { answer true }.with('YES')
    r.space(1)
    r.anchor.callback { answer false }.with('NO')
  end
end

Wee.runcc(Page)

The interesting part of this example is the anchor tag callback handler of component Page. When clicked it will display a YES/NO message box which will return true if you clicked on YES, otherwise false. When it returns, execution resumes exactly at the point after the callcc call. In fact, callcc returns the return value of the called component. A called component returns by calling the answer method (it behaves mostly like the regular return statement). The concrete mechanism how this all works out is a bit more complicated as it involves installing and removing several decorations to delegate rendering and catch exceptions upon answer, but this is totally unrelated to continuations.

So how would it look without the use of continuations? Well, the callback handler would turn from:

if callcc YesNoMessageBox.new('Really delete?')
  callcc InfoMessageBox.new('Deleted!')
else
  callcc InfoMessageBox.new('Deleted action aborted')
end

into:

call YesNoMessageBox.new('Really delete?') do |res|
  if res
    call InfoMessageBox.new('Deleted!')
  else
    call InfoMessageBox.new('Deleted action aborted')
  end
end

Actually in this concrete example it’s not hard to rewrite the code without using continuations. But for more complex examples it would get much worse. For example, using continations, a simple sequential flow like the following one:

callcc page1
callcc page2
callcc page3

would turn into the much less readable equivalent using CPS (“continuation passing style”):

call page1 do
  call page2 do
    call page3
  end
end

Which one would you prefer? The good thing: In Wee you can use both, that’s why I have the two methods call and callcc, the latter not to mix up with Kernel.callcc.

Performance and Memory Usage

The reason why continuations were basically unsupported in Wee for a very long time was that they leaked memory. Since a few months there circulated patches on the ruby-core mailing list that seem to fix those leaks. Actually the reason seams to be that some parts of the stack are not overwritten when calling a function and as such old values keep referenced: a leak is born! I haven’t tried those patches, but I think they work. Instead I tried Ruby 1.9.1. And wow! It’s incredible! Memory usage stays constant at 12 MB, regardless of the number of requests. And performance scales nearly linearily as I increase the number of threads.

For 10000 requests (with one thread) Ruby 1.9.1 takes 16 seconds and requires 12 MB of memory. The same example with Ruby 1.8.7 grows to 329 MB of memory and takes 56 seconds. That’s an increase in performance of factor 3.5 and a 27-fold reduction of memory.

The example I’m using for the benchmark is the following 2-level nested callcc component call:

class Benchmark < Wee::Component

  #
  # calls Called2 then returns
  #
  class Called1 < Wee::Component
    def render(r)
      r.anchor.callback { callcc Called2.new; answer }.with('back')
    end
  end

  class Called2 < Wee::Component
    def render(r)
      r.anchor.callback { answer }.with('back')
    end
  end

  def render(r)
    r.anchor.callback { callcc Called1.new }.with("click")
  end
end

Conclusion

Continuations seem to be stable enough and not too expensive in terms of memory and performance in Ruby 1.9 so that there is nothing against using them (wisely) within Wee. This makes Wee the only web framework for Ruby to my knowledge that uses continuations. Coupled with other great features provided by Wee, for example reusable components, backtracking or the programmatic HTML generation, this undoubtly allows Wee to be called one of the most advanced web application frameworks for Ruby. Worth to note is that Wee does not focus on RESTful multi-million page-view serving, scaling applications. Instead Wee focuses on very complex applications, similar as found in traditional GUIs, and to get the job done quick and beautiful.

Postgres 8.4 window functions

2009-01-22T08:05:23.342068+01:00

As you can read here, PostgreSQL will contain so called window functions which is part of the SQL 2008 specification. Now that I know how this feature is named, I know what I was missing for the past 10 years. Window functions allow you to produce aggregated values from each row over a specific set of rows (a partition).

Assuming a table staff defined as

create table staff (
    name    varchar(30),
    dept    varchar(30),
    salaray int
);

we can now show each staff person together with the average salary of the department he/she belongs to using a very simple SQL statement like

select name, dept, salary,
       avg(salary) over (partition by dept)
from staff;

Without using window functions the necessary SQL statement is considerably more complex and requires ugly subselects and joins:

select s.name, s.dept, s.salary,
       a.avg_salary
from staff s,
     (select dept, avg(salary) as avg_salary
      from staff group by dept) as a
where s.dept = a.dept

Window functions can do a lot more, so for example can produce cummulative sums easily.

From VirtualBox to Kvm/Qemu

2008-12-04T10:12:20.06375+01:00

VirtualBox is a nice piece of software that lets you easily run another operating system (e.g. Windows) from your host system which in my case is Linux at the moment. But for some obscure timer-related reasons, VirtualBox cannot run DragonFlyBSD, the system of my desire, whereas all the other “emulators” like VMware, Qemu or KVM have no problems running DragonFly.

I tried to patch DragonFly to be able to run within VirtualBox, inspired by an idea of someone else. It’s not perfect, but it works. Nevertheless, the rest of this article is about how to migrate from VirtualBox to Kernel Virtual Machine (KVM5).

O'Browser - OCaml in the Browser

2008-11-18T10:36:30.589387+01:00

O’Browser is an Objective Caml bytecode interpreter written in Javascript. It is capable of running a stripped down version of OCaml’s standard library. Some example applications can be seen here.

The implementation of the runtime counts around 6000 lines of Javascript code, plus another 7000 lines of OCaml code for the standard library. That’s quite a lot when compared against the approx. 4000 lines of Ruby code, including inline Javascript code of the Ruby core classes, of RubyJS. But it’s definitively a great way to run little OCaml applications within the browser, for those that use OCaml. Unfortunately I never made it to write any larger application in OCaml, I only ever used SML (MLTon) in a more extensive way.

Before I started to work on RubyJS (that’s now around 2 years ago), I also thought about compiling OCaml down to Javascript. My idea was to emit Javascript code instead of running OCaml bytecode within Javascript, and I think together with OCaml’s object model, this approach could be a viable alternative to Google’s Java to Javascript compiler when it comes to performance and static typing. RubyJS can’t simply give you all those static type guarantees and due to it’s dynamic typing system of Ruby where everything is a method call, it can’t be all that fast. But then, we all use Ruby or a similar dynamic language and it is just fast enough, so why stick with Java?