Around 4 years have passed since the last release of Wee.

What is Wee?

Wee is a Seaside-inspired 1 web-framework for building highly-dynamic component-based web-applications.

What’s new?

• 100% Rack based
• Support for continuations
• JQuery AJAX support
• Ruby 1.9 ready

Distinctive Features

Wee is not just another web framework. It is completely different.

Continuations

You can use continuations to model the page flow of the your application, in the same way as you’d call a method in a GUI application. Continuations are optional; nothing in Wee depends on them. Example:

# you can write code like this:

if callcc YesNoMessageBox.new('Really?')
  # do something
else
  # something else
end

Without continuations one has to use (ugly) Continuation Passing Style (CPS):

call YesNoMessageBox.new('Really?') do |res|
  if res
    # do something
  else
    # something else
  end
end

Backtracking

Backtracking means that the user can naturally use the browsers back button despite Wee’s statefulness. This is not taken for granted for stateful web frameworks. Behind the scenes, Wee keeps multiple states of the application around with only little help by the programmer.

Components

Contrary to the widely used model/view/controller (MVC) paradigm, Wee tighly couples the controller and the view within a component. Components itself are highly decoupled from the rest of the application and can be reused to construct more complex applications. The powerful programmatic HTML rendering approach further reduces wasting the programmers mind by avoiding switching files (controller/view) or languages (Ruby/HTML). Therefore in Wee everything is written in Ruby. The programmatic HTML renderer is not just a simple HTML builder, it provides very powerful methods for easily generating HTML constructs and registering callbacks. Example:

#
# Generating a <select> tag
#

# select an object from these items
items = [1, 2, 3, 4]

# the labels shown to the user
labels = items.map {|i| i.to_s}

# render it
r.select_list(items).labels(labels).callback {|choosen| p choosen}

# render a multi-select list, with objects 2 and 4 selected
r.select_list(items).multi.labels(labels).selected([2,4])

Call/Answer mechanism

From callback handlers you can call other components which in turn replace the current components view. The called component can later return (or answer) back to the calling componenent. Example:

class YesNoMessageBox < Wee::Component
  def initialize(msg)
    super()
    @msg = msg
  end

  def render(r)
    r.bold(@msg)
    r.form do
      r.submit_button.value('YES').callback { answer true }
      r.space
      r.submit_button.value('No').callback { answer false }
    end
  end
end

# Use call (or callcc) as in the Continuation section above
# to call a component:
call YesNoMessageBox.new('Really?')

The classical Hello World Example

require 'rubygems' if RUBY_VERSION < "1.9"
require 'wee'

class HelloWorld < Wee::Component
 def initialize
   super
   add_decoration Wee::PageDecoration.new(title="Hello World")
 end

 def render(r)
   r.h1 "Hello World from Wee!"
   r.div.onclick_callback { p "clicked" }.with("click here")
 end
end

Wee.run(HelloWorld) if __FILE__ == $0
# Now point your browser to http://localhost:2000/

Installation

gem install wee

More Resources

• http://rubyforge.org/projects/wee

• http://www.ntecs.de/projects/wee/doc/rdoc/

• http://github.com/mneumann/wee

After Sun (Java: JRuby), Microsoft (.NET: IronRuby), Apple (ObjectiveC: MacRuby) and GemStone (Smalltalk: Maglev) showed their interest in Ruby, another big player – no less than SAP – is working on support for Ruby: BlueRuby – a RubyVM that integrates seamlessly with the ABAP environment.

The last couple of days I spent refactoring my web application framework Wee, a Seaside-like framework for Ruby that I started back in 2004 with a lot of mental help from Avi Bryant (one of the main guys behind Seaside). It’s now approaching a 2.0 release. Wee is now fully Rack based (Rack is a commonly used Ruby Webserver Interface) and the code is in general even cleaner than it was before (huge parts were rewritten). Plus one cool feature: Continuations.

Continuations

Yesterday I finally thought it’s time to put back in continuations mainly because of some very interesting developments (i.e. patches) for Ruby 1.8.x, which seems to fix memory leaks that occured when using continuations.

Continuations in Wee were never used as extensively as they were used in Seaside. And it’s interesting to read that Seaside 2.8 reduced heavy usage of continuations – something that Wee did from the beginning :)

An example says more than thousands words, so here we go:

require 'wee'

class Page < Wee::Component
  def initialize
    add_decoration Wee::PageDecoration.new('Title')
    super
  end
  def render(r)
    r.anchor.callback {
      if callcc YesNoMessageBox.new('Really delete?')
        callcc InfoMessageBox.new('Deleted!')
      else
        callcc InfoMessageBox.new('Deleted action aborted')
      end
    }.with("delete?")
  end
end

class InfoMessageBox < Wee::Component
  def initialize(msg)
    @msg = msg
    super()
  end

  def render(r)
    r.h1(@msg)
    r.anchor.callback { answer }.with('OK')
  end
end

class YesNoMessageBox < InfoMessageBox
  def render(r)
    r.h1(@msg)
    r.anchor.callback { answer true }.with('YES')
    r.space(1)
    r.anchor.callback { answer false }.with('NO')
  end
end

Wee.runcc(Page)

The interesting part of this example is the anchor tag callback handler of component Page. When clicked it will display a YES/NO message box which will return true if you clicked on YES, otherwise false. When it returns, execution resumes exactly at the point after the callcc call. In fact, callcc returns the return value of the called component. A called component returns by calling the answer method (it behaves mostly like the regular return statement). The concrete mechanism how this all works out is a bit more complicated as it involves installing and removing several decorations to delegate rendering and catch exceptions upon answer, but this is totally unrelated to continuations.

So how would it look without the use of continuations? Well, the callback handler would turn from:

if callcc YesNoMessageBox.new('Really delete?')
  callcc InfoMessageBox.new('Deleted!')
else
  callcc InfoMessageBox.new('Deleted action aborted')
end

into:

call YesNoMessageBox.new('Really delete?') do |res|
  if res
    call InfoMessageBox.new('Deleted!')
  else
    call InfoMessageBox.new('Deleted action aborted')
  end
end

Actually in this concrete example it’s not hard to rewrite the code without using continuations. But for more complex examples it would get much worse. For example, using continations, a simple sequential flow like the following one:

callcc page1
callcc page2
callcc page3

would turn into the much less readable equivalent using CPS (“continuation passing style”):

call page1 do
  call page2 do
    call page3
  end
end

Which one would you prefer? The good thing: In Wee you can use both, that’s why I have the two methods call and callcc, the latter not to mix up with Kernel.callcc.

Performance and Memory Usage

The reason why continuations were basically unsupported in Wee for a very long time was that they leaked memory. Since a few months there circulated patches on the ruby-core mailing list that seem to fix those leaks. Actually the reason seams to be that some parts of the stack are not overwritten when calling a function and as such old values keep referenced: a leak is born! I haven’t tried those patches, but I think they work. Instead I tried Ruby 1.9.1. And wow! It’s incredible! Memory usage stays constant at 12 MB, regardless of the number of requests. And performance scales nearly linearily as I increase the number of threads.

For 10000 requests (with one thread) Ruby 1.9.1 takes 16 seconds and requires 12 MB of memory. The same example with Ruby 1.8.7 grows to 329 MB of memory and takes 56 seconds. That’s an increase in performance of factor 3.5 and a 27-fold reduction of memory.

The example I’m using for the benchmark is the following 2-level nested callcc component call:

class Benchmark < Wee::Component

  #
  # calls Called2 then returns
  #
  class Called1 < Wee::Component
    def render(r)
      r.anchor.callback { callcc Called2.new; answer }.with('back')
    end
  end

  class Called2 < Wee::Component
    def render(r)
      r.anchor.callback { answer }.with('back')
    end
  end

  def render(r)
    r.anchor.callback { callcc Called1.new }.with("click")
  end
end

Conclusion

Continuations seem to be stable enough and not too expensive in terms of memory and performance in Ruby 1.9 so that there is nothing against using them (wisely) within Wee. This makes Wee the only web framework for Ruby to my knowledge that uses continuations. Coupled with other great features provided by Wee, for example reusable components, backtracking or the programmatic HTML generation, this undoubtly allows Wee to be called one of the most advanced web application frameworks for Ruby. Worth to note is that Wee does not focus on RESTful multi-million page-view serving, scaling applications. Instead Wee focuses on very complex applications, similar as found in traditional GUIs, and to get the job done quick and beautiful.

Christopher Nelson talked about Ruby in the Browser at this years’ RubyConf 2008. He mentioned RubyJS, my Ruby to Javascript compiler, as well as a new library called Red, which also translates Ruby to Javascript, but in a slightly different way.

There are lots of interesting articles about RubyJS and rubyjs_on_rails on his blog.

This picture is from last years RubyConf (2007) and shows Matz (the creator of Ruby ;-) in the middle surrounded by Nick Sutterer on the right and myself on the left.

I guess our friend Helder from Brasil took the photo. Both Nick and me nearly waited one year to get this picture. Thanks Helder!

In this morning I made a benchmark that compares an example written in Javascript using Prototype against the equivalent version in RubyJS (which uses plain Ruby code).

The RubyJS version:

sum = 0
500_000.times {|i| sum += i}

The Javascript/Prototype version:

var sum = 0;
(500000).times(function(i) {sum += i;});

Not only is the Ruby(JS) version more readable (IMHO), but it is also faster.

Great news! Yesterday I tried to compile a C extension on Windows for Ruby and figured out the following problem myself (as described here):

The problem with this flexibility is that programs compiled and linked against version 6.0 cannot be called or used in programs linked with version 7.1 or 8.0, and vice versa.

So, if you’re not the owner of an old VC++ 6.0 compiler you will not succeed (the C++ compiler is free, but you just can’t find a download for it anymore).

This morning I came across HotRuby, a virtual machine for YARV (Ruby 1.9) bytecode for Javascript and Flash. When I started implementing RubyJS I also considered this approach (also using Ocaml bytecode :), but then abandoned it due to performance issues.

I quickly implemented the benchmark they give using RubyJS. It takes 11.57 seconds in HotRuby but only 4 seconds in RubyJS. The same benchmark executed in Ruby 1.8 takes around 24 seconds!!! But only 0.12 seconds if you replace the “+=” by ”«”, so I consider this a very bad benchmark because it only benchmarks the performance of String allocation in the underlying VM/interpreter and not that of RubyJS/HotRuby. A benchmark with a lot of method calls, as is usual in a regular Ruby application, would be a much better benchmark. I think this would show even better results for RubyJS.

A few minutes ago I announced version 1.0.0 of Cplus2Ruby. I use it extensively in the pulsed neural net simulator called Yinspire that I am currently developing.

Why the lucky stiff, a great Rubyist and Artist, has extended my DragonFly checkpointing interface for Ruby as you can read here. He also talks about the new 1.2 release of the great DragonFlyBSD.

A few months ago I implemented CplusRuby, a library for gluing C and Ruby together. Cplus2Ruby (note the "2" in it's name) is very similar except that it glues C++ instead of C with Ruby. One new feature is that mixing in modules now basically works, except for nested modules and open modules.

I read the term “Monkey Patching” a few days ago for the first time. But I didn’t knew what it means. Then, I heard it again in the chaosradio podcast about Moderne Webentwicklung (modern web development). So Monkey Patching is when you extend an existing class and add or change a method, for example to fix a bug, without needing to modify the source code itself (which you might not have access to). In Ruby this is easy, thanks to it’s open classes.

It’s nice to hear a new name for a concept that you are using since 9 years :)

I finally made it to put the slides of my RubyConf 07 talk RubyJS - Efficient Ruby to Javascript Compilation online.

I am looking for someone to sponsor me, so that I can continue work on RubyJS, my Ruby to Javascript compiler and finish the Google Web Toolkit port, called RWT. Interested? Then read my proposal and contact me.

ruby -rsocket -e's=TCPServer.new(5**5);loop{_=s.accept;_<<"HTTP/1.0 \
200 OK\r\n\r\n#{File.read(_.gets.split[1])rescue nil}";_.close}'

Then, to get the message of the day, point your browser to http://localhost:3125/etc/motd.

Today I gave a presentation within a small circle of students at my university. Guess about what. Ruby, of course! The slides and everything else can be found here. Don’t ask me why the PDF looks ugly. It’s converted from HTML :)

Started by Dave, replied by Matz, it ends lovely:

Oh, I’m having a bad day. More medicine is called for. Sorry.

Get well soon. Keep yourself warm, take nutritious food, and rest, including staying away from your computer ;-)

I wrote an implementation of the Arc Challenge in Ruby using my web-framework Wee. The sourcecode is available here. I haven’t used continuations, because I think I broke continuation support in some of the later versions of Wee (just because I haven’t used continuations in my Wee apps). Once Rubinius gets (even) more mature, and my spare time allows, I’d like to get back to some ideas of Wee and make use of continuations.

The following sentence in the article Garbage Collection is Why Ruby on Rails is Slow: Patches to Improve Performance 5x; Memory Profiling made me shudder:

“Complex Rails request can allocate hundreds or even thousands of megabytes of memory, making GC runs dozens of times.”

WTF?! Hundreds or even thousands of megabytes for a single (but complex) Rails request? Then there is seriously something wrong!

I’m currently reading the great Programming Erlang book. At the end of chapter 8, the author invites the reader to implement a ring benchmark in Erlang. Then implement the same in another language, compare the results and finally blog about it. That’s what I am doing here ;-). I’ve choosen Ruby because it’s the language I am mostly familiar with (and I am too lazy to implement it using C and MPI).

The Ruby version uses Fibers (Co-Routines) and as such needs Ruby 1.9. It has half the number of lines as the Erlang version and is IMHO easier to understand, thanks to Arrays and loops. The Erlang program uses processes and is around 4 times as fast.

The sourcecode for both languages you can find here.

A recent article asked the question whether dynamic languages can scale or not. I think the question can’t be answered that easily. Just because a project that used Python failed, doesn’t prove that dynamic languages can’t scale. There are just too many reasons for project failure. And I think Smalltalk proved that there can be big commercial projects that build on dynamic languages. The best thing of course is to reduce or even avoid the need of scaling, by careful architecturing and refactoring and by using the right tool. And IMHO lines of code isn’t a very good measurement for scaling.

I don’t think static typing is the silver bullet. It’s still far too limited, at least when using it in an imperative language.

One sentence in the above mentioned article made me feel somewhat sick:

“Languages like Ruby and Scala introduce new programming constructs and paradigms that are beyond the cognitive abilities of the average programmer”

At first I think, Java is far beyond the cognitive abilities of the average programmer. The whole environment a Java programmer usually has to use is incredible complex (build systems Maven, Ant, web frameworks, OO mapper) counting probably many millions lines of code.

The cited sentence also means that we need stupid programmers that don’t extend their knowledge. I couldn’t live in a setting like that. I expect from people and especially from programmers to be flexible, to be able to learn new languages. It’s the only way to prove that they have really understood the ideas behind the language and not just it’s syntax and how to apply syntax to specific problems. If they aren’t flexible enough, they will soon be replaced by machines. It’s that simple. So instead of accepting that programmers are limited, challenge them and give them a possibility to enhance their knowledge. But never accept sentences like the one above.

Lately I had an idea about lazily parsing JSON. Parsing JSON comes along with building a data structure in the host language (e.g. Ruby). For larger JSON documents this becomes expensive, especially if you’re only using a few values of the JSON. To do this efficiently, it’s important to specify forward skips as in the following example:

{/*20*/
 a: "test",
 b: "abc",
}

The 20 here means that the closing ”}” is 20 bytes later, so that the JSON parser can skip 20 bytes (after writing down the current location) and continue parsing. It’d create a special Hash object, which would lazily parse the inner JSON upon access.

Of course this requires that the JSON document is kept available (either in memory or on disk). Even for a large JSON document, the memory space to keep it in memory is usually far less than the memory used for all the Ruby values.

Another idea I had was that of a streaming JSON parser, similar to what exists for XML (SAX or expat). This would allow for very (space) efficient extraction of values out of a JSON document (of any size). Well, maybe I’ll rewrite my C++ JSON parser into a streaming parser one day.

It’s really true. Sometimes you (in this case I) are too stupid to see the simplest things. That’s why it is so important to talk about problems. This morning I talked with my dad about the strange performance problems I had with my neural net simulator embedded in Ruby. He came to a simple possible reason that the nets that I simulate might not be identical.

Well, I wrote my own JSON parser in C++. I use reference counting to avoid memory leaks and to allow sharing of objects, while I don’t use this sharing at all. But at least it’s possible. You can spend a lot of time with these issues, just to be sure that you don’t loose any memory. This can even be a performance bottle neck, because C++ strings for example are copied by default to overcome the problem of “who owns the memory”. That’s all ugly and bullshit!

For some reasons I implemented the jsonHash class as a linked list. It was easier at that time. Especially was it easier with this implementation to make sure that the reference counting holds true. But of course, a lookup turned into O(n) ;-).

Then I changed the file format for neural nets. Yeah and there I used a very big hash with 10000’s of entries. In Ruby no problem. But with my O(n) jsonHash C++ implementation this just didn’t worked anymore (O(n^2) runtime). So I had to change the fileformat again, using arrays. The crux is that the Ruby version still used the “old” file format that had this big hash.

Now it is that in Ruby 1.9 hashes retain order, but I think I executed the conversion script with Ruby 1.8 which doesn’t retain order. So the neural net entities were created in a different order, and connected in a different order as well. This made a difference of 10 seconds, not loading the net but simulating the net. 10 seconds seems to be not much, but that’s around 40%!

Yeah, I’m happy ;-)

The last couple of days I’ve spent improving the performance of my neural net simulator that I wanted to embed in Ruby (for easier scripting). The pure C++ version is very fast until I embed it into Ruby, in which case performance drops significantly. Here some numbers:

20.98 seconds, pure C++
24.16 seconds, C++ embedded in Ruby
26.16 seconds, C++ embedded in Ruby with wrapping C++ objects in Ruby objects
36.90 seconds, C++ embedded in Ruby with loading code written in Ruby (implies wrapping)

Note that in the last case, the loading code is written in Ruby. The time spent in loading the net is not significant (under 1 second)!

So how comes it that the same code embedded in Ruby just runs slower? At first I thought it must have to do with some compiler flags like -fPIC or -shared. But no, that’s not the reason!

My latest theory is that of memory fragmentation. Loading a net in Ruby allocates a lot of objects making further memory allocations slower. That’s the only reason I now can think of. Would be nice to have two separate memory frames, one for Ruby and one for my C++ application so that fragmenting one doesn’t hurt the other. In my C++ application I have to allocate a lot of memory for the priority queues, mainly expand them in size to hold more entries. That’s the only downside of the priority queues I use. The need for expanding an array. Maybe I use some more advanced data structure, something what I found under the name DSplay. A DSplay uses three structures, a splay tree which holds the newest items, a calendar queue (for one year) for the medium items and a linked list for all the “far future” items. The good thing is that this combination doesn’t need that much allocation. A free list allocator is enough. Of course I’d use an implicit heap instead of the splay tree, which would make it a lot faster IMO.

It’s a long standing tradition that new versions of Ruby are released at Christmas. The same happened this year ;-)

You can get Ruby 1.9.0 here. What is new?

• Bytecode interpreted. A lot faster!!!
• Support for m18n (multilingulization). Strings keep track of their encoding. No need to encode everything in UTF-8!
• Native threads.
• A few syntax changes. New Hash literal: {a: 4, b: 5}. New lambda: -> (a, b) { … }
• Semantic changes. Block arguments are always block local. New constant lookup rules.
• a lot more, as you can read here.

Actually, I am using Ruby 1.9 since some time. So I am prepared. Nevertheless it will take some time until 1.9 becomes the default!

A nice audio presentation by Avi Bryant about what Ruby can learn from Smalltalk.

Finally, the recording of my talk at RubyConf 2007 is online here. Slightly too technical and too monotone voice :). The other talks can be found here.

Matz Keynote at RubyConf 2007 was really amusing. In the first part he talked for example about the Ruby community or introduced the Love/Hate Ratio which shows that Ruby is the most lovely language on earth :).

In the second part he talked about what’s new in Ruby 1.9. Traditionally, Ruby releases are made on christmas, and this years christmas will bring us Ruby 1.9 he said. New features will be for example M17N (Ruby strings are encoding aware!), new syntax for lambdas which David A. Black doesn’t like (the syntax seems to be pretty ugly at the first sight, but on a closer look becomes a bit prettier) or external iterators.

Part three was mainly about the new YARV virtual machine, which gets rid of Matz ;-). YARV improves performance noticeable as shown here and also memory usage as shown for example here.

In the last part, Matz talked about our origin, who we are and that we will go Enterprisey ;-). My impression from this conference was that, yeah Ruby is now Enterprisey, with all those companies like Sun, Microsoft, Apple or TI working on/with Ruby. And that’s not Matz’s fault, blame Ruby on Rails :). Matz is “only” responsible for designing one of the most beautiful languages and for beeing such a kind person.

It’s Sunday and the RubyConf 2007 held in Charlotte, North Carolina is almost over. Todays talks are not that well visited as those on the last two days. I think the attendees are just tired. I was very surprised when I came into the big room on Friday. Around 500 people! Wow! So many people so that I had to sit on the floor ;-). The talks till noon were in the big room, then it was split into three rooms for separate tracks. I couldn’t relax until my talk on Saturday about RubyJS was over. I got some quite good feedback from some people. That made me happy. And now I’m tired :)

Very nice people! And the city… wow!

During the last two days I made huge progress on completing RubyJS. The last 30 commits added tons of test-cases and features:

• Implement Exceptions and begin, rescue, ensure statements
• Implement case statement and the “===” method for serveral classes.
• Complete implementation of iterators and break, next
• Implement String interpolation
• Produce HTML containing the Javascript code of all test-cases suitable for running easily inside a browser. Test cases succeed in IE6, Firefox, Opera and Konqueror.
• Implement Range class
• Optimize conditionals and unless
• Add useful methods like “gsub” to class String and fix “inspect”.
• Implement send, kind_of?, instance_of? and respond_to? methods
• Implement zsuper (super without arguments)

The only thing that is now missing is a more complete core library and method_missing. The latter is a huge benefit against plain old Javascript.

Just a few hours after I announced CplusRuby on ruby-talk the first bug-fixes and extension went in. Wow! That’s one of the reasons I love Open-Source and why I think Open-Source can produce much better software. One the one hand, you can share good stuff with others and make them happy. Of course, you also feel good because you get credits for your work. And on the other hand, you have to work less, because others send in their changes :). In the end, everyone is happy.

Now what changed in the few hours since I made the announcement?

• It should work on Windows and other platforms than Linux+BSD
• Two new types :int and :char_p
• C code generation fixed.

In my last article I wrote about using Ruby and C to implement a high-performance pulsed neural net simulator. The first result is a library called CplusRuby which makes it easy to mix Ruby and C. Take a look at the README and/or at the following example:

require 'cplusruby'

class NeuralEntity < CplusRuby
  property :id
end

class Neuron < NeuralEntity
  property :potential,        :float
  property :last_spike_time,  :float
  property :pre_synapses,     :value

  method_c :stimulate, %(float at, float weight), %{
    // this is C code
    selfc->potential += at*weight;
  }

  def initialize
    self.pre_synapses = []
  end
end

# generate C file, compile it and load the .so
CplusRuby.evaluate("inspire.cc", "-O3", "-lstdc++")

if __FILE__ == $0
  n = Neuron.new
  n.id = "n1"
  n.potential = 1.0
  n.stimulate(1.0, 2.0)
  p n.potential # => 3.0
end

Note that the crux of CplusRuby is high-performance. The properties form a C structure which is attached to the Ruby object. The C functions can access those values directly, and C functions are itself properties, as such you can call other C method-functions directly without going through Ruby, which is sloooow! On the other hand, you can access everything from Ruby as well. Of course all garbage collecting stuff is generated for you automatically!

You wouldn’t use Ruby for a high-performance pulsed neural network simulator, would you? The one I wrote is actually in C++, but I really consider to rewrite it in Ruby plus C.

The reason is that there is so much code that isn’t performance critial at all (loading and dumping of neural nets for example) and which is quite ugly to do in C++. Ruby on the other hand gives me a lot of things for free:

• a garbage collector
• dynamic arrays
• strings
• exceptions

The performance of the garbage collector isn’t a problem at all because during the simulation no new objects are created. As such, the garbage collector will only run very few times. The use of a garbage collector even makes it quite easy to modify the neural net at runtime without introducing memory leaks. You get headache to try the same in C++.

Then the performance of dynamic arrays and strings in Ruby is comparable to the equivalents in C++ (std::vector and std::string). And you don’t get memory leaks :).

Then everything else gets so much easier. I can reuse the JSON parser of Ruby. I can make a web-service out of the simulator easily. New neuron types could be prototyped in Ruby and then later converted to C for high performance. And a lot more.

If you like to take a look at the development version of Ruby (1.9)

svn co http://svn.ruby-lang.org/repos/ruby/trunk ruby
cd ruby
autoconf configure.in > configure
chmod +x configure
./configure --prefix=$HOME/usr
gmake # or make

After a few hours of hacking, I implemented and refined the ideas first mentioned in my article HTML in Ruby and released XML in Ruby as you can read here.

For my blogging software I am using Tenjin as template engine. I also tried Kwartz which is also very nice. With Tenjin you embed Ruby into HTML code, in the same way as you’d do in Erb or PHP. This turns out to get pretty ugly. Kwartz completely separates HTML from presentation logic. It is a nice concept, but the presentation logic file can get quite complex for just simple things.

So why not embed HTML in Ruby?

# Example for embedding HTML in Ruby :)

class MyView

  def render(posts)
    for post in posts
      render_post(post)
    end
  end

  def render_post(post)
    #<div id="${post.id}">
    if post.abstract?
      #<p>${post.abstract}</p>
    else
      #<p>${post.body}</p>
    end
    #</div>
  end

end

I think you got the idea, right? I just misuse comments to embed HTML in Ruby. This is very easy to parse and looks quite nice in an editor and it should be easy to tell vim to colorize the embedded HTML correctly. Note that everything that starts with “#<” would be embedded HTML so that you’d still be able to use regular Ruby comments.

Finally, I wrote my own blogging software. The one you see while you read this! The old one is no longer available, but most of the permalinks should stay valid, except a few. I also migrated all articles from my very old Rublog into this new one. For now, commenting is not yet implemented, but will come soon. This time, it is protected by a captcha.

Why I wrote my own blogging software? Well, I used typo before. But then this started to get really really slow for submitting new articles. Slow means that I had to wait up to a few minutes! Then it got a lot of spam. And as I looked into my database I saw some ten-thousands of sessions inside my database. What the hell! This is very bad for taking backups! That was the time when I decided to write my own.

Typo looks very nice from the outside, but in the end, I don't use all of it's features. Even worse, those features that I need are not implemented, or maybe I'm just ignorant. Mostly, a good spam filter for comments. Comments are just bullshit (I learned this word from Linus ;-) if you get a lot of them and most of them are spam.

My blogging engine is very slim. As such it doesn't use Rails at all! It supports all major markup languages: Textile (RedCloth), Markdown (BlueCloth, Maruku) and RDoc. It has nice code syntax highlighting using my vim colorize script. The articles are stored in the file system. I consider this a feature, as it makes incremental backups very easy (and efficient). Another feature is that you will never be able to delete or overwrite an article as I keep every version. Of course it has RSS feeds for articles and comments. And soon it will get full-text indexing.

Another thing that is different in my blog is that I only allow editing an article or creating a new one over HTTPS. And I do authentification via the HTTP server. That keeps my application slim. And I really don't know why it is used so little. It's a great method and a very simple one.

This will become my first visit of the United States. At RubyConf I'll give a talk about Efficient Ruby to Javascript Compilation and Applications. Take a look at the agenda for other presentations.

NASA's open source Ruby software. Just for the record :)

http://g3d-ruby.rubyforge.org/

Coool!

I've been using Kwartz lately and it's great! The good thing is that you can view your templates in your browser without the need to preprocess them first. Of course Kwartz only works for HTML or XML templates, but that's fine with me.

Well, I'm currently working on our PeopleSearch engine. It's part of a student project at Fraunhofer Gesellschaft, one of the big research labs in Germany. Below, there is a screenshot:

For the prototype that we want to show tomorrow, I took the logo of my Guuglehupf search engine that I wrote 5 years ago in MLton. The logo is of course copyright by my friend W. Maier, who supported me in Guuglehupf. So it's just a temporary solution :).

The credo of this project is: Java sucks! Java sucks! Java sucks! And this is not even my own opinion, as I coded in Ruby from the beginning, but if you compare the results from the guys that used Java to those of Ruby, it's incredible. Spending one week with Java for almost nothing is natural. Uhh, and don't try to read the code :).

The first week was mostly spend for exploration. We wrote extractors for various pages, collected ideas. This second week was more structured. We made plans and had more concrete ideas. But we are still in exploration mode. There are so many free variables left. Extracting and caching is easy. Now the more advanced ideas have to come into our minds. To realize those ideas we need Ruby, or we will fail. From now I can tell that Mongrel, Ferret and Hpricot are great tools for such kind of task. I use them all. For testing I use RSpec, but I'm not yet sure it fullfills all my need.

A simple reason: You can stream it!

require 'yaml'

a_big_array_of_hashes = [
  {...},
  {...},
  ... ]

File.open('/tmp/out', 'w+') do |f|
  a_big_array_of_hashes.each do |h|
    f.puts h.to_yaml
  end
end

# now read it back again

YAML::each_document(File.open('/tmp/out')) do |doc|
  p doc
end

Great stuff! Okay, in JSON it can be done as well, but maybe you need to write it yourself.

I'm currently extracting all German persons that are listed in Wikipedia. For this I used WWW::Mechanize, a library that I initially wrote, but now is maintained and extended by someone else. After fetching and parsing around 100 pages, the process took around 500 MB and pipes began to fail and stuff like that :). Mechanize keeps a history of all pages, so it's better to call history.clear from time to time, or maybe there is even an option to set the size of history. I found out about this after switching to using Hpricot directly (it is used by Mechanize anyway). So now memory stays constant with a maximum of around 30 MB.

The Hpricot HTML parser and XPath/CSS implementation is just - GREAT! So easy to use, so powerful, and I think it's also fast! Thanks _why!

At first I tried scrubyt!, but I never got it running. And after reading the documentation, I was quite confused. I think it tries to do too much and too much magic behind the scenes.

15 minutes ago, I read this article about DictMixing's in Python. In Ruby there is no special mixin for Hashes (Ruby's equivalent to Python dictionaries). But there is Enumerable. A Hash simply has the methods [] []= delete has_key? keys and a few more, but those are the most important ones.

77 lines later, I had a complete equivalent to Ian Bickings FSDict shown here:

class FSDict
  include Enumerable

  class << self
    alias [] new
  end

  def initialize(path)
    @path = path
  end

  def [](item)
    fn = path_for(item)
    unless File.exists?(fn)
      raise "File #{fn} does not exist"
    end
    if File.directory?(fn)
      self.class.new(fn)
    else
      File.read(fn)
    end
  end

  def []=(item, value)
    delete(item) if has_key?(item)
    fn = path_for(item)
    if value.kind_of?(FSDict)
      Dir.mkdir(fn)
      f = self[item]
      value.each {|k,v| f[k] = v} # recurse
    else
      File.open(fn, 'w+') {|f| f << value}
    end
  end

  def delete(item)
    fn = path_for(item)
    unless File.exists?(fn)
      raise "File #{fn} does not exist"
    end
    if File.directory?(fn)
      each {|k,v| delete(k)} # recurse
      Dir.rmdir(fn)
    else
      File.unlink(fn)
    end
  end

  def has_key?(item)
    File.exists?(File.join(@path, item))
  end

  alias include? has_key?

  def keys
    map {|k,v| k} # inefficient!
  end

  def each
    Dir.open(@path) do |d|
      d.each do |fn|
        next if fn == '.' or fn == '..'
        yield fn, self[fn]
      end
    end
  end

  private

  def path_for(item)
    File.join(@path, item)
  end
end

if $0 == __FILE__
  FSDict['/tmp']['dest'] = FSDict['/tmp']['bar']
end

By including the Enumerable mixin, you can use all kind of predefined methods, e.g.

FSDict['/tmp'].select {|i| i.is_a?(String) && i =~ /def/}

Well, of course using select or each is highly inefficient if your directory contains huge files, because every file is read-in and returned as string :)

Linus Thorvalds wrote git in around two weeks. The concepts behind are really simple (as is true for most source code versioning systems that use content addressing by cryptographic hashes). Huh, but it's now around 70k lines of code. Because I'm hacking on a similar version control system, I made some tests. I added the whole DragonFlyBSD source tree (/usr/src) into a git repository (git add . && git commit). Adding it took around 4.5 minutes and commiting it another 1.5 minutes, so in total it took 6 minutes. The git repository uses 154 MB. The same with my very early version control system takes around 5 minutes and consumes 139 MB. The original files consume 387 MB in total.

Did I mentioned that it's written in Ruby?

Okay, I am in the progress of porting Google Web Toolkit to RubyJS. The first 2500 lines of code (that includes a lot of comments) went pretty well. So far the basic is up and running. 99% of the DOM code (which I split up into Event and Element classes and renamed some methods accordingly) is functional. UIObject and Widget classes as well. But then it really starts to become Java-centric. Iterators, a lot of interfaces (e.g. class Label implments following interfaces: SourcesClickEvents, SourcesMouseEvents, SourcesMouseWheelEvents, HasHorizontalAlignment, HasText, HasWordWrap) and big switch statements make it now more difficult to continue with a mostly line-by-line translation. While I could just port everything line-by-line, I don't think this is the Ruby way. Right now I am looking at the Tree.java file, whose onBrowserEvent method (the Widgets central event dispatch method) contains a big, no a huge switch statement. As long as each case is followed by a break that's just fine. But it's not the case. What I really find ugly is this piece of code:

case Event.ONKEYDOWN:
/* lots of code removed */

  // Intentional fallthrough.
case Event.ONKEYUP:
  if (eventType == Event.ONKEYUP) {
    /* ... */
  }

  // Intentional fallthrough.
case Event.ONKEYPRESS:

How can a human understand that. It's as bad as goto.

Instead of huge switch statements, I have invented a event dispatch mechanism, which comes at no memory overhead (or only very little, at least no new object exists) and only one additional method call. So you can write:

mapEvent(Event::ONCLICK, `#<self>.#<m:onClick>`)
mapEvent(Event::ONMOUSEUP, `#<self>.#<m:onMouseDown>`)

and the ONCLICK event will trigger the onClick() call. If this extra method call would be too much overhead, this could be removed by moving the code into the Event dispatcher code (which currently calls onBrowserEvent).

I'm not yet sure how I will name my library which is similar to Google Web Toolkit and is based on my RubyJS. Yesterday, I implemented some DOM related stuff in under two hours. GWT uses class methods of the DOM class everywhere, e.g.:

DOM.setElementAttribute(Element elem, String attr, String value)

I don't follow this approach, because I think it's more elegant to move them into an Element class and make it an instance method instead:

el = Element.createDiv
el.setAttribute("a", "b")

The same applies to Events. This allows me to write for example:

Element['root'].appendChild(
  Element.createDiv.setInnerHTML('abc').
          setAttribute('title', 'tooltip'))

instead of:

root = DOM.getElementById('root')
elem = DOM.createDiv
DOM.setInnerHTML(elem, 'abc')
DOM.setElementAttribute(elem, 'title', 'tooltip')
DOM:appendChild(root, elem)

which looks much less like OO in my hence opinion.

I'm reading Michael Barbosas bachelor thesis about why companies adopt Ruby on Rails. To summarize the key adoption factors:

• Increase in productivity
• Open source nature
• Rapid prototyping
• Joy in development
• Cost
• Maintainability
• MVC
• Ruby programming language
• Convention over configuration

"Not" adoption factors were:

• Performance
• Scalability
• Hype (ha ha :)
• Learning curve
• Interoperability

I'm leaving out some other factors.

Sorry, I just don't buy it that Hype is not an adoption factor, especially for those coming from Java-land. Cost of course is an adoption factor, but usually it's seen as the-more-the-better ;-). Funny is that the Ruby programming language should have been an adoption factor for the companies. Of course they say it now, but before, they even didn't care about Ruby let alone that they didn't knew about Ruby, so I doubt this factor strongly.

You can also read that 81.5% use Linux for deployment. 10.3% goes to an unnamed operating system (which is of course FreeBSD ;-), and the rest is divided equally towards Windows and MacOSX.

Hm, you can also read that 66.4% use Apache and only 24.7% Lighttpd. From my feelings I'd say that it would be the other way round.

Well done!

So I finished work on RubyJS for now. It's mostly complete. Further changes are done on demand. Today I prototyped a first GWT port and thought about API issues. I'm not yet sure whether I want to port it. I'm more into taking a good idea as a basis and then do the stuff myself.

For example what in GWT is:

Button b = new Button("Hello", new ClickListener() {
  public void onClick(Widget sender) {
    Window.alert("...");
  }
});

becomes in RubyJS:

b = Button.new("Hello")
b.on_click do
  Window.alert("...")
end

Or:

b = Button.new("Hello").on_click do
  Window.alert("...")
end

I think there is much room for improvements.

The Javascript code for the core library is currently at ~9 KB. Compare this with the Prototype library which is at ~70 KB. And those 9 KB are without (space) optimizations turned on. So there is still much room left.

The last couple of hours I added the following features to RubyJS:

• Iterators
• Modules
• Whole program code-generation, including automatic dependency checking (that sounds complicated, but it's nothing more than introspecting which classes/modules to generate code for and in which order)

Again some statistics:

> cat *.rb | wc -l
    2450

> cat *.rb | grep -E -e '^[[:space:]]*#' | wc -l
     623

Again, the implementation of iterators in my new RubyJS version is much better than in the old version. At first one must understand how iterators works.

def m
  p(yield 1)
  p(yield 2)
end

m {|i| i+1}

The example above will print out 2 and 3. So the result of the iterator is the result of yield. Now there are two keywords in Ruby next and break that are useful in the context of iterators:

m {|i|
  next if i == 1
  i+1
}

m {|i|
  break if i == 1
  i+1
}

The first iterator using next will print out 3, whereas the second which uses break will not print out anything. So next is for iterators what is return for methods. And as such, as iterators are implemented as Javascript functions, it maps directly to return. The second using break is special. It breaks out of the calling method, so the second line in method m is not called anymore (and even the "p" call in the first line is not reached). This has to be implemented with an exception that has to be catched in the method m and converted into a return. Now that's pretty general as return is a statement in Javascript and sometimes we want it to be used inside an expression. This is only possible if we use an exception instead.

Now the equivalent converted to Javascript:

function m(block)
{
  try {
    this.p(block(1));
    this.p(block(2));
  } catch(e) {
    if (e === Return) return e.value;
    else throw(e);
  }     
}

m(function(i) { return i+1; });

m(function(i) { if (i==1) return nil; return i+1; });

m(function(i) { if (i==1) throw(Return(nil)); return i+1; });

Total number of lines:

> cat *.rb | wc -l
    1587

Total number of lines that are comments:

> cat *.rb | grep -E -e '^[[:space:]]*#' | wc -l
     394

So around 1/4 of the total number of lines are comments. That doesn't say anything about quality though. But it shows that it's not a quick hack :). It's code that is well described, especially the compiler, which needs most explanation. This is important that I can continue work on RubyJS after I come back from holidays in a few weeks :)

This popped up into my head just a few seconds ago while thinking about RubyJS. I'm a big fan of MLton, a whole program optimizing Standard ML compiler. I'm even listed there as user, as I wrote my Guuglehupf search engine using MLton.

So in RubyJS I also do whole program "optimization". Yes, really! Because I can take a look at the whole application that is going to be translated into Javascript. That's a huge benefit, as you can for example obfuscate a lot better than any other obfuscator that works on a file by file basis could ever do. As I wrote in my last blog entry you can also do unused method removal.

After 3 (in words "three") days of hard work and 1500 lines of code, the new completely rewritten version of RubyJS is nearing completion. I have learned a lot from the old version, and that's why I took the approach to rewrite everything from scratch. It has paid out. The code now is much cleaner and it's so much easier to extend (add optimizations). The generated code is much more dense and should execute much faster! This is thanks to direct method calls, instead of going an indirection over a rubyjs_send method. In the old version of RubyJS this indirection was needed to make method_missing working, but now I took a better approach, which lets you generate code like obj.method() instead of rubyjs_send(obj, method).

If you'd take a look at the implementation of rubyjs_send you'd see a lot of if statements, that check the type of the receiver and then dispatch to the method. While this makes it possible to use the special values "null" and "undefined" as receicer (they are special in Javascript as they don't have any attributes or prototype associated with them) it is just slow. I assume the new version to be around one order of magnitude faster in the end, as there is absolutely no overhead in calling a method, thanks to the introduction of "nil", which is a special object. This of course introduces new problems like initializing instance variables to "nil" before using them, but I figured this out.

There are still some things missing in the new version, but they are pretty straight forward to implement, especially as I've already implemented them in the old version. So it's mostly copy and paste and then modify a few bits.

The new version generates not only very condensed code, it also generates unreadable Javascript code. This protects your future client-side Ruby knowledge from being decompiled. You will be able to tell RubyJS to not generate tables that contain mappings between the Ruby method name (unobfuscated) and the method name used in the generated code. Those tables are required if you want to use send or methods etc. I think they are really not used that often, so it's worthwhile to have such an option, and it makes it nearly impossible to decompile it back to understandable code :). There are further options for generating faster code, omitting arity checks and so on.

What has changed as well was the method calling convention. In the old version I used:

function(self, args, block)

whereas I now use:

function(block, arg1, arg2)

All arguments are part of the function signature, except splat (catch-all) arguments, that collect all further arguments. In the new version they are constructed via Javascripts "arguments". That's by the way an ugly thing. I thought it would be a real Javascript Array but it isn't. So you have to loop and copy every element to a real Array. That's ugly, but it's the price you have to pay. The advantage is, that in the new version you no longer create an array per method call, which was a big overhead!

I really think given another day of hard work, RubyJS is (mostly) feature complete and I can start working on doing something "real" with it. Last night (this morning :) I took a look at Google Web Toolkit, especially at the API and class hierarchy. While it's Java, they did it right! It's not unusual to see over-designed Java APIs, but this for sure is not one of them. Of course in Ruby it will be much easier as you don't need all those interfaces. Porting the widgets code and DOM to Ruby I assume will take me another 1-2 days. I will not port it line-by-line, I'll write my own code, just copying most of the API, which I don't want to reinvent.

I'm also very interested in a comparison towards Prototype. This gives an infrastructure to write Javascript code in a more Ruby-like fashion. This is all part of the core libraries in RubyJS and is written in Ruby (with inline Javascript where needed). I'm interested in see how the generated code size compares to the size of Prototype AND how the execution speed compares. I think size-wise there will be no big difference, maybe RubyJS will generate even less code, due to variable name obfuscation. Performance-wise I assume RubyJS to be not more than twice as slow as Prototype. With optimizations turned on, I think there will be no big difference at all. Of course I'm hoping to see it run faster, but it's hard using a dynamically typed language like Ruby to generate more efficient code than hand-written code :).

In a thread about this new Javascript Rails port I read about the fact that Javascript doesn't have method_missing, which is abused to ad-absurdum in Rails. So this made it harder to proper port it to Javascript. In RubyJS you have method_missing thanks to the compilation step! You can really do a lot when compiling. I will also do method removal of unsed methods. Again thanks to compilation and static method call recording.

You might not believe it, but it's really true! The code given below is Ruby code ;-)

function(_g,_b,_c,_d){var _a,_e,_f,_h;_a=_h=nil;
if(this.$c===undefined)this.$c=nil;if(arguments.
length<2)throw('ArgumentError');if(_c===undefined)
_c=_b;if(_d===undefined)_d=2;_e=[];for(_f=4;_f<
arguments.length;_f++)_e.push(arguments[_f]);;
this.__invoke(_g,'$a',rubyjs_splat(_e));this.$b(
nil,1,2,3);this.$b(nil,2);_b.$a(nil);_b=_c=_d=1;
_h=1;_h=this.$c;this.$c=1;if((_f=this.$c.$d(nil,4),
_f!==false&&_f!==nil)){_b=_c}else{if((_f=true,_f!==
false&&_f!==nil)){}};_b=((_f=true,_f!==false&&_f!==
nil)?1:5);while((_f=true,_f!==false&&_f!==nil)){
_b=_b.$e(nil,1);_b=_b.$e(nil,2)};_a=3;return _a}

Well, not 100% true, but it is generated from Ruby code using RubyJS. The Ruby method is actually very complicated, using splat arguments, block arguments etc. I assume that the equivalent method in hand-written Javascript would be around the same size. I can easily omit arity checks and stuff like that to reduce size even more.

Now I solved the hardest part of my Ruby to Javascript Compiler. The implementation of Ruby's metaclass model. Last time, it broke my head, so I took another approach. This time I wanted to do it right, and after some hard thinking and a little grain of magic I did it. Object is a Class. Class is a subclass of Object. Class itself is a Class. Who doesn't get confused?!

This morning, I read this blog entry. It's about programming in a Ruby-style fashion, in Javascript. I think my RubyJS Ruby-to-Javascript compiler I'm currently greatly improving is a better solution though.

In Ruby really everything is an expression (as well as an object), while in Javascript there is a distinction between statements and expressions. For example "if" or "while" are statements. In Ruby you can write for example:

a = if a > 0 then 1 else 0 end

Which is equivalent in Javascript with:

a = a > 0 ? 1 : 0

The above is of course valid Ruby code, too. So if you want to write a Ruby to Javascript translator, use "?:" instead of "if". But how to map "while" to an expression in Javascript? Well, the easiest approach is using a function:

while(cond) { stmts; }

// maps to

(function(){ 
  while(cond) { stmts; }
})()

That makes it very easy to translate between Ruby and Javascript. The whole method body is one expression, so the Ruby code:

def meth()
  stmt1
  stmt2
  stmt3
end

maps to the following Javascript:

function() {
  return (stmt1, stmt2, stmt3);
}

where each statement (stmt1, stmt2, stmt3) is translated into an expression.

Of course functions have some overhead.

What I've not thought about is "return" and "throw", which are statements as well, so that the approach given above does not work for them.

Some guys from Google did it. When I read "they ported it line-by-line in 6 month" I really felt sorry with them. How can you do that? Why do you want to port something line-by-line? That means, shut up your mind and begin to translate, line by line! At least that's my feeling when I read line-by-line.

I think they could and should have invested those 6 months in something better, maybe they could have created a better version of Rails (in Javascript).

RubyJS is a Ruby to Javascript compiler I wrote a few month ago. One of it's main features is that you can use all meta-programming tricks you know from Ruby, as before it compiles the Ruby code to Javascript it executes the top-level code (require, attr_accessor and stuff like that). So you have the full power of Ruby.

Now I took a second thought on how to compile the code in a most efficient way. The old version used rubyjs_send() calls everywhere. It's the equivalent to Ruby's send. That's of course inefficient, at least a bit :). And of course the code grows. The only problem we face is to reliably implement method_missing as this does not exist in Javascript. One method is to check an attribute for null-ness before calling it. But that introduces a lot of code. The other approach, which I am going to use, is very simple. Every attribute (i.e. method/function) we call is defined in Javascript. The compiler knows all method calls, so it knows every method name that will ever be called. Now, we simply generate for each method name an attribute in the Object.prototype, from which all objects inherit. Every attribute is assigned the method_missing function. Subclasses can overwrite this of course. Simple eh? And efficient, hopefully. Don't know how prototypes are implemented in Javascript, hopefully thru a reference and not by copying every attribute.

It's just that I don't have so much time implementing all those changes and writing something like GWT. Or are there any sponsors? :)

New code by our Ruby genius Mauricio, which allows to call Ocaml code from Ruby.

This new book might be a nice read, especially because our Matz (Creator of Ruby) has written a chapter.

Sass looks nice.

Maruku is really a great markup language. It’s based on Markdown, but has a lot of more (useful) features. It will be part of my new blogging engine ;-).

Okay, I did a simple performance benchmark on my laptop.

My 31 lines Erlang webserver shown below using the undocumented option {packet, http} as described here is very fast!

-module(http).
-export([start/0]).

start() ->
  {ok, LSock} =
    gen_tcp:listen(8081, [binary, {packet, http}, {reuseaddr, true}, {active, false}, {backlog, 30}]),
  accept(LSock).

accept(LSock) ->
  {ok, Sock} = gen_tcp:accept(LSock),
  Pid = spawn(?MODULE, request(Sock)),
  accept(LSock).

request(Sock) ->
  {ok, {http_request, Method, Path, Version}} = gen_tcp:recv(Sock, 0),
  headers(Sock).

headers(Sock) ->
  case gen_tcp:recv(Sock, 0) of
      {ok, {http_header, _, _, _, _}} -> headers(Sock);
      {error, {http_error, _}} -> headers(Sock);
      {ok, http_eoh} -> body(Sock)
  end.

body(Sock) ->
  gen_tcp:send(Sock, [<<"HTTP/1.1 200 OK\r\n">>,
     ["Connection", ": ", "Keep-Alive", "\r\n",
      "Content-Length: 7\r\n",
      "Content-Type: text/plain\r\n"],
    <<"\r\n">>, <<"1234567">>]),
  request(Sock).

Save this code in a file http.erl. Then start up Erlang "erl" and type "c(http)." followed by "http:start().". After that, you can run apache-bench on it:

    ab -n 50000 -c 100 -k http://127.0.0.1:8081/test

The result on my very old laptop is 3600 requests/sec.

Now the same with Mongrel:

require 'rubygems'
require 'mongrel'

class TestHandler < Mongrel::HttpHandler
  def process(req, resp)
    resp.start(200) do |head, out|
      head["Content-Length"] = "7"
      out.write("1234567")
    end
  end
end

Mongrel::Configurator.new :host => '127.0.0.1' do
  listener :port => 8081 do
    uri "/test", :handler => TestHandler.new
    run
    join
  end
end

The same benchmark with Mongrel gives only around 500 requests/sec. Of course Mongrel does a little more than the simple Erlang web server shown above. And I think Mongrel doesn't employ connection keep alive. But if you see that lighttpd serves only at 1050 requests/sec, then it's impressive what the Erlang web server can handle. Okay, lighttpd has to load the file from disk (7 bytes), that's much less efficient than serving directly from memory.

I really dislike Javascript. That’s why I wrote RubyJS some time ago. With RubyJS you can basically translate your Ruby scripts to Javascript. Of course there are some minor problems due to their incompatible object models etc. For example Ruby has integers and floats, while Javascript only has numbers. Then Ruby has mutable strings while Javascript has not. Of course all this can be simulated by RubyJS easily but of course this comes not for free. For immutable strings there are two choices, either restrict RubyJS to use immutable operations on strings or box every Javascript string into a RubyString object. On the other hand, Ruby’s Arrays perfectly fit to Javascript arrays. Hashes do not!

Today I took a look at other languages. For example, would it be possible to convert Ocaml to Javascript? Haskell? For Haskell there even exists an approach using YHC (York Haskell Compiler). But, while Haskell is a very nice language, it lacks some OO support, that can be found in Ocaml. How about Erlang? It should be possible using the BEAM bytecode and translating it to Javascript. There is only one problem. Lists. There are not native in Javascript, but are used heavily in Erlang. After some time, I took a deeper look at Ocaml. It shouldn’t be that hard to translate Ocaml’s bytecode or it’s lambda-terms (-dlambda option) to Javascript. The advantage would be that you have a strongly typed langauge. No more ugly Javascript errors. But then, you don’t have open classes (or open modules). That’s a pain if you are coming from Ruby (or Javascript).

RubyJS is now working pretty well! The tests are now compared with the results of running the same test with the “real” Ruby interpreter.

Now I’m working on object dispatch. What I mainly want is to treat Javascript DOM nodes as RubyJS objects. So you can do:

def test
  e = `document.getElementById('id')`
  p e.class # => DOM::Element
end

I avoid to wrap the objects into my own objects and use singleton classes. This makes it much easier to pass those objects around between Javascript code and RubyJS code (no conversion needed). And it should save some memory. Only problem is that Konqueror doesn’t implement ‘instanceof’ for the DOM objects. But it’s possible to do the same in Konqueror via signature testing.

I lately came aross Google Web Toolkit. It’s amazing. Except, well, Java. If you write the server side in Java, perfect! If not, then you have a mixed feeling. But really, it’s a great toolkit, except that I’d like to be more free in how and what is generated, and I’ve not yet understood why they are generating all those files. Then I dislike Java’s package naming which leads to very deeply nested directories (which is okay if you use an IDE).

Then I saw Pyjamas aka py-gwt. Wow! Impressive! Much better to use Python instead of Java. And more important, it’s easier to modify parts of the generator!

I also read about converting Ruby to Javascript as described here.

Before I started my own :). You can get the code from here: http://ntecs.de/hg-projects/. Or use Mercurial to get your own repository:

hg clone static-http://ntecs.de/hg-projects/rubyjs/

So what is RubyJS?

In simple words: It converts Ruby to JS. You have to write some kind of “static” code, i.e. the converter introspects the classes as they are in-memory and then converts this version to Javascript. You cannot neither add/remove methods at runtime nor add classes at runtime. Method calls inside the same class (self as receiver) can be compile-time checked (not yet, but simple to add in one line). Inheritance and Mixins are supported. Compile-time constant lookup. Class methods as well (and inheritance of class methods). My brain still hurts from thinking about Ruby’s Metaclass model. Even the core classes are written in Ruby (with inline JavaScript). Method calls on immediate objects like true, false, nil and numbers are supported. Other objects can be boxed (numbers can be as well if you like). So it converts “1 + 2” into a method call, instead of assuming that JavaScript’s ”+” operator will work right!

Of course a lot is still missing, mainly compiling constrol structure, blocks, instance variables and of course the whole core library.

But it is really cool, because you can use all sort of Ruby’s meta-programming tricks, like “attr_accessor” for free. Or “require”. It executes as Ruby before it is being translated.

Update 1

Instance variables and super calls are now implemented.

Update 2

Compile time selection of method for calls within the same object (includes “super” calls).

Update 3

implemented iterators, while, break

In my last article I only tested sequential performance. Later I wrote a multi-threaded version and noticed that performance dropped drown a lot (from 3000 to 2000 page requests/second). Well, multiple threads are essential as the performed action usually takes longer and might block.

I made some basic non-professional performance benchmarks of a simple “Hello World” application between Mongrel (proxied through Lighttpd) against a SCGI-adapter via Lighttpd. This shows that the SCGI adapter is more than twice as fast. I also benchmarked AJP13 – the Apache JServ Protocol, a proprietary binary protocol. It is implemented in Lighttpd 1.5. There must be something very buddy, because it is around 100 times slower :). And it’s definitively not my implementation. There are two major differences between SCGI and AJP13. First, AJP13 reuses connections (but does no multiplexing like FastCGI). This should be a little faster. Second, it encodes common headers like “Content-length” as a single integer. But it has some very ugly inconsistencies for my taste. The SCGI implementation in Lighttpd 1.5 seems to be less production-ready as the same for Lighttpd 1.4. For example, when a SCGI connection is closed, lighttpd tells you about it, even if it is part of the SCGI protocol that the connection must be closed at the end of the transfer. One advantage of a structured response back to the server (like AJP13 does) is that the response parsing step in lighttpd could be easily ommited. And of course less data is sent across the wire. Less memory usage. Less connection-establishment overhead. So, once lighttpd 1.5 gets production-quality, I might write my own high-performance Ruby adapter.

If you want to learn more about Ruby’s internals, the Ruby Hacking Guide is what you are looking for. Well, the english translation is far from being complete. So feel free to read the japaneese original.

At the time when Rails RJS Templates were announced, I though: “Wow, that’s cool stuff”. Since then, I used it in several projects. Well, “used” means here, that in most cases I simply made an element showing an effect etc. I never really thought about why I am using RJS. Because it easily plays together with Rails link_to_remote helper methods. Maybe that’s the reason why RJS is being used. The link_to_remote already takes so many arguments, and in my experience all that stuff really makes a view looking pretty ugly. You see a lot of Ruby code inside your HTML, which makes it very hard to see the structure of the HTML (and the sense of the Ruby). But that’s another topic. Back to why “I consider RJS harmful”.

Actually, it’s so easy to do the same on the Javascript side. Just add an AJAX-onSuccess event and put there your code. So I really don’t see any advantage of being able to write in Ruby:

page.select('#content p').each do |element|
  element.hide
end

instead of the same in Javascript:

// jQuery
$('#content p').hide();

The Javascript is even much more terse. In general I think that it’s a bad idea to generate Javascript code on the Ruby-side. Better design your Javascript application well! And don’t mix up Ruby with Javascript.

In the web-application that I’m currently writing, I use JSON. This results in a cleaner separation between server and client side. You have to define clean interfaces and your Javascript (as well as your Ruby application) looks less like a hack.

Another advantage is that JSON should lead to less bytes being transfered over the wire, and it should be much easier (and faster) to generate. It’s also more secure, because you could get rid of Javascript’s eval by using a JSON parser. Keep in mind, JSON is pure data! You could even get rid of your XMLRPC or SOAP API’s completely and only use JSON-RPC. IMHO it’s much easier to implement (hey, I wrote my own JSON generator in just 1 minute using a few lines of code ;-) and much faster than any XML approach and you don’t have that ugly base64 encoding for binary strings in XMLRPC (or XML in general) where you explicitly have to know whether your string contains binary data or not.

Another thing: Someone had to implement that Javascript generating stuff (called RJS in Rails). I’ve not yet seen a use-case where this amount of work would be justified. Less code is always better!

With this patch you can add you C functions as methods which get passed an arbitrary argument upon each call. See what it can do in case of struct.c.

The Friday before my Iran trip, once again I had nothing to do, so I decided to make a competition with “our little ape” Sergey. Who is faster in implementing The Blocks Problem. That’s unfair I know, not only because I should be a more experienced programmer, but also because of Ruby vs. C#. Most problems occured due to the problem description being hard to read and understand. But then, after 1-2 hours I had a correct solution. It can be found here.

As I’ve nothing to do here in Yerevan, I started working on yet another O-R mapper for Ruby. The goal is to keep it small and understandable. Maybe below 200 lines in total (just including PostgreSQL support). And, all this belongs_to, has_many or has_one stuff is no longer needed. Those relations are detected automatically! If you have a database defined according to the naming schema, you can just start off, without writing any line of Ruby code.

Furthermore, only those attributes you set will be written to the database. In this case, either NULL or a DEFAULT value specified in SQL is used. And I’d like to implement the detection of modified attributes, so that only them are updated, not all!

This year we again have the European Ruby Conference in the beautiful city Munich. Read more.

One thing that most annoys me in using Rails are the generators. While wizards are quite different than generators (wizards generate code you don’t understand), they have one thing in common: They generate code, code that you have to maintain, code that contains lots of redundancy, duplicate code. If you know The Pragmatic Programmer read The Evils of Duplication. While code generation enables rapid prototyping and development, once this code has been generated, it has to be changed by hand. For very customized applications this is no problem, but for applications that are kind of “redundant”, you’d better use a different approach. I generally prefer an inheritance based template model, where you can overwrite parts which differ from a base “class”. But I also know, that it’s hard if not impossible to build reusable components, at least for the web. I tried it with Wee and would call it a failure (not Wee, but reusable components).

TupleSpaces are an easy and nice way to:

• distribute data across multiple nodes
• decouple the nodes (by using simple tuples)
• let nodes behave data-driven

Two downsides of Rinda, an implementation of TupleSpaces in Ruby:

• The get operation does not scale for many tuples!
• No transactions.

But, if you have too many tuples, you are probably using the TupleSpace in a wrong way. One way to avoid that, is to use credits which are consumed/produces whenever you put/get a tuple out of the space. But that again couples the two processes a bit more.

With no transactions, I mean, that it might happen, that one process takes a tuple out of the space, and later fails, so that the tuple is “lost”. It shouldn’t be too hard to implement transaction, e.g. by using sqlite as storage and then simply use database transactions. By using type definitions for tuples, one could then create on-the-fly the corresponding database table and insert the tuple there (that even works without type definition by only using the number of elements in a tuple as information). Using type information one could improve lookup speed.

TupleSpaces are easy to use. But they are very centralized. And as such, possibly inefficient and a single point of failure.

Shattered Ruby uses Ogre3d and lets you write 3D games easily in an object oriented manner in Ruby.

Yesterday night I hacked a persistent B-tree implementation in pure Ruby. Only key deletion is missing, insertion and find works well. Of course it’s so unoptimized, that smaller block-sizes are faster than larger. This is because on insertion of a key, I write the block back to disk and not keep a cache of dirty blocks in memory. Nevertheless I get a 100-1000 key insertion rate per second. It’s 300 lines of code, and well it’s a hack! I use an “internal” memory management system, meaning that I keep a pointer to the first free block in the header of the Btree file, and each free block points to the next free block. It’s the simplest way to go.

I’m thinking of rewriting it in C using mmap, so that the OS takes care of block caching/management. Don’t know whether this would be portable on Windows, but who cares :). I think it could be pretty fast. But of course you can’t implement your own buffer management algorithms.

B-Trees are really easy to implement! And after that, you really have a small database implemented. No really, you just have to add some meta-data, use multiple B-trees for further indices and of course, which seems to be a bit harder, implement transactions, and you have your own little database. Of course you don’t have query optimization etc. but as SQL is missing anyway, you do it “by hand”. Of course, what else is missing is concurrency. BUT, maybe it’s better to think about avoiding concurrency at all and gain scalability from locality? Let each process use it’s own database, containing non-overlapping entities. Let it operate on it’s owning entities, and from time to time balance (migrate) the number of entities across the databases.

What else is missing in my B-tree implementation? I want to be able to store duplicate keys. Currently only unique keys are allowed. This is fine for the primary key, but not for additional sorting indices. Hm, and how about multiple indices withint one and the same file? Should be easy!

Is to read the tutorial while at the same time trying it out in your browser. Really great stuff by why the lucky stiff.

This paper describes an idea (generating Javascript, SQL etc. from one source) I tried to implement about 10 years ago (only the idea was pretty much the same while their approach is really good!). I failed, because my approach ought to be too general. While it’s a very interesting approach, I think it will not become widely accepted. My dream has ever been to have one model from which you can automatically generate the SQL (including check constraints), the server-side validation code as well as client-side validation code. But they all use very different “languages” (SQL, Javascript, Ruby), so the only way is to agree upon a common description language and transform this into each language. Of course, that’s not that easy, and you loose expressiveness of the respective languages.

Several month ago, I worked a bit on Nemo, which is a Ruby implementation of Mewa based on Wee. There we tried to unify the description of storage and server-side validation constraints. In most cases they were mostly the same. But it’s really hard to get it right.

Aaron Patterson has taken over the development of my WWW::Mechanize library. With this library you can fetch and parse HTML pages, trigger form-submissions and stuff like that. In a single word – you can automate. It’s quite handy for simple stuff, but is limited if there’s lots of use of javascript in the pages.

It has now it’s own project page at rubyforge.

Read here and here about a Ruby on Rails setup which gets 2.5 million hits per day. That are around 30 requests per second on average. Quite a lot!

In the last few hours, I implemented a pure Ruby FTP server, in less than 200 lines. You can browse directories and download files. Passive mode is not yet working, just because I’m lazy to read the whole RFC in one go. But the FTP protocol is pretty simple.

Why would you want a pure Ruby FTP implementation? Well, there is a reason. Definitely! You can do lots of cool things with FTP, other than downloading files. What I’m planning is some kind of categorized, non-hierarchical filesystem. Another idea was to use it as a simple versioned filesystem. Everytime you store a file with the same name, it would record the changes. It’s so easy to implement, if you use Ruby. And FTP works really everywhere. I quickly tested it with Konqueror and on MacOSX. Just works! For now it’s using the filesystem as backend, but that will go away soon and become pluggable (if time permits).

What I’d like to implement is a poor-man’s image categorization program. Every folder is a category. You just drop in the images and they will be stored on the server under this category. You can assign an image to multiple categories, just by putting it in multiple folders. Subfolder are sub-categories. Damn easy. And it should be pretty fast when it runs on the same machine.

We’re giving a three days RubyOnRails Workshop in Munich, Germany, from Apr 12 – Apr 14. Language will be, guess what, German. Register soon :).

Some interesting stuff (from my personal point of view of course), is listed below:

• Librend: Realtime 3d apps in Ruby.
• Mongrel: A Fast As Hell Mostly Ruby Web Server
• Nitro/Og 0.27.0: Lots of good new stuff!

The same procedure as every year. This X-mas, matz has released Ruby 1.8.4. Thanks matz and merry X-mas to the whole Ruby community!

Liquid templates shine through their simplicity and readability. They do not contain executable code and are as such better suited for designers, but also for end users (no security risk).

A good summary by Mauricio Fernandez is available here.

Thanks to Sven C. Koehler, the EuRuKo05 videos are online here.

Looks really nice and very useful.

Link

Read more here.

Since a few hours Artima’s new e-zine Ruby Code & Style is online. Read the excellent articles!

http://www.pragmaticprogrammer.com/titles/ctrubyqt/

http://www.artima.com/rubycs/articles/pdf_writer.html

I can’t write all Wee code on my own. So here is a list of things that needs to be done:

• Make session-id cookies available to both Wee::Request and Wee::PagelessRequest, controlled by an application-wide setting.
• Finish the implementation of pretty-URLs (this rulez!). Thanks to Joao Pedrosa who is working on this.
• Implement a FilterDecoration and it’s subclass AuthorizationDecoration (that’s very easy, ~20 lines of code).
• Implement a Seaside-like root_for traversal, which allows each component in the tree to add stuff inside the <head> tag. This is triggered by the root-component from it’s render method.
• Refactor the Wee::Session class and it’s subclasses.
• Implement further adaptors, most useful would be a FastCGI adaptor. This would need to factor out the WEBrick dependent parts.
• Implement ResouceHandlers, which make it possible to generate images (or other external resources) from within your components.
• Use it in your projects and write tutorials ;-)

I’m trying to marry my child Wee with some of Rails features, mainly REST-like, pretty URLs. But before, I have to understand the differencies in how they work.

Disclaimer: This is my understanding of how Rails work. I might be wrong.

Rails

Rails model is very simple compared to that of Wee. The controller class and the method to invoke is extracted directly from the URL. For example:

/blog/show

would invoke BlogController#show. Now imaging the following URL:

/blog/show/4

This would invoke BlogController#show with @params = {‘id’ => 4}. I don’t know how Rails knows that 4 maps to the ‘id’ key, but that’s another story.

So you have these parts of a URL:

/controller/action/arguments

In Rails, there’s also no (conceptual) distinction between performing an action and rendering, as is the case in Wee. Both are inseparable from each other (in Rails).

Another difference to Wee is, that the controller classes are stateless, and instances of it might be reused (pooling). On the other hand this implicates that you can’t store information inside a controller instance across requests.

Wee

In Wee, we differenciate between an action phase, where callbacks are invoked and as such possibly modify the state of components, and a render phase, which ideally is side-effect free and whose sole purpose is to render the component to HTML (or whatever format you like).

Imagine you look at a simple Wee application which displays an HTML anchor tag. If you click on that anchor, you’ll trigger an action phase, which eventually will find the registered callback associated with the anchor you have clicked on and invokes it. At the end of the action phase, Wee will forward you to a new (automatically generated) URL, which when requested by your browser, will trigger the render phase. This in turn renders the whole component tree. "Why redirect", you might ask. Well, this simply avoids that the same callback will be invoked again if you hit your browsers reload button. Whenever you hit on reload, this will only trigger a render phase event.

Additionally, at the end of each action phase, a snapshot of the component tree is taken, so that you can go back to older states (called back-tracking). The information about which state of the component tree we refer to, is stored as page_id inside the URL. This page_id increases whenever an action phase is performed.

So, an URL in Wee basically consists of the following parts:

session_id page_id [callback_id]

where callback_id is optional. If it is given, the URL triggers an action phase. Otherwise, a render phase.

Compared to Rails controllers, Wee’s components are composites, meaning that they may contain sub-components which itself might contain sub-components and so on. And in Wee there’s only ever one root component, whereas in Rails there are usually multiple controllers. All this makes it nearly impossible to have REST-like URLs in Wee. Also due to the reason that a sub-component cannot be rendered on it’s own, whereas a controller in Rails builds a whole page. So it does not make sense to have URLs in Wee like:

/1.2.3
# == @root_component.children[1].children[2].children[3]

Marrying Wee and Rails aproaches

The concept of multiple top-level controllers can be easily added onto Wee by using the following RootComponent class:

# NOTE: not fully functional code!

class RootComponent < Wee::Component
  def initialize
    super()
    @controllers = {
      'blog' => add_child(BlogComponent.new),
      'list' => add_child(ListComponent.new)
    }
  end

  def render
    controller = # extract information from URL
    r.render @controllers[controller]
  end
end

This RootComponent merely acts as a dispatcher. It looks at the URL and extracts the desired controller out of it and then forwards to it.

Likewise, we could map the Rails action part (the "show" in "/blog/show"), to invoke render_xxx (e.g. render_show) of the controller-component (I use this term now, to distinguish it from a "regular" Wee component and to mark the similarity with a Rails controller):

  def render
    controller = # extract information from URL
    action = # extract Rails action part from URL

    component = @controllers[controller]

    component.with_renderer {
      component.send("render_" + action)
    }
  end

And even further, we could also extract the additional arguments of the URL for use inside the called render method. We could use a custom parse_arguments method here in each controller-component.

Now lets look at a simple example:

class BlogController < Wee::Component
  def parse_arguments(str)
    # for /blog/show/5, str would be "5"
    @params = ...   # e.g. {'id' => 5}
  end

  def render_show
    entry = BlogEntry.get(@params['id'])
    # render it
  end

  def render_list
    BlogEntry.find_all do |entry|
      # render it
    end
  end
end

What else would be needed is to tell which controller/action pair should be used. This could for example be specified each time when generating an anchor or form tag:

r.anchor.controller('blog/show/5').callback { .... }.with('show')

Note that this would first invoke the callback and then render ‘blog/show/5’. Of course you could ommit in this case the callback. But specifying the controller/action pair each time is tedious, as this would have to be done in each sub-component, too, which completely breaks the concept of a component. So the second approach is much better:

r.anchor.callback {
  ... do something
  request.controller = 'blog/show/5'
}

where the request.controller setting is carried inside the URL as long as you assign a new value to it.

Note that you can still use sub-components with this approach, but they would no longer be subject of REST-like URLs.

Read yourself here.

For all of you that experience slowliness with any type of application that uses sockets in Ruby (e.g. WEBrick), try this:

Socket.do_not_reverse_lookup = true

I’ve now enabled this by default for my Wee::WEBrickAdaptor class, which results in a huge difference in performance.

This night, I hacked up a Table and a Pager component for Wee. The Table component is 140 lines of code (including all HTML generation, nothing else except CSS styles are required), the Pager is 100 lines long.

Some days ago, I wrote an OgScaffolder component. It’s damn easy to use. Just pass an Og domain class to OgScaffolder.new, and what you get is a regular Wee component. Look at the screenshot below.

Below I’d like to show some Nemo screenshots, to show it’s potential. Note that the whole generic editor component (the first screenshot) is less than 200 lines very clean Ruby code! Thanks Kevin ;-)

Ploticus is a nice tool to generate diagrams. I wrote a nice wrapper in 143 lines of Ruby code, which makes it easy to use. See yourself:

require 'ploticus'

pl = Ploticus.new

pl.data = (0..10).map {|x| [x, x**2]}

pl.area {|a|
  a.title = "The title of this plot"
  a.xrange = [0,10]
  a.yrange = [0,100]
}

pl.xaxis {|a|
  a.stubs = "incremental"
  a.label = "x"
}

pl.yaxis {|a|
  a.stubs = "incremental 10"
  a.label = "x^2"
  a.grid = {:color => "blue"}
}

pl.lineplot {|a|
  a.xfield = 1    # use first column for x
  a.yfield = 2    # use second column for y
  a.linedetails = {:color => "red"}
}

puts pl.plot!

And the result is:

You can download my ruby-ploticus library from: ntecs.de/viewcvs/viewcvs/ruby-ploticus/

A more advanced example with four plots on a page can be found here. The diagram it generates is shown below:

I hacked up a little library which I use in some small projects to automate login into web-pages, extracting data from them and similar things. Web::Unit didn’t work for my purposes, and it was easier to reinvent the wheel than to hack Web::Unit (of course I first tried to hack it…).

It depends on narf-lib (>= 0.6.3), more exactly, it’s version of htmltools, and requires ruby-current (due to limitations in the net/http library of 1.8.2). If you want to try it out with Ruby 1.8.2, just get the net/*.rb files from ruby-current and use these instead.

Features

• Cookies (very preliminary, no expire etc.)
• automatic redirect
• Forms, Links

Be aware that it’s just a few-hour hack, to get my project done.

It’s still very generic and misses lots of usability methods. For example, I use find all the time, or have to pass the link that should be clicked to method click.

Example

Here’s a simple example, that logs into your rubyforge.org acount (pass your username and password as command-line argument).

require 'mechanize'

agent = WWW::Mechanize.new {|a| a.log = Logger.new(STDERR) }
page = agent.get('http://rubyforge.org/')
link = page.links.find {|l| l.node.text =~ /Log In/ }
page = agent.click(link)
form = page.forms[1]
form.fields.find {|f| f.name == 'form_loginname'}.value = ARGV[0]
form.fields.find {|f| f.name == 'form_pw'}.value = ARGV[1]
page = agent.submit(form, form.buttons.first)
puts page.body

Get it!

www.ntecs.de/viewcvs/viewcvs/Mechanize/

I wrote this little shell-script to easen the task of installing Ruby locally into $HOME/usr. It downloads the stable snapshot via FTP, unpacks, compiles and installs Ruby, then installs a recent version of RubyGems. Run it like this:

wget -O - http://ruby-installer.ntecs.de | sh

Or using fetch instead of wget:

fetch -o - http://ruby-installer.ntecs.de | sh

Be sure that you add $HOME/usr/bin to your PATH after installing it (put this into .shrc):

export PATH=$HOME/usr/bin:$PATH
export RUBYOPT=-rubygems

Or for csh (put this into .cshrc):

setenv PATH $HOME/usr/bin:$PATH
setenv RUBYOPT -rubygems

BTW, if you pass the script "current" as argument, it will install the current snapshotof Ruby.

Yesterday, I spent the time to create a web-based presentation maker (yes, in less than one day!). It’s a web application written in Wee and Ruby, with which you can edit slides (change, add, remove, reorder) and view a presentation inside a browser. Let’s look at a screenshot first (click on it to see it in full size):

The screenshot shows the edit mode. If you click on ‘Back’, then the left pane is hidden and you can proceed with presenting. If you click with the mouse on the title, the next overlay (or slide, if we’re on the last overlay) is shown. I want to add keyboard navigation, too, but I couldn’t get the JavaScript working. Note that the whole application is in no way JavaScript-driven, I only need JavaScript to trigger an "open(url)" call, so that Wee will display a new page. It would work completely without JavaScript if I would display anchor links for "next overlay" and so on.

Features

• The presentations file format uses Ruby’s documentation markup (RDoc), which is a very easy markup language (nearly plain text). So you even don’t need the web-based editor to create the presentation, you could also use your favourit editor (vim of course ;-)
• It’s easy to display colorized source code. Just insert:

  !!colorize:ruby
  def this_method
  end

• You can start applications from within the presentation by clicking on a link. Note that the application is started on the server-side, which is intended, as you’ll run the presentation maker application from your very own machine. Example (the execute button in the screenshot):

  !!exec:hidden
  cd $HOME/Dev/svn/public/ForSys && ruby test_lout.rb

• You can create Postscript and PDF output for printing. A Table of Content is included, and in the PDF version all links are clickable (of course all images are included, too).
• Of course you can use your own style-sheet.

Try it!

viewcvs.ntecs.de/PresentationMaker/

And don’t forget to read the README file.

I recently came across this thread on Slashdot, as it was mentioned on ruby-talk mailinglist. Someone has written a 15 lines P2P program in Python. I tried to port it to Ruby, but either it was too late this night (3am), or the Python’s list comprehensions confused me, so I got stucked with this little piece of code:

require 'openssl'
require 'xmlrpc/client'

def hmac(key, msg) OpenSSL::HMAC.hexdigest(OpenSSL::Digest::Digest.new('md5'), key, msg) end

passwd = "abc"
url = "http://127.0.0.1:3001"

p XMLRPC::Client.new2(url).call("f", hmac(passwd, url), 0, [])

It should query a tinyp2p.py server started with:

python tinyp2p.py abc server 127.0.0.1 3001

But it failed for some unknown reason. So I stopped at this point.

Today, Florian Gross posted a 6 lines P2P program in Ruby which is shown below:

# Server: ruby p2p.rb password server server-uri merge-servers
# Sample: ruby p2p.rb foobar server druby://localhost:1337 druby://foo.bar:1337
# Client: ruby p2p.rb password client server-uri download-pattern
# Sample: ruby p2p.rb foobar client druby://localhost:1337 *.rb
require'drb';F,D,C,P,M,U,*O=File,Class,Dir,*ARGV;def s(p)F.split(p[/[^|].*/])[-1
]end;def c(u);DRbObject.new((),u)end;def x(u)[P,u].hash;end;M=="client"&&c(U).f(
x(U)).each{|n|p,c=x(n),c(n);(c.f(p,O[0],0).map{|f|s f}-D["*"]).each{|f|F.open(f,
"w"){|o|o<<c.f(p,f,1)}}}||(DRb.start_service U,C.new{def f(c,a=[],t=2)c==x(U)&&(
t==0&&D[s(a)]||t==1&&F.read(s(a))||p(a))end;def y()(p(U)+p).each{|u|c(u).f(x(u),
p(U))rescue()};self;end;private;def p(x=[]);O.push(*x).uniq!;O;end}.new.y;sleep)

I think I would understand this code better than the Python code, but I will not try it ;-)

Preliminary documention of Wee is online. There is not much as of now, only the documentation of Wee::Component is nearly complete.

www.ntecs.de/viewcvs/viewcvs/*checkout*/Wee/trunk/doc/rdoc/index.html

I’m playing around with DragonFly's checkpointing support. Many thanks to Matt Dillon, who now made a syscall:

leaf.dragonflybsd.org/mailarchive/users/2004-11/msg00191.html

I wrote a Ruby wrapper around the sys_checkpoint syscall. You can find the sources here: www.ntecs.de/viewcvs/viewcvs/DragonFly/checkpoint/

The next step is to put this into the Wee sources, so that you can shutdown and reload Wee applications that use continuations. Of course, this only works on DragonFlyBSD. But I don’t think this is much of a problem…

I’m proud to announce the first public "release" of postgres-pr. It is a library to access PostgreSQL from Ruby without the need of any C library.

• You can use it only with newer 7.x databases that use wire-protocol 3.
• Lot’s of stuff is missing, only the wire-protocol is quite complete.

Quick Example:

> gem install postgres-pr
> irb -r rubygems

Then in the interactive Ruby interpreter type (replace DBNAME and DBUSER accordingly):

require 'postgres-pr/connection'
c = PostgresPR::Connection.new('DBNAME', 'DBUSER')
c.query('SELECT 1+2')                  # => [["3"]]

Have fun!

There’s a problem with objects registered for backtracking whose snapshots have a reference back to the registered object itself.

Let’s look at some scenarios. I’ll call the object to register "obj" and it’s snapshot "snap". I further assume that "snap" references "obj".

Scenario 1 (Good)

1. We register "obj" for backtracking.
2. "obj" goes out of scope -> it gets garbage collected.
3. We take a snapshot. No snapshot of "obj" is taken.

Here, no problem.

Scenario 2 (Worse)

1. We register "obj" for backtracking.
2. We take a snapshot "snap".
3. Snapshot "snap" goes out of scope.
4. Now "obj" could go out of scope, too.

Note that as long as a snapshot of "obj" exists, "obj" cannot go out of scope.

Scenario 3 (Worst)

1. We register "obj" for backtracking.
2. We take a snapshot "snap".
3. Before "snap" goes out of scope, we take another snapshot. This new snapshot also includes "obj".

As long as at least one snapshot exists that was taken after the initial "snap" snapshot, "obj" will not be garbage collected. It’s included in every further snapshot.

This might happen in Wee as we store a fixed number of snapshots (there’s always one available). So you have to take care to not introduce a reference back to the registered object from the snapshot.

Solutions

• Whenever possible, avoid to register objects in dynamically created components (e.g. MessageBox).
• The snapshot of a registered object which is taken by calling method take_snapshot of the objects’s class should never contain a reference back to the referenced object. If this happens, you’ll introduce memory-leaks (for the duration of the session).

As my dad is giving a guest lecture about the mathematics behind building society savings (Bausparen) at University of Karlsruhe, he wanted to give his students some tools they can play with easily.

Last evening, we then build the calculator shown in the picture below. It’s a portable web application using Ruby and my new Wee framework. I spent most time on generating the diagram (it’s done using gnuplot). The rest was straightforward, once we had the underlying calculations working.

The total code size is 360 lines of code (including HTML), of which 110 are used for the calculation and for generating the diagram. Lots of lines of code are currently spent in conversion and validation routines (german number format).

You can try it out online here: ntecs.de/wee/ehf/

Ruby can’t store continuations or blocks on disk! That was one reason why I was initially against using continuations (and blocks) in Wee. But now I have found some solutions to the problem.

Blocks

In the rendering-phase, you usually define some actions and input elements. This is currently done this way:

def inc; @cnt += 1 end
def dec; @cnt -= 1 end

def render_content_on(r)
  r.anchor.action(:inc).with('++')
  r.anchor.action(:dec).with('--')
end

While with blocks, you can write:

def render_content_on(r)
  r.anchor.action { @cnt += 1 }.with('++')
  r.anchor.action { @cnt -= 1 }.with('--')
end

When it comes to storing a session on disk, we simply mark those pages for which a block handler was specified, as "needing another render-pass before being able to respond to an action". So, in the rare case of needing to shutdown the computer, we don’t loose any information. It’s just that the user has to reload the page.

Continuations

We can rebuild block handlers, by executing their definition (render-phase). But that does not apply to continations. If you serialize a session with active continuations, Wee will replace the continations with marker objects, so that we can later detect that there was an active contination going on. If this happens, Wee will notify the user about this situation and will return to the calling component and call it’s method fallback_from_cc. If you want, you can still use old-style call behaviour by passing the name of the return method. Then you won’t loose any information.

Another idea is to use DragonFly's checkpointing support. This would be a 100% solution for the whole problem.

The picture shows page 10 of Curt Hibbs presentation about the Ruby on Rails web framework. Very good presentation!

The whole long night, my laptop was running and stressing a Wee application to find memory leaks in the current continuation implementation. I started with 25 stress processes, after 14 hours, there were only 16 of them still alive. The other 9 processes died for a yet unknown reason (probably a connection related problem). All in all, the processes created around 7.5 million requests, in 14 hours or 50000 seconds, and memory consumption did not exceed the magic 20 MB mark. This time I used Ruby from the stable branch.

This is an average of 150 requests per second or if we don’t count the redirects, which make the half of the requests, this is still an impressive performance of 75 whole page views per second (on a 1300 MHz Centrino laptop, running KDE with lot’s of open applications). Of course this time, the web application was a very simple one, it showed only a chain of dialog boxes.

Avi Bryant wrote some comments about Wee.

This morning, I added support for continuation-based programming to Wee. It’s now one step closer to Seaside2 or Borges. But I still support non-continuation-based apps. After changing half a dozen of lines, it was done, or so I thought. But then, I realized that you can’t call continuations across threads. After wrapping each session into it’s own thread and communicating with the application via two queues, and setting Thread.abort_on_exception=true to find some minor bugs, it really worked!

The rest of the day I fighted against some memory issues. To detect them, I was in the need for a simple stress testing application. Thus, I patched WebUnit, to support redirects and <br/> tags, then wrote a memory observer with GNUplot output and finally the stress tester.

I run the tests with all recent versions of Ruby, then I realized that the leak must be in my program and not in Ruby. More or less quickly I discovered that I had a reference back to the continuation in the called component, which resulted in the continuation not being freed, as the continuation has itself a reference to the component on the procedure stack. Assigning nil fixed the bug.

Below is the memory usage of Wee with continuations running under ruby-current as of today. I stressed it with 50 processes, i.e. 50 active sessions (= 50 Ruby threads), and a total of 100000 requests (in 10 minutes). It stayed below 22MB, that’s 500kB per session. It still slightly increases over the time, not sure whether this is GC related or not (fragmentation?). I’ll have to stress it over night ;-)

I finally got backtracking right. I fighted against a hard to find bug, and it turned out, that I was modifying a snapshot (e.g. arr = []; arr << 1). Method freeze is your friend! Now, backtracking is much more flexible. You can register individual objects for beeing backtracked (nothing new), but you can now have user-defined methods for taking or applying a snapshot of this object. For this purpose, you have to define take_snapshot and apply_snapshot, very similar to marshal_dump and marshal_load.

Backtracking decorations is now possible, too, but you still have to do this on your own. This is useful, so that "call/answer" calls can be undone correctly.

Decorations

A Delegate decoration is used to implement the call/answer mechanism. I’ll explain this by demonstrating a little example:

def render_content_on(r)
  r.anchor.action(:confirm_quit).with do
    r.text('Quit')
  end
end

def confirm_quit
  call MessageBox.new('Do you really want to quit?'), :quit
end

def quit(res)
  exit if res
end

If you click on the anchor named Quit, method confirm_quit is called. This then replaces the calling component with a message box (or at least, only the message box is shown and can handle actions, not the calling component), with two buttons OK and Cancel. If you now click on one of them, the calling component is restored and it’s method quit is invoked with the return code of the message box. How the message box can be implemented is shown below:

class MessageBox < Wee::Component
  def initialize(msg)
    @msg = msg
  end

  def ok
    answer true
  end

  def cancel
    answer false
  end

  def render_content_on(r)
    r.text @msg
    r.form do
      r.submit_button.value('OK').action(:ok)
      r.space
      r.submit_button.value('Cancel').action(:cancel)
    end
  end
end

In further versions of Wee, it will be possible to register an action and specify some default arguments. This would make the methods ok and cancel superfluous:

r.submit_button.value('OK').action(:answer, true)
r.space
r.submit_button.value('Cancel').action(:answer, false)

Uhm, 12 minutes later… it’s working ;-) Found by the way two other bugs.

So how is this call/answer mechanism working? Well, it really is only half a dozen lines of code:

class Wee::Component
  attr_accessor :caller   # who is calling us?

  # call another component
  def call(component, return_method=nil)
    component.caller = [self, return_method]
    add_decoration(Wee::Delegate.new(component))
  end

  # return from a call
  def answer(*args)
    component, return_method = self.caller
    component.remove_first_decoration
    component.send(return_method, *args) if return_method
  end
end

Get it! Try it!

So now, get Wee and try it out: viewcvs.ntecs.de/Wee/

Or play with it here.

Now it can handle 10 complete requests (which are actually two, one is a redirect) per second more, simply by changing the line

val.dup rescue val

into:

case val
when Fixnum, nil, true, false
  val
else
  val.dup rescue val
end

The second is up to 14 times faster for objects that can’t be cloned (Fixnum, nil, true, false). This results in an increase from 97 req/sec to 160 req/sec in the action-phase of Wee (for the example described in WeeFramework). Next, I will do component render caching (for leaf components).

After another complete rewrite of my initial prototype, and after trying out several different design choices (and asking Avi Bryant lots of questions), my web framework is now in a very good shape. And it’s got a name: Wee.

The core is now around 600 lines of code. The HTML related libraries put another 500 lines of code.

The trickiest thing I implemented was the StateRegistry, where you can register objects for being backtracked, and where you can take snapshots of all registered objects. All this is done with extensive use of weak-references and finalizers. The serialization part of the StateRegistry was tricky too, as you can’t simply serialize object-ids.

Features

Fully serializable

If you shut down the application, all state gets stored to disk. You can then reload the whole application from disk (possibly on another machine) with exactly the same state where you left off. Of course you could also store the sessions in a database table. Should be simple to implement any kind of load-balancing, as each session is completely independent, so that you could simply transfer sessions to another machine.

Backtracking

If you want, you can make the back-button of your browser work correctly together with your web-application. Imagine you have a simple counter application, which shows the current count and two links inc and dec with which you can increase or decrease the current count. Starting with an inital count of 0, you increase the counter up to 8, then click three times the back button of your browser (now displays 5) and finally decrease by one, then the counter really shows the expected 4, instead of 7 (as clicking the back button does usually not send a HTTP request, and the last state of your application was 8).

Only individual objects are backtracked, those that are registered, not the whole component. That’s the easiest (from an application programmers perspective) and most flexible way. And it’s fast and uses little memory.

You can decide yourself whether you want infinite backtracking or only backtracking up to n pages, with whatever replacement strategy you want, least recently used (LRU), least frequently used (LFU) etc.

Acceptable Performance and Memory Usage

Memory consumption stayed constant in all of my tests (with Ruby 1.8.x, not with 1.9!) by around 7 MB. On my Centrino 1300 MHz laptop, I can currently render 86 pages per second and can handle 97 action requests per second. That means it can handle 45 complete page views per second. The example consist of 20 counter components, with backtracking each of them, WEBrick as webserver (100% pure Ruby).

Reusable Components

Wee components are like widgets in a GUI. Once written, you can use them everywhere. They are completely independent and do not interfere with other components. Components encapsulate state, a view and actions. Of course you can use an external model or use templates for rendering.

Programmatic HTML Generation

You don’t have to leave Ruby to generate the HTML code. This is especially useful for highly dynamic components. But of course, you could also use templates.

Example

Below is the screenshot and sourcecode of a simple editable counter component that is encapsulated 10 times in the main component. You can try it out live here (it’s quite slow as it gets proxied through Apache as I’m too lazy to open my firewall).

require 'wee'

class Counter < Wee::Component
  def initialize(cnt)
    @cnt = cnt
    @show_edit_field = false
    session.register_object_for_backtracking(self)
  end

  def dec
    @cnt -= 1
  end

  def inc
    @cnt += 1
  end

  def render_content_on(r)
    r.form.action(:submit).with do
      r.anchor.action(:dec).with("--")
      r.space

      if @show_edit_field
        r.text_input.assign(:cnt=).value(@cnt).size(6)
      else
        r.anchor.action(:submit).with(@cnt)
      end

      r.space
      r.anchor.action(:inc).with("++")
    end
  end

  def submit
    @show_edit_field = !@show_edit_field
  end

  def cnt
    @cnt
  end

  def cnt=(val)
    @cnt = val.to_i if val =~ /^\d+$/
  end
end

class Main < Wee::Component
  def initialize
    @counters = (1..10).map {|i| Counter.new(i)}
    children.push(*@counters)
  end

  def render_content_on(r)
    r.page.title("Counter Test").with do
      @counters.each { |cnt| r.render(cnt)  }
    end
  end
end

Yesterday evening, after my brother demonstrated me his good-looking Python WebWare application, I decided to write my own Web Framework in Ruby. The idea of this project is in my head for a very long time, it started to get more concrete in a thread of mine on ruby-talk of subject WebShare (;-) but then I lost it again for some time.

Then Armin asked me if I would like to do a presentation about Ruby and the Web at EuRuKo04, but I couldn’t decide on a single powerful framework which rulez the world (in my eyes).

Why not Rails, IOWA or Borges?

I never got behind Rails and why it should be so good (luckily I’m not the only one). IOWA instead I know for serveral years and like it’s underlying model. What I don’t like in IOWA is listed below (this should not seen as criticism):

• The build-in templating stuff that you can’t get around,
• that IOWA components are in some ways different than pure Ruby classes,
• it’s code layout (yeah, in this way I’m an aesthet),
• the use of class-variables and too much magic like "inherited" or import instead of require,
• but most importantly, that I don’t understand all details of the code

The Borges framework is very nice and powerful, but has had some severe problems with memory leaks in the past, probably due to using continuations (I am not sure whether they were able to fix all these issues). Borges is a port of Seaside 2 from Smalltalk (IIRC, IOWA was a port of Seaside 1). Currently, there seems to be no further activity in improving Borges and make in en par with Seaside. Furthermore I heard on a mailing list that it is very slow due to continations or whatever else, and as such not suitable for larger applications. And you can’t migrate sessions between processes, at least not in Ruby, as Continations are not serializable.

My goals

In short, I wanted something that looks like Borges, but with the underlying model of IOWA. And of course which is developed in Ruby and for Ruby from the first second on.

I pretty exactly knew how a component in my WebFramework (put in better name) should look like. See the example section below.

So what actually were my goals?

1. Web applications and components should look very concise and clean. Components should in no way look different than usual classes in Ruby. You can use instance variables as you would use them in ordinary Ruby classes. Initialization is no magic and works as usual.
2. Make it easy to develop reusable components.
3. No configuration files or special setup required. Works out of the box.
4. To be able to migrate sessions and serialize them and the whole components tree to a file.
5. Be able to not just use this framework for HTML content generation, but also for example for emails. Imagine Python’s mailman. This could be implemented "easily" with this framework by using long-term sessions and a special email module that extracts the session and action id out of the email. Then, the only difference to the mailman web-module would be the content generation (plain text for email vs. html for web), the actions would stay the same. This is why I did not have chosen continuations for my framework, as they are not good for long-term sessions, because you loose them if you have to shutdown your server (and sessions for email usually endure several days).
6. Have a nice framework that I fully understand.

Example

Below, I implement a simple counter component and the main component that displays ten counters.

require 'web'

class MainPage < Web::Component
  def setup
    @body = "<html><body><p>hello world</p>%s</html>"
    @counter_list = (0..10).map {Counter.new(self)}
  end

  def render
    @body % (@counter_list.map {|c| c.render}.join("<br>"))
  end
end

class Counter < Web::Component
  def setup
    @counter = 0
  end

  def render
  %{ <a href="#{ action_url(:dec) }">--</a> #{ @counter }
     <a href="#{ action_url(:inc) }">++</a> }
  end

  def inc
    @counter += 1
  end

  def dec
    @counter -= 1
  end
end

Using templates

Of course you could use templates, too. I quickly implemented this in a few lines of very tricky Ruby code.

require 'rdoc/template'
module Templating
  def template
    file = caller.first.split(':').first
    file = file[0..(-1-File.extname(file).size)] + ".tmpl"
    tmpl = TemplatePage.new(File.read(file))
    proc {|values| tmpl.write_html_on('', values) }
  end
end

Now let’s look at how the templated version of the counter component would look like. You don’t even have to specify the name of the template.

class TemplatedCounter < Counter
  include Templating

  def render; template[
    'dec' => action_url(:dec),
    'inc' => action_url(:inc),
    'counter' => @counter.to_s]
  end
end

The template should be in the same directory as the component. Of course only if you use my approach shown here, which I like very much, as this allows to put each component in it’s own directory, together with it’s template, and you don’t need to setup template paths etc.

<a href="%dec%">--</a> %counter% <a href="%inc%">++</a>

Starting the application

All examples above just implemented the components. To start the application using Ruby’s WEBrick webserver, you have to write:

require 'web/webrick'
Web::Application.new {|app|
  app.name = 'Counter'
  app.session_factory {
    Web::Session.new(app) { |sess| sess.root_component = MainPage.new }
  }
}.start

Today, I wrote a little application that parses a ChangeLog file and constructs a RSS news feed. The source is available here.

The RSS feed for the Ruby ChangeLog is available here:

• www.ntecs.de/ruby-devel-cvs.rss?cnt=20 (20 entries)
• www.ntecs.de/ruby-devel-cvs.rss?full (all entries)

Random Rollbacks

From time to time our EuRuKo04 Wiki experienced strange and very annoying random rollbacks. After I installed the newest version of Instiki from CVS, which claims to fix this bug, I thought everything is now fine. But only a few days later matju told us that again a random rollback has happened! Now I decided to put an end to these annoying rollbacks and removed the rollback method completely from the controller. Rollbacks, either random or by intention, can now no more happen.

But why was the solution in the CVS of Instiki not enough to prevent random rollbacks? At first let me say that the rollback action is triggered by a simple HTTP GET request. When a web-spider comes along, it usually tries to follow this URL and voila, a random rollback has happened! The CVS version now tried to fix this by first showing a JavaScript message box which forwards to the rollback URL on submission. This means that the URL of the rollback action is no longer inside the HTML, but inside the JavaScript section (which itself is inside the HTML ;-). Two reasons why this could be not enough:

1. Maybe some web-spiders even crawl the JavaScript sources. I don’t believe this is the reason, because it’s very hard to distinguish URLs in JavaScript from simple strings especially as the rollback URL is a relative one.
2. Web-spiders look out for modifications of pages they’ve visited in the past. So in our case, they’ve the rollback URL in their database and visit this page regularily to see if it has changed.

Conclusion: Don’t do actions via HTTP GET unless you disallow web-spiders or you use session URLs which have a limited time span.

45 Gigabytes

This morning my bother called me by phone: "Hey Mike, what has happended with our server? I can’t read my emails!". In the same breath he argued whether the reason could be a filled up harddisk. Quickly I logged in to the server and followed his suggestion and displayed the harddisk capacity via df. Boom! What has happended? My /data and /backup partions had no space left. Then I did a du -ch inside my home directory (which resides on the /data partion) and voila, the ~/instiki/storage/2500 directory was nearly 45 GB in size. I looked into it and saw many many snapshot files. So many, that a rm *.snapshot did not succeed due to too many arguments on the command line. Ruby does not have these limitations, so the one-liner ruby -rfileutils -e ‘Dir.glob("*.snapshot") {|i| File.delete i}’ did the job.

Conclusion: Madeleine, the main-memory database used by Instiki to store the Wiki, should definitly delete old snapshot files. It might be that a newer version of Madeleine has this already implemented.

In {Part I}[PythonDecorators] of the article, I hacked the Ruby interpreter and modified one single line to implement Python Decorators in Ruby. Now I’ll show some more examples in both Python and Ruby.

The examples are taken from the article Decorate this.

Wrapping

Let me show the Python example first.

def wrapwith(obj):
    def decorator(f):
        def _wrapper(*args, **kwargs):
          print "##", obj
          return f(*args, **kwargs)
        return _wrapper
    return decorator

@wrapwith(42)
def f(x): return x*2

# let's use it
print f(4)

Now the same example in Ruby (with my single-line patch applied):

def wrapwith(obj, method_id)
  old_method = method(method_id)
  new_method = proc {|*args|
    print "##", obj, "\n"
    old_method.call(*args)
  }
  self.class.send(:define_method, method_id, new_method)
end

wrapwith 42,
def f(x) x*2 end

# let's use it
print f(4)

For me, the Python example is harder to understand, as three nested function definitions are involved. Now, the Ruby example is just fine, but in Ruby 2.0 we can even do better:

def f(x) x*2 end

def wrap:f(x)
  print "##", 42, "\n"
  super # calls "f"
end

print f(4)

For all examples, the output will be:

## 42
8

Prototype-based OO

Below, I only show how to apply prototype-based OO with Decorators in Python, not how to implement it. For the complete Python example see Decorate this.

class Foo:
    def __init__(self):
        self.x = 42

foo = Foo()

@addto(foo)
def print_x(self):
    print self.x

foo.print_x()   # => 42

The same in Ruby:

class Foo
  def initialize
    @x = 42
  end
end

foo = Foo.new

def foo.print_x
  print @x
end

foo.print_x     # => 42

Note that the Ruby example is complete whereas in the Python example the implementation of @addto is missing. But what if you want to add a whole bunch of methods to an object? In Ruby it’s as easy:

foo = Foo.new

class << foo
  def m1
    1
  end

  def m2
    2
  end
end

foo.m1   # => 1
foo.m2   # => 2

start summary::

Is Ruby human-understandable? Understandable by humans with absolutely no experience in computer programming? How can we as experienced programmers at all answer such a question? Well, we can’t! The only way to determine whether a language is easily understood by novices is to actually ask them.

end summary::

To find an answer to this question I performed some tests with my beloved girlfriend. She’s an absolute novice in this area as she’d never seen any source code of a computer program before. I gave her a page of paper with examples written on it and told her to write down what would be the output for each example. Then I left the room to prevent any communication during the test.

The results were promising! She mastered 4 of 7 tests without my help. Tests two and seven were obvious after introducing := for assignment, and = for equality. And the failure of test 6 had more to do with the English word "select" than with Ruby (keep in mind that English is not our native tongue). Wow, she even implicitly used lazy evaluation ;-)

The Tests

Test 1

print "hello world"

She passed this test easily.

Test 2

result = (3*10) + 5
print result

She pointed out a problem that I wasn’t aware of. From her point of view, the output should be "(3*10) + 5" and not "35". I discussed this issue with her and found out that it would be easier for her to understand if the "=" would be a ":=". Once I changed the tests according to this problem, she was able to solve them.

Test 3

5.times {
  print "hallo welt"
}

No problems here!

Test 4

5.downto(1) { |counter|
  print counter
}

Again, no problems!

Test 5

elements = ["Hydrogen", "Oxygen", "Helium", "Ferrite"]
elements.each { |element|
  print element
}

She even mastered this test without problems.

Test 6

one_to_ten = [1,2,3,4,5,6,7,8,9,10]
even = one_to_ten.select { |i| i.even? }

print even

Once I told her that select translates to the German word auswählen (choose) she mastered this test easily.

Test 6b

Same as test 5, plus the following line:

print elements.select { |element| element == 'Ferrite' }

I created this test after test 6 failed. She mastered this one easily and thereafter was able to solve test 6.

Test 7

counter = 1

loop {
  print counter
  counter = counter + 1
}

This test raised the same issue as test 2: Assignment vs. Equality. Her result was "1 + 1 + 1 + …", which is not all that wrong. Of course it’s wrong in Ruby, but remember Haskell’s lazy evaluation:

{- create an infinite list of ones -}
ones = 1 : ones

She mastered the test after I introduced ":=" for assignment instead of "=".

Test 7b

counter = 1

loop {
  break if counter > 10
  print counter
  counter = counter + 1
}

I created this one after she solved test 7 successfully. Of course it was a mere child’s play for her ;-)

Discussing Loops

I took the opportunity and asked her about loops.

loop {
  ...
}

loop {
  ...
  break if cond
  ...
}

while 1 {
  ...
}

while true {
  ...
}

while cond {
   ...
}

repeat
  ...
until cond

She easily understood the loops that use the loop keyword, but not the others. That’s exactly what I prefer for most of the loops I write in my programs. And IMO they are much more readable and less error prone than while or repeat until loops.

I remember a statement of a mathematics professor at University of Tuebingen who told us that the human mind is not good in understanding negation (the "not" operator from logics). And at least for my mind, he’s correct. Whenever I can, I prefer positive logic over negative:

# this one is easy to understand (for me)
loop {
  break if cond
}

# tell me when this loop aborts!
while not cond {
  ...
}

And whenever I see something like "while 1", especially in so called very high level languages, I ask myself whether they got caught in C forever.

Finally, I asked her about for loops:

for i in 1..10 do
  ...
end

As she hasn’t studied mathematics, she had problems understanding the for … in notation. Luckily, Ruby supports customized loops like "1.upto(10)" which reads easily even for absolute beginners.

I read about Python Decorators on Lambda-the-Ultimate. They will be part of Python 2.4. But, was it really worth to add special syntax for something that can be done without problems in a different way? Isn’t Python breaking it’s own principle "There’s only one way to do it"? Will Python become a second Perl? Which so-called "feature" will be added next? Don’t get me wrong, Python is a valuable language!

The Ruby language instead is pretty constant since it’s very beginnings (version 1.0 of December 1996). There are no visible changes in the language (but in the library!) except the introduction of class-variables in 1.4 (or was it 1.6?).

Decorators in Python

Instead of:

def f(...)
  ...
f = synchronize(f)

you can now write:

@synchronize
def f(...)
  ...

The part after the at-sign is simply a function (synchronize is our case).

Decorators in Ruby

def f
  ...
end
synchronize :f

That’s pretty the same as the first Python example shown above. But can we simulate the second Python example in Ruby? Yes we can! We could do this in pure Ruby, but that would be a bit more advanced. Therefore I’ll show you how to do it with a minor change to the Ruby interpreter. Minor means that the change does not break any existing code and that only one line needs to be changed:

--- eval.c    2 Aug 2004 08:52:53 -0000  1.686
+++ eval.c    16 Aug 2004 12:52:28 -0000
@@ -3701,7 +3701,7 @@
                rb_add_method(rb_singleton_class(ruby_class),
                              node->nd_mid, defn, NOEX_PUBLIC);
            }
-         result = Qnil;
+         result = ID2SYM(node->nd_mid);
        }
        break;

With this patch applied, a method definition will return the method name as symbol. Let’s try this:

x =
def f
end
p x # => :f

Now we can "simulate" the second Python example in Ruby:

synchronize def f
  ...
end

# or with the decorator on a separate line

synchronize \
def f
  ...
end

Note that the backslash after the synchronize in the second part of the example is neccessary! This tells Ruby that the argument is on the next line.

And we can even cascade "decorators":

private memoize synchronize \
def f
  ...
end

Of course this only works if each involved "decorator" method passes the method-id through.

I will show you in this article how to increase performance of an algorithm by a factor of 10^2086, simply by adding two single lines while not changing any existing code! Yeah, 10^2086 that’s a one with 2086 zeros.

Here is the "un-optimized" algorithm, the well known Fibonacci function:

def fib(n)
  if (0..1).include? n
    1
  else
    fib(n-2) + fib(n-1)
  end
end

Okay, it’s fast enough for small numbers, but how long would it take to calculate fib of 10000? Well, it would take so much longer than the life of the whole universe that I can’t even describe how much longer it would take (billions of billions of billions of … times longer).

To make the algorithm above run 10^2086 times faster than before, we add the following two lines:

# paste here the algorithm from above
require 'memoize'
memoize 'fib'

Now let’s calculate fib(10_000):

p fib(10_000)

And after approx. 3 seconds, the result is displayed on the screen:

5443837311356528133873426099375038013538918455469596702624771584120858
2865622349017083051547938960541173822675978026317384359584751116241439
... 26 lines snipped ...
0765629245998136251652347093963049734046445106365304163630823669242257
761468288461791843224793434406079917883360676846711185597501

Wow that’s a bit longish number ;-)

For those who are interested about the performance characteristics, it’s a change from O(Fib(n)) to O(n).

Implementing Memoize

As I couldn’t find Robert Feldt’s memoize module for Ruby anymore, I decided to write my own. It took me no more than 10 minutes:

$MEMOIZE_CACHE = Hash.new

def memoize(*methods)
  methods.each do |meth|
    mc = $MEMOIZE_CACHE[meth] = Hash.new
    old = method(meth)
    new = proc {|*args|
      if mc.has_key? args
        mc[args]
      else
        mc[args] = old.call(*args)
      end
    }
    self.class.send(:define_method, meth, new)
  end
end

Wow! That really impressed me! A vim clone written in pure Ruby called Ruvi. It’s incredible fast. My feeling is, that it’s even much more responsive than vim itself. And Ruby syntax high-lighting is very good as you can see on the image below (click on it to see it full-sized), even auto-indent is supported.

start summary:: In diesem Artikel versuche ich mal die im Titel genannten Begriffe zu erklären. Diese Begriffe bilden die Grundlage nicht nur der Objektorientierten Programmierung, sondern finden vielmehr auch Anwendung in anderen Disziplinen wie etwa der Biologie, und dies schon Jahrhunderte bevor die Entwicklung der Rechenmaschinen begann (ja, ich spreche von Computern ;-). end summary::

Im folgenden sind die einzelnen Begriffe erklärt:

Klasse (Schema/Schablone)

Eine Klasse beschreibt das Verhalten und die Eigenschaften all seiner Objekte. Zum Beispiel, dass jedes Haus ein Dach besitzt oder die Eigenschaft das Häuser in der Regel durch ihre Anzahl von Stockwerken beschrieben werden können.

Eine Klasse beschreibt dagegen nicht aus wievielen Stockwerken nun ein bestimmtes Haus besteht (das Nachbarhaus z.B.). Ein ganz bestimmtes Haus ist nämlich schon eine Ausprägung dieser Klasse "Haus", ein Objekt also. Man sollte hierbei gut aufpassen nicht die Klasse "Haus" mit dem Objekt "mein Haus" zu verwechseln.

Klassen sind abstrakte Beschreibungen der Objekte. Genauso gut könnte man jedes einzelne Objekt aufs neue beschreiben. Da viele Objekte sich jedoch sehr ähnlich sind ("jedes Haus hat ein Dach"), fasst man ähnliche Objekte in Klassen zusammen und beschreibt diese nur einmal.

Objekte (Ausprägung)

Objekte hingegen sind, anders als Klassen, nicht abstrakt, sie existieren wirklich! Objekte einer Klasse besitzen diesselben Verhaltensweisen (Methoden), jedoch hat jedes Objekt seine eigenen Attributwerte die es von den anderen Objekten unterscheidet. So ist z.B. jeder Mensch in Deutschland eindeutig durch die konkreten Werte von Geburtsdatum, Geburtsort usw. identifizierbar. Dies sind seine Attributwerte. Klassen beschreiben nur, welche Attribute Objekte besitzen können, nicht jedoch deren aktuelle Werte (z.B. Berlin als Geburtsort).

Attribute (Zustand)

Objekte derselben Klasse unterscheiden sich oft nur in ihren Attributwerten. Jedes Objekt hat seine eigenen Attributwerte, die jedoch mit denen eines anderen Objektes durchaus übereinstimen können. So gibt es viele Objekte der Klasse Mensch (man sagt auch "vom Typ Mensch"), die in Berlin (Attributwert des Attributs Geburtsort) geboren sind.

Ein weiteres wichtiges Attribut von Lebewesen ist der sogenannte "genetische Fingerabdruck", oder kurz, die DNS. Wieder muss unterschieden werden zwischen der Beschreibung, dass jedes Lebewesen einen genetischen Fingerabdruck besitzt (das geschieht in der Klasse), und des konkreten Wertes, der wiederum Bestandteil des zugehörigen Objektes ist.

Methoden (Verhalten)

Methoden beschreiben wie sich Objekte verhalten (können). Dabei wird dieses Verhalten für alle Objekte einer Klasse beschrieben, ist jedoch abhängig von den Attributwerten des konkreten Objektes.

Methoden beschreiben meist gewisse Aktionen. So hat zum Beispiel die Aktion geh_nach_hause eines Objektes der Klasse Mensch zur Folge, das dieser nach Hause geht. Logisch ist, das dieses Verhalten sehr stark von dem Attributwert Wohnort des Objektes abhängt. Zwei Menschen verhalten sich demnach anders, wenn der eine in Bonn und der andere in Berlin wohnt und sie aufgefordert werden nach Hause zu gehen. Anstatt von "Aufforderung" spricht man hier auch vom "Aufrufen einer Methode eines Objektes". In unserem Fall wird die Methode geh_nach_hause beider Menschen aufgerufen.

Prinzipien der Objektorientierten Programmierung

Kapselung

Was ich noch erwähnen sollte - und einige werden mir jetzt sicherlich widersprechen - ist, dass die Werte der Attribute nur dem Objekt selbst bekannt sind, solange es diese nicht preisgibt. Preisgeben kann es diese nur, wenn dafür eine spezielle Methode definiert wird. Das leuchtet ein wenn man sich einmal in Gedanken vorstellt wie die Polizei versucht die Identität einer Wasserleiche herauszufinden. Die Polizei kann ja auch nicht einfach die Werte der Attribute "Name" oder "Wohnort" abfragen. Selbst wenn die "Leiche" noch leben würde, hätte sie (die Leiche) die volle Kontrolle darüber, ob sie nun der Polizei verrät wo sie wohnt und wie sie heisst oder ob sie dies nicht tut oder gar lügt. Da eine Leiche jedoch in aller Regel tot ist (so die Definition ;-)), fällt diese Möglichkeit eh weg.

Also nochmal zum mitschreiben: Die einzigste Möglichkeit mit Objekten zu interagieren besteht darin, dessen Methoden aufzurufen, also dem Objekt mitzuteilen was wir von ihm wollen. Wir kommunizieren im Prinzip mit dem Objekt, daher sprechen wir auch häufig vom "Senden einer Nachricht" an ein Objekt anstelle von "Methodenaufruf".

Dieses Prinzip, also dass die Attributwerte nur dem Objekt bekannt sind, führt auch sofort zu einem der Grundpfeiler der Objektorientierten Programmierung (OOP), nämlich der Kapselung. Der Zustand (also die Attributwerte) eines Objektes kann ausschliesslich durch Methoden verändert bzw. abgefragt werden. Leider findet dieses ungemein wichtige Prinzip in vielen Programmiersprachen nicht die Bedeutung die es verdient. Anders in Ruby, Smalltalk, Eiffel und einigen anderen Sprachen.

Ach wie wär’s schön wenn man den Zustand von Objekten direkt abfragen könnte. Das würde nämlich bedeuten, das es keine Lügen und keine Missverständnisse mehr gäbe. Aber leider gäbs dann auch keine Kommunikation mehr!

Vererbung

Ein weiterer wichtiger Aspekt der OOP neben der Kapselung ist die Vererbung. Vererbung ist sehr leicht zu erklären und zu verstehen wie ich meine. Klassen können von anderen Klassen erben. So könnten wir zum Beispiel die Klasse Katze von der Klasse Lebewesen erben lassen um auszudrücken, dass jede Katze ein Lebewesen ist und somit diesselben Eigenschaften wie ein Lebenwesen besitzt. Das spart uns ne Menge Zeit, da wir ja auch noch Hunde und Fische definieren wollen, die allesamt von der Klasse Lebenwesen erben. Das ist genauso wie in der Biologie! Natürlich können wir dann noch zusätzliche Eigenschaften für Katzen definieren und sogar die von der Klasse Lebewesen geerbten Eigenschaften durch eigene überschreiben (letzteres ist das dritte wichtige Prinzip der OOP, die Polymorphie, auf die ich aber hier nicht weiter eingehen werde).

Polymorphie

Nein, das erklär ich jetzt nicht! Polymorphie (griechisch für "vieldeutig") ist selbst ein sehr vieldeutiger Begriff. So gibt es u.a. statische und dynamische Polymorphie. Aber das interessiert uns hier nicht weiter.

Der Praktische Teil

Wir wissen nun also ungefähr was Klassen und Objekte sind und kennen auch deren Eigenschaften wie Attribute und Methoden. Nun wollen wir aber, um es besser verstehen zu können, das ganze anhand einem Beispiel vertiefen, und zwar indem wir die Tasten schwingen und gleich richtig in die Objektorientierte Programmierung (OOP) einsteigen. Und was wäre dafür besser geeignet als Ruby? Nein wirklich, schreibt diese Beispiele von mir aus in einer anderen Sprache und gebt es einem Neuling und lasst ihn entscheiden…

Und Los gehts!

Wir bauen uns ein Lebewesen… Ein Lebewesen soll ein Attribut haben, und zwar seine DNS.

class Lebewesen
  def init(dns)
    @dns = dns
  end
end

Wir haben also oben die Klasse Lebewesen definiert, mit einer Methode init. Diese Methode dient dazu, den Zustand des Objektes initial, also quasi gleich nach der "Geburt" zu definieren, da andernfalls das Objekt mit keinem oder nur mit einem undefinierten Zustand leben muss, was wir schliesslich nicht wollen.

In unserem Fall bezeichnet @dns das Attribut "DNS". Attribute, also die Variablen in denen der Attributwert gespeichert wird, beginnen in Ruby immer mit einem at-Zeichen (@). Damit kann man sie nicht mit lokalen Variablen verwechseln.

Wird nun die init Methode aufgerufen, so weisen wir der Zustandsvariablen @dns den Wert zu, den wir der Methode beim Aufruf als Parameter übergeben haben, hier ist das der Wert des Parameters dns.

Um nun ein neues Lebewesen Objekt zu erzeugen dessen Zustand initial leer ist, rufen wir die allocate Methode der Klasse Lebewesen auf. Das geht in Ruby, da dort Klassen in Wirklichkeit auch Objekte sind, Klassen-Objekte eben ;-). Und wen ich jetzt total verwirrt habe, der denke sich einfach eine spezielle Operation, mit der man von einer Klasse ein "frisches", d.h. leeres Objekt erzeugen kann (z.B. den new Operator aus C++ oder Konstruktoren).

# wir erzeugen ein neues "frisches" Lebewesen
neuesLebewesen = Lebewesen.allocate

Nun haben wir ein frisches Lebewesen erschaffen und haben es in der Variablen neuesLebewesen vorerst zwischengespeichert. Jetzt wollen wir ihm noch seine DNS zuweisen. Dazu müssen wir seinen Zustand, genauer seine Zustandsvariable @dns, ändern. Da wir aber selbstverständlich nicht direkt auf den Zustand eines Objektes zugreifen können (Gott sei Dank! Wer will schon gerne das sich seine DNS so einfach ändern lässt), müssen wir das Objekt höftlich fragen "es solle doch seine DNS auf den von uns genannten Wert setzen". Klar, dafür ist ja auch die init Methode gedacht:

# und initialisieren seinen internen Zustand
neuesLebewesen.init("AA-B-AC-DA")

Wenn du das allocate oben nicht verstanden hast, egal! Es ist eine vordefinierte Methode, die jedes Klassenobjekt automatisch "erbt". Lebewesen oben im Quellcode ist nämlich das Objekt das die Klasse "Lebewesen" beschreibt. Und auf Objekte können wir bekanntermassen Methoden anwenden, nicht auf Klassen, da Klassen abstrakte Dinge sind die so nicht existieren (ausser in unseren Gedanken oder in Form von Objekten die wiederrum Klassen beschreiben). Auf keinen Fall jetzt durcheinander kommen, das ist alles nicht soo wichtig!

Schön wäre es doch nun, ein Objekt bei dessen Erschaffung gleich mit seinem Zustand initialisieren zu können. Das ist sogar noch einfacher in Ruby und geht so:

class Lebewesen
  def initialize(dns)
    @dns = dns
  end
end

# neues Lebewesen erzeugen und initialisieren
neuesLebewesen = Lebewesen.new("AA-B-AC-DA")

Wieder ist new eine bereits vordefinierte Methode in Ruby, die im Prinzip allocate aufruft um ein neues Objekt zu erzeugen. Danach ruft sie die initialize Methode des neu erzeugten Objektes auf und gibt ihr all die Parameter mit die sie selbst bekommen hat. Es ist also ungefähr äquivalent zu:

l = Lebewesen.allocate
l.initialize("AA-B-AC-DA")

Nur funktioniert das nicht ganz, da initialize standardmässig von Ruby als private deklariert wird, d.h. sie kann nicht von ausserhalb des Objektes aufgerufen werden, ähnlich wie die Attribute nach aussen hin abgeschottet sind. Aber das nur am Rande.

Warum gerade initialize? Nun, das ist einfach Konvention. new ruft eben diese Methode auf! Aber wie wir gesehen haben, sind wir nicht gezwungen diese Methode zu definieren.

Klonen Leichtgemacht

Tja, wer hätte es geahnt, aber wir in Ruby können schon über 10 Jahre lang Lebewesen klonen, und zwar ohne das dabei Frankensteins entstehen ;-) Oder etwa doch?

frank = Lebewesen.new('AA-BB-CC')
frankenstein = frank.clone

p frank.id == frankenstein.id  # => false
p frank == frankenstein        # => true

Das p in den letzen zwei Zeilen ist eine Kurzform für print. Es stellt sich heraus, das wir zwei Objekte erzeugt haben (sie haben unterschiedliche ids), die jedoch ansonsten identisch sind.

Dieser Einschub war aber eher spasseshalber gedacht.

Es regnet Hunde und Katzen

Soweit so gut. Jetzt wollen wir uns paar kleine Kätzchen definieren. Eine Katze hat neben der DNS noch einen Namen.

class Katze < Lebewesen
  def initialize(dns, name)
    super(dns)
    @name = name
  end

  def miau
    print "miau, ich bin " + @name
  end
end

punkti = Katze.new("AA-BB", "Punkti")
flocki = Katze.new("AC-DC", "Flocki")

punkti.miau   # => "miau, ich bin Punkti"
flocki.miau   # => "miau, ich bin Flocki"

Neu ist hier die Verwendung von super. Dies ruft die Methode der Superklasse, in diesem Fall also initialize der Klasse Lebewesen auf. Klar sollte sein, dass die Klasse Katze von der Klasse Lebewesen erbt.

Zusätzlich wollen wir jetzt aber den Namen der Katze abfragen können. Wir definieren also eine Methode name, die ganz einfach das Attribut @name zurückliefert.

class Katze
  def name
    @name
  end
end

Zu beachten gilt, dass in Ruby der letzte Wert einer Methode als Rückgabewert zurückgegeben wird, @name also im Beispiel oben.

Jetzt brauchen wir noch Hunde. Hunde haben der Einfachheit halber keinen Namen (och, bin ich fies ;-). Dafür können sie bellen und man kann sie Katzen jagen lassen.

class Hund < Lebewesen
  def wau
    print "wau wau"
  end

  def jage(katze)
    print "ich jage " + katze.name
  end
end

Wir müssen auch gar keine initialize Method definieren da diese aus der Klasse Lebewesen geerbt wird.

Dann lasst uns mal spielen ;-)

bello = Hund.new("AA-ZZ")   # auch Hunde haben eine DNS

punkti.miau                 # => "miau, ich bin Punkti"
bello.wau                   # => "wau wau"
bello.jage(punkti)          # => "ich jage Punkti"

Und was passiert wenn wir eine Katze bellen lassen? Einfach ausprobieren:

punkti.wau
# => NoMethodError: undefined method `wau' for #<Katze:0x81e9748>

Da Ruby weiss dass punkti eine Katze ist, und Katzen keine Methode wau definieren, teilt es uns mittels einer Ausnahme (Exception) mit, dass diese Methode nicht existiert. Genauso gut können wir einen Hund einen anderen Hund jagen lassen, oder ein beliebiges anderes Objekt. Da ein Hund jedoch keine Methode name hat, wird Ruby uns wieder eine Ausnahme melden. Das heisst also, dass wir die Methode jage mit einem Objekt als Parameter aufrufen müssen, für das die Methode name definiert ist. Diese Vorgehensweise nennt man auch liebevoll Duck Typing. Dave Thomas hat diesen Begriff geprägt, wenn ich mich recht erinnere.

"Everything that walks like a duck, quacks like a duck, is a duck"

Gemeint damit ist, dass wir nicht überprüfen ob ein Objekt der Klasse Ente angehört wie dies in Java, C++ und vielen anderen Sprachen der Fall ist (das nennt man übrigens Tag-typing), sondern einfach nur testen ob dieses Objekt die Methoden watschel und quack definiert. Wenn ja, dann handelt es sich für uns um eine Ente, ansonsten nicht.

Today I wrote RuBackup, my own backup program, of course in Ruby. I am now using it instead of Flexbackup, which is a very decent backup program written in Perl that I used before.

It creates CPIO archives (new portable format, w/o CRC), and compresses them on the fly if desired. Note that it’s all pure Ruby, so that you don’t have to install the cpio program.

You can download the sources here. Make sure that you get both files cpio.rb and rubackup.rb. Of course it comes without any warranties. USE AT YOUR OWN RISK!

Below is a short example how to use RuBackup:

require 'rubackup'
require 'enumerator'
require 'find'

BACKUP_DIR = '/backup/%s'

Collection.new('projects', BACKUP_DIR) {
  Find.to_enum(:find, '/data/projects').to_a
}.backup_monthly

Collection.new('maildirs-home', BACKUP_DIR) {
  files = []
  Dir.glob('/data/home/*/Maildir') do |md|
    Find.find(md) {|path|
      if File.directory?(path) && %w(.Spam .Trash).include?(File.basename(path))
        Find.prune # skip those directories
      else
        files << path
      end
    }
  end
  files
}.backup_monthly

Collection.new('rcs', BACKUP_DIR) {
  Dir['/**/RCS'].select {|path| File.directory?(path)}.
  collect {|path| Find.to_enum(:find, path).to_a}.flatten
}.backup_weekly

Collection.new('etc', BACKUP_DIR) {
  Find.to_enum(:find, '/etc').to_a +
  Find.to_enum(:find, '/usr/local/etc').to_a
}.backup_weekly

Call this script daily (via crontab or /etc/periodic/daily) to create daily incremental backups. When backup_monthly is specified, it will create a full backup every first day of the month, or in the case of backup_weekly, every first day of the week.

Note that I’m using Find.find most of the time in the example above instead of Dir.glob, as the latter does not include dotfiles by default.

Why another backup program?

I wrote RuBackup, as I wanted to be able to perform an arbitrary number of incremental backups. This wasn’t possible with flexbackup or at least I didn’t knew how. Furthermore, I wanted to be able to specify exactly which files to backup and which not. For this purpose, Ruby is very powerful.

In FightingWithSubversion I wrote that the development version of ViewCVS was not able to colorize Ruby sources. Now I found out how to solve this problem:

--- lib/viewcvs.py.old  2004/08/09 20:04:49
+++ lib/viewcvs.py  2004/08/09 20:06:06
@@ -1102,6 +1102,7 @@
   '.pp' : 'pascal',
   '.ps' : 'postscript',
   '.py' : 'python',
+  '.rb' : 'ruby',
   '.s' : 'asm',
   '.scheme' : 'scheme',
   '.scm' : 'scheme',

Why not use Ruby as a host language for implementing an assembler? Each mnemonic becomes a method in Ruby, registers are represented by objects of class Int32Register and can either be accessed through methods (r( i ), r0, r1, …) or constants (R[ i ], R0, R1, …).

Just for fun, I did a proof of concept in Ruby. The assembler is not yet ready to generate MIPS machine code, except for a few opcodes. Instead, it simply outputs MIPS assembly source code.

One advantage is, that you are not limited to a very restricted macro language, but have the full power of Ruby to generate assembly code. It’s very simple to create more complex instructions like the downto_zero macro shown below, which first "executes" the block (i.e. the instructions in the block are assembled), then decrements the counter register given as an argument to it, and stops when it reaches zero.

require 'mips'

def move(dst, src)
  if src.is_a? Integer
    addi dst, src, r0
  else
    add dst, src, r0 # it's a Register
  end
end

def downto_zero(counter_reg)
  loop_start = label()
  yield
  subi counter_reg, counter_reg, 1
  bgtz counter_reg, loop_start
end

# load R1-R3 with words from (R10)
for i in 1..3
  lw R[i], (i-1)[R10]
end

# calculate 1+2+3+4+5 in r2
move r1, 5
move r2, 0
downto_zero(r1) {
  add r2, r2, r1
}

And this is the assembly code it generates:

                LW R1, 0(R10)
                LW R2, 1(R10)
                LW R3, 2(R10)
                ADDI R1, R0, #5
                ADDI R2, R0, #0
tmp_00000:
                ADD R2, R2, R1
                SUBI R1, R1, #1
                BGTZ R1, tmp_00000

Here is the source. Note that it’s just a hack.

Nothing big or world-changing.

rcrchive.net/rcr/RCR/RCR265

My first thought was: Where is the Linux/BSD version! as the homepage says that it’s available for Windows and Intel Linux, but there was no link to download the free trial version for Linux.

The IDE is written using the FOX Toolkit (or at least I strongly think so), which is cross-platform. FOX has a Windows-like look on all supported platforms. It’s a lightweight toolkit because it draws the widgets itself and not uses any native widgets as for example WxWindows does.

For my personal taste, it’s too much overloaded with features. And it’s missing mechanisms to extend it. And of course, it’s not free (neither source nor cost). The syntax high-lighting parser seems to be pretty well done, even though I didn’t tried it extensively. The debugger seems to be very good too (Disclaimer: I never really needed/used a debugger ;-).

The company developing the Arachno IDE is located in Berlin, Germany. The same IDE seems to be available for Python, PHP and Perl. If you like it, buy it :-)

BTW, I made this screenshot sitting in the garden with my laptop (still) running FreeBSD, accessing a Windows box via my wireless network using VNC. Gimp then did the rest.

start summary::

Rite will be a complete rewRite of the Ruby interpreter, whereas Ruby2 will be the new revision of the Ruby language.

Rite:

• bytecode
• generational GC
• native threads
• new regular expression engine (Onigurama)
• m17n (multilinguization)

Ruby2:

• real keyword arguments (not just hashes)
• new block variable scope
• pre/post/wrap methods
• selector namespaces (?)
• a few other modifications that clean up the language (but no dramatic changes, Ruby will stay the same)

This is mainly a repetition of Matz’s ideas presented at RubyConf 2003 "How Ruby Sucks" (did Guido van Rossum ever told you How Python Sucks? :-). More information on this subject is given in Rite or RiteSuggestions.

end summary::

New Local Variable Scope

Block parameters are block-local, whereas all other variable assignments inside the block do no longer create block-local variables.

But what about block-local variables other than the parameters?

Solution 1

By using more block-parameters than actually used for the arguments. For example:

ary.each do |i, aLocal, anotherLocal|
  ...
end

This seems odd to me, as it’s no longer clear how each behaves, which variables are actually parameters, and which are not.

Solution 2

Introduce new syntax for block-local variables. For example:

ary.each do |i| <aLocal, anotherLocal>
  ...
end

Or more verbose:

ary.each do |i|
  local aLocal, anotherLocal
  ...
end

A local * or <*> would then mean, that all used variables are local to the block.

Keyword arguments

Old hash syntax for method calls

Will the following become obsolete?

def foo(hash)
  ...
end

foo("port" => 8080, "url" => "...")

Instead (Rite):

def foo(**hash)
  ...
end

Or explicitly (Rite):

def foo(port: nil, url: nil)
  ...
end

foo(port: 8080, url: "...")

I would argue for making it obsolete, i.e. removing it, for the sake of clearness.

No parentheses?

Will it still be possible to ommit parentheses? Will the following work?

foo port: 8080, url: "..."

I don’t like this when used in combination with keyword arguments, as it reminds me of Smalltalk, but is completely different (first part is a method instead of a receiver/object in ST).

Passing Arguments Through

Will this work?

def wrap_a(*args, **keys, &block)
  a(*args, **keys, &block)
end

Of course, it should!

Complex argument lists

Will this be possible?

def foo(a, x=5, *args, b: 42, c: nil, **keys, &block)
  ...
end

foo(1)
# => a=1, x=5, args=[], b=42, c=nil, keys={}, block=nil

foo(1, 2, 3, 4, c: 5, port: 8080) {...}
# => a=1, x=2, args=[3,4], b=42, c=5, keys={port: 8080}, block={...}

I really don’t want to write a parser for this ;-)

Don’t care argument list

As argument lists become more complicated, wouldn’t it be nice to have a syntax for don’t care? This is especially useful for pre and post methods:

def foo(a, *args, b: 42, c: nil, **keys, &block)
  ...
end

def foo:pre(...)
  # don't care about the arguments
end

Don’t need to write:

def foo:pre(*a, **b, &p)
  ...
end

Maybe … isn’t appropriate, as it reminds me of ellipses in C++. Probably no big win in clearness, as usually pre and post methods are used for meta-programming and not by the programmer itself.

Another possible solution is to not perform arity checks for pre or post methods, so that argument lists can be ommited, when not needed:

def foo(a, b, c)
  ...
end

def foo:pre
  puts "pre"
end

But that would make them a bit more special than normal methods are, which is IMHO not a good choice.

Pre/Post/Wrap Methods

Great improvement for meta-programming like Aspect Oriented Programming (AOP).

start summary::

There are lots of different mechanisms for preventing unwanted concurrent data-access, which might lead to race-conditions. Ordered from low to high-level some of them are:

1. Semaphores
2. Monitors
3. Tuple Spaces
4. Join Calculus
5. Data-Flow Synchronization

end summary::

The last three are not only used for synchronization purposes, but also for communcation.

Semaphores

Semaphores are the most basic building block for synchronization. But it’s hard to use them in large, complex applications.

Monitors

Monitors instead are a synchronization construct (usually) built into the language. It’s behaviour is very similar to that of a mutex, but unlike a mutex which can be used in a very fine-grained manner, monitors are tied to methods. This fact might lead to increased use of condition variables and to more synchronization overhead.

Furthermore, monitors make the language much more complex, and introduce implicit assumptions. My knowledge of Java’s concurrency primitives is too little, but would you be able to tell, without looking at the specs, whether or not it’s possible to access an instance variable declared as public inside a synchronized class? This is exactly the point why C or C++ does not have monitors, as it is impossible to decide whether another object has access to a public instance variable, whose access should be serialized. I strongly believe in the pure OO way followed by Eiffel & Co (Smalltalk, Sather, Ruby …), where objects react solely on messages (methods) and instance variables are not direclty accessible from outside of the object, only through methods (a good compiler can then optimize the method-call away, e.g. similar what inline does in C++)

That monitors need not be built into the language is demonstrated by the next example:

require "monitor"

mon = Monitor.new         # create Monitor object
cond1 = mon.new_cond      # create new condition variable
cond2 = mon.new_cond      # create another new cv

Thread.start do
  mon.synchronize do
    # inside the monitor
    ...
    cond1.wait
    ...
    cond2.signal
    ...
  end # leave monitor
end

# proceed with next thread here...

The advantage of this approach is, that you can have multiple monitors inside your objects, each one acting on another set of methods. And, you can decide yourself when a monitor should be entered, which needn’t be the begin of a method. Cross-object monitors are possible as well.

The example above didn’t showed the use of monitors in combination with methods. Now, in Ruby, this becomes even easier:

class Queue
  include MonitorMixin

  def initialize
    super()           # initalize Monitor
    @cond = new_cond  # create cv
  end

  def get
    synchronize { ... @cond.wait  ... }
  end

  def put
    synchronize { ... @cond.signal ... }
  end
end

Tuple Spaces

Parts of this section are taken from my book "Ruby Developers Guide", Chapter 5.

A TupleSpace is a space or kind of database in which you can store arbitrary tuples of any length and type (may of course depend on the implemention). Three operations are defined on a TupleSpace:

in

Removes a matching tuple from the TupleSpace and returns it to the caller.

rd

Same as in but does not remove the tuple from the TupleSpace.

out

Puts a tuple into the TupleSpace where it is stored until a matching in operation is issued.

All three operations are performed atomically, that is, they are thread-safe. Pattern-matching is used to specify which tuples to fetch from the TupleSpace. Multiple identical tuples are allowed inside a TupleSpace, similar as multiple tokens are allowed to sit in a place for a general S/T petrinet.

Producer-Consumer Example using TupleSpaces

An implementation in Ruby is given below. Note that the methods write and take are used for in and out respectively.

require "rinda/tuplespace"

ts = TupleSpace.new

# init with 2 credits
2.times { ts.write ['credit'] }

# producer thread
Thread.new {
  loop do
    ts.take ['credit']

    sleep 1
    puts "P: car produced"

    ts.write ['car']
    puts "P: car delivered"
  end
}

# consumer thread
Thread.new {
  loop do
    ts.write ['credit']
    puts "C: car ordered"

    ts.take ['car']
    puts "C: car received"

    sleep 3
  end
}

Both the consumer and the producer thread synchronize to each other using a TupleSpace. The tokens in the "Cars" place (see figure below) are represented in the TupleSpace by car tuples, and the "Credit" tokens by credit tuples. The other places of the petrinet are not used in the TupleSpace example.

With Rinda, Ruby’s version of Linda, we used in the example above, it’s very easy to even distribute the TupleSpace accross a TCP/IP network. Just replace the second line with the following two lines:

DRb.start_service
ts = DRbObject.new(nil, "druby://host_running_ts:0")

Then start anywhere on the network a remote TupleSpace:

require "rinda/tuplespace"

DRb.start_service("druby://hostname:0", TupleSpace.new)
gets   # [return] to exit

Rinda and DRb (Distributed Ruby) are part of the Ruby distribution. DRb is comparable to Java’s RMI (Remote Method Invocation), but is (a) much easier to use, and (b) written in pure Ruby in around 500 lines (the full package with lots of other useful libraries including SSL, ACLs, different object id converters, is around 2500 lines).

Join Calculus

The Join Calculus is similar to a TupleSpace where you can wait on a set of tuples at once. Only if the join condition of a function is fulfilled (usually this means waiting for more than one tuple) it is executed.

JoCaml

JoCaml is a dialect of Objective Caml and implements the Join Calculus. It is best suited for concurrent and distributed applications.

In the example below, we define a concurrent counter.

let def count! n | inc () = count (n+1) | reply to inc
     or count! n | get () = count n | reply n to get
;;

The count! is an asynchronous channel, which means that an invocation will return immediatly back to the caller and does not return any value. Each asynchronous channel can be seen as a named tuple in a TS. A synchronous channel (without an !) instead blocks the caller until the execution terminates, or if it’s definition is used in a join pattern like count! n | inc (), the reply to directive must be used to return to the caller of inc.

Now, let’s use the counter. We first have to initialize it by issuing spawn { count 0 }. In TupleSpace-terms, this would store a ["count!", 0] tuple in the TS. To increase the counter we then issue an inc() operation. If there’s a count! tuple (as there is), this triggers the execution of the first line of the definition of our counter, and consumes the count! tuple, binding the n in the definition to the tuple’s value (0 in our case). It then proceeds with executing the right hand side of the definition (everything after the =). The vertical bar separating the count (n+1) and reply to inc means that both terms are executed in parallel, i.e. in different threads. With the count (n+1), we store back the increased tuple to the TS, similar as we did at the beginning with the value zero, and with reply to inc we return to the caller of inc. Exactly the same happens when calling get (except that it returns the current counter value to the caller).

The count! channel is used as both a mutex and a storage for the counter value.

Join Patterns for Ruby

This is my implementation of Join Patterns for Ruby using a TupleSpace provided by Rinda. You can get it here. It’s nothing more than a proof of concept and should not be used for serious development.

Below we implement the concurrent counter in Ruby.

js.let_def "count! n | inc ()" do
  send "count!", @n+1
  reply_to "inc"
end

js.let_def "count! n | get ()" do
  send "count!", @n
  reply_to "get", @n
end

# initialize counter with 4
js.send "count!", 4

# you now can increment from everywhere

js.send "inc"
js.send "inc"

p (js.send "get") # => 6

Data-Flow Synchronization

Mozart/Oz

Mozart/Oz is a multi-paradigm language which uses data-flow for thread synchronization. One of it’s features are logic variables. A logic variable is either bound to a value or unbound. Once it is bound, it stays bound forever (inside it’s scope).

This type of variable is often used in functional languages, with the difference that variables in most of these languages (Haskell, ML) are always bound to some value.

Now, if you access an unbound variable in Mozart/Oz, the accessing thread will block until a value gets bound to it (from another thread of course). See the example below:

declare Cond1 Cond2 in

% Thread T1
thread
   {Show 'T1 a'}
   {Delay 1000}    % Wait a second
   Cond1 = true    % signal Cond1
   if Cond2 then   % wait for Cond2
      {Show 'T1 b'}
   end
end

% Thread T2
thread
   if Cond1 then   % wait for Cond1
      {Show 'T1'}
   end
   Cond2 = true    % signal Cond2
end

The output would be:

'T1 a'
'T1'
'T1 b'

This is only a very primitve example that uses logic variables as signal semaphores. Of course you can do a lot more with them, and very elegantly. If you’re interested, have a look at the Mozart/Oz manual, which provides you with many good examples.

PACT XPP

The PACT processor is another good example of data driven synchronization, this time implemented in hardware. It consists of an array of say, 32x32 cells, where each cell itself is configurable (FPGA) as either an ALU with a specific funtion, a register or another computation unit. An ALU cell for example has two inputs (two operands) and one output (result). Those inputs and outputs can be connected to other inputs and outputs of other cells.

As soon as data comes in from outside the processor, it flows through this network. Data may branch-off and flow in real-parallel (not quasi!) through different parts of the processor at the same time, or join back later into a single path.

So there may be situations, where an ALU is waiting for a second operand to complete its computation. But as it is usually not known during compile time when a packet arrives, a signaling mechanism is used at runtime to tell the ALU when an operand arrives at an input. Only if both operands are available, it begins its work and produces after n cycles a result on the output channel.

It may also happen that one path to input A of an ALU is much longer than the path to input B of the same ALU. This leads to a block of the whole short path to input B until the ALU has taken the packets from it’s inputs (it’s very similar to the consumer/producer problem). So the network inside the PACT processor can be seen and perfectly modeled as a petrinet, where each place only accepts one token. It might even be (long time since I read the PACT manual) that there are buffers between the outputs and inputs of the cells, so that the short path of the above example would not block immediatly if there is a packet-jam somewhere.

Here’s a Ruby implementation of the simple Haskell algorithm given on page 2 of An Unbounded Spigot Algorithm for the Digits of Pi.

def calc_pi
  q, r, t, k = 1, 0, 1, 1
  loop do
    n = (3*q+r) / t
    if (4*q+r) / t == n then
      yield n
      q, r, t, k = 10*q, 10*(r-n*t), t, k
    else
      q, r, t, k = q*k, q*(4*k+2)+r*(2*k+1), t*(2*k+1), k+1
    end
  end
end

calc_pi {|n| print n; $stdout.flush }

The algorithm generates an infinite number of the digits of Pi.

The article translates a former Python article about design patterns (Factory, Proxy, Template Method) to Ruby and introduces the following additional patterns: Singleton, Visitor and Observer. The reader gains some knowlege how Ruby works internally, for example, that class bodies contain executable code and are executed in the context of class Module or that method_missing is called for unknown methods, and so on…

Sorry, but the article is in German.

www.linuxenterprise.de/itr/online_artikel/psecom,id,407,nodeid,9.html

During the preparation of my Ruby presentation for GPN2, I made a nice observation that did fascinate me the last nights over. While using Ruby’s Proc objects I never thought much about it’s [] method, which is an alias for the call method. I thought, the [] method is there due to it’s similarity to the () calling syntax. Today, I believe that matz had other reasons that led to this decision than just their similarity. The following example tries to clarify this a bit:

Imagine you have written a method that works on arrays, using the [] method to access specific elements. Let’s call this method process:

def process(anArray)
  ...
  anArray[i]  # access i'th element
  ...
end

At the beginning there’s no problem with using this method on arrays, as your arrays are small:

arr = Array.new
for i in 1..1000
  arr[i] = i**2
end
process(arr)

After some time, your array grows, but not the number of elements in it, say only a few elements in your array are defined, but with indices in the range from 1 to 1 billion.

arr = Array.new
arr[23] = 66.5
arr[25_333] = 4
arr[1_000_000_000] = 8
process(arr)

You’ll probably not have enough memory to try out this example, as arrays in Ruby will automatically grow to their largest index. In the example above, we would need around 4 GB of memory, for only three elements.

Clearly, what you need is a sparse array. In Ruby, that’s simply a Hash:

arr = Hash.new
arr[23] = 66.5
arr[25_333] = 4
arr[1_000_000_000] = 8
process(arr)

Notice how nothing except the first line has changed.

Defining a default value for undefined elements in the hash is simple again:

arr = Hash.new(default_value)
...
...

This would return default_value for non existing indices/keys.

Imagine you now want to call the process method with an array that contains all numbers from 1 to 1 billion in reversed order. Neither arrays nor sparse arrays will help here, due to their high memory consumption. What we need are functions, or in Ruby, Proc objects.

arr = Proc.new {|i| 1_000_000_000 - i}
process(arr)

The function (or Proc object) in the above example can be seen as virtual array. We can get the value of the i’th element in the same way we would have done with an array:

arr[i]

which of course in the case of a Proc object is equivalent to:

arr.call(i)

This nice consistency is what I was talking about at the beginning of the article. Did you get it :-)

Today (Sunday 25 May 2003) I gave an introductional talk about Ruby at GPN2 (2nd Gulasch Programming Night).

The presentation slides can be found here:

www.ntecs.de/talks/RubyQuickStart/RubyQuickStart.pdf

www.ntecs.de/talks/RubyQuickStart/html/index.html

Today, I implemented Julia sets in Ruby and C. I first prototyped a Ruby version, then quickly recognized that it is simply too slow to generate larger images. So I translated it to C which resulted in a performance boost of factor 180!

Here’s the fractal created by the example.

The Ruby version is given below. It creates PNM bitmap files which can be viewed for example with display on Un*x.

Feel free to modify the parameters, especially c, rx and ry. And of course the colormap.

require "complex"

c = Complex(-0.75, 0.136)          # the Julia set parameter
iters = 50                         # max iterations
w, h = 400, 300                    # bitmap extents
rx, ry = -1.0 .. 1.0, -1.0 .. 1.0  # view area

sx = (rx.last - rx.first).abs / w  # x-scale
sy = (ry.last - ry.first).abs / h  # y-scale

colmap = Array.new(256, "\000"*3)  # color map

# initialize colormap
(1..255).each {|i| colmap[i] = [10+i*10, 0, rand*100].pack("CCC") }

# write PNM header
print "P6\n#{w} #{h}\n255\n"

# do the real stuff
for j in 0...h
  for i in 0...w
    n, zn = 0, Complex(sx*i+rx.first, sy*j+ry.first)
    while n <= iters
      # sequence runs away to infinity
      break if (zn-c).abs > 2

      # calculate next iteration
      zn = zn**2 + c; n += 1
    end

    # draw/print pixel
    print colmap[n]
  end
end

The same example written more compactly with only 228 bytes :-)

require"complex";c,m,w,h=Complex(-0.75,0.136),50,200,100;puts "P6\n#{w} #{h}\n255";(0...h).each{|j|(0...w).each{
|i|n,z=0,Complex(0.9*i/w,0.9*j/h);while n<=m&&(z-c).abs<=2;z=z*z+c;n+=1 end;print [20+n*10,0,rand*99].pack("C*")}}

Or on the command line:

ruby -rcomplex -e'c,m,w,h=Complex(-0.75,0.136),50,200,100;puts"P6\n#{w} #{h}\n255";(0...h).each{|j|(0...w).each{|i|n,\
z=0,Complex(0.9*i/w,0.9*j/h);while n<=m&&(z-c).abs<=2;z=z*z+c;n+=1 end;print [20+n*10,0,rand*99].pack("C*")}}'|display

This months editon (11/2003) of the German c’t magazin contains an article about Lindenmayer systems. I’ve read about them some years ago in an IBM science jounal, and implemented it afterwards in OCaml. Today I implemented them in Ruby.

Lindenmayer systems are similar to Markov or Semi-Thue systems (both are very simple text replacing systems), with the exception that at each production step, all matching rules are applied, whereas Markov or Semi-Thue systems apply only one rule at a time.

All three systems have in common that they have production rules and a start symbol. Markov systems further have special terminating rules (unlike Semi-Thue) and rules have a specific order in which they are tried (Semi-Thue systems may apply the rules in random order). Due to the specific order, Markov systems are deterministic, Semi-Thue usually not. A Markov system is incorrect if no more rules are applicable, it must terminate by explicitly "calling" the terminating rule (usually denoted as dot).

Here’s an example of a simple Markov system (or algorithm) that can substract two numbers:

|-| ::= -
|-  ::= .|
-|  ::= .-|
-   ::= `e
`e  ::= .`e

The `e denotes the empty symbol (epsilon), the dot marks a terminating rule. Read ::= as bebecomes or produces.

Applying the Markov algorithm onto the start symbol |-|| will lead to:

|-|| ==> -|   (rule 1)
-|   ==> -|   (rule 3)

You see that it correctly subtracts two from one, giving negative one (-|). You can even express any algorithm as Markov or Semi-Thue system, as they are Turing-complete.

During the Computer Science I reading, I wrote a Markov simulator, which is available here: raa.ruby-lang.org/list.rhtml?name=markov

Lindenmayer Systems

I told above that they are very similar to Markov systems, with the difference that at each step any matching rule is applied. Look at the following example which should clarify this (the second step is important):

Rules:
  F => (F+F)

Start symbol:
  F

Markov:
  1.  F     => (F+F)
  2.  (F+F) => ((F+F)+F)

Lindenmayer:
  1.  F     => (F+F)
  2.  (F+F) => ((F+F)+(F+F))

The subsitution step of a Lindenmayer system can be written in only a few lines of Ruby code:

class String
  def explode; scan(/./m) end
end

class Producer
  def initialize(rules)
    @rules = rules
  end

  def produce(str, n=1)
    n.times { str = str.explode.map {|s| @rules[s] || s}.to_s }
    str
  end
end

# use it
l = Producer.new { "F" => "(F+F)" }
l.produce(start="F", n=2)

Once you have produced a pattern with a Lindenmayer system, you could interpret every character of it as a command to a turtle graphics engine, with the following meanings:

F

Forward one step with pen down

f

Forward one step, but do not draw (pen up)

+

Turn right for a predefined angle

-

Turn left for a predefined angle

[

Push current state on stack (position, angle etc.)

]

Pop state from stack and make it current

Implementing this and the turtle engine is another 100 lines of Ruby code.

Examples

One well known Lindenmayer system is the Koch curve. It is constructed by the rule F ::= F+F—F+F, the start symbol F—F—F and an angle of 60 degree. The picture shows a Koch curve after the 5th iteration.

Botanic ferns or grasses are other examples of Lindenmayer systems.

The fern below (4th iteration) was constructed by the rule F ::= FF-[-F+F+F]+[+F-F-F], the start symbol F and an angle of 22 degree.

The grass below (7th iteration) was constructed by the two rules X ::= F[+X][-X]FX and F ::= FF, the start symbol X and an angle of 29.7 degree.

The X symbol as well as all other unknown symbols are ignored by the turtle graphics engine.

Downloads

You can download the sources for the turtles engine and the Lindenmayer systems here (you must put all sources into one directory):

www.ntecs.de/downloads/Turtle.tgz

www.ntecs.de/downloads/Lindenmayer.tgz

To view the turtles, you’ll need RMagick as well:

raa.ruby-lang.org/list.rhtml?name=rmagick