7. April, 2009
in
»
»
by Michael Neumann

Around 4 years have passed since the last release of Wee.

What is Wee?

Wee is a Seaside-inspired 1 web-framework for building highly-dynamic component-based web-applications.

What’s new?

  • 100% Rack based
  • Support for continuations
  • JQuery AJAX support
  • Ruby 1.9 ready

Distinctive Features

Wee is not just another web framework. It is completely different.

Continuations

You can use continuations to model the page flow of the your application, in the same way as you’d call a method in a GUI application. Continuations are optional; nothing in Wee depends on them. Example:

# you can write code like this:

if callcc YesNoMessageBox.new('Really?')
  # do something
else
  # something else
end

Without continuations one has to use (ugly) Continuation Passing Style (CPS):

call YesNoMessageBox.new('Really?') do |res|
  if res
    # do something
  else
    # something else
  end
end

Backtracking

Backtracking means that the user can naturally use the browsers back button despite Wee’s statefulness. This is not taken for granted for stateful web frameworks. Behind the scenes, Wee keeps multiple states of the application around with only little help by the programmer.

Components

Contrary to the widely used model/view/controller (MVC) paradigm, Wee tighly couples the controller and the view within a component. Components itself are highly decoupled from the rest of the application and can be reused to construct more complex applications. The powerful programmatic HTML rendering approach further reduces wasting the programmers mind by avoiding switching files (controller/view) or languages (Ruby/HTML). Therefore in Wee everything is written in Ruby. The programmatic HTML renderer is not just a simple HTML builder, it provides very powerful methods for easily generating HTML constructs and registering callbacks. Example:

#
# Generating a <select> tag
#

# select an object from these items
items = [1, 2, 3, 4]

# the labels shown to the user
labels = items.map {|i| i.to_s}

# render it
r.select_list(items).labels(labels).callback {|choosen| p choosen}

# render a multi-select list, with objects 2 and 4 selected
r.select_list(items).multi.labels(labels).selected([2,4])

Call/Answer mechanism

From callback handlers you can call other components which in turn replace the current components view. The called component can later return (or answer) back to the calling componenent. Example:

class YesNoMessageBox < Wee::Component
  def initialize(msg)
    super()
    @msg = msg
  end

  def render(r)
    r.bold(@msg)
    r.form do
      r.submit_button.value('YES').callback { answer true }
      r.space
      r.submit_button.value('No').callback { answer false }
    end
  end
end

# Use call (or callcc) as in the Continuation section above
# to call a component:
call YesNoMessageBox.new('Really?')

The classical Hello World Example

require 'rubygems' if RUBY_VERSION < "1.9"
require 'wee'

class HelloWorld < Wee::Component
 def initialize
   super
   add_decoration Wee::PageDecoration.new(title="Hello World")
 end

 def render(r)
   r.h1 "Hello World from Wee!"
   r.div.onclick_callback { p "clicked" }.with("click here")
 end
end

Wee.run(HelloWorld) if __FILE__ == $0
# Now point your browser to http://localhost:2000/

Installation

gem install wee

More Resources

  • http://rubyforge.org/projects/wee

  • http://www.ntecs.de/projects/wee/doc/rdoc/

  • http://github.com/mneumann/wee

25. February, 2009
in
»
»
by Michael Neumann

The last couple of days I spent refactoring my web application framework Wee, a Seaside-like framework for Ruby that I started back in 2004 with a lot of mental help from Avi Bryant (one of the main guys behind Seaside). It’s now approaching a 2.0 release. Wee is now fully Rack based (Rack is a commonly used Ruby Webserver Interface) and the code is in general even cleaner than it was before (huge parts were rewritten). Plus one cool feature: Continuations.

Continuations

Yesterday I finally thought it’s time to put back in continuations mainly because of some very interesting developments (i.e. patches) for Ruby 1.8.x, which seems to fix memory leaks that occured when using continuations.

Continuations in Wee were never used as extensively as they were used in Seaside. And it’s interesting to read that Seaside 2.8 reduced heavy usage of continuations – something that Wee did from the beginning :)

An example says more than thousands words, so here we go:

require 'wee'

class Page < Wee::Component
  def initialize
    add_decoration Wee::PageDecoration.new('Title')
    super
  end
  def render(r)
    r.anchor.callback {
      if callcc YesNoMessageBox.new('Really delete?')
        callcc InfoMessageBox.new('Deleted!')
      else
        callcc InfoMessageBox.new('Deleted action aborted')
      end
    }.with("delete?")
  end
end

class InfoMessageBox < Wee::Component
  def initialize(msg)
    @msg = msg
    super()
  end

  def render(r)
    r.h1(@msg)
    r.anchor.callback { answer }.with('OK')
  end
end

class YesNoMessageBox < InfoMessageBox
  def render(r)
    r.h1(@msg)
    r.anchor.callback { answer true }.with('YES')
    r.space(1)
    r.anchor.callback { answer false }.with('NO')
  end
end

Wee.runcc(Page)

The interesting part of this example is the anchor tag callback handler of component Page. When clicked it will display a YES/NO message box which will return true if you clicked on YES, otherwise false. When it returns, execution resumes exactly at the point after the callcc call. In fact, callcc returns the return value of the called component. A called component returns by calling the answer method (it behaves mostly like the regular return statement). The concrete mechanism how this all works out is a bit more complicated as it involves installing and removing several decorations to delegate rendering and catch exceptions upon answer, but this is totally unrelated to continuations.

So how would it look without the use of continuations? Well, the callback handler would turn from:

if callcc YesNoMessageBox.new('Really delete?')
  callcc InfoMessageBox.new('Deleted!')
else
  callcc InfoMessageBox.new('Deleted action aborted')
end

into:

call YesNoMessageBox.new('Really delete?') do |res|
  if res
    call InfoMessageBox.new('Deleted!')
  else
    call InfoMessageBox.new('Deleted action aborted')
  end
end

Actually in this concrete example it’s not hard to rewrite the code without using continuations. But for more complex examples it would get much worse. For example, using continations, a simple sequential flow like the following one:

callcc page1
callcc page2
callcc page3

would turn into the much less readable equivalent using CPS (“continuation passing style”):

call page1 do
  call page2 do
    call page3
  end
end

Which one would you prefer? The good thing: In Wee you can use both, that’s why I have the two methods call and callcc, the latter not to mix up with Kernel.callcc.

Performance and Memory Usage

The reason why continuations were basically unsupported in Wee for a very long time was that they leaked memory. Since a few months there circulated patches on the ruby-core mailing list that seem to fix those leaks. Actually the reason seams to be that some parts of the stack are not overwritten when calling a function and as such old values keep referenced: a leak is born! I haven’t tried those patches, but I think they work. Instead I tried Ruby 1.9.1. And wow! It’s incredible! Memory usage stays constant at 12 MB, regardless of the number of requests. And performance scales nearly linearily as I increase the number of threads.

For 10000 requests (with one thread) Ruby 1.9.1 takes 16 seconds and requires 12 MB of memory. The same example with Ruby 1.8.7 grows to 329 MB of memory and takes 56 seconds. That’s an increase in performance of factor 3.5 and a 27-fold reduction of memory.

The example I’m using for the benchmark is the following 2-level nested callcc component call:

class Benchmark < Wee::Component

  #
  # calls Called2 then returns
  #
  class Called1 < Wee::Component
    def render(r)
      r.anchor.callback { callcc Called2.new; answer }.with('back')
    end
  end

  class Called2 < Wee::Component
    def render(r)
      r.anchor.callback { answer }.with('back')
    end
  end

  def render(r)
    r.anchor.callback { callcc Called1.new }.with("click")
  end
end

Conclusion

Continuations seem to be stable enough and not too expensive in terms of memory and performance in Ruby 1.9 so that there is nothing against using them (wisely) within Wee. This makes Wee the only web framework for Ruby to my knowledge that uses continuations. Coupled with other great features provided by Wee, for example reusable components, backtracking or the programmatic HTML generation, this undoubtly allows Wee to be called one of the most advanced web application frameworks for Ruby. Worth to note is that Wee does not focus on RESTful multi-million page-view serving, scaling applications. Instead Wee focuses on very complex applications, similar as found in traditional GUIs, and to get the job done quick and beautiful.

21. February, 2008
21. February, 2008
in
»
by Michael Neumann

The last day I hacked on another piece of Erlang code. I gave it the name http_hub. It’s not that easy to explain what it actually does. Clients can connect to it, as it behaves for them like a HTTP server. The client’s request is forwarded to a backend server. But no answer is sent back to the client. Instead http_hub waits on another port (e.g. 14000) for another HTTP request from the backend server (usually a single worker node in a cluster will answer), which includes the response that should be sent back to the client. Once this arrives, the client is answered with the included response. If no request on port 14000 arrives within a specified time, an error response is sent back to the client.

The code is here.

19. February, 2008
19. February, 2008
in
by Michael Neumann

During the last two hours I implemented a hyper-fast (4000 files with ~110 MB in total in around 4 seconds) link extractor. It reads file names of HTML files from stdout, extracts all anchor href attributes and writes them line-by-line in the file with the same name + “.links”. The sources are here.

7. February, 2008
7. February, 2008
in
»
»
by Michael Neumann

I wrote an implementation of the Arc Challenge in Ruby using my web-framework Wee. The sourcecode is available here. I haven’t used continuations, because I think I broke continuation support in some of the later versions of Wee (just because I haven’t used continuations in my Wee apps). Once Rubinius gets (even) more mature, and my spare time allows, I’d like to get back to some ideas of Wee and make use of continuations.

2. February, 2008
2. February, 2008
in
»
by Michael Neumann

The following sentence in the article Garbage Collection is Why Ruby on Rails is Slow: Patches to Improve Performance 5x; Memory Profiling made me shudder:

“Complex Rails request can allocate hundreds or even thousands of megabytes of memory, making GC runs dozens of times.”

WTF?! Hundreds or even thousands of megabytes for a single (but complex) Rails request? Then there is seriously something wrong!

16. November, 2007
16. November, 2007
in
by Michael Neumann

In this post on Artima, Bruce Ekkel writes about his bad experience with web-sites that mix HTML and AJAX components. Especially some travel-sites that use this technology behave badly on other browsers than Internet Explorer, he says.

From my own experience I fully agree with him in this point. When I wanted to fly to Greece earlier this year I noticed that the GermanWings web site is not usable on any browser other than Internet Explorer because it displays an advertisement banner in front of the “choose your destination” dialog in the wrong moment. I guess thats clearly a CSS problem.

I disagree that this has anything to do with AJAX. At first, he mixes up AJAX with Javascript. AJAX is not Javascript! Most pages use Javascript for displaying a “date selector” dialog, and as he talks about new technologies, Javascript is clearly not one of them. Javascript is actually a damn old technology and date selectors exist for a very long time, so nothing fancy here. I think that especially for the user there is nothing easier to use than a Javascript-based date selector (unless you take technologies like Seaside or Wee into account ;-).

15. September, 2007
15. September, 2007
in
»
by Michael Neumann

After a few hours of hacking, I implemented and refined the ideas first mentioned in my article HTML in Ruby and released XML in Ruby as you can read here.

13. September, 2007
13. September, 2007
in
»
by Michael Neumann

For my blogging software I am using Tenjin as template engine. I also tried Kwartz which is also very nice. With Tenjin you embed Ruby into HTML code, in the same way as you’d do in Erb or PHP. This turns out to get pretty ugly. Kwartz completely separates HTML from presentation logic. It is a nice concept, but the presentation logic file can get quite complex for just simple things.

So why not embed HTML in Ruby?

# Example for embedding HTML in Ruby :)

class MyView

  def render(posts)
    for post in posts
      render_post(post)
    end
  end

  def render_post(post)
    #<div id="${post.id}">
    if post.abstract?
      #<p>${post.abstract}</p>
    else
      #<p>${post.body}</p>
    end
    #</div>
  end

end

I think you got the idea, right? I just misuse comments to embed HTML in Ruby. This is very easy to parse and looks quite nice in an editor and it should be easy to tell vim to colorize the embedded HTML correctly. Note that everything that starts with “#<” would be embedded HTML so that you’d still be able to use regular Ruby comments.

13. September, 2007
13. September, 2007
in
by Michael Neumann

Nginx (aka. “EngineX”) is a high-performance web server similar to Lighttpd (aka. “Lighty”).

I migrated to Lighty from Apache long ago. Now it was time to migrate from Lighty to Nginx. Why? Well, I prefer it’s configuration over that of Lighty. It’s more intuitive. I just like it a lot!

The migration went seamless. And the configuration is now much more readable.

31. August, 2007
31. August, 2007
in
by Michael Neumann

While working on pay4co.de (soon to be released), I saw the need for Captchas. Because I am lazy I found reCAPTCHA, which provides a service that you can include in your page. Not only that, it also helps to digitalize scanned in books, as the words the user has to type in come directly from scanned-in books. Here is the API Documentation.

A Capchta decoder for various captchas in use you can find here.

30. August, 2007
30. August, 2007
in
»
by Michael Neumann

I've been using Kwartz lately and it's great! The good thing is that you can view your templates in your browser without the need to preprocess them first. Of course Kwartz only works for HTML or XML templates, but that's fine with me.

22. August, 2007
22. August, 2007
in
»
by Michael Neumann

I'm currently extracting all German persons that are listed in Wikipedia. For this I used WWW::Mechanize, a library that I initially wrote, but now is maintained and extended by someone else. After fetching and parsing around 100 pages, the process took around 500 MB and pipes began to fail and stuff like that :). Mechanize keeps a history of all pages, so it's better to call history.clear from time to time, or maybe there is even an option to set the size of history. I found out about this after switching to using Hpricot directly (it is used by Mechanize anyway). So now memory stays constant with a maximum of around 30 MB.

21. August, 2007
21. August, 2007
in
»
by Michael Neumann

The Hpricot HTML parser and XPath/CSS implementation is just - GREAT! So easy to use, so powerful, and I think it's also fast! Thanks _why!

At first I tried scrubyt!, but I never got it running. And after reading the documentation, I was quite confused. I think it tries to do too much and too much magic behind the scenes.

6. August, 2007
11. September, 2007
in
»
»
by Michael Neumann

I'm not yet sure how I will name my library which is similar to Google Web Toolkit and is based on my RubyJS. Yesterday, I implemented some DOM related stuff in under two hours. GWT uses class methods of the DOM class everywhere, e.g.:

DOM.setElementAttribute(Element elem, String attr, String value)

I don't follow this approach, because I think it's more elegant to move them into an Element class and make it an instance method instead:

el = Element.createDiv
el.setAttribute("a", "b")

The same applies to Events. This allows me to write for example:

Element['root'].appendChild(
  Element.createDiv.setInnerHTML('abc').
          setAttribute('title', 'tooltip'))

instead of:

root = DOM.getElementById('root')
elem = DOM.createDiv
DOM.setInnerHTML(elem, 'abc')
DOM.setElementAttribute(elem, 'title', 'tooltip')
DOM:appendChild(root, elem)

which looks much less like OO in my hence opinion.

4. August, 2007
4. August, 2007
in
by Michael Neumann

YSlow analyses your web page and tells you how and where you could improve it to make it faster (e.g. load times).

9. July, 2007
9. July, 2007
in
by Michael Neumann

Google Gears is a browser extension that embeds an SQLite database into a browser to enable web application to even run when there is no internet connectivity. I think this is a great idea, once it (would) come by default with every web-browser.

I'd like to think about alternatives. I think the easiest alternative is to have a local web-server running on your system, which serves the client-side web-application even if there is no internet-connectivity. This of course requires parts (or the whole) of the server side to run under this local web-server. And thats my idea. Imagine you have your photos stored on Flickr. Do you trust them? I feel better when I have my photos at least backed up on my local system. And it's also much faster to show other people your photos once they are cached locally. So the idea is to have "two" servers. One locally and one globally. They synchronize each other, so that you can fully work offline without any need to change one line of the client side code. This of course means that you better open-source your "application", or you encrypt it so that no one can decipher it, if you prefer to protect your investments.

23. June, 2007
11. September, 2007
in
»
»
by Michael Neumann

Okay, I did a simple performance benchmark on my laptop.

My 31 lines Erlang webserver shown below using the undocumented option {packet, http} as described here is very fast!

-module(http).
-export([start/0]).

start() ->
  {ok, LSock} =
    gen_tcp:listen(8081, [binary, {packet, http}, {reuseaddr, true}, {active, false}, {backlog, 30}]),
  accept(LSock).

accept(LSock) ->
  {ok, Sock} = gen_tcp:accept(LSock),
  Pid = spawn(?MODULE, request(Sock)),
  accept(LSock).

request(Sock) ->
  {ok, {http_request, Method, Path, Version}} = gen_tcp:recv(Sock, 0),
  headers(Sock).

headers(Sock) ->
  case gen_tcp:recv(Sock, 0) of
      {ok, {http_header, _, _, _, _}} -> headers(Sock);
      {error, {http_error, _}} -> headers(Sock);
      {ok, http_eoh} -> body(Sock)
  end.

body(Sock) ->
  gen_tcp:send(Sock, [<<"HTTP/1.1 200 OK\r\n">>,
     ["Connection", ": ", "Keep-Alive", "\r\n",
      "Content-Length: 7\r\n",
      "Content-Type: text/plain\r\n"],
    <<"\r\n">>, <<"1234567">>]),
  request(Sock).

Save this code in a file http.erl. Then start up Erlang "erl" and type "c(http)." followed by "http:start().". After that, you can run apache-bench on it:

    ab -n 50000 -c 100 -k http://127.0.0.1:8081/test

The result on my very old laptop is 3600 requests/sec.

Now the same with Mongrel:

require 'rubygems'
require 'mongrel'

class TestHandler < Mongrel::HttpHandler
  def process(req, resp)
    resp.start(200) do |head, out|
      head["Content-Length"] = "7"
      out.write("1234567")
    end
  end
end

Mongrel::Configurator.new :host => '127.0.0.1' do
  listener :port => 8081 do
    uri "/test", :handler => TestHandler.new
    run
    join
  end
end

The same benchmark with Mongrel gives only around 500 requests/sec. Of course Mongrel does a little more than the simple Erlang web server shown above. And I think Mongrel doesn't employ connection keep alive. But if you see that lighttpd serves only at 1050 requests/sec, then it's impressive what the Erlang web server can handle. Okay, lighttpd has to load the file from disk (7 bytes), that's much less efficient than serving directly from memory.

23. August, 2006
27. June, 2007
in by Michael Neumann

Read this. Having discovered it today, I have to say, I really like it :)

28. February, 2005
28. February, 2005
in
»
»
by Michael Neumann
I can’t write all Wee code on my own. So here is a list of things that needs to be done:
  • Make session-id cookies available to both Wee::Request and Wee::PagelessRequest, controlled by an application-wide setting.
  • Finish the implementation of pretty-URLs (this rulez!). Thanks to Joao Pedrosa who is working on this.
  • Implement a FilterDecoration and it’s subclass AuthorizationDecoration (that’s very easy, ~20 lines of code).
  • Implement a Seaside-like root_for traversal, which allows each component in the tree to add stuff inside the <head> tag. This is triggered by the root-component from it’s render method.
  • Refactor the Wee::Session class and it’s subclasses.
  • Implement further adaptors, most useful would be a FastCGI adaptor. This would need to factor out the WEBrick dependent parts.
  • Implement ResouceHandlers, which make it possible to generate images (or other external resources) from within your components.
  • Use it in your projects and write tutorials ;-)
26. February, 2005
26. February, 2005
in
»
»
by Michael Neumann
I’m trying to marry my child Wee with some of Rails features, mainly REST-like, pretty URLs. But before, I have to understand the differencies in how they work.

Disclaimer: This is my understanding of how Rails work. I might be wrong.

Rails

Rails model is very simple compared to that of Wee. The controller class and the method to invoke is extracted directly from the URL. For example:

/blog/show

would invoke BlogController#show. Now imaging the following URL:

/blog/show/4

This would invoke BlogController#show with @params = {‘id’ => 4}. I don’t know how Rails knows that 4 maps to the ‘id’ key, but that’s another story.

So you have these parts of a URL:

/controller/action/arguments

In Rails, there’s also no (conceptual) distinction between performing an action and rendering, as is the case in Wee. Both are inseparable from each other (in Rails).

Another difference to Wee is, that the controller classes are stateless, and instances of it might be reused (pooling). On the other hand this implicates that you can’t store information inside a controller instance across requests.

Wee

In Wee, we differenciate between an action phase, where callbacks are invoked and as such possibly modify the state of components, and a render phase, which ideally is side-effect free and whose sole purpose is to render the component to HTML (or whatever format you like).

Imagine you look at a simple Wee application which displays an HTML anchor tag. If you click on that anchor, you’ll trigger an action phase, which eventually will find the registered callback associated with the anchor you have clicked on and invokes it. At the end of the action phase, Wee will forward you to a new (automatically generated) URL, which when requested by your browser, will trigger the render phase. This in turn renders the whole component tree. "Why redirect", you might ask. Well, this simply avoids that the same callback will be invoked again if you hit your browsers reload button. Whenever you hit on reload, this will only trigger a render phase event.

Additionally, at the end of each action phase, a snapshot of the component tree is taken, so that you can go back to older states (called back-tracking). The information about which state of the component tree we refer to, is stored as page_id inside the URL. This page_id increases whenever an action phase is performed.

So, an URL in Wee basically consists of the following parts:

session_id page_id [callback_id]

where callback_id is optional. If it is given, the URL triggers an action phase. Otherwise, a render phase.

Compared to Rails controllers, Wee’s components are composites, meaning that they may contain sub-components which itself might contain sub-components and so on. And in Wee there’s only ever one root component, whereas in Rails there are usually multiple controllers. All this makes it nearly impossible to have REST-like URLs in Wee. Also due to the reason that a sub-component cannot be rendered on it’s own, whereas a controller in Rails builds a whole page. So it does not make sense to have URLs in Wee like:

/1.2.3
# == @root_component.children[1].children[2].children[3]

Marrying Wee and Rails aproaches

The concept of multiple top-level controllers can be easily added onto Wee by using the following RootComponent class:

# NOTE: not fully functional code!

class RootComponent < Wee::Component
  def initialize
    super()
    @controllers = {
      'blog' => add_child(BlogComponent.new),
      'list' => add_child(ListComponent.new)
    }
  end

  def render
    controller = # extract information from URL
    r.render @controllers[controller]
  end
end

This RootComponent merely acts as a dispatcher. It looks at the URL and extracts the desired controller out of it and then forwards to it.

Likewise, we could map the Rails action part (the "show" in "/blog/show"), to invoke render_xxx (e.g. render_show) of the controller-component (I use this term now, to distinguish it from a "regular" Wee component and to mark the similarity with a Rails controller):

  def render
    controller = # extract information from URL
    action = # extract Rails action part from URL

    component = @controllers[controller]

    component.with_renderer {
      component.send("render_" + action)
    }
  end

And even further, we could also extract the additional arguments of the URL for use inside the called render method. We could use a custom parse_arguments method here in each controller-component.

Now lets look at a simple example:

class BlogController < Wee::Component
  def parse_arguments(str)
    # for /blog/show/5, str would be "5"
    @params = ...   # e.g. {'id' => 5}
  end

  def render_show
    entry = BlogEntry.get(@params['id'])
    # render it
  end

  def render_list
    BlogEntry.find_all do |entry|
      # render it
    end
  end
end

What else would be needed is to tell which controller/action pair should be used. This could for example be specified each time when generating an anchor or form tag:

r.anchor.controller('blog/show/5').callback { .... }.with('show')

Note that this would first invoke the callback and then render ‘blog/show/5’. Of course you could ommit in this case the callback. But specifying the controller/action pair each time is tedious, as this would have to be done in each sub-component, too, which completely breaks the concept of a component. So the second approach is much better:

r.anchor.callback {
  ... do something
  request.controller = 'blog/show/5'
}

where the request.controller setting is carried inside the URL as long as you assign a new value to it.

Note that you can still use sub-components with this approach, but they would no longer be subject of REST-like URLs.

1. February, 2005
1. February, 2005
in
»
»
by Michael Neumann
Read yourself here.
26. January, 2005
12. September, 2007
in
»
»
by Michael Neumann
This night, I hacked up a Table and a Pager component for Wee. The Table component is 140 lines of code (including all HTML generation, nothing else except CSS styles are required), the Pager is 100 lines long.

Some days ago, I wrote an OgScaffolder component. It’s damn easy to use. Just pass an Og domain class to OgScaffolder.new, and what you get is a regular Wee component. Look at the screenshot below.

Below I’d like to show some Nemo screenshots, to show it’s potential. Note that the whole generic editor component (the first screenshot) is less than 200 lines very clean Ruby code! Thanks Kevin ;-)

18. January, 2005
18. January, 2005
in
»
by Michael Neumann
I hacked up a little library which I use in some small projects to automate login into web-pages, extracting data from them and similar things. Web::Unit didn’t work for my purposes, and it was easier to reinvent the wheel than to hack Web::Unit (of course I first tried to hack it…).

It depends on narf-lib (>= 0.6.3), more exactly, it’s version of htmltools, and requires ruby-current (due to limitations in the net/http library of 1.8.2). If you want to try it out with Ruby 1.8.2, just get the net/*.rb files from ruby-current and use these instead.

Features

  • Cookies (very preliminary, no expire etc.)
  • automatic redirect
  • Forms, Links

Be aware that it’s just a few-hour hack, to get my project done.

It’s still very generic and misses lots of usability methods. For example, I use find all the time, or have to pass the link that should be clicked to method click.

Example

Here’s a simple example, that logs into your rubyforge.org acount (pass your username and password as command-line argument).

require 'mechanize'

agent = WWW::Mechanize.new {|a| a.log = Logger.new(STDERR) }
page = agent.get('http://rubyforge.org/')
link = page.links.find {|l| l.node.text =~ /Log In/ }
page = agent.click(link)
form = page.forms[1]
form.fields.find {|f| f.name == 'form_loginname'}.value = ARGV[0]
form.fields.find {|f| f.name == 'form_pw'}.value = ARGV[1]
page = agent.submit(form, form.buttons.first)
puts page.body

Get it!

www.ntecs.de/viewcvs/viewcvs/Mechanize/

11. January, 2005
12. September, 2007
in
»
by Michael Neumann
Yesterday, I spent the time to create a web-based presentation maker (yes, in less than one day!). It’s a web application written in Wee and Ruby, with which you can edit slides (change, add, remove, reorder) and view a presentation inside a browser. Let’s look at a screenshot first (click on it to see it in full size):

The screenshot shows the edit mode. If you click on ‘Back’, then the left pane is hidden and you can proceed with presenting. If you click with the mouse on the title, the next overlay (or slide, if we’re on the last overlay) is shown. I want to add keyboard navigation, too, but I couldn’t get the JavaScript working. Note that the whole application is in no way JavaScript-driven, I only need JavaScript to trigger an "open(url)" call, so that Wee will display a new page. It would work completely without JavaScript if I would display anchor links for "next overlay" and so on.

Features

  • The presentations file format uses Ruby’s documentation markup (RDoc), which is a very easy markup language (nearly plain text). So you even don’t need the web-based editor to create the presentation, you could also use your favourit editor (vim of course ;-)
  • It’s easy to display colorized source code. Just insert:
    !!colorize:ruby
    def this_method
    end
  • You can start applications from within the presentation by clicking on a link. Note that the application is started on the server-side, which is intended, as you’ll run the presentation maker application from your very own machine. Example (the execute button in the screenshot):
    !!exec:hidden
    cd $HOME/Dev/svn/public/ForSys && ruby test_lout.rb
  • You can create Postscript and PDF output for printing. A Table of Content is included, and in the PDF version all links are clickable (of course all images are included, too).
  • Of course you can use your own style-sheet.

Try it!

viewcvs.ntecs.de/PresentationMaker/

And don’t forget to read the README file.

29. November, 2004
29. November, 2004
in
»
by Michael Neumann
Preliminary documention of Wee is online. There is not much as of now, only the documentation of Wee::Component is nearly complete.

www.ntecs.de/viewcvs/viewcvs/*checkout*/Wee/trunk/doc/rdoc/index.html

16. November, 2004
16. November, 2004
in
»
by Michael Neumann
There’s a problem with objects registered for backtracking whose snapshots have a reference back to the registered object itself.

Let’s look at some scenarios. I’ll call the object to register "obj" and it’s snapshot "snap". I further assume that "snap" references "obj".

Scenario 1 (Good)

  1. We register "obj" for backtracking.
  2. "obj" goes out of scope -> it gets garbage collected.
  3. We take a snapshot. No snapshot of "obj" is taken.

Here, no problem.

Scenario 2 (Worse)

  1. We register "obj" for backtracking.
  2. We take a snapshot "snap".
  3. Snapshot "snap" goes out of scope.
  4. Now "obj" could go out of scope, too.

Note that as long as a snapshot of "obj" exists, "obj" cannot go out of scope.

Scenario 3 (Worst)

  1. We register "obj" for backtracking.
  2. We take a snapshot "snap".
  3. Before "snap" goes out of scope, we take another snapshot. This new snapshot also includes "obj".

As long as at least one snapshot exists that was taken after the initial "snap" snapshot, "obj" will not be garbage collected. It’s included in every further snapshot.

This might happen in Wee as we store a fixed number of snapshots (there’s always one available). So you have to take care to not introduce a reference back to the registered object from the snapshot.

Solutions

  • Whenever possible, avoid to register objects in dynamically created components (e.g. MessageBox).
  • The snapshot of a registered object which is taken by calling method take_snapshot of the objects’s class should never contain a reference back to the referenced object. If this happens, you’ll introduce memory-leaks (for the duration of the session).
31. October, 2004
12. September, 2007
in
»
»
by Michael Neumann
As my dad is giving a guest lecture about the mathematics behind building society savings (Bausparen) at University of Karlsruhe, he wanted to give his students some tools they can play with easily.

Last evening, we then build the calculator shown in the picture below. It’s a portable web application using Ruby and my new Wee framework. I spent most time on generating the diagram (it’s done using gnuplot). The rest was straightforward, once we had the underlying calculations working.

The total code size is 360 lines of code (including HTML), of which 110 are used for the calculation and for generating the diagram. Lots of lines of code are currently spent in conversion and validation routines (german number format).

You can try it out online here: ntecs.de/wee/ehf/

31. October, 2004
31. October, 2004
in
»
by Michael Neumann
Ruby can’t store continuations or blocks on disk! That was one reason why I was initially against using continuations (and blocks) in Wee. But now I have found some solutions to the problem.

Blocks

In the rendering-phase, you usually define some actions and input elements. This is currently done this way:

def inc; @cnt += 1 end
def dec; @cnt -= 1 end

def render_content_on(r)
  r.anchor.action(:inc).with('++')
  r.anchor.action(:dec).with('--')
end

While with blocks, you can write:

def render_content_on(r)
  r.anchor.action { @cnt += 1 }.with('++')
  r.anchor.action { @cnt -= 1 }.with('--')
end

When it comes to storing a session on disk, we simply mark those pages for which a block handler was specified, as "needing another render-pass before being able to respond to an action". So, in the rare case of needing to shutdown the computer, we don’t loose any information. It’s just that the user has to reload the page.

Continuations

We can rebuild block handlers, by executing their definition (render-phase). But that does not apply to continations. If you serialize a session with active continuations, Wee will replace the continations with marker objects, so that we can later detect that there was an active contination going on. If this happens, Wee will notify the user about this situation and will return to the calling component and call it’s method fallback_from_cc. If you want, you can still use old-style call behaviour by passing the name of the return method. Then you won’t loose any information.

Another idea is to use DragonFly's checkpointing support. This would be a 100% solution for the whole problem.

30. October, 2004
12. September, 2007
in
»
by Michael Neumann
The whole long night, my laptop was running and stressing a Wee application to find memory leaks in the current continuation implementation. I started with 25 stress processes, after 14 hours, there were only 16 of them still alive. The other 9 processes died for a yet unknown reason (probably a connection related problem). All in all, the processes created around 7.5 million requests, in 14 hours or 50000 seconds, and memory consumption did not exceed the magic 20 MB mark. This time I used Ruby from the stable branch.

This is an average of 150 requests per second or if we don’t count the redirects, which make the half of the requests, this is still an impressive performance of 75 whole page views per second (on a 1300 MHz Centrino laptop, running KDE with lot’s of open applications). Of course this time, the web application was a very simple one, it showed only a chain of dialog boxes.

30. October, 2004
30. October, 2004
in
»
by Michael Neumann
Avi Bryant wrote some comments about Wee.
29. October, 2004
29. October, 2004
in
»
by Michael Neumann
This morning, I added support for continuation-based programming to Wee. It’s now one step closer to Seaside2 or Borges. But I still support non-continuation-based apps. After changing half a dozen of lines, it was done, or so I thought. But then, I realized that you can’t call continuations across threads. After wrapping each session into it’s own thread and communicating with the application via two queues, and setting Thread.abort_on_exception=true to find some minor bugs, it really worked!

The rest of the day I fighted against some memory issues. To detect them, I was in the need for a simple stress testing application. Thus, I patched WebUnit, to support redirects and <br/> tags, then wrote a memory observer with GNUplot output and finally the stress tester.

I run the tests with all recent versions of Ruby, then I realized that the leak must be in my program and not in Ruby. More or less quickly I discovered that I had a reference back to the continuation in the called component, which resulted in the continuation not being freed, as the continuation has itself a reference to the component on the procedure stack. Assigning nil fixed the bug.

Below is the memory usage of Wee with continuations running under ruby-current as of today. I stressed it with 50 processes, i.e. 50 active sessions (= 50 Ruby threads), and a total of 100000 requests (in 10 minutes). It stayed below 22MB, that’s 500kB per session. It still slightly increases over the time, not sure whether this is GC related or not (fragmentation?). I’ll have to stress it over night ;-)

28. October, 2004
28. October, 2004
in
»
by Michael Neumann
I finally got backtracking right. I fighted against a hard to find bug, and it turned out, that I was modifying a snapshot (e.g. arr = []; arr << 1). Method freeze is your friend! Now, backtracking is much more flexible. You can register individual objects for beeing backtracked (nothing new), but you can now have user-defined methods for taking or applying a snapshot of this object. For this purpose, you have to define take_snapshot and apply_snapshot, very similar to marshal_dump and marshal_load.

Backtracking decorations is now possible, too, but you still have to do this on your own. This is useful, so that "call/answer" calls can be undone correctly.

Decorations

A Delegate decoration is used to implement the call/answer mechanism. I’ll explain this by demonstrating a little example:

def render_content_on(r)
  r.anchor.action(:confirm_quit).with do
    r.text('Quit')
  end
end

def confirm_quit
  call MessageBox.new('Do you really want to quit?'), :quit
end

def quit(res)
  exit if res
end

If you click on the anchor named Quit, method confirm_quit is called. This then replaces the calling component with a message box (or at least, only the message box is shown and can handle actions, not the calling component), with two buttons OK and Cancel. If you now click on one of them, the calling component is restored and it’s method quit is invoked with the return code of the message box. How the message box can be implemented is shown below:

class MessageBox < Wee::Component
  def initialize(msg)
    @msg = msg
  end

  def ok
    answer true
  end

  def cancel
    answer false
  end

  def render_content_on(r)
    r.text @msg
    r.form do
      r.submit_button.value('OK').action(:ok)
      r.space
      r.submit_button.value('Cancel').action(:cancel)
    end
  end
end

In further versions of Wee, it will be possible to register an action and specify some default arguments. This would make the methods ok and cancel superfluous:

r.submit_button.value('OK').action(:answer, true)
r.space
r.submit_button.value('Cancel').action(:answer, false)

Uhm, 12 minutes later… it’s working ;-) Found by the way two other bugs.

So how is this call/answer mechanism working? Well, it really is only half a dozen lines of code:

class Wee::Component
  attr_accessor :caller   # who is calling us?

  # call another component
  def call(component, return_method=nil)
    component.caller = [self, return_method]
    add_decoration(Wee::Delegate.new(component))
  end

  # return from a call
  def answer(*args)
    component, return_method = self.caller
    component.remove_first_decoration
    component.send(return_method, *args) if return_method
  end
end

Get it! Try it!

So now, get Wee and try it out: viewcvs.ntecs.de/Wee/

Or play with it here.

27. October, 2004
27. October, 2004
in
»
by Michael Neumann
Now it can handle 10 complete requests (which are actually two, one is a redirect) per second more, simply by changing the line
val.dup rescue val

into:

case val
when Fixnum, nil, true, false
  val
else
  val.dup rescue val
end

The second is up to 14 times faster for objects that can’t be cloned (Fixnum, nil, true, false). This results in an increase from 97 req/sec to 160 req/sec in the action-phase of Wee (for the example described in WeeFramework). Next, I will do component render caching (for leaf components).

27. October, 2004
12. September, 2007
in
»
by Michael Neumann
After another complete rewrite of my initial prototype, and after trying out several different design choices (and asking Avi Bryant lots of questions), my web framework is now in a very good shape. And it’s got a name: Wee.

The core is now around 600 lines of code. The HTML related libraries put another 500 lines of code.

The trickiest thing I implemented was the StateRegistry, where you can register objects for being backtracked, and where you can take snapshots of all registered objects. All this is done with extensive use of weak-references and finalizers. The serialization part of the StateRegistry was tricky too, as you can’t simply serialize object-ids.

Features

Fully serializable

If you shut down the application, all state gets stored to disk. You can then reload the whole application from disk (possibly on another machine) with exactly the same state where you left off. Of course you could also store the sessions in a database table. Should be simple to implement any kind of load-balancing, as each session is completely independent, so that you could simply transfer sessions to another machine.

Backtracking

If you want, you can make the back-button of your browser work correctly together with your web-application. Imagine you have a simple counter application, which shows the current count and two links inc and dec with which you can increase or decrease the current count. Starting with an inital count of 0, you increase the counter up to 8, then click three times the back button of your browser (now displays 5) and finally decrease by one, then the counter really shows the expected 4, instead of 7 (as clicking the back button does usually not send a HTTP request, and the last state of your application was 8).

Only individual objects are backtracked, those that are registered, not the whole component. That’s the easiest (from an application programmers perspective) and most flexible way. And it’s fast and uses little memory.

You can decide yourself whether you want infinite backtracking or only backtracking up to n pages, with whatever replacement strategy you want, least recently used (LRU), least frequently used (LFU) etc.

Acceptable Performance and Memory Usage

Memory consumption stayed constant in all of my tests (with Ruby 1.8.x, not with 1.9!) by around 7 MB. On my Centrino 1300 MHz laptop, I can currently render 86 pages per second and can handle 97 action requests per second. That means it can handle 45 complete page views per second. The example consist of 20 counter components, with backtracking each of them, WEBrick as webserver (100% pure Ruby).

Reusable Components

Wee components are like widgets in a GUI. Once written, you can use them everywhere. They are completely independent and do not interfere with other components. Components encapsulate state, a view and actions. Of course you can use an external model or use templates for rendering.

Programmatic HTML Generation

You don’t have to leave Ruby to generate the HTML code. This is especially useful for highly dynamic components. But of course, you could also use templates.

Example

Below is the screenshot and sourcecode of a simple editable counter component that is encapsulated 10 times in the main component. You can try it out live here (it’s quite slow as it gets proxied through Apache as I’m too lazy to open my firewall).

require 'wee'

class Counter < Wee::Component
  def initialize(cnt)
    @cnt = cnt
    @show_edit_field = false
    session.register_object_for_backtracking(self)
  end

  def dec
    @cnt -= 1
  end

  def inc
    @cnt += 1
  end

  def render_content_on(r)
    r.form.action(:submit).with do
      r.anchor.action(:dec).with("--")
      r.space

      if @show_edit_field
        r.text_input.assign(:cnt=).value(@cnt).size(6)
      else
        r.anchor.action(:submit).with(@cnt)
      end

      r.space
      r.anchor.action(:inc).with("++")
    end
  end

  def submit
    @show_edit_field = !@show_edit_field
  end

  def cnt
    @cnt
  end

  def cnt=(val)
    @cnt = val.to_i if val =~ /^\d+$/
  end
end

class Main < Wee::Component
  def initialize
    @counters = (1..10).map {|i| Counter.new(i)}
    children.push(*@counters)
  end

  def render_content_on(r)
    r.page.title("Counter Test").with do
      @counters.each { |cnt| r.render(cnt)  }
    end
  end
end
15. October, 2004
15. October, 2004
in
»
by Michael Neumann
Yesterday evening, after my brother demonstrated me his good-looking Python WebWare application, I decided to write my own Web Framework in Ruby. The idea of this project is in my head for a very long time, it started to get more concrete in a thread of mine on ruby-talk of subject WebShare (;-) but then I lost it again for some time.

Then Armin asked me if I would like to do a presentation about Ruby and the Web at EuRuKo04, but I couldn’t decide on a single powerful framework which rulez the world (in my eyes).

Why not Rails, IOWA or Borges?

I never got behind Rails and why it should be so good (luckily I’m not the only one). IOWA instead I know for serveral years and like it’s underlying model. What I don’t like in IOWA is listed below (this should not seen as criticism):

  • The build-in templating stuff that you can’t get around,
  • that IOWA components are in some ways different than pure Ruby classes,
  • it’s code layout (yeah, in this way I’m an aesthet),
  • the use of class-variables and too much magic like "inherited" or import instead of require,
  • but most importantly, that I don’t understand all details of the code

The Borges framework is very nice and powerful, but has had some severe problems with memory leaks in the past, probably due to using continuations (I am not sure whether they were able to fix all these issues). Borges is a port of Seaside 2 from Smalltalk (IIRC, IOWA was a port of Seaside 1). Currently, there seems to be no further activity in improving Borges and make in en par with Seaside. Furthermore I heard on a mailing list that it is very slow due to continations or whatever else, and as such not suitable for larger applications. And you can’t migrate sessions between processes, at least not in Ruby, as Continations are not serializable.

My goals

In short, I wanted something that looks like Borges, but with the underlying model of IOWA. And of course which is developed in Ruby and for Ruby from the first second on.

I pretty exactly knew how a component in my WebFramework (put in better name) should look like. See the example section below.

So what actually were my goals?

  1. Web applications and components should look very concise and clean. Components should in no way look different than usual classes in Ruby. You can use instance variables as you would use them in ordinary Ruby classes. Initialization is no magic and works as usual.
  2. Make it easy to develop reusable components.
  3. No configuration files or special setup required. Works out of the box.
  4. To be able to migrate sessions and serialize them and the whole components tree to a file.
  5. Be able to not just use this framework for HTML content generation, but also for example for emails. Imagine Python’s mailman. This could be implemented "easily" with this framework by using long-term sessions and a special email module that extracts the session and action id out of the email. Then, the only difference to the mailman web-module would be the content generation (plain text for email vs. html for web), the actions would stay the same. This is why I did not have chosen continuations for my framework, as they are not good for long-term sessions, because you loose them if you have to shutdown your server (and sessions for email usually endure several days).
  6. Have a nice framework that I fully understand.

Example

Below, I implement a simple counter component and the main component that displays ten counters.

require 'web'

class MainPage < Web::Component
  def setup
    @body = "<html><body><p>hello world</p>%s</html>"
    @counter_list = (0..10).map {Counter.new(self)}
  end

  def render
    @body % (@counter_list.map {|c| c.render}.join("<br>"))
  end
end

class Counter < Web::Component
  def setup
    @counter = 0
  end

  def render
  %{ <a href="#{ action_url(:dec) }">--</a> #{ @counter }
     <a href="#{ action_url(:inc) }">++</a> }
  end

  def inc
    @counter += 1
  end

  def dec
    @counter -= 1
  end
end

Using templates

Of course you could use templates, too. I quickly implemented this in a few lines of very tricky Ruby code.

require 'rdoc/template'
module Templating
  def template
    file = caller.first.split(':').first
    file = file[0..(-1-File.extname(file).size)] + ".tmpl"
    tmpl = TemplatePage.new(File.read(file))
    proc {|values| tmpl.write_html_on('', values) }
  end
end

Now let’s look at how the templated version of the counter component would look like. You don’t even have to specify the name of the template.

class TemplatedCounter < Counter
  include Templating

  def render; template[
    'dec' => action_url(:dec),
    'inc' => action_url(:inc),
    'counter' => @counter.to_s]
  end
end

The template should be in the same directory as the component. Of course only if you use my approach shown here, which I like very much, as this allows to put each component in it’s own directory, together with it’s template, and you don’t need to setup template paths etc.

<a href="%dec%">--</a> %counter% <a href="%inc%">++</a>

Starting the application

All examples above just implemented the components. To start the application using Ruby’s WEBrick webserver, you have to write:

require 'web/webrick'
Web::Application.new {|app|
  app.name = 'Counter'
  app.session_factory {
    Web::Session.new(app) { |sess| sess.root_component = MainPage.new }
  }
}.start