Porting Crystal to DragonFly

Crystal is a statically-typed language with a Ruby-inspired syntax and philosophy. “A language for humans and computers” as the homepage states. Want a simple echo server? There we go with a one-liner:

require "socket"; TCPServer.open("localhost", 4000) {|s| s.accept{|c| loop {c.puts(c.gets||break)}}}

Nice! But it’s not available on DragonFly, so let’s change that! It took me roughly a full day to port Crystal to DragonFly. In this article, I describe the steps to accomplish this goal.

Patching the source

Let’s get started and download the source code of Crystal:

git clone https://github.com/crystal-lang/crystal.git
cd crystal

The next thing I do is to type git grep freebsd. This reveals all the files that need to be adapted and have a corresponding version for DragonFly added. Crystal uses compile-time meta-programming where code within “{% … %}” is evaluated at compile time. This is similar to a pre-processor and indeed is used in a similar way to provide platform specialization. There is a couple of files where the only thing I have to change is the following:

-{% if flag?(:freebsd) || flag?(:openbsd) %}
+{% if flag?(:freebsd) || flag?(:openbsd) || flag?(:dragonfly) %}
   ...
 {% elsif flag?(:linux) %}
    ...
 {% end %}

Pretty easy! More effort requires porting Crystal’s lib_c and add support for Dragonfly. I start off with copying the definitions for FreeBSD:

cpdup src/lib_c/x86_64-freebsd src/lib_c/x86_64-dragonfly

Then I go through each file and compare the definitions with the system header files found on DragonFly in /usr/include. While there are subtle differences between DragonFly and FreeBSD most of the definitions are still the same. Same root, and you know, standards like POSIX. Same same but different. Still, half day is spent on carefully reviewing every macro definition. This results in a first patch that looks like this:

From c5c4ee379240fe12e8d6e434a2edaa6f58f5efcc Mon Sep 17 00:00:00 2001
From: Michael Neumann <mneumann@ntecs.de>
Date: Fri, 24 Apr 2020 18:06:51 +0200
Subject: [PATCH] Fixes for DragonFly

---
 src/lib_c/x86_64-dragonfly/c/arpa/inet.cr    |  2 -
 src/lib_c/x86_64-dragonfly/c/dirent.cr       | 18 ++-----
 src/lib_c/x86_64-dragonfly/c/errno.cr        |  4 +-
 src/lib_c/x86_64-dragonfly/c/fcntl.cr        |  9 ++--
 src/lib_c/x86_64-dragonfly/c/pwd.cr          |  4 ++
 src/lib_c/x86_64-dragonfly/c/signal.cr       | 10 ++--
 src/lib_c/x86_64-dragonfly/c/sys/mman.cr     | 12 ++---
 src/lib_c/x86_64-dragonfly/c/sys/resource.cr |  4 +-
 src/lib_c/x86_64-dragonfly/c/sys/socket.cr   |  4 +-
 src/lib_c/x86_64-dragonfly/c/sys/stat.cr     | 53 +++++++-------------
 src/lib_c/x86_64-dragonfly/c/sys/types.cr    | 18 ++-----
 src/lib_c/x86_64-dragonfly/c/sysctl.cr       |  2 +-
 src/lib_c/x86_64-dragonfly/c/time.cr         |  3 +-
 13 files changed, 57 insertions(+), 86 deletions(-)

diff --git a/src/lib_c/x86_64-dragonfly/c/arpa/inet.cr b/src/lib_c/x86_64-dragonfly/c/arpa/inet.cr
index afac8795f..e32f93837 100644
--- a/src/lib_c/x86_64-dragonfly/c/arpa/inet.cr
+++ b/src/lib_c/x86_64-dragonfly/c/arpa/inet.cr
@@ -2,8 +2,6 @@ require "../netinet/in"
 require "../stdint"
 
 lib LibC
-  fun htons(x0 : UInt16T) : UInt16T
-  fun ntohs(x0 : UInt16T) : UInt16T
   fun inet_ntop(x0 : Int, x1 : Void*, x2 : Char*, x3 : SocklenT) : Char*
   fun inet_pton(x0 : Int, x1 : Char*, x2 : Void*) : Int
 end
diff --git a/src/lib_c/x86_64-dragonfly/c/dirent.cr b/src/lib_c/x86_64-dragonfly/c/dirent.cr
index 0028289a3..51986d65e 100644
--- a/src/lib_c/x86_64-dragonfly/c/dirent.cr
+++ b/src/lib_c/x86_64-dragonfly/c/dirent.cr
@@ -6,21 +6,11 @@ lib LibC
   DT_DIR = 4
 
   struct Dirent
-    {% if flag?(:freebsd11) %}
-      d_fileno : UInt
-    {% else %}
-      d_fileno : ULong
-      d_off : ULong
-    {% end %}
-    d_reclen : UShort
+    d_fileno : InoT
+    d_namlen : UShort
     d_type : UChar
-    {% if flag?(:freebsd11) %}
-      d_namlen : UChar
-    {% else %}
-      d_pad0 : UChar
-      d_namlen : UShort
-      d_pad1 : UShort
-    {% end %}
+    d_unused1 : UChar
+    d_unused2 : UInt
     d_name : StaticArray(Char, 256)
   end

...snip (300 lines)...

You can see this patch in it’s entirety in Pull Request #9178.

Two notable remarks: DragonFly’s libc does not export ntohs() and htons() as functions as they are macros in DragonFly. That means I cannot use these functions from Crystal and have to roll them on my own in Crystal. This is rather easy given that they use bswap16 on DragonFly unconditionally (we support only x86_64 atm). Another difference is that we don’t implement fdatasync(2) so I have to use fsync(2) instead.

Cross-compiling Crystal

Since a couple of days I am working on a FreeBSD 12.1 system so I used this as the host system. We need the following packages installed:

pkg ins gmake pkgconf crystal llvm90

Note that we need Crystal to compile Crystal. This process is called bootstrapping and it is the reason why I can’t just compile Crystal on DragonFly - there simply isn’t a Crystal compiler for DragonFly (yet).

Before we start cross-compiling to DragonFly we bootstrap a fresh Crystal compiler from the Crystal compiler obtained via FreeBSD ports. This is as simple as hitting gmake. While it builds, I write down all the commands that the build system uses to build the compiler. I was expecting hundreds of lines…. but no, just the following four lines:

# Line 1
c++ -c -o src/llvm/ext/llvm_ext.o src/llvm/ext/llvm_ext.cc \
    -I/usr/local/llvm90/include -std=c++11 -fno-exceptions \
    -D__STDC_CONSTANT_MACROS -D__STDC_FORMAT_MACROS -D__STDC_LIMIT_MACROS

# Line 2
cc -fPIC -c -o src/ext/sigfault.o src/ext/sigfault.c

# Line 3
ar -rcs src/ext/libcrystal.a src/ext/sigfault.o

# Line 4
CRYSTAL_CONFIG_PATH="/home/mneumann/Dev/crystal/src" \
CRYSTAL_CONFIG_LIBRARY_PATH="" \
CRYSTAL_CONFIG_BUILD_COMMIT="b25cad6bd" \
./bin/crystal build  -o .build/crystal src/compiler/crystal.cr \
    -D without_openssl -D without_zlib

1. The first line builds the LLVM extension that Crystal uses for code generation.

2. The second line compiles C code that Crystal calls early on during startup to setup a signal handler for SIGSEGV and SIGBUS signals. The signalhandler will catch stack overflows and segmentation violations and print out a nice stack trace. This is just a handful lines of C code.

3. The third line produces a library src/ext/libcrystal.a that contains the signal handler code (line 2). I assume that when you compile a Crystal program, it will get statically linked to this library, whereas it won’t get statically linked (nor dynamically) to the LLVM code (line 1). This gives “smaller” executables - a hello world executable in Crystal is roughtly 1 MB in size.

4. The fourth and last line bootstraps the Crystal compiler. It is written in itself so you need an existing Crystal compiler here. Once this finishes, the resulting Crystal compiler is stored as .build/crystal.

To port Crystal to DragonFly we only have to cross-compile the Crystal compiler (last line) to an object file, then compile the C and C++ files (lines 1 to 3) on a DragonFly system and finally link everything together into an executable. Compiling the C/C++ code and the final linking is all done on the DragonFly system.

To cross-comile the Crystal compiler to an object file, I execute the following line on FreeBSD:

./bin/crystal build -o crystal-dragonfly \
    --cross-compile                      \
    --target "x86_64-unknown-dragonfly"  \
    src/compiler/crystal.cr              \
    -D without_openssl -D without_zlib -D without_playground

This tells the Crystal compiler to --cross-compile the src/compiler/crystal.cr file to --target x86_64-unknown-dragonfly. As a result it will produce the object file crystal-dragonfly.o. In addition to that, it prints out instructions on how to link the object file into a final executable:

cc crystal-dragonfly.o -o 'crystal-dragonfly'  \
    -rdynamic /usr/home/mneumann/Dev/crystal/src/llvm/ext/llvm_ext.o \
    `"/usr/local/bin/llvm-config90" --libs --system-libs --ldflags 2> /dev/null` \
    -lstdc++ -lpcre -lm -lgc-threaded -lpthread \
    /usr/home/mneumann/Dev/crystal/src/ext/libcrystal.a -levent -lpthread \
    -L/usr/lib -L/usr/local/lib

We need that on the DragonFly system.

Linking to an executable

Let’s copy the crystal-dragonfly.o to a DragonFly system, install required packages pkg ins gmake pkgconf llvm90, download the Crystal source with the DragonFly patches from above, and then produce an executable:

c++ -c -o src/llvm/ext/llvm_ext.o src/llvm/ext/llvm_ext.cc \
    -I/usr/local/llvm90/include -std=c++11 -fno-exceptions \
    -D__STDC_CONSTANT_MACROS -D__STDC_FORMAT_MACROS -D__STDC_LIMIT_MACROS

cc -fPIC -c -o src/ext/sigfault.o src/ext/sigfault.c

ar -rcs src/ext/libcrystal.a src/ext/sigfault.o

cc crystal.dragonfly.o -o 'crystal'  \
    -rdynamic ./src/llvm/ext/llvm_ext.o \
    `"/usr/local/bin/llvm-config90" --libs --system-libs --ldflags 2> /dev/null` \
    -lstdc++ -lpcre -lm -lgc-threaded -lpthread \
    ./src/ext/libcrystal.a -levent -L/usr/lib -L/usr/local/lib

Note that the first three lines are the exact same command that we were also executing on the FreeBSD system, whereas the last line is the command that Crystal told us when we were cross-compiling the compiler. I didn’t even try to touch the Makefile as it was so simple to compile everything manually. Once the command above finishes, I have a crystal compiler that I can run on DragonFly:

mneumann:~/Dev/crystal % ./crystal version
Crystal 0.35.0-dev (2020-04-24)

LLVM: 9.0.1
Default target: x86_64-portbld-dragonfly5.9

Task accomplished?

Fixing the operating system

At this point I felt great and though I am done. A small hello world program just compiled fine. The next logical step was to let Crystal compile itself. To accomplish that, I copied it to /usr/local/bin/crystal and ran gmake. The compiler ran for one or two seconds until it quit with a huge stack trace! What was going wrong?

At first I suspected that the signal handling code did something wrong. No. Then I thought, maybe it’s libgc-threaded so I tried libgc instead. A bug in the Boehm garbage collector is unlikely but maybe it just doesn’t work well with DragonFly? I tried to patch a few more random things here and there without success. Finally I started gdb and it became clear that this really wasn’t a “bug”, crystal was just using a large amount of stack space and as it turned out later on, a little bit too much for DragonFly in it’s current version. My understanding is that the Crystal parser and compiler uses recursion quite heavily and grew beyond 4MB.

To track it further down, I compared FreeBSD’s libpthread with that of DragonFly. The stack size of the main thread on DragonFly reported 4MB whereas on FreeBSD it reported something around 512MB or whatever is specified as the process’s resource limit. To fix this hard coded 4MB limit, I made a patch against libthread_xu, the libpthread implementation of DragonFly. My patch uses getrlimit(RLIMIT_STACK) for the main thread’s stack size instead of the hard coded 4MB. This is the same approach that FreeBSD takes. With this patch applied I was able to successfully bootstrap Crystal on a DragonFly system!

Fixing unit tests

This is still work in-progress. To run the Crystal test suite, invoke:

gmake spec

A large amount of the test cases pass. Though there are some hangs which needs to be fixed. For instance, the spec for IO.pipe() “raises if trying to read to an IO not opened for reading” fails, as pipe(2) on DragonFly creates two bidirectional endpoints and neither one is special cased for read- or write-only operation, meaning, reading from the “write”-end is perfectly legal on DragonFly and blocks until data is available via the “read”-end.

I have also seen problems with TCPSocket that I have to dive into. There might be also problems with libevent.

Conclusion

Big thumbs up for the Crystal project, not only for creating such a wonderful pragmatic language, but also for making it that easy to cross-compile and even more thumbs up for the simplistic build system. In addition to that I found the overall code quality quite high, everything looked polished and easy to understand. Some 10+ years of experience in Ruby, including that of writing my own Ruby to Javascript transpiler, might have helped me understanding the parts of the codebase that I touched, plus having ported Rust to DragonFly. The only “negative” aspect I can mention here is the use of that much stack space in the compiler.