Setting up a Webserver on DragonFly : NTECS Consulting

In the previous article DragonFly on Hetzner Cloud we covered how to install DragonFly BSD on a Hetzner Cloud server. This article dives into setting up a web server on DragonFly, including Let’s Encrypt certificates, specifically when using Hetzner DNS Robot. It also covers setting up SSH access to the server, setting up time servers, updating packages, etc.

The server I am operating on is c1.ntecs.de which stands for 1st cloud server. Once it is up and running, it will be aliased as www.ntecs.de and also serve HTTP for my second domain www.michaelonroad.de.

Setting up network

root@c1 $ echo dhcpcd_enable=YES >> /etc/rc.conf
root@c1 $ service dhcpcd start

Setting up SSH

To transfer the public key to my server c1.ntecs.de, I use py38-magic-wormhole (termbin.com would do as well as the key is public):

root@c1 $ pkg ins py38-magic-wormhole

As root on c1.ntecs.de I run:

root@c1 $ mkdir /root/.ssh
root@c1 $ workhole ssh invite

It’s now waiting for someone (hopefully it’s me) to send it’s ssh key and enter the correct one-time-code. IMHO it’s actually a bit unsafe, as if someone can guess the one-time-code with a chance of 1 to 65000, he can gain access to my server. Never mind.

On my FreeBSD laptop, I also need wormhole installed. Here it is package py39-magic-wormhole. Then I create a public/private key pair:

laptop $ ssh-keygen -f .ssh/c1.ntecs.de/root

and send it over to the server:

laptop $ wormhole ssh accept **CODE**

Here, you have to provide the correct **CODE**. It will ask you which ssh key to send over to the other side. Choose the one you have just created.

In addition, on my FreeBSD system, I configure .ssh/config to use the key for c1.ntecs.de:

Host c1.ntecs.de
  IdentityFile ~/.ssh/c1.ntecs.de/root

Or as I will later add another user:

Match host c1.ntecs.de user root
  IdentityFile ~/.ssh/c1.ntecs.de/root

Now I should be able to log in with

laptop $ ssh root@c1.ntecs.de

Finally, I remove py38-magic-wormhole from the server again:

root@c1 $ pkg remove -y py38-magic-wormhole
root@c1 $ pkg autoremove

It’s probably much much easier to just copy and paste your public key via the Hetzner Robot Console. As I just found out about wormhole a couple of days ago I wanted to give it a try :)

Updating packages

Next, we want to update the packages, cleanup no longer needed packages (cdrtools for instance) and install vim and tmux.

root@c1 $ pkg remove -y cdrtools
root@c1 $ pkg autoremove -y
root@c1 $ pkg upgrade -y
root@c1 $ pkg ins -y vim tmux

Configuring the system

We want to enable NTP time synchronization using dntpd and setting up a hostname:

root@c1 $ echo dntpd_enable=YES >> /etc/rc.conf
root@c1 $ echo hostname=c1.ntecs.de >> /etc/rc.conf
root@c1 $ service hostname restart
root@c1 $ service dntpd start

Setting up the timezone:

root@c1 $ tzsetup

Choose your timezone from the dialog.

Configuring Let’s Encrypt

On my old server I am using acme.sh but wasn’t really happy with it. After some research I found lego which is a ACME Client written in Go that has all features I need and is easy to use:

root@c1 $ pkg ins lego

In order to configure ACME with lego, you need to modify the lego.sh and deploy.sh scripts and populate domains.txt (all located under /usr/local/etc/lego).

Certificate creation

We first update the /usr/local/etc/lego/lego.sh script. This is what gets called to setup a certification account, request certificates and renew certificates. Once it’s working, we will enable it with periodic.

Patch lego.sh:

--- lego.sh.sample      2022-06-26 22:13:40.000000000 +0200
+++ lego.sh     2022-09-13 22:12:33.576483000 +0200
@@ -1,7 +1,8 @@
 #!/bin/sh -e
 
 # Email used for registration and recovery contact.
-EMAIL=""
+EMAIL="mneumann@ntecs.de"
+HETZNER_API_KEY="********"
 
 BASEDIR="/usr/local/etc/lego"
 SSLDIR="/usr/local/etc/ssl/lego"
@@ -24,10 +25,11 @@
 fi
 
 run_or_renew() {
+       env HETZNER_API_KEY=${HETZNER_API_KEY} \
        /usr/local/bin/lego --path "${SSLDIR}" \
                --email="${EMAIL}" \
                $(printf -- "--domains=%s " $line) \
-               --http --http.webroot="/usr/local/www/lego" \
+               --dns hetzner \
                $1
 }

Here I am using dns-01 challenges with the Hetzner DNS Robot API. You need to create an API key first with your Hetzner DNS Robot.

Next we populate the domains.txt:

echo www.ntecs.de ntecs.de c1.ntecs.de michaelonroad.de www.michaelonroad.de >> /usr/local/etc/lego/domains.txt

This will create a certificate for www.ntecs.de with subject altnames for all the other domains listed thereafter. There will be only one certificate and the file will be named after www.ntecs.de.

We are now ready to run lego for the first time (make sure you run it as user _lego):

root@c1  $ su -fm _lego
_lego@c1 $ /usr/local/etc/lego/lego.sh run

You should now see a certificate and a private key for www.ntecs.de generated as well as some internal book-keeping for lego:

root@c1 $ ls -l /usr/local/etc/ssl/lego/certificates
total 24
-rw-------  1 root  _lego  5394 Sep 14 13:37 www.ntecs.de.crt
-rw-------  1 root  _lego  3751 Sep 14 13:37 www.ntecs.de.issuer.crt
-rw-------  1 root  _lego   233 Sep 14 13:37 www.ntecs.de.json
-rw-------  1 root  _lego   227 Sep 14 13:37 www.ntecs.de.key

Certificate deployment

We want to adapt the /usr/local/etc/lego/deploy.sh script to copy the certificates and private keys into /usr/local/etc/ssl/certs and /usr/local/etc/ssl/private and then restart the web server.

root@c1 $ mkdir -p /usr/local/etc/ssl/{certs,private}
root@c1 $ chmod o+w /usr/local/etc/lego/deploy.sh

We also make the following changes to deploy.sh:

--- deploy.sh.sample    2022-06-26 22:13:40.000000000 +0200
+++ deploy.sh   2022-09-13 22:58:52.080816000 +0200
@@ -1,6 +1,7 @@
 #!/bin/sh -e
 
 SSLDIR="/usr/local/etc/ssl"
+RELOAD_SERVICE=h2o
 
 copy_certs () {
   local certdir certfile domain keyfile rc
@@ -26,5 +27,5 @@
 
 if copy_certs
 then
-  output=$(service nginx reload 2>&1) || (echo "$output" && exit 1)
+  output=$(service ${RELOAD_SERVICE} reload 2>&1) || (echo "$output" && exit 1)
 fi

By default it will reload nginx but as we will be using h2o we have to fix it there.

Before continuing with the web server configuration, let’s run deploy.sh now (as root). It will copy over the certificates but then fail to reload h2o, which is fine.

Setting up the web server

Install and enable the h2o web server:

root@c1 $ pkg ins -y h2o
root@c1 $ echo h2o_enable=YES >> /etc/rc.conf

Create a dhparam.pem file (this may take quite long time):

root@c1 $ openssl dhparam -out /usr/local/etc/ssl/dhparam.pem 4096

Edit /usr/local/etc/h2o/h2o.conf:

--- h2o.conf.sample	2022-07-03 14:09:42.000000000 +0200
+++ h2o.conf	2022-09-13 23:05:17.376781000 +0200
@@ -5,9 +5,9 @@
 user: www
 pid-file: /var/run/h2o.pid
 # log normal access to file
-access-log: /var/log/h2o/access.log
+access-log: "| cat /dev/null"
 # send errors to syslog
-error-log:  "| logger -i -p daemon.err -t h2o"
+error-log:  "| cat /dev/null"
 
 # as of 2017-12-01 the following TLS config and headers, with
 # DNS CAA records and custom diffie-hellmann parameters via
@@ -22,9 +22,8 @@
     # using at least TLS1.2 restricts many older devices
     minimum-version: TLSv1.1
     dh-file: /usr/local/etc/ssl/dhparam.pem
-    # generate your own certificates with security/acme-client
-    certificate-file: /usr/local/etc/ssl/acme/example.org/fullchain.pem
-    key-file: /usr/local/etc/ssl/acme/private/example.org/privkey.pem
+    certificate-file: /usr/local/etc/ssl/certs/www.ntecs.de.crt
+    key-file: /usr/local/etc/ssl/private/www.ntecs.de.key
     cipher-preference: server
     cipher-suite: ECDHE-ECDSA-CHACHA20-POLY1305:ECDHE-RSA-CHACHA20-POLY1305:ECDHE-ECDSA-AES128-GCM-SHA256:ECDHE-RSA-AES128-GCM-SHA256:ECDHE-ECDSA-AES256-GCM-SHA384:ECDHE-RSA-AES256-GCM-SHA384:DHE-RSA-AES128-GCM-SHA256:DHE-RSA-AES256-GCM-SHA384:ECDHE-ECDSA-AES128-SHA256:ECDHE-RSA-AES128-SHA256:ECDHE-ECDSA-AES128-SHA:ECDHE-RSA-AES256-SHA384:ECDHE-RSA-AES128-SHA:ECDHE-ECDSA-AES256-SHA384:ECDHE-ECDSA-AES256-SHA:ECDHE-RSA-AES256-SHA:DHE-RSA-AES128-SHA256:DHE-RSA-AES128-SHA:DHE-RSA-AES256-SHA256:DHE-RSA-AES256-SHA:ECDHE-ECDSA-DES-CBC3-SHA:ECDHE-RSA-DES-CBC3-SHA:EDH-RSA-DES-CBC3-SHA:AES128-GCM-SHA256:AES256-GCM-SHA384:AES128-SHA256:AES256-SHA256:AES128-SHA:AES256-SHA:DES-CBC3-SHA:!DSS
 
@@ -48,50 +47,12 @@
   application/atom+xml: .xml
   "text/html; charset=utf-8": .html
 
-# per-host configurations
 hosts:
-  # a basic fileserver
-  www.example.org:
-    # enable Apache-style directory listings
-    file.dirlisting: on
-    file.send-gzip: on
+  www.ntecs.de:
     paths:
       "/":
-        file.dir: "/var/www/www.example.org"
-      # a simple permanent URL redirect
-      "/blog":
-        redirect:
-          status: 301
-          url: https://blog.example.org/
-      # a password-restricted url
-      "/server-status":
-        mruby.handler: |
-          require "htpasswd.rb"
-          Htpasswd.new("/usr/local/etc/h2o/private/htpasswd", "example.org")
-        status: ON
-      # redireect Lets Encrypt ACME protocol to a specific challenge directory
-      "/.well-known/acme-challenge":
-        file.dir: "/var/www/acme"
-  # virtual directory layout to support serving FreeBSD packages built by poudriere
-  pkg.example.org:
-    paths:
-      "/poudriere":
-        file.dir: "/usr/local/poudriere/data/logs/bulk"
-      "/FreeBSD:10:amd64":
-        file.dir: "/usr/local/poudriere/data/packages/10_amd64-default/"
-      "/FreeBSD:11:amd64":
-        file.dir: "/usr/local/poudriere/data/packages/11_amd64-default/"
-  # a simple ruby-powered embedded JSON API
-  api.example.net:
-    paths:
-      "/ok.json":
         mruby.handler: |
           Proc.new do |env|
             [200, {'content-type' => 'application/json'}, ['{"status":"ok"}']]
           end
-  # a websockets-aware reverse proxy
-  ws.example.net:
-    paths:
-      "/":
-        proxy.websocket: ON
-        proxy.reverse.url: "http://localhost:1080/"
+

Let’s test the configuration now:

root@c1 $ service h2o configtest

This might fail, as there is a “bug” in the rc.d/h2o script.

Either run it directly:

root@c1 $ /usr/local/etc/rc.d/h2o configtest

Or fix it by editing /usr/local/etc/rc.d/h2o and adding env "${h2o_env}" before running the command:

h2o_configtest() {
	env "${h2o_env}" "${command}" -c "${h2o_config}" -t
}

Or wait until the h2o port gets updated.

If the configtest succeeds, start h2o:

root@c1 $ service h2o start

When everything works well, and the DNS records for www.ntecs.de correctly point to the server, we should not be able to point our browser to www.ntecs.de and see it delivering the JSON {"status": "ok"}.

Note that I have disabled any logging of accesses and errors via:

access-log: "| cat /dev/null"
error-log:  "| cat /dev/null"

I do this for privacy reasons (DSGVO) and because I don’t really need the log information. If you turn on logging, don’t forget to rotate your logs.

Advanced web server configuration

In h2o.conf I disable the two HTTP headers:

# header.set: "X-Content-Type-Options: nosniff"
# header.set: "Content-Security-Policy: default-src https:"

because they lead to problems in one of my web sites that use:

<link ref="stylesheet" href=".css">

Changing that to

<link ref="stylesheet" href=".css" type="text/css">

should mitigate the nosniff problem, but I am too lazy atm to fix it.

As I am not using POST HTTP requests, I limit the body to just 1 MiB (from 10):

limit-request-body: 1048576 # 1MiB

Finally, this is my hosts configuration:

hosts:
  "ntecs.de:80":
    paths:
      "/":
        redirect:
          status: 301
          url: "http://www.ntecs.de:80/"

  "ntecs.de:443":
    paths:
      "/":
        redirect:
          status: 301
          url: "http://www.ntecs.de:443/"

  "www.ntecs.de:80":
    paths:
      "/":
        file.dir: "/var/www-data/www.ntecs.de"

  "www.ntecs.de:443":
    paths:
      "/":
        file.dir: "/var/www-data/www.ntecs.de"

  "michaelonroad.de:80":
    paths:
      "/":
        redirect:
          status: 301
          url: "http://www.michaelonroad.de:80/"

  "michaelonroad.de:443":
    paths:
      "/":
        redirect:
          status: 301
          url: "https://www.michaelonroad.de:443/"

  "www.michaelonroad.de:80":
    paths:
      "/":
        file.dir: "/var/www-data/www.michaelonroad.de"

  "www.michaelonroad.de:443":
    paths:
      "/":
        file.dir: "/var/www-data/www.michaelonroad.de"

  c1.ntecs.de:
    paths:
      "/status.json":
        mruby.handler: |
          Proc.new do |env|
            [200, {'content-type' => 'application/json'}, ['{"status":"ok"}']]
          end

As I often type in just the domain name without www. prefix, I add redirections to the domain with www.. I do this for both port 80 and 443 (SSL). What I don’t do is to automatically redirect port 80 to 443. If someone wants to see my page unencrypted via port 80 he or she can still do. Modern browsers will anyway try to use 443 and warn about unecrypted HTTP.

The actual website data is located under /var/www-data/$DOMAIN which is owned by user www-data. Let’s create the user:

root@c1 $ pw user add \
   -n www-data \
   -u 81 \
   -d /var/www-data \
   -g www \
   -L default \
   -m \	
   -s /usr/local/libexec/git-core/git-shell \
   -w no

Within /var/www-data, for each domain I have a git repository, e.g. www.ntecs.de.git that contains the whole static website. On my laptop, I have a copy of that repo and whenever I update the site via a static site generator, I commit the changes into that repo and git push them efficiently to the server. Then I check it out into the /var/www-data/www.ntecs.de directory.

This looks roughly like this:

laptop $ git --git-dir=./deploy-site.git push -u \
    www-data@c1.ntecs.de:/var/www-data/www.ntecs.de.git master
laptop $ ssh www-data@c1.ntecs.de \
    "git --git-dir=/var/www-data/www.ntecs.de.git --work-tree=/var/www-data/www.ntecs.de checkout -f master"

Note that you first have to create a bare git repo like this:

root@c1 $ cd /var/www-data
root@c1 $ git init --bare www.ntecs.de.git
root@c1 $ chown -R www-data:www www.ntecs.de.git

You furthermore have to setup a ssh-key for user www-data and properly configure git-shell. I haven’t done the git-shell configuration yet, so I am using a normal tcsh shell instead.

Configuring periodic tasks

We want daily cleanup of HAMMER2

root@c1 $ echo daily_clean_hammer2_enable=YES >> /etc/periodic.conf

This is on by default, so you don’t need to configure it.

If you want, you can enable daily HAMMER2 snapshots by setting:

daily_snapshot_hammer2_enable="YES"

I don’t do this as none of the data needs to be backed up.

Important is to enable lego in /etc/periodic.conf:

root@c1 $ echo weekly_lego_enable=YES >> /etc/periodic.conf
root@c1 $ echo weekly_lego_renewscript=/usr/local/etc/lego/lego.sh >> /etc/periodic.conf
root@c1 $ echo weekly_lego_deployscript=/usr/local/etc/lego/deploy.sh >> /etc/periodic.conf

This will renew certificates every week, deploy them and reload h2o. We quickly want to test if the period scripts work as expected, so let’s run them:

root@c1 $ periodic weekly

Afterwards, start mail to see the weekly report:

root@c1:~ # mail
Mail version 8.1 6/6/93.  Type ? for help.
"/var/mail/root": 3 messages 3 new
>N  1 root@c1.ntecs.de      Thu Sep 15 05:01 1065/95275 "c1.ntecs.de daily security run output"
 N  2 root@c1.ntecs.de      Thu Sep 15 05:01  80/2959  "c1.ntecs.de daily run output"
 N  3 root@c1.ntecs.de      Thu Sep 15 14:54  29/960   "c1.ntecs.de weekly run output"
&

It’s the email numbered 3, so type 3 and hit enter. Once done hit x for exit (or q for quit which will save the mails in a mbox file).

Good that I checked it, as I am seeing the following message:

Checking Let's Encrypt certificate status:
2022/09/15 14:54:44 Error while loading the certificate for domain www.ntecs.de
        open /usr/local/etc/ssl/lego/certificates/www.ntecs.de.crt: permission denied

Inspecting the file

root@c1:~ # ls -la /usr/local/etc/ssl/lego/certificates/www.ntecs.de.crt
-rw-------  1 root  _lego  5394 Sep 14 13:37 /usr/local/etc/ssl/lego/certificates/www.ntecs.de.crt

shows that it has the wrong file permissions. The problem is, I was running lego.sh once as root and not as user _lego, so it was creating the file as root. Later on, the periodic script runs lego.sh correctly as user _lego and can’t access that file anymore. To fix it, I delete the whole lego/certificates directory and properly run lego.sh:

root@c1  $ rm -rf /usr/local/etc/ssl/lego/certificates
root@c1  $ su -fm _lego
_lego@c1 $ /usr/local/etc/lego/lego.sh run

As root I am now running periodic weekly again and validate the email report via mail. It’s now printing success:

Checking Let's Encrypt certificate status:
2022/09/15 15:05:05 [www.ntecs.de] The certificate expires in 89 days, the number of days defined to perform the renewal is 30: no renewal.
Deploying Let's Encrypt certificates:

Open tasks

Setting up git-shell properly, restricting ssh access for user www-data to just `git.
Creating a HAMMER2 snapshot and comparing the changes.
Daily package updates / audits with pkg upgrade --dry-run and pkg audit --fetch --recursive
Enable Log rotation
Setting up sshlockout
Setting up a mail server
Configure doas
Automating the whole web server setup with a simple script.