Home Blog Art Ref sheet Tech Buttons Contact

Migrating my servers to OpenBSD (with loadbalancing)

This post was published on 15 Mar 2025

Update 16th of March, 2025: After posting this on Mastodon, I was given some tips on how I can improve my config, you can find more about that in the update blog post.

Sometimes, I get random ideas that don’t really make much sense in the grand scheme of things. I had a very stable setup for hosting my websites on an LXC in my apartment (which I made available on the Internet using a method I outlined here a few months back), so there was not really any need to change the setup; besides, it’s not like my websites require a really good uptime or anything, they’re just personal websites and nothing mission critical. Nevertheless, I had the urge to try out OpenBSD, mostly because I hadn’t really used a BSD before and I really wanted to learn how BSD works! We use pfSense at work, but you don’t really get to see the CLI much there. Thus, I downloaded OpenBSD and tried it out! I also wanted to kind of “challenge” myself to only (or mostly) use the native tools already available in OpenBSD. No caddy, no nginx, no HAProxy, no certbot.

At any rate, the migration was a success and this website is now running entirely on OpenBSD! Now I just need to find some nice 88x31 “Powered by OpenBSD” or so buttons for my button page ^v^

Why OpenBSD

OpenBSD touts itself as being a very secure operating system which, in theory, should make it ideal for servers. Additionally, it supposedly follows the UNIX philosophy (like doing one thing and doing it well) a bit more closely than modern-day Linux. It also has a fantastic firewall, namely pf, which is what special firwalling operating systems like OPNsense or pfSense use in the background (they’re both based on OpenBSD). Additionally, its man pages are said to be (one of) the best ones you can find; and while I still find them a bit lacking sometimes, I definitely have to agree that they’re very pleasant to read and helped me a ton. So much so, in fact, that I learnt around 80% of what I needed to get this setup working through the built-in man pages! My biggest problem was knowing where to find stuff. For example, if you want to create some self-signed certificates using the openssl command, the OpenBSD man pages very helpfully provide examples for that – but in man ssl instead of man openssl.

A New VPS

I currently have two VPS, one from netcup and one from IONOS. They’re both (seemingly) quite decent VPS providers and netcup also lets you upload your own ISO, so installing OpenBSD on one of their VPS wouldn’t have been a problem at all (I am not sure if IONOS allows custom ISOs). Nevertheless, I wanted to find out if there was maybe a VPS provider that specialised in OpenBSD – and there is indeed, namely obsda.ms. For €69 – nice – per year you can get a (pretty tiny) VPS from them with 1 CPU vCore, 1 GB of RAM and 50 GB of storage. They’re based out of Amsterdam (I am always happy to see more EU-based providers!) and seem really chill. I especially like how, unlike all German providers I have used, they do not require your legal name nor your address; and their TOS are the shortest TOS I have ever seen:

Terms of Use: Don’t run Tor/I2P nodes, don’t convert to Linux, traffic is fair-use and be kind to your neighbour.

I am also very much a fan of the fact that they “[…] donate €10 per VM and €15 per VM for every renewal to the OpenBSD Foundation[.] […]”. They seem to be run by a Dutch company called “High5!”. The VM was also set up much more quickly than I had initially anticipated, taking around ~2 hours if I remember correctly. So if you’re looking for a nice little VPS provider that provides OpenBSD VMs, then I can highly recommend obsda.ms!

The New Setup – With Problems

Now, I am pretty sure my setup is actually just … kind of terrible. There are probably a myriad of ways whatever I am doing is bad practice or inefficient or [negative adjective here], but I am still learning! So if you know more about this and end up reading something you feel like is awful, I would love to hear from you on how I could improve upon it! Shoot me an email over on me [ at ] hexaitos [ dot ] com, replacing the brackets with the actual symbol.

My websites are now running on both the VPS (osprey) and an OpenBSD VM (openbateleur) I have running on a PVE node at home. Both are running httpd as their webserver. openbateleur is connected to osprey via a WireGuard connection and osprey is running relayd which handles TLS certificates and provides load balancing between the webservers on osprey itself and on openbateleur. This, however, means that (at least for now), TLS is terminated on the VPS, which is something I still want to change soon (I just have to figure out how).

Regular HTTP traffic is always routed to a designated acme server (which, in my case, is localhost, i. e. osprey) for serving ACME challenges. HTTP traffic is routed between a series of pre-defined backend servers using relayd’s loadbalance mode. In addition, it uses the check tcp mode for health checks and ignores a backend if it is offline.

I have a bunch of webservers running serving different websites, with each site having its own port. bateleur.org for example has 10000 and some other stuff has 10001 and 10002. These only listen either on localhost (in the case of the VPS) or on the WireGuard interface (in case of the VM at home). For a full example configuration of /etc/relay.conf, scroll down to the bottom of the article.

Problem #1 – Several Domains

The biggest hurdle I came across was getting relayd to behave with several domains. Let’s say I have two domains, www.domain1.tld and www.domain2.tld wherein the first domain should be routed to port 10000 and the second one to 10001 on the defined backend servers. The only way I could get this working was to define a list of backend and acme servers for each domain / webroot and route the requests depending on the HTTP Host header.

In relayd, you can define tables of servers such as <my_servers> { 10.10.10.5 10.10.10.6 10.10.10.7 } and then use the loadbalance, roundrobin etc. modes to route between them. This works perfectly fine as long as you only have one domain on your external IP. If you have several domains (on the same external IP) and route them all to the same table, relayd is confused and just sends everything to the first (I think) domain you routed to. The following configuration gave me problems (only shown here with HTTP instead HTTP+HTTPS) with domain1.tld -> Port 10000 and domain2.tld -> Port 10001.

relayd.conf snippet 
table <webservers> { 10.5.0.11 localhost }

http protocol "http" {
        pass request header "Host" value "domain1.tld" forward to <webservers>
        pass request header "Host" value "www.domain1.tld" forward to <webservers>

        pass request header "Host" value "domain2.tld" forward to <webservers>
        pass request header "Host" value "www.domain2.tld" forward to <webservers>
}

relay "http_relay" {
    listen on $ipv4 port 80
    listen on $ipv6 port 80
    protocol "http"

    forward to <webservers> port 10000
    forward to <webservers> port 10001
}
</details>
I feel like I am missing something here because I don’t _think_ that having to create a separate table for every domain should be necessary? Maybe there’s a more elegant solution, but this was the only way to get it working (which, unfortunately, made the config much longer): 
table <servers_domain1> { 10.5.0.11 localhost }
table <servers_domain2> { 10.5.0.11 localhost }

http protocol "http" {
        pass request header "Host" value "domain1.tld" forward to <servers_domain1>
        pass request header "Host" value "www.domain1.tld" forward to <servers_domain1>

        pass request header "Host" value "domain2.tld" forward to <servers_domain2>
        pass request header "Host" value "www.domain2.tld" forward to <servers_domain2>
}

relay "http_relay" {
    listen on $ipv4 port 80
    listen on $ipv6 port 80
    protocol "http"

    forward to <servers_domain1> port 10000
    forward to <servers_domain2> port 10001
}

Problem #2 – Dual Stack

I also had a bit of trouble getting dual stack to work. My first problem was that, for some reason, no IPv6 traffic could pass through to the VPS at all, despite it definitely having both an IPv4 and an IPv6 address. I knew it had to be a problem with my pf configuration because once I turned off pf with pfctl -d, it worked (my websites weren’t reachable via IPv6 yet but I could at least ssh into the server using IPv6). As it turns out, IPv6 seems to require your allowing icmp6! I did not know about that, but after adding pass in inet6 proto icmp6 from any to any to my firewall config, IPv6 worked! As you can tell, I have not yet dealt with IPv6 much and I wish I could work with it some more, but even at work my boss is like, “Nah, IPv4 works fine, we don’t have the time to spend on unimportant things like this.”

Anyway, relayd itself was not yet listening on IPv6 as I had only added one listen statement at that point for the IPv4. I, therefore, create two variables – ipv4 and ipv6 – and set their values to the public IP addresses of my server. Then, in both my HTTPS and my HTTP relay block, I added a second listen statement for IPv6 like so:

relayd.conf snippet 
# === HTTP RELAY ===#
relay "http_relay" {
    listen on $ipv4 port 80
    listen on $ipv6 port 80
    protocol "http_acme"

    forward to <acme_domain1> port 10000
    forward to <acme_domain2> port 10001
}

# === HTTPS RELAY ===#
relay "https_relay" {
    listen on $ipv4 port 443 tls
    listen on $ipv6 port 443 tls
    protocol "https"

    forward to <servers_domain1> port 10000 mode loadbalance check tcp
    forward to <servers_domain2> port 10001 mode loadbalance check tcp
}

This… did not work. Checking the config using relayd -n threw an error about it not being able to find the certificate https4:443 for some reason. I am not quite sure why it was looking for that certificate in the first place. After some googling, I found one other person who had the same problem as me but who never got a reply to his question – they did, however, mention that splitting the https portion of the relay into two blocks fixed the issue… and it did indeed. Thus, my config now looks as follows:

relayd.conf snippet 
# === HTTP RELAY ===#
relay "http_relay" {
    listen on $ipv4 port 80
    listen on $ipv6 port 80
    protocol "http_acme"

    forward to <acme_domain1> port 10000
    forward to <acme_domain2> port 10001
}

# === HTTPS RELAYS ===#
relay "https_relay" {
    listen on $ipv4 port 443 tls
    protocol "https"

    forward to <servers_domain1> port 10000 mode loadbalance check tcp
    forward to <servers_domain2> port 10001 mode loadbalance check tcp
}

relay "https_relay_v6" {
    listen on $ipv6 port 443 tls
    protocol "https"

    forward to <servers_domain1> port 10000 mode loadbalance check tcp
    forward to <servers_domain2> port 10001 mode loadbalance check tcp
}

Problem #3 – ACME

ACME also gave me some problems. First, I had to figure out how to properly manage certificates. For now, I have decided to have relayd handle TLS connections and then pass regular HTTP traffic to the defined backends. This is not ideal and I am going to figure out a way of having the entire traffic encrypted soon. For my personal websites it doesn’t matter too much, but I want to use relayd for some other things soon (things that require a login) and for that, I would very much like everything fully encrypted.

However, getting the certificates turned out to be a bit more complicated than I had hoped. The VPS is running httpd (just like my VM at home) and I, therefore, decided to make it the designated ACME server. Thus, all regular HTTP traffic will be sent to localhost (i. e. the VPS itself) so that acme-client (OpenBSD’s own ACME client, something like certbot) can put the ACME challenges into the webroot and the VPS’ httpd server can serve them and verify that I own the domains.

I would have preferred to only send requests to /.well-known/acme-challenge/* to localhost, but I have not yet found a way of making that possible with multiple domains. With only one domain, match request path "/.well-known/acme-challenge/**" forward to <acme> in the HTTP protocol block worked perfectly; however, when using several domains and matching them using the HTTP HOST header, the path matching doesn’t seem to work properly (or I am doing something wrong). Therefore, I now have to have an <acme_domain> { localhost } table for every domain. I once again feel like this is not how you’re supposed to do things, but it works for now, even if it makes the config even longer still.

Additionally, I had some problems getting ACME to work in general. I copied the acme-client.conf example configuration from /etc/examples over to /etc and changed it to my needs. The man page says the following:

challengedir path

The directory in which the challenge file will be stored. If its not specified, a default of /var/www/acme will be used.

The /var/www/acme directory already existed and I, therefore, instructed httpd to do the following:

httpd.conf 
server "domain1.tld" {
        listen on localhost port 10000
        root "/htdocs/domain1.tld"

	location "/.well-known/acme-challenge/*" {
		root "/acme"
		request strip 2
	}
}

This, however, did not work. I tried playing around with the permissions of the acme directory to no avail. Whenever I ran acme-client -v domain1.tld, it would throw an error and tell me the acme challenge files weren’t found and my domain was unable to be verified. After some more trial and error and some googling, I came across a solution that finally worked. Instead of using /var/www/acme as the challenge directory, I created a new directory using mkdir /var/www/htdocs/acme and then changed its owner by running chown -R www:www /var/www/htdocs/acme. Afterwards, I had to change the above-mentioned challengedir option in the /etc/acme-client.conf to challengedir "/var/www/htdocs/acme". I also had to change the server configuration. However, this also wasn’t enough to make it work quite how I wanted it to; my domain could finally be verified, but relayd had trouble finding the certificates. Looking at relayd.conf’s man page reveals the following:

keypair name

The relay will attempt to look up a private key in /etc/ssl/private/name:port.key and a public certificate in /etc/ssl/name:port.crt, where port is the specified port that the relay listens on. If these files are not present, the relay will continue to look in /etc/ssl/private/name.key and /etc/ssl/name.crt. This option can be specified multiple times for TLS Server Name Indication. If not specified, a keypair will be loaded using the specified IP address of the relay as name. See ssl(8) for details about TLS server certificates.

As you can see, relayd requires its certificates to be in a special format containing the domain name the certificate is for and the port the relay listens on. The default configuration of acme-client however has the following:

domain key "/etc/ssl/private/domain1.tld.key"
domain full chain certificate "/etc/ssl/domain1.tld.fullchain.pem"

Not only is the format wrong (.pem instead of .crt), but the file naming is also not quite right as it’s missing the port. I, therefore, had to change the file names and ended up with the following two config files that finally worked:

httpd.conf & acme-client.conf 
# === /etc/httpd.conf === #
server "domain1.tld" {
        listen on localhost port 10000
        root "/htdocs/domain1.tld"

	location "/.well-known/acme-challenge/*" {
		root "/htdocs/acme"
		request strip 2
	}
}

# === /etc/acme-client.conf === #
domain domain1.tld {
	alternative names { www.domain1.tld }
	domain key "/etc/ssl/private/domain1.tld:443.key"
	domain full chain certificate "/etc/ssl/domain1.tld:443.crt"
	sign with letsencrypt
}

Last but very much not least, acme-client does not do automatic renewls on its own, you have to write your own cronjob. Luckily, its man page has the answer:

A cron(8) job can renew the certificate as necessary. On renewal, httpd(8) is reloaded:

~ * * * * acme-client example.com && rcctl reload httpd

In this case, we don’t have to reload httpd as that isn’t what’s actually handling HTTPS; instead, we need to adjust the cronjob so that it reloads relay instead and make one entry in our crontab for every domain. Below is an example that checks every day at midnight if a certificate needs to be renewed and reloads relayd:

0 0 * * * acme-client domain1.tld && rcctl reload relayd
0 0 * * * acme-client domain2.tld && rcctl reload relayd

Todo And Summary

All in all, despite all of the problems, this project was a lot of fun even if I am still not entirely content with how it’s set up yet. One thing I would really like to still get working is HTTP -> HTTPS redirects, but only for paths that do not contain the ACME challenge directory – I haven’t yet figured out a good way of doing that. So for now, no automatic HTTP -> HTTPS redirects. I would also much like to have the backends talk to the VPS using HTTPS instead of plain HTTP as well at some point. I will also be adding a third backend server just for the heck of it and I will be moving away from using Cloudflare’s proxy and I’ll start with my Gitea instance soon.

This would have gone much faster if I had used the tools I am already familiar with (such as caddy or HAProxy), but I wanted to learn some new stuff and I think I definitely have! Even though I am fairly certain my setup will raise some eyebrows with some folk (sorry :<), it was still plenty of fun learning about OpenBSD, how it differs from Linux and how to use its built-in tools for making this loadbalancing web server / relay thing.

Full Config

Here are all the important config files with plenty of comments.

/etc/relayd.conf 
# ===== PORTS ===== #
# domain1.tld        -> 10000
# domain2.tld        -> 10001
# ================= #
#
# ===== SERVERS ===== #
# localhost -> current machine (obvs?)
# 10.5.0.11 -> OpenBSD VM via WireGuard
# ================= #


# ===== EXTERNAL IPS ===== #
ipv4="203.0.113.52"
ipv6="2001:db8:0:e291::1000:1"

# The following definitions are necessary ... for some reason? Even though the IPs are the same, using only one table (such as one called <webservers>)
# with multiple domains causes the routing to just ... not work. It is therefore seemingly needed to create a new table for every new domain / website / web root
table <servers_domain1> { 10.5.0.11 localhost }
table <servers_domain2> { 10.5.0.11 localhost }

# Because the encryption is handled only by relayd (so far, at least), we must make sure that ACME requests are handled by *only* one server
# I have designated this server to `localhost` because it just makes most sense, I would think.
# Same problem here as above - one table per domain / website / web root
table <acme_domain1> { localhost }
table <acme_domain2> { localhost }

http protocol "http_acme" {
        # Block all requests that aren't excplicitly allowed below
        # This is mostly so that going to the server's IP won't show anything
        block request header "Host" value "*"

        # Pass all regular HTTP requests to the designated ACME server
        # Didn't really work with many domains so, disabled for now
        #match request path "/.well-known/acme-challenge/**" forward to <acme>

        # Specifically allow the following domains to pass and route them to their specified servers
        # == DOMAIN1.TLD == #
        pass request header "Host" value "domain1.tld" forward to <acme_domain1>
        pass request header "Host" value "www.domain1.tld" forward to <acme_domain1>

        # == DOMAIN2.TLD == #
        pass request header "Host" value "domain2.tld" forward to <acme_domain2>
        pass request header "Host" value "www.domain2.tld" forward to <acme_domain2>
}

http protocol "https" {
        # Some recommended TCP options for SSL that I found
        tcp { nodelay, sack, socket buffer 65536, backlog 100 }

        # Block all requests that aren't excplicitly allowed below
        # This is mostly so that going to the server's IP won't show anything
        block request header "Host" value "*"

        # Load TLS keypairs
        # Keypairs have to be in the following format: /etc/ssl/hostname:port.crt & /etc/ssl/private/hostname:port.key
        tls keypair "domain1.tld"
        tls keypair "domain2.tld"

        # Specifically allow the following domains to pass and route them to their specified servers
        # == DOMAIN1.TLD == #
        pass request header "Host" value "domain1.tld" forward to <servers_domain1>
        pass request header "Host" value "www.domain1.tld" forward to <servers_domain1>

        # == DOMAIN2.TLD == #
        pass request header "Host" value "domain2.tld" forward to <servers_domain2>
        pass request header "Host" value "www.domain2.tld" forward to <servers_domain2>
}

# ===== RELAYS ===== #
#
# === HTTP RELAY ===#
# This relay is for sending regular HTTP requests to the backend servers
# It also handles ACME requests (as those obviously have to be handled by HTTP
relay "http_relay" {
    listen on $ipv4 port 80
    listen on $ipv6 port 80
    protocol "http_acme"

    # Forward ACME requests to the specified server + port
    forward to <acme_domain1> port 10000
    forward to <acme_domain2> port 10001
}

# === HTTPS RELAY ===#
# This relay is for HTTPS traffic
# Unlike with HTTP, having *two* listen statements (one for IPv4 and one for IPv6) in the same relay block… also just doesn't work for some reason. I have found someone else online who had the same problem and the fix was just to have two blocks.
relay "https_relay" {
    listen on $ipv4 port 443 tls
    protocol "https"

    forward to <servers_domain1> port 10000 mode loadbalance check tcp
    forward to <servers_domain2> port 10001 mode loadbalance check tcp
}

relay "https_relay_v6" {
    listen on $ipv6 port 443 tls
    protocol "https"

    forward to <servers_domain1> port 10000 mode loadbalance check tcp
    forward to <servers_domain2> port 10001 mode loadbalance check tcp
}
/etc/httpd.conf (On VPS) 
server "domain1.tld" {
    listen on localhost port 10000
    root "/htdocs/domain1.tld"

	location "/.well-known/acme-challenge/*" {
		root "/htdocs/acme"
		request strip 2
	}
}

server "domain2.tld" {
    listen on localhost port 10001
    root "/htdocs/domain2.tld"

	location "/.well-known/acme-challenge/*" {
		root "/htdocs/acme"
		request strip 2
	}
}
/etc/httpd.conf (WireGuard backends) 
server "domain1.tld" {
    listen on * port 10000
    root "/htdocs/domain1.tld"
}

server "domain2.tld" {
    listen on * port 10001
    root "/htdocs/domain2.tld"
}
/etc/acme-client.conf 
authority letsencrypt {
        api url "https://acme-v02.api.letsencrypt.org/directory"
        account key "/etc/acme/letsencrypt-privkey.pem"
}

authority letsencrypt-staging {
        api url "https://acme-staging-v02.api.letsencrypt.org/directory"
        account key "/etc/acme/letsencrypt-staging-privkey.pem"
}

domain domain1.tld {
	alternative names { www.domain1.tld }
	domain key "/etc/ssl/private/domain1.tld:443.key"
	domain full chain certificate "/etc/ssl/domain1.tld:443.crt"
	sign with letsencrypt
}

domain domain2.tld {
	alternative names { www.domain2.tld }
	domain key "/etc/ssl/private/domain2.tld:443.key"
	domain full chain certificate "/etc/ssl/domain2.tld:443.crt"
	sign with letsencrypt
}