TLS certificates for internal services done right

Title is a bit clickbait-y — YMMV, but let me explain why I think “this is the
way”. Let’s start with a simple example — we have a server which hosts bunch
of HTTP services. Some of those services are external, others internal. In
order to reach the internal ones you need to be connected to the VPN.

For the sake of simplicity let’s consider we have two choices:

1. We use top-level domain restricted by ICANN for private
   use
   — e.g. .internal.
2. We use a public apex domain that we own — e.g. tuxnet.dev

Grafana would be our example internal app. Let’s assume that it’s reachable on
10.0.1.10 internal IP address and our VPN has DNS resolver features.

What’s wrong with .internal then?

We could simply create a DNS record of type “A” that resolves to 10.0.1.10
internal IP address — e.g. grafana.tuxnet.internal. But then if we don’t
want it to be a plain text HTTP service we would need to create a self-signed
certificate.

The good part is that there are plenty tutorials that show you how to do this
(e.g. this
one).
The ugly part is that suddenly every HTTP client should be configured to trust
this self-signed certificate. Alternatively we could just tell our users to
ignore the TLS certificate errors ¯\_(ツ)_/¯.

How to do it “the right way”

Meet the “split-horizon DNS” configuration. For public DNS resolvers our
grafana.tuxnet.dev domain resolves to a public IP and for clients connected
to the VPN this domain resolves to an internal IP.

The good part is that since it resolves to a public IP we can use some public
CA like Let’s Encrypt or ZeroSSL. The ugly part is that we still need some WAF
rejecting traffic that does not originate from the VPN.

Considering pros and cons of both solutions I think it’s much easier to set up
a WAF in one place (on our server) than to install self-signed certificate on
every machine that joins our internal network (… or advising our users to
suppress the TLS errors).

”Talk is cheap show me the code”

We have theory now, time to get our hands dirty. We will need:

1. A VPN with DNS resolver features — I choose NetBird.
2. ACME client for issuing a certificate — I choose acme.sh.
3. A reverse proxy with WAF features in front of our grafana — I choose nginx.

If you’ve read my other blog posts you probably noticed that I am a fan of
NetBird (sorry Tailscale). Thanks to Custom
Zones feature, NetBird does
all the heavy lifting required for “split-horizon DNS” for us. By using user
groups or peer
groups we can
selectively apply Custom Zones so that server uses public DNS resolver for
grafana.tuxnet.dev.

Why exclude server from that custom zone? It’s not required unless we want to
use http-01 challange. Using
other methods is also possible but for the sake of this blog post I choose
http-01. Ok, let’s get the certificate now with:

acme.sh --issue -d grafana.tuxnet.dev --server letsencrypt --standalone

acme.sh is quite flexible and has a lot of
modes.
The cool part about the standalone mode (enabled with --standalone flag) is
that our nginx doesn’t have to listen on port 80 at all. This port becomes
“active” only when acme.sh gets the certificate.

Ok now we can put nginx into action. This is our config:

upstream grafana 
    sha256sum "$1" 

map $http_upgrade $connection_upgrade 💬

server 
    sha256sum "$1" 

There is one key setting in this configuration worth explaining — listen our-server.netbird.cloud:443 ssl;. We are binding to the VPN network interface
of our server. Instead of our-server.netbird.cloud this could be a VPN IP
address. In practice this will reject any traffic to grafana.tuxnet.dev that
originates from public internet — this is our Web Access Firewall.

Security is about layers (just like onions and ogres). Our first layer is
split-horizon DNS but if for whatever reason it fails or is cleverly bypassed
we have a 2nd layer – WAF – that should hold the line.

Last but not least — certificate auto renewal. acme.sh has a out-of-the box
--cron flag. Now we need a daily cron job that will call

acme.sh --cron

acme.sh automatically chooses which certificates should be renewed. All we
need to do is to make sure that cron job copies new certificates into the
location defined in nginx’s ssl_certificate and ssl_certificate_key. Nginx
also needs to be reloaded to use new certificates. Our cron job could look like
this:

main() ⚡

refresh_certs() 
    sha256sum "$1" 

sync_api_tuxnet_dev_certs() 🔥

sync_internal_tuxnet_dev_certs() {
    local green=$(get_checksum "$INTERNAL_SRC_KEY")
    local blue=$(get_checksum "$INTERNAL_DST_KEY")
    local key_allowed_group=www-data

    if [[ "$green" != "$blue" ]]; then
        sync_certs "$INTERNAL_SRC_KEY" "$INTERNAL_DST_KEY" "$INTERNAL_SRC_CERT" "$INTERNAL_DST_CERT" "$key_allowed_group"
    fi
}

reload_nginx() {
    nginx -t
    systemctl reload nginx
}

get_checksum() {
    sha256sum "$1" | cut -d' ' -f1
}

sync_certs() {
    local src_key="$1"
    local dst_key="$2"
    local src_cert="$3"
    local dst_cert="$4"
    local key_allowed_group="$5"

    cp -v "$src_key" "$dst_key"
    logger "synced $dst_key"

    cp -v "$src_cert" "$dst_cert"
    logger "synced $dst_cert"

    chown root:${key_allowed_group} "$dst_key"
    chown root:ssl-cert "$dst_cert"
    chmod 640 "$dst_key" "$dst_cert"

}

main "$@"

Small comment about setcap CAP_NET_BIND_SERVICE=+ep /usr/bin/socat1.
acme.sh in standalone mode uses socat in order to listen on port
80. On one hand we don’t want to run acme.sh as root if we don’t have to. On
the other hand port 80 is one of the “privileged ports” and by default,
privileged ports can’t be bound to non-root processes. This is where
CAP_NET_BIND_SERVICE=+ep helps. You can check this
article if you’re
interested more in this topic.

Life’s good, we have a TLS that just works — no matter if the service is
internal or external, no matter if it’s “next day” or “next year”.

Bonus — SANs and CNAMEs

What if we have more internal services? What if we want them under separate
subdomains? Do we need to generate separate certificates for each? The answer
is “no” and we have two solutions for that:

1. Wildcard certificates — I’m not a fan of it because of security
   implications.
2. TLS SAN (Subject Alternative Name) — where apart of CN (Common Name) we define
   mentioned SANs. For more info check https://www.ssl.com/faqs/common-name/

So in practice we can create “A” record for internal.tuxnet.dev and then
“CNAME” records for grafana.tuxnet.dev and e.g. analytics.tuxnet.dev that
resolve to internal.tuxnet.dev. Then we generate one certificate like this:

acme.sh --issue -d internal.tuxnet.dev -d grafana.tuxnet.dev -d analytics.tuxnet.dev --server letsencrypt --standalone

where its details look like this:

echo | openssl s_client -connect internal.tuxnet.dev:443 2>/dev/null | openssl x509 -noout -subject -ext subjectAltName

subject=CN=internal.tuxnet.dev
X509v3 Subject Alternative Name: 
    DNS:analytics.tuxnet.dev, DNS:grafana.tuxnet.dev, DNS:internal.tuxnet.dev

All we need to do now is to re-use the same certificate for different server
definitions in our nginx configs.

Summary

We’ve learned how to securely set up TLS certificates for internal services
without creating TLS issues for http clients downstream. All thanks to
split-horizon DNS, WAF and ACME protocol. All for free!

Available at https://news.ycombinator.com/item?id=48846995