How to create secure certificates

THIS ARTICLE IS CURRENTLY IN DRAFT - it is still being developed. Please feel free to add constructive comments and corrections below.

The problem 🔗︎

Argh! Why is it so hard to create and manage trusted certificates for “internal” services! While you can create your own “self-signed” certificates, all modern browsers now mark these as insecure and try to stop you from accessing them. This is wrong. Browsers should allow access to self-signed certificates if they point to a non-routable IP address (192.168.. or 10...* for example) or an invalid root domain such as *.something.local.

Also, we really probably don’t want to expose all of our internal servers to the bad, wide Internet - this is generally a really bad idea unless you are good at securing things and have the time to keep making sure they stay secure as things update.

Possible fixes 🔗︎

You could manually add a new (self-signed) root certificate to all devices needing access to your internal services so that your self-signed certificates are trusted - try getting that past the rest of the family!

The only other alternative is to use a trusted CA. Since I’m assuming you are doing this for testing or for use at home, I also assume that you don’t want to spend lots of money. Trusted certificates usually cost - a lot! Often US$100 per year or more.

However, there is one supplier that issues free trusted certificates. Let’s Encrypt. This is a great service for a great price. But it comes with some overheads.

It’s never simple 🔗︎

OK, so assuming we want to use Let’s Encrypt (LE), what issues do we now need to overcome?

1. Firstly, we have to have a publicly known domain address. You cannot issue a publicly trusted certificate to an IP address or non-routable domain name.

   Note that certificates are generally issued to specific domain names so that www.thing.com and thing.com are different names. We don’t want to mess with all of that all the time, especially if we are doing lots of tests. So we can now use a “wildcard” certificate for *.thing.com

2. Next, we have to have a way for the Let’s Encrypt servers to verify that our domain name is actually ours to do something with.

   By default, the LE service wants to have access back to our server in order to verify that it is ours. This would bring us back to the issue of exposing our server to the Internet, something that we prefer not to do unless we really have to.

   Thankfully LE now has an alternative called DNS-01. Unfortunately, this requires our DNS (Domain Name Service) to support a particular type of secure API. For that we can use Cloudflare or any of the other DNS services listed on the LE website.

3. Finally, we have to renew the certificate every 3 months since that is all that Let’s Encrypt allows us to keep a certificate for. This is a pain but it does have some security benefits.

Let’s look at the details of how to do this.

1. Get a domain 🔗︎

We need a domain and we need it to be one that we can, at least to some degree, control. So some of the free “dynamic DNS” services probably won’t cut it. However, we can get our very own domain for a few £/$ per year so go ahead and do that. Save yourself some pain in the next bit by using Cloudflare themselves to register your domain.

Once we have a domain, we need to let it be managed by a DNS that supports the DNS-01 verification API. You might be lucky in that your domain registrar already supports that. In most cases, they won’t. So now you have to hand over control of the DNS settings to someone like Cloudflare who’s free service is plenty for what we need. Your domain registrar will tell you how to change the name servers that control your domain. Don’t worry, your registrar remains the overall controller so renewals are not an issue.

As we will be using a wildcard certificate, we don’t need to worry about setting up specific names at this point.

Also, as we will be using DNS-01 verification, we don’t need to point anything at our public IP address. Use the DNS settings to point the default “A” DNS entry at a dummy IP address like 10.10.10.10. That’s fine and it means that we won’t be leaking any information about our private network.

2a. Install a client for Let’s Encrypt 🔗︎

There are lots of clients for LE, see the list on their website. For this post, I’m going to use a Raspberry Pi as I have one running permanently controlling my home automation system. You could also use a NAS or any PC. Even some routers such as the ever excellent Ubiquity EdgeRouter Lite can be used.

I am going to use a 3rd-party BASH (Linux command-line) script as this is a lot simpler than the official Python based script.

From your Pi, follow the instructions to install the script. Start a remote command line using an SSH client from a convenient computer. Log in using an ID that is allowed to do administration on the Pi. Then install the script using curl https://get.acme.sh | sh. Note that I didn’t bother to become root as I need to use the certificate with a Node.JS service that isn’t run globally as root (this is best practice, don’t run things as root as that opens up additional security issues). I also ran alias acme.sh=~/.acme.sh/acme.sh manually rather than logging out and back in again to pick up the defined alias.

2b. Get the Cloudflare API details 🔗︎

So, we’re ready now right? Not quite. First we need somewhere to actually run the client tool that will initially get our first certificate and that will then run periodically to renew the certificate.

As we are using DNS-01 validation with Cloudflare, we need the API access details. Jump ahead in the acme.sh instructions to the part on DNS API integration since that’s what we need in order to avoid exposing our internal servers to the Internet. We need a configuration file. The instructions are in the the dnsapi folder.

Log into your Cloudflare account, go to “My Profile” under the little person icon top-right. Pick up your verified email address then scroll down to the bottom “API Keys” & click on “View” against the “Global API Key”. While you are there, turn on 2-factor authentication to protect your account and services. Keep this information safe! If someone gets hold of it, they can change your DNS and other settings.

Issue the commands export CF_Key="sdfsdfsdfljlbjkljlkjsdfoiwje" && export CF_Email="[email protected]" which temporarily puts the security information into environment variables.

2c. Get our first certificate 🔗︎

Now we are ready to try and get our first certificate. Run the following from your SSH command line:

acme.sh --issue --dns dns_cf -d example.com -d *.example.com

Obviously replacing the domain names with your own. The entry that starts with * gives you a wildcard certificate so that you can use this certificate with any sub-domain like www.example.com or fred.example.com.

3. Certificate renewals 🔗︎

As mentioned, LE certificates expire every 90 days. Thankfully, the script we just ran not only does gets us our first certificate, it sets up a script to renew the certificate every 60 days - giving some extra time for the occasional renewal failures. You can adjust the renewal in the configuration file if you really want to. You don’t have to worry about restarting this if your device reboots.

Once the script has run for the first time, go ahead and run the following so that the script itself auto-updates:

acme.sh --upgrade --auto-upgrade

Using names instead of IP addresses 🔗︎

New we have a certificate that we can use for any service simply by referencing that service by name.

Note, though, that you can no longer reference your services by IP address.

For example pi.example.com but not 192.168.1.20.

Using an IP address will throw an error in modern browsers.

So a slight wrinkle in our effort to get rid of browser errors since, most home servers are access via an IP address not a name. How do we fix that? We have a couple of options.

1. Use your router’s DHCP or DNS service to define names for local services.

2. Install a DNS service on a local device.

3. Update the hosts file on every client device.

   This may work if you only have one or two laptops and nothing else but even then it is clunky. With mobile devices, it won’t be possible anyway.

Both 1 and 2 will require some configuration. What we are doing is creating names that point to local IP addresses. How you do this, depends on the router or DNS server you have.

In either case, we really want a fixed IP address to work with so that our Pi (or other device running our services) is always at the same address. All routers should have the ability to do this so look for the DHCP settings. You will need to know the MAC address of the device, it looks something like b8:27:eb:df:49:7e (you may see it with upper case letters and/or without the colons). The router’s DHCP pages will have a list of active “leases” that will show that information. You use that address, which is defined by a network interface on a device, to issue a fixed IP address via the DHCP service. Make sure that your fixed addresses don’t overlap with the range defined for DHCP to issue dynamically.

Sometimes, your DHCP server will let you define a name to go with this configuration. That’s great because you can now specify the name to include the domain that the certificate is issued to. e.g. pi.example.com.

The Ubiquiti EdgeRouter’s will let you define names manually using a Wizard called “DNS Host Names”. Some other routers may give you access to the router’s hosts file.

If you can’t do that on your router, you will need a DNS server that will let you define names to IP addresses. That’s a bit more complex and beyond the scope of this, already rather long, blog post I’m afraid.

Using your certificate 🔗︎

Now you have the certificate and can access your servers via names instead of IP addresses so you are ready - finally - to configure the services to use the certificates.

You can configure most TCP/IP based services to use TLS (Transport Layer Security) which is what we will mostly want our certificate for. Most people will be familiar with accessing web pages over HTTPS which is TLS applied to HTTP. But we can also use the certificate to secure communications for file transfers (FTPS or SFTP), Email (SMTPS, IMAPS, etc.).

Node-RED 🔗︎

Node-RED is a service built over NodeJS and ExpressJS. It creates a web server that we can secure using our certificate. The same certificate will also be used to help to secure websocket communications.

Once you’ve changed the settings below, remember to access Node-RED using the server name instead of the IP address. You will need to restart the Node-RED service.

settings.js 🔗︎

This file is where we configure Node-RED to use HTTPS. Note that the settings are the same as those from NodeJS so you can check out any other settings in the NodeJS documentation.

The settings.js file is found in your userDir folder which is generally ~/.node-red if installed according to the instructions on the Node-RED website. ~ is the “home” folder for the user ID running the Node-RED service.

...
module.exports = {
    ...
    const path = require('path');
    const fs   = require('fs');
    ...
    https: {
        // Don't forget to adjust the paths below
        // according to your installation
        key: fs.readFileSync(
            path.join(
                '..','.acme.sh','<server_name>', '<server_name>.key'
            )
        ),
        cert: fs.readFileSync(
            path.join(
                '..','.acme.sh','<server_name>', 'fullchain.cer'
            )
        ),
    },
    ...
}

Where <server_name> is something like pi2.example.com - whatever you have defined as the name associated with the IP address.

MQTT broker Mosquitto 🔗︎

NOTE: This section is not complete as I’ve not managed to get this working as yet.

Many IoT systems make use of MQTT for publish and subscribe handling of data from devices. The more we embed IoT into our lives, the more important it is to secure the MQTT brokers. By enabling encrypted communications via TLS and then configuring user ID’s and strong passcodes between devices and the broker, we can help make things a lot more secure.

Mosquitto is one of the most common MQTT brokers due to its small size and high performance.

On a Linux system, Mosquitto configuration files are found in /etc/mosquitto. Try not to edit /etc/mosquitto/mosquitto.conf, instead add your own file to /etc/mosquitto/conf.d. It will be loaded automatically and will not be overridden when Mosquitto is upgraded.

/etc/mosquitto/conf.d/custom.conf 🔗︎

Webmin 🔗︎

Webmin is a really helpful remote administration tool for Linux servers. Being web-based means that you can use your certificate with it. Replace the default certificate by going to Webmin > Webmin Configuration > SSL Settings (https://<domain_name>:10000/webmin/edit_ssl.cgi?xnavigation=1).

Then change the default Private key file path with the the one generated by the acme.sh script (it ends with .key). Also change the certificate file setting to be the file fullchain.cer in the same folder.

Now restart Webmin from the main Webmin configuration page or issue the command sudo systemctl restart webmin.

No more certificate errors as long as you remember to use your server name not IP address.

Web servers 🔗︎

There are so many posts about configuring any of the regular web servers with certificates that I’m not going to repeat them here.

Just remember that you have all the certificate and key files that you need so you only need to make the folder available (read-only) to the user ID that runs the web server. Then to configure the appropriate files.

Router (Ubiquiti EdgeRouter) 🔗︎

There is a dedicated script for EdgeOS which you may wish to use for simplicity. Otherwise, you will need to remember to securely transfer the fullchain.cer and xxx.key files to the router’s filing system each time they are updated.

You can change to custom key and certificate files using the “Config Tree” service / gui.

InfluxDB and Telegraf 🔗︎

InfluxDB is a timeseries database that is very efficient at recording data over time. Great for sensor data. All access is, by default, over HTTP so it is possible to configure it to use HTTPS.

Telegraf is from the same vendor as InfluxDB. It can also be configured to talk to InfluxDB over HTTPS.

Grafana 🔗︎

NOTE: Grafana seems to want to be able to write to something - either the folder or the cert/key files - that it shouldn’t do so currently is failing. I haven’t had time to resolve this as yet.

/etc/grafana/grafana.ini

[server]
# Protocol (http, https, socket)
protocol = https

# https certs & key file
cert_file = /home/pi/.acme.sh/<server_name>/fullchain.cer
cert_key = /home/pi/.acme.sh/<server_name>/<server_name>.key