it-security on Much Ado About IT

How to create secure certificates

Julian Knight — Sun, 14 Oct 2018 15:54:20 +0000

Generate certificates for Node-RED that are trusted by all modern browsers. This will let you access Node-RED (and other services) over an encrypted HTTPS link.

THIS ARTICLE IS CURRENTLY IN DRAFT - it is still being developed. Please feel free to add constructive comments and corrections below.

The problem

Argh! Why is it so hard to create and manage trusted certificates for “internal” services! While you can create your own “self-signed” certificates, all modern browsers now mark these as insecure and try to stop you from accessing them. This is wrong. Browsers should allow access to self-signed certificates if they point to a non-routable IP address (192.168.. or 10...* for example) or an invalid root domain such as *.something.local.

Also, we really probably don’t want to expose all of our internal servers to the bad, wide Internet - this is generally a really bad idea unless you are good at securing things and have the time to keep making sure they stay secure as things update.

Possible fixes

You could manually add a new (self-signed) root certificate to all devices needing access to your internal services so that your self-signed certificates are trusted - try getting that past the rest of the family!

The only other alternative is to use a trusted CA. Since I’m assuming you are doing this for testing or for use at home, I also assume that you don’t want to spend lots of money. Trusted certificates usually cost - a lot! Often US$100 per year or more.

However, there is one supplier that issues free trusted certificates. Let’s Encrypt. This is a great service for a great price. But it comes with some overheads.

It’s never simple

OK, so assuming we want to use Let’s Encrypt (LE), what issues do we now need to overcome?

Firstly, we have to have a publicly known domain address. You cannot issue a publicly trusted certificate to an IP address or non-routable domain name.

Note that certificates are generally issued to specific domain names so that www.thing.com and thing.com are different names. We don’t want to mess with all of that all the time, especially if we are doing lots of tests. So we can now use a “wildcard” certificate for *.thing.com
Next, we have to have a way for the Let’s Encrypt servers to verify that our domain name is actually ours to do something with.

By default, the LE service wants to have access back to our server in order to verify that it is ours. This would bring us back to the issue of exposing our server to the Internet, something that we prefer not to do unless we really have to.

Thankfully LE now has an alternative called DNS-01. Unfortunately, this requires our DNS (Domain Name Service) to support a particular type of secure API. For that we can use Cloudflare or any of the other DNS services listed on the LE website.
Finally, we have to renew the certificate every 3 months since that is all that Let’s Encrypt allows us to keep a certificate for. This is a pain but it does have some security benefits.

Let’s look at the details of how to do this.

1. Get a domain

We need a domain and we need it to be one that we can, at least to some degree, control. So some of the free “dynamic DNS” services probably won’t cut it. However, we can get our very own domain for a few £/$ per year so go ahead and do that. Save yourself some pain in the next bit by using Cloudflare themselves to register your domain.

Once we have a domain, we need to let it be managed by a DNS that supports the DNS-01 verification API. You might be lucky in that your domain registrar already supports that. In most cases, they won’t. So now you have to hand over control of the DNS settings to someone like Cloudflare who’s free service is plenty for what we need. Your domain registrar will tell you how to change the name servers that control your domain. Don’t worry, your registrar remains the overall controller so renewals are not an issue.

As we will be using a wildcard certificate, we don’t need to worry about setting up specific names at this point.

Also, as we will be using DNS-01 verification, we don’t need to point anything at our public IP address. Use the DNS settings to point the default “A” DNS entry at a dummy IP address like 10.10.10.10. That’s fine and it means that we won’t be leaking any information about our private network.

2a. Install a client for Let’s Encrypt

There are lots of clients for LE, see the list on their website. For this post, I’m going to use a Raspberry Pi as I have one running permanently controlling my home automation system. You could also use a NAS or any PC. Even some routers such as the ever excellent Ubiquity EdgeRouter Lite can be used.

I am going to use a 3rd-party BASH (Linux command-line) script as this is a lot simpler than the official Python based script.

From your Pi, follow the instructions to install the script. Start a remote command line using an SSH client from a convenient computer. Log in using an ID that is allowed to do administration on the Pi. Then install the script using curl https://get.acme.sh | sh. Note that I didn’t bother to become root as I need to use the certificate with a Node.JS service that isn’t run globally as root (this is best practice, don’t run things as root as that opens up additional security issues). I also ran alias acme.sh=~/.acme.sh/acme.sh manually rather than logging out and back in again to pick up the defined alias.

2b. Get the Cloudflare API details

So, we’re ready now right? Not quite. First we need somewhere to actually run the client tool that will initially get our first certificate and that will then run periodically to renew the certificate.

As we are using DNS-01 validation with Cloudflare, we need the API access details. Jump ahead in the acme.sh instructions to the part on DNS API integration since that’s what we need in order to avoid exposing our internal servers to the Internet. We need a configuration file. The instructions are in the the dnsapi folder.

Log into your Cloudflare account, go to “My Profile” under the little person icon top-right. Pick up your verified email address then scroll down to the bottom “API Keys” & click on “View” against the “Global API Key”. While you are there, turn on 2-factor authentication to protect your account and services. Keep this information safe! If someone gets hold of it, they can change your DNS and other settings.

Issue the commands export CF_Key="sdfsdfsdfljlbjkljlkjsdfoiwje" && export CF_Email="xxxx@sss.com" which temporarily puts the security information into environment variables.

2c. Get our first certificate

Now we are ready to try and get our first certificate. Run the following from your SSH command line:

acme.sh --issue --dns dns_cf -d example.com -d *.example.com

Obviously replacing the domain names with your own. The entry that starts with * gives you a wildcard certificate so that you can use this certificate with any sub-domain like www.example.com or fred.example.com.

3. Certificate renewals

As mentioned, LE certificates expire every 90 days. Thankfully, the script we just ran not only does gets us our first certificate, it sets up a script to renew the certificate every 60 days - giving some extra time for the occasional renewal failures. You can adjust the renewal in the configuration file if you really want to. You don’t have to worry about restarting this if your device reboots.

Once the script has run for the first time, go ahead and run the following so that the script itself auto-updates:

acme.sh --upgrade --auto-upgrade

Using names instead of IP addresses

New we have a certificate that we can use for any service simply by referencing that service by name.

Note, though, that you can no longer reference your services by IP address.

For example pi.example.com but not 192.168.1.20.

Using an IP address will throw an error in modern browsers.

So a slight wrinkle in our effort to get rid of browser errors since, most home servers are access via an IP address not a name. How do we fix that? We have a couple of options.

Use your router’s DHCP or DNS service to define names for local services.
Install a DNS service on a local device.
Update the hosts file on every client device.

This may work if you only have one or two laptops and nothing else but even then it is clunky. With mobile devices, it won’t be possible anyway.

Both 1 and 2 will require some configuration. What we are doing is creating names that point to local IP addresses. How you do this, depends on the router or DNS server you have.

In either case, we really want a fixed IP address to work with so that our Pi (or other device running our services) is always at the same address. All routers should have the ability to do this so look for the DHCP settings. You will need to know the MAC address of the device, it looks something like b8:27:eb:df:49:7e (you may see it with upper case letters and/or without the colons). The router’s DHCP pages will have a list of active “leases” that will show that information. You use that address, which is defined by a network interface on a device, to issue a fixed IP address via the DHCP service. Make sure that your fixed addresses don’t overlap with the range defined for DHCP to issue dynamically.

Sometimes, your DHCP server will let you define a name to go with this configuration. That’s great because you can now specify the name to include the domain that the certificate is issued to. e.g. pi.example.com.

The Ubiquiti EdgeRouter’s will let you define names manually using a Wizard called “DNS Host Names”. Some other routers may give you access to the router’s hosts file.

If you can’t do that on your router, you will need a DNS server that will let you define names to IP addresses. That’s a bit more complex and beyond the scope of this, already rather long, blog post I’m afraid.

Using your certificate

Now you have the certificate and can access your servers via names instead of IP addresses so you are ready - finally - to configure the services to use the certificates.

You can configure most TCP/IP based services to use TLS (Transport Layer Security) which is what we will mostly want our certificate for. Most people will be familiar with accessing web pages over HTTPS which is TLS applied to HTTP. But we can also use the certificate to secure communications for file transfers (FTPS or SFTP), Email (SMTPS, IMAPS, etc.).

Node-RED

Node-RED is a service built over NodeJS and ExpressJS. It creates a web server that we can secure using our certificate. The same certificate will also be used to help to secure websocket communications.

Once you’ve changed the settings below, remember to access Node-RED using the server name instead of the IP address. You will need to restart the Node-RED service.

settings.js

This file is where we configure Node-RED to use HTTPS. Note that the settings are the same as those from NodeJS so you can check out any other settings in the NodeJS documentation.

The settings.js file is found in your userDir folder which is generally ~/.node-red if installed according to the instructions on the Node-RED website. ~ is the “home” folder for the user ID running the Node-RED service.

...
module.exports = {
 ...
 const path = require('path');
 const fs = require('fs');
 ...
 https: {
 // Don't forget to adjust the paths below
 // according to your installation
 key: fs.readFileSync(
 path.join(
 '..','.acme.sh','<server_name>', '<server_name>.key'
 )
 ),
 cert: fs.readFileSync(
 path.join(
 '..','.acme.sh','<server_name>', 'fullchain.cer'
 )
 ),
 },
 ...
}

Where <server_name> is something like pi2.example.com - whatever you have defined as the name associated with the IP address.

MQTT broker Mosquitto

NOTE: This section is not complete as I’ve not managed to get this working as yet.

Many IoT systems make use of MQTT for publish and subscribe handling of data from devices. The more we embed IoT into our lives, the more important it is to secure the MQTT brokers. By enabling encrypted communications via TLS and then configuring user ID’s and strong passcodes between devices and the broker, we can help make things a lot more secure.

Mosquitto is one of the most common MQTT brokers due to its small size and high performance.

On a Linux system, Mosquitto configuration files are found in /etc/mosquitto. Try not to edit /etc/mosquitto/mosquitto.conf, instead add your own file to /etc/mosquitto/conf.d. It will be loaded automatically and will not be overridden when Mosquitto is upgraded.

/etc/mosquitto/conf.d/custom.conf

Webmin

Webmin is a really helpful remote administration tool for Linux servers. Being web-based means that you can use your certificate with it. Replace the default certificate by going to Webmin > Webmin Configuration > SSL Settings (https://<domain_name>:10000/webmin/edit_ssl.cgi?xnavigation=1).

Then change the default Private key file path with the the one generated by the acme.sh script (it ends with .key). Also change the certificate file setting to be the file fullchain.cer in the same folder.

Now restart Webmin from the main Webmin configuration page or issue the command sudo systemctl restart webmin.

No more certificate errors as long as you remember to use your server name not IP address.

Web servers

There are so many posts about configuring any of the regular web servers with certificates that I’m not going to repeat them here.

Just remember that you have all the certificate and key files that you need so you only need to make the folder available (read-only) to the user ID that runs the web server. Then to configure the appropriate files.

Router (Ubiquiti EdgeRouter)

There is a dedicated script for EdgeOS which you may wish to use for simplicity. Otherwise, you will need to remember to securely transfer the fullchain.cer and xxx.key files to the router’s filing system each time they are updated.

You can change to custom key and certificate files using the “Config Tree” service / gui.

InfluxDB and Telegraf

InfluxDB is a timeseries database that is very efficient at recording data over time. Great for sensor data. All access is, by default, over HTTP so it is possible to configure it to use HTTPS.

Telegraf is from the same vendor as InfluxDB. It can also be configured to talk to InfluxDB over HTTPS.

Grafana

NOTE: Grafana seems to want to be able to write to something - either the folder or the cert/key files - that it shouldn’t do so currently is failing. I haven’t had time to resolve this as yet.

/etc/grafana/grafana.ini

[server]
# Protocol (http, https, socket)
protocol = https

# https certs & key file
cert_file = /home/pi/.acme.sh/<server_name>/fullchain.cer
cert_key = /home/pi/.acme.sh/<server_name>/<server_name>.key

How to secure Node-RED

Julian Knight — Sun, 30 Sep 2018 13:48:14 +0000

Node-RED is increasingly used in situations that require reasonable security. Up to now, however, the information required to secure it correctly has been fragmented. This article aims to give an end-to-end outline to enable anyone to secure their installation.

THIS ARTICLE IS CURRENTLY IN DRAFT - it is still being developed. Please feel free to add constructive comments and corrections below.

I am going to attempt to describe the various options and configurations that you will need to think about in order to secure an instance of Node-RED. This is probably going to be a long article!

Please see my other articles Making Node-RED available over the Internet and Secure Home Automation Controls via a Telegram Bot for other ideas about the security of Node-RED in regard to use over the Internet and how to avoid having to worry about some of the issues dealt with here.

Warning and disclaimer

This is my best view of securing Node-RED. I’ve not been a professional developer in a long time and I am not a professional security analyst (I’m a IT technology and information security manager).

So you must not assume that this article covers every issue. You must also not assume, even if you follow every best practice, that an instance of Node-RED will be “Secure”. It may not be for many reasons.

Get your system and its infrastructure tested to destruction by professionals.

Hopefully though, this article will at least let you see the issues and have a reasonable go at making your instance of Node-RED a little more secure.

Terminology

A few words and phrases that will be used along with the least technical descriptions that I can come up with.

Term	Meaning
Encryption	Changing human readable information into something that requires a special key before it can be understood.
Authentication	Providing a secured identifier to a system to prove that the person accessing the system is who they claim to be.
Authorisation	Controlling what information a user of a system is allowed to see and what actions they are allowed to take.
TLS	A defined protocol for securing connections between systems and applications. In basic use, it provides a minimum level of authentication of a server (via a certificate chain of trust) and facilitates encrypted communications with that server.It also has additional mechanisms that will also provide a minimum level of trust of the client application. However, this is rarely used in Internet web applications.TLS may be used to secure many different application and system interactions, not just between a browser and a server.
HTTP(S)	The main web protocol that lets browsers get human readable information from web servers. HTTPS is HTTP secured with TLS (used to also allow a protocol called SSL but that is deprecated as insecure). Ideally, every connection to a server from a browser should be over HTTPS since the use of HTTP allows some pretty bad things to happen.

The basic architecture of Node-RED

To help us understand what needs to be secured, here is the basic technical architecture of Node-RED. From the outside in. This is only one way to view the architecture of course, I’ve tried to keep things as simple as possible.

Your flows, Dashboard and other pages served by Node-RED

This is the “code” that you (or your users/administrators) have put together.

Node-RED is a general purpose tool and so is capable of creating web pages and various other types of connections such as websockets, TCP/UDP connections and much more.

Anything that is created at this level either must be secured by Node-RED’s settings or by your code.
The Node-RED runtime API

This is the bit of magic that makes everything work. At some point, this will be independent to the administration side but at the moment (~ v0.19) it isn’t.
The Node-RED Administration user interface

This is a web application (web page plus websocket connections) that lets you build your own applications using Node-RED. It uses various 3rd-party libraries to do some of the heavy lifting (D3, JQuery, etc.)
The Node-RED core and contributed nodes

Here is all of the clever custom code, generally tucked out of the way so you don’t need to worry about it.
ExpressJS

ExpressJS is a library for NodeJS that does the heavy lifting of providing a web server.
NodeJS

NodeJS is a collection of JavaScript and C code that lets people build application servers across different operating systems. It lets people use JavaScript (typically a browser language) to create their server applications.
Operating System

Apart from the hardware, we are now at the bottom of the architecture.

No point in securing everything else unless you have secured things at this level.

In order to secure Node-RED, we have to pay some attention to all of these layers. Thankfully, the guys to created Node-RED have thought about many of the details and made it at least somewhat easier to provide a basic level of security.

Settings that you need to think about

Other considerations

NodeJS and package vulnerabilities

As Node-RED itself needs to have backwards compatibility with older versions of NodeJS, it may be forced to use packages (AKA modules or libraries) that are now out-of-date. Similarly, nodes that you rely on may not get updated as regularly as needed to stay ahead of vulnerabilities.

NodeJS itself is very rapidly changing and unless you are updating it regularly (e.g. weekly) it may also have outstanding vulnerabilities.

To mitigate these issues, you need to:

Update NodeJS weekly or daily in sensitive installations. You will also want a set of regression tests - test that are run after every update - if you want users to be able to rely on rapid update cycles.
npm itself may have vulnerabilities and may need to be updated out of cycle from NodeJS.
Regularly run npm outdated and npm audit on both the location you install Node-RED (global by default) and the location where you install contributed nodes. Update nodes regularly or consider the impact of nodes that have dependencies failing audits. Note that npm audit is only available on newer versions of npm.

Issues and limitations

Specifics

References

Node-RED Official documentation on security
Making Node-RED available over the Internet
Secure Home Automation Controls via a Telegram Bot
Why should we use HTTPS?
1. Google’s take
2. Cloudflare’s take
3. The US Government’s take
4. A leading security experts take (Troy Hunt)

Presentation: Security and Governance in the Cloud

Julian Knight — Mon, 28 Nov 2016 12:54:25 +0000

Here is a presentation that I did recently for NHS CIO’s and CCIO’s. It is all about how NHS England has followed a journey to cloud services and the IT Security & Information Governance issues we had to deal with along the way. It tries to also show other NHS organisations how they might work towards similar aims.