Aymeric on Software

Because we needed another of these blogs...

Let’s Encrypt With Nginx

The Objectives

One of the major hurdles hampering the deployment of HTTPS on smaller websites like this one, has always been the price of certificates. As much I would have liked to get one, I could hardly justify the cost. That’s why a year ago, when the Let’s Encrypt project was announced, with the promise of free domain certificates, I was particularly excited. I decided to migrate my websites as soon as the project reached the public beta phase, last week.

Even if your site does not require HTTPS for security reasons, it is worth considering:

  • to provide additional privacy to your users. Unscrupulous ISPs have started to inject javascript or cookies into third party HTTP pages. They cannot do this with HTTPS pages.

  • to benefit from the performance improvements of HTTP/2. Even though in theory HTTP/2 does not require TLS, all major browsers have decided to boycott the “plaintext” version of the protocol.

In this blog post, I intend to explain how I migrated this Octopress blog hosted with nginx from HTTP to HTTPS and obtained an A+ Grade from SSL Labs.

Intall Let’s Encrypt

This part of the process is very well covered by the Let’s Encrypt documentation. Since there are no packages for Ubuntu Server LTS at the moment, I used the source code approach. In this case, we end up using the letsencrypt-auto command instead of letsencrypt directly. letsencrypt-auto is a wrapper script that ensures that the tool and its dependencies are up to date, prior to running any letsencrypt commands.

1
2
3
4
5
6
7
8
9
# Optionally install git
sudo apt-get install git-core

# Checkout the let's encrypt git repository into /srv/letsencrypt
git clone https://github.com/letsencrypt/letsencrypt /srv/letsencrypt

# Run the tool at least once
cd /srv/letsencrypt
./letsencrypt-auto --help

When I did this, I happened to have some kind of InsecurePlatform Python warning.

1
2
3
4
5
Creating virtual environment...
Updating letsencrypt and virtual environment dependencies...../home/aymericb/.local/share/letsencrypt/local/lib/python2.7/site-packages/pip/_vendor/requests/packages/urllib3/util/ssl_.py:90: InsecurePlatformWarning: A true SSLContext object is not available. This prevents urllib3 from configuring SSL appropriately and may cause certain SSL connections to fail. For more information, see https://urllib3.readthedocs.org/en/latest/security.html#insecureplatformwarning.
  InsecurePlatformWarning
./home/aymericb/.local/share/letsencrypt/local/lib/python2.7/site-packages/pip/_vendor/requests/packages/urllib3/util/ssl_.py:90: InsecurePlatformWarning: A true SSLContext object is not available. This prevents urllib3 from configuring SSL appropriately and may cause certain SSL connections to fail. For more information, see https://urllib3.readthedocs.org/en/latest/security.html#insecureplatformwarning.
  InsecurePlatformWarning

This warning comes from urllib3. I ignored it and everything was fine!

Getting the Certificate

At this stage you are ready to request a certificate for the website. Let’s Encrypt uses the ACME protocol, which requires serving a bespoke generated file on the web server to confirm ownership of a domain. The letsencrypt tool aims to automate the whole setup, so that writing ./letsencrypt-auto --apache -d blog.barthe.ph should be sufficient to get the entire web server up and running, if you happen to use Apache.

Unfortunately, the plugin for nginx is not stable yet and cannot be used. The setup will be slightly more complicated. We will use the --webroot switch, telling letsencrypt where to find our web server files, and afterwards, we will have to modify the nginx configuration files ourselves.

Getting the Let’s Encrypt certificate is the easy part:

1
./letsencrypt-auto certonly --webroot -w /srv/blog.barthe.ph/www -d blog.barthe.ph

You are asked to provide an email address (used to warn you when your certificate is about to expire), and agree to the terms and conditions.

Afterwards, you need to manually edit the nginx configuration files using your favorite editor (e.g. nano -w /etc/nginx/sites-available/blog.barthe.ph). The TLS certificate and the private keys are located in /etc/letsencrypt/live/blog.barthe.ph.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
# Add the following to redirect HTTP requests to HTTPS
server {
        server_name blog.barthe.ph;
        listen 80;
        return 301 https://$server_name$request_uri;
}
# Modify the existing HTTP setup to use HTTPS
server {
        server_name blog.barthe.ph;
        listen 443 ssl;

        # Basic TLS setup
        ssl_certificate /etc/letsencrypt/live/blog.barthe.ph/fullchain.pem;
        ssl_certificate_key /etc/letsencrypt/live/blog.barthe.ph/privkey.pem;
        ssl_protocols TLSv1 TLSv1.1 TLSv1.2;

        # keep the rest of the site configuration unchanged
        # [...]
 }

You’ll notice that in the configuration, I chose to support TLS 1.0 and greater. I wanted to be more aggressive but TLS 1.1 is not that well supported. It does not work on OS X 10.8 for instance. However, I did drop support for the older SSLv3 protocol (which preceded TLS 1.0) and is still supported by most websites. The consequence is that my site is no longer compatible with older versions of Internet Explorer (IE6 to IE8). Supporting SSLv3 without weakening the security of modern TLS 1.2 browsers is becoming increasingly difficult because of downgrade attacks such as POODLE.

In order for the changes to take effect you should invoke sudo service nginx reload. If you botched the configuration file, the site will continue running with the existing configuration and you should get an error in /var/log/nginx/error.log.

Once that’s done, you should have a web server running with HTTPS, and that redirects older HTTP URLs to their HTTPS equivalents. W00t!

A+ Grade Security

When testing the Octopress server on different browsers, I got some warnings about mixed content. It means that although my website was configured to serve its content through HTTPS, it still referenced URLs that used HTTP, either for internal URLs or externals URLs (such as Google Fonts, etc…). Chrome in particular was the most strict of all the browsers about this. After spending some time searching and replacing all references to HTTP URLs, I managed to fix all the warnings.

The next step was to edit /etc/nginx/sites-available/blog.barthe.ph to improve performance, by enabling TLS session caching:

1
2
ssl_session_cache shared:SSL:10m;
ssl_session_timeout 20m;

Let’s now focus on improving security and get an A+ grade on SSL Labs.

Let’s start by overriding the default prime used for Diffie Hellman. The default prime is usually too small, and also since it is a shared prime, some pre-computed attacks like Logjam are possible. Beware, the openssl command will take a long time; from 20 minutes to a few hours.

1
2
3
4
# Be patient…
openssl dhparam -out /etc/ssl/private/dhparams_4096.pem 4096

# Add "ssl_dhparam /etc/ssl/private/dhparams_4096.pem;" to nginx config

Now, let’s add support for OCSP stapling. OCSP is a mechanism that can be used by the browser to confirm that a certificate has not been revoked. This normally involves an extra connection to the certificate authority, unless the website uses stapling.

1
2
3
ssl_stapling on;
ssl_stapling_verify on;
ssl_trusted_certificate /etc/letsencrypt/live/blog.barthe.ph/fullchain.pem;

Let’s enable HSTS. This prevents a browser that connected to the website at least once before, from ever accepting an HTTP connection from the same domain.

1
add_header Strict-Transport-Security max-age=31536000;  # Valid for 1 year

Finally let’s massage the cipher suite to only use safe and secure ciphers.

1
2
ssl_prefer_server_ciphers on;
ssl_ciphers 'ECDHE-ECDSA-AES256-GCM-SHA384:... SEE BELOW';

In order to obtain the list of ciphers separated by :, use openssl ciphers -V to dump all available ciphers. Then do as follow:

  • Delete all references to DES, 3DES and RC4, and just keep AES ciphers.

  • Delete all references to MD5.

  • Put all ciphers that use SHA1, whose name ends with _SHA and are SSLv3 at the bottom.

  • Put ciphers with larger key lengths on top.

  • Put ciphers with forward secrecy on top. That would be ECDHE or DHE, the DH stands for Diffie-Hellman, and the E for Ephemeral, which means a new key is negotiated for each connection. Stealing the private key will not allow an attacked to decrypt past TLS sessions.

  • Favor GCM over CBC as a mode of operation. This is faster, and CBC brought security issues like POODLE in the past.

  • Favor Elleptic Curves (anything with EC) over plain RSA, DSA, or DHE. Elliptic curves are faster because they require smaller primes than traditional crypto, and as far as we know the security behind the maths is solid.

Here’s my list:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
0xC0,0x2C - ECDHE-ECDSA-AES256-GCM-SHA384 TLSv1.2 Kx=ECDH     Au=ECDSA Enc=AESGCM(256) Mac=AEAD
0xC0,0x30 - ECDHE-RSA-AES256-GCM-SHA384 TLSv1.2 Kx=ECDH     Au=RSA  Enc=AESGCM(256) Mac=AEAD
0xC0,0x24 - ECDHE-ECDSA-AES256-SHA384 TLSv1.2 Kx=ECDH     Au=ECDSA Enc=AES(256)  Mac=SHA384
0xC0,0x28 - ECDHE-RSA-AES256-SHA384 TLSv1.2 Kx=ECDH     Au=RSA  Enc=AES(256)  Mac=SHA384
0x00,0xA3 - DHE-DSS-AES256-GCM-SHA384 TLSv1.2 Kx=DH       Au=DSS  Enc=AESGCM(256) Mac=AEAD
0x00,0x9F - DHE-RSA-AES256-GCM-SHA384 TLSv1.2 Kx=DH       Au=RSA  Enc=AESGCM(256) Mac=AEAD
0x00,0x6B - DHE-RSA-AES256-SHA256   TLSv1.2 Kx=DH       Au=RSA  Enc=AES(256)  Mac=SHA256
0x00,0x6A - DHE-DSS-AES256-SHA256   TLSv1.2 Kx=DH       Au=DSS  Enc=AES(256)  Mac=SHA256
0xC0,0x2B - ECDHE-ECDSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH     Au=ECDSA Enc=AESGCM(128) Mac=AEAD
0xC0,0x2F - ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH     Au=RSA  Enc=AESGCM(128) Mac=AEAD
0xC0,0x23 - ECDHE-ECDSA-AES128-SHA256 TLSv1.2 Kx=ECDH     Au=ECDSA Enc=AES(128)  Mac=SHA256
0xC0,0x27 - ECDHE-RSA-AES128-SHA256 TLSv1.2 Kx=ECDH     Au=RSA  Enc=AES(128)  Mac=SHA256
0x00,0xA2 - DHE-DSS-AES128-GCM-SHA256 TLSv1.2 Kx=DH       Au=DSS  Enc=AESGCM(128) Mac=AEAD
0x00,0x9E - DHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=DH       Au=RSA  Enc=AESGCM(128) Mac=AEAD
0x00,0x67 - DHE-RSA-AES128-SHA256   TLSv1.2 Kx=DH       Au=RSA  Enc=AES(128)  Mac=SHA256
0x00,0x40 - DHE-DSS-AES128-SHA256   TLSv1.2 Kx=DH       Au=DSS  Enc=AES(128)  Mac=SHA256          
0xC0,0x2E - ECDH-ECDSA-AES256-GCM-SHA384 TLSv1.2 Kx=ECDH/ECDSA Au=ECDH Enc=AESGCM(256) Mac=AEAD
0xC0,0x32 - ECDH-RSA-AES256-GCM-SHA384 TLSv1.2 Kx=ECDH/RSA Au=ECDH Enc=AESGCM(256) Mac=AEAD
0xC0,0x26 - ECDH-ECDSA-AES256-SHA384 TLSv1.2 Kx=ECDH/ECDSA Au=ECDH Enc=AES(256)  Mac=SHA384
0xC0,0x2A - ECDH-RSA-AES256-SHA384  TLSv1.2 Kx=ECDH/RSA Au=ECDH Enc=AES(256)  Mac=SHA384
0xC0,0x31 - ECDH-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH/RSA Au=ECDH Enc=AESGCM(128) Mac=AEAD
0xC0,0x2D - ECDH-ECDSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH/ECDSA Au=ECDH Enc=AESGCM(128) Mac=AEAD
0xC0,0x29 - ECDH-RSA-AES128-SHA256  TLSv1.2 Kx=ECDH/RSA Au=ECDH Enc=AES(128)  Mac=SHA256
0xC0,0x25 - ECDH-ECDSA-AES128-SHA256 TLSv1.2 Kx=ECDH/ECDSA Au=ECDH Enc=AES(128)  Mac=SHA256
0x00,0x3D - AES256-SHA256           TLSv1.2 Kx=RSA      Au=RSA  Enc=AES(256)  Mac=SHA256
0x00,0x9C - AES128-GCM-SHA256       TLSv1.2 Kx=RSA      Au=RSA  Enc=AESGCM(128) Mac=AEAD
0x00,0x3C - AES128-SHA256           TLSv1.2 Kx=RSA      Au=RSA  Enc=AES(128)  Mac=SHA256
0xC0,0x0A - ECDHE-ECDSA-AES256-SHA  SSLv3 Kx=ECDH     Au=ECDSA Enc=AES(256)  Mac=SHA1
0xC0,0x14 - ECDHE-RSA-AES256-SHA    SSLv3 Kx=ECDH     Au=RSA  Enc=AES(256)  Mac=SHA1
0xC0,0x09 - ECDHE-ECDSA-AES128-SHA  SSLv3 Kx=ECDH     Au=ECDSA Enc=AES(128)  Mac=SHA
0xC0,0x13 - ECDHE-RSA-AES128-SHA    SSLv3 Kx=ECDH     Au=RSA  Enc=AES(128)  Mac=SHA1
0x00,0x39 - DHE-RSA-AES256-SHA      SSLv3 Kx=DH       Au=RSA  Enc=AES(256)  Mac=SHA1
0x00,0x38 - DHE-DSS-AES256-SHA      SSLv3 Kx=DH       Au=DSS  Enc=AES(256)  Mac=SHA1          
0x00,0x33 - DHE-RSA-AES128-SHA      SSLv3 Kx=DH       Au=RSA  Enc=AES(128)  Mac=SHA1
0x00,0x32 - DHE-DSS-AES128-SHA      SSLv3 Kx=DH       Au=DSS  Enc=AES(128)  Mac=SHA1          
0xC0,0x22 - SRP-DSS-AES-256-CBC-SHA SSLv3 Kx=SRP      Au=DSS  Enc=AES(256)  Mac=SHA1
0xC0,0x21 - SRP-RSA-AES-256-CBC-SHA SSLv3 Kx=SRP      Au=RSA  Enc=AES(256)  Mac=SHA1
0xC0,0x20 - SRP-AES-256-CBC-SHA     SSLv3 Kx=SRP      Au=SRP  Enc=AES(256)  Mac=SHA1
0xC0,0x1F - SRP-DSS-AES-128-CBC-SHA SSLv3 Kx=SRP      Au=DSS  Enc=AES(128)  Mac=SHA1
0xC0,0x1E - SRP-RSA-AES-128-CBC-SHA SSLv3 Kx=SRP      Au=RSA  Enc=AES(128)  Mac=SHA1          
0xC0,0x1D - SRP-AES-128-CBC-SHA     SSLv3 Kx=SRP      Au=SRP  Enc=AES(128)  Mac=SHA1
0xC0,0x05 - ECDH-ECDSA-AES256-SHA   SSLv3 Kx=ECDH/ECDSA Au=ECDH Enc=AES(256)  Mac=SHA1
0xC0,0x0F - ECDH-RSA-AES256-SHA     SSLv3 Kx=ECDH/RSA Au=ECDH Enc=AES(256)  Mac=SHA1
0xC0,0x0E - ECDH-RSA-AES128-SHA     SSLv3 Kx=ECDH/RSA Au=ECDH Enc=AES(128)  Mac=SHA1
0xC0,0x04 - ECDH-ECDSA-AES128-SHA   SSLv3 Kx=ECDH/ECDSA Au=ECDH Enc=AES(128)  Mac=SHA1
0x00,0x35 - AES256-SHA              SSLv3 Kx=RSA      Au=RSA  Enc=AES(256)  Mac=SHA1
0x00,0x2F - AES128-SHA              SSLv3 Kx=RSA      Au=RSA  Enc=AES(128)  Mac=SHA1	

Automatic Updates of the Certificate

As a matter of policy the certificates emitted by Let’s Encrypt are only valid for a short period of 90 days. This explains why the project focuses so much on automation.

It is possible to renew the certificate by executing the following commands, which I added to a /srv/letsencrypt/cron-update.sh script

1
2
3
#!/bin/sh
/srv/letsencrypt/letsencrypt-auto certonly --webroot -w /srv/blog.barthe.ph/www -d blog.barthe.ph --renew-by-default  --agree-tos
service nginx reload

Unfortunately, you’ll find out that the letsencrypt-auto command fails with the following error:

1
Failed authorization procedure. blog.barthe.ph (http-01): urn:acme:error:unauthorized :: The client lacks sufficient authorization :: Error parsing key authorization file: Invalid key authorization: 231 parts

I found out that the ACME protocol attempts to validate the ownership using HTTP and is unable to follow the HTTP to HTTPS 301 redirection we set up at the beginning. So I had to slightly change my setup in /etc/nginx/sites-available/blog.barthe.ph.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
# The following lines prevent Let's Encrypt ACME protocol from working 
#server {
#        server_name blog.barthe.ph;
#        listen 80;
#        return 301 https://$server_name$request_uri;
#}
#
# Replace by these lines, to continue serving /.well-known/ files on port 80
server {
        server_name blog.barthe.ph;
        listen 80;
        location /.well-known/ {
                root /srv/blog.barthe.ph/www;
                try_files $uri $uri/ =404;
        }
        location / {
                return 301 https://$server_name$request_uri;
        }
}

The new version let all ACME requests to /.well-known/ be served normally over HTTP by nginx, but redirects all the others to HTTPS. Once that change is done and nginx reloaded, executing the letsencrypt-auto successfully updates the certificate.

The next is to add a CRON job, by typing crontab -e (as root). I set up mine to update the certificate once a month, as follow:

1
2
3
# contrab -e
# m h  dom mon dow   command
0 11 7 * * /srv/letsencrypt/cron-update.sh

Conclusion

So far I’m relatively happy with Let’s Encrypt. I have chosen to run the letsencrypt-auto run with root privileges, but if that bothers you, there is a project named Let’s Encrypt no sudo which aims to prevent that.

For websites with more traffic, you could delay the HTTP to HTTPS redirection until you have fully tested the HTTPS version of the site. This is a good opportunity to fix all mixed URL scheme content issues.

Finally, I published my final /etc/nginx/sites-available/blog.barthe.ph config file as a Gist.

Edit. I initially made a mistake and wrote return 301 https://blog.barthe.ph instead of return 301 https://$server_name$request_uri in the nginx configuration. The consequence is that URLs starting with http would be redirected to the home page of the blog, instead of the https counterpart page. This defect was somehow masked by HSTS, because after browsing once to the site, the browser would redirect to the correct pages.

Migrated to Kimsufi

Originally this website was powered by a shared hosting plan on Dreamhost, but I migrated to a fully hosted solution over a year ago.

After considering many options, I chose a Dedibox SC gen 2, a dedicated server solution from Online.net a French server manged hosting provider. The machine features a fairly low spec CPU, a 500GB hybrid SSD-HDD disk, 2GB of RAM and unlimited bandwidth with guaranteed throughput.

All that for €12 a month. It is much cheaper that all other hardware of VPS solutions I could find, and in fact about the same price as the Dreamhost hosting plan it replaced.

France seems to have two companies (Online with Dedibox and OVH with Kimsufi) dedicated to offer very low cost hosting solutions. The prices can be low thanks to custom dedicated hardware that is both small and power efficient. As you can see below, a Dedibox SC is barely larger than a 3.5 inch HDD. My gut feeling is that the emergence of these offers in France is linked to the three strike piracy law and the disappearance of Megaupload. A lot of these servers were used as seed boxes, at least in the beginning…

The hosting on Dedibox turned out to be very good and sufficient for my needs. A properly configured nginx server serving mostly static pages has no problem serving the 20,000 hits per month my sites require. I initially hesitated using Amazon S3 for some of the most popular pages, but dropped the plan entirely. I also have a few dynamic pages, but these are mostly for personal use. The PHP code of Exposition, my unfinished photo gallery software, works a lot faster than on Dreamhost.

The support on Dedibox is not great though. I had some billing issue at the beginning and it took forever for it to be resolved. I’m not sure I would trust them for something critical, like a business.

In the past few days, I made the switch to their competitor Kimsufi. I was running out of space on the 500GB disk of the Dedibox, because of my large photo collection. I opted for a KS-2, which is the exact same price price as the Dedibox SC Gen 2. It is normally fitted with a 1TB disk, but I got 2TB disk instead (surprise!). The KS-2 is also fitted with 4GB of RAM instead of the 2GB of the Dedibox.

The rest is worse though. On Dedibox the bandwidth is guaranteed, whereas on Kimsufi it’s best effort. In practice, I have not noticed any differences. The disk on the Dedibox is a hybrid SDD; Kimsufi uses a traditional slower spinning disk. Dedibox offers free monitoring services, like email notifications when the machine is turned on or off. Kimsufi offers nothing. OVH has a full featured iOS app that used to support Kimsufi users, but is now restricted to the pro tier of OVH hosting. Shame on you OVH! So I ended up configuring a free pingdom.com account for basic monitoring.

In the end, it’s a pretty good deal if you need the storage or the RAM. 2TB of “Cloud Storage” (it should be 1TB) for €12 month and similar fees with no limits in bandwidth is pretty good. It’s more than what you would get from Amazon S3 or Dropbox for the same price. But if you’re fine with 500GB or only 2GB of RAM, I would recommend the Dedibox offer instead.

Let’s hope the new hosting is as reliable as the previous one…