Let's Encrypt - Free SSL/TLS Certificates

Let's Encrypt

Preliminary :

The very 1st thing you should check if you consider enabling HTTPS / SSL / TLS on your server is :
Initial setup :
PUBLIC_IP:80  [Varnish] --> PUBLIC_IP:8080 [Lighttpd]
PUBLIC_IP:443 [SSHd]
Available methods :
Once this TCP 443 question is under control, there are several methods to proceed :
Varnish + Hitch
  • Varnish for HTTP caching, Hitch for SSL/TLS termination (1, 2)
  • This failed for me because of a "containerized" VM.
    My server is actually a VPS running on OpenVZ :
  • the steps of this procedure are shown below, not only for future reference, but also because I re-used entire steps (like the Certbot stuff)
Varnish + Nginx

Varnish + Hitch :

This procedure is described in :
 * https://letsencrypt.org/getting-started/
 * https://letsencrypt.org/docs/client-options/
 * https://certbot.eff.org/lets-encrypt/debianstretch-other

 * https://docs.varnish-software.com/tutorials/hitch-letsencrypt/
 * https://hitch-tls.org/



Step 1 - Install Hitch
apt install hitch
Step 2 - Add certbot passthrough VCL
cat << EOF > /etc/varnish/letsencrypt.vcl
vcl 4.1;

backend certbot {
    .host = "127.0.0.1";
    .port = "8080";
}

sub vcl_recv {
    if (req.url ~ "^/\.well-known/acme-challenge/") {
        set req.backend_hint = certbot;
        return(pipe);
    }
}

sub vcl_pipe {
    if (req.backend_hint == certbot) {
        set req.http.Connection = "close";
        return(pipe);
    }
}
EOF
sed -ri 's/ {4}/\t/g' /etc/varnish/letsencrypt.vcl && cat /etc/varnish/letsencrypt.vcl
installing Hitch creates a _hitch user account (hence changes in /etc/passwd and /etc/shadow
then load this new file : emacs /etc/varnish/default.vcl add: include "/etc/varnish/letsencrypt.vcl"; Step 3 - Configure and start Varnish
systemctl edit --full varnish
change ExecStart=/usr/sbin/varnishd -a :6081 -a localhost:8443,proxy -f /etc/varnish/default.vcl -s malloc,256m into ExecStart=/usr/sbin/varnishd -a :80 -a localhost:8443,proxy -f /etc/varnish/default.vcl -s malloc,256m here (because already had Varnish configured), change : ExecStart=/usr/sbin/varnishd -a :80 -T localhost:6082 -f /etc/varnish/default.vcl -S /etc/varnish/secret -s malloc,256m into ExecStart=/usr/sbin/varnishd -a :80 -T localhost:6082 -a localhost:8443,proxy -f /etc/varnish/default.vcl -S /etc/varnish/secret -s malloc,256m Step 4 - Prepare hitch
cat << EOF > /usr/local/bin/hitch-deploy-hook
#!/bin/bash
# Full path to pre-generated Diffie Hellman Parameters file
dhparams=/etc/hitch/dhparams.pem

if [[ "\${RENEWED_LINEAGE}" == "" ]]; then
    echo "Error: missing RENEWED_LINEAGE env variable." >&2
    exit 1
fi

umask 077
cat \${RENEWED_LINEAGE}/privkey.pem \
\${RENEWED_LINEAGE}/fullchain.pem \
\${dhparams} > \${RENEWED_LINEAGE}/hitch-bundle.pem
EOF
chmod a+x /usr/local/bin/hitch-deploy-hook sed -ri 's/ {4}/\t/g' /usr/local/bin/hitch-deploy-hook && cat /usr/local/bin/hitch-deploy-hook openssl dhparam 2048 | tee /etc/hitch/dhparams.pem
grep -E '^backend' /etc/hitch/hitch.conf ==> nothing (???) not installed, or I removed it by mistake ? see example :
cp /usr/share/doc/hitch/examples/hitch.conf.example /etc/hitch/hitch.conf
sample /etc/hitch/hitch.conf :
# Run 'man hitch.conf' for a description of all options.

frontend = {
	host = "127.0.0.1"
	port = "443"
}
#backend = "[127.0.0.1]:6086"    # 6086 is the default Varnish PROXY port.
backend = "[localhost]:8443"
workers = 4                     # number of CPU cores

daemon = on
user = "_hitch"
group = "_hitch"

# Enable to let clients negotiate HTTP/2 with ALPN. (default off)
# alpn-protos = "http/2, http/1.1"

# run Varnish as backend over PROXY; varnishd -a :80 -a localhost:6086,PROXY ..
write-proxy-v2 = on             # Write PROXY header


pem-file = "/etc/letsencrypt/live/example.com/hitch-bundle.pem"
systemctl enable hitch systemctl start hitch journalctl -u hitch -r mkdir -p /var/lib/hitch/ {bind-socket}: Address already in use: [127.0.0.1]:443 ==> configure SSLH to listen on public IP only : /etc/default/sslh DAEMON_OPTS="--user sslh --listen my.public.IP.address:443 --ssh 127.0.0.1:22 --ssl 127.0.0.1:443 --pidfile /var/run/sslh/sslh.pid" systemctl restart sslh systemctl start hitch systemctl status hitch Oct 14 21:23:59 MyVPS hitch[3335]: {core} Daemonized as pid 3337. Oct 14 21:23:59 MyVPS hitch[3335]: {core} Loading certificate pem files (1) Oct 14 21:23:59 MyVPS hitch[3335]: {core} hitch 1.4.4 starting Oct 14 21:23:59 MyVPS systemd[1]: Started Hitch TLS unwrapping daemon. Oct 14 21:23:59 MyVPS systemd[1]: hitch.service: Failed to set invocation ID on control group /system.slice/hitch.service, ignoring: Operation not permitted
the certbot part :
Step 5 - Install and run certbot
apt install certbot
certbot certonly --standalone --preferred-challenges http --http-01-port 8080 -d example.com -d www.example.com --deploy-hook="/usr/local/bin/hitch-deploy-hook" --post-hook="systemctl reload hitch"
Please read the Terms of Service at
https://letsencrypt.org/documents/LE-SA-v1.2-November-15-2017.pdf.


Saving debug log to /var/log/letsencrypt/letsencrypt.log
Plugins selected: Authenticator standalone, Installer None
Obtaining a new certificate
Performing the following challenges:
http-01 challenge for example.com
http-01 challenge for www.example.com
Cleaning up challenges
Running post-hook command: systemctl reload hitch
Hook command "systemctl reload hitch" returned error code 1
Error output from systemctl:
hitch.service is not active, cannot reload.
/!\ port number 8080 may be already in use. Configure same port in /etc/varnish/letsencrypt.vcl systemctl status lighttpd systemctl stop lighttpd systemctl start lighttpd
certbot certonly --standalone --preferred-challenges http --http-01-port 8080 -d example.com -d www.example.com --deploy-hook="/usr/local/bin/hitch-deploy-hook" --pre-hook="systemctl stop lighttpd" --post-hook="systemctl start lighttpd"
IMPORTANT NOTES:
 - The following errors were reported by the server:

   Domain: example.com
   Type:   connection
   Detail: Fetching
   http://example.com/.well-known/acme-challenge/6OXqb8UQdVUX-_t2rwknP-bFzgfWEO7snu8R5WpFfbI:
   Error getting validation data

   Domain: www.example.com
   Type:   connection
   Detail: Fetching
   http://www.example.com/.well-known/acme-challenge/v7VKRrLEC32cRRHjsb4PYqNBebEmo_Aq8FaTCv2SBzw:
   Error getting validation data

   To fix these errors, please make sure that your domain name was
   entered correctly and the DNS A/AAAA record(s) for that domain
   contain(s) the right IP address. Additionally, please check that
   your computer has a publicly routable IP address and that no
   firewalls are preventing the server from communicating with the
   client. If you're using the webroot plugin, you should also verify
   that you are serving files from the webroot path you provided.
host -t A example.com
example.com has address my.public.IP.address
host -t A www.example.com
www.example.com has address my.public.IP.address
certbot certonly --standalone --preferred-challenges http --http-01-port 8080 -d example.com -d www.example.com --deploy-hook="/usr/local/bin/hitch-deploy-hook" --pre-hook="systemctl stop lighttpd" --post-hook="systemctl start lighttpd"
Saving debug log to /var/log/letsencrypt/letsencrypt.log
Plugins selected: Authenticator standalone, Installer None
Running pre-hook command: systemctl stop lighttpd
Obtaining a new certificate
Performing the following challenges:
http-01 challenge for example.com
http-01 challenge for www.example.com
Waiting for verification...
Cleaning up challenges
Running deploy-hook command: /usr/local/bin/hitch-deploy-hook
Hook command "/usr/local/bin/hitch-deploy-hook" returned error code 1
Error output from hitch-deploy-hook:
Error: missing RENEWED_LINEAGE env variable.

Running post-hook command: systemctl start lighttpd

IMPORTANT NOTES:
 - Congratulations! Your certificate and chain have been saved at:
   /etc/letsencrypt/live/example.com/fullchain.pem
   Your key file has been saved at:
   /etc/letsencrypt/live/example.com/privkey.pem
   Your cert will expire on 2020-01-12. To obtain a new or tweaked
   version of this certificate in the future, simply run certbot
   again. To non-interactively renew *all* of your certificates, run
   "certbot renew"
 - If you like Certbot, please consider supporting our work by:

   Donating to ISRG / Let's Encrypt:   https://letsencrypt.org/donate
   Donating to EFF:                    https://eff.org/donate-le
missing RENEWED_LINEAGE env variable : https://community.letsencrypt.org/t/certbot-renewed-lineage-and-renewed-domains-shell-variable-clarification/33184/3 https://github.com/certbot/certbot/issues/3502 ==>/etc/letsencrypt/live/$domain
export RENEWED_LINEAGE='/etc/letsencrypt/live/example.com'
TODO: enable automatic renewal sudo systemctl enable certbot-renew.timer sudo systemctl start certbot-renew.timer ==> not available systemctl list-unit-files | grep certbot certbot.service static certbot.timer enabled systemctl enable certbot.timer && systemctl start certbot.timer ==> ok Step 6 - Start hitch add to /etc/hitch/hitch.conf pem-file = "/etc/letsencrypt/live/example.com/hitch-bundle.pem" systemctl restart hitch ==========================================8<========================================================= now let's finish configuring our webserver Lighttpd on 8080 only : server.bind = "127.0.0.1" ==> quotes are mandatory!! ==========================================8<========================================================= ... ... Can't work with Hitch because it requires a "modern" kernel, whereas my server is actually a "containerized VM (OpenVZ), with an old kernel I can't upgrade ==> STOP WORKING ON THIS SOLUTION, LET'S UNDO CHANGES
systemctl stop hitch && systemctl disable hitch
restore Varnish conf :
systemctl edit --full varnish
/etc/systemd/system/varnish.service
systemctl restart varnish && systemctl status varnish
userdel -r _hitch apt purge hitch empty + remove /etc/hitch

Varnish + Nginx :

Here's the setup we're going to configure :

Certificates : stuff I already have from previous steps :
ll /etc/letsencrypt/live/example.com
total 12K
lrwxrwxrwx 1 root root   39 Oct 14 20:15 cert.pem -> ../../archive/example.com/cert2.pem
lrwxrwxrwx 1 root root   40 Oct 14 20:15 chain.pem -> ../../archive/example.com/chain2.pem
lrwxrwxrwx 1 root root   44 Oct 14 20:15 fullchain.pem -> ../../archive/example.com/fullchain2.pem
-rw------- 1 root root 5.6K Oct 14 20:21 hitch-bundle.pem
lrwxrwxrwx 1 root root   42 Oct 14 20:15 privkey.pem -> ../../archive/example.com/privkey2.pem
-rw-r--r-- 1 root root  692 Oct 14 20:08 README
Get more certificates :
This is because :
  • there are distinct websites hosted on the same domain example.com : www.example.com, ww2.example.com, ...
  • the certificates I requested so far (see previous certbot commands) don't cover them all
--deploy-hook="/usr/local/bin/hitch-deploy-hook" now useless, clean this
certbot certonly --standalone --preferred-challenges http --http-01-port 8080 -d example.com -d www.example.com --deploy-hook="/usr/local/bin/hitch-deploy-hook" --pre-hook="systemctl stop lighttpd" --post-hook="systemctl start lighttpd"
certbot certonly --standalone --preferred-challenges http --http-01-port 8080 -d example.com,ww2.example.com,www.example.com,ww3.example.com,ww4.example.com,ww5.example.com,ww6.example.com,ww7.example.com --deploy-hook="/usr/local/bin/hitch-deploy-hook" --pre-hook="systemctl stop lighttpd" --post-hook="systemctl start lighttpd"
Saving debug log to /var/log/letsencrypt/letsencrypt.log
Plugins selected: Authenticator standalone, Installer None

- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
You have an existing certificate that contains a portion of the domains you
requested (ref: /etc/letsencrypt/renewal/example.com.conf)

It contains these names: example.com, www.example.com

You requested these names for the new certificate: example.com,
ww2.example.com, www.example.com, ww3.example.com,
ww4.example.com, ww5.example.com, ww6.example.com,
ww7.example.com.

Do you want to expand and replace this existing certificate with the new
certificate?
- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
(E)xpand/(C)ancel: E
Running pre-hook command: systemctl stop lighttpd
Renewing an existing certificate
Performing the following challenges:
http-01 challenge for example.com
http-01 challenge for www.example.com
http-01 challenge for ww2.example.com
http-01 challenge for ww3.example.com
http-01 challenge for ww4.example.com
http-01 challenge for ww5.example.com
http-01 challenge for ww6.example.com
http-01 challenge for ww7.example.com
Waiting for verification...
Cleaning up challenges
Running deploy-hook command: /usr/local/bin/hitch-deploy-hook
Running post-hook command: systemctl start lighttpd

IMPORTANT NOTES:
 - Congratulations! Your certificate and chain have been saved at:
   /etc/letsencrypt/live/example.com/fullchain.pem
   Your key file has been saved at:
   /etc/letsencrypt/live/example.com/privkey.pem
   Your cert will expire on 2020-01-12. To obtain a new or tweaked
   version of this certificate in the future, simply run certbot
   again. To non-interactively renew *all* of your certificates, run
   certbot renew
   
				
Install Nginx :
Nginx is a little bit annoying since, upon installing, it tries to start listening on port TCP 80 automatically. Which fails since this port is used by Varnish.
Workaround :
  1. systemctl stop varnish
  2. apt install nginx
  3. systemctl stop nginx
  4. systemctl start varnish
Configure Nginx :
/etc/nginx/nginx.conf (sources : 1, 2, 3) :
user www-data;
worker_processes auto;
pid /run/nginx.pid;
include /etc/nginx/modules-enabled/*.conf;

stream {
	upstream stream_backend {
		server	127.0.0.1:80;
		}

	server {
		listen			127.0.0.1:443 ssl;
		proxy_pass		stream_backend;

		ssl_certificate		/etc/letsencrypt/live/example.com/fullchain.pem;
		ssl_certificate_key	/etc/letsencrypt/live/example.com/privkey.pem;
#		ssl_protocols		SSLv3 TLSv1 TLSv1.1 TLSv1.2;
		ssl_protocols		TLSv1 TLSv1.1 TLSv1.2;

#		ssl_ciphers		HIGH:!aNULL:!MD5;
		# get the full list of available ciphers with : openssl ciphers
		ssl_ciphers		ECDHE-ECDSA-AES128-GCM-SHA256:ECDHE-RSA-AES128-GCM-SHA256:ECDHE-ECDSA-AES256-GCM-SHA384:ECDHE-RSA-AES256-GCM-SHA384:ECDHE-ECDSA-CHACHA20-POLY1305:ECDHE-RSA-CHACHA20-POLY1305:DHE-RSA-AES128-GCM-SHA256:DHE-RSA-AES256-GCM-SHA384;

		ssl_prefer_server_ciphers	off;
		ssl_session_cache		shared:SSL:20m;
		ssl_session_timeout		4h;
		ssl_handshake_timeout		10s;

		# get this with :
		# curl https://ssl-config.mozilla.org/ffdhe2048.txt > /etc/nginx/dhparam.pem
		ssl_dhparam	/etc/nginx/dhparam.pem;
		}
	}

events {
	worker_connections 768;
	# multi_accept on;
	}
Then :
  • Start it :
    systemctl start nginx
  • Check :
    systemctl status nginx
  • Debug :
    journalctl -u nginx -r
Reboot the universe :
systemctl restart lighttpd && systemctl restart varnish && systemctl restart nginx && systemctl restart sslh
Check your configuration :
Test the whole setup with ssllabs.com.