Let's Encrypt - Httqm's Docs

As described in the Let's Encrypt : setup article, I have :

applied for and received certificates
configured my "web stack" so that it can receive + handle + reply to HTTPS request

The resulting setup is :

This means I'm currently serving the same content both via HTTP and HTTPS. What I want to do now is :

continue serving content via HTTPS
when receiving a request, for any content, via HTTP, reply with a HTTP 301 code to instruct the client to renew its request via HTTPS

To do so, I'll have to configure whatever is listening on my.public.IP.address:80 —so far, it's Varnish— to handle HTTP requests and send the appropriate replies.

I have several solutions :

configure Varnish (the part listening on my.public.IP.address:80) so that it returns the HTTP 301 codes
OR :
1. configure Varnish so that it only listens on 127.0.0.1:80
2. configure Nginx to listen both on
  - 127.0.0.1:443 : already active
  - my.public.IP.address:80 : replace Varnish
3. configure Nginx to perform the HTTP to HTTPS redirection

I've chosen the 2^nd option, which gives :

For Varnish : listen on 127.0.0.1:80 only :

edit etc/systemd/system/varnish.service b/etc/systemd/system/varnish.service

change

ExecStart=/usr/sbin/varnishd -a :80 -f /etc/varnish/default.vcl -S /etc/varnish/secret -s malloc,64m -p thread_pool_min=10 -p thread_pool_max=20

into

ExecStart=/usr/sbin/varnishd -a 127.0.0.1:80 -f /etc/varnish/default.vcl -S /etc/varnish/secret -s malloc,64m -p thread_pool_min=10 -p thread_pool_max=20

don't restart it yet

For Nginx : add listening on my.public.IP.address:80 + redirect HTTP to HTTPS :

Many blog articles simply suggest adding a code block (see below) into Nginx's configuration, which is right but still lacks details IMHO :

server {
	listen 80 default_server;
	listen [::]:80 default_server;
	server_name _;
	return 301 https://$host$request_uri;
}

This code block must be enclosed within an http {} block (source) or it will cause an error :

"server" directive is not allowed here in /etc/nginx/nginx.conf:lineNumber

http {
	
		insert the server {  } block here
	
	}

Finally, for me, the operation was to :

edit /etc/nginx/nginx.conf so that it contains :

http {
	server {
		listen	my.public.IP.address:80;
		return	301 https://$host$request_uri;
		}
	}

restart Varnish (which frees my.public.IP.address:80) :
systemctl restart varnish.service
restart Nginx :
systemctl restart nginx

Final words :

This article describes a setup which is pretty similar to mine. There's an interesting discussion in the comments regarding whether Varnish is still a valid solution today whereas it does not (and never will) support TLS.

Varnish developers have chosen not to implement TLS to follow the Unix philosophy : Do only one thing and do it well.. Varnish is actually a very good (and fast !) HTTP cache server. For TLS termination functionality, there are already many dedicated tools : HAProxy, Nginx, Pound, Hitch, ...

It's been a strange situation where I got simultaneously :

a message in my web browser stating that my certificate was expired (as well as expiration notice received by mail a few days earlier)
certbot refusing to renew my certificate saying
```
Cert not yet due for renewal
```

I must have done something wrong, but can't figure what.

What I tried :

Commands below were run on the 2020-01-13.

Naively trying to renew the certificate :

certbot renew

Saving debug log to /var/log/letsencrypt/letsencrypt.log

- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
Processing /etc/letsencrypt/renewal/example.com.conf
- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
Cert not yet due for renewal

- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -

The following certs are not due for renewal yet:
	/etc/letsencrypt/live/example.com/fullchain.pem expires on 2020-03-13 (skipped)		My browser said it was expired 
No renewals were attempted.
- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -

Let's get details about my certificate :

certbot certificates

Saving debug log to /var/log/letsencrypt/letsencrypt.log

- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
Found the following certs:
	Certificate Name: example.com
Domains: example.com www.example.com www1.example.com www2.example.com www3.example.com www4.example.com www5.example.com www6.example.com
Expiry Date: 2020-03-13 02:41:21+00:00 (VALID: 59 days)
Certificate Path: /etc/letsencrypt/live/example.com/fullchain.pem
Private Key Path: /etc/letsencrypt/live/example.com/privkey.pem
- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -

What about my webserver configuration ?

grep ssl_certificate /etc/nginx/nginx.conf

	ssl_certificate			/etc/letsencrypt/live/example.com/fullchain.pem;
	ssl_certificate_key		/etc/letsencrypt/live/example.com/privkey.pem;

Now we just have to revoke this certificate and ask for a new one.

This is not the normal process.

certbot revoke --cert-path /etc/letsencrypt/live/example.com/fullchain.pem

Saving debug log to /var/log/letsencrypt/letsencrypt.log

- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
Would you like to delete the cert(s) you just revoked?
- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
(Y)es (recommended)/(N)o: y

- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
Deleted all files relating to certificate example.com.
- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -

- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
Congratulations! You have successfully revoked the certificate that was located
at /etc/letsencrypt/live/example.com/fullchain.pem

- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -

Clean directories :
mv /etc/letsencrypt/live{,_DELETEME}
request a new certificate
Not clear why I had to do this : :
cd /etc/letsencrypt/live && ln -s example.com-0001/ example.com
restart everything

Preliminary :

The very 1^st thing you should check if you consider enabling HTTPS / SSL / TLS on your server is :

Is the port TCP 443 available to receive web traffic ?

Initial setup :

PUBLIC_IP:80  [Varnish] --> PUBLIC_IP:8080 [Lighttpd]
PUBLIC_IP:443 [SSHd]

Available methods :

Once this TCP 443 question is under control, there are several methods to proceed :

Varnish + Hitch

Varnish for HTTP caching, Hitch for SSL/TLS termination (1, 2)
This failed for me because of a "containerized" VM.
My server is actually a VPS running on OpenVZ :
- OpenVZ runs containers, just like LXC and Docker do.
- this means "my kernel" is not "my kernel" : it's the kernel of the host system. This is also why I can not update it. (more about VPS and OpenVZ vs KVM)
- Hitch couldn't start on my VPS because of unmet requirements about the kernel and/or systemd versions.
the steps of this procedure are shown below, not only for future reference, but also because I re-used entire steps (like the Certbot stuff)

Varnish + Nginx

Varnish for HTTP caching, Nginx for SSL/TLS termination
this method (based on changes made with the previous) finally worked and is available here.

Varnish + Hitch :

This procedure is described in :
 * https://letsencrypt.org/getting-started/
 * https://letsencrypt.org/docs/client-options/
 * https://certbot.eff.org/lets-encrypt/debianstretch-other

 * https://docs.varnish-software.com/tutorials/hitch-letsencrypt/
 * https://hitch-tls.org/



Step 1 - Install Hitch
apt install hitch


Step 2 - Add certbot passthrough VCL

cat << EOF > /etc/varnish/letsencrypt.vcl
vcl 4.1;

backend certbot {
	.host = "127.0.0.1";
	.port = "8080";
}

sub vcl_recv {
	if (req.url ~ "^/\.well-known/acme-challenge/") {
		set req.backend_hint = certbot;
		return(pipe);
	}
}

sub vcl_pipe {
	if (req.backend_hint == certbot) {
		set req.http.Connection = "close";
		return(pipe);
	}
}
EOF

sed -ri 's/ {4}/\t/g' /etc/varnish/letsencrypt.vcl && cat /etc/varnish/letsencrypt.vcl


installing Hitch creates a _hitch user account (hence changes in /etc/passwd and /etc/shadow



then load this new file :
emacs /etc/varnish/default.vcl
add:
include "/etc/varnish/letsencrypt.vcl";


Step 3 - Configure and start Varnish
systemctl edit --full varnish

change
ExecStart=/usr/sbin/varnishd -a :6081 -a localhost:8443,proxy -f /etc/varnish/default.vcl -s malloc,256m
into
ExecStart=/usr/sbin/varnishd -a :80 -a localhost:8443,proxy -f /etc/varnish/default.vcl -s malloc,256m


here (because already had Varnish configured), change :
ExecStart=/usr/sbin/varnishd -a :80 -T localhost:6082 -f /etc/varnish/default.vcl -S /etc/varnish/secret -s malloc,256m
into
ExecStart=/usr/sbin/varnishd -a :80 -T localhost:6082 -a localhost:8443,proxy -f /etc/varnish/default.vcl -S /etc/varnish/secret -s malloc,256m




Step 4 - Prepare hitch
cat << EOF > /usr/local/bin/hitch-deploy-hook
#!/bin/bash
# Full path to pre-generated Diffie Hellman Parameters file
dhparams=/etc/hitch/dhparams.pem

if [[ "\${RENEWED_LINEAGE}" == "" ]]; then
	echo "Error: missing RENEWED_LINEAGE env variable." >&2
	exit 1
fi

umask 077
cat \${RENEWED_LINEAGE}/privkey.pem \
\${RENEWED_LINEAGE}/fullchain.pem \
\${dhparams} > \${RENEWED_LINEAGE}/hitch-bundle.pem
EOF

chmod a+x /usr/local/bin/hitch-deploy-hook
sed -ri 's/ {4}/\t/g' /usr/local/bin/hitch-deploy-hook && cat /usr/local/bin/hitch-deploy-hook

openssl dhparam 2048 | tee /etc/hitch/dhparams.pem


grep -E '^backend' /etc/hitch/hitch.conf
	==> nothing (???) not installed, or I removed it by mistake ?

see example : cp /usr/share/doc/hitch/examples/hitch.conf.example /etc/hitch/hitch.conf
sample /etc/hitch/hitch.conf :
# Run 'man hitch.conf' for a description of all options.

frontend = {
	host = "127.0.0.1"
	port = "443"
}
#backend = "[127.0.0.1]:6086"	# 6086 is the default Varnish PROXY port.
backend = "[localhost]:8443"
workers = 4                     # number of CPU cores

daemon = on
user = "_hitch"
group = "_hitch"

# Enable to let clients negotiate HTTP/2 with ALPN. (default off)
# alpn-protos = "http/2, http/1.1"

# run Varnish as backend over PROXY; varnishd -a :80 -a localhost:6086,PROXY ..
write-proxy-v2 = on             # Write PROXY header


pem-file = "/etc/letsencrypt/live/example.com/hitch-bundle.pem"



systemctl enable hitch
systemctl start hitch

journalctl -u hitch -r
mkdir -p /var/lib/hitch/

{bind-socket}: Address already in use: [127.0.0.1]:443
	==> configure SSLH to listen on public IP only :

/etc/default/sslh
DAEMON_OPTS="--user sslh --listen my.public.IP.address:443 --ssh 127.0.0.1:22 --ssl 127.0.0.1:443 --pidfile /var/run/sslh/sslh.pid"

systemctl restart sslh
systemctl start hitch
systemctl status hitch


Oct 14 21:23:59 MyVPS hitch[3335]: {core} Daemonized as pid 3337.
Oct 14 21:23:59 MyVPS hitch[3335]: {core} Loading certificate pem files (1)
Oct 14 21:23:59 MyVPS hitch[3335]: {core} hitch 1.4.4 starting
Oct 14 21:23:59 MyVPS systemd[1]: Started Hitch TLS unwrapping daemon.
Oct 14 21:23:59 MyVPS systemd[1]: hitch.service: Failed to set invocation ID on control group /system.slice/hitch.service, ignoring: Operation not permitted

the certbot part :

Step 5 - Install and run certbot
apt install certbot

certbot certonly --standalone --preferred-challenges http --http-01-port 8080 -d example.com -d www.example.com --deploy-hook="/usr/local/bin/hitch-deploy-hook" --post-hook="systemctl reload hitch"
Please read the Terms of Service at
https://letsencrypt.org/documents/LE-SA-v1.2-November-15-2017.pdf.


Saving debug log to /var/log/letsencrypt/letsencrypt.log
Plugins selected: Authenticator standalone, Installer None
Obtaining a new certificate
Performing the following challenges:
http-01 challenge for example.com
http-01 challenge for www.example.com
Cleaning up challenges
Running post-hook command: systemctl reload hitch
Hook command "systemctl reload hitch" returned error code 1
Error output from systemctl:
hitch.service is not active, cannot reload.


/!\ port number 8080 may be already in use. Configure same port in /etc/varnish/letsencrypt.vcl

systemctl status lighttpd
systemctl stop lighttpd
systemctl start lighttpd


certbot certonly --standalone --preferred-challenges http --http-01-port 8080 -d example.com -d www.example.com --deploy-hook="/usr/local/bin/hitch-deploy-hook" --pre-hook="systemctl stop lighttpd" --post-hook="systemctl start lighttpd"
IMPORTANT NOTES:
 - The following errors were reported by the server:

	Domain: example.com
	Type:   connection
	Detail: Fetching
	http://example.com/.well-known/acme-challenge/6OXqb8UQdVUX-_t2rwknP-bFzgfWEO7snu8R5WpFfbI:
	Error getting validation data

	Domain: www.example.com
	Type:   connection
	Detail: Fetching
	http://www.example.com/.well-known/acme-challenge/v7VKRrLEC32cRRHjsb4PYqNBebEmo_Aq8FaTCv2SBzw:
	Error getting validation data

	To fix these errors, please make sure that your domain name was
	entered correctly and the DNS A/AAAA record(s) for that domain
	contain(s) the right IP address. Additionally, please check that
	your computer has a publicly routable IP address and that no
	firewalls are preventing the server from communicating with the
	client. If you're using the webroot plugin, you should also verify
	that you are serving files from the webroot path you provided.


host -t A example.com
example.com has address my.public.IP.address

host -t A www.example.com
www.example.com has address my.public.IP.address





certbot certonly --standalone --preferred-challenges http --http-01-port 8080 -d example.com -d www.example.com --deploy-hook="/usr/local/bin/hitch-deploy-hook" --pre-hook="systemctl stop lighttpd" --post-hook="systemctl start lighttpd"
Saving debug log to /var/log/letsencrypt/letsencrypt.log
Plugins selected: Authenticator standalone, Installer None
Running pre-hook command: systemctl stop lighttpd
Obtaining a new certificate
Performing the following challenges:
http-01 challenge for example.com
http-01 challenge for www.example.com
Waiting for verification...
Cleaning up challenges
Running deploy-hook command: /usr/local/bin/hitch-deploy-hook
Hook command "/usr/local/bin/hitch-deploy-hook" returned error code 1
Error output from hitch-deploy-hook:
Error: missing RENEWED_LINEAGE env variable.

Running post-hook command: systemctl start lighttpd

IMPORTANT NOTES:
 - Congratulations! Your certificate and chain have been saved at:
	/etc/letsencrypt/live/example.com/fullchain.pem
	Your key file has been saved at:
	/etc/letsencrypt/live/example.com/privkey.pem
	Your cert will expire on 2020-01-12. To obtain a new or tweaked
	version of this certificate in the future, simply run certbot
	again. To non-interactively renew *all* of your certificates, run
	"certbot renew"
 - If you like Certbot, please consider supporting our work by:

	Donating to ISRG / Let's Encrypt:   https://letsencrypt.org/donate
	Donating to EFF:                    https://eff.org/donate-le


missing RENEWED_LINEAGE env variable :
	https://community.letsencrypt.org/t/certbot-renewed-lineage-and-renewed-domains-shell-variable-clarification/33184/3
	https://github.com/certbot/certbot/issues/3502
	==>/etc/letsencrypt/live/$domain
	export RENEWED_LINEAGE='/etc/letsencrypt/live/example.com'



sudo systemctl enable certbot-renew.timer
sudo systemctl start certbot-renew.timer
	==> not available
systemctl list-unit-files | grep certbot
certbot.service                        static
certbot.timer                          enabled

systemctl enable certbot.timer && systemctl start certbot.timer
	==> ok

Step 6 - Start hitch

add to /etc/hitch/hitch.conf
pem-file = "/etc/letsencrypt/live/example.com/hitch-bundle.pem"

systemctl restart hitch


==========================================8<=========================================================
now let's finish configuring our webserver

Lighttpd on 8080 only :

server.bind = "127.0.0.1"	==> quotes are mandatory!!

==========================================8<=========================================================
...
...


Can't work with Hitch because it requires a "modern" kernel, whereas my server is actually a "containerized VM (OpenVZ), with an old kernel I can't upgrade 
==> STOP WORKING ON THIS SOLUTION, LET'S UNDO CHANGES

systemctl stop hitch && systemctl disable hitch
restore Varnish conf :
systemctl edit --full varnish
	/etc/systemd/system/varnish.service
systemctl restart varnish && systemctl status varnish

userdel -r _hitch
apt purge hitch
empty + remove /etc/hitch

Varnish + Nginx :

Here's the setup we're going to configure :

Certificates : stuff I already have from previous steps :

ll /etc/letsencrypt/live/example.com

total 12K
lrwxrwxrwx 1 root root   39 Oct 14 20:15 cert.pem -> ../../archive/example.com/cert2.pem
lrwxrwxrwx 1 root root   40 Oct 14 20:15 chain.pem -> ../../archive/example.com/chain2.pem
lrwxrwxrwx 1 root root   44 Oct 14 20:15 fullchain.pem -> ../../archive/example.com/fullchain2.pem
-rw------- 1 root root 5.6K Oct 14 20:21 hitch-bundle.pem
lrwxrwxrwx 1 root root   42 Oct 14 20:15 privkey.pem -> ../../archive/example.com/privkey2.pem
-rw-r--r-- 1 root root  692 Oct 14 20:08 README

Get more certificates :

This is because :

there are distinct websites hosted on the same domain example.com : www.example.com, ww2.example.com, ...
the certificates I requested so far (see previous certbot commands) don't cover them all

certbot certonly --standalone --preferred-challenges http --http-01-port 8080 -d example.com,ww2.example.com,www.example.com,ww3.example.com,ww4.example.com,ww5.example.com,ww6.example.com,ww7.example.com --pre-hook="systemctl stop lighttpd" --post-hook="systemctl start lighttpd"

Saving debug log to /var/log/letsencrypt/letsencrypt.log
Plugins selected: Authenticator standalone, Installer None

- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
You have an existing certificate that contains a portion of the domains you
requested (ref: /etc/letsencrypt/renewal/example.com.conf)

It contains these names: example.com, www.example.com

You requested these names for the new certificate: example.com,
ww2.example.com, www.example.com, ww3.example.com,
ww4.example.com, ww5.example.com, ww6.example.com,
ww7.example.com.

Do you want to expand and replace this existing certificate with the new
certificate?
- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
(E)xpand/(C)ancel: E
Running pre-hook command: systemctl stop lighttpd
Renewing an existing certificate
Performing the following challenges:
http-01 challenge for example.com
http-01 challenge for www.example.com
http-01 challenge for ww2.example.com
http-01 challenge for ww3.example.com
http-01 challenge for ww4.example.com
http-01 challenge for ww5.example.com
http-01 challenge for ww6.example.com
http-01 challenge for ww7.example.com
Waiting for verification...
Cleaning up challenges
Running deploy-hook command: /usr/local/bin/hitch-deploy-hook
Running post-hook command: systemctl start lighttpd

IMPORTANT NOTES:
 - Congratulations! Your certificate and chain have been saved at:
	/etc/letsencrypt/live/example.com/fullchain.pem
	Your key file has been saved at:
	/etc/letsencrypt/live/example.com/privkey.pem
	Your cert will expire on 2020-01-12. To obtain a new or tweaked
	version of this certificate in the future, simply run certbot
	again. To non-interactively renew *all* of your certificates, run
	certbot renew

Install Nginx :

Nginx is a little bit annoying since, upon installing, it tries to start listening on port TCP 80 automatically. Which fails since this port is used by Varnish.

Workaround :

systemctl stop varnish
apt install nginx
systemctl stop nginx
systemctl start varnish

Configure Nginx :

/etc/nginx/nginx.conf (sources : 1, 2, 3) :

user www-data;
worker_processes auto;
pid /run/nginx.pid;
include /etc/nginx/modules-enabled/*.conf;

stream {
	upstream stream_backend {
		server	127.0.0.1:80;
		}

	server {
		listen			127.0.0.1:443 ssl;
		proxy_pass		stream_backend;

		ssl_certificate		/etc/letsencrypt/live/example.com/fullchain.pem;
		ssl_certificate_key	/etc/letsencrypt/live/example.com/privkey.pem;
#		ssl_protocols		SSLv3 TLSv1 TLSv1.1 TLSv1.2;
		ssl_protocols		TLSv1 TLSv1.1 TLSv1.2;

#		ssl_ciphers		HIGH:!aNULL:!MD5;
		# get the full list of available ciphers with : openssl ciphers
		ssl_ciphers		ECDHE-ECDSA-AES128-GCM-SHA256:ECDHE-RSA-AES128-GCM-SHA256:ECDHE-ECDSA-AES256-GCM-SHA384:ECDHE-RSA-AES256-GCM-SHA384:ECDHE-ECDSA-CHACHA20-POLY1305:ECDHE-RSA-CHACHA20-POLY1305:DHE-RSA-AES128-GCM-SHA256:DHE-RSA-AES256-GCM-SHA384;

		ssl_prefer_server_ciphers	off;
		ssl_session_cache		shared:SSL:20m;
		ssl_session_timeout		4h;
		ssl_handshake_timeout		10s;

		# get this with :
		# curl https://ssl-config.mozilla.org/ffdhe2048.txt > /etc/nginx/dhparam.pem
		ssl_dhparam	/etc/nginx/dhparam.pem;
		}
	}

events {
	worker_connections 768;
	# multi_accept on;
	}

Then :

Start it :
systemctl start nginx
Check :
systemctl status nginx
Debug :
journalctl -u nginx -r

Reboot the universe :

systemctl restart lighttpd && systemctl restart varnish && systemctl restart nginx && systemctl restart sslh

Check your configuration :

Test the whole setup with ssllabs.com.

Let's Encrypt - Free SSL/TLS Certificates

Redirect HTTP traffic to HTTPS

Situation

Details

Solution