Varnish - How to ... ?

Error 503 Backend fetch failed

Backend fetch failed
Guru Meditation:

XID: 98441

Varnish cache server

• overloaded
• down for maintenance
• not fully functional for another reason

step 1 — identify your backend :

step 2 — make sure the backend is not overloaded :

step 3 — make sure the backend is not down for maintenance :

step 4 — investigate the Guru Meditation itself :

Method 1 — restart the daemon :

Method 2 — ban content with varnishadm :

varnishadm ban req.http.host == example.com '&&' req.url '~' '\\.png$'

(todo ;-)

1. sub detectHotlinking {
   	if(req.http.host == "my.website.tld" && req.http.referer ~ "(BADDOMAIN\.com|IMAGESUCKERDOMAIN\.com)" && req.url ~ "^/path/to/whatever.jpg$") {
   		return (synth (444, ""));
   		}
   	}

   • the req.http.host == "" test is there because, in my setup, Varnish serves several distinct virtualhosts but I want these hotlinking rules to apply to THIS virtualhost only.
   • the 444 code has no special meaning. You can choose any number you like, as long as what is output by detectHotlinking matches what vcl_synth expects as input (see below).

2. sub vcl_recv {
   	
   	call detectHotlinking;
   	
   	}

3. sub vcl_synth {
   	# "Hotlinking is BAD¹⁰⁰⁰⁰"
   	if (resp.status == 444) {
   		set resp.status = 302;
   		set resp.http.location = "/pictures/hotlinking.png";
   		}
   	}

   • this is where the 444 discussed above is matched. This function generally ends as a giant switch / case block handling distinct responses to different behaviors detected by one or more subs.
   • in the specific case of fighting hotlinking, the HTTP 302 code is interesting because the response is not cached. This means ALL requests to http://my.website.tld/path/to/whatever.jpg will be re-evaluated individually and served accordingly. Indeed, the web cache matches a request with a response without considering the referrer header, possibly leading to over-blocking.

• leave things as-is. Technically, this is ok. But if you're trying to take things seriously about your website, this is not a good thing, though :
  • nobody likes dead links
  • search engines may decrease your site notation if it has too many 404
• or let everybody aware —and especially search engines, actually— that the corresponding resource is gone forever and that it should be removed from the indexed resources for your website. This is exactly what the 410 status code is about.

1. add a function to declare removed resource and define what to do when receiving a request for such resource :

   sub declareRemovedResources {
   	if(req.http.host == "my.website.tld" && req.url ~ "path/to/.*[dD]ocument.*\.pdf") {
   	    return (synth (410, ""));		this could be any arbitrary code. 410 chosen for consistency
   	    }
   	}

2. run this function upon receiving requests :

   sub vcl_recv {
   	
   	call declareRemovedResources;
   	
   	}

3. apply the defined behavior :

   sub vcl_synth {
   	
   	if (resp.status == 410) {		this matches the arbitrary code described above
   	    set resp.status = 410;		this is the actual HTTP response code that will be returned
   	    }
   	
   
   	return(deliver);
   	}

4. check the config file
5. restart Varnish :
   systemctl restart varnish.service
6. check + restart :
   configFile='/etc/varnish/default.vcl'; varnishd -C -f "$configFile" 2>/dev/null && systemctl restart varnish.service || echo "error in '$configFile'"

• Varnish is setup to serve HTML contents on several virtualhosts :
  • A.example.com
  • B.example.com
  • ... and some others
• In the current setup, all logs are written to a single logfile by varnishncsa : /var/log/varnish/varnishncsa.log
• A.example.com and B.example.com are websites I'd like to get statistics about. The other virtualhosts are there for technical reasons or tests, and I don't need stats about them.
  To list these virtualhosts :

  1. grep 'example.com' /var/log/varnish/varnishncsa.log | grep -Ev 'A|B|C|'
  2. complete the 2^nd grep list until you caught them all.
• The stats software I'll use (AWStats) expects distinct logfiles per virtualhost to do its job. So I'll need 3 distinct logfiles :
  • the 1^st for A.example.com
  • the 2^nd for B.example.com
  • and a 3^rd for everything else (which won't be analyzed)
• varnishncsa itself is controlled by systemd. To write 3 distinct logfiles, the solution is to run 3 varnishncsa instances (sources : 1, 2, 3), fired by 3 custom service unit files.

1. Let's start by having a look at the current varnishncsa service unit file :
   cat /etc/systemd/system/varnishncsa.service

   [Unit]
   Description=Varnish Cache HTTP accelerator NCSA logging daemon
   After=varnish.service
   
   [Service]
   RuntimeDirectory=varnishncsa
   Type=forking
   PIDFile=/run/varnishncsa/varnishncsa.pid
   User=varnishlog
   Group=varnish
   ExecStart=/usr/bin/varnishncsa -a -w /var/log/varnish/varnishncsa.log -D -P /run/varnishncsa/varnishncsa.pid
   ExecReload=/bin/kill -HUP $MAINPID
   
   [Install]
   WantedBy=multi-user.target

2. stop the "old" configuration :
   systemctl stop varnishncsa.service
3. /etc/systemd/system/varnishncsa.service must not stay in place, otherwise it'll interfere with the system unit files we're about to create. Move it somewhere else or (my advice) : create a local Git repository, commit it as the "before" situation then use it to build the first specific service unit file.
4. Create the 3 specific service unit files (they must be named after whatever.service, details on custom service unit files) :
   • /etc/systemd/system/varnishncsa.A.service

     [Unit]
     Description=Varnish Cache HTTP accelerator NCSA logging daemon (A)
     After=varnish.service
     
     [Service]
     RuntimeDirectory=varnishncsa
     Type=forking
     PIDFile=/run/varnishncsa/varnishncsa_A.example.com.pid		looks redundant with -P below but seems necessary 
     User=varnishlog
     Group=varnish
     ExecStart=/usr/bin/varnishncsa -q "ReqHeader:Host eq 'A.example.com'" -D -a -w /var/log/varnish/A.example.com.log -P /run/varnishncsa/varnishncsa_A.example.com.pid
     ExecReload=/bin/kill -HUP $MAINPID
     
     [Install]
     WantedBy=multi-user.target

   • /etc/systemd/system/varnishncsa.B.service

     [Unit]
     Description=Varnish Cache HTTP accelerator NCSA logging daemon (B)
     After=varnish.service
     
     [Service]
     RuntimeDirectory=varnishncsa
     Type=forking
     PIDFile=/run/varnishncsa/varnishncsa_B.example.com.pid
     User=varnishlog
     Group=varnish
     ExecStart=/usr/bin/varnishncsa -q "ReqHeader:Host eq 'B.example.com'" -D -a -w /var/log/varnish/B.example.com.log -P /run/varnishncsa/varnishncsa_B.example.com.pid
     ExecReload=/bin/kill -HUP $MAINPID
     
     [Install]
     WantedBy=multi-user.target

   • /etc/systemd/system/varnishncsa.other.service

     [Unit]
     Description=Varnish Cache HTTP accelerator NCSA logging daemon (other: not 'A' and not 'B')
     After=varnish.service
     
     [Service]
     RuntimeDirectory=varnishncsa
     Type=forking
     PIDFile=/run/varnishncsa/varnishncsa_other.example.com.pid
     User=varnishlog
     Group=varnish
     ExecStart=/usr/bin/varnishncsa -q "ReqHeader:Host !~ '(A|B).example.com'" -D -a -w /var/log/varnish/other.example.com.log -P /run/varnishncsa/varnishncsa_other.example.com.pid
     ExecReload=/bin/kill -HUP $MAINPID
     
     [Install]
     WantedBy=multi-user.target

5. Start it all :
   chmod 664 /etc/systemd/system/varnishncsa.*.service; systemctl daemon-reload && systemctl start varnishncsa.A.service; systemctl start varnishncsa.B.service; systemctl start varnishncsa.other.service
6. If everything works fine (and provided there is traffic coming to the websites), you should see some logfiles :
   ll /var/log/varnish/*.example.com.log
7. Don't forget to logrotate your new log files.

https://unix.stackexchange.com/questions/463321/how-to-split-varnishncsa-logs-into-separate-under-systemd-ubuntu-16-04#answer-463446
So for each unit file, 1 per vhost, you'd add your exec line like so:
unit file #1

ExecStart=/usr/bin/varnishncsa -q "ReqHeader ~ '^Host: somedomain1.com'" -D -a -w /var/log/varnish/somedomain1.log -P /run/varnishncsa/varnishncsa_vhost1.pid -F '%%{X-Forwarded-For}i %%l %%u %%t "%%r" %%s %%b "%%{Referer}i" "%%{User-agent}i"'

unit file #2

ExecStart=/usr/bin/varnishncsa -q "ReqHeader ~ '^Host: somedomain2.com'" -D -a -w /var/log/varnish/somedomain2.log -P /run/varnishncsa/varnishncsa_vhost2.pid -F '%%{X-Forwarded-For}i %%l %%u %%t "%%r" %%s %%b "%%{Referer}i" "%%{User-agent}i"'

configFile='/etc/varnish/default.vcl'; varnishd -C -f "$configFile" 2>/dev/null && echo "$configFile : OK" || echo "$configFile : KO"

Important notes before we start :

Configuration :

sub vcl_recv {
	
	call detectHotLinking;
	

sub detectHotLinking {
	if(req.http.host == "my.site.tld" && req.http.referer ~ "evil\.hotlinker\.com") {
		return (synth (750, ""));
		}
	}

sub vcl_synth {
	if (resp.status == 750) {
		set resp.status = 301;
		set resp.http.Location = "http://images.google.com";
		}
	return(deliver);
	}

Test :

More filters :

1. telnet varnishHost 6082
2. if required, please authenticate
3. backend.list

   200 1427
   Backend name			Refs	Admin	Probe
   host9(10.0.16.19,,80)	12	probe	Healthy 5/5
   host10(10.0.16.22,,80)	10	probe	Healthy 5/5
   host11(10.0.16.23,,80)	6	probe	Healthy 5/5
   host12(10.0.16.24,,80)	10	probe	Healthy 5/5
   host21(10.0.16.117,,80)	7	probe	Healthy 5/5
   host134(10.0.16.134,,80)	6	probe	Healthy 5/5
   host167(10.0.16.167,,80)	2	probe	Sick 0/5

sub vcl_deliver {
	if (obj.hits > 0) {
		set resp.http.X-Cache = "HIT";
		}
	else {
		set resp.http.X-Cache = "MISS";
		}
	}

1. First, you should have a /etc/varnish/secret file containing a secret key.
2. Then, upon opening a telnet session, you are presented with a challenge :

   Connected to localhost.localdomain.
   Escape character is '^]'.
   107 59
   uzaiqheubccbpimwyyevwqfedtxuqdwm
   
   Authentication required.
   
   

   The CLI 107 status code means that authentication is requested.
3. To authenticate (assuming the secret key is foo), you'll have to compute the response with :
   challenge='uzaiqheubccbpimwyyevwqfedtxuqdwm'; secret=$(cat /etc/varnish/secret); echo -e "${challenge}\n${secret}\n${challenge}" | sha256sum

   2bcd7855d7d63870aecaa7f7c5eeeae3166581644f8216698558d78f19c3bdc2

   The documentation states that the string to be sha256sum'd is made of : challenge + newline + secret + challenge + newline, while the command above adds a newline after the secret, and no newline after the final challenge. This is because the secret is read by cat (which adds no newline), and the whole string is output by echo (that DOES add a trailing newline).

4. Back to the telnet session, enter :
   auth 2bcd7855d7d63870aecaa7f7c5eeeae3166581644f8216698558d78f19c3bdc2
   Then you get :

   200 239
   -----------------------------
   Varnish Cache CLI 1.0
   -----------------------------
   Linux,2.6.32-042stab106.4,x86_64,-smalloc,-smalloc,-hcritbit
   varnish-4.0.3 revision b8c4a34
   
   Type 'help' for command list.
   Type 'quit' to close CLI session.
   
   

   The CLI 200 status code means OK.