Keepalived - Loadbalancing & High-Availability

mail

How to switch the VIP to another host ?

On the host owning the VIP :
systemctl restart keepalived
This will make the current host briefly unavailable, and a new master will be chosen among the remaining ones.
mail

Who owns the VIP ? Aka Which is the active node ?

If the VIP is a hostname :

vip=$(host vip.company.tld | awk '/has address/ { print $NF }'); for i in {01..2}; do server="server$i.company.tld"; echo -n "$server : "; ssh -q "$server" "ip a | grep -Eq \"^ +inet $vip\" && echo 'has VIP' || echo 'no VIP'"; done

server01.company.tld : no VIP
server02.company.tld : has VIP

Subtleties of this command line :

/has address/
this is because
host vip.company.tld
may output several lines : 
vip.company.tld is an alias for foo.bar.baz.
foo.bar.baz has address 12.34.56.78

If the VIP is a regular IP address :

vip='12.34.56.78'; for i in {01..2}; do server="server$i.company.tld"; echo -n "$server : "; ssh -q "$server" "ip a | grep -Eq \"^ +inet $vip\" && echo 'has VIP' || echo 'no VIP'"; done

server01.company.tld : no VIP
server02.company.tld : has VIP

Subtleties of this command line :

grep -Eq \"^ +inet $vip\"
this is there to match this IP address :
inet 12.34.56.78/32 scope global eth0
but not this one :
inet xx.xx.xx.xx/23 brd 12.34.56.78 scope global eth1
from the output of ip a.
mail

Both hosts are reported as master

Situation

On both servers, the state file (/var/run/keepalived/state, details) reports the local keepalived as the master.

Details

This _may_ be caused by a synchronous systemctl restart keepalived on both hosts, not letting enough time to daemons to negotiate who's the master.

Solution

On the host you want to become the backup :
  1. systemctl stop keepalived
  2. sleep 1
  3. systemctl start keepalived

ssh -t ha02-prp "sudo systemctl stop keepalived && sleep 1 && sudo systemctl start keepalived"

mail

keepalived

Usage :

	+------------------+                     +------------------+
	|      host A      |                     |      host B      |
	|                  |     negotiation     |                  |
	|   +----------+   | <=================> |   +----------+   |
	|   |keepalived|   |                     |   |keepalived|   |
	|   +----------+   |                     |   +----------+   |
	+------------------+                     +------------------+
  • keepalived is installed on both nodes, with a similar (but not identical) configuration.
  • Nodes negotiate to elect the master.
  • The master owns the VIP.
  • keepalived is configured with a directory containing scripts to fire upon becoming the master.
  • when becoming the master, keepalived :
    • gets the VIP
    • lists scripts from the specified directory
    • fires them
  • when keepalived stops while being the master, the VIP switches to another host
  • AFAIK, there is no easy/clean way to force the VIP to be on either host. To do so, stop (restart) keepalived on the master host : the VIP will jump to another node (details).
  • keepalived relies on VRRP (an alternative to CARP, implemented in ucarp)
keepalived --version
	Keepalived v1.2.13 (05/28,2014)


keepalived.conf.SYNOPSIS
	https://github.com/acassen/keepalived/blob/master/doc/keepalived.conf.SYNOPSIS

Configuration :

Sample /etc/keepalived/keepalived.conf : 
global_defs {
	vrrp_check_unicast_src
	}

vrrp_script check_things {
	script "path/to/script.sh"
	interval 2
	weight 2
	}

vrrp_instance cluster {
	interface eth0
	virtual_router_id 1,
	unicast_src_ip 10.201.36.9		my IP
	unicast_peer {
		10.201.36.8			my peer's IP
		}
	priority 102
	nopreempt
	virtual_ipaddress_excluded {
		10.201.36.14			the VIP we're sharing (details)
		}
	track_script {
		check_things
		}

	notify "/usr/local/sbin/keepalived_state.sh"
	notify_stop "/usr/local/sbin/keepalived_state.sh unknown_type unknown_name STOPPED"
	}

Directives

Flag Usage
notify path/to/script scripts that keepalived invokes after changing state (sources : 1, 2)
notify_stop path/to/script [user group ... ] script to be called on VRRP service stop (source)
track_script List of scripts (referred to via the name of their corresponding vrrp_script entry) monitoring the state of other daemons. (source) :
  • If a configured script returns a non-zero exit code f times in succession, keepalived changes the state of the VRRP instance or group to FAULT, removes the virtual IP address 10.0.0.10 from eth0, reduces the priority value by w and stops sending multicast VRRP packets.
  • If the script subsequently returns a zero exit code r times in succession, the VRRP instance or group exits the FAULT state and transitions to the MASTER or BACKUP state depending on its new priority.
Looks like 
track_script {
	check_things
	}
automagically translates into : 
vrrp_script check_things {
	script "/usr/local/etc/keepalived_toggle.d/things"
	}
virtual_ipaddress { } a VIP definition block, up to 20 IP addresses
virtual_ipaddress_excluded { }
vrrp_script Adds a script to be executed periodically (sources : 1, 2)