How to set up PfSense High Availability (hardware redundancy)

What is High Availability in PfSense?

High Availability (HA) in PfSense comes down to hardware redundancy, essentially having a hot spare instantly taking over a router that becomes unavailable, aka failover. Instantly in this case being one or two seconds, without firewall states being broken, so your file will just continue downloading and your video will continue streaming.

If you find this article useful feel free to click some of the ads on this page. It won’t make me rich but it will tell me someone appreciated my work. If you’re really euphoric about it there’s also the PayPal donate button on the right ;)

HA is a combination of three different techniques:
– Common Address Redundancy Protocol (CARP), a protocol that allows for network devices to share virtual IP addresses;
– firewall state synchronisation (pfsync);
– settings synchronisation (XMLRPC sync).

CARP requires at least three public IP addresses: one virtual address for outgoing traffic and one for each node (=PfSense router). Since there’s no such thing as redundancy with only one node you will need at least two. You can add more if you like.

What is HA used for in PfSense?

Most use cases describe one of these purposes:
– failover in case of a hardware problem;
– reboots without downtime;
– internet connectivity redundancy (fall-back internet connections).

I will be describing mono-WAN HA here, so just one internet connection. If someone will sponsor me more connections I’ll be happy to document multi-WAN HA here ;)

Notes

– PfSense documents HA here. It’s a starting point but in my opinion the article is in places ambiguous and overly complicated.
– VHIDs are important – if you’re in a network using VHIDs make sure they are not in use or you will end up confused. If unsure pick a random high number, like 178 or 49, instead of beginning at 1.
– The PfSense installations need to be on the exact same version. If they aren’t discrepancies can occur in the xml configuration files resulting in weird errors.
– Sorry for the watermarking; some sackwasher has been copying my articles and is selling them behind a paywall.

I am setting this up on VirtualBox. We need three machines: PfSense1; PfSense2; and a client.
Each PfSense firewall gets three interfaces:
1. a bridged connection to a physical nic (both firewall vms can bridge to the same nic if necessary) for the WAN connection;
2. an internal network called LAN for the LAN connection;
3. an internal network called SYNC for the pfsync and XMLRPC connection.

You can change the VirtualBox internal network name by just clicking the name and typing an other one. You can define groups in the same way: just type or select the network name.

Note that if you’re testing on VirtualBox you must set all network interfaces to Promiscuous Mode ‘Allow All’.

(CARP troubleshooting here: https://doc.pfsense.org/index.php/CARP_Configuration_Troubleshooting)

It is not necessary to use two different internal networks for LAN and SYNC; however I like to approach the physical situation as much as possible.

The client needs one network interface, connected to the LAN internal network.

What do I need?

As suggested it’s best to use physical hardware. Use two machines with at least three network interfaces each: one for the internet connection (WAN), one for the LAN connection and one for the synchronisation (OPT1, renamed to SYNC for easy recognition).

The reason to use a separate physical network interface for synchronisation is that pfsync (firewall state synchronisation) is not encrypted and pfsync with XMLRPC can take a huge bite out of your available bandwidth. For testing purposes it is ok to use the LAN interface for synchronisation but your shouldn’t use it for production.

My example setup

PfSense names the third interface OPT1 by default; I’ve renamed those to SYNC.

Set up your two PfSense vms as regular firewalls. Use this ip addressing table:

PfSense 1
nic	purpose	ip	mac
en0	wan	83.167.196.114/28	08002729F030
em1	lan	192.168.1.1/24	0800279D9AC1
em2	sync	192.168.4.1/24	08002739F94B
PfSense 2
nic	purpose	ip	mac
em0	wan	83.167.196.125/28	080027854A05
em1	lan	192.168.1.2/24	080027FBE5CA
em2	sync	192.168.4.2/24	0800275510E6

The 83.167.196.114/28 network is my public IP range. The public IP address my clients will be using to go online is 83.167.196.126.

Why 192.168.4.0/24 instead of 192.168.2.0/24? Because 1.2 is easier confused with 2.1 than 4.1 ;) The /24 subnet is a PfSense recommendation. I guess this doesn’t really matter but I haven’t tried.

The steps

In theory the order of the steps is not important. In practice it helps if you stick to it.

Before you begin, set up two identical PfSense machines using the IPs mentioned for PfSense 1 and 2 above as examples. Use your own IPs, not mine. Set them up like you would with normal PfSense routers. Both should work individually as functioning routers. The only differences between them should be the LAN ip, the hostname and if you’re setting up dhcp you should do so on just one of them. The two firewalls should use the exact same version.

1. state and settings synchronisation (pfsync)
2. create virtual IPs (CARP)
3. change outbound NAT
4. setting DHCP and DNS
5. test
6. 1:1 IP mapping for internet facing servers

State synchronisation

For state sync we’ll be using a separate network. First assign the interfaces to be used to PfSense: Interfaces > Assignments. Click Add to add your third network adpater. This will give you the OPT1 interface.

Click OPT1 to edit its settings.

Enabled	checked
Description	SYNC
IPv4 Configuration Type	Static IPv4
IPv6 Configuration Type	None
IPv4 Address	192.168.4.1/24
IPv4 Upstream gateway	None
Block private networks and loopback addresses	not checked
Block bogon networks	not checked

Click Save and then do the same on the other firewall, using IP address 192.168.4.2.

On both firewalls add two rules to allow traffic on the SYNC interface: go to Firewall > Rules > Sync and click Add.

Action	Pass
Interface	SYNC
Address Family	IPv4
Protocol	CARP
Source	SYNC net
Destination	SYNC address
Description	Allow CARP

Click Save, Apply and add another rule:

Action	Pass
Interface	SYNC
Address Family	IPv4
Protocol	TCP (do not choose PFSYNC here)
Source	SYNC net
Destination	SYNC address
Destination Port Range	your PfSense web interface port number here. Mine is on port 80. Remember to change it if you change your web interface port number.
Description	Allow XMLRPC Sync

Save, Apply. The rules should look like this:

Go to System > High Avail. Sync. There are two sections here: State Synchronization Settings (pfsync) and Configuration Synchronization Settings (XMLRPC Sync).

[State Synchronization Settings (pfsync)]
Fill out this section on both firewalls.

Synchronization state	checked
Synchronize Interface	SYNC
pfsync Synchronize Peer IP	192.168.4.2

On the first firewall enter the IP address of the second firewall as peer IP and vice versa. They should point to each other.

[Configuration Synchronization Settings (XMLRPC Sync)]
Fill out this section only on the first firewall. This will become your master; the other firewall will be the slave. Do not fill out this section on the second firewall.

Synchronize Config to IP: 192.168.4.2
Click Toggle All so everything is selected then click Save.

Let’s check if synchronisation is working: go to Firewall > Aliases and create a new alias. Anything will do. Then check if the alias also appears on the second firewall. This should be nearly instantaneous (<2 seconds).

If it doesn’t work, check:
– Are the firewall web interfaces running on the same protocols and ports?
– Is the admin password set correctly? (User Manager > Users > admin.)
– Are the firewall rules to allow synch set to use the correct interface (SYNC)?
– If you’re using VMs, are the firewalls on the same internal network?

Remember that if it does work, any changes are synchronised to the other firewall. So:
– removing rules from the master firewall will also remove them from the slave;
– as a rule of thumb only make changes on the master firewall;
– if you edit rules on the slave they will not be synchronised to the master.

Note that if you change the management web interface port you do not need to change it here; you just need to change the firewall rule allowing the traffic.

Do not proceed untill synchronisation is working correctly.

create virtual IPs (CARP)

On the master (first) firewall go to Firewall > Virtual IPs and click Add.

Type	CARP
Address(es)	83.167.196.126/28 (the public IP address your firewalls will both be listening on and your clients’ outgoing public IP address – obviously 83.167.196.114/28 is not my real address).
Virtual Password	Enter some password here. Use the same password on the other firewall.
VHID Group	Choose a number that’s not in use in your entire network.
Advertising frequency	1
Skew	0
Description	WAN VIP

Add another CARP address on the LAN interface.

Type	CARP
Address(es)	192.168.1.3/24
Virtual Password	Enter some password here. Use the same password on the other firewall. Does not need to be the same as the WAN VIP VHID password.
VHID Group	Choose a number that’s not in use in your entire network. Choose a different one than the WAN VIP VHID Group number.
Advertising frequency	1
Skew	0
Description	LAN VIP

The result should look like this:

Now verify that on the slave firewall the same virtual IPs were made automatically. Click the edit icon next to one of the VIPs. If they were not created go back and check your synchonization configuration.

Verify that pfSense has automatically set the skew value on the slave firewall to 100 (or in any case the master firewall’s skew value plus 100).

Change outbound NAT

Set your client to use the master pfSense firewall as its default gateway. Use a service such as What Is My IP to check your outbound public IP address. It should show the master pfSense firewall’s public IP address.

We need to change this because we want to use the shared public IP (the WAN VIP) so that if the master goes down the clients won’t need to change any configuration and sessions remain alive.

Go to Firewall > NAT > Outbound. Note that at the moment ‘Automatic outbound NAT rule generation’ is selected. Change this to ‘Manual Outbound NAT rule generation’ and click Save.

The Mappings list will look a bit different. Find your LAN IP ranges (there should be two) and click the edit icon next to the first. The only thing you need to change here is the Translation Address: change it from Interface Address to the WAN VIP address.

Do the same for the other LAN network mapping. It should end up looking like this:

The slave pfSense firewall should have these settings synchronised automatically.

Set your client to use the LAN VIP (192.168.1.3) as its default gateway and verify (using a tool such as What Is My IP) that your outgoing public IP address has now changed to the WAN VIP.

Setting DHCP and DNS

If you’ll be using your pfSense firewall as a DNS resolver you must change its outgoing network interface to your WAN VIP: go to Services > DNS Resolver > General Settings and under ‘Outgoing Network Interfaces’ select your WAN VIP.

Under Services > DHCP Server > LAN set DNS server and Gateway to the LAN VIP (192.168.1.3) and set the Failover peer IP to the slave firewall (192.168.1.2). The Slave firewall will automatically have the master firewall IP configured.

Test

Make sure your client is receiving its network configuration from the LAN VIP and let a continuous ping run to an external address. Go to Status > CARP (failover). The master firewall should show two addresses with MASTER. The slave should show Backup statuses.

Click the ‘Enter Persistent CARP Maintenance Mode’ button.

Now the master should show Backup and the slave should say Master. Click the button again to leave persistent CARP maintenance mode.

During the CARP maintenance mode your client’s ping shouldn’t show more than one or two missed answers; it will probably have missed not one. You can use the persistent CARP maintenance mode to perform hardware maintenance, for instance, or reboot the machine.

Of course you can always just disconnect one or more network cables from the firewall and see what happens.

If things don’t work like they should, check if you are using VHIDs that are unique in your network. If you aren’t sure try one of the higher VHIDs and/or contact your network admin.

1:1 IP mapping for internet facing servers

Let’s say you have web server that needs to be available on a separate public IP address (within your range, of course): IP address 83.167.196.116, which falls within your 83.167.196.114/28 range, needs to point to your webserver which has internal IP address 192.168.1.100.

Start out by adding a new CARP virtual IP address: Firewall > Virtual IPs > Add.

Type	CARP
Interface	WAN
Address(es)	Your webserver’s public facing IP address, e.g. 83.167.196.116/28 – note that like it says this does not specify a CIDR range; the subnet is used to identify the address’s subnet.
Virtual IP Password	Choose one.
VHD Group	Pick one that’s unique in your network.
Advertising frequency	1, Skew 0
Description	Webserver (or whatever)

Click Save; verify that the new CARP VIP is synchronised to the slave firewall(s) and that their Skew value is set to 100 (or in any case higher than the master’s).

To make sure that your webserver will also be available on its public IP address (and/or domain name) from inside your LAN we must enable NAT reflection 1:1: go to System > Advanced > Firewall & NAT, scroll down to Network Address Translation and check Enable NAT Reflection for 1:1 NAT and Enable automatic outbound NAT for Reflection. As this is not replicated to the slave firewalls you must set this manually on the other firewall(s).

Create a WAN firewall rule to allow port 80 (or whatever ports or aliases you need) to the webserver: Firewall > Rules > WAN > Add.

Make sure that you set the Interface to WAN and the Destination to your webserver’s internal IP address.

At this point your webserver should be available from the internet and from within your LAN and your webserver will use its own dedicated CARP VIP for internet traffic.