IPVS Connection Synchronization

This page describes connection synchronization implemented in IPVS and its usage.

1. Introduction

In order to make load balancer failover transparent to client applications, the primary load balancer need synchronize its state (e.g. connection information) to the load balancer(s). When the backup load balancer takes over after the primary one fails, the backup load balancer have the state of most connections, so that almost all connections can continue to access the service through the backup load balancer.

Since the load balancer handles packets at a very high speed and the connection state changes frequently, we need design an efficient mechanism to transfer the state, then this connection synchronization will not introduce too much performance penalty to the load balancer.

For example, the primary must transfer the following state of each connection to the backup(s):

	<Protocal, CIP:CPort, VIP:VPort, RIP:RPort, Flags, State>

which need 24bytes at least. Supposing the load balancer is working at 5,000 connections/second, then the load balancer needs to transfer 120KBytes at least to the backup every second.

2. Implemenation

In order to keep performance penalty of connection synchronization small, we adopt a strategy to implement it inside the kernel, so that a lot of overhead in memory copying and context switching between the kernel and the user-space can be avoided. And, the connection synchronization is implemented inside the ipvs code, so that all the LVS user-space monitoring programs can easily use this feature.

We use UDP multicast to propogate the creation/changes of connections from the primary load balancer to a set of backup load balancers. The primary load balancer puts the changes of connections in the sync queue. An ipvs syncmaster daemon is started inside the kernel on the primary load balancer, and it multicasts connection state in the queue periodically. An ipvs syncbackup daemon is started inside the kernel too on each backup load balancer, and it accepts multicast messages and create/change corresponding connections.

This feature is available at ipvs 0.9.2 (for kernel 2.4) or later.

3. Usage

Here is simple intructions to use IPVS connection synchronization. On the primary load balancer, run

	ipvsadm --start-daemon=master --mcast-interface=eth0

On the backup load balancers, run 

	ipvsadm --start-daemon=backup --mcast-interface=eth0

To stop the daemon, run 

	ipvsadm --stop-daemon

See ipvsadm man page for more information about the usage of ipvs connection synchronization.

Note that the feature of connection synchronization is under experiment now, and there is some performance penalty when connection synchronization, because a highly loaded load balancer may need multicast a lot of connection information. If the daemon is not started, the performance will not be affected.