Persistence Handling in LVS
This page contains information about persistence handling in LVS for sticky connection applications, such as http cookie, https, ftp, and so on.
Introduction
In the LVS cluster that we discuss in other documents, we have assumed that each network connection is independent of every other connection, so that each connection can be assigned to a server independently of any past, present or future assignments. However, there are times that two connections from the same client must be assigned to the same server either for functional or for performance reasons.
FTP is an example for a functional requirement for connection affinity. The client establishs two connections to the server, one is a control connection (port 21) to exchange command information, the other is a data connection (usually port 20) that transfer bulk data. For active FTP, the client informs the server the port that it listens to, the data connection is initiated by the server from the server's port 20 and the client's port. LinuxDirector could examine the packet coming from clients for the port that client listens to, and create any entry in the hash table for the coming data connection. But for passive FTP, the server tells the clients the port that it listens to, the client initiates the data connection connectint to that port. For the VS/TUN and the VS/DR, LinuxDirector is only on the client-to-server half connection, so it is imposssible for LinuxDirector to get the port from the packet that goes to the client directly.
SSL (Secure Socket Layer) is an example of a protocol that has connection affinity between a given client and a particular server. When a SSL connection is made, port 443 for secure Web servers and port 465 for secure mail server, a key for the connection must be chosen and exchanged. The later connections from the same client are granted by the server in the life span of the SSL key.
Persistent Port Solution
Our current solution to connection affinity is to add persistent port feature in LVS. In the persistent port, when a client first accesses the service, LinuxDirector will create a connection template between the given client and the selected server, then create an entry for the connection in the hash table. The template expires in a configurable time, and the template won't expire until all its connections expire. The connections for any port from the client will send to the server before the template expires. The timeout of persistent templates can be configured by users, and the default is 300 seconds.
Here is some information about the implementation of persistent port feature. For persistent ftp service, the template is created in the form like <cip, 0, vip, 0, rip, 0>, where cip is client IP address, vip is virtual IP address and rip is real server IP address. Then, ftp data connection for any port can be caught. For other persistent services, the template is created in the form like <cip, 0, vip, vport, rip, rport>, which grants that any connection from the same client for the persistent service will be sent to the same server, but different persistent services is irrelavant.
Although the persistent port may lead to slight load imbalance among servers, it is a good solution to connection affinity.
Persistent Granularity
In the persistent port feature by default, we have assumed that persistent granularity is per client. However, here is some application scenerios that we need persistent granularity is larger than per client. For example, we want to build a persistent service based on the LVS cluster, but some ISPs use the non-persistent proxy servers, which means that connections from the same clients will be sent to the different proxy servers. In the default, LinuxDirector will treat them from the different client and sent them to different servers, then the clients behind the non-persistent proxy servers cannot access our persistent service.
Here we use some conjesture to solve this problem. We assume that non-persistent proxy servers are often located in the same network, for example the network netmask is 255.255.255.0. We add the persistent netmask into persistent services. The client source address is masked with this netmask for the purpose of creating and accessing the templates, then all clients within the peristent netmask will access the same server. The default persistent mask is 255.255.255.255, which means that the persistent granularity is per client.
Examples
The following is some configuration example related to persistence.
1. Sticky http example of VS/NAT
ipvsadm -A -t virtualdomain:www -p
ipvsadm -a -t virtualdomain:www -r 192.168.1.2 -m
ipvsadm -a -t virtualdomain:www -r 192.168.1.3 -m
ipvsadm -a -t virtualdomain:www -r 192.168.1.4 -m
2. FTP example of VS/NAT
ipvsadm -A -t virtualdomain:ftp -p 540
ipvsadm -a -t virtualdomain:ftp -r 192.168.1.2 -m
ipvsadm -a -t virtualdomain:ftp -r 192.168.1.3 -m
ipvsadm -a -t virtualdomain:ftp -r 192.168.1.4 -m
3. Sticky http example of VS/TUN
ipvsadm -A -t virtualdomain:www -p
ipvsadm -a -t virtualdomain:www -r 192.168.1.2 -i
ipvsadm -a -t virtualdomain:www -r 192.168.1.3 -i
ipvsadm -a -t virtualdomain:www -r 192.168.1.4 -i
4. FTP example of VS/DR
ipvsadm -A -t virtualdomain:ftp -p 540
ipvsadm -a -t virtualdomain:ftp -r 192.168.1.2 -g
ipvsadm -a -t virtualdomain:ftp -r 192.168.1.3 -g
ipvsadm -a -t virtualdomain:ftp -r 192.168.1.4 -g
5. Catch-all persistence example of VS/DR
In some applications, all servers run the same two or more services, it requires that once a clients access one server of a server, all connections for other services from the same client must be sent to the same server in the specified time. We can use the port zero here to catch all persistent services, the configuration commands are as follows:
ipvsadm -A -t virtualdomain:0 -p
ipvsadm -a -t virtualdomain:0 -r 192.168.1.2 -g
ipvsadm -a -t virtualdomain:0 -r 192.168.1.3 -g
ipvsadm -a -t virtualdomain:0 -r 192.168.1.4 -g
6. Persistent granularity example
ipvsadm -A -t virtualdomain:www -p -M 255.255.255.0
ipvsadm -a -t virtualdomain:www -r 192.168.1.2
ipvsadm -a -t virtualdomain:www -r 192.168.1.3
ipvsadm -a -t virtualdomain:www -r 192.168.1.4