lvs cluster, NAT mode and DR mode, keepalive

Main function: Multiple back-end servers are combined into a high-availability and high-performance server cluster, and client requests are distributed to the back-end servers through a load balancing algorithm to achieve high availability and load balancing.

Alibaba's SLB server loab balance is implemented using lvs+keepalive.

Cluster and Distributed

System expansion methods:

Vertical expansion: Expand upward to enhance the performance of the computer. Bottleneck: The limitation of the computer itself; the performance bottleneck of the hardware itself

Horizontal expansion: Expand outward and add equipment. Run multiple services in parallel, rely on the network to solve internal communication problems, cluster cluster

Cluster: A single system composed of multiple computers combined to solve a specific problem

Types of clusters: Three types

LB：load balance Load balancing cluster, composed of multiple hosts, each host only bears a part of the access requests

HA：High availability: When designing a system, certain measures are taken to ensure that the entire system can still operate normally even if a component or part of the system fails. In order to maintain the availability, reliability, and fault tolerance of the system

HPC：High-performance computing High-performance clusters have higher requirements for response time and processing power

System reliability index

MYBF：Mean Time Between Failure Mean Time Between Failures

MTTR：Mean Time Restoration repair Mean time for failure recovery

A=MTBF/(MTBF+MTTR)

The A index should be between 0 and 1. The A index is a measure of system availability. 0 means the system is less available, and 1 means the system is more available.

The A index must be as close to 1 as possible (98%-99% qualified; 90%-95% unqualified)

All are in hours (365 days in a year, 8760 hours)

Downtime and planned time are not counted

Unplanned time, also known as failure time, is the total time from when a failure occurs to when it is resolved. Unplanned time is an indicator that we must pay attention to.

LVS applicable scenarios:

Small clusters do not need to use lvs, use nginx, large clusters use lvs

Terms in lvs clusters

VS: the logical name of the vital server LVS service, which is the IP address and port used when we access the LVS cluster externally

DS: director server The main server in the LVS cluster, that is, the scheduler (that is, the proxy server of nginx), is the core of the cluster. The scheduler is used to receive client requests and forward them to the backend server.

RS: real server LVS cluster real server, backend server, used to receive requests forwarded by DS and respond to the results

CIP: client ip client address, that is, the client address that initiates the request

VIP: virtual ip lvs The ip address used by the cluster, providing a virtual ip address for external cluster access

DIP: director ip address of the scheduler in the cluster, used to communicate with RS

RIP: real ip The ip address of the backend server in the cluster

How LVS works

NAT mode: The mode is responded to the client by the scheduler (address translation)

DR mode (direct routing mode): the real server responds directly to the client

TUN mode: tunnel mode

Commonly used modes: NAT mode and DR mode

NAT Mode

In NAT mode, LVS will modify the target IP address and port in the request message from the client to the internal IP address and port of LVS, and then forward the request to the backend server.

When the response result is returned to the client, the response message is processed by LVS and the target IP and port are modified to the client's IP address and port.

Principle: First, when the load balancer receives the client's request data packet, it decides to which backend real server (RS) to send the request based on the scheduling algorithm.
The load balancer then changes the destination IP address and port of the request data packet sent by the client to the IP address (RIP) of the backend real server.
After the real server responds to the request, it checks the default route and sends the response data packet to the load balancer. After receiving the response packet, the load balancer
Change the source address of the packet to the virtual address (VIP) and send it back to the client.

Advantages: The servers in the cluster can use any operating system that supports TCP/IP, as long as the load balancer has a valid IP address.

Disadvantages: Limited scalability. When the number of server nodes increases too much, all requests and responses need to go through the load balancer.
Therefore, the load balancer will become the bottleneck of the entire system.

Address Translation

Address translation: Intranet - Extranet What is converted is the source IP address SNAT

External network - internal network conversion is the destination IP address DNAT

lvs tools

ipvsadmTools, tools used to configure and manage LVS clusters

-A Add virtual server vip

-D Delete virtual server address

-s specifies the load balancing scheduling algorithm

-a Add real server

-d delete real server

-t specifies the address and port of vip

-r specifies the address and port of rip

-m Use NAT mode

-g Use DR mode

-i Use tunnel mode

-w Set weight

-p 60 The connection is maintained for 60 seconds. Set the connection maintenance time

-l list view

-n Digital display

algorithm

Poll rr

Weighted Round Robin wrr

Minimum connection lc

Weighted Least Connection (wlc)

experiment

Data Flow

nginx 1 RS1 192.168.233.20

nginx2 RS2 192.168.233.30

test1 scheduler ens33 192.168.233.10 ens36 12.0.0.1

test2 client 12.0.0.10

step

1. Scheduler configuration (test1 192.168.233.10)

yum -y install ipvsadm* -y

Configure ens33

systemctl restart network

Configure ens36

cd /etc/sysconfig/network-scripts/

cp ifcfg-ens33 ifcfg-ens36

vim ifcfg-ens36

systemctl restart network

2. RS configuration (nginx1 and nginx2)

Configure nginx1 192.168.233.20 to modify the gateway

systemctl restart network

Configure nginx2 192.168.233.30 to modify the gateway

systemctl restart network

vim /usr/local/nginx/html/index.html

Modify the content of the visited page

Check whether the access is connected

3. Address translation (test1 192.168.233.10)

iptables -t nat -vnL Check whether there is a policy in the nat table

iptables -t nat -A POSTROUTING -s 192.168.233.0/24 -o ens36 -j SNAT --to 12.0.0.1

Convert the device address from 192.168.233.0/24 to 12.0.0.1

ipvsadm -C clears the original policy

ipvsadm -A -t 12.0.0.1:80 -s rr specifies the address and port of the vip

First add VIP, virtual server IP and port, then add real server

ipvsadm -a -t 12.0.0.1:80 -r 192.168.233.20:80 -m

-a Add real server

-t specifies vip address

-r specifies the address and port of the real server

-m specifies the mode as NAT mode

ipvsadm -ln View

ipvsadm-save

ipvsadm -D -t 192.168.233.10:80 delete policy

ipvsadm -d -r 192.168.233.20: -t 12.0.0.1:80 Delete the node server

5. Enable routing forwarding function

vim /etc/sysctl.conf

net.ipv4.ip_forward=1

4. Client (test2 12.0.0.10)

Modify the IP address and gateway of test2

Experimental Results

Weighted Round Robin

DR mode

concept

It is direct routing mode

NAT mode: The scheduler is the most important in the entire LVS cluster. In NAT mode, it is responsible for receiving requests, forwarding traffic according to the load balancing algorithm, and sending responses to clients.

DR mode: The scheduler is still responsible for receiving requests and forwarding traffic to the RS according to the load balancing algorithm. The response is directly sent by the RS to the client.

Direct Routing: It is a layer 2 forwarding mode. The layer 2 forwards data frames based on the source MAC address and the destination MAC address. The source IP and destination IP of the data packet will not be modified. It is forwarded based on the MAC address of the data packet.

In DR mode, LVS also maintains a virtual IP address, and all requests are sent to this VIP. Since Layer 2 forwarding is used, when the client's request reaches the scheduler, an RS is selected according to the load balancing algorithm, and the destination MAC of the VIP server is modified to the MAC address of the RS. After RS processes the request, it directly sends the response to the client based on the source MAC address of the client in the message, without going through the scheduler.

Principle: First, when the load balancer receives the client's request data packet, it decides to which backend real server (RS) to send the request based on the scheduling algorithm.
The load balancer then changes the destination MAC address of the request data packet sent by the client to the MAC address of the backend real server (R-MAC).
After the real server responds to the request, it checks the default route and sends the response data packet directly to the client without going through the load balancer.

Advantages: The load balancer is only responsible for distributing request packets to the backend node server, while RS sends the response packet directly to the user.
Therefore, a large amount of data flow to the load balancer is reduced, and the load balancer is no longer the bottleneck of the system and can handle a huge amount of requests.

Disadvantages: The load balancer and the real server RS need to have a network card connected to the same physical network segment and must be in the same LAN environment.

Data Flow Diagram

question:

1. VIP address conflict

Reason: The scheduler is configured with VIP, and the RS is also configured with a VIP address. The VIP address conflicts, because the scheduler and RS are in the same network segment, which will cause ARP communication disorder. Because it is broadcast throughout the LAN, all devices receive it.

How to block the loopback response of lo so that only the local physical IP address responds.

Solution: Modify the kernel parameters:arp_ignore=1

The system's physical IP address will respond to ARP requests, and the lo loopback interface will not respond to ARP requests

2. When the message is returned, the VIP address is still there. How can the client receive the response?

Solution:arp_announce=2

The system does not use the source address of the IP packet to respond to the ARP request, but directly sends the IP address of the physical interface.

Implementation of DR mode

nginx1 RS (real IP) 192.168.233.20

nginx2 RS 192.168.233.30

vip 192.168.233.100

Scheduler 192.168.233.10

Client 192.168.233.40

step

1. Scheduler configuration (test1 192.168.233.10)

yum -y install ipvsadm* -y

Add virtual network card ens33:0

Modify the response parameters of the scheduler

vim /etc/sysctl.conf

net.ipv4.ip_forward=0
net.ipv4.conf.all.send_redirects=0
net.ipv4.conf.default.send_redirects=0
net.ipv4.conf.ens33.send_redirects=0

sysctl -p

Add a policy

cd /opt

ipvsadm -A -t 192.168.233.100:80 -s rr

ipvsadm -a -t 192.168.233.100:80 -r 192.168.233.20:80 -g

ipvsadm -a -t 192.168.233.100:80 -r 192.168.233.30:80 -g

ipvsadm-save >/etc/sysconfig/ipvsadm

systemctl restart ipvsadm

ipvsadm -ln

2. RS configuration (nginx1 and nginx2) [Modify both]

Modify the display content of static pages

vim /usr/local/nginx/html/index.html

systemctl restart nginx

Add loopback address

cd /etc/sysconfig/network-scripts/

cp ifcfg-lo ifcfg-lo:0

vim ifcfg-lo:0

Add lo:0 interface as vip

route add -host 192.168.233.100 dev lo:0

Set the IP address 192.168.233.100 to the loopback interface, as the VIP of LVS, forward it to RS through the routing mode, so that the VIP can identify the real server

Modify the kernel response of RS real server

vim /etc/sysctl.conf

net.ipv4.conf.lo.arp_ignore=1
net.ipv4.conf.lo.arp_announce=2
net.ipv4.conf.all.arp_ignore=1
net.ipv4.conf.all.arp_announce=2

Experimental Results

Modify the vip polling algorithm

Modify the weight in the policy poll

Summarize

The difference between lvs and nginx for load balancing

LVS is a layer 4 forwarding, which uses kernel-mode ip+port and can only be a layer 4 proxy

nginx four-layer proxy, or seven-layer proxy

Summary of the experiment

lvs (DR mode) + nginx + tomcat

LVS implements layer 4 forwarding + nginx implements layer 7 forwarding (dynamic). Accessing the VIP address of LVS can achieve dynamic and static separation.

Data Flow Diagram

Experimental Procedure

Based on the above DR mode experiment, dynamic and static separation is achieved

1. Tomcat part

1. Create dynamic pages in tomcat1 and tomcat2 respectively

<%@ page language="java" import="java.util.*" pageEncoding="UTF-8"%>
<html>
<head>
<title>JSP test1 page</title>
</head>
<body>
<% out.println("Dynamic page 1, http://www.test1.com");%>
</body>
</html>

2. Add sites in tomcat1 and tomcat2 respectively

cd conf

vim server.xml

Delete the original site first

Check whether the port is enabled

Visit 192.168.233.40:8080/index.jsp

2. nginx part

Configure nginx2 and nginx3

cd /usr/local/nginx/conf/

cp nginx.conf nginx.conf.bak.2024.07.08

vim nginx.conf

upstream tomcat {
server 192.168.233.40:8080 weight=1;
server 192.168.233.50:8080 weight=1;
}

location ~ .*.jsp$ {
           proxy_pass http://tomcat;
           proxy_set_header HOST $host;
           proxy_set_header X-Real-IP $remote_addr;
           proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
       }

Then systemctl restart nginx

Configure nginx1 proxy

Be a four-layer agent

cd /usr/local/nginx/conf

vim nginx.conf

Then systemctl restart nginx

Experimental Results

Visit 192.168.100:81 static page

Visit 192.168.233.100:82/index.jsp dynamic page

Three working modes of LVS

NAT DR TUN

Advantages Address translation, simple configuration Best performance WAN, can achieve data packet forwarding over longer distances

Disadvantages Performance bottleneck Does not support cross-segment dedicated channel, need to open VPN (cost money)

RS requirements No restrictions Must prohibit ARP responses from non-physical interfaces Must support tunnel mode

Number of RS 10-20 100 100

Interview questions:

1. Briefly describe the three modes and differences of LVS

The table above

2. How to solve keepalive split-brain problem