openeuler集群架构

发表于 分类于 ，
openEuler HCIP实验手册集群架构部分

openeuler集群架构

lvs配置实践

lvs/net模式配置

软件版本
nginx1.21.5
ipvsadm1.31
节点IP1IP2系统功能CPU内存硬盘
node110.80.20.1openeuler20.03client2核心4GB20GB
node210.80.20.210.10.20.2openeuler20.03lvs2核心4GB20GB
node310.80.20.310.10.20.3openeuler20.03nginx2核心4GB20GB
node410.80.20.410.10.20.4openeuler20.03nginx2核心4GB20GB

node3、node4

配置好网络后，将ip2的dns设置为lvs的ip2的地址：

1
2
3
4
5
6
# node3
# nmcli con mod ens224 ipv4.address 10.10.20.3/16 ipv4.gateway 10.10.20.2
# nmcli con down ens224 && nmcli con up ens224
# node4
# nmcli con mod ens224 ipv4.address 10.10.20.4/16 ipv4.gateway 10.10.20.2
# nmcli con down ens224 && nmcli con up ens224

下载安装nginx：

1
2
3
# dnf install -y nginx
# systemctl enable nginx --now
# systemctl status nginx

配置访问页面：

1
2
3
4
5
6
7
8
# node3
# echo "nginx web 10.10.20.3" > /usr/share/nginx/html/index.html
# curl 10.10.20.3
nginx web 10.10.20.3
# node4
# echo "nginx web 10.10.20.4" > /usr/share/nginx/html/index.html
# curl 10.10.20.4
nginx web 10.10.20.4

关闭ip1网卡：

1
# ifdown ens160

node2

开启流量转发功能：

1
2
3
# sed -i "s/ip_forward=0/ip_forward=1/g" /etc/sysctl.conf
# sysctl -p | grep ip_forward
net.ipv4.ip_forward = 1

访问nginx：

1
2
3
4
# curl 10.10.20.3
nginx web 10.10.20.3
# curl 10.10.20.4
nginx web 10.10.20.4

下载安装ipvsadm：

1
# dnf install -y ipvsadm

创建配置文件，启动ipvsadm：

1
2
3
# touch /etc/sysconfig/ipvsadm
# systemctl enable ipvsadm --now
# systemctl status ipvsadm

创建轮训的算法：

1
# ipvsadm -A -t 10.80.20.2:80 -s rr

添加轮训的后端服务器：

1
2
# ipvsadm -a -t 10.80.20.2:80 -r 10.10.20.3 -m
# ipvsadm -a -t 10.80.20.2:80 -r 10.10.20.4 -m

查看配置：

1
2
3
4
5
6
7
# ipvsadm -Ln
IP Virtual Server version 1.2.1 (size=4096)
Prot LocalAddress:Port Scheduler Flags
  -> RemoteAddress:Port           Forward Weight ActiveConn InActConn
TCP  10.80.20.2:80 rr
  -> 10.10.20.3:80                Masq    1      0          0         
  -> 10.10.20.4:80                Masq    1      0          0

node1

访问测试：

1
2
3
4
5
6
7
8
9
10
11
# for i in {1..10}; do curl 10.80.20.2; done
nginx web 10.10.20.4
nginx web 10.10.20.3
nginx web 10.10.20.4
nginx web 10.10.20.3
nginx web 10.10.20.4
nginx web 10.10.20.3
nginx web 10.10.20.4
nginx web 10.10.20.3
nginx web 10.10.20.4
nginx web 10.10.20.3

node2

轮训修改为加权轮训算法：

1
2
3
4
5
6
7
8
# ipvsadm -E -t 10.80.20.2:80 -s wrr
# ipvsadm -Ln
IP Virtual Server version 1.2.1 (size=4096)
Prot LocalAddress:Port Scheduler Flags
  -> RemoteAddress:Port           Forward Weight ActiveConn InActConn
TCP  10.80.20.2:80 wrr
  -> 10.10.20.3:80                Masq    1      0          7         
  -> 10.10.20.4:80                Masq    1      0          7

修改权重，nginx3为2，node4为1：

1
2
3
4
5
6
7
8
9
# ipvsadm -e -t 10.80.20.2:80 -r 10.10.20.3 -m -w 2
# ipvsadm -e -t 10.80.20.2:80 -r 10.10.20.4 -m -w 1
# ipvsadm -Ln
IP Virtual Server version 1.2.1 (size=4096)
Prot LocalAddress:Port Scheduler Flags
  -> RemoteAddress:Port           Forward Weight ActiveConn InActConn
TCP  10.80.20.2:80 wrr
  -> 10.10.20.3:80                Masq    2      0          0         
  -> 10.10.20.4:80                Masq    1      0          0

node1

访问测试：

1
2
3
4
5
6
7
8
9
10
11
# for i in {1..10}; do curl 10.80.20.2; done
nginx web 10.10.20.4
nginx web 10.10.20.3
nginx web 10.10.20.3
nginx web 10.10.20.4
nginx web 10.10.20.3
nginx web 10.10.20.3
nginx web 10.10.20.4
nginx web 10.10.20.3
nginx web 10.10.20.3
nginx web 10.10.20.4

lvs/dr模式配置

软件版本
nginx1.21.5
ipvsadm1.31
节点IP1IP2VIP系统功能CPU内存硬盘
node110.80.20.1openeuler20.03client2核心4GB20GB
node210.80.20.210.10.20.210.10.20.100openeuler20.03lvs2核心4GB20GB
node310.80.20.310.10.20.310.10.20.100openeuler20.03nginx2核心4GB20GB
node410.80.20.410.10.20.410.10.20.100openeuler20.03nginx2核心4GB20GB

node3、node4

添加vip：

1
# nmcli con add type dummy ifname dummy2 ipv4.method manual ipv4.address 10.10.20.100/32

修改网关：

1
2
# nmcli con modify ens224 ipv4.gateway 10.10.20.1
# ifdown ens224 && ifup ens224

配置内核转发：

1
2
3
4
5
6
7
8
9
10
11
# cat >> /etc/sysctl.conf << EOF
net.ipv4.conf.all.arp_ignore=1
net.ipv4.conf.all.arp_announce=2
net.ipv4.conf.dummy2.arp_ignore=1
net.ipv4.conf.dummy2.arp_announce=2
EOF
# sysctl -p | tail -4
net.ipv4.conf.all.arp_ignore = 1
net.ipv4.conf.all.arp_announce = 2
net.ipv4.conf.dummy2.arp_ignore = 1
net.ipv4.conf.dummy2.arp_announce = 2

下载安装nginx：

1
2
3
# dnf install -y nginx
# systemctl enable nginx --now
# systemctl status nginx

配置访问页面：

1
2
3
4
5
6
7
8
# node3
# echo "nginx web 10.10.20.3" > /usr/share/nginx/html/index.html
# curl 10.10.20.3
nginx web 10.10.20.3
# node4
# echo "nginx web 10.10.20.4" > /usr/share/nginx/html/index.html
# curl 10.10.20.4
nginx web 10.10.20.4

关闭ip1网卡：

1
# ifdown ens160

node2

添加vip：

1
# nmcli con add type dummy ifname dummy2 ipv4.method manual ipv4.address 10.10.20.100/32

修改网关：

1
2
# nmcli con modify ens224 ipv4.gateway 10.10.20.1
# ifdown ens224 && ifup ens224

下载安装ipvsadm：

1
# dnf install -y ipvsadm

创建配置文件，启动ipvsadm：

1
2
3
# touch /etc/sysconfig/ipvsadm
# systemctl enable ipvsadm --now
# systemctl status ipvsadm

创建轮训的算法：

1
# ipvsadm -A -t 10.10.20.100:80 -s rr

添加轮训的后端服务器：

1
2
# ipvsadm -a -t 10.10.20.100:80 -r 10.10.20.3
# ipvsadm -a -t 10.10.20.100:80 -r 10.10.20.4

查看配置：

1
2
3
4
5
6
7
# ipvsadm -Ln
IP Virtual Server version 1.2.1 (size=4096)
Prot LocalAddress:Port Scheduler Flags
  -> RemoteAddress:Port           Forward Weight ActiveConn InActConn
TCP  10.10.20.100:80 rr
  -> 10.10.20.3:80                Route   1      0          0         
  -> 10.10.20.4:80                Route   1      0          0

node1

访问测试：

1
2
3
4
5
6
7
8
9
10
11
# for i in {1..10}; do curl 10.10.20.100; done
nginx web 10.10.20.4
nginx web 10.10.20.3
nginx web 10.10.20.4
nginx web 10.10.20.3
nginx web 10.10.20.4
nginx web 10.10.20.3
nginx web 10.10.20.4
nginx web 10.10.20.3
nginx web 10.10.20.4
nginx web 10.10.20.3

nginx反向代理和负载均衡实践

软件版本
nginx1.21.5
节点IP1IP2系统功能CPU内存硬盘
node110.80.20.1openeuler20.03client2核心4GB20GB
node210.80.20.210.10.20.2openeuler20.03nginx2核心4GB20GB
node310.80.20.310.10.20.3openeuler20.03nginx2核心4GB20GB
node410.80.20.410.10.20.4openeuler20.03nginx2核心4GB20GB

node3、node4

下载安装nginx：

1
2
3
# dnf install -y nginx
# systemctl enable nginx --now
# systemctl status nginx

配置访问页面：

1
2
3
4
5
6
7
8
# node3
# echo "nginx web 10.10.20.3" > /usr/share/nginx/html/index.html
# curl 10.10.20.3
nginx web 10.10.20.3
# node4
# echo "nginx web 10.10.20.4" > /usr/share/nginx/html/index.html
# curl 10.10.20.4
nginx web 10.10.20.4

关闭ip1网卡：

1
# ifdown ens160

node2

下载安装nginx：

1
2
3
# dnf install -y nginx
# systemctl enable nginx --now
# systemctl status nginx

添加负载均衡配置：

1
2
3
4
5
6
7
8
9
10
11
12
# vim /etc/nginx/conf.d/lb.conf
upstream 10.10.20.2 {
    server 10.10.20.3:80;
    server 10.10.20.4:80;
}
server {
    listen 80;
    server_name localhost;
    location / {
        proxy_pass http://10.10.20.2;
    }
}
1
# nginx -s reload

node1

客户端访问：

1
2
3
4
5
6
7
8
9
10
11
# for i in {1..10}; do curl 10.10.20.2; done
nginx web 10.10.20.3
nginx web 10.10.20.4
nginx web 10.10.20.3
nginx web 10.10.20.4
nginx web 10.10.20.3
nginx web 10.10.20.4
nginx web 10.10.20.3
nginx web 10.10.20.4
nginx web 10.10.20.3
nginx web 10.10.20.4

node3、node4

配置虚拟主机：

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
# vim /etc/nginx/conf.d/vhost.conf
server {
    listen 0.0.0.0:80;
    root /data/Nginx/;
    server_name localhost;
    index index80.html;
}
server {
    listen 0.0.0.0:81;
    root /data/Nginx/;
    server_name localhost;
    index index81.html;
}
server {
    listen 0.0.0.0:82;
    root /data/Nginx/;
    server_name localhost;
    index index82.html;
}
1
# nginx -s reload

创建虚拟主机资源文件：

1
2
3
4
5
6
7
8
9
10
# node3
# mkdir -p /data/Nginx
# echo "hello 10.10.20.3:80" > /data/Nginx/index80.html
# echo "hello 10.10.20.3:81" > /data/Nginx/index81.html
# echo "hello 10.10.20.3:82" > /data/Nginx/index82.html
# node4
# mkdir -p /data/Nginx
# echo "hello 10.10.20.4:80" > /data/Nginx/index80.html
# echo "hello 10.10.20.4:81" > /data/Nginx/index81.html
# echo "hello 10.10.20.4:82" > /data/Nginx/index82.html

node2

修改负载均衡配置：

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
# vim /etc/nginx/conf.d/lb.conf
upstream 10.10.20.2 {
    server 10.10.20.3:80;
    server 10.10.20.3:81;
    server 10.10.20.3:82;
    server 10.10.20.4:80;
    server 10.10.20.4:81;
    server 10.10.20.4:82;
}
server {
    listen 80;
    server_name localhost;
    location / {
        proxy_pass http://10.10.20.2;
    }
}
1
# nginx -s reload

node1

客户端访问测试：

1
2
3
4
5
6
7
8
9
10
11
# for i in {1..10}; do curl 10.10.20.2; done
hello 10.10.20.3:80
hello 10.10.20.3:81
hello 10.10.20.3:82
hello 10.10.20.4:80
hello 10.10.20.4:81
hello 10.10.20.4:82
hello 10.10.20.3:80
hello 10.10.20.3:81
hello 10.10.20.3:82
hello 10.10.20.4:80

node2

负载均衡配置ip hash算法：

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
# vim /etc/nginx/conf.d/lb.conf
upstream 10.10.20.2 {
    ip_hash;
    server 10.10.20.3:80;
    server 10.10.20.3:81;
    server 10.10.20.3:82;
    server 10.10.20.4:80;
    server 10.10.20.4:81;
    server 10.10.20.4:82;
}
server {
    listen 80;
    server_name localhost;
    location / {
        proxy_pass http://10.10.20.2;
    }
}
1
# nginx -s reload

node1

客户端访问测试：

1
2
3
4
5
6
7
8
9
10
11
# for i in {1..10}; do curl 10.10.20.2; done
hello 10.10.20.4:82
hello 10.10.20.4:82
hello 10.10.20.4:82
hello 10.10.20.4:82
hello 10.10.20.4:82
hello 10.10.20.4:82
hello 10.10.20.4:82
hello 10.10.20.4:82
hello 10.10.20.4:82
hello 10.10.20.4:82

node3、node4

创建generic hash算法资源文件：

1
2
3
4
# node3
# echo "10.10.20.3 generic hash" > /data/Nginx/test.txt
# node4
# echo "10.10.20.4 generic hash" > /data/Nginx/test.txt

node2

负载均衡恢复默认算法：

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
# vim /etc/nginx/conf.d/lb.conf
upstream 10.10.20.2 {
    server 10.10.20.3:80;
    server 10.10.20.3:81;
    server 10.10.20.3:82;
    server 10.10.20.4:80;
    server 10.10.20.4:81;
    server 10.10.20.4:82;
}
server {
    listen 80;
    server_name localhost;
    location / {
        proxy_pass http://10.10.20.2;
    }
}
1
# nginx -s reload

node1

客户端访问测试：

1
2
3
4
5
6
7
8
9
10
11
# for i in {1..10}; do curl 10.10.20.2/test.txt; done
10.10.20.3 generic hash
10.10.20.3 generic hash
10.10.20.3 generic hash
10.10.20.4 generic hash
10.10.20.4 generic hash
10.10.20.4 generic hash
10.10.20.3 generic hash
10.10.20.3 generic hash
10.10.20.3 generic hash
10.10.20.4 generic hash

node2

负载均衡配置generic hash算法：

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
# vim /etc/nginx/conf.d/lb.conf
upstream 10.10.20.2 {
    hash $request_uri;
    server 10.10.20.3:80;
    server 10.10.20.3:81;
    server 10.10.20.3:82;
    server 10.10.20.4:80;
    server 10.10.20.4:81;
    server 10.10.20.4:82;
}
server {
    listen 80;
    server_name localhost;
    location / {
        proxy_pass http://10.10.20.2;
    }
}
1
# nginx -s reload

node1

客户端访问测试：

1
2
3
4
5
6
7
8
9
10
11
# for i in {1..10}; do curl 10.10.20.2/test.txt; done
10.10.20.4 generic hash
10.10.20.4 generic hash
10.10.20.4 generic hash
10.10.20.4 generic hash
10.10.20.4 generic hash
10.10.20.4 generic hash
10.10.20.4 generic hash
10.10.20.4 generic hash
10.10.20.4 generic hash
10.10.20.4 generic hash

node2

负载均衡配置random算法：

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
# vim /etc/nginx/conf.d/lb.conf
upstream 10.10.20.2 {
    random two;
    server 10.10.20.3:80;
    server 10.10.20.3:81;
    server 10.10.20.3:82;
    server 10.10.20.4:80;
    server 10.10.20.4:81;
    server 10.10.20.4:82;
}
server {
    listen 80;
    server_name localhost;
    location / {
        proxy_pass http://10.10.20.2;
    }
}
1
# nginx -s reload

node1

客户端访问测试：

1
2
3
4
5
6
7
8
9
10
11
# for i in {1..10}; do curl 10.10.20.2; done
hello 10.10.20.4:81
hello 10.10.20.4:80
hello 10.10.20.3:82
hello 10.10.20.3:82
hello 10.10.20.4:81
hello 10.10.20.4:81
hello 10.10.20.3:81
hello 10.10.20.3:82
hello 10.10.20.4:81
hello 10.10.20.4:80

node2

负载均衡配置权重和备份：

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
# vim /etc/nginx/conf.d/lb.conf
upstream 10.10.20.2 {
    server 10.10.20.3:80 weight=2;
    server 10.10.20.3:81;
    server 10.10.20.3:82;
    server 10.10.20.4:80 backup;
    server 10.10.20.4:81 backup;
    server 10.10.20.4:82 backup;
}
server {
    listen 80;
    server_name localhost;
    location / {
        proxy_pass http://10.10.20.2;
    }
}
1
# nginx -s reload

node1

客户端访问测试：

1
2
3
4
5
6
7
8
9
10
11
# for i in {1..10}; do curl 10.10.20.2; done
hello 10.10.20.3:82
hello 10.10.20.3:80
hello 10.10.20.3:80
hello 10.10.20.3:81
hello 10.10.20.3:82
hello 10.10.20.3:80
hello 10.10.20.3:80
hello 10.10.20.3:81
hello 10.10.20.3:82
hello 10.10.20.3:80

node3

停止nginx：

1
# systemctl stop nginx

node1

客户端访问测试：

1
2
3
4
5
6
7
8
9
10
11
# for i in {1..10}; do curl 10.10.20.2; done
hello 10.10.20.4:80
hello 10.10.20.4:81
hello 10.10.20.4:82
hello 10.10.20.4:80
hello 10.10.20.4:81
hello 10.10.20.4:82
hello 10.10.20.4:80
hello 10.10.20.4:81
hello 10.10.20.4:82
hello 10.10.20.4:80

haproxy配置实践

软件版本
nginx1.21.5
haproxy2.2.16
节点IP1IP2系统功能CPU内存硬盘
node110.80.20.1openeuler20.03client2核心4GB20GB
node210.80.20.210.10.20.2openeuler20.03haproxy2核心4GB20GB
node310.80.20.310.10.20.3openeuler20.03nginx2核心4GB20GB
node410.80.20.410.10.20.4openeuler20.03nginx2核心4GB20GB

通过haproxy实现简单负载均衡调度功能

node3、node4

下载安装nginx：

1
2
3
# dnf install -y nginx
# systemctl enable nginx --now
# systemctl status nginx

配置虚拟主机：

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
# vim /etc/nginx/conf.d/vhost.conf
server {
    listen 0.0.0.0:80;
    root /data/Nginx/;
    server_name localhost;
    index index80.html;
}
server {
    listen 0.0.0.0:81;
    root /data/Nginx/;
    server_name localhost;
    index index81.html;
}
server {
    listen 0.0.0.0:82;
    root /data/Nginx/;
    server_name localhost;
    index index82.html;
}
1
# nginx -s reload

创建虚拟主机资源文件：

1
2
3
4
5
6
7
8
9
10
11
12
# node3
# mkdir -p /data/Nginx
# echo "hello 10.10.20.3:80" > /data/Nginx/index80.html
# echo "hello 10.10.20.3:81" > /data/Nginx/index81.html
# echo "hello 10.10.20.3:82" > /data/Nginx/index82.html
# echo "10.10.20.3 generic hash" > /data/Nginx/test.txt
# node4
# mkdir -p /data/Nginx
# echo "hello 10.10.20.4:80" > /data/Nginx/index80.html
# echo "hello 10.10.20.4:81" > /data/Nginx/index81.html
# echo "hello 10.10.20.4:82" > /data/Nginx/index82.html
# echo "10.10.20.4 generic hash" > /data/Nginx/test.txt

关闭ip1网卡：

1
# ifdown ens160

node2

下载安装haproxy：

1
# dnf install -y haproxy

修改haproxy配置文件：

1
2
3
4
5
6
7
8
9
10
11
# cp /etc/haproxy/haproxy.cfg{,.bak}
# vim /etc/haproxy/haproxy.cfg
# 尾行，修改backend http_back部分
backend http_back
    balance     roundrobin
    server  node1 10.10.20.3:80 check
    server  node2 10.10.20.3:81 check
    server  node3 10.10.20.3:82 check
    server  node4 10.10.20.4:80 check
    server  node5 10.10.20.4:81 check
    server  node6 10.10.20.4:82 check
1
2
# systemctl enable haproxy --now
# systemctl status haproxy

node1

客户端访问测试：

1
2
3
4
5
6
7
8
9
10
11
# for i in {1..10}; do curl 10.80.20.2; done
hello 10.10.20.3:80
hello 10.10.20.3:81
hello 10.10.20.3:82
hello 10.10.20.4:80
hello 10.10.20.4:81
hello 10.10.20.4:82
hello 10.10.20.3:80
hello 10.10.20.3:81
hello 10.10.20.3:82
hello 10.10.20.4:80

haproxy监控页面配置

node2

修改haproxy配置文件，开启监控页面：

1
2
3
4
5
6
7
8
9
10
# vim /etc/haproxy/haproxy.cfg
# 尾行，添加配置
listen admin_stat
    bind 0.0.0.0:8443
    mode http
    stats refresh 30s
    stats uri /haproxy_stats
    stats realm Haproxy\ Statistics
    stats auth openEuler:Huawei@123
    stats hide-version

  • refresh 30s：刷新频率。

  • uri /haproxy_stats：监控页面url。

  • openEuler:Huawei@123：登录页面认证信息。

1
# systemctl restart haproxy

浏览器访问：http://10.80.20.2:8443/haproxy_stats

haproxy日志相关配置

node2

修改haproxy配置文件，定义日志：

1
2
3
# vim /etc/haproxy/haproxy.cfg
# 10行，修改配置
    log         127.0.0.1 local3 info

修改rsyslog配置：

1
2
3
4
5
# vim /etc/rsyslog.conf
# 尾行，添加配置
local3.*                                                /var/log/haproxy.log
$ModLoad imudp
$UDPServerRun 514
1
2
# systemctl restart haproxy rsyslog
# tail -f /var/log/haproxy.log

node4

开启防火墙：

1
# systemctl status firewalld

关闭防火墙，等待恢复：

1
# systemctl stop firewalld

haproxy acl配置

node2

修改配置文件，将txt结尾的url发送到指定的主机组：

1
2
3
4
5
6
7
8
9
10
11
12
# vim /etc/haproxy/haproxy.cfg
# 33~35行，修改配置
frontend main
    bind *:80
    acl test url_reg -i \.txt$
    use_backend test if test
    default_backend         http_back
# 尾行，添加配置
backend test
    balance     roundrobin
    server test1 10.10.20.3:80/test.txt check
    server test2 10.10.20.4:80/test.txt check
1
# systemctl restart rsyslog

node1

客户端访问测试：

1
2
3
4
5
6
7
8
9
10
11
# for i in {1..10}; do curl 10.80.20.2/test.txt; done
10.10.20.4 generic hash
10.10.20.4 generic hash
10.10.20.3 generic hash
10.10.20.3 generic hash
10.10.20.3 generic hash
10.10.20.4 generic hash
10.10.20.4 generic hash
10.10.20.4 generic hash
10.10.20.3 generic hash
10.10.20.3 generic hash

keepalived配置实践

keepalived实现nginx的高可用集群

软件版本
nginx1.21.5
keepalived2.0.20
节点IP1IP2VIP系统功能CPU内存硬盘
node110.80.20.1openeuler20.03client2核心4GB20GB
node210.80.20.210.10.20.210.80.20.100openeuler20.03nginx、keepalived2核心4GB20GB
node310.80.20.310.10.20.310.80.20.100openeuler20.03nginx、keepalived2核心4GB20GB

node2、node3

下载安装nginx：

1
2
3
# dnf install -y nginx
# systemctl enable nginx --now
# systemctl status nginx

配置虚拟主机：

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
# vim /etc/nginx/conf.d/vhost.conf
server {
    listen 0.0.0.0:80;
    root /data/Nginx/;
    server_name localhost;
    index index80.html;
}
server {
    listen 0.0.0.0:81;
    root /data/Nginx/;
    server_name localhost;
    index index81.html;
}
server {
    listen 0.0.0.0:82;
    root /data/Nginx/;
    server_name localhost;
    index index82.html;
}
1
# nginx -s reload

创建虚拟主机资源文件：

1
2
3
4
5
6
7
8
9
10
# node2
# mkdir -p /data/Nginx
# echo "hello 10.10.20.2:80" > /data/Nginx/index80.html
# echo "hello 10.10.20.2:81" > /data/Nginx/index81.html
# echo "hello 10.10.20.2:82" > /data/Nginx/index82.html
# node3
# mkdir -p /data/Nginx
# echo "hello 10.10.20.3:80" > /data/Nginx/index80.html
# echo "hello 10.10.20.3:81" > /data/Nginx/index81.html
# echo "hello 10.10.20.3:82" > /data/Nginx/index82.html

下载安装keepalived：

1
# dnf install -y keepalived

node2

配置keepalived主节点：

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
# cp /etc/keepalived/keepalived.conf{,.bak}
# vim /etc/keepalived/keepalived.conf
! Configuration File for keepalived

global_defs {
   router_id Nginx1
}

vrrp_instance Nginx {
    state MASTER
    interface ens160
    virtual_router_id 51
    priority 225
    advert_int 1
    authentication {
        auth_type PASS
        auth_pass Huawei@1
    }
    virtual_ipaddress {
        10.80.20.100/24
    }
}
1
2
3
4
# systemctl enable keepalived --now
# systemctl status keepalived
# ip a | grep 10.80.20.100
    inet 10.80.20.100/24 scope global ens160

node3

配置keepalived备节点：

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
# cp /etc/keepalived/keepalived.conf{,.bak}
# vim /etc/keepalived/keepalived.conf
! Configuration File for keepalived

global_defs {
   router_id Nginx2
}

vrrp_instance Nginx {
    state BACKUP
    interface ens160
    virtual_router_id 51
    priority 100
    advert_int 1
    authentication {
        auth_type PASS
        auth_pass Huawei@1
    }
    virtual_ipaddress {
        10.80.20.100/24
    }
}
1
2
# systemctl enable keepalived --now
# systemctl status keepalived

node1

客户端访问测试：

1
2
3
4
# for i in {80..82}; do curl 10.80.20.100:${i}; done
hello 10.10.20.2:80
hello 10.10.20.2:81
hello 10.10.20.2:82

node2

关闭node2节点：

1
# poweroff

node1

客户端访问测试：

1
2
3
4
# for i in {80..82}; do curl 10.80.20.100:${i}; done
hello 10.10.20.3:80
hello 10.10.20.3:81
hello 10.10.20.3:82

node2、node3

keepalived主配置添加nginx检测：

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
# vim /etc/keepalived/keepalived.conf
! Configuration File for keepalived

global_defs {
   router_id Nginx1
}

vrrp_script nginx_check {
    scrpt "/etc/keepalived/check.sh"
    interval 1
    weight -5
    fail 3
}

vrrp_instance Nginx1 {
    state BACKUP
    interface ens160
    virtual_router_id 51
    priority 100
    advert_int 1
    authentication {
        auth_type PASS
        auth_pass Huawei@1
    }
    virtual_ipaddress {
        10.80.20.100/24
    }
    track_script {
        nginx_check
    }
}

keepalived备配置添加nginx检测：

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
# vim /etc/keepalived/keepalived.conf
! Configuration File for keepalived

global_defs {
   router_id Nginx1
}

vrrp_script nginx_check {
    scrpt "/etc/keepalived/check.sh"
    interval 1
    weight -5
    fail 3
}

vrrp_instance Nginx1 {
    state MASTER
    interface ens160
    virtual_router_id 51
    priority 225
    advert_int 1
    authentication {
        auth_type PASS
        auth_pass Huawei@1
    }
    virtual_ipaddress {
        10.80.20.100/24
    }
    track_script {
        nginx_check
    }
}

添加nginx检测脚本：

1
2
3
4
5
6
7
8
9
10
11
12
13
# vim /etc/keepalived/check.sh
#!/bin/bash
systemctl status nginx | grep "active (running)" > /dev/null
if [ $? -ne 0 ]; then
  systemctl restart nginx &> /dev/null
  sleep 1
  systemctl status nginx | grep "active (running)" > /dev/null
  if [ $? -ne 0 ]; then
    systemctl stop keepalived
  else
    exit
  fi
fi      
1
2
# chmod +x /etc/keepalived/check.sh
# systemctl restart keepalived

node1

客户端访问测试，nginx停掉会自动重启：

1
2
3
4
# for i in {80..82}; do curl 10.80.20.100:${i}; done
hello 10.10.20.2:80
hello 10.10.20.2:81
hello 10.10.20.2:82

keepalived+lvs实现nginx集群

软件版本
nginx1.21.5
keepalived2.0.20
ipvsadm1.31
节点IP1IP2VIP系统功能CPU内存硬盘
node110.80.20.1openeuler20.03client2核心4GB20GB
node210.80.20.210.10.20.210.80.20.100openeuler20.03lvs、keepalived2核心4GB20GB
node310.80.20.310.10.20.310.80.20.100openeuler20.03lvs、keepalived2核心4GB20GB
node410.80.20.410.80.20.100openeuler20.03nginx2核心4GB20GB
node510.80.20.510.80.20.100openeuler20.03nginx2核心4GB20GB

node4、node5

1
2
3
4
5
6
7
8
# nmcli con add type dummy ifname dummy2 ipv4.method manual ipv4.address 10.80.20.100/32
# cat >> /etc/sysctl.conf << EOF
net.ipv4.conf.all.arp_ignore=1
net.ipv4.conf.all.arp_announce=2
net.ipv4.conf.dummy2.arp_ignore=1
net.ipv4.conf.dummy2.arp_announce=2
EOF
# sysctl -p

下载安装nginx：

1
2
3
# dnf install -y nginx
# systemctl enable nginx --now
# systemctl status nginx

配置页面：

1
2
3
4
5
6
7
8
# node4
# echo "nginx web 10.80.20.4" > /usr/share/nginx/html/index.html
# curl 10.80.20.4
nginx web 10.80.20.4
# node5
# echo "nginx web 10.80.20.5" > /usr/share/nginx/html/index.html
# curl 10.80.20.5
nginx web 10.80.20.5

node2、node3

下载安装keepalived和ipvsadm：

1
# dnf install -y keepalived ipvsadm

修改主keepalived的lvs：

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
# cp /etc/keepalived/keepalived.conf{,.bak}
# vim /etc/keepalived/keepalived.conf
! Configuration File for keepalived

global_defs {
   router_id Cluster1
}

vrrp_instance Nginx {
    state MASTER
    interface ens160
    mcast_src_ip ens224
    virtual_router_id 51
    priority 225
    advert_int 1
    authentication {
        auth_type PASS
        auth_pass 1111
    }
    virtual_ipaddress {
        10.80.20.100/24
    }
}

virtual_server 10.80.20.100 80 {
    delay_loop 6
    lb_algo rr
    lb_kind DR
    persistence_timeout 50
    protocol TCP

    real_server 10.80.20.4 80 {
        weight 1
        TCP_CHECK {
            connect_timeout 3
            retry 3
            delay_before_retry 3
        }
    }

    real_server 10.80.20.5 80 {
        weight 2
        TCP_CHECK {
            connect_timeout 3
            retry 3
            delay_before_retry 3
        }
    }
}
1
2
# systemctl enable keepalived --now
# systemctl status keepalive

修改备keepalived的lvs：

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
# cp /etc/keepalived/keepalived.conf{,.bak}
# vim /etc/keepalived/keepalived.conf
! Configuration File for keepalived

global_defs {
   router_id Cluster2
}

vrrp_instance Nginx {
    state BACKUP
    interface ens160
    mcast_src_ip ens224
    virtual_router_id 51
    priority 100
    advert_int 1
    authentication {
        auth_type PASS
        auth_pass 1111
    }
    virtual_ipaddress {
        10.80.20.100/24
    }
}

virtual_server 10.80.20.100 80 {
    delay_loop 6
    lb_algo rr
    lb_kind DR
    persistence_timeout 50
    protocol TCP

    real_server 10.80.20.4 80 {
        weight 1
        TCP_CHECK {
            connect_timeout 3
            retry 3
            delay_before_retry 3
        }
    }

    real_server 10.80.20.5 80 {
        weight 2
        TCP_CHECK {
            connect_timeout 3
            retry 3
            delay_before_retry 3
        }
    }
}
1
2
# systemctl enable keepalived --now
# systemctl status keepalived

node1

客户端访问测试：

1
2
# curl 10.80.20.100
nginx web 10.80.20.4

redis基础操作实践

redis操作

软件版本
redis6.2.7
节点IP1系统功能CPU内存硬盘
node110.80.20.1openeuler20.03redis2核心4GB20GB

node1

下载安装redis：

1
2
3
# dnf install -y redis6
# systemctl enable redis --now
# systemctl status redis

登录redis：

1
2
3
# redis-cli

127.0.0.1:6379>

创建数据：

1
2
3
4
5
6
7
8
127.0.0.1:6379> set test1 openeuler1
OK

127.0.0.1:6379> set test2 openeuler2
OK

127.0.0.1:6379> set test3 openeuler3
OK

查看全部key：

1
2
3
4
127.0.0.1:6379> keys *
1) "test3"
2) "test1"
3) "test2"

查看具体key对应的value：

1
2
127.0.0.1:6379> get test1
"openeuler1"

设置key的过期时间为2秒：

1
2
3
4
5
127.0.0.1:6379> expire test1 2
(integer) 1

127.0.0.1:6379> get test1
(nil)

移动key到其它库：

1
2
3
4
5
6
7
8
9
10
11
12
13
127.0.0.1:6379> move test2 1
(integer) 1

127.0.0.1:6379> get test2
(nil)

127.0.0.1:6379> select 1
OK

127.0.0.1:6379[1]> get test2
"openeuler2"

127.0.0.1:6379[1]> exit

设置密码：

1
2
3
# vim /etc/redis/redis.conf
# 903行，修改配置
requirepass Huawei@123
1
# systemctl restart redis

验证：

1
2
3
4
5
6
7
8
9
10
11
12
# redis-cli

127.0.0.1:6379> keys *
(error) NOAUTH Authentication required.

127.0.0.1:6379> auth Huawei@123
OK

127.0.0.1:6379> keys *
1) "test3"

127.0.0.1:6379> exit

修改持久化存储配置：

1
2
3
4
5
6
7
# vim /etc/redis/redis.conf
# 433行，修改配置
dbfilename snapshot.rdb
# 456行，修改配置
dir /var/lib/redis
# 386行，添加配置
save 10 1
1
2
3
# systemctl restart redis
# ls /var/lib/redis/
dump.rdb

创建数据：

1
2
3
4
5
6
7
8
9
# redis-cli -a Huawei@123

127.0.0.1:6379> set a1 b1
OK

127.0.0.1:6379> set a2 b2
OK

127.0.0.1:6379> exit
1
2
# ls /var/lib/redis/
dump.rdb  snapshot.rdb

redis为wordpress提供缓存

软件版本
redis6.2.7
节点IP1系统功能CPU内存硬盘
node110.80.20.1openeuler20.03redis2核心4GB20GB
node210.80.20.2openeuler20.03wordpress2核心4GB20GB
node310.80.20.3openeuler20.03mysql2核心4GB20GB

node1

下载安装redis：

1
2
3
# dnf install -y redis6
# systemctl enable redis --now
# systemctl status redis

修改配置文件：

1
2
3
4
5
# vim /etc/redis/redis.conf
# 75行，修改配置
bind 10.80.20.1
# 903行，修改配置
requirepass Huawei@123
1
2
3
# systemctl restart redis
# ss -tlunp | grep redis
tcp     LISTEN   0        128           10.80.20.1:6379          0.0.0.0:*       users:(("redis-server",pid=19515,fd=6))

node2

下载安装php：

1
# dnf install -y php

修改配置文件：

1
2
3
4
# vim /etc/php-fpm.d/www.conf
# 尾行，添加配置
listen.allowed_clients = 127.0.0.1
listen = 9000
1
2
# systemctl enable php-fpm --now
# systemctl status php-fpm

下载安装nginx：

1
# dnf install -y nginx

修改配置文件：

1
2
3
4
5
6
7
8
9
10
11
12
13
# vim /etc/nginx/conf.d/wordpress.conf
server {
    listen 0.0.0.0:80;
    root /data/WordPress;
    server_name localhost;
    index index.php;
    location ~ \.php$ {
        fastcgi_pass 127.0.0.1:9000;
        fastcgi_index index.php;
        fastcgi_param SCRIPT_FILENAME $document_root$fastcgi_script_name;
        include fastcgi_params;
    }
}
1
2
# systemctl enable nginx --now
# systemctl status nginx

创建测试文件：

1
2
3
4
5
# mkdir -p /data/WordPress
# vim /data/WordPress/index.php
<?php
phpinfo();
?>

浏览器访问：http://10.80.20.2/

node3

下载安装mysql：

1
2
3
# dnf install -y mysql5-server
# systemctl enable mysqld --now
# systemctl status mysqld

开放权限：

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
# mysql

mysql> CREATE user 'root'@'%' IDENTIFIED WITH mysql_native_password BY 'Huawei@123';
Query OK, 0 rows affected (0.00 sec)

mysql> GRANT ALL PRIVILEGES ON *.* TO 'root'@'%' WITH GRANT OPTION;
Query OK, 0 rows affected (0.00 sec)

mysql> SELECT user,host FROM mysql.user where user='root';
+------+-----------+
| user | host      |
+------+-----------+
| root | %         |
| root | localhost |
+------+-----------+
2 rows in set (0.00 sec)

mysql> exit

node2

下载php连接mysql插件：

1
# dnf install -y php-mysqlnd

创建测试文件：

1
2
3
4
5
6
7
8
9
# vim /data/WordPress/conn_mysql.php
<?php
$con = mysqli_connect("10.80.20.3","root","Huawei@123");
if ($con)
    echo "OK\n";
else
    echo "NOT OK\n";
$con->close();
?>

访问测试：

1
2
# curl http://10.80.20.2/conn_mysql.php
OK

下载安装wordpress：

下载地址：https://cn.wordpress.org/download/releases/

1
2
3
4
# cd /home/
# mkdir wordpress && cd wordpress
# wget https://cn.wordpress.org/wordpress-6.1.1-zh_CN.tar.gz
# tar -xzvf wordpress-6.1.1-zh_CN.tar.gz

node3

创建wordpress数据库：

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
# mysql

mysql> CREATE DATABASE wordpress;
Query OK, 1 row affected (0.00 sec)

mysql> CREATE USER wp@'%' IDENTIFIED BY 'Huawei@123';
Query OK, 0 rows affected (0.00 sec)

mysql> GRANT ALL PRIVILEGES ON wordpress.* TO 'wp'@'%';
Query OK, 0 rows affected (0.00 sec)

mysql> FLUSH PRIVILEGES;
Query OK, 0 rows affected (0.00 sec)

mysql> exit

node2

拷贝文件：

1
2
# cd /home/wordpress/wordpress
# cp -ar * /data/WordPress/

浏览器访问：http://10.80.20.2/wp-admin/setup-config.php

开始安装：

1
2
3
4
数据库名：wordpress
用户名：wp
密码：Huawei@123
数据库主机：10.80.20.3

将配置写入文件：

1
# vim /data/WordPress/wp-config.php

建站信息：

1
2
3
4
站点标题：openeuler
用户名：openeuler
密码：Huawei@123
您的电子邮箱地址：test@test.com

安装完成：

登录：

进入主页：

wordpress配置添加redis：

1
2
3
4
5
# vim /data/WordPress/wp-config.php
# 50行，添加配置
define("FS_METHOD","direct");
define("FS_CHMOD_DIR","0777");
define("FS_CHMOD_FILE","0777");
1
# chmod -R 777 /data/WordPress/

添加Redis Object Cache插件：

插件—>安装插件

安装wp-cli：

1
2
3
4
5
6
7
# curl -O https://raw.githubusercontent.com/wp-cli/builds/gh-pages/phar/wp-cli.phar
# chmod +x wp-cli.phar
# mv wp-cli.phar /usr/local/bin/wp
# cd /data/WordPress/
# wget https://downloads.wordpress.org/plugin/redis-cache.2.5.0.zip
# unzip redis-cache.2.5.0.zip
# wp plugin activate redis-cache

刷新页面：

添加wordpress配置：

1
2
3
4
5
6
7
8
# vim /data/WordPress/wp-config.php
# 50行，添加配置
define('WP_REDIS_HOST','10.80.20.1');
define('WP_REDIS_PORT',6379);
define('WP_REDIS_PASSWORD','Huawei@123');
define('WP_REDIS_TIMEOUT',1);
define('WP_REDIS_READ_TIMEOUT',1);
define('WP_REDIS_DATABASE',0);

查看插件状态：

插件—>Redis Object Cache（Settings）

node1

查看keys：

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
10.80.20.1:6379> keys *
 1) "wp:userlogins:openeuler"
 2) "wp:comment:last_changed"
 3) "wp:options:notoptions"
 4) "wp:redis-cache:metrics"
 5) "wp:site-transient:update_plugins"
 6) "wp:comment:get_comments-fbb001ab5aa3f7e15b8511e52af82c41-0.78696500 1701326926"
 7) "wp:post_meta:3"
 8) "wp:options:user_count"
 9) "wp:comment:get_comments-e96df111ec1dc11a41982d65e4d21821-0.78696500 1701326926"
10) "wp:default:is_blog_installed"
11) "wp:posts:last_changed"
12) "wp:comment:get_comments-1b516a23ca8801fb482159bb8e30ecd2-0.78696500 1701326926"
13) "wp:posts:wp_query-21b313226e504da56f1725101390cade-0.76763700 17013269260.76712000 1701326926"
14) "wp:site-transient:theme_roots"
15) "wp:users:1"
16) "wp:options:alloptions"
17) "wp:comment:get_comments-46612f2dd3b604b784c3eb3dac699d60-0.78696500 1701326926"
18) "wp:user_meta:1"
19) "wp:options:can_compress_scripts"
20) "wp:terms:last_changed"
21) "wp:posts:3"
22) "wp:site-transient:update_core"
23) "wp:userslugs:openeuler"
24) "wp:site-options:1-notoptions"
25) "wp:terms:get_terms-3c007108e9cd085ab2b09f1e06edd5fc-0.76712000 1701326926"
26) "wp:options:uninstall_plugins"
27) "wp:useremail:test@test.com"
28) "wp:site-transient:update_themes"
29) "wp:comment:get_comments-bbafb40d4aed04b7b4890f46f3eeac25-0.78696500 1701326926"
30) "wp:posts:wp_query-ead82d7f3b812b48d821b60d252c028d-0.76763700 17013269260.76712000 1701326926"