shell脚本

shell脚本

shell脚本编程入门

节点IP系统功能CPU内存硬盘
node110.80.10.1centos7.9shell4核心8GB20GB

什么是shell:

  • shell除了命令,它同时也是一门脚本编程语言。

  • shell无需编译即可运行,写完即可运行。

  • 学习一门编程语言,语法和程序设计的重要性。

shell脚本语言的种类:

  • shell命令比较通用,命令很多用c语言编写的,例如ls。

  • shell脚本有bash shell、c shell等多种shell,每种shell脚本的语法不一样,其中bash shell是使用最广泛的。

node1

为什么说bash shell是应用最广泛的?

我们使用root登录系统,默认使用shell是bash shell:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
# cat /etc/passwd
root:x:0:0:root:/root:/bin/bash
bin:x:1:1:bin:/bin:/sbin/nologin
daemon:x:2:2:daemon:/sbin:/sbin/nologin
adm:x:3:4:adm:/var/adm:/sbin/nologin
lp:x:4:7:lp:/var/spool/lpd:/sbin/nologin
sync:x:5:0:sync:/sbin:/bin/sync
shutdown:x:6:0:shutdown:/sbin:/sbin/shutdown
halt:x:7:0:halt:/sbin:/sbin/halt
mail:x:8:12:mail:/var/spool/mail:/sbin/nologin
operator:x:11:0:operator:/root:/sbin/nologin
games:x:12:100:games:/usr/games:/sbin/nologin
ftp:x:14:50:FTP User:/var/ftp:/sbin/nologin
nobody:x:99:99:Nobody:/:/sbin/nologin
systemd-network:x:192:192:systemd Network Management:/:/sbin/nologin
dbus:x:81:81:System message bus:/:/sbin/nologin
polkitd:x:999:998:User for polkitd:/:/sbin/nologin
sshd:x:74:74:Privilege-separated SSH:/var/empty/sshd:/sbin/nologin
postfix:x:89:89::/var/spool/postfix:/sbin/nologin
chrony:x:998:996::/var/lib/chrony:/sbin/nologin

默认的sh是指向bash shell:

1
2
# ll -h /bin/sh
lrwxrwxrwx. 1 root root 4 Sep 30 04:15 /bin/sh -> bash

交互式和非交互式shell:

  • 日常敲命令是属于交互式的。

  • 课程上主要学习的shell脚本是一种非交互式shell,然后使用sh运行它。

第一个shell程序:

1
2
3
4
# yum install -y vim
# mkdir -p /data/shell
# vim /data/shell/hello.sh
echo "Hello World!"

怎么样运行shell脚本?三种方法都可以:

1
2
# bash /data/shell/hello.sh
Hello World!
1
2
# sh /data/shell/hello.sh
Hello World!
1
2
3
# chmod a+x /data/shell/hello.sh
# /data/shell/hello.sh
Hello World!

具有执行权限的#!说明:

  • linux中具有执行权限的脚本程序,我们都需要使用#!指定是什么程序。

  • 如果是bash shell程序,在脚本的第一行加入:#!/bin/sh。

  • 如果是python程序,在脚本的第一行加入:#!/usr/bin/python。

shell脚本注释和顺序执行

node1

bash shell脚本中的注释:

1
2
3
4
5
# vim /data/shell/hello.sh
#!/bin/sh
#hello world
echo "Hello World!"
#hello world
1
2
# bash /data/shell/hello.sh
Hello World!

shell脚本的顺序执行,无语法:

1
2
3
4
# vim /data/shell/order.sh
echo "student"
ls -lh /
df -h
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
# bash /data/shell/order.sh
student
total 16K
lrwxrwxrwx. 1 root root 7 Sep 30 04:15 bin -> usr/bin
dr-xr-xr-x. 5 root root 4.0K Oct 6 18:50 boot
drwxr-xr-x 3 root root 19 Dec 9 18:47 data
drwxr-xr-x 20 root root 3.2K Dec 9 18:30 dev
drwxr-xr-x. 74 root root 8.0K Dec 9 18:44 etc
drwxr-xr-x. 2 root root 6 Apr 11 2018 home
lrwxrwxrwx. 1 root root 7 Sep 30 04:15 lib -> usr/lib
lrwxrwxrwx. 1 root root 9 Sep 30 04:15 lib64 -> usr/lib64
drwxr-xr-x. 2 root root 6 Apr 11 2018 media
drwxr-xr-x. 2 root root 6 Apr 11 2018 mnt
drwxr-xr-x. 2 root root 6 Apr 11 2018 opt
dr-xr-xr-x 139 root root 0 Dec 9 18:30 proc
dr-xr-x---. 3 root root 163 Dec 9 18:50 root
drwxr-xr-x 23 root root 640 Dec 9 18:47 run
lrwxrwxrwx. 1 root root 8 Sep 30 04:15 sbin -> usr/sbin
drwxr-xr-x. 2 root root 6 Apr 11 2018 srv
dr-xr-xr-x 13 root root 0 Dec 9 18:30 sys
drwxrwxrwt. 8 root root 172 Dec 9 18:31 tmp
drwxr-xr-x. 13 root root 155 Sep 30 04:15 usr
drwxr-xr-x. 19 root root 267 Sep 30 04:45 var
Filesystem Size Used Avail Use% Mounted on
devtmpfs 3.8G 0 3.8G 0% /dev
tmpfs 3.9G 0 3.9G 0% /dev/shm
tmpfs 3.9G 12M 3.8G 1% /run
tmpfs 3.9G 0 3.9G 0% /sys/fs/cgroup
/dev/mapper/centos-root 17G 1.9G 16G 12% /
/dev/sda1 1014M 151M 864M 15% /boot
tmpfs 781M 0 781M 0% /run/user/0

顺序执行编写apache的yum安装脚本:

1
2
3
4
5
# vim /data/shell/install_httpd.sh
yum install -y httpd
systemctl start httpd
systemctl enable httpd
echo 'it works! shell study' > /var/www/html/index.html
1
2
3
# bash /data/shell/install_httpd.sh
# curl 127.0.0.1
it works! shell study

顺序执行编写nginx编译安装脚本:

1
2
3
4
5
6
7
8
9
10
# systemctl stop httpd
# vim /data/shell/install_nginx.sh
yum install -y wget gcc pcre pcre-devel zlib zlib-devel
cd /usr/local/src/
wget http://nginx.org/download/nginx-1.12.2.tar.gz
tar -xzvf nginx-1.12.2.tar.gz
cd nginx-1.12.2
./configure --prefix=/usr/local/nginx
make
make install
1
# bash /data/shell/install_nginx.sh

语法再强调:

  • 顺序执行无语法,是最简单的,同时也是最常用的。

  • bash shell的语法,可以控制程序执行顺序、循环执行、模块执行等。

  • 语法每门语言都不一致。

程序设计强调:

  • 脚本语法也是一门编程语言,任何编程语言都需程序设计。

  • 程序设计思路都适用。

编写输出1-100:

  • 一个数1。

  • 判断这个数是不是小于等于100。

  • 小于等于100输出。

  • 大于100就退出。

  • 这个数加1。

  • 判断这个数是不是小于等于100。

  • 小于等于100输出。

  • 大于100就退出。

shell脚本中的变量

编程界中的变量:

  • 变量,字面上含义就是可变化的量。

  • 与变量相对应的是常量,常量例如”Hello World”,不可改变的。

  • 变量可以给个变量名,假如为name,值是可变的。

变量具有变量名:

  • shell脚本每个变量都有一个变量名,变量名有它的命名规则。

  • 命名规则:必须由大写字母、小写字母、下划线、数字,并且首字母不能是数字。

1
2
3
4
1name:不符合要求。
nam,e:不符合要求。
my_name:符合要求。
_name:符合要求。

值的类型:

  • 如果有学过java等,值的类型会分为整型、浮点型、字符串型、布尔型等,而且使用变量需要指定类型。

  • shell默认的变量类型都是字符串,无需指定类型。

变量种类:

  • 内部变量,也就是系统自带的。

  • 自定义变量,脚本中可以自定义变量 。

  • 自定义变量语法:左边是变量名、右边是值。

node1

报错:

1
2
3
# vim /data/shell/myvar.sh
1name="student"
echo $my_name
1
2
# bash /data/shell/myvar.sh
/data/shell/myvar.sh: line 1: 1name=student: command not found

正常,不报错:

1
2
3
# vim /data/shell/myvar.sh
my_name="student"
echo $my_name
1
2
# bash /data/shell/myvar.sh
student

shell自定义变量注意:

  • =左右不能出现空格,学习过java等语法,会这样定义变量name = “student”。

  • shell中name会被理解成是一个命令。

变量默认是字符串,不会对数字进行运算:

1
2
3
# vim /data/shell/myvar.sh
age=2+2
echo $age
1
# bash /data/shell/myvar.sh

值是可变的:

1
2
3
4
5
# vim /data/shell/myvar.sh
my_name="student"
echo $my_name
my_name="teacher"
echo $my_name
1
2
3
# bash /data/shell/myvar.sh
student
teacher

内部变量,系统自带的,可以直接用,无需定义。env命令可以查看内部变量,变量名都是大写字母:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
# env
XDG_SESSION_ID=1
HOSTNAME=node1
TERM=xterm
SHELL=/bin/bash
HISTSIZE=1000
SSH_CLIENT=10.80.0.1 3420 22
SSH_TTY=/dev/pts/0
USER=root
LS_COLORS=rs=0:di=01;34:ln=01;36:mh=00:pi=40;33:so=01;35:do=01;35:bd=40;33;01:cd=40;33;01:or=40;31;01:mi=01;05;37;41:su=37;41:sg=30;43:ca=30;41:tw=30;42:ow=34;42:st=37;44:ex=01;32:*.tar=01;31:*.tgz=01;31:*.arc=01;31:*.arj=01;31:*.taz=01;31:*.lha=01;31:*.lz4=01;31:*.lzh=01;31:*.lzma=01;31:*.tlz=01;31:*.txz=01;31:*.tzo=01;31:*.t7z=01;31:*.zip=01;31:*.z=01;31:*.Z=01;31:*.dz=01;31:*.gz=01;31:*.lrz=01;31:*.lz=01;31:*.lzo=01;31:*.xz=01;31:*.bz2=01;31:*.bz=01;31:*.tbz=01;31:*.tbz2=01;31:*.tz=01;31:*.deb=01;31:*.rpm=01;31:*.jar=01;31:*.war=01;31:*.ear=01;31:*.sar=01;31:*.rar=01;31:*.alz=01;31:*.ace=01;31:*.zoo=01;31:*.cpio=01;31:*.7z=01;31:*.rz=01;31:*.cab=01;31:*.jpg=01;35:*.jpeg=01;35:*.gif=01;35:*.bmp=01;35:*.pbm=01;35:*.pgm=01;35:*.ppm=01;35:*.tga=01;35:*.xbm=01;35:*.xpm=01;35:*.tif=01;35:*.tiff=01;35:*.png=01;35:*.svg=01;35:*.svgz=01;35:*.mng=01;35:*.pcx=01;35:*.mov=01;35:*.mpg=01;35:*.mpeg=01;35:*.m2v=01;35:*.mkv=01;35:*.webm=01;35:*.ogm=01;35:*.mp4=01;35:*.m4v=01;35:*.mp4v=01;35:*.vob=01;35:*.qt=01;35:*.nuv=01;35:*.wmv=01;35:*.asf=01;35:*.rm=01;35:*.rmvb=01;35:*.flc=01;35:*.avi=01;35:*.fli=01;35:*.flv=01;35:*.gl=01;35:*.dl=01;35:*.xcf=01;35:*.xwd=01;35:*.yuv=01;35:*.cgm=01;35:*.emf=01;35:*.axv=01;35:*.anx=01;35:*.ogv=01;35:*.ogx=01;35:*.aac=01;36:*.au=01;36:*.flac=01;36:*.mid=01;36:*.midi=01;36:*.mka=01;36:*.mp3=01;36:*.mpc=01;36:*.ogg=01;36:*.ra=01;36:*.wav=01;36:*.axa=01;36:*.oga=01;36:*.spx=01;36:*.xspf=01;36:
MAIL=/var/spool/mail/root
PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/root/bin
PWD=/root
LANG=en_US.UTF8
HISTCONTROL=ignoredups
SHLVL=1
HOME=/root
LOGNAME=root
SSH_CONNECTION=10.80.0.1 3420 10.80.10.1 22
LESSOPEN=||/usr/bin/lesspipe.sh %s
XDG_RUNTIME_DIR=/run/user/0
_=/usr/bin/env
1
2
3
4
5
6
# echo $HOSTNAME
node1
# echo $shell

# echo $PATH
/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/root/bin

内部变量是系统自带的,改变内部变量需要慎重。PATH=’’:所有命令都无法执行。

shell的字符串拼接

node1

shell定义字符串,建议值使用双引号:

1
2
3
4
5
6
7
# vim /data/shell/myvar.sh
name=teacher
echo $name
name='teacher'
echo $name
name="teacher"
echo $name
1
2
3
4
# bash /data/shell/myvar.sh
teacher
teacher
teacher

字符串拼接:

1
2
3
4
5
6
7
# vim /data/shell/myvar.sh
name="student"
echo "my name is $name" #拼接成功
echo 'my name is $name' #单引号里面无法使用变量
echo "my name is $name666" #拼接失败
echo "my name is ${name}666" #大括号拼接成功
echo "my name is 666$name" #拼接成功
1
2
3
4
5
6
# bash /data/shell/myvar.sh
my name is student
my name is $name
my name is
my name is student666
my name is 666student

加引号和不加引号的区别,推荐加引号:

1
2
# echo 1    2    3
1 2 3

相当于echo跟三个参数:

1
2
# echo "1   2   3"
1 2 3

加引号脚本:

1
2
3
4
# vim /data/shell/myvar.sh
number="1 2 3"
echo $number
echo "$number"
1
2
3
# bash /data/shell/myvar.sh
1 2 3
1 2 3

拼接换行符、制表符:

  • \n:代表换行符。

  • \t:代表制表符tab。

1
2
3
4
# vim /data/shell/myvar.sh
echo "student\t23456\tteacher"
echo -e "student\t12345\tteacher"
echo -e "student\t12345\tteacher\nclassmate\tclassmate\tclassmate"
1
2
3
4
5
# bash /data/shell/myvar.sh
student\t23456\tteacher
student 12345 teacher
student 12345 teacher
classmate classmate classmate

编译安装nginx使用变量,脚本还待完善:

1
2
3
4
5
6
7
8
9
10
# vim /data/shell/install_nginx.sh
nginxdownloadurl="http://nginx.org/download/nginx-1.12.2.tar.gz"
yum install -y wget gcc pcre pcre-devel zlib zlib-devel
cd /usr/local/src/
wget "$nginxdownloadurl"
tar -zxvf nginx-1.12.2.tar.gz
cd nginx-1.12.2
./configure --prefix=/usr/local/nginx
make
make install
1
2
3
# rm -rf /usr/local/nginx/
# rm -rf /usr/local/src/nginx-*
# bash /data/shell/install_nginx.sh

变量方便修改:

1
2
3
4
# vim /data/shell/myvar.sh
name="student"
echo "my name is $name"
echo "$name is 12345, location: north"
1
2
3
# bash /data/shell/myvar.sh
my name is student
student is 12345, location: north

shell条件判断之数字判断

条件判断,之前顺序执行无语法:

  • 接下来学习的条件判断、循环判断、函数等都要学习语法。

  • 条件判断可以控制命令的运行,不再是单纯的按顺序执行。

  • 学习条件判断之前需要先了解shell返回值。

node1

shell的返回值,运行一条命令,都会有一个返回值。0代表执行正常,非0代表命令执行异常,返回值放到$?变量中:

1
2
3
4
5
6
7
8
# ls /
bin boot data dev etc home lib lib64 media mnt opt proc root run sbin srv sys tmp usr var
# echo $?
0
# ls /nononono
ls: cannot access /nononono: No such file or directory
# echo $?
2

自编写shell脚本如何设置返回值:

  • 显示使用exit。

  • 没有使用exit的话,以最后一条命令的返回值作为脚本的返回值。

1
2
3
# vim /data/shell/myreturn.sh
echo "text"
exit 2
1
2
3
4
# bash /data/shell/myreturn.sh
text
# echo $?
2

bash shell条件判断的语法:

1
2
3
4
# 格式一
if 条件; then
代码
fi
1
2
3
4
5
6
# 格式二
if 条件; then
代码
else
代码
fi

条件可以有数字判断、字符串判断、文件判断等。

条件判断之数字判断语法:

  • $number1 -eq $nubmer2:是否相等,equal。

  • $number1 -ne $nubmer2:是否不相等,not equal。

  • $number1 -gt $nubmer2:是否大于,greater than。

  • $number1 -ge $nubmer2:是否大于等于,greater or equal。

  • $number1 -lt $nubmer2:是否小于,less than。

  • $number1 -le $nubmer2:是否小于等于,less or equal 。

test命令或者[ ]命令可测试判断,返回0代表判断成立,返回非0代表判断不成立:

1
2
3
4
5
6
# test 2 -eq 2
# echo $?
0
# test 3 -eq 2
# echo $?
1

编程中习惯使用[ ]中括号:

1
2
3
# [ 2 -eq 2 ]
# echo $?
0

最简单的条件判断案例:

1
2
3
4
5
6
7
# vim /data/shell/myif.sh
number1=3
number2=3
if [ $number1 -eq $number2 ]; then
echo "$number1 equal $number2"
echo "in if"
fi
1
2
3
# bash /data/shell/myif.sh
3 equal 3
in if

数字判断案例,注意缩进习惯:

1
2
3
4
5
6
7
8
9
10
11
# vim /data/shell/myif.sh
number1=3
number2=4
if [ $number1 -eq $number2 ]; then
echo "$number1 equal $number2"
echo "in if"
echo "student"
else
echo "$number1 not equal $number2"
echo "in else"
fi
1
2
3
# bash /data/shell/myif.sh
3 not equal 4
in else

不同shell语法及判断实战

node1

bash shell数字判断案例:

1
2
3
4
5
6
7
8
9
# vim /data/shell/myif.sh
number1=3
number2=3
if [ $number1 -eq $number2 ]; then
echo "$number1 equal $number2"
echo "student"
else
echo "$number1 not equal $number2"
fi
1
2
3
# bash /data/shell/myif.sh
3 equal 3
student

为什么说每种shell的语法不一样?底下为c shell判断的语法:

1
2
3
4
5
6
7
8
# yum install -y tcsh
# vim /data/shell/mycif.sh
if ( 3 == 3 ) then
echo "3 equal 3"
echo "student"
else
echo "in else"
endif
1
2
3
# tcsh /data/shell/mycif.sh
3 equal 3
student

检测端口是否可通的命令,端口通的话返回0,端口不通返回非0:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
# yum install -y nmap
# nc -zv 127.0.0.1 22
Ncat: Version 7.50 ( https://nmap.org/ncat )
Ncat: Connected to 127.0.0.1:22.
Ncat: 0 bytes sent, 0 bytes received in 0.04 seconds.
# echo $?
0
# nc -zv 127.0.0.1 80
Ncat: Version 7.50 ( https://nmap.org/ncat )
Ncat: Connection refused.
# echo $?
1
# nc -w 2 -zv www.baidu.com 81
Ncat: Version 7.50 ( https://nmap.org/ncat )
Ncat: Connection to 220.181.38.149 failed: Connection timed out.
Ncat: Trying next address...
Ncat: Connection to 220.181.38.150 failed: Connection timed out.
Ncat: Trying next address...
Ncat: Connection to 240e:83:205:5a:0:ff:b05f:346b failed: Network is unreachable.
Ncat: Trying next address...
Ncat: Network is unreachable.
# echo $?
1
  • -z:测试端口连接状态。

  • -v:输出详细的状态。

  • -w:设置超时时间,返回值非0。

脚本,判断nginx是否存活,不存活则启动:

1
2
3
4
5
6
7
8
9
10
11
# vim /data/shell/detect_nginx.sh
myip=127.0.0.1
myport=80
nc -w 2 -zv $myip $myport
if [ $? -eq 0 ]; then
echo "port is alived"
else
echo "port is not alived"
echo "run nginx"
/usr/local/nginx/sbin/nginx
fi
1
2
3
4
5
6
7
8
9
10
# bash /data/shell/detect_nginx.sh
Ncat: Version 7.50 ( https://nmap.org/ncat )
Ncat: Connection refused.
port is not alived
run nginx
# bash /data/shell/detect_nginx.sh
Ncat: Version 7.50 ( https://nmap.org/ncat )
Ncat: Connected to 127.0.0.1:80.
Ncat: 0 bytes sent, 0 bytes received in 0.01 seconds.
port is alived

shell条件判断之字符串判断

node1

条件判断语法:

1
2
3
4
5
if 条件; then
条件成立执行的命令,可以有多个命令
else
条件不成立执行的命令,可以多个命令
fi

条件判断之字符串判断:

  • $str1 == $str2:字符串是否相等。

  • $str1 != $str2:字符串是否不相等。

  • -z “$str”:字符串长度是否为0,zero。

  • -n “$str”:字符串长度是否不为0,not zero。

条件判断之字符串判断测试:

1
2
3
4
5
6
7
8
9
# vim /data/shell/mytest.sh
[ "student" == "student" ]
echo $?
[ "student" == "student " ]
echo $?
[ -z "student" ]
echo $?
[ -n "student" ]
echo $?
1
2
3
4
5
# bash /data/shell/mytest.sh
0
1
1
0

字符串判断举例,建议变量要加双引号:

1
2
3
4
5
6
7
# vim /data/shell/mystrtest.sh
myname="student"
if [ "$myname" == "student" ]; then
echo "my name is student"
else
echo "my name is not student"
fi
1
2
# bash /data/shell/mystrtest.sh
my name is student

变量不加双引号的话,可能会有语法错,未定义:

1
2
3
4
5
6
# vim /data/shell/mystrtest.sh
if [ $myname == "student" ]; then
echo "your name is student"
else
echo "your name is not student"
fi
1
2
3
# bash /data/shell/mystrtest.sh
/data/shell/mystrtest.sh: line 1: [: ==: unary operator expected
your name is not student

判断rpm包是否已安装,升级有可能会导致系统问题:

1
2
3
4
5
# rpm -qa pcre-devel
pcre-devel-8.32-17.el7.x86_64
# result=$(rpm -qa pcre-devel)
# echo $result
pcre-devel-8.32-17.el7.x86_64
1
2
3
# result=$(rpm -qa pcre-devel)
# echo $result
pcre-devel-8.32-17.el7.x86_64

安装rpm包,已安装则跳过:

1
2
3
4
5
6
7
8
9
# vim /data/shell/mystrapp.sh
softpack="pcre-devel"
soft_result=$(rpm -qa $softpack)
if [ -z "$soft_result" ]; then
echo "${softpack} is not find"
yum install -y ${softpack}
else
echo "${softpack} is find"
fi
1
2
# bash /data/shell/mystrapp.sh
pcre-devel is find

shell条件判断之文件判断

文件、目录、权限判断语法:

  • -e “$path”:判断文件或目录是否存在 ,exits。

  • -f “$path”:判断文件是否存在,file。

  • -d “$path”:判断目录是否存在 ,directory。

  • -w “$path”:判断是否可写,write。

  • -r “$path”:判断是否可读,read。

  • -x “$path”:判断是否可执行,execute。

node1

文件判断测试:

1
2
3
4
5
6
7
8
9
10
11
12
13
# vim /data/shell/filetest.sh
[ -e "/etc/passwd" ]
echo $?
[ -e "/etc/" ]
echo $?
[ -e "/tmp/nononono" ]
echo $?
[ -f "/tmp/" ]
echo $?
[ -x "/etc/passwd" ]
echo $?
[ -x "/bin/bash" ]
echo $?
1
2
3
4
5
6
7
# bash /data/shell/filetest.sh
0
0
1
1
1
0

举例如下:

1
2
3
4
5
6
7
# vim /data/shell/filetest.sh
filename="/etc/passwd"
if [ -f "$filename" ]; then
echo "$filename is a file"
else
echo "$filename is not a file"
fi
1
2
# bash /data/shell/filetest.sh
/etc/passwd is a file

nginx安装脚本改善,判断是否已经安装nginx:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
# vim /data/shell/install_nginx.sh
if [ -d "/usr/local/nginx" ]; then
echo "nginx is installed. exit"
exit 1
fi
nginxdownloadurl="http://nginx.org/download/nginx-1.12.2.tar.gz"
yum install -y wget gcc pcre pcre-devel zlib zlib-devel
cd /usr/local/src/
wget "$nginxdownloadurl"
tar xzvf nginx-1.12.2.tar.gz
cd nginx-1.12.2
./configure --prefix=/usr/local/nginx
make
make install
1
2
# rm -rf /usr/local/nginx/ /usr/local/src/nginx-*
# bash /data/shell/install_nginx.sh

shell的复杂判断

shell复杂判断:

  • 复杂判断:判断多个条件。

  • 多个条件其中一个成立。

  • 多个条件都要成立。

复杂判断语法:

  • ||:或的意思,两个条件只要一个条件成立就好。

  • &&:与的意思,两个条件都要成立。

node1

复杂判断使用,或判断:

1
2
3
4
5
6
7
# vim /data/shell/myif.sh
name="nonono"
if [ "$name" == "student" ] || [ "$name" == "teacher" ]; then
echo "$name is my want"
else
echo "in else"
fi
1
2
# bash /data/shell/myif.sh
in else

复杂判断使用,与判断:

1
2
3
4
5
6
7
8
# vim /data/shell/myif.sh
age=80
if [ $age -gt 10 ] && [ $age -lt 75 ]; then
echo "age > 10 and age < 75"
echo "working"
else
echo "in else"
fi
1
2
# bash /data/shell/myif.sh
in else

多复杂判断:

1
2
3
4
5
6
7
# vim /data/shell/myif.sh
age=0
if [ $age -gt 2 ] && [ $age -lt 10 ] || [ $age -eq 100 ]; then
echo "age is > 2 and age is < 10 or age == 100"
else
echo "in else"
fi
1
2
# bash /data/shell/myif.sh
in else

编译安装脚本中的make && make install怎么样理解?

&&代表两个条件都成立,所以当make运行失败的话,make install就不会再运行,所以等价于:

1
2
3
4
make
if [ $? -eq 0 ];then
make install
fi

nginx安装使用&&部署:

1
2
3
4
5
6
7
8
9
10
11
12
13
# rm -rf /usr/local/nginx/ /usr/local/src/nginx*
# vim /data/shell/install_nginx.sh
if [ -e "/usr/local/nginx" ]; then
echo "nginx is installed. exit script"
exit 1
fi
nginxdownloadurl="http://nginx.org/download/nginx-1.12.2.tar.gz"
yum install - wget gcc pcre pcre-devel zlib zlib-devel
cd /usr/local/src/
wget "$nginxdownloadurl"
tar -zxvf nginx-1.12.2.tar.gz
cd nginx-1.12.2
./configure --prefix=/usr/local/nginx && make && make install
1
# bash /data/shell/install_nginx.sh

shell多重判断及优化

node1

多重判断语法elif,else if:

1
2
3
4
5
6
7
8
9
if 条件1; then
命令,条件1成立执行。
elif 条件2; then
命令,条件1不成立,条件2成立执行。
elif 条件3; then
命令,条件1不成立,条件2不成立,条件3成立执行。
else
命令,以上条件都不成立执行。
fi

案例如下,只能进入一个命令块:

1
2
3
4
5
6
7
8
9
10
# vim /data/shell/myifif.sh
number1=10
if [ $number1 -le 0 ]; then
echo "$number1 <= 0"
elif [ $number1 -le 10 ]; then
echo "$number1 > 0"
echo "$number1 <= 10"
else
echo "$number1 > 10"
fi
1
2
3
# bash /data/shell/myifif.sh
10 > 0
10 <= 10

字符串案例:

1
2
3
4
5
6
7
8
9
# vim /data/shell/myifif.sh
choice="No"
if [ "$choice" == "no" ] || [ "$choice" == "No" ]; then
echo "choice is no"
elif [ "$choice" == "yes" ]; then
echo "choice is yes"
else
echo "not no not yes"
fi
1
2
# bash /data/shell/myifif.sh
choice is no

if做多重判断有个问题,就是当条件比较多的时候比较麻烦,可以借用case去改善它。

case多重判断语法,case还支持正则,后续会讲解正则:

1
2
3
4
5
6
7
8
9
10
11
12
# vim /data/shell/myifif.sh
choice="student"
case "$choice" in
"yes" | "Yes")
echo "in yes";;
"no" | "No")
echo "in no";;
"student" | "teacher")
echo "in student";;
*)
echo "in other other";;
esac
1
2
# bash /data/shell/myifif.sh
in student

shell读取用户输入

读取用户输入:

  • 选择性输入,例如yes or no。

  • 密码输入,不适合回显。

node1

read命令读取用户输入,输入的内容放到变量里:

1
2
3
4
5
# vim /data/shell/myread.sh
read name
echo "read name, name is $name"
read age
echo "read age, age is $age"
1
2
3
4
5
# bash /data/shell/myread.sh
abc
read name, name is abc
123
read age, age is 123

read命令的返回值,中断为非0:

1
2
3
4
# read name
^C
# echo $?
130

read -p选项可提示用户输入什么内容:

1
2
3
4
5
# vim /data/shell/myread.sh
read -p "Please input your name: " name
echo "read name, name is $name"
read -p "Please input your age: " age
echo "read age, age is $age"
1
2
3
4
5
# bash /data/shell/myread.sh
Please input your name: abc
read name, name is abc
Please input your age: 123
read age, age is 123

read -s选项输入密码,输入不显示:

1
2
3
# vim /data/shell/mypwd.sh
read -s -p "Please input your password: " mypwd
echo "read pwd, pwd is $mypwd"
1
2
# bash /data/shell/mypwd.sh
Please input your password: read pwd, pwd is 123

更改nginx安装脚本,支持输入安装目录:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
# rm -rf /usr/local/nginx/ /usr/local/src/nginx*
# vim /data/shell/install_nginx.sh
read -p "Please input you installpath: " installpath
if [ -e "$installpath" ]; then
echo "nginx is installed. exit script"
exit 1
fi
nginxdownloadurl="http://nginx.org/download/nginx-1.12.2.tar.gz"
yum install -y wget gcc pcre pcre-devel zlib zlib-devel
cd /usr/local/src/
wget "$nginxdownloadurl"
tar xzvf nginx-1.12.2.tar.gz
cd nginx-1.12.2
./configure --prefix=$installpath && make && make install
1
2
# bash /data/shell/install_nginx.sh
Please input you installpath: /usr/local/nginx

shell运算符和命令行传参

node1

let命令实现简单运算,不支持浮点数:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
# vim /data/shell/mycount.sh
# shell默认是字符串
number=1+1
echo $number
# 加
let number=4+2
echo $number
# 减
let number=4-2
echo $number
# 乘
let number=4*2
echo $number
# 除
let number=4/2
echo $number
# 余
let number=4%2
echo $number
# 指数运算
let number=2**10
echo $number
# 无浮点
let number=5/2
echo $number
# 不支持浮点数,报错
let number=5/2.0
1
2
3
4
5
6
7
8
9
10
# bash /data/shell/mycount.sh
1+1
6
2
8
2
0
1024
2
/data/shell/mycount.sh: line 34: let: number=5/2.0: syntax error: invalid arithmetic operator (error token is ".0")

bc命令运行浮点数,bc命令支持浮点数:

1
2
3
# yum install -y bc
# echo "scale=2; 5/3.0" | bc
1.66

shell脚本传参,使用以下变量接收传入的参数:

  • echo $0:脚本名。

  • echo $1:第一个参数值。

  • echo $2:第二个参数值。

  • echo $n:第n个参数值。

  • echo $#:参数的个数。

  • echo $$:进程pid。

  • echo $@:所有参数。

shell脚本传参:

1
2
3
4
5
6
7
8
9
# vim /data/shell/myparam.sh
myscriptname=$0
mycity=$1
myname=$2
myage=$3
echo "script name is $myscriptname"
echo "city is $mycity"
echo "name is $myname"
echo "param length $#"
1
2
3
4
5
# bash /data/shell/myparam.sh zbc student 123
script name is /data/shell/myparam.sh
city is zbc
name is student
param length 3

shell传参数的方式安装nginx:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
# rm -rf /usr/local/src/nginx-* /usr/local/nginx
# vim /data/shell/install_nginx.sh
if [ $# -eq 0 ]; then
echo "no parameter"
exit 1
fi
installpath=$1
if [ -e "$installpath" ]; then
echo "nginx is installed. exit script"
exit 1
fi
nginxdownloadurl="http://nginx.org/download/nginx-1.12.2.tar.gz"
yum install -y wget gcc pcre pcre-devel zlib zlib-devel
cd /usr/local/src/
wget "$nginxdownloadurl"
tar -zxvf nginx-1.12.2.tar.gz
cd nginx-1.12.2
./configure --prefix=$installpath && make && make install
1
# bash /data/shell/install_nginx.sh /usr/local/nginx

shell脚本之while循环

shell循环:

  • shell循环有while循环、for循环。

  • while循环偏向于解决有规律的问题,输出从1到100。

  • for循环偏向于解决重复性的问题,循环处理文本中的每一行。

while循环和if判断共同点和区别:

  • while循环也有条件判断,当条件成立的时候,会循环执行。当条件不成立退出。

  • if判断当条件成立时,会执行一次,然后退出。当条件不成立时直接退出。

node1

if判断语法:

1
2
3
4
if 条件; then
echo "student"
echo "run"
fi

while循环语法:

1
2
3
4
5
while 条件; do
echo 'student'
echo 'change 条件'
改变条件
done

while条件不能改变的话,会形成死循环,条件不成立不执行:

1
2
3
4
5
# vim /data/shell/mywhile.sh
while [ 4 -gt 3 ]; do
echo "student"
sleep 1
done
1
2
3
4
5
6
7
8
9
10
11
# bash /data/shell/mywhile.sh
student
student
student
student
student
student
student
student
student
^C

死循环也可以用true:

1
2
3
4
5
# vim /data/shell/mywhile.sh
while true; do
echo 'student'
sleep 1
done
1
2
3
4
5
# bash /data/shell/mywhile.sh
student
student
student
^C

while循环改善,改变条件:

1
2
3
4
5
6
7
# vim /data/shell/mywhile.sh
number=1
while [ $number -le 3 ]; do
echo "student"
let number=$number+1
echo "number is $number"
done
1
2
3
4
5
6
7
# bash /data/shell/mywhile.sh
student
number is 2
student
number is 3
student
number is 4

输出1-100的实现:

1
2
3
4
5
6
# vim /data/shell/mywhile.sh
number=1
while [ $number -le 100 ]; do
echo "$number"
let number=$number+1
done
1
# bash /data/shell/mywhile.sh

shell的while循环工作中个人是用得比较少的,用得比较多的是for循环。

shell脚本之for循环

shell的for循环:

  • for循环对于批量管理服务器非常有用。

  • for默认以所有的空白字符进行分隔:tab、空格、回车,去循环处理。

node1

for循环语法,分隔成几段就循环几次:

1
2
3
for line in xxx yyy zzz; do
循环处理每个字符串,循环三次。
done

输出student001到student100:

1
2
3
4
# vim /data/shell/myfor.sh
for line in $(seq -w 100); do
echo "student$line"
done
1
# bash /data/shell/myfor.sh

一个字符串一个字符串处理,空格、Tab都识别:

1
2
3
4
# vim /data/shell/myfor.sh
for name in student teacher classmate; do
echo "name is $name"
done
1
2
3
4
# bash /data/shell/myfor.sh
name is student
name is teacher
name is classmate

循环读取文件中的每一行,输出结果有问题:

1
2
3
4
# vim /data/shell/myfor.sh
for line in $(cat /etc/passwd); do
echo "read it. line is $line"
done
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
# bash /data/shell/myfor.sh
read it. line is root:x:0:0:root:/root:/bin/bash
read it. line is bin:x:1:1:bin:/bin:/sbin/nologin
read it. line is daemon:x:2:2:daemon:/sbin:/sbin/nologin
read it. line is adm:x:3:4:adm:/var/adm:/sbin/nologin
read it. line is lp:x:4:7:lp:/var/spool/lpd:/sbin/nologin
read it. line is sync:x:5:0:sync:/sbin:/bin/sync
read it. line is shutdown:x:6:0:shutdown:/sbin:/sbin/shutdown
read it. line is halt:x:7:0:halt:/sbin:/sbin/halt
read it. line is mail:x:8:12:mail:/var/spool/mail:/sbin/nologin
read it. line is operator:x:11:0:operator:/root:/sbin/nologin
read it. line is games:x:12:100:games:/usr/games:/sbin/nologin
read it. line is ftp:x:14:50:FTP
read it. line is User:/var/ftp:/sbin/nologin
read it. line is nobody:x:99:99:Nobody:/:/sbin/nologin
read it. line is systemd-network:x:192:192:systemd
read it. line is Network
read it. line is Management:/:/sbin/nologin
read it. line is dbus:x:81:81:System
read it. line is message
read it. line is bus:/:/sbin/nologin
read it. line is polkitd:x:999:998:User
read it. line is for
read it. line is polkitd:/:/sbin/nologin
read it. line is sshd:x:74:74:Privilege-separated
read it. line is SSH:/var/empty/sshd:/sbin/nologin
read it. line is postfix:x:89:89::/var/spool/postfix:/sbin/nologin
read it. line is chrony:x:998:996::/var/lib/chrony:/sbin/nologin
read it. line is apache:x:48:48:Apache:/usr/share/httpd:/sbin/nologin

解决上述问题需要改变IFS内置变量:

1
2
3
4
5
6
7
# vim /data/shell/myfor.sh
OLDIFS=$IFS
IFS=$'\n'
for line in $(cat /etc/passwd); do
echo "read it. line is $line"
done
IFS=$OLDIFS
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
# bash /data/shell/myfor.sh
read it. line is root:x:0:0:root:/root:/bin/bash
read it. line is bin:x:1:1:bin:/bin:/sbin/nologin
read it. line is daemon:x:2:2:daemon:/sbin:/sbin/nologin
read it. line is adm:x:3:4:adm:/var/adm:/sbin/nologin
read it. line is lp:x:4:7:lp:/var/spool/lpd:/sbin/nologin
read it. line is sync:x:5:0:sync:/sbin:/bin/sync
read it. line is shutdown:x:6:0:shutdown:/sbin:/sbin/shutdown
read it. line is halt:x:7:0:halt:/sbin:/sbin/halt
read it. line is mail:x:8:12:mail:/var/spool/mail:/sbin/nologin
read it. line is operator:x:11:0:operator:/root:/sbin/nologin
read it. line is games:x:12:100:games:/usr/games:/sbin/nologin
read it. line is ftp:x:14:50:FTP User:/var/ftp:/sbin/nologin
read it. line is nobody:x:99:99:Nobody:/:/sbin/nologin
read it. line is systemd-network:x:192:192:systemd Network Management:/:/sbin/nologin
read it. line is dbus:x:81:81:System message bus:/:/sbin/nologin
read it. line is polkitd:x:999:998:User for polkitd:/:/sbin/nologin
read it. line is sshd:x:74:74:Privilege-separated SSH:/var/empty/sshd:/sbin/nologin
read it. line is postfix:x:89:89::/var/spool/postfix:/sbin/nologin
read it. line is chrony:x:998:996::/var/lib/chrony:/sbin/nologin
read it. line is apache:x:48:48:Apache:/usr/share/httpd:/sbin/nologin

应用:使用for循环批量判断包是否已安装:

1
2
3
4
5
6
7
8
9
# vim /data/shell/mypackcheck.sh
for softpack in wget gcc pcre pcre-devel zlib zlib-devel; do
soft_result=$(rpm -qa $softpack)
if [ -z "$soft_result" ];then
yum install -y $softpack
else
echo "$softpack is installed"
fi
done
1
2
3
4
5
6
7
# bash /data/shell/mypackcheck.sh
wget is installed
gcc is installed
pcre is installed
pcre-devel is installed
zlib is installed
zlib-devel is installed

循环的break和continue

shell循环中的break和continue,语法适用于while、for:

  • break直接结束循环,循环立即退出。

  • continue可以用来跳过一次循环,跳过后循环继续,直到循环停止。

node1

break语法:

1
2
3
4
5
6
7
# vim /data/shell/mybreak.sh
for line in student teacher classmate roommate workmate; do
echo $line
if [ "$line" == "teacher" ];then
break
fi
done
1
2
3
# bash /data/shell/mybreak.sh
student
teacher

continue用来跳过一次循环:

1
2
3
4
5
6
7
# vim /data/shell/mycontinue.sh
for line in student teacher classmate roommate workmate; do
if [ "$line" == "teacher" ];then
continue
fi
echo $line
done
1
2
3
4
5
# bash /data/shell/mycontinue.sh
student
classmate
roommate
workmate

shell编程之函数

shell函数:

  • 模块化。

  • 函数只定义,不执行。

  • 函数定义完后可以被多次调用。

函数名:

  • 函数名的命令规则跟变量名是一样的。

函数的语法:

1
2
3
function 函数名() {
具体的功能
}

node1

函数定义function,但shell中function可有可无:

1
2
3
4
5
6
7
8
9
# vim /data/shell/myfunc.sh
function echo_hello() {
echo "hello world"
echo -e "student\n"
}
echo "call function one"
echo_hello
echo "call function two"
echo_hello
1
2
3
4
5
6
7
8
# bash /data/shell/myfunc.sh
call function one
hello world
student

call function two
hello world
student

函数支持传递参数:

  • $1:第一个参数。

  • $2:第二个参数。

  • $n:第n个参数。

函数传递参数使用:

1
2
3
4
5
6
7
8
# vim /data/shell/myfunc.sh
function echo_name_age() {
name=$1
age=$2
echo "name is $name.age is $age"
}
echo "run function with parameters"
echo_name_age ABC 123
1
2
3
# bash /data/shell/myfunc.sh
run function with parameters
name is ABC.age is 123

函数的退出码和应用

脚本的退出码:

  • 使用exit或者以最后命令的退出码为准。

  • 函数使用return定义退出码,0表示正常,非0表示异常。

  • 无return以最后一条命令。

node1

1
2
3
4
5
6
7
8
9
# vim /data/shell/myreturn.sh
function echo_name_age() {
name=$1
age=$2
echo "name is $name, age is $age"
return 6
}
echo_name_age abc 123
echo $?
1
2
3
# bash /data/shell/myreturn.sh
name is abc, age is 123
6

使用函数检查rpm包是否已安装:

1
2
3
4
5
6
7
8
9
10
11
12
13
# vim /data/shell/myreturn.sh
function check_rpm_package() {
package_name=$1
result=$(rpm -qa $package_name)
if [ -z "$result" ]; then
yum install -y $package_name
else
echo "$package_name is installed"
fi
}
for packname in pcre-devel pcre zlib zlib-devel; do
check_rpm_package $packname
done
1
2
3
4
5
# bash /data/shell/myreturn.sh
pcre-devel is installed
pcre is installed
zlib is installed
zlib-devel is installed

grep和普通正则表达式

文本处理三剑客:

  • grep。

  • awk。

  • sed。

node1

示例文本文件:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
# vim /tmp/student
student1 56 hangzhou
student2 10 shenzhen
student3 22 shanghai
zhangsan1 33 hangzhou
zhangsan2 34 foreign
zhangsan3 14 beijing
student.test
student888
ABCZ
whoiAm
#mark1
#mark2
mark3#
test test
学习

grep命令的基础使用:

  • grep全称:global regular expression and print。

  • 全局正则表达式搜索然后打印。

包含hangzhou的会整行打印:

1
2
3
4
5
6
7
8
9
# cat /tmp/student | grep "hangzhou"
student1 56 hangzhou
zhangsan1 33 hangzhou
# grep 'student' /tmp/student
student1 56 hangzhou
student2 10 shenzhen
student3 22 shanghai
student.test
student888

针对grep到的会有颜色:

1
# alias grep='grep --color=auto'

grep为什么会和正则一起讲?

  • grep普通的匹配无法快速的过滤我们想要的内容。

  • 正则表达式可以让我们查找字符串更加方便。

正则表达式:

  • 正则表达式是一种语法。

  • 正则表达式是用约定好的某些符号去表示某个含义。

正则表达式种类:

  • 普通正则表达式,定了一些基础的符号及其含义。

  • 扩展正则表达式,除了基础的符号,又扩展了更多的符号+?。

  • 定义的符号越多,我们过滤出想要的内容就越快,但需要去记住更多的语法。

常用普通正则表达式,用某种约定好的符号去表示某个含义:

  • .:任意一个字符 。

  • * :前面一个字符出现0次或者多次,跟shell通配符要区分。

1
2
3
4
5
6
7
8
9
10
11
12
# cat /tmp/student | grep 'student.'
student1 56 hangzhou
student2 10 shenzhen
student3 22 shanghai
student.test
student888
# cat /tmp/student | grep 'student.*'
student1 56 hangzhou
student2 10 shenzhen
student3 22 shanghai
student.test
student888
  • [abc]:中括号内的任意一个字符。

  • [^abc]:非中括号内的任意字符。

1
2
3
4
5
6
7
8
# cat /tmp/student | grep 'student[123]'
student1 56 hangzhou
student2 10 shenzhen
student3 22 shanghai
# cat /tmp/student | grep 'student[^12]'
student3 22 shanghai
student.test
student888
  • [0-9]:数字。

  • [a-z]:小写字母。

  • [A-Z]:大写字母。

  • [a-zA-Z]:所有字母。

  • [a-zA-Z0-9]:所有字母+数字。

  • [^0-9]:非数字。

1
2
3
4
5
6
7
8
# cat /tmp/student | grep 'student[0-9]'
student1 56 hangzhou
student2 10 shenzhen
student3 22 shanghai
student888
# cat /tmp/student | grep '[A-Z]'
ABCZ
whoiAm
  • [[:alpha:]]:字母。

  • [[:lower:]]:小写字母。

  • [[:upper:]]:大写字母。

  • [[:digit:]]:数字。

  • [[:alnum:]]:字母加数字。

  • [[:space:]]:任意空白。

  • [[:punct:]]:特殊字符。

  • [^[:alpha:]]:非大小写字母。

1
2
3
4
5
6
7
# cat /tmp/student | grep '[[:upper:]]'
ABCZ
whoiAm
# cat /tmp/student | grep '3[[:space:]]'
student3 22 shanghai
zhangsan1 33 hangzhou
zhangsan3 14 beijing
  • ^xx:以xx开头的。

  • xx$:以xx结尾的。

1
2
3
4
5
6
# cat /tmp/student | grep '^#'
#mark1
#mark2
# cat /tmp/student | grep 'g$'
zhangsan3 14 beijing
# cat /tmp/student | grep '^$'

正则字符当成普通字符使用\,加反斜杠:

1
2
# cat /tmp/student | grep '\.'
student.test

正则的通用性:

正则表达式在编程界是通用的,也就是java、python、php等都是使用同一套正则表达式,同样或者类似。

扩展正则表达式

正则表达式种类:

  • 普通正则表达式,定了一些基础的符号及其含义。

  • 扩展正则表达式,除了基础的符号,又扩展了更多的符号+和?等。

node1

示例文本文件:

1
2
3
4
5
6
7
8
9
10
11
12
13
# vim /tmp/student
student1 31 hangzhou
student2 22 shenzhen
student3 29 shanghai
zhangsan1 33 hangzhou
zhangsan2 34 foreign
zhangsan3 18 beijing
student.test
student888
studentstudent
ABCZ
whoiAm
<body><replaceHost>0</replaceHost><replacePort>0</replacePort><seq>1366120110119</seq>1366120110119</body>

grep注意事项:

  • grep默认只支持普通正则。

  • 如果使用扩展正则,则需要使用egrep或grep -E。

扩展正则:

  • ?:前面字符出现0或者1次。

  • +:前面字符出现1或者多次。

1
2
3
4
5
6
7
8
9
# cat /tmp/student |egrep 'student1?'
student1 31 hangzhou
student2 22 shenzhen
student3 29 shanghai
student.test
student888
studentstudent
# cat /tmp/student |egrep 'student1+'
student1 31 hangzhou
  • {n}:前面字符匹配n次。

  • {a,b}:前面字符匹配a到b次。

  • {,b}:前面字符匹配0次到b次。

  • {a,}:前面字符匹配a或a+次。

1
2
3
4
5
6
# cat /tmp/student |egrep 'student1{4}'
# cat /tmp/student |egrep 'student8{3}'
student888
# cat /tmp/student |egrep 'student8{1,4}'
student888
# cat /tmp/student |egrep 'student8{4,}'

student分组匹配:

1
2
# cat /tmp/student |egrep '(student){2}'
studentstudent

分组匹配反向引用\1\2:

1
2
3
4
5
6
7
8
9
10
# cat /tmp/student | egrep '<replacePort>.*</replacePort>'
<body><replaceHost>0</replaceHost><replacePort>0</replacePort><seq>1366120110119</seq>1366120110119</body>
# cat /tmp/student | egrep '<seq>.*</seq>'
<body><replaceHost>0</replaceHost><replacePort>0</replacePort><seq>1366120110119</seq>1366120110119</body>
# cat /tmp/student | egrep '<(replacePort)>.*</\1>'
<body><replaceHost>0</replaceHost><replacePort>0</replacePort><seq>1366120110119</seq>1366120110119</body>
# cat /tmp/student | egrep '<(replaceHost)>.*</\1>'
<body><replaceHost>0</replaceHost><replacePort>0</replacePort><seq>1366120110119</seq>1366120110119</body>
# cat /tmp/student | egrep '<(seq)>(.*)</\1>\2'
<body><replaceHost>0</replaceHost><replacePort>0</replacePort><seq>1366120110119</seq>1366120110119</body>

grep命令常用参数

node1

示例文本文件:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
# vim /tmp/student
student1 31 hangzhou
student2 22 shenzhen
student3 29 shanghai
zhangsan1 33 hangzhou
zhangsan2 34 foreign
zhangsan3 18 beijing
begin ask
name
student or teacher
lisi1
lisi2
#mark
#mark2
STUDENT

grep反向查找:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
# cat /tmp/student | grep -v 'student'
zhangsan1 33 hangzhou
zhangsan2 34 foreign
zhangsan3 18 beijing
begin ask
name
lisi1
lisi2
#mark
#mark2
STUDENT
# cat /tmp/student| grep -v '^#'
student1 31 hangzhou
student2 22 shenzhen
student3 29 shanghai
zhangsan1 33 hangzhou
zhangsan2 34 foreign
zhangsan3 18 beijing
begin ask
name
student or teacher
lisi1
lisi2
STUDENT

grep忽略大小写:

1
2
3
4
5
6
# cat /tmp/student | grep -i 'student'
student1 31 hangzhou
student2 22 shenzhen
student3 29 shanghai
student or teacher
STUDENT

grep仅仅打印匹配,不打印整行:

1
2
3
4
5
# cat /tmp/student | grep -o 'student.'
student1
student2
student3
student

统计字符n的数量有多少个:

1
2
# cat /tmp/student | grep -o n | wc -l
19

grep打印出上下文:

1
2
# cat /tmp/student | grep 'name'
name

打印下2行:

1
2
3
4
# cat /tmp/student | grep -A 2 'name'
name
student or teacher
lisi1

打印上2行:

1
2
3
4
# cat /tmp/student | grep -B 2 'name'
zhangsan3 18 beijing
begin ask
name

打印上下2行:

1
2
3
4
5
6
# cat /tmp/student | grep -C 2 'name'
zhangsan3 18 beijing
begin ask
name
student or teacher
lisi1

grep递归查找:

1
2
3
4
5
# grep -r 'transmission' /etc/
Binary file /etc/udev/hwdb.bin matches
/etc/protocols:tcp 6 TCP # transmission control protocol
/etc/services:fax 4557/tcp # FAX transmission service (old)
/etc/mime.types:application/news-transmission

去抓取nginx的最新稳定下载链接:

1
2
3
4
# curl -s 'http://nginx.org/en/download.html'
# curl -s 'http://nginx.org/en/download.html' | egrep -o 'Stable version.*'
# curl -s 'http://nginx.org/en/download.html' | egrep -o 'Stable version.*' | egrep -o 'href="[^"\]+"'
# curl -s 'http://nginx.org/en/download.html' | egrep -o 'Stable version.*' | egrep -o 'href="[^"\]+"' | grep 'tar.gz"$'

awk列处理入门

grep和awk:

  • grep偏向于行处理。

  • awk偏向于列处理,会把一行分成多个列。

awk语言:

  • awk同时也是一门编程语言,也有变量、数组、判断、循环等语法,awk内置了很多实用函数。

  • 要讲解的awk的内容:列处理、列过滤。

node1

示例文本文件:

1
2
3
4
5
6
7
# vim /tmp/student
student1 31 hangzhou
student2 22 shenzhen
student3 29 shanghai
zhangsan1 33 hangzhou
zhangsan2 34 foreign
zhangsan3 18 beijing

awk简单使用,默认会使用空白字符隔成一列一列的。

打印姓名:

1
2
3
4
5
6
7
# cat /tmp/student | awk '{print $1}'
tudent1
student2
student3
zhangsan1
zhangsan2
zhangsan3

打印城市:

1
2
3
4
5
6
7
# cat /tmp/student | awk '{print $3}'
hangzhou
shenzhen
shanghai
hangzhou
foreign
beijing

awk内置变量:

  • $0:整行。

  • $1:第一列。

  • $2:第二列。

  • $n:第n列。

  • $NF:最后一列 。

  • NR:当前处理到第几行 。

1
2
3
4
5
6
7
8
9
10
11
12
13
14
# cat /tmp/student |awk '{print $NF}'
hangzhou
shenzhen
shanghai
hangzhou
foreign
beijing
# cat /tmp/student |awk '{print NR}'
1
2
3
4
5
6

awk打印多列、字符串拼接:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
# cat /tmp/student | awk '{print NR,$0,$1}'
1 tudent1 31 hangzhou tudent1
2 student2 22 shenzhen student2
3 student3 29 shanghai student3
4 zhangsan1 33 hangzhou zhangsan1
5 zhangsan2 34 foreign zhangsan2
6 zhangsan3 18 beijing zhangsan3
# cat /tmp/student | awk '{print $1,$2}'
tudent1 31
student2 22
student3 29
zhangsan1 33
zhangsan2 34
zhangsan3 18
# cat /tmp/student |awk '{print "name is "$1", " "age is "$2"."}'
name is tudent1, age is 31.
name is student2, age is 22.
name is student3, age is 29.
name is zhangsan1, age is 33.
name is zhangsan2, age is 34.
name is zhangsan3, age is 18.

awk针对列过滤,整数比较:

  • number1 > number2。

  • number1 < number2。

  • number1 == number2。

  • number1 >= number2。

  • number1 <= number2。

1
2
3
4
5
# cat /tmp/student | awk '$2>30'
tudent1 31 hangzhou
zhangsan1 33 hangzhou
zhangsan2 34 foreign
# cat /tmp/student | awk '$2==52'

awk针对列过滤,字符串比较:

1
2
3
4
5
6
7
8
# cat /tmp/student | awk '$3=="hangzhou"'
tudent1 31 hangzhou
zhangsan1 33 hangzhou
# cat /tmp/student | awk '$3!="hangzhou"'
student2 22 shenzhen
student3 29 shanghai
zhangsan2 34 foreign
zhangsan3 18 beijing

awk针对列过滤,使用正则:

1
2
3
4
5
6
7
8
# cat /tmp/student | awk '$3 ~ /sh/'
student2 22 shenzhen
student3 29 shanghai
# cat /tmp/student | awk '$3 ~ /(sh)|(hang)/'
tudent1 31 hangzhou
student2 22 shenzhen
student3 29 shanghai
zhangsan1 33 hangzhou

awk默认:

  • 如果没有过滤的化,会全部行数处理。

  • 如果没有动作的话,就会打印整行{print $0}。

awk过滤加处理:

1
2
3
4
# cat /tmp/student | awk '$2>30 {print $1}'
tudent1
zhangsan1
zhangsan2

awk列处理常用知识

node1

示例文本文件:

1
2
3
4
5
6
7
# vim /tmp/student
student1 28 hangzhou
student2 30 shenzhen
student3 32 shanghai
zhangsan1 33 hangzhou
zhangsan2 34 foreign
zhangsan3 18 beijing

awk的BEGIN、END:

  • BEGIN{}大括号里:只运行一次,在文本处理开始前运行。

  • {}大括号里:针对每一行进行处理。

  • END{}大括号里:只运行一次,在文本处理结束后运行。

举例说明:

1
2
3
4
5
6
7
8
9
# cat /tmp/student | awk '{print $1} BEGIN {print "begin"} END {print "end"}'
begin
student1
student2
student3
zhangsan1
zhangsan2
zhangsan3
end

BEGIN可以用来做浮点数运算:

1
2
# awk 'BEGIN {printf("%.2f",5/3)}'
1.67

awk使用-F指定分隔符:

  • 默认分隔符是以多个空白字符作为分隔:空格、tab。

  • -F参数可指定分隔符,后面跟正则表达式,支持扩展正则。

1
2
3
4
5
6
7
8
# sed -i 's/ /:/g' /tmp/student
# cat /tmp/student | awk -F':' '{print $3}'
hangzhou
shenzhen
shanghai
hangzhou
foreign
beijing

awk多个单字符分隔:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
# cat /tmp/student | awk -F'[:;]' '{print $3}'
hangzhou
shenzhen
shanghai
hangzhou
foreign
beijing
# cat /tmp/student | awk -F':|;' '{print $3}'
hangzhou
shenzhen
shanghai
hangzhou
foreign
beijing

awk支持多字符分隔,如果多个相同的分隔符在一起要配合+使用:

1
2
3
4
5
6
7
# cat /tmp/student | awk -F ':+|;' '{print $3}'
hangzhou
shenzhen
shanghai
hangzhou
foreign
beijing

awk传参:

1
2
3
4
5
6
7
8
9
# sed -i 's/:/ /g' /tmp/student
# cat /tmp/student | awk -v age=30 '$2>age {print $0,age}'
student3 32 shanghai 30
zhangsan1 33 hangzhou 30
zhangsan2 34 foreign 30
# cat /tmp/student | awk -v age1=30 -v age2=40 '$2>age1 && $2<age2'
student3 32 shanghai
zhangsan1 33 hangzhou
zhangsan2 34 foreign

awk的复杂过滤:

1
2
3
4
# cat /tmp/student | awk '$3=="hangzhou" || $3=="shenzhen"'
student1 28 hangzhou
student2 30 shenzhen
zhangsan1 33 hangzhou

awk的判断语法

node1

示例文本文件:

1
2
3
4
5
6
7
# vim /tmp/student
student1 31 hangzhou
student2 22 shenzhen
student3 29 shanghai
zhangsan1 33 hangzhou
zhangsan2 34 foreign
zhangsan3 18 beijing

awk的判断语法,每种语言都有自己的判断语法:

  • awk ‘{ if(条件){条件成立执行}else{条件不成立执行} }’。

  • 多条语句,需要以;分隔。

数字判断条件:

  • number1 > number2。

  • number1 < number2。

  • number1 == number2。

  • number1 >= number2。

  • number1 <= number2。

字符串简单判断:

  • str1 == str2。

  • str1 != str2。

正则判断条件:

  • str1 ~ /str2/。

awk的判断例子:

1
2
3
4
5
6
7
8
9
# cat /tmp/student |awk '{ if($2>30) {print $0} }'
student1 31 hangzhou
zhangsan1 33 hangzhou
zhangsan2 34 foreign
# cat /tmp/student | awk '{ if($3=="foreign") {print $0} }'
zhangsan2 34 foreign
# cat /tmp/student |awk '{ if($3~/^hang/) {print $0} }'
student1 31 hangzhou
zhangsan1 33 hangzhou

awk数组循环语法

数组基础:

  • 数字变量只能定义一个数,数组可以定义多个数。

  • 数组有索引和值,一般的索引是使用0、1、2递增。

  • awk数组也支持使用字符串为索引,类似于字典。

node1

shell最简单的数组定义:

1
2
3
4
# numbers[0]=2
# numbers[1]=4
# numbers[2]=6
# numbers[3]=8

第一个数:

1
2
# echo ${numbers[0]}
2

第二个数:

1
2
# echo ${numbers[1]}
4

第n+1个数:

1
2
# echo ${numbers[n]}
2

所有数字:

1
2
# echo ${numbers[*]}
2 4 6 8

删除变量:

1
2
# unset numbers
# echo $number

数组赋值:

1
2
3
# numbers=(2 4 6 8)
# echo ${numbers[0]}
2

shell关联数组支持用字符串为索引:

1
2
3
4
5
6
7
# declare -A info
# info['name']="student"
# info['age']=123
# echo ${info['name']}
student
# echo ${info['age']}
123

awk循环语法:

1
2
3
for(i in 数组变量){
print i,数组变量[i]
}

array[0]++相当于:

1
array[0] = array[0] + 1

awk的数组和循环打印,awk的数组还是比较常用的:

1
2
3
4
# awk 'BEGIN{ array[0]=2; array[1]=4; print array[1] }'
4
# awk 'BEGIN{ array[0]=2; array[1]=4; print array[0] }'
2

打印索引和值:

1
2
3
4
# awk 'BEGIN{ array[0]=2; array[1]=4; array[2]=6; for(i in array){print i,array[i]} }'
0 2
1 4
2 6

简单数组的字义,数字默认值为0:

1
2
3
# awk 'BEGIN{ array[0]++;array[0]++; array[1]--; for(i in array){print i,array[i]} }'
0 2
1 -1

awk数组字符串索引:

1
2
3
# awk 'BEGIN{ info["age"]=123; info["name"]="student"; for(i in info){print i,info[i]} }'
age 123
name student

awk统计每个城市出现的次数:

1
2
3
4
5
6
7
8
9
10
11
12
# cat /tmp/student | awk '{print $3}'| sort | uniq -c
1 beijing
1 foreign
2 hangzhou
1 shanghai
1 shenzhen
# cat /tmp/student |awk '{ count[$3]++ }END{ for(city in count){print city,count[city]} }'
foreign 1
shanghai 1
beijing 1
hangzhou 2
shenzhen 1

awk分析nginx日志

node1

根据访问ip进行统计:

1
# cat access.log | awk '{ count[$1]++ }END{ for(ip in count){print ip,count[ip]} }'
1
# cat access.log | awk '{ count[$1]++ }END{ for(ip in count){print ip"\t"count[ip]} }'|sort -rnk 2

统计nginx的响应状态码。

各个状态码数量:

1
# cat access.log | awk '{count[$9]++}END{for(ip in count){print ip,count[ip]}}' 

比例统计:

1
# cat access.log | awk '{count[$9]++}END{for(status in count){print status,count[status]/NR*100"%"}}'

比例统计保留整数:

1
# cat access.log | awk '{count[$9]++}END{for(status in count){print status"\t"int(count[status]/NR*100)"%"}}'

根据ua统计:

1
# cat access.log | awk -F'"' '{print $(NF-1)}' | less
1
# cat access.log | awk -F'"' '{count[$(NF-1)]++}END{for(ua in count){print ua,count[ua]}}'

根据时间统计,统计每分钟的访问量、每秒钟的访问量。

统计每分钟的请求数:

1
# cat access.log | awk '{print $4}' | awk -F':' '{print $1":"$2":"$3}' | awk '{count[$1]++}END{for(time in count){print time,count[time]}}'

每秒钟请求,并发:

1
# cat access.log | awk '{count[$4]++}END{ for(time in count){print time,count[time]} }'

nginx日志过滤。

状态码,正常请求:

1
# cat access.log | awk '$9~/^2/'

状态码,处理异常:

1
# cat access.log | awk '$9~/^5/'

过滤含有iphone的ua:

1
# cat access.log | awk -F'"' '$(NF-1) ~ /iPhone/'

sed文本操作入门

sed说明:

  • vim可编辑文本,shell脚本中无法使用vim。

  • sed命令可对文本进行更改、删除、添加、打印,可以直接修改文本文件。

node1

示例文本文件:

1
2
3
4
5
6
7
8
9
# vim /tmp/student
Port 22 Port 22 Port 22
PermitRootLogin yes
Port 22
PasswordAuthentication yes
ListenAddress 10.175.201.36
sedsedsed
//note
nouse

sed语法:

  • sed ‘过滤+动作’ 文件路径。

sed过滤,默认无过滤,每一行都处理:

  • /^Port/:正则过滤Port。

  • 2,$:指定行数,n代表第n行,$代表最后一行。

  • /PermitRootLogin/,/ListenAddress/:包含PermitRootLogin的行开始,包含ListenAddress的行结束。

sed动作:

  • p:打印动作。

  • a:在行下面添加。

  • i:在行上面添加。

  • d:删除。

  • s/str1/str2/g:全局查找替换,str1替换成st2。

  • s/str1/str2/:查找替换,每行第一次出现的替换。

sed打印动作,跟-n结合使用,默认打印两次:

1
2
3
# cat /tmp/student | sed -n '/^Port/p'
Port 22 Port 22 Port 22
Port 22

sed默认不支持扩展正则,sed -r支持扩展正则:

1
2
3
# cat /tmp/student | sed -r -n '/^Port+/p'
Port 22 Port 22 Port 22
Port 22

打印第三行到最后一行:

1
2
3
4
5
6
7
# cat /tmp/student | sed -n '3,$p'
Port 22
PasswordAuthentication yes
ListenAddress 10.175.201.36
sedsedsed
//note
nouse

PermitRootLogin开始sed结束:

1
2
3
4
5
6
# cat /tmp/student | sed -n '/PermitRootLogin/,/sed/p'
PermitRootLogin yes
Port 22
PasswordAuthentication yes
ListenAddress 10.175.201.36
sedsedsed

sed添加文本,每行行上增加:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
# cat /tmp/student | sed 'i student'
student
Port 22 Port 22 Port 22
student
PermitRootLogin yes
student
Port 22
student
PasswordAuthentication yes
student
ListenAddress 10.175.201.36
student
sedsedsed
student
//note
student
nouse

需要增加空格,使用反斜杠:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
# cat /tmp/student | sed 'a \ shanghai'
Port 22 Port 22 Port 22
shanghai
PermitRootLogin yes
shanghai
Port 22
shanghai
PasswordAuthentication yes
shanghai
ListenAddress 10.175.201.36
shanghai
sedsedsed
shanghai
//note
shanghai
nouse
shanghai

过滤指定行添加:

1
2
3
4
5
6
7
8
9
10
# cat /tmp/student | sed '/sedsedsed/a \ teacher add'
Port 22 Port 22 Port 22
PermitRootLogin yes
Port 22
PasswordAuthentication yes
ListenAddress 10.175.201.36
sedsedsed
teacher add
//note
nouse

sed删除,过滤指定行删除:

1
2
3
4
5
6
7
# cat /tmp/student | sed '/^Port/d'
PermitRootLogin yes
PasswordAuthentication yes
ListenAddress 10.175.201.36
sedsedsed
//note
nouse

sed文本替换和应用

node1

示例文本文件:

1
2
3
4
5
6
7
8
9
10
# vim /tmp/student
Port 22 Port 22 Port 22
PermitRootLogin yes
Port 22
PasswordAuthentication yes
ListenAddress 10.175.201.36
sedsedsed
//note
nouse
Port

全局替换:

1
2
3
4
5
6
7
8
9
10
# cat /tmp/student | sed 's/22/1111/g'
Port 1111 Port 1111 Port 1111
PermitRootLogin yes
Port 1111
PasswordAuthentication yes
ListenAddress 10.175.201.36
sedsedsed
//note
nouse
Port

每行第一次出现替换:

1
2
3
4
5
6
7
8
9
10
# cat /tmp/student | sed 's/22/1111/'
Port 1111 Port 22 Port 22
PermitRootLogin yes
Port 1111
PasswordAuthentication yes
ListenAddress 10.175.201.36
sedsedsed
//note
nouse
Port

sed正则替换,支持普通正则和扩展正则-r:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
# cat /tmp/student | sed 's/ListenAddress .*/ListenAddress 192.168.0.1/'
Port 22 Port 22 Port 22
PermitRootLogin yes
Port 22
PasswordAuthentication yes
ListenAddress 192.168.0.1
sedsedsed
//note
nouse
Port
# cat /tmp/student | sed -r 's/(sed){3}/\1/'
Port 22 Port 22 Port 22
PermitRootLogin yes
Port 22
PasswordAuthentication yes
ListenAddress 10.175.201.36
sed
//note
nouse
Port

sed分隔符可以有多种选择,一般都用/:

1
2
3
4
5
6
7
8
9
10
# cat /tmp/student |sed 's#//#;#'
Port 22 Port 22 Port 22
PermitRootLogin yes
Port 22
PasswordAuthentication yes
ListenAddress 10.175.201.36
sedsedsed
;note
nouse
Port

过滤加替换:

1
2
3
4
5
6
7
8
9
10
# cat /tmp/student | sed '1,2s/22/1111/g'
Port 1111 Port 1111 Port 1111
PermitRootLogin yes
Port 22
PasswordAuthentication yes
ListenAddress 10.175.201.36
sedsedsed
//note
nouse
Port

正则过滤加替换:

1
2
3
4
5
6
7
8
9
10
# cat /tmp/student | sed '/^Port/s/Port/PORT/g'
PORT 22 PORT 22 PORT 22
PermitRootLogin yes
PORT 22
PasswordAuthentication yes
ListenAddress 10.175.201.36
sedsedsed
//note
nouse
Port

sed支持直接更改文本文件,使用-i选项:

1
2
3
4
5
6
7
8
9
10
11
12
# sed 's/22/1111/g' /tmp/student
Port 1111 Port 1111 Port 1111
PermitRootLogin yes
Port 1111
PasswordAuthentication yes
ListenAddress 10.175.201.36
sedsedsed
//note
nouse
Port
# sed -i 's/22/1111/g' /tmp/student
# sed -i '/PermitRootLogin/a PermitStudent' /tmp/student

sed更改文本文件记得备份:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
# cp /tmp/student /tmp/student.bak
# sed -i 's/1111/22/g' /tmp/student
# diff -u /tmp/student /tmp/student.bak
--- /tmp/student 2022-11-01 22:24:21.457340338 +0800
+++ /tmp/student.bak 2022-11-01 22:24:15.149135015 +0800
@@ -1,7 +1,7 @@
-Port 22 Port 22 Port 22
+Port 1111 Port 1111 Port 1111
PermitRootLogin yes
PermitStudent
-Port 22
+Port 1111
PasswordAuthentication yes
ListenAddress 10.175.201.36
sedsedsed
# vimdiff /tmp/student /tmp/student.bak

nginx安装脚本,使用sed优化:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
# vim /data/shell/test.sh
rm -rf /opt/nginx
installpath=/opt/nginx
if [ -e "$installpath" ]; then
echo "nginx is installed. exit script"
exit 1
fi
nginxdownloadurl="http://nginx.org/download/nginx-1.14.1.tar.gz"
yum install -y wget gcc pcre pcre-devel zlib zlib-devel
cd /usr/local/src/
wget "$nginxdownloadurl"
targzname=$(echo $nginxdownloadurl | sed 's#.*/##')
tar xzvf $targzname
dirname=$(echo $targzname | sed 's/.tar.gz//')
cd $dirname
./configure --prefix=$installpath && make && make install
1
# bash /data/shell/test.sh

find文件查找入门

find简单介绍:

  • find命令用来搜索指定文件。

  • 搜索到指定文件后可执行某些动作,例如rm操作。

node1

文件准备:

1
2
3
4
5
6
7
# rm -rf /tmp/student; mkdir /tmp/student; cd /tmp/student
# for line in $(seq 10);do
touch file_$line
mkdir dir_$line
done
# ln -s /tmp/student/file_9 /tmp/student/file_link
# chown nobody:nobody file_7

find语法:

  • find 目录 选项 动作。

find选项说明:

  • 选项可按文件类型、更改时间、名字等进行查找。

  • 无选项默认全查找。

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
# find /tmp/student/
/tmp/student/
/tmp/student/file_1
/tmp/student/dir_1
/tmp/student/file_2
/tmp/student/dir_2
/tmp/student/file_3
/tmp/student/dir_3
/tmp/student/file_4
/tmp/student/dir_4
/tmp/student/file_5
/tmp/student/dir_5
/tmp/student/file_6
/tmp/student/dir_6
/tmp/student/file_7
/tmp/student/dir_7
/tmp/student/file_8
/tmp/student/dir_8
/tmp/student/file_9
/tmp/student/dir_9
/tmp/student/file_10
/tmp/student/dir_10
/tmp/student/file_link

find动作说明:

  • 动作默认print,输出查找到的文件路径。

  • 动作可以自定义。

根据文件类型查找-type:

  • f:普通文件。

  • d:目录。

  • l:链接文件。

  • b:块设备文件。

  • c:字符设备文件。

  • p:管道文件。

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
# find /tmp/student/ -type f
/tmp/student/file_1
/tmp/student/file_2
/tmp/student/file_3
/tmp/student/file_4
/tmp/student/file_5
/tmp/student/file_6
/tmp/student/file_7
/tmp/student/file_8
/tmp/student/file_9
/tmp/student/file_10
# find /tmp/student/ -type d
/tmp/student/
/tmp/student/dir_1
/tmp/student/dir_2
/tmp/student/dir_3
/tmp/student/dir_4
/tmp/student/dir_5
/tmp/student/dir_6
/tmp/student/dir_7
/tmp/student/dir_8
/tmp/student/dir_9
/tmp/student/dir_10
# find /tmp/student/ -type l
/tmp/student/file_link

find根据文件名字查找,只支持通配符:

1
2
# find /tmp/student/ -name "file_9"
/tmp/student/file_9

根据文件的用户、用户组来查找:

1
2
3
4
# find /tmp/student/ -type f -user nobody
/tmp/student/file_7
# find /tmp/student/ -type f -group nobody
/tmp/student/file_7

find反向查找:

1
2
3
4
5
6
7
8
9
10
# find /tmp/student/ -type f ! -user nobody
/tmp/student/file_1
/tmp/student/file_2
/tmp/student/file_3
/tmp/student/file_4
/tmp/student/file_5
/tmp/student/file_6
/tmp/student/file_8
/tmp/student/file_9
/tmp/student/file_10

find命令帮助:

1
# find --help

find文件查找加动作

node1

linux文件时间,可用stat查看:

1
2
3
4
5
6
7
8
9
# stat /tmp/student/file_6
File: ‘/tmp/student/file_6’
Size: 0 Blocks: 0 IO Block: 4096 regular empty file
Device: fd00h/64768d Inode: 557442 Links: 1
Access: (0644/-rw-r--r--) Uid: ( 0/ root) Gid: ( 0/ root)
Access: 2023-02-27 23:27:51.366900158 +0800
Modify: 2023-02-27 23:27:51.366900158 +0800
Change: 2023-02-27 23:27:51.366900158 +0800
Birth: -
  • atime:access time访问时间。

  • mtime:modify time修改时间,比较常用。内容修改更新这个时间。

  • ctime:change time,包含内容修改或者属性修改(文件属主、文件权限)。

修改文件mtime和ctime:

1
2
3
4
5
6
7
8
9
10
# echo 'abc' >> /tmp/student/file_6
# stat /tmp/student/file_6
File: ‘/tmp/student/file_6’
Size: 4 Blocks: 8 IO Block: 4096 regular file
Device: fd00h/64768d Inode: 557442 Links: 1
Access: (0644/-rw-r--r--) Uid: ( 0/ root) Gid: ( 0/ root)
Access: 2023-02-27 23:27:51.366900158 +0800
Modify: 2023-02-27 23:29:43.481399161 +0800
Change: 2023-02-27 23:29:43.481399161 +0800
Birth: -

修改文件ctime:

1
2
3
4
5
6
7
8
9
10
# chown nobody:nobody file_6
# stat /tmp/student/file_6
File: ‘/tmp/student/file_6’
Size: 4 Blocks: 8 IO Block: 4096 regular file
Device: fd00h/64768d Inode: 557442 Links: 1
Access: (0644/-rw-r--r--) Uid: ( 99/ nobody) Gid: ( 99/ nobody)
Access: 2023-02-27 23:27:51.366900158 +0800
Modify: 2023-02-27 23:29:43.481399161 +0800
Change: 2023-02-27 23:29:57.003579955 +0800
Birth: -

atime不生效原因,由于atime频繁更改会对文件io产生影响,因此很多线上系统atime的修改是禁止的。挂载的时候使用noatime。

1
2
例如:
/dev/mapper/centos-root / xfs defaults,noatime 0 0

根据修改时间查找。

3分钟前修改:

1
# find /tmp/student/ -type f -mmin +3

6天前修改的:

1
2
3
4
# find /tmp/student/ -type f -mtime +6
# touch -d "365 days ago" /tmp/student/file_9
# find /tmp/student/ -type f -mtime +6
/tmp/student/file_9

相对文件时间查找,比file_6更新的文件:

1
2
3
4
5
6
7
8
9
10
11
12
13
# echo 'abc' >> /tmp/student/file_10
# find /tmp/student/ -type f -newer file_6
/tmp/student/file_10
# find /tmp/student/ -type f ! -newer file_6
/tmp/student/file_1
/tmp/student/file_2
/tmp/student/file_3
/tmp/student/file_4
/tmp/student/file_5
/tmp/student/file_6
/tmp/student/file_7
/tmp/student/file_8
/tmp/student/file_9

绝对时间查找,比指定时间更新的文件:

1
2
3
4
5
6
7
8
9
10
11
# find /tmp/student/ -type f -newermt '2018-11-25 13:15:00'
/tmp/student/file_1
/tmp/student/file_2
/tmp/student/file_3
/tmp/student/file_4
/tmp/student/file_5
/tmp/student/file_6
/tmp/student/file_7
/tmp/student/file_8
/tmp/student/file_9
/tmp/student/file_10

find结合xargs:

1
# find /tmp/student/ -type f -newer file_6 -print | xargs rm

find结合xargs存在问题:

1
2
3
# touch '/tmp/student/blank  student'
# find /tmp/student/ -type f -name "blank*"
/tmp/student/blank student

报错,因为默认有空格作为分隔符:

1
2
3
# find /tmp/student/ -type f -name "blank*" | xargs ls -l
ls: cannot access /tmp/student/blank: No such file or directory
ls: cannot access student: No such file or directory

使用-print0和-0代表以\0作为分隔符:

1
2
3
4
# find /tmp/student/ -type f -name "blank*" -print0 | xargs -0 ls -l
-rw-r--r-- 1 root root 0 Nov 2 06:30 /tmp/student/blank student
# find /tmp/student/ -type f -print0 | xargs -0 grep 'abc'
/tmp/student/file_6:abc

统计各个字符出现的次数

node1

统计基础:

1
2
3
4
5
# cat /etc/ssh/sshd_config | grep -o a
# cat /etc/ssh/sshd_config | grep -o a | wc
153 153 306
# cat /etc/ssh/sshd_config | grep -o a | wc -l
153

使用{a..z}可以循环获取26个小写字母:

1
2
3
4
# vim /data/shell/countchar.sh
for i in {a..z}; do
echo "read it $i"
done
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
# bash /data/shell/countchar.sh
read it a
read it b
read it c
read it d
read it e
read it f
read it g
read it h
read it i
read it j
read it k
read it l
read it m
read it n
read it o
read it p
read it q
read it r
read it s
read it t
read it u
read it v
read it w
read it x
read it y
read it z

循环统计每个字母出现的次数:

1
2
3
4
# vim /data/shell/countchar.sh
for i in {a..z}; do
cat /etc/ssh/sshd_config | grep -o $i | wc -l
done
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
# bash /data/shell/countchar.sh
153
34
86
94
324
30
46
116
178
1
24
88
39
223
234
59
0
138
271
240
81
31
37
13
74
8

把字母打印出来更加直观:

1
2
3
4
5
# vim /data/shell/countchar.sh
for i in {a..z}; do
countresult=$(cat /etc/ssh/sshd_config | grep -o $i | wc -l)
echo -e "$i\t$countresult"
done
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
# bash /data/shell/countchar.sh
a 153
b 34
c 86
d 94
e 324
f 30
g 46
h 116
i 178
j 1
k 24
l 88
m 39
n 223
o 234
p 59
q 0
r 138
s 271
t 240
u 81
v 31
w 37
x 13
y 74
z 8

统计大写字母:

1
2
3
4
5
# vim /data/shell/countchar.sh
for i in {A..Z}; do
result=`cat /etc/ssh/sshd_config | grep -o $i | wc -l`
echo -e "$i\t$result"
done
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
# bash /data/shell/countchar.sh
A 68
B 3
C 38
D 10
E 33
F 17
G 14
H 13
I 23
J 0
K 17
L 38
M 24
N 14
O 11
P 44
Q 0
R 20
S 37
T 29
U 18
V 2
W 1
X 5
Y 4
Z 0

统计每个数字:

1
2
3
4
5
# vim /data/shell/countchar.sh
for i in {0..9}; do
result=`cat /etc/ssh/sshd_config | grep -o $i | wc -l`
echo -e "$i\t$result"
done
1
2
3
4
5
6
7
8
9
10
11
# bash /data/shell/countchar.sh
0 16
1 18
2 8
3 3
4 1
5 5
6 2
7 0
8 1
9 1

统计每个字母、每个数字出现的次数:

1
2
3
4
5
# vim /data/shell/countchar.sh
for i in {0..9} {a..z} {A..Z}; do
countresult=$(cat /etc/ssh/sshd_config | grep -o $i | wc -l)
echo -e "$i\t$countresult"
done
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
# bash /data/shell/countchar.sh
0 16
1 18
2 8
3 3
4 1
5 5
6 2
7 0
8 1
9 1
a 153
b 34
c 86
d 94
e 324
f 30
g 46
h 116
i 178
j 1
k 24
l 88
m 39
n 223
o 234
p 59
q 0
r 138
s 271
t 240
u 81
v 31
w 37
x 13
y 74
z 8
A 68
B 3
C 38
D 10
E 33
F 17
G 14
H 13
I 23
J 0
K 17
L 38
M 24
N 14
O 11
P 44
Q 0
R 20
S 37
T 29
U 18
V 2
W 1
X 5
Y 4
Z 0

注意如果统计到正则字符需要加反斜杠\:

1
2
# cat /etc/ssh/sshd_config | grep --color '\.' | wc -l
18

shell批量下发公钥

登录服务器步骤(从一台linux登录到另外一台linux):

  • 登录服务器需要有两个步骤,一个输入yes、一个输入密码。

  • 试想如果有上千台服务器?每台都要输入yes、输入密码。

节点IP功能CPU内存硬盘
node110.80.10.1server4核心8GB20GB
node210.80.10.2client4核心8GB20GB

node1

修改配置ssh免输入yes,输入密码进行连接:

1
2
3
# sed -i 's/#   StrictHostKeyChecking ask/StrictHostKeyChecking no/g' /etc/ssh/ssh_config
# ssh 10.80.10.2
# exit

ssh免输入密码的实现,方式1:

1
2
3
4
# yum install -y sshpass
# sshpass -p "toortoor" ssh 10.80.10.1
# logout
# sshpass -p "toortoor" ssh 10.80.10.2 'hostname'

批量管理:

1
2
3
# vim /tmp/test
10.80.10.1
10.80.10.2
1
2
3
4
5
# vim /data/shell/allserver.sh
for ip in $(cat /tmp/test); do
echo "run command in $ip"
sshpass -p "toortoor" ssh $ip 'uptime'
done
1
2
3
4
5
# bash /data/shell/allserver.sh
run command in 10.80.10.1
22:18:01 up 3:47, 3 users, load average: 0.38, 0.27, 0.19
run command in 10.80.10.2
22:18:01 up 1 min, 0 users, load average: 0.27, 0.14, 0.05

ssh免输入密码的实现2,通过密钥:

  • 生成公钥(锁)、私钥(钥匙),公钥放被登录的服务器上,私钥需要自己保留。

  • 手动把公钥上传到被登录的服务器上,私钥就能进行无密码登录。

1
2
3
4
# ssh-keygen -t rsa
回车
回车
回车

借助ssh-copy-id来实现公钥上传,需要输入服务器的密码:

1
2
3
# ssh-copy-id -i /root/.ssh/id_rsa.pub 10.80.10.2
# ssh 10.80.10.2
# logout

node2

删除.ssh目录:

1
# rm -rf /root/.ssh/

node1

sshpass自动输入密码:

1
# sshpass -p 'toortoor' ssh-copy-id -i /root/.ssh/id_rsa.pub 10.80.10.2

node2

删除.ssh目录:

1
# rm -rf /root/.ssh/

node1

上千台服务器批量下发公钥,配置文件:

1
2
3
# vim /root/.host
node1 10.80.10.1 toortoor
node2 10.80.10.2 toortoor

shell脚本实现批量下发公钥:

1
2
3
4
5
6
7
8
9
10
11
12
# vim /data/shell/allserver.sh
OLDIFS=$IFS
IFS=$'\n'

for line in $(cat /root/.host); do
myip=$(echo $line | awk '{print $2}')
mypwd=$(echo $line | awk '{print $3}')
echo "$myip $mypwd"
sshpass -p $mypwd ssh-copy-id -i /root/.ssh/id_rsa.pub $myip
done

IFS=$OLDIFS
1
# bash /data/shell/allserver.sh

验证是否都能无密码登录:

1
2
3
4
# ssh 10.80.10.1
# logout
# ssh 10.80.10.2
# logout

shell批量管理服务器

node1

配置文件格式:

1
2
3
# vim /data/shell/serverlist
node1 10.80.10.1
node2 10.80.10.2

批量修改主机名:

1
2
3
4
5
6
7
8
9
10
11
12
13
# vim /data/shell/allserver.sh
OLDIFS=$IFS
IFS=$'\n'

for line in $(cat /data/shell/serverlist); do
myhost=$(echo $line | awk '{print $1}')
myip=$(echo $line | awk '{print $2}')
echo "myhost is $myhost. myip is $myip"
ssh $myip "hostnamectl set-hostname $myhost"
ssh $myip "echo $myip $myhost >> /etc/hosts"
done

IFS=$OLDIFS
1
2
3
# bash /data/shell/allserver.sh
myhost is node1. myip is 10.80.10.1
myhost is node2. myip is 10.80.10.2

批量执行shell命令:

1
2
3
4
5
6
7
8
9
10
11
12
# vim /data/shell/allserver.sh
OLDIFS=$IFS
IFS=$'\n'

for line in $(cat /data/shell/serverlist); do
myhost=$(echo $line | awk '{print $1}')
myip=$(echo $line | awk '{print $2}')
echo "myhost is $myhost. myip is $myip"
ssh $myip "uptime; ip addr show"
done

IFS=$OLDIFS
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
# bash /data/shell/allserver.sh
myhost is node1. myip is 10.80.10.1
22:22:57 up 3:52, 3 users, load average: 0.21, 0.19, 0.17
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000
link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
inet 127.0.0.1/8 scope host lo
valid_lft forever preferred_lft forever
inet6 ::1/128 scope host
valid_lft forever preferred_lft forever
2: ens33: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP group default qlen 1000
link/ether 00:0c:29:c3:f0:68 brd ff:ff:ff:ff:ff:ff
inet 10.80.10.1/16 brd 10.80.255.255 scope global noprefixroute ens33
valid_lft forever preferred_lft forever
inet6 fe80::3bf:b5b5:3a15:39fd/64 scope link noprefixroute
valid_lft forever preferred_lft forever
myhost is node2. myip is 10.80.10.2
22:22:57 up 6 min, 0 users, load average: 0.06, 0.09, 0.05
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000
link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
inet 127.0.0.1/8 scope host lo
valid_lft forever preferred_lft forever
inet6 ::1/128 scope host
valid_lft forever preferred_lft forever
2: ens33: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP group default qlen 1000
link/ether 00:0c:29:94:e5:2c brd ff:ff:ff:ff:ff:ff
inet 10.80.10.2/16 brd 10.80.255.255 scope global noprefixroute ens33
valid_lft forever preferred_lft forever
inet6 fe80::3a73:b636:db6a:82f4/64 scope link noprefixroute
valid_lft forever preferred_lft forever

批量观察服务器日志:

1
2
3
4
5
6
7
8
9
10
11
12
# vim /data/shell/allserver.sh
OLDIFS=$IFS
IFS=$'\n'

for line in $(cat /data/shell/serverlist); do
myhost=$(echo $line | awk '{print $1}')
myip=$(echo $line | awk '{print $2}')
echo "myhost is $myhost. myip is $myip"
ssh $myip "timeout 1 tail -f /var/log/messages"
done

IFS=$OLDIFS
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
# bash /data/shell/allserver.sh
myhost is node1. myip is 10.80.10.1
Dec 9 22:22:33 node1 systemd: Started Session 19 of user root.
Dec 9 22:22:33 node1 systemd-logind: Removed session 19.
Dec 9 22:22:33 node1 systemd-logind: New session 20 of user root.
Dec 9 22:22:33 node1 systemd: Started Session 20 of user root.
Dec 9 22:22:33 node1 systemd-logind: Removed session 20.
Dec 9 22:22:56 node1 systemd-logind: New session 21 of user root.
Dec 9 22:22:56 node1 systemd: Started Session 21 of user root.
Dec 9 22:22:57 node1 systemd-logind: Removed session 21.
Dec 9 22:23:19 node1 systemd-logind: New session 22 of user root.
Dec 9 22:23:19 node1 systemd: Started Session 22 of user root.
myhost is node2. myip is 10.80.10.2
Dec 9 22:22:34 node2 systemd-logind: Removed session 14.
Dec 9 22:22:34 node2 systemd: Removed slice User Slice of root.
Dec 9 22:22:57 node2 systemd: Created slice User Slice of root.
Dec 9 22:22:57 node2 systemd-logind: New session 15 of user root.
Dec 9 22:22:57 node2 systemd: Started Session 15 of user root.
Dec 9 22:22:57 node2 systemd-logind: Removed session 15.
Dec 9 22:22:57 node2 systemd: Removed slice User Slice of root.
Dec 9 22:23:20 node2 systemd: Created slice User Slice of root.
Dec 9 22:23:20 node2 systemd-logind: New session 16 of user root.
Dec 9 22:23:20 node2 systemd: Started Session 16 of user root.

批量上传文件:

1
2
3
4
5
6
7
8
9
10
11
12
13
# echo "test scp" >> /tmp/student.txt
# vim /data/shell/allserver.sh
OLDIFS=$IFS
IFS=$'\n'

for line in $(cat /data/shell/serverlist); do
myhost=$(echo $line | awk '{print $1}')
myip=$(echo $line | awk '{print $2}')
echo "myhost is $myhost. myip is $myip"
scp /tmp/student.txt $myip:/usr/local/src/
done

IFS=$OLDIFS
1
2
3
4
5
6
7
# bash /data/shell/allserver.sh
myhost is node1. myip is 10.80.10.1
student.txt 100% 9 3.5KB/s 00:00
myhost is node2. myip is 10.80.10.2
student.txt 100% 9 1.5KB/s 00:00
# ls /usr/local/src/student.txt
/usr/local/src/student.txt

批量拉取文件:

1
2
3
4
5
6
7
8
9
10
11
12
# vim /data/shell/allserver.sh
OLDIFS=$IFS
IFS=$'\n'

for line in $(cat /data/shell/serverlist); do
myhost=$(echo $line | awk '{print $1}')
myip=$(echo $line | awk '{print $2}')
echo "myhost is $myhost. myip is $myip"
scp $myip:/usr/local/src/student.txt /tmp/student.txt.$myip
done

IFS=$OLDIFS
1
2
3
4
5
6
7
# bash /data/shell/allserver.sh
myhost is node1. myip is 10.80.10.1
student.txt 100% 9 3.6KB/s 00:00
myhost is node2. myip is 10.80.10.2
student.txt 100% 9 2.7KB/s 00:00
# ls /tmp/
student.txt student.txt.10.80.10.1 student.txt.10.80.10.2 test

shell批量探测http接口

node1

使用curl探测http接口:

1
2
3
4
5
# curl http://www.baidu.com
<!DOCTYPE html>
<!--STATUS OK--><html> <head><meta http-equiv=content-type content=text/html;charset=utf-8><meta http-equiv=X-UA-Compatible content=IE=Edge><meta content=always name=referrer><link rel=stylesheet type=text/css href=http://s1.bdstatic.com/r/www/cache/bdorz/baidu.min.css><title>百度一下,你就知道</title></head> <body link=#0000cc> <div id=wrapper> <div id=head> <div class=head_wrapper> <div class=s_form> <div class=s_form_wrapper> <div id=lg> <img hidefocus=true src=//www.baidu.com/img/bd_logo1.png width=270 height=129> </div> <form id=form name=f action=//www.baidu.com/s class=fm> <input type=hidden name=bdorz_come value=1> <input type=hidden name=ie value=utf-8> <input type=hidden name=f value=8> <input type=hidden name=rsv_bp value=1> <input type=hidden name=rsv_idx value=1> <input type=hidden name=tn value=baidu><span class="bg s_ipt_wr"><input id=kw name=wd class=s_ipt value maxlength=255 autocomplete=off autofocus></span><span class="bg s_btn_wr"><input type=submit id=su value=百度一下 class="bg s_btn"></span> </form> </div> </div> <div id=u1> <a href=http://news.baidu.com name=tj_trnews class=mnav>新闻</a> <a href=http://www.hao123.com name=tj_trhao123 class=mnav>hao123</a> <a href=http://map.baidu.com name=tj_trmap class=mnav>地图</a> <a href=http://v.baidu.com name=tj_trvideo class=mnav>视频</a> <a href=http://tieba.baidu.com name=tj_trtieba class=mnav>贴吧</a> <noscript> <a href=http://www.baidu.com/bdorz/login.gif?login&tpl=mn&u=http%3A%2F%2Fwww.baidu.com%2f%3fbdorz_come%3d1 name=tj_login class=lb>登录</a> </noscript> <script>document.write('<a href="http://www.baidu.com/bdorz/login.gif?login&tpl=mn&u='+ encodeURIComponent(window.location.href+ (window.location.search === "" ? "?" : "&")+ "bdorz_come=1")+ '" name="tj_login" class="lb">登录</a>');</script> <a href=//www.baidu.com/more/ name=tj_briicon class=bri style="display: block;">更多产品</a> </div> </div> </div> <div id=ftCon> <div id=ftConw> <p id=lh> <a href=http://home.baidu.com>关于百度</a> <a href=http://ir.baidu.com>About Baidu</a> </p> <p id=cp>©2017 Baidu <a href=http://www.baidu.com/duty/>使用百度前必读</a> <a href=http://jianyi.baidu.com/ class=cp-feedback>意见反馈</a> 京ICP证030173号 <img src=//www.baidu.com/img/gs.gif> </p> </div> </div> </div> </body> </html>
# echo $?
0

curl -m选项设置超时时间,响应超时,卡住:

1
# curl http://www.baidu.com:82

设置2s超时:

1
2
3
4
# curl -m 2 http://www.baidu.com:82
curl: (7) Failed connect to www.baidu.com:82; Operation now in progress
# echo $?
7

curl -s选项静默输出,不会输出下载速度:

1
2
3
4
# curl -s www.baidu.com > /tmp/aaa
# cat /tmp/aaa
<!DOCTYPE html>
<!--STATUS OK--><html> <head><meta http-equiv=content-type content=text/html;charset=utf-8><meta http-equiv=X-UA-Compatible content=IE=Edge><meta content=always name=referrer><link rel=stylesheet type=text/css href=http://s1.bdstatic.com/r/www/cache/bdorz/baidu.min.css><title>百度一下,你就知道</title></head> <body link=#0000cc> <div id=wrapper> <div id=head> <div class=head_wrapper> <div class=s_form> <div class=s_form_wrapper> <div id=lg> <img hidefocus=true src=//www.baidu.com/img/bd_logo1.png width=270 height=129> </div> <form id=form name=f action=//www.baidu.com/s class=fm> <input type=hidden name=bdorz_come value=1> <input type=hidden name=ie value=utf-8> <input type=hidden name=f value=8> <input type=hidden name=rsv_bp value=1> <input type=hidden name=rsv_idx value=1> <input type=hidden name=tn value=baidu><span class="bg s_ipt_wr"><input id=kw name=wd class=s_ipt value maxlength=255 autocomplete=off autofocus></span><span class="bg s_btn_wr"><input type=submit id=su value=百度一下 class="bg s_btn"></span> </form> </div> </div> <div id=u1> <a href=http://news.baidu.com name=tj_trnews class=mnav>新闻</a> <a href=http://www.hao123.com name=tj_trhao123 class=mnav>hao123</a> <a href=http://map.baidu.com name=tj_trmap class=mnav>地图</a> <a href=http://v.baidu.com name=tj_trvideo class=mnav>视频</a> <a href=http://tieba.baidu.com name=tj_trtieba class=mnav>贴吧</a> <noscript> <a href=http://www.baidu.com/bdorz/login.gif?login&tpl=mn&u=http%3A%2F%2Fwww.baidu.com%2f%3fbdorz_come%3d1 name=tj_login class=lb>登录</a> </noscript> <script>document.write('<a href="http://www.baidu.com/bdorz/login.gif?login&tpl=mn&u='+ encodeURIComponent(window.location.href+ (window.location.search === "" ? "?" : "&")+ "bdorz_come=1")+ '" name="tj_login" class="lb">登录</a>');</script> <a href=//www.baidu.com/more/ name=tj_briicon class=bri style="display: block;">更多产品</a> </div> </div> </div> <div id=ftCon> <div id=ftConw> <p id=lh> <a href=http://home.baidu.com>关于百度</a> <a href=http://ir.baidu.com>About Baidu</a> </p> <p id=cp>©2017 Baidu <a href=http://www.baidu.com/duty/>使用百度前必读</a> <a href=http://jianyi.baidu.com/ class=cp-feedback>意见反馈</a> 京ICP证030173号 <img src=//www.baidu.com/img/gs.gif> </p> </div> </div> </div> </body> </html>

curl -v输出接口的请求和响应详细信息:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
# curl -m 2 -v http://www.baidu.com
* About to connect() to www.baidu.com port 80 (#0)
* Trying 110.242.68.4...
* Connected to www.baidu.com (110.242.68.4) port 80 (#0)
> GET / HTTP/1.1
> User-Agent: curl/7.29.0
> Host: www.baidu.com
> Accept: */*
>
< HTTP/1.1 200 OK
< Accept-Ranges: bytes
< Cache-Control: private, no-cache, no-store, proxy-revalidate, no-transform
< Connection: keep-alive
< Content-Length: 2381
< Content-Type: text/html
< Date: Tue, 01 Nov 2022 14:47:25 GMT
< Etag: "588604c1-94d"
< Last-Modified: Mon, 23 Jan 2017 13:27:29 GMT
< Pragma: no-cache
< Server: bfe/1.0.8.18
< Set-Cookie: BDORZ=27315; max-age=86400; domain=.baidu.com; path=/
<
<!DOCTYPE html>
<!--STATUS OK--><html> <head><meta http-equiv=content-type content=text/html;charset=utf-8><meta http-equiv=X-UA-Compatible content=IE=Edge><meta content=always name=referrer><link rel=stylesheet type=text/css href=http://s1.bdstatic.com/r/www/cache/bdorz/baidu.min.css><title>百度一下,你就知道</title></head> <body link=#0000cc> <div id=wrapper> <div id=head> <div class=head_wrapper> <div class=s_form> <div class=s_form_wrapper> <div id=lg> <img hidefocus=true src=//www.baidu.com/img/bd_logo1.png width=270 height=129> </div> <form id=form name=f action=//www.baidu.com/s class=fm> <input type=hidden name=bdorz_come value=1> <input type=hidden name=ie value=utf-8> <input type=hidden name=f value=8> <input type=hidden name=rsv_bp value=1> <input type=hidden name=rsv_idx value=1> <input type=hidden name=tn value=baidu><span class="bg s_ipt_wr"><input id=kw name=wd class=s_ipt value maxlength=255 autocomplete=off autofocus></span><span class="bg s_btn_wr"><input type=submit id=su value=百度一下 class="bg s_btn"></span> </form> </div> </div> <div id=u1> <a href=http://news.baidu.com name=tj_trnews class=mnav>新闻</a> <a href=http://www.hao123.com name=tj_trhao123 class=mnav>hao123</a> <a href=http://map.baidu.com name=tj_trmap class=mnav>地图</a> <a href=http://v.baidu.com name=tj_trvideo class=mnav>视频</a> <a href=http://tieba.baidu.com name=tj_trtieba class=mnav>贴吧</a> <noscript> <a href=http://www.baidu.com/bdorz/login.gif?login&tpl=mn&u=http%3A%2F%2Fwww.baidu.com%2f%3fbdorz_come%3d1 name=tj_login class=lb>登录</a> </noscript> <script>document.write('<a href="http://www.baidu.com/bdorz/login.gif?login&tpl=mn&u='+ encodeURIComponent(window.location.href+ (window.location.search === "" ? "?" : "&")+ "bdorz_come=1")+ '" name="tj_login" class="lb">登录</a>');</script> <a href=//www.baidu.com/more/ name=tj_briicon class=bri style="display: block;">更多产品</a> </div> </div> </div> <div id=ftCon> <div id=ftConw> <p id=lh> <a href=http://home.baidu.com>关于百度</a> <a href=http://ir.baidu.com>About Baidu</a> </p> <p id=cp>©2017 Baidu <a href=http://www.baidu.com/duty/>使用百度前必读</a> <a href=http://jianyi.baidu.com/ class=cp-feedback>意见反馈</a> 京ICP证030173号 <img src=//www.baidu.com/img/gs.gif> </p> </div> </div> </div> </body> </html>
* Connection #0 to host www.baidu.com left intact
# curl -m 2 -I http://www.baidu.com
HTTP/1.1 200 OK
Accept-Ranges: bytes
Cache-Control: private, no-cache, no-store, proxy-revalidate, no-transform
Connection: keep-alive
Content-Length: 277
Content-Type: text/html
Date: Tue, 01 Nov 2022 14:47:37 GMT
Etag: "575e1f59-115"
Last-Modified: Mon, 13 Jun 2016 02:50:01 GMT
Pragma: no-cache
Server: bfe/1.0.8.18

探测http接口有以下几个方向:

  • curl的返回值可判断接口是否假死或者不通,检测不出异常的响应,假如服务器响应5开头的状态码。

  • 根据响应状态码来监控。

  • 根据响应的内容监控。

所有url写入配置文件:

1
2
3
4
5
6
# vim /tmp/urls
www.baidu.com
www.baidu.com:82
www.baidu.com:81
127.0.0.1
127.0.0.1:81

批量探测http接口:

1
2
3
4
5
6
7
8
9
# vim /data/shell/urls
for one_url in $(cat /tmp/urls); do
curl -s -m 2 $one_url > /dev/null
if [ $? -eq 0 ];then
echo "$one_url is normal"
else
echo "$one_url is unnormal"
fi
done
1
2
3
4
5
6
# bash /data/shell/urls
www.baidu.com is normal
www.baidu.com:82 is unnormal
www.baidu.com:81 is unnormal
127.0.0.1 is normal
127.0.0.1:81 is unnormal

shell多进程简单实现

多线程fifo的那个理解起来会比较难。上万个http接口探测的话很慢。

node1

以sleep代替响应慢的情况。

1
2
3
4
5
# vim /data/shell/test.sh
for number in $(seq 6); do
echo $number
sleep 2
done
1
2
3
4
5
6
7
# bash /data/shell/test.sh
1
2
3
4
5
6

多进程的方式&,使用wait等待所有进程运行结束:

1
2
3
4
5
6
7
8
# vim /data/shell/test.sh
for number in $(seq 6); do
(
echo $number
sleep 2
) &
done
wait
1
2
3
4
5
6
7
# bash /data/shell/test.sh
4
3
5
1
6
2

进程数不控制的话,容易导致系统崩溃:

1
2
3
4
5
6
7
8
# vim /data/shell/test.sh
for number in $(seq -w 100); do
(
echo $number
sleep 10
) &
done
wait
1
# bash /data/shell/test.sh

支持控制进程数量,并发进程10个:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
# vim /data/shell/test.sh
i=0
for number in $(seq 33); do
let i=$i+1
let j=$i%10
(
echo $number
sleep 5
) &
if [ $j -eq 0 ];then
wait
fi
done
wait
1
bash /data/shell/test.sh

多进程探测http接口,并发数12:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
# vim /tmp/urls
www.baidu.com
www.baidu.com:82
www.baidu.com:81
127.0.0.1
127.0.0.1:81
www.baidu.com
www.baidu.com
www.baidu.com
www.baidu.com:82
www.baidu.com:81
127.0.0.1
127.0.0.1:81
www.baidu.com:82
www.baidu.com:81
127.0.0.1
127.0.0.1:81
www.baidu.com:82
www.baidu.com:81
127.0.0.1
127.0.0.1:81
www.baidu.com
www.baidu.com:82
www.baidu.com:81
127.0.0.1
127.0.0.1:81
www.baidu.com
www.baidu.com:82
www.baidu.com:81
127.0.0.1
127.0.0.1:81
www.baidu.com
www.baidu.com
www.baidu.com
www.baidu.com:82
www.baidu.com:81
127.0.0.1
127.0.0.1:81
www.baidu.com:82
www.baidu.com:81
127.0.0.1
127.0.0.1:81
www.baidu.com:82
www.baidu.com:81
127.0.0.1
127.0.0.1:81
www.baidu.com
www.baidu.com:82
www.baidu.com:81
127.0.0.1
127.0.0.1:81
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
# vim /data/shell/test.sh
i=0
for one_url in $(cat /tmp/urls); do
let i=$i+1
let j=$i%12
(
curl -m 2 -s $one_url > /dev/null
if [ $? -eq 0 ];then
echo "$one_url is normal"
else
echo "$one_url is unnormal"
fi
) &
if [ $j -eq 0 ];then
wait
fi
done
wait
1
# bash /data/shell/test.sh