原文：

blog.csdn.net/weixin_43291944/article/details/100180464

                                      

1. grep

                                                                                   

grep-global regular expression print - 全局正则表达式打印

可用于数据查找定位

 

curl -s https://testerhome.com | grep href

curl -s https://testerhome.com | grep href | grep -o "http[^\"]*"

curl -s -I https://testerhome.com/topics/feed

curl -s https://testerhome.com | grep href | grep -o "http[^\"]*" | while read line;do curl -s -I $line | grep 200 && echo $line || echo ERR $line;done

 

4. 最终结果展示

 

2. awk   

                                           

 

awk的字段数据处理

 

• -F 参数指定字段分隔符

• BEGIN{FS=‘_’} 也可以表示分隔符

 

$0 代表原来的行$1 代表第一个字段$N 代表第N个字段$NF 代表最后一个字段

 

下面以一个在nginx.log中查找返回状态码非200的请求响应数目的需求为例，演示awk的基础用法

 

有一份nginx.log文件，打开后内容格式如下：

220.181.108.111 - - [05/Dec/2018:00:11:42 +0000] "GET /topics/15225/show_wechat HTTP/1.1" 200 1684 "-" "Mozilla/5.0 (compatible; Baiduspider/2.0; +http://www.baidu.com/search/spider.html)" 0.029 0.029 .216.244.66.241 - - [05/Dec/2018:00:11:42 +0000] "GET /topics/10052/replies/85845/reply_suggest HTTP/1.1" 301 5 "-" "Mozilla/5.0 (compatible; DotBot/1.1; http://www.opensiteexplorer.org/dotbot, help@moz.com)" 0.016 0.016 .216.244.66.241 - - [05/Dec/2018:00:11:42 +0000] "GET /topics/10040?order_by=created_at HTTP/1.1" 301 5 "-" "Mozilla/5.0 (compatible; DotBot/1.1; http://www.opensiteexplorer.org/dotbot, help@moz.com)" 0.002 0.002 .216.244.66.241 - - [05/Dec/2018:00:11:42 +0000] "GET /topics/10043/replies/85544/reply_suggest HTTP/1.1" 301 5 "-" "Mozilla/5.0 (compatible; DotBot/1.1; http://www.opensiteexplorer.org/dotbot, help@moz.com)" 0.001 0.001 .216.244.66.241 - - [05/Dec/2018:00:11:44 +0000] "GET /topics/10075/replies/89029/edit HTTP/1.1" 301 5 "-" "Mozilla/5.0 (compatible; DotBot/1.1; http://www.opensiteexplorer.org/dotbot, help@moz.com)" 0.001 0.001 .216.244.66.241 - - [05/Dec/2018:00:11:44 +0000] "GET /topics/10075/replies/89631/edit HTTP/1.1" 301 5 "-" "Mozilla/5.0 (compatible; DotBot/1.1; http://www.opensiteexplorer.org/dotbot, help@moz.com)" 0.001 0.001 .216.244.66.241 - - [05/Dec/2018:00:11:45 +0000] "GET /topics/10075?order_by=created_at HTTP/1.1" 301 5 "-" "Mozilla/5.0 (compatible; DotBot/1.1; http://www.opensiteexplorer.org/dotbot, help@moz.com)" 0.000 0.000 .216.244.66.241 - - [05/Dec/2018:00:11:45 +0000] "GET /topics/10075?order_by=like HTTP/1.1" 301 5 "-" "Mozilla/5.0 (compatible; DotBot/1.1; http://www.opensiteexplorer.org/dotbot, help@moz.com)" 0.001 0.001 .223.71.41.98 - - [05/Dec/2018:00:11:46 +0000] "GET /cable HTTP/1.1" 101 60749 "-" "Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:63.0) Gecko/20100101 Firefox/63.0" 2608.898 2608.898 .113.87.161.17 - - [05/Dec/2018:00:11:39 +0000] "GET /cable HTTP/1.1" 101 3038 "-" "Mozilla/5.0 (Windows NT 6.1; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/67.0.3396.62 Safari/537.36" 112.418 112.418 .216.244.66.241 - - [05/Dec/2018:00:11:46 +0000] "GET /topics/10079/replies/119591/edit HTTP/1.1" 301 5 "-" "Mozilla/5.0 (compatible; DotBot/1.1; http://www.opensiteexplorer.org/dotbot, help@moz.com)" 0.001 0.001 .216.244.66.241 - - [05/Dec/2018:00:11:46 +0000] "GET /topics/10089?locale=zh-TW HTTP/1.1" 301 5 "-" "Mozilla/5.0 (compatible; DotBot/1.1; http://www.opensiteexplorer.org/dotbot, help@moz.com)" 0.002 0.002 .

 

观察log内容，可以发现，以空格为分隔符，状态码在第九个字段位置；这里我们用awk命令从第九个字段位置开始匹配非200的状态码并打印出来。命令：

 

awk '$9!~/200/{print $9}' nginx.log

[avbxb9efockpz ~]$ awk '$9!~/200/{print $9}' nginx.log301301301301301301301301301......#剩余部分省略

 

再对取出的数据进行排序->去重->按数字的倒叙进行排列。命令：

 

awk '$9!~/200/{print $9}' nginx.log | sort | uniq -c | sort -nr

 

命令含义：

sort: 按从小到大进行排序uniq -c :去重（相邻）-nr: 按数字进行倒叙排序-n:按数字进行排序

结果展示：

[sqavbxb9efockpz ~]$ awk '$9!~/200/{print $9}' nginx.log | sort | uniq -c | sort -nr    433 101    304 301    266 404    152 302      7 401      5 304      2 499      2 422      1 500

1 nobody2 root3 daemon4 _uucp5 _taskgated6 _networkd7 _installassistant8 _lp9 _postfix......

用户信息：

 

cat /etc/passwd | awk -F ':' 'BEGINE{userindex=0}{user[userindex]=$1;userindex++}END{for(i=0;i<NR;i++)print i+1, user[i+10]}' |less

3. sed                                                                                  

 

sed:stream editor   根据定位到的数据行修改数据

 

[16210504@izuf60jasqavbxb9efockpz ~]$ echo "aaa|bbb}|cccbbb" | sed 's/bbb/BBB/'aaa|BBB}|cccbbb[16210504@izuf60jasqavbxb9efockpz ~]$ echo "aaa|bbb}|cccbbb" | sed 's#bbb#BBB#'aaa|BBB}|cccbbb

 

• 若想讲目标中所有的字段都替换，需要在命令最后加上g:

 

[16210504@izuf60jasqavbxb9efockpz ~]$ echo "aaa|bbb}|cccbbb" | sed 's/bbb/BBB/g'aaa|BBB}|cccBBB

• sed还可以修改文件中的内容，现在有文件text.txt,内容如下：

   

[16210504@izuf60jasqavbxb9efockpz ~]$ cat text.txthello bash worldhi~ testergo go go go!

 

用sed 's/hello/HELLO/' text.txt 命令将文件中的hello替换成HELLO：

 

[16210504@izuf60jasqavbxb9efockpz ~]$ sed 's/hello/HELLO/' text.txtHELLO bash worldhi~ testergo go go go!

 

但是此时我们打开源text.txt文件发下源文件内容并未改变：

 

[16210504@izuf60jasqavbxb9efockpz ~]$ cat text.txthello bash worldhi~ testergo go go go!

[16210504@izuf60jasqavbxb9efockpz ~]$ sed -i.bak 's/hello/HELLO/' text.txt[16210504@izuf60jasqavbxb9efockpz ~]$ ls1  1.sh  Allen_qin  nginx.log  test  text.txt  text.txt.bak  while_test[16210504@izuf60jasqavbxb9efockpz ~]$ cat text.txtHELLO bash worldhi~ testergo go go go![16210504@izuf60jasqavbxb9efockpz ~]$ cat text.txt.bakhello bash worldhi~ testergo go go go!

a:b:c:d:

要将其中每行末尾的：都替换成@，将a替换成A，并在文本末尾加上“Sed Test”
命令：

sed -i -e 's/:/@/g' \-i -e 's/a/A/' \-i -e '$a Sed Test' 1.txt

 

实例演示：

 

[16210504@izuf60jasqavbxb9efockpz ~]$ sed -i -e 's/:/@/g' -i -e 's/a/A/' -i -e '$a Sed Test' 1.txt[16210504@izuf60jasqavbxb9efockpz ~]$ cat 1.txtA@b@c@d@Sed Test

如喜欢本文，请点击右上角，把文章分享到朋友圈

因公众号更改推送规则，请点“在看”并加“星标”第一时间获取精彩技术分享

·END·