nginx可以配置访问日志，如果我们要对日志文件进行统计分析，在linux环境下可以借助awk命令完成。
日志格式配置如下所示：
log_format access_json '{"@timestamp":"$time_iso8601",''"host":"$server_addr",''"clientip":"$remote_addr",''"size":$body_bytes_sent,''"responsetime":$request_time,''"upstreamtime":"$upstream_response_time",''"upstreamhost":"$upstream_addr",''"login_user":"$cookie_fine_login_users",''"http_host":"$host",''"url":"$uri",''"request":"$request",''"method":"$request_method",''"domain":"$host",''"xff":"$http_x_forwarded_for",''"referer":"$http_referer",''"user_agent":"$http_user_agent",''"status":"$status"}';access_log /usr/local/openresty/nginx/logs/access.log access_json;
访问日志示例
{"@timestamp":"2024-05-21T19:11:31+08:00","host":"10.1.1.1","clientip":"10.2.6.1","size":10600,"responsetime":0.523,
"upstreamtime":"0.522","upstreamhost":"10.1.11.11:8080","login_user":"-","http_host":"abc.com.cn",
"url":"/irj/servlet/prt/portal/prtroot/com.sap.ip.bi.web.portal.integration.launcher",
"request":"GET /irj/servlet/prt/portal/prtroot/com.sap.ip.bi.web.portal.integration.launcher?BI_COMMAND-BI_COMMAND_TYPE=UPDATE&BI_COMMAND-TARGET_DATA_PROVIDER_REF=DP_1&BI_COMMAND-TARGET_DIALOG_REF=SELECTOR&PAGE_ID=1_T0k1OUFpT0VyQ1d3WUVVa0dfa0g5QS0tWWNJejk2TmU1ZU03SzNiR2NqNnFkUS0t&REQUEST_ID=195 HTTP/1.1",
"method":"GET","domain":"abc.com.cn","xff":"-","referer":"http://abc.com.cn/irj/servlet/prt/portal/prtroot/com.sap.ip.bi.web.portal.integration.launcher?BI_COMMAND-BI_COMMAND_TYPE=UPDATE&PAGE_ID=1_T0k1OUFpT0VyQ1d3WUVVa0dfa0g5QS0tWWNJejk2TmU1ZU03SzNiR2NqNnFkUS0t&REQUEST_ID=7",
"user_agent":"Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/125.0.0.0 Safari/537.36 Edg/125.0.0.0","status":"200"}{"@timestamp":"2024-05-21T19:11:37+08:00","host":"10.1.1.1","clientip":"10.3.9.1","size":405,"responsetime":0.035,
"upstreamtime":"0.034","upstreamhost":"10.1.11.11:8080","login_user":"-","http_host":"abc.com.cn",
"url":"/irj/servlet/prt/portal/prtroot/com.sap.ip.bi.web.portal.integration.launcher","request":"POST /irj/servlet/prt/portal/prtroot/com.sap.ip.bi.web.portal.integration.launcher HTTP/1.1",
"method":"POST","domain":"abc.com.cn","xff":"-","referer":"http://abc.com.cn/irj/servlet/prt/portal/prtroot/com.sap.ip.bi.web.portal.integration.launcher?BI_COMMAND-BI_COMMAND_TYPE=UPDATE&PAGE_ID=2_a3JxVW1LM09oWWNSOTVnWGpOMmRWdy0tKnZzbDZkQXV4NVdvVFlIQ0dvSXFYdy0t&REQUEST_ID=1",
"user_agent":"Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/125.0.0.0 Safari/537.36 Edg/125.0.0.0","status":"200"}
1.统计调用uri次数最多的前20个
awk -F',"' '{s[$10]+=1}END{for(i in s){print substr(i,7),s[i]}}'
access.log | sort -r -n -k2 | head -n 202.统计size 大小最大的前20个
awk -F',"' '{s[substr($4,7)]+=1}END{for(i in s){print i,s[i]}}'
access.log | sort -r -n -k1 | head -n 203.统计特定url size大小最大的前n个
根据1中的命令获取的目标uri，比如：/irj/servlet/prt/portal/prtroot/com.sap.ip.bi.web.portal.integration.launcher
将指定uri内容导出到临时文件t1.txt
grep '"url":"/irj/servlet/prt/portal/prtroot/com.sap.ip.bi.web.portal.integration.launcher' access.log > t1.txt统计size>1000的调用次数之和
awk -F',"' '{s[substr($4,7)]+=1}END{for(i in s){if((i+0)>=1000)print i,s[i]}}' t1.txt | sort -n -r -k1 | awk '{sum+=$2}END{print sum}'
