一、报错分析
原始报错信息：
JSON parse error, original data now in message field {:message=>"Unrecognized character escape 'x' (code 120)\n at [Source: (String)\"{\"@timestamp\":\"2022-06-29T10:02:00+08:00\",\"@source\":\"172.1.2.13\",\"hostname\":\"nb001\",\"ip\":\"121.8.185.98\",\"xff\":\"-\",\"client\":\"121.8.185.98\",\"remote_user\":\"-\",\"request_method\":\"POST\",\"request_body\": \"{\\x22equipmentId\\x22:1501805378853269505}\",\"scheme\":\"https\",\"domain\":\"gw.xxx.com\",\"referer\":\"-\",\"request\":\"/api/equipment/findWaitPlan\",\"args\":\"-\",\"size\":69,\"status\": 200,\"responsetime\":0.075,\"upstreamtime\":\"0.076\",\"upstreamaddr\":\"172.1.2.12:31149\",\"upstream_stat\"[truncated 205 chars]; line: 1, column: 204]", :exception=>LogStash::Json::ParserError, :data=>"{\"@timestamp\":\"2022-06-29T10:02:00+08:00\",\"@source\":\"172.1.2.13\",\"hostname\":\"nb001\",\"ip\":\"121.8.185.98\",\"xff\":\"-\",\"client\":\"121.8.185.98\",\"remote_user\":\"-\",\"request_method\":\"POST\",\"request_body\": \"{\\x22equipmentId\\x22:1501805378853269505}\",\"scheme\":\"https\",\"domain\":\"gw.xxx.com\",\"referer\":\"-\",\"request\":\"/api/equipment/findWaitPlan\",\"args\":\"-\",\"size\":69,\"status\": 200,\"responsetime\":0.075,\"upstreamtime\":\"0.076\",\"upstreamaddr\":\"172.1.2.12:31149\",\"upstream_status\": \"200\",\"http_user_agent\":\"Apache-HttpClient/4.5.6 (Java/1.8.0_131)\",\"http_host\":\"gw.xxx.com\",\"url\":\"/api/equipment/findWaitPlan\",\"connection_requests\": 4,\"https\":\"on\"}"}上述有说解析的nginx日志的第一行204个字符（"[truncated 205 chars]; line: 1, column: 204]",）无法解析：
发现就是json中的\"request_body\": \"{\\x22equipmentId\\x22:1501805378853269505}\" request_body的只在转义的时候变成了\\x22。
二、解决办法
在生成json格式的nginx日志上明确指定escape=json
log_format json escape=json '{"@timestamp":"$time_iso8601",''"@source":"$server_addr",''"hostname":"$hostname",''"ip":"$remote_addr",''"xff":"$http_x_forwarded_for",''"client":"$remote_addr",''"remote_user":"$remote_user",''"request_method":"$request_method",''"request_body": "$request_body",''"scheme":"$scheme",''"domain":"$server_name",''"referer":"$http_referer",''"request":"$request_uri",''"args":"$args",''"size":$body_bytes_sent,''"status": $status,''"responsetime":$request_time,''"upstreamtime":"$upstream_response_time",''"upstreamaddr":"$upstream_addr",''"upstream_status": "$upstream_status",''"http_user_agent":"$http_user_agent",''"http_host":"$host",''"url":"$uri",''"connection_requests": $connection_requests,''"https":"$https"''}';三、改后效果
再次查看结果：
发现logstash已经解析成功了：
{"args" => "","upstreamaddr" => "172.1.2.12:31149","connection_requests" => 1,"request_method" => "POST","upstream_status" => "200","log" => {"offset" => 349425307,"file" => {"path" => "/usr/local/nginx/logs/access.log"}},"tags" => [[0] "beats_input_codec_json_applied",[1] "_grokparsefailure"],"remote_user" => "","container" => {"id" => "access.log"},"xff" => "","agent" => {"version" => "7.17.4","ephemeral_id" => "4e4e1f17-99bc-4ba1-b431-df953c9594d8","type" => "filebeat","id" => "3f299217-a27a-4fb0-ac7d-f80c561d6f2a","name" => "nb001","hostname" => "nb001"},"@version" => "1","client" => "218.85.130.106","http_user_agent" => "","geoip" => {"city_name" => "Xiamen","country_name" => "China","location" => {"lat" => 24.4798,"lon" => 118.0819},"region_name" => "Fujian","coordinates" => [[0] 118.0819,[1] 24.4798],"ip" => "218.85.130.106"},"@timestamp" => 2022-06-29T10:36:02.940Z,"https" => "on","status" => 200,"referer" => "","@source" => "172.1.2.13","request_body" => "{\"experimentId\":\"d03ac7b7548a49828d207431a48ef916\",\"subExperimentId\":\"f95dd19af6e34904b32ef3cc76124ba4\",\"resourceId\":\"1413404267903803394\",\"orgId\":\"616d1ccee4b0ab7641c47a2e\",\"userId\":\"9ea996f86eb3459688390789bc151974\"}","request" => "/api/open/platform/heartbeats","size" => 112,"responsetime" => 0.068,"upstreamtime" => "0.067","url" => "/api/open/platform/heartbeats","ecs" => {"version" => "1.12.0"},"scheme" => "https","domain" => "gw.xxx.com","http_host" => "gw.xxx.com","ip" => "218.85.130.106","hostname" => "nb001","input" => {"type" => "filestream"}
}
