使用ELK收集nginx access日志,利用Grafana做出一套可视化图表
操作系统:centos 7.6
docker版本:19.03.12
ip地址:192.168.31.196
关于elk的搭建,请参考以下3篇文章:
docker安装elasticsearch和head插件
docker安装logstash
docker安装kibana
线上nginx直接用yum安装的
yum install -y nginx
默认的nginx日志格式,需要改为指定的格式
http {
log_format main ' $remote_addr - $remote_user [$time_local] "$request" '
' $status $body_bytes_sent "$http_referer" '
' "$http_user_agent" "$http_x_forwarded_for" ' ;
log_format aka_logs
'{"@timestamp":"$time_iso8601",'
'"host":"$hostname",'
'"server_ip":"$server_addr",'
'"client_ip":"$remote_addr",'
'"xff":"$http_x_forwarded_for",'
'"domain":"$host",'
'"url":"$uri",'
'"referer":"$http_referer",'
'"args":"$args",'
'"upstreamtime":"$upstream_response_time",'
'"responsetime":"$request_time",'
'"request_method":"$request_method",'
'"status":"$status",'
'"size":"$body_bytes_sent",'
'"request_body":"$request_body",'
'"request_length":"$request_length",'
'"protocol":"$server_protocol",'
'"upstreamhost":"$upstream_addr",'
'"file_dir":"$request_filename",'
'"http_user_agent":"$http_user_agent"'
'}' ;
#access_log /var/log/nginx/ access.log main;
access_log /var/log/nginx/access.log aka_logs;
注意:这里的nginx的所有访问日志,统一在/var/log/nginx/access.log。
如果你的环境,单独为每一个虚拟主机分配了access日志,那么就需要在对应的虚拟主机配置文件,应用格式aka_logs即可。
注意:由于本文,直接使用logstash采集nginx日志,并发送给elasticsearch。
如果是使用filebeat收集nginx日志,请查阅参考文章中的配置说明。
新建文件nginx.conf
/data/elk7/logstash/config/conf.d/nginx.conf
内容如下:
input {
file {
## 修改你环境nginx日志路径
path => " /var/log/nginx/access.log "
ignore_older => 0
codec => json
}
}
filter {
geoip {
#multiLang => " zh-CN "
target => " geoip "
source => " client_ip "
database => "/usr/share/logstash/GeoLite2-City.mmdb"
add_field => [ " [geoip][coordinates] " , " %{[geoip][longitude]} " ]
add_field => [ " [geoip][coordinates] " , " %{[geoip][latitude]} " ]
# 去掉显示 geoip 显示的多余信息
remove_field => [ " [geoip][latitude] " , " [geoip][longitude] " , " [geoip][country_code] " , " [geoip][country_code2] " , " [geoip][country_code3] " , " [geoip][timezone] " , " [geoip][continent_code] " , " [geoip][region_code] " ]
}
mutate {
convert => [ " size " , " integer " ]
convert => [ " status " , " integer " ]
convert => [ " responsetime " , " float " ]
convert => [ " upstreamtime " , " float " ]
convert => [ " [geoip][coordinates] " , " float " ]
# 过滤 filebeat 没用的字段,这里过滤的字段要考虑好输出到es的,否则过滤了就没法做判断
remove_field => [ " ecs " , " agent " , " host " , " cloud " , " @version " , " input " , " logs_type " ]
}
# 根据http_user_agent来自动处理区分用户客户端系统与版本
useragent {
source => " http_user_agent "
target => " ua "
# 过滤useragent没用的字段
remove_field => [ " [ua][minor] " , " [ua][major] " , " [ua][build] " , " [ua][patch] " , " [ua][os_minor] " , " [ua][os_major] " ]
}
}
output {
elasticsearch {
hosts => "192.168.31.196"
#user => " elastic "
#password => " password "
index => " logstash-nginx-%{+YYYY.MM.dd} "
}
}
注意2个位置:
/usr/share/logstash/GeoLite2-City.mmdb,这个是地图数据文件
elasticsearch {xx} 这里填写elasticsearch信息,由于我的elasticsearch 没有开启认证,因此不需要用户名和密码,请根据实际情况填写。
GeoLite2-City.mmdb是IP信息解析和地理定位的。官方下载是需要收费的,因此我找了一个免费下载地址。
下载地址1:https://pan.baidu.com/s/1sjtdvPV
备用下载地址:
链接:https://pan.baidu.com/s/1eJkDqq2nvV3vETyUfOBypw
提取码:2jq5
下载完成后,将此文件,上传到/data/elk7/logstash目录,待会启动logstash,就会挂载进去。
以新的方式启动logstash
docker rm - f logstash
docker run - d \
--name= logstash \
--restart= always \
-p 5044 : 5044 \
-v /data/elk7/logstash:/usr/share/ logstash \
-v /var/log/messages:/var/log/ messages \
-v /var/log/nginx:/var/log/ nginx \
logstash: 7.5 . 1
等待30秒,查看logstash日志是否有错误
docker logs -f logstash
访问head插件,查看索引是否生成
http: // 192.168.31.196:9100/
有出现logstash-nginx 索引,说明成功了。
关于grafana的安装,请参考链接:
https://www.cnblogs.com/xiao987334176/p/9930517.html
添加elasticsearch数据源,这里输入elasticsearch的url
如果elasticsearch需要认证,在下面的Auth设置中,Basic auth开启,输入用户名和密码。
输入索引值,时间戳,选择版本:7.0+
如果测试通过,则会添加成功。
进入grafana容器,安装2个插件,用来支持展示图表的。
grafana-cli plugins install grafana-piechart- panel
grafana -cli plugins install grafana-worldmap-panel
重启grafana
docker restart grafana
模板下载地址为:https://grafana.com/grafana/dashboards/11190
下载最新版本,导入json文件,选择2个数据源
刷新页面,效果如下:
本次参考链接:https://grafana.com/grafana/dashboards/11190
转自:https://www.cnblogs.com/Lqdream/p/16918607.html