业务服务器免装插件,使用rsync+nxlog同步+采集应用日志并接入到GrayLog5.1
AppServer业务服务器上的应用日志需要接入到GrayLog,进行字段分析及业务日志异常告警
但是AppServer业务服务器为重要生产环境,不建议安装filebeat和nxlog等其他日志采集的Agent插件程序
所以考虑将AppServer业务服务器上的应用日志实时同步到某个中间服务器上,然后再发往GrayLog
所以有了如下拓扑图的这种方案
(图片点击放大查看)
利用rsync将AppServer业务服务器上应用日志同步到群晖NAS上,然后NAS利用NFS或者其他协议挂载到GrayLog服务器上,在GrayLog服务器上面安装nxlog读取NFS挂载目录下的日志文件,即可实现日志采集读取
主要环境说明
具体步骤如下
ssh-keygen -t rsa -b 4096 -C
"appserver"
cd
/root/.ssh/
cp id_rsa id_rsa_appserver.pem
cat /etc/ssh/sshd_config | grep authorized_keys
cat id_rsa.pub >> /root/.ssh/authorized_keys
sz /root/.ssh/id_rsa_appserver.pem
(图片点击放大查看)
(图片点击放大查看)
假设AppServer业务服务器的应用日志为/var/log/nginx/*.log
选择通过SSH连接至rsync shell模式,然后
认证策略:通过SSH密钥验证
备份模式:镜像
备份到/LogSpace NAS的本地路径
(图片点击放大查看)
(图片点击放大查看)
(图片点击放大查看)
(图片点击放大查看)
(图片点击放大查看)
运行频率为每小时一次
(图片点击放大查看)
(图片点击放大查看)
(图片点击放大查看)
(图片点击放大查看)
yum install nfs-utils
showmount -e 192.168.31.140
mkdir /log_collector
mount -t nfs 192.168.31.140:/volume1/LogSpace /log_collector/ -o proto=tcp -o nolock
echo
"mount -t nfs 192.168.31.140:/volume1/LogSpace /log_collector/ -o proto=tcp -o nolock"
>> /etc/rc.local
chmod a+x /etc/rc.d/rc.local
(图片点击放大查看)
下面就可以在GrayLog安装nxlog去读取采集/log_collector/下的日志了
/log_collector/var/
log
/nginx/*.
log
上面的同步方式可针对实时性要求不高的场景 由于群晖NAS上ABB(active backup for business)rsync备份只能每小时同步一次
业务日志的实时性要求比较强,所以决定直接rsync+crontab定时任务这种方式来实时同步日志,再nxlog进行读取,如下图所示
(图片点击放大查看)
mkdir /root/.ssh
cd
/root/.ssh/
上传id_rsa_appserver.pem
mkdir /LogSpace
rsync --progress -avz --perms --chmod=ugo+r -og --chown=root:root -e
"ssh -p 22 -i /root/.ssh/id_rsa_appserver.pem"
root@192.168.31.230:/var/
log
/nginx/ /LogSpace/
先确认rsync同步是否OK
报错:Permissions 0644
for
'/root/.ssh/id_rsa_appserver.pem'
are too open.
解决办法:chmod 0600 /root/.ssh/id_rsa_appserver.pem
(图片点击放大查看)
cat /opt/logfiles_rsync.sh
#!/bin/bash
LOCK=/var/
log
/appserverlogs_rsync_record.log
echo
"同步日期:"
>>
${LOCK}
2>&1
echo
`date
'+%Y-%m-%d_%T'
` >>
${LOCK}
2>&1
echo
"================= AppServer logs Rsync starting==============================="
>>
${LOCK}
2>&1
rsync --progress -avz --perms --chmod=ugo+r -og --chown=root:root -e
"ssh -p 22 -i /root/.ssh/id_rsa_appserver.pem"
root@192.168.31.230:/var/
log
/nginx/ /LogSpace/ >>
${LOCK}
2>&1
echo
"================= AppServer logs Rsync Finished==============================="
>>
${LOCK}
2>&1
#chmod -R 755 /LogSpace/ >> ${LOCK} 2>&1
#chown -R root:root /LogSpace/ >> ${LOCK} 2>&1
(图片点击放大查看)
crontab -e
添加每1分钟定时同步应用日志到本地
*/59 * * * * /usr/sbin/ntpdate ntp.aliyun.com >> /tmp/ntp_sync.log
*/1 * * * * /opt/logfiles_rsync.sh
(图片点击放大查看)
然后记得权限
chmod 777 /opt/logfiles_rsync.sh
yum localinstall nxlog-ce*.rpm
vim /etc/nxlog/nxlog.conf
(图片点击放大查看)
(图片点击放大查看)
修改如下几处
1)、运行用户改为root
User root
Group root
2)、Input Output 和Route配置
<Extension _syslog>
Module xm_syslog
</Extension>
<Extension _json>
Module xm_json
</Extension>
<Input 192.168.31.230_appserver_log>
Module im_file
File
"/LogSpace/*.log"
SavePos TRUE
ReadFromLast TRUE
<Exec>
$Hostname
= file_name();
$Message
=
$raw_event
;
</Exec>
</Input>
<Output toGraylog>
Module om_udp
Host 192.168.31.170
Port 2514
Exec to_syslog_bsd();
Exec parse_syslog(); to_json();
</Output>
########################################
# Routes #
########################################
<Route udproute>
Path 192.168.31.230_appserver_log => toGraylog
</Route>
systemctl
enable
nxlog
systemctl start nxlog
systemctl status nxlog
(图片点击放大查看)
排错命令
journalctl -xe -u nxlog
tail -f /var/
log
/nxlog/nxlog.log
json日志解析,可以获取到日志的实际路径文件名字段
(图片点击放大查看)
(图片点击放大查看)
步骤省略,最后的效果
(图片点击放大查看)
https://docs.nxlog.co/userguide/integrate/graylog.html
参考如下链接
https://www.mongodb.com/docs/manual/tutorial/configure-scram-client-authentication/
use admin
db.createUser({user:
"admin"
,
pwd
:
"Admin@2023!"
, roles: [
"root"
]})
db.auth(
"admin"
,
"Admin@2023!"
)
#创建graylog数据库并设置密码
use graylog
db.createUser({
user:
"graylog"
,
pwd
:
"Graylog2023!"
,
"roles"
: [{
"role"
:
"dbOwner"
,
"db"
:
"graylog"
}, {
"role"
:
"readWrite"
,
"db"
:
"graylog"
}]
})
(图片点击放大查看)
vim /etc/mongod.conf
修改如下行
security:
authorization: enabled
(图片点击放大查看)
vim /etc/graylog/server/server.conf
连接配置信息修改成
mongodb_uri = mongodb://graylog:Graylog2023!@localhost/graylog
systemctl restart mongod
systemctl restart graylog-server.service
(图片点击放大查看)
(图片点击放大查看)
mongosh --port 27017 --authenticationDatabase
"admin"
-u
"admin"
-p
mongosh --port 27017 --authenticationDatabase
"graylog"
-u
"graylog"
-p