Graylog 分布式日志管理系统-集群搭建

简介

Graylog是一个开源的日志聚合、分析、审计、展现和预警工具。低成本，高性能。

Graylog与ELK对比

Graylog需要把日志源采集到graylog-server,经过处理后的数据，使用ES进行存储。
Graylog提供了Web端，相当于Kibana。

集群架构分配参考

官方文档架构图：https://docs.graylog.org/en/3.3/pages/architecture.html

部署节点

IP	作用
192.168.122.71	Graylog、MongoDB，Nginx
192.168.122.72	Graylog、MongoDB
192.168.122.73	Graylog、MongoDB
192.168.122.74	Elasticsearch
192.168.122.75	Elasticsearch
192.168.122.76	Elasticsearch

前置工作

查看防火墙已关闭

sudo systemctl status firewalld

…

Active: inactive (dead)

…

查看SELinux已关闭

cat /etc/selinux/config

…

SELINUX=disabled

…

查看jdk版本

java -version

openjdk version “1.8.0_161”

…

71,72,73 搭建 Graylog+MongoDB 集群同步操作

安装epel-release (epel源)、pwgen

sudo yum install epel-release pwgen -y

RedHat系统安装Epel报错以及解决方案

搭建MongoDB集群

MongoDB的用途是什么？

Graylog使用MongoDB来存储您的配置数据，而不是您的日志数据。仅存储元数据，例如用户信息或流配置。您的任何日志消息都不会存储在MongoDB中。这就是为什么MongoDB对系统没有太大影响的原因，并且您不必为扩展它而过分担心。通过我们推荐的设置架构，MongoDB可以与您的Graylog服务器进程一起运行，并且几乎不使用任何资源。

sudo touch /etc/yum.repos.d/mongodb-org.repo 
sudo vim /etc/yum.repos.d/mongodb-org.repo

添加内容:

[mongodb-org-4.0] 
name=MongoDB Repository baseurl=https://repo.mongodb.org/yum/redhat/$releasever/mongodb-org/4.0/x86_64/ 
gpgcheck=1 
enabled=1 
gpgkey=https://www.mongodb.org/static/pgp/server-4.0.asc

安装MongoDB

sudo yum install -y mongodb-org

启动MongoDB

sudo mongod --config /etc/mongod.conf

关闭MongoDB

mongo
use admin 
db.shutdownServer()

创建文件

mkdir -p /work/mongo/data 
mkdir -p /work/mongo/log 
touch /work/mongo/log/mongod.log

备份并编辑配置

sudo cp /etc/mongod.conf /etc/mongod.conf.bak 
sudo vim /etc/mongod.conf

编辑 mongod.conf 配置

systemLog:
  destination: file
  logAppend: true
  path: /work/mongo/log/mongod.log
  #path: /var/log/mongodb/mongod.log

storage:
  dbPath: /work/mongo/data
  #dbPath: /var/lib/mongo
  journal:
    enabled: true


#配置远程连接
net:
  port: 27017
  bindIp: 0.0.0.0  # Enter 0.0.0.0,:: to bind to all IPv4 and IPv6 addresses or, alternatively, use the net.bindIpAll setting.

#replication（集群配置）:
replication:
  replSetName: rs0

mongoDB重启，参考上面先关闭mongoDB，再启动。

初始化mongodb集群（在其中一台操作）

mongo
>rs.initiate( {
   _id : "rs0",
   members: [ { _id : 0, host : "192.168.122.71:27017" } ]
})
exit

验证集群配置

mongo
rs0:PRIMARY> rs.conf()

将其他节点加入集群，并查看集群配置

rs0:PRIMARY>  rs.add("192.168.122.72")
rs0:PRIMARY>  rs.add("192.168.122.73")
rs0:PRIMARY> rs.status()

创建graylog数据库，并添加graylog用户，赋予readWrite和dbAdmin权限

rs0:PRIMARY> use graylog
switched to db graylog
rs0:PRIMARY> db.createUser( {
    user: "graylog",
     pwd: "xxxxx",
     roles: [ { role: "readWrite", db: "graylog" } ]
   });
rs0:PRIMARY> db.grantRolesToUser( "graylog" , [ { role: "dbAdmin", db: "graylog" } ])
rs0:PRIMARY> show users
rs0:PRIMARY> db.auth("graylog","xxxxx")

Graylog 的ES

所有日志数据都存储在Elasticsearch中。

ES 设置索引过期

https://docs.graylog.org/en/3.0/pages/configuration/index_model.html

Elasticsearch集群部署

graylog3.0 使用的elasticsearch不低于5.6.13版（且暂不支持7.x）

主机74,75,76同步操作

编译安装软件包

cd /wrok
wget https://artifacts.elastic.co/downloads/elasticsearch/elasticsearch-6.4.3.tar.gz
tar -zxf elasticsearch-6.4.3.tar.gz

创建数据目录

mkdir -p  /work/elasticsearch-6.4.3/data

编辑ES配置

sudo vim /work/elasticsearch-6.4.3/config/elasticsearch.yml

三台机分别如下

#graylog01：
cluster.name: graylog
node.name: graylog01
node.master: true
node.data: true
bootstrap.memory_lock: false
bootstrap.system_call_filter: false
path.data: /work/elasticsearch-6.4.3/data 
path.logs: /work/elasticsearch-6.4.3/logs
network.host: 0.0.0.0
http.port: 9200
discovery.zen.ping.unicast.hosts: ["192.168.122.74:9300", "192.168.122.75:9300","192.168.122.76:9300"]


#graylog02
cluster.name: graylog
node.name: graylog02
node.master: true
node.data: true
bootstrap.memory_lock: false
bootstrap.system_call_filter: false
path.data: /work/elasticsearch-6.4.3/data 
path.logs: /work/elasticsearch-6.4.3/logs
network.host: 0.0.0.0
http.port: 9200
discovery.zen.ping.unicast.hosts: ["192.168.122.74:9300", "192.168.122.75:9300","192.168.122.76:9300"]

#graylog03
cluster.name: graylog
node.name: graylog03
node.master: true
node.data: true
bootstrap.memory_lock: false
bootstrap.system_call_filter: false
path.data: /work/elasticsearch-6.4.3/data 
path.logs: /work/elasticsearch-6.4.3/logs
network.host: 0.0.0.0
http.port: 9200
discovery.zen.ping.unicast.hosts: ["192.168.122.74:9300", "192.168.122.75:9300","192.168.122.76:9300"]

启动Elasticsearch

bin/elasticsearch -d

ES启动失败如果遇到报错

[2020-09-09T15:27:08,646][ERROR][o.e.b.Bootstrap          ] [graylog01] node validation exception
[2] bootstrap checks failed
[1]: max file descriptors [65535] for elasticsearch process is too low, increase to at least [65536]
[2]: max virtual memory areas vm.max_map_count [65530] is too low, increase to at least [262144]

[1] increase to at least [65536]的解决办法

sudo vim /etc/security/limits.conf

添加如下：

* - nofile 65536 
* - memlock unlimited
或者
* soft nofile 65536
* hard nofile 65536
* soft nproc 4096
* hard nproc 4096

同时注释掉当前用户（如有）

#user – nproc 65535

#user – nofile 65535

#* soft core 0 #* hard core 0

最后重连登陆生效

[2] increase to at least [262144]的解决办法

sudo vi /etc/sysctl.conf

添加下面配置：

vm.max_map_count=655360

并执行命令：

sudo sysctl -p

然后，重新启动elasticsearch，即可启动成功。

Graylog集群安装

参考：https://docs.graylog.org/en/3.0/pages/configuration/multinode_setup.html

主机71,72,73同步操作

sudo rpm -Uvh https://packages.graylog2.org/repo/packages/graylog-3.0-repository_latest.rpm

生成密钥，并加密Graylog的登陆密码（这里为admin）

pwgen -N 1 -s 96
M39BrdTsF7EmzLc1x0iejVoCn3QAYuvgSc5OkitRspJBmBCL2XasAK2LgW5uvok0v2QT3gM8hgaNbNTED1UOjAgCSQVPznLy

echo -n admin | sha256sum
8c6976e5b5410415bde908bd4dee15dfb167a9c873fc4bb8a81f6f2ab448a918

编辑Graylog配置文件

参考：https://docs.graylog.org/en/3.0/pages/configuration/server.conf.html?highlight=server.conf

sudo vim /etc/graylog/server/server.conf

修改如下:

#主节点is_master = true，其他两个节点的配置文件中设置为is_master = false
is_master = true

#密钥
password_secret = M39BrdTsF7EmzLc1x0iejVoCn3QAYuvgSc5OkitRspJBmBCL2XasAK2LgW5uvok0v2QT3gM8hgaNbNTED1UOjAgCSQVPznLy
#加密后的登陆密码
root_password_sha2 = 8c6976e5b5410415bde908bd4dee15dfb167a9c873fc4bb8a81f6f2ab448a918
#时区
root_timezone = Asia/Shanghai
# 网络访问相关，重要
http_bind_address = 0.0.0.0:9000
# 配置Nginx代理的外网地址
http_publish_uri = http://192.168.122.71:9100/
#高亮
allow_highlighting = true
# ES 连接配置
elasticsearch_hosts = http://192.168.122.74:9200,http://192.168.122.75:9200,http://192.168.122:9200
# mongodb 连接配置，设置验证
mongodb_uri = mongodb://graylog:graylog@192.168.122.71:27017,192.168.122.71:27017,192.168.122:27017/graylog
# 或者，不设置验证
mongodb_uri = mongodb://192.168.122.71:27017,192.168.122.71:27017,192.168.122:27017/graylog

启动服务

sudo chkconfig --add graylog-server
sudo systemctl daemon-reload
sudo systemctl enable graylog-server.service

sudo systemctl start graylog-server.service

sudo systemctl status graylog-server.service

sudo systemctl restart graylog-server.service

安装Nginx

请参考:

https://blog.csdn.net/gotheon/article/details/90409492

https://blog.csdn.net/w410589502/article/details/70787468

设置指定的安装nginx目录

./configure \
--prefix=/work/graylog-nginx

配置Graylog 负载均衡

官方文档：https://docs.graylog.org/en/3.0/pages/configuration/web_interface.html

sudo vim /work/nginx/conf/nginx.conf

再http { } 中添加配置

upstream graylog_servers {
        least_conn;
        server 192.168.122.71:9000 max_fails=3 fail_timeout=30s;
        server 192.168.122.72:9000 max_fails=3 fail_timeout=30s;
        server 192.168.122:9000 max_fails=3 fail_timeout=30s;
    }
  
   
server {
           listen       9100;
           server_name 192.168.122.71:9100;

           location / {
                   proxy_set_header Host $http_host;
                   proxy_set_header X-Forwarded-Host $host;
                   proxy_set_header X-Forwarded-Server $host;
                   proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
                   proxy_set_header X-Graylog-Server-URL http://$server_name/;
                   proxy_pass http://graylog_servers;
           }
    }

Graylog访问地址就是上面配置的

http://192.168.122.71:9100

账户admin，密码amdin

Nginx相关命令操作

在线加载配置
cd /work/nginx/sbin sudo nginx -s reload
查看版本
nginx -v
停止nginx服务
sudo nginx -s stop
检查配置文件语法是否有误
sudo nginx -t
查看nginx 安装了哪些模块
nginx -V

Graylog生成的日志文件在哪里？

您可以在下面的目录下找到Graylog的日志数据，其中包含时间戳，级别和异常消息。这对于调试或服务器无法启动很有用。

cd /var/log/graylog-server/server.log

如何使用Logstash 转发到 Graylog?

1. Graylog只能处理自己处理的消息。无法处理Elasticsearch数据库中的外来消息 (不能直接发到ES)。

2. Graylog包含了kibana和logstash的功能，执行日志处理（logstash）并提供Web UI（kibana）。

3. 使用logstash的gelf output插件:

安装插件

bin/logstash-plugin install logstash-output-gelf

Logstash配置

output {
  gelf {
    host => "graylog_ip_address"
    port => 12201
  }
  stdout { codec => rubydebug }
}

Graylog-Inputs 配置为GELF UDP，监听端口12201

本文链接：https://www.cnblogs.com/levi125/p/13685855.html