Filebeat收集 json格式日志

一、Filebeat 简介

Filebeat：ELK 协议栈的新成员，一个轻量级开源日志文件数据搜集器，基于go语言开发。

我们之前使用logstach去收集client日志，但是会占用较多的资源，拖慢服务器，后续轻量级的filebeat诞生,我们今天的主角是 Filebeat 版本为 6.2.3.

前面提到 Filebeat 已经完全替代了 Logstash-Forwarder 成为新一代的日志采集器，同时鉴于它轻量、安全等特点，越来越多人开始使用它，那么Filebeat又有哪些优势呢，见下面对比图。

logstach与Filebeat的对比图

二、基于 Filebeat 架构的配置部署详解

传送门： ELKstack 一键自动化部署

wget https://artifacts.elastic.co/downloads/beats/filebeat/filebeat-6.2.3-linux-x86_64.tar.gz
tar -zxf filebeat-6.2.3-linux-x86_64.tar.gz
mv filebeat-6.2.3-linux-x86_64 /usr/local/filebeat
cat >/usr/local/filebeat/filebeat.yml<<EOF
filebeat.prospectors:
- input_type: log
  paths:
    - /var.log/nginx/*.log
output.logstash:
  hosts: ["192.168.56.13:5044"]
EOF
cd /usr/local/filebeat/
nohup /usr/local/filebeat/filebeat & >/dev/null
## 第一行定义抓取日志类型  log
## 第二行 定义要抓取的日志文件 可以用正则
## 第三行 输出到 logstach，并配置地址及端口
## 最后 启动filebeat

三、修改 nginx 的日志格式json

tail /etc/nginx/nginx.conf

log_format access_json '{"@timestamp":"$time_iso8601",'
        '"host":"$server_addr",'
        '"clientip":"$remote_addr",'
        '"size":$body_bytes_sent,'
        '"responsetime":$request_time,'
        '"upstreamtime":"$upstream_response_time",'
        '"upstreamhost":"$upstream_addr",'
        '"http_host":"$host",'
        '"url":"$uri",'
        '"domain":"$host",'
        '"xff":"$http_x_forwarded_for",'
        '"referer":"$http_referer",'
        '"status":"$status"}';
    access_log  /var/log/nginx/access.log  access_json;
#记得写在`http` 标签内
#跟进实际情况，判断是否注销原有日志定义内容
#nginx 的日志里含有中文的话，会将中文转为 Unicode 编码，如果不转的话，加入 `escape=json` 参数就可以了[图1]

重新加载配置，生产reload即可。
systemctl restart nginx.service

图1

四、配置 filebeat.yml

cp /usr/local/filebeat/filebeat.yml{,.bak}     #备份下配置文件
grep -v "#"  /usr/local/filebeat/filebeat.yml | grep -v "^$"  #查看配置
#=========================== Filebeat prospectors =============================

filebeat.prospectors:

- input_type: log

  paths:
    - /var/log/nginx/*access*.log
exclude_lines: ["^DBG","^$"] #不收取的
document_type: system-log-5611 #类型，会在每条日志中插入标记

output.logstash:
  hosts: ["192.168.56.13:5044"]

# logstash conf 配置文件输入 要加入  codec => "json" 否则看不到效果

五、查看效果

变更前：[主要看meassage 里面都写在一行，太乱了]

{
  "_index": "logstash-2019.01.22",
  "_type": "doc",
  "_id": "nQNtdWgBhuwVZ0iJm1FU",
  "_version": 1,
  "_score": null,
  "_source": {
    "tags": [
      "beats_input_codec_plain_applied"
    ],
    "message": "{\"@timestamp\":\"2019-01-22T19:58:54+08:00\",\"host\":\"192.168.56.11\",\"clientip\":\"192.168.56.1\",\"size\":0,\"responsetime\":0.000,\"upstreamtime\":\"-\",\"upstreamhost\":\"-\",\"http_host\":\"192.168.56.11\",\"url\":\"/index.html\",\"domain\":\"192.168.56.11\",\"xff\":\"-\",\"referer\":\"-\",\"status\":\"304\"}",
    "source": "/var/log/nginx/access.log",
    "host": "qiuyuetao",
    "@version": "1",
    "@timestamp": "2019-01-22T11:59:01.250Z",
    "offset": 8433,
    "beat": {
      "version": "6.2.3",
      "hostname": "qiuyuetao",
      "name": "qiuyuetao"
    }
  },
  "fields": {
    "@timestamp": [
      "2019-01-22T11:59:01.250Z"
    ]
  },
  "sort": [
    1548158341250
  ]
}

变更后：[meassage被拆分成多个关键字，方便我们出图和数据比对]

{
  "_index": "logstash-2019.01.22",
  "_type": "doc",
  "_id": "pgN4dWgBhuwVZ0iJ61Hm",
  "_version": 1,
  "_score": null,
  "_source": {
    "domain": "192.168.56.11",
    "offset": 10881,
    "size": 0,
    "tags": [
      "beats_input_codec_json_applied"
    ],
    "url": "/index.html",
    "http_host": "192.168.56.11",
    "beat": {
      "name": "qiuyuetao",
      "version": "6.2.3",
      "hostname": "qiuyuetao"
    },
    "status": "304",
    "source": "/var/log/nginx/access.log",
    "upstreamhost": "-",
    "@timestamp": "2019-01-22T04:11:23.310Z",
    "clientip": "192.168.56.1",
    "xff": "-",
    "referer": "-",
    "upstreamtime": "-",
    "@version": "1",
    "host": "192.168.56.11",
    "responsetime": 0
  },
  "fields": {
    "@timestamp": [
      "2019-01-22T04:11:23.310Z"
    ]
  },
  "sort": [
    1548130283310
  ]
}

至此Filebeat收集 json格式日志 完~~~