Filebeat收集 json格式日志

2018年1月21日16:47:18 发表评论 5,340 views
广告也精彩

一、Filebeat 简介

Filebeat:ELK 协议栈的新成员,一个轻量级开源日志文件数据搜集器,基于go语言开发。

我们之前使用logstach去收集client日志,但是会占用较多的资源,拖慢服务器,后续轻量级的filebeat诞生,我们今天的主角是 Filebeat 版本为 6.2.3.

Filebeat收集 json格式日志
前面提到 Filebeat 已经完全替代了 Logstash-Forwarder 成为新一代的日志采集器,同时鉴于它轻量、安全等特点,越来越多人开始使用它,那么Filebeat又有哪些优势呢,见下面对比图。

logstach与Filebeat的对比图

Filebeat收集 json格式日志

二、基于 Filebeat 架构的配置部署详解

传送门: ELKstack 一键自动化部署

wget https://artifacts.elastic.co/downloads/beats/filebeat/filebeat-6.2.3-linux-x86_64.tar.gz
tar -zxf filebeat-6.2.3-linux-x86_64.tar.gz
mv filebeat-6.2.3-linux-x86_64 /usr/local/filebeat
cat >/usr/local/filebeat/filebeat.yml<<EOF
filebeat.prospectors:
- input_type: log
  paths:
    - /var.log/nginx/*.log
output.logstash:
  hosts: ["192.168.56.13:5044"]
EOF
cd /usr/local/filebeat/
nohup /usr/local/filebeat/filebeat & >/dev/null
## 第一行定义抓取日志类型  log
## 第二行 定义要抓取的日志文件 可以用正则
## 第三行 输出到 logstach,并配置地址及端口
## 最后 启动filebeat

三、修改 nginx 的日志格式json

tail /etc/nginx/nginx.conf

log_format access_json '{"@timestamp":"$time_iso8601",'
        '"host":"$server_addr",'
        '"clientip":"$remote_addr",'
        '"size":$body_bytes_sent,'
        '"responsetime":$request_time,'
        '"upstreamtime":"$upstream_response_time",'
        '"upstreamhost":"$upstream_addr",'
        '"http_host":"$host",'
        '"url":"$uri",'
        '"domain":"$host",'
        '"xff":"$http_x_forwarded_for",'
        '"referer":"$http_referer",'
        '"status":"$status"}';
    access_log  /var/log/nginx/access.log  access_json;
#记得写在`http` 标签内
#跟进实际情况,判断是否注销原有日志定义内容
#nginx 的日志里含有中文的话,会将中文转为 Unicode 编码,如果不转的话,加入 `escape=json` 参数就可以了[图1]

重新加载配置,生产reload即可。
systemctl restart nginx.service

图1
Filebeat收集 json格式日志
Filebeat收集 json格式日志

四、配置 filebeat.yml

cp /usr/local/filebeat/filebeat.yml{,.bak}     #备份下配置文件
grep -v "#"  /usr/local/filebeat/filebeat.yml | grep -v "^$"  #查看配置
#=========================== Filebeat prospectors =============================

filebeat.prospectors:

- input_type: log

  paths:
    - /var/log/nginx/*access*.log
exclude_lines: ["^DBG","^$"] #不收取的
document_type: system-log-5611 #类型,会在每条日志中插入标记

output.logstash:
  hosts: ["192.168.56.13:5044"]

# logstash conf 配置文件输入 要加入  codec => "json" 否则看不到效果

Filebeat收集 json格式日志

五、查看效果

Filebeat收集 json格式日志

变更前:[主要看meassage 里面都写在一行,太乱了]

{
  "_index": "logstash-2019.01.22",
  "_type": "doc",
  "_id": "nQNtdWgBhuwVZ0iJm1FU",
  "_version": 1,
  "_score": null,
  "_source": {
    "tags": [
      "beats_input_codec_plain_applied"
    ],
    "message": "{\"@timestamp\":\"2019-01-22T19:58:54+08:00\",\"host\":\"192.168.56.11\",\"clientip\":\"192.168.56.1\",\"size\":0,\"responsetime\":0.000,\"upstreamtime\":\"-\",\"upstreamhost\":\"-\",\"http_host\":\"192.168.56.11\",\"url\":\"/index.html\",\"domain\":\"192.168.56.11\",\"xff\":\"-\",\"referer\":\"-\",\"status\":\"304\"}",
    "source": "/var/log/nginx/access.log",
    "host": "qiuyuetao",
    "@version": "1",
    "@timestamp": "2019-01-22T11:59:01.250Z",
    "offset": 8433,
    "beat": {
      "version": "6.2.3",
      "hostname": "qiuyuetao",
      "name": "qiuyuetao"
    }
  },
  "fields": {
    "@timestamp": [
      "2019-01-22T11:59:01.250Z"
    ]
  },
  "sort": [
    1548158341250
  ]
}

变更后:[meassage被拆分成多个关键字,方便我们出图和数据比对]

{
  "_index": "logstash-2019.01.22",
  "_type": "doc",
  "_id": "pgN4dWgBhuwVZ0iJ61Hm",
  "_version": 1,
  "_score": null,
  "_source": {
    "domain": "192.168.56.11",
    "offset": 10881,
    "size": 0,
    "tags": [
      "beats_input_codec_json_applied"
    ],
    "url": "/index.html",
    "http_host": "192.168.56.11",
    "beat": {
      "name": "qiuyuetao",
      "version": "6.2.3",
      "hostname": "qiuyuetao"
    },
    "status": "304",
    "source": "/var/log/nginx/access.log",
    "upstreamhost": "-",
    "@timestamp": "2019-01-22T04:11:23.310Z",
    "clientip": "192.168.56.1",
    "xff": "-",
    "referer": "-",
    "upstreamtime": "-",
    "@version": "1",
    "host": "192.168.56.11",
    "responsetime": 0
  },
  "fields": {
    "@timestamp": [
      "2019-01-22T04:11:23.310Z"
    ]
  },
  "sort": [
    1548130283310
  ]
}

至此Filebeat收集 json格式日志 完~~~

  • QQ精品交流群
  • weinxin
  • 微信公众号
  • weinxin
广告也精彩
admin

发表评论

:?: :razz: :sad: :evil: :!: :smile: :oops: :grin: :eek: :shock: :???: :cool: :lol: :mad: :twisted: :roll: :wink: :idea: :arrow: :neutral: :cry: :mrgreen: