Filebeat

Created by Seokje Park at 2020.02.21

keyword

  • centralized logging

  • log shipper

What is Beats ?

  • Beats are collection of lightweight and open source log shipper

  • Beats are built on top of a Go framework called libbeat.

  • It acts as agents installed on the different servers for collecting lgos or metrics

  • Filebeat : Collects log files

  • Packetbeat : Collects network data

  • Metricbeat : Collects server metrics

  • Once collected, the data is sent to either directly into Elasticsearch or to Logstash.

Quick start

Described on AWS Linux(AMI) basis.

  • Download and install the public signing key:

    $ sudo rpm --import https://packages.elastic.co/GPG-KEY-elasticsearch

  • Create elastic.repo and add the following lines in your /etc/yum.repos.d/ directory:

    [elastic-7.x]
name=Elastic repository for 7.x packages
baseurl=https://artifacts.elastic.co/packages/7.x/yum
gpgcheck=1
gpgkey=https://artifacts.elastic.co/GPG-KEY-elasticsearch
enabled=1
autorefresh=1
type=rpm-md

  • Modify filebeat.yml in /etc/filebeat/filebeat.yml:

    filebeat.inputs:
  - type: log
    enabled: true
    paths:
      - /home/ec2-user/.pm2/[APPNAME]*.log
output.elasticsearch:
  hosts: ["<es_url>"]
username: "elastic"
password: "<password>"
setup.kibana:
  host: "<kibana_url>"

  • Add the following line /etc/filebeat/modules.d/system.yml.disabled: (Optional)

    This way you send log events in Elasticsearch with a UTC timestamp. Kibana can simply convert from UTC to whatever timezone you browser is in at request time.

    var.convert_timezone: true

  • Enable logstash module

    $ sudo filebeat modules enable logstash

  • Setup and start Filebeat

    $ sudo filebeat setup # A new index will be created in Elasticsearch which you can define in Kibana
$ sudo service filebeat start
$ sudo chkconfig --add filebeat # Start filebeat automatically during boot

  • Kibana Management > Index Patterns create index pattern filebeat-*

  • In Kibana Discover, change current index pattern to filebeat-* and set date filters.

  • Click Refresh. if you see data, you made it. Congratulation!

참고

PreviousMigration from MongoDB to ElasticsearchNextDatabase

Last updated