Technology Sharing

한어Русский языкEnglishFrançaisIndonesianSanskrit日本語DeutschPortuguêsΕλληνικάespañolItalianoSuomalainenLatina

Common plug-ins for ELK filter module

Commonly used plug-ins for logstash filter module:

filter: represents the data processing layer, including data formatting, data type conversion, data filtering, etc., and supports regular expressions

1. grok splits several large text fields into several small fields (?<field name> regular expression) Field name: the content matched by the regular expression
2. date unifies and formats the time format in the data
3. mutate can rename, delete, replace and modify fields in events. For example, you can remove some useless fields or add custom fields.
4. multiline unifies multiple lines of data and aggregates them into a single line

GROK: Regular Expression Capture Plugin

Use text fragment segmentation to segment log events, which are divided into built-in regular expressions and custom regular expressions.

Built-in regular expression call: %(built-in regular expression: field name)
 

Custom regular expression call: (?<field name>custom regular)

         

multiline: Unify multiple lines of data and aggregate them into a single line

pattern (match lines by regular expression)
negate (false|true, whether to negate. False means not to negate, and merge the lines matched by the regular expression according to the setting of what
True means negation, and the lines matched by the regular expression are not merged according to the setting of what)
what (previous|next, previous means upward merge, next means downward merge

date: unifies the format of the timestamp @timestamp of the log event collected by logstash and the actual printing time of the log

1. First configure the grok plug-in to split the log time and print the time field
2. Use match to match the time format of the log time field in the date plugin configuration
3. Then use target to output to the @timestamp field to unify the time format
 

ELK optimized Filebeat deployment

Install NGINX

cd /etc/yum.repos.d/
上传nginx.repo文件
yum install -y nginx
systemctl enable --now nginx
cd /usr/share/nginx/html
  #准备测试页面
echo '<h1>this is web page</h1>' > test.html
echo '<h1>this is web2 page</h1>' > test1.html

Install Filebeat

上传软件包 filebeat-6.7.2-linux-x86_64.tar.gz 到/opt目录
tar xf filebeat-6.7.2-linux-x86_64.tar.gz
mv filebeat-6.7.2-linux-x86_64 /usr/local/filebeat

Set the main configuration file of filebeat

cd /usr/local/filebeat
cp filebeat.yml filebeat.yml.bak
vim filebeat.yml
filebeat.inputs:
- type: log         #指定 log 类型，从日志文件中读取消息
  enabled: true     #24行
  paths:
    - /var/log/nginx/access.log       #28行指定监控的日志文件
    - /var/log/nginx/error.log
  tags: ["filebeat"]		#设置索引标签
  fields:           #46行可以使用 fields 配置选项设置一些参数字段添加到 output 中
    service_name: nginx
    log_from: 192.168.73.130 
 
--------------output-------------------
(全部注释掉)
 
----------------Logstash output---------------------
output.logstash:    #162行
  hosts: ["192.168.73.120:5044"]      #164行指定 logstash 的 IP 和端口

Modify Logstash configuration

cd /etc/logstash/conf.d
vim filebeat.conf
input {
    beats {
        port => "5044"
    }
}
 
#filter {}
 
output {
     elasticsearch {
                  hosts => ["192.168.73.80:9200", "192.168.73.100:9200", "192.168.73.110:9200"]   #集群els地址
                  index => "nginx-%{+yyyy.MM.dd}"
     }
 
}
 
 
logstash -t -f filebeat.conf  #检查文件

Start and configure filebeat

192.168.73.130

Start ./filebeat -e -c filebeat.yml

Start logstash

192.168.73.120

Start logstashlogstash -f filebeat.conf