Introduction

This article will introduce the common distributed system log collection components EFK Build , And the handling of some common problems .

summary

EFK（ElasticSearch、Fluentd、Kibana） It is a common distributed system log collection scheme ,es For storing data ,kibana Used to show data , Support various search and dimension aggregation .fluentd For the log collection tool , Support data collection from various data sources , Filter data 、 analysis 、 transformation 、 After being structured , write in es.

Maybe what you hear more is ELK, The difference is L, refer to logstash, It is another log collection tool , The two have little difference in function , There are some performance comparison articles on the Internet , Overall ,fluentd better .

Component building

background

In the background of shangyun ,es Components can be purchased directly from cloud resources . Tencent cloud es establish 、 The use of reference Cloud official website ES product . establish es example , You can use visualization tools kibana（ Open the Internet kibana Pay attention to safety , You can only open intranet access , Or set the white list of access network segments ）.

Because the team uses based on k8s Container deployment of , In order to collect the logs in the container , We mount data volumes , Map the log path in the container to the fixed location of the local disk of the node . In order to ensure that the logs of each node can be collected in time , We go through daemonset The way , Deploy fluentd, Ensure that each node has a log collection process .fluentd According to the configuration file , take tail The log to is written to the target at a certain interval es example .

Key steps

1. Deploy td-agent.conf To configure

fluentd The most troublesome part of the deployment process is , The configuration file .fluentd After collecting the logs , According to the configuration file , Process and output logs . therefore , We first need to deploy a configmap, Set the configuration file to td-agent.conf Mount the file name to the specified path of the container （/etc/fluent/config.d） Next （subPath）.configmap Profile contents , Use |- Define complex attributes .

data:   
    td-agent.conf:|-

Here are td-agent.conf Example ：

<match fluent.**>
  @type null
</match>
<source>
  @id xx-containers.log
  @type tail
  path /var/log/**/log.log
  pos_file /var/log/xx.log.pos
  tag log.**
  <parse>
        @type multiline
        format_firstline /^\[\d{4}\/\d{2}\/\d{2} /
        format1 /^\[(?<logtime>[^\]]*)\] \[(?<level>[^ ]*)\] (?<position>[\s\S\d]*): \[Action=(?<action>[^ ]*)\|RequestId=(?<reqId>[^|]*)\|AppId=(?<appid>[^ ]*)\|TfAid=(?<aid>[^|]*)\|TfUid=(?<uid>[^|]*)\|(?<context>[^\]]*)\] (?<message>[\s\S]*)$/
  </parse>
  emit_unmatched_lines false
  read_from_head false
</source>
<source>
  @id xx-containers.log
  @type tail
  path /var/log/**/vlog.log
  pos_file /var/log/xx.log.pos
  tag xx.*
  <parse>
    @type regexp
    expression /(?<logtime>\d{4}-[01]\d-[0-3]\d [0-2]\d:[0-5]\d:[0-5]\d\.\d{3})[\t ]*(?<level>[^\t ]*)[\t ]*(?<position>[^\t ]*)[\t ]*(?<message>[\s\S]*)$/
  </parse>
  emit_unmatched_lines false
  read_from_head false
</source>
<filter **>
  @type record_transformer
  enable_ruby true
  <record>
    message ${record.dig("position")}:${record.dig("message")}
  </record>
</filter>
<filter **>
  @type record_transformer
  <record>
    namespace ${tag_parts[4]}
  </record>
</filter>
<filter **>
  @type record_transformer
  <record>
    module ${tag_parts[5]}
  </record>
</filter>
<filter **>
  @type record_transformer
  <record>
    service_name ${tag_parts[4]}:${tag_parts[5]}
  </record>
</filter>
<match **>
   @type elasticsearch_dynamic
   @log_level debug
   include_tag_key true
   type_name _doc
   host xx
   port xx
   user xx
   password xx
   logstash_format true
   logstash_prefix logstash-xx-test
   <buffer>
        @type file
        path /var/log/fluentd/buffer
        flush_interval 5s
        flush_thread_count 10
        retry_forever false
        retry_max_times 1
		retry_wait 1s
        chunk_limit_size 8MB
        total_limit_size 256MB
        queue_limit_length 32
        compress gzip
   </buffer>
</match>

source

For input source , Key configuration ：@type Choose to tail plug-in , Read path （ Support for wildcards ） File contents under the path .pos_file Record the current read file location . The content goes through parse Parsing plug-ins , Press @type multiline Multi line parsing .format_firstline Parsing for multiple lines , The first line of regular rules ,format1 Regular for line content . Multiple input sources can be configured . This step is because the log format is complex , Try again and again to avoid failure , You can start with   Online verification .

read_from_head： by true, Will read from the file header , The default is false. Attention should be paid to , The first access is due to the volume of historical logs , May trigger es Of  circuit_breaking_exception. Recommended setting is false, If it already exists pos_file Specified file , You need to remove , To read from the end of the file .

filter

For the filter , adopt tag matching , Qualified records , Here we use @type record_transformer plug-in unit , It can realize the conversion of fields in records , Including increase 、 Delete 、 Change . If it involves expression operation , Need to specify enable_ruby true. For example, the field content may be empty , You can use dig, for example ${record.dig("position")} , Avoid exceptions . See ：https://docs.fluentd.org/filter/record_transformer.

match

matching tag, The output type specifies @type elasticsearch. The following configuration involves expressions , Such as logstash_prefix logstash-${tag_parts[3]}-test

, Need to use @type elasticsearch_dynamic.

in the light of es Different versions , Pay attention to the configuration details .es 7.5 above ,type_name Required , Otherwise, the system will report an error ：

as the final mapping would have more than 1 type: [_doc, fluentd]

• host、port、user、password Fill in the applied ES Cluster information .
• logstash_format：true,fluentd Will take logstash Format to forward structured log data .
• logstash_prefix：index Name prefix , You can add ${tag} Placeholders to identify different data sources . For example, data from different environments .
• Or use index_name logstash.${tag}.%Y%m%d Configure the specified index name .
• buffer： Configure buffer , Periodically brush the contents of the buffer to es.
• If the log volume is too large , May trigger BufferOverflowError. Configurable ：queue_full_action drop_oldest_chunk solve .

Personal experience ,fluentd The configuration of may encounter many problems , Most of the questions can be answered on the official website .

Deploy fluentd service

Mount log path

You need to attach the data volume to the disk path of the log generated by the business with the local path fluentd In container .

Mount profile

Will step 1 Created configmap, To mount in a container .fluentd Image needs to be selected fluentd-elasticsearch, Data volume mount concerns volumes、volumeMounts To configure .

apiVersion: apps/v1beta2
kind: DaemonSet
metadata:
  annotations:
    deprecated.daemonset.template.generation: "6"
    description: fluentd Log collection component 
  generation: 7
  labels:
    k8s-app: fluentd-es
    qcloud-app: fluentd-es
  name: fluentd-es
  namespace: kube-public
spec:
  revisionHistoryLimit: 10
  selector:
    matchLabels:
      k8s-app: fluentd-es
      qcloud-app: fluentd-es
  template:
    metadata:
      creationTimestamp: null
      labels:
        k8s-app: fluentd-es
        qcloud-app: fluentd-es
    spec:
      containers:
      - image: ccr.ccs.tencentyun.com/k8s-comm/fluentd-elasticsearch:v2.5.2
        imagePullPolicy: IfNotPresent
        name: fluentd-es
        resources:
          limits:
            cpu: "1"
            memory: 2Gi
          requests:
            cpu: 500m
            memory: 256Mi
        securityContext:
          privileged: false
        terminationMessagePath: /dev/termination-log
        terminationMessagePolicy: File
        volumeMounts:
        - mountPath: /var/log
          name: varlog
        - mountPath: /var/lib/docker/containers
          name: varlibdockercontainers
        - mountPath: /etc/fluent/config.d
          name: config-volume
      dnsPolicy: ClusterFirst
      restartPolicy: Always
      schedulerName: default-scheduler
      securityContext: {}
      serviceAccount: efk
      serviceAccountName: efk
      terminationGracePeriodSeconds: 30
      volumes:
      - hostPath:
          path: /var/log
          type: DirectoryOrCreate
        name: varlog
      - hostPath:
          path: /var/lib/docker/containers
          type: DirectoryOrCreate
        name: varlibdockercontainers
      - configMap:
          defaultMode: 420
          name: td-agent-config
        name: config-volume
  updateStrategy:
    type: OnDelete

reference

1. fluentd Online testing tool for text parsing rules ：https://fluentular.herokuapp.com/
2. fluentd Official documents ：https://docs.fluentd.org/
3. EFK、ELK difference ：https://cloud.tencent.com/developer/article/1770741