Level up logs and ELK - VRR FileBeat configuration

Articles index:

1. Introduction (Everyone)
2. JSON as logs format (Everyone)
3. Logging best practices with Logback (Targetting Java DEVs)
4. Logging cutting-edge practices (Targetting Java DEVs) 
5. Contract first log generator (Targetting Java DEVs)
6. ElasticSearch VRR Estimation Strategy (Targetting OPS)
7. VRR Java + Logback configuration (Targetting OPS)
8. VRR FileBeat configuration (Targetting OPS)
9. VRR Logstash configuration and Index templates (Targetting OPS)
10. VRR Curator configuration (Targetting OPS)
11. Logstash Grok, JSON Filter and JSON Input performance comparison (Targetting OPS)

VRR FileBeat configuration

Filebeat doesn't need much configuration for JSON log files, just our typical agreement between parties:

• DEVs agree to 
• use JSON for logs, 
• VRR as log retention strategy, 
• "imp" JSON field for VRR "importance" fields with values LOW, IMP, CRIT
• no "imp" field means LOW importance
• OPS agree to
• take this file and use retention and replication depending on those fields
• add "service" in filebeat for application name
• add "environment" in filebeat where applicable
• add "logschema":"vrr" to distinguish a common approach for logs.

As contract is mostly the same for all applications, Filebeat configuration is very reusable, one entry per application and box.

This file, in a working example, can be found here.

- type: log
  enabled: true <- important ;)
  paths:
    - /path/to/logFile.json 
  encoding: utf-8
  fields:
    logschema: vrr <- this value will be reused in Logstash configuration
    service: leveluplogging <- application / service name
    environment: production <- optional, very.

Next: 9 - VRR Logstash configuration and Index templates