Articles index:
- Introduction (Everyone)
- JSON as logs format (Everyone)
- Logging best practices with Logback (Targetting Java DEVs)
- Logging cutting-edge practices (Targetting Java DEVs)
- Contract first log generator (Targetting Java DEVs)
- ElasticSearch VRR Estimation Strategy (Targetting OPS)
- VRR Java + Logback configuration (Targetting OPS)
- VRR FileBeat configuration (Targetting OPS)
- VRR Logstash configuration and Index templates (Targetting OPS)
- VRR Curator configuration (Targetting OPS)
- Logstash Grok, JSON Filter and JSON Input performance comparison (Targetting OPS)
Contract first log generator - HTML and Java generation from descriptor file.
As a result of my previous article,Logging best practices with Logback and Logging cutting-edge practices, I came to the conclusion that we were putting to much weight into developers alone to generate top quality logs.
Nowadays it wouldn't be acceptable by many people to live with code full of hard-coded keys that need to be kept in mind, types to relate and never mistake, and explain all this to other colleagues not only from development teams, but from support, pre-sales, etc...
I see no better alternative than creating a description file that relates domain, type and explanation, something I could convert to HTML and code alike:
version: 1project-name:coins-jdk8-example
mappings:
- name: amount
type: java.lang.Integer
description: Amount of money to match, in minimum representation (no decimals).
- name: combinations
type: java.lang.Integer
description: Total number of combinations of change.
- name: coins
type: int
description: Number of coins in a combination.
Generated code:
package com.navid.codegen;
import static net.logstash.logback.argument.StructuredArguments.keyValue;
import java.lang.Integer;
import java.lang.Iterable;
import net.logstash.logback.argument.StructuredArgument;
public final class LoggerUtils {
public static StructuredArgument kvAmount(Integer amount) {
return keyValue("amount",amount);
}
public static StructuredArgument aAmount(Iterable amount) {
return new net.logstash.logback.marker.ObjectAppendingMarker("amount",amount);
}
public static StructuredArgument aAmount(Integer... amount) {
return new net.logstash.logback.marker.ObjectAppendingMarker("amount",amount);
}
public static StructuredArgument kvCombinations(Integer combinations) {
return keyValue("combinations",combinations);
}
public static StructuredArgument aCombinations(Iterable combinations) {
return new net.logstash.logback.marker.ObjectAppendingMarker("combinations",combinations);
}
public static StructuredArgument aCombinations(Integer... combinations) {
return new net.logstash.logback.marker.ObjectAppendingMarker("combinations",combinations);
}
public static StructuredArgument kvCoins(int coins) {
return keyValue("coins",coins);
}
public static StructuredArgument aCoins(Iterable coins) {
return new net.logstash.logback.marker.ObjectAppendingMarker("coins",coins);
}
public static StructuredArgument aCoins(int... coins) {
return new net.logstash.logback.marker.ObjectAppendingMarker("coins",coins);
}
}
And HTML description code
After a second evolution, talking with my colleagues while I was presenting this idea, they feed back with the idea of adding complete log sentences instead of just key-type-description triplets.
This would be specially useful for non-coders and other teams, as they would know what to look for in the logs without needing to read it all.
version: 1
project-name: coins-jdk8-example
mappings:
- name: amount
type: java.lang.Integer
description: Amount of money to match, in minimum representation (no decimals).
- name: combinations
type: java.lang.Integer
description: Total number of combinations of change.
- name: coins
type: int
description: Number of coins in a combination.
- name: iid
type: java.util.UUID
description: Interaction id, basically like a request id.
sentences:
- code: ResultCombinations
message: "Number of combinations of getting change"
variables:
- amount
- combinations
extradata: {}
defaultLevel: info
- code: ResultMinimum
message: "Minimum number of coins required"
variables:
- amount
- coins
extradata: {}
defaultLevel: info
context:
- iid
Generated code (removing repeated code from above)
public static void auditResultCombinations(Logger logger, Integer amount, Integer combinations) {
logger.info("Number of combinations of getting change {} {}",kvAmount(amount),kvCombinations(combinations));
}
public static void auditResultCombinations(TriConsumer logger, Integer amount,
Integer combinations) {
logger.accept("Number of combinations of getting change {} {}",kvAmount(amount),kvCombinations(combinations));
}
public static void auditResultMinimum(Logger logger, Integer amount, int coins) {
logger.info("Minimum number of coins required {} {}",kvAmount(amount),kvCoins(coins));
}
public static void auditResultMinimum(TriConsumer logger, Integer amount, int coins) {
logger.accept("Minimum number of coins required {} {}",kvAmount(amount),kvCoins(coins));
}
public static void setContextIid(UUID iid) {
org.slf4j.MDC.put("ctx.iid",String.valueOf(iid));
}
public static void removeContextIid() {
org.slf4j.MDC.remove("iid");
}
public static void resetContext() {
org.slf4j.MDC.clear();
}
public interface MonoConsumer {
void accept(String var1);
}
public interface BiConsumer {
void accept(String var1, Object var2);
}
public interface TriConsumer {
void accept(String var1, Object var2, Object var3);
}
public interface ManyConsumer {
void accept(String var1, Object... var2);
}
Generated code follows all the principles I have been elaborating before:
- As developers are not typing the name of the Structured Argument, they won't make typos, upper-lower case mistakes, etc. Same concept won't be reused alongside code. Same concept will always have same name, perfectly written each time.
- As generated code is strongly typed, we are removing another danger from the road. At the same time, generated code allows single, collection and arrays of the same type, trying to make it comfortable for the developer at the same time.
- Compatible with the idea of forbidding developers to use their own StructuredArguments key-value pairs on the code.
- Generates HTML code to share with other teams, explaining mappings and sentences to search for.
Role responsibilities are now as follows:
Business / Product Owner / Product Manager / B.I. Analyst- Ask for introduction of main KPIs that they could later on map in charts or dashboards in order to measure whether the system is doing that it is suppose to do in business words.
This value is usually offered from developers to business, this time they would know they have a tool to ask for this information themselves.
Developers:
- Introduce information related to performance, audit of actions, transactions, etc...
- Introduce those sentences asked by other teams to make their life easier.
Maintainers/ On Callers/ Supports
- Introduce or ask for introduction for some specific well known errors in logs, being them documented first in the descriptor files. We could be aiming for a collection of error codes.
- They can elaborate log based alerts using descriptor file to know what's available.
Missing or future features:
- Maven plugin (currently only Gradle is supported)
- Support for more JDK versions (currently 1.7 and 1.8 supported).
- Improve HTML aspect.
- Redo the tool, it's my first time in Kotlin and I am a bit ashamed to be honest.
- Support for other languages, even not JVM based.
- Support for other Java logging framework.
- Create Elastalert configuration for error messages, maybe for other products as well.
- Suport for varargs and collections in sentences part.
- Extradata field to do something really.
- MDC Closeable function
To be honest, the more I work with this approach, the more I like it. I cannot but invite you to try:
Gradle plugin with working examples: https://github.com/albertonavarro/loggergenerator-gradle-plugin
Standalone tool: https://github.com/albertonavarro/loggergenerator
Next: 6 - ElasticSearch VRR Estimation Strategy
