Level up logs and ELK - Logging Cutting-Edge Practices - Documenting and sharing your most important logged information

 

Articles index:

1. Introduction (Everyone)
2. JSON as logs format (Everyone)
3. Logging best practices with Logback (Targetting Java DEVs)
4. Logging cutting-edge practices (Targetting Java DEVs) 
5. Contract first log generator (Targetting Java DEVs)
6. ElasticSearch VRR Estimation Strategy (Targetting OPS)
7. VRR Java + Logback configuration (Targetting OPS)
8. VRR FileBeat configuration (Targetting OPS)
9. VRR Logstash configuration and Index templates (Targetting OPS)
10. VRR Curator configuration (Targetting OPS)
11. Logstash Grok, JSON Filter and JSON Input performance comparison (Targetting OPS)

Logging cutting-edge practices: Sharing your log strategy with other teams and departments

Once you have your ELK cluster in place, and your team is making an application (or several) logging into it, you still have a problem to solve.

Given either of the following scenarios:

• A team newcomer is asked to investigate something that involves logs.
• CTO has granted Support access to ELK so they can have some insights to application activity or problems.
• Another team is investigating a cross-application problem.

Assuming they know the service, host and other common OPS-depending attributes...

• What attributes can they query in ELK?
• What to they mean?
• What text does come with the data they need?
• How do they know what data is correlated?

Bonus question

• How do you enforce that same attribute has same name in all your logs? Moreover, how do you enforce it's always the same type?

I'll be proposing a number of different solutions in a iterative evolution, pick the one you prefer.

Baseline:

You are using  latest log format and best practices from Level up logs and ELK - Logging best practices with Logback

logger.info("New Market Status {} {}", kv("marketId", 1234), kv("newStatus", "ready"));

Cons:

• Name or case could differ for same concept in different names.
• Type mismatch is possible, same name, different type.
• Enforcing usage as policy.
• Not documentation available for sharing with other teams or for your own reference.

 

Iteration #1: Constant names

First problem to solve is ensuring we are not using different keys for the same concept.
Initially there's nothing that avoided this to happen:

logger.info("New Market Status {} {}", kv("marketId", 1234), kv("newStatus", "ready"));

[...] 

logger.info("Editing Market Status {} {}", kv("id", 1234), kv("newstatus", "ready"));

On one hand, no developer is going to manually check all the names for any variable to enforce consistency of names. On the other, even lower/upper case mismatch could ruin your queries as they are stored in case sensitive fields.

First iteration would make us think of a set of constant Strings or Enums to help us to deal with variety of names. Documentation about meaning, usage, required type and description could be explicit in comments.

public class LoggerUtils {
    public static final String MK_ID = "marketId"; //String - Market Id
    public static final String MK_NEW_STATUS = "newStatus"; //MarketStatus.java enum - New Market Status
}

Finally code would look like this (Enforced by code review and good will only)

logger.info("New Market Status {} {}", kv(MK_ID, 1234), kv(MK_NEW_STATUS, "ready"));

[...] 

logger.info("Editing Market Status {} {}", kv(MK_ID, 1234), kv(MK_NEW_STATUS, "ready"));

Cons:

• Name or case could differ for same concept in different names.
• Type mismatch is possible: same name, different type.
• Enforce usage as policy.
• Not documentation available for sharing with other teams or for your own reference.

Iteration #2: Enforcing types

This problem comes from accidentally reusing same name for different kinds of items, being "id" a paradigmatic example.

logger.info("New Market Status {} {}", kv("id", 1234), kv("newStatus", "ready"));

[...] 

logger.info("User has signed up {}", kv("id", "email@forexample.com"));

Unless forced, first occurrence of a type defines the type for that index. Therefore you will find an error as following in Logstash logs/output:

logstash_1       | [2018-11-15T22:08:29,392][WARN ][logstash.outputs.elasticsearch] 

Could not index event to Elasticsearch. {

:status=>400, 

:action=>[

  "index", {

    :_id=>nil, 

    :_index=>"vrr-leveluplogging-low-2018-11-15", 

    :_type=>"doc", 

    :_routing=>nil

  }, 

  #<LogStash::Event:0x9b41f3e>

], 

:response=>{

  "index"=>{

    "_index"=>

    "vrr-leveluplogging-low-2018-11-15", 

    "_type"=>"doc", 

    "_id"=>"hJZrGWcBSDbbRtN6HHw3", 

    "status"=>400, 

    "error"=>{

      "type"=>"illegal_argument_exception", 

      "reason"=>"mapper [id] of different type, current_type [long], merged_type [text]"

}}}}

If you check Kibana's Management -> Index Patterns -> vrr-* (in our example)


Field "id" has been defined as "number", therefore when Logstash tries to put a "string" it fails.

Once introduced the problem, let's try the solution.

import net.logstash.logback.argument.StructuredArgument;
import static net.logstash.logback.argument.StructuredArguments.kv;

public class LoggerUtils {

    public static StructuredArgument logMarketId(Long id) {
        return kv("marketId", id);
    }

    public static StructuredArgument logCustId(String id) {
        return kv("custId", id);
    }

}

We could create a helper class that centralized the responsibility of not reusing names, and name-type relationship.

It would be used like this now (old and new version):
logger.info("User has signed up {}", kv("id", "email@forexample.com"));

logger.info("User has signed up {}", logCustId("email@forexample.com"));

Yet you need to enforce creating these functions and its usage.

Cons:

• Name or case could differ for same concept in different names
• Type mismatch is possible, same name, different type
• Enforce usage as policy
• Not documentation available for sharing with other teams or for your own reference

Please notice that a field accepting a type, e.g. "id" accepting "number", also implies that it can take an array of the same type (and not only a single value); it will be handled by ElasticSearch transparently. They will be indexed as searches will return the same document for a query of either of the values in the array.

Iteration #3: Enforcing usage

How to avoid that a random developer adds their own log lines, breaking all the rules above? Good and Rogue developer below:

import net.logstash.logback.argument.StructuredArgument;
import org.slf4j.Logger;import org.slf4j.LoggerFactory; 
import static example.LoggerUtils.logCustId;
import static net.logstash.logback.argument.StructuredArguments.kv; 

[...] 

logger.info("Good developer using custom LoggerUtils {}", logCustId("email@forexample.com")); 

[...] 

logger.info("Roge developer not using custom LoggerUtils {}", kv("custId",1234L));


Fortunately for us, Checkstyle already allows to control arbitrary imports, and also allows suppresion of this rule for a given class. Let's put it together.

Checkstyle configuration forbidding usage of kv import in the project:

<module name="Checker">
    <property name="charset" value="UTF-8">

    <module name="SuppressionFilter">
        <property name="file" value="${config_loc}/suppressions.xml">
    </property></module>
    
    <module name="TreeWalker">
        <module name="IllegalImport">
            <property name="illegalClasses" value="net\.logstash\.logback\.argument\.StructuredArguments.*">
            <property name="regexp" value="true">
        </property></property></module>
    </module>
</property></module>

Making an exception with LoggerUtils class:

<suppressions>
    <suppress checks="IllegalImport" files="LoggerUtils\.java"></suppress>

</suppressions>

You could argue that there are ways to avoid this prohibition, I agree, always are.
Goal here is not to avoid by all means that a Rogue developer logged anything he liked. Goal is to avoid mistakes, and programmatically bypass this checks should really shine in a code review.

As a result of applying above configuration in our favorite build tool (Maven, Gradle, etc..) you would find a broken build (if you configure it so, and so I recommend) and an explanation as follows (Gradle output):

:compileJava
:processResources UP-TO-DATE
:classes

[ant:checkstyle] [ERROR] /home/alberto/code/leveluplogging/leveluplogging/src/main/java/example/ErrorType.java:9:1: 

Illegal import - net.logstash.logback.argument.StructuredArguments.kv. [IllegalImport]
:checkstyleMain FAILED

FAILURE: Build failed with an exception.

Cons:

• Name or case could differ for same concept in different names.
• Type mismatch is possible: same name, different type.
• Enforce usage as policy.  
• Not documentation available for sharing with other teams or for your own reference.

 

Iteration #4: Contract-first LoggerUtils code generation

A solution that kept the goodness of the three previous iterations, but focused on a "documentation first" approach (analogy of contract-first for developers) would involve, in my opinion, the following elements:

1. A description file that explains in the closest to human language possible what are you logging that could be of any interest to anyone.
2. A code generator application that took the description file and made some equivalent to LoggerUtils in your favorite language. 
3. A documentation generator application that took the description file and made a full human readable documentation in your favorite markup / text format.
4. A plugin to attack previous two steps in your build chain (Gradle plugin, Maven plugin..)

This topic will take me long enough to deserve another article. See link below "Next"
 




Next: 5 - Contract first log generator

Lombok, cleaning up your code

Much time without writing, ok, I have been working hard in company projects a bit, and the rest of the time trying to improve our tools and processes.

I want to introduce you to "Lombok", a nice tool that could bring light to the darkness of certain classes everyone has seen at least once.

The gain in this case is almost out of discussion, you annotate your code and magically it gets powers and kicks the ass to the bad guys. They are not dependencies and either aspects, because Lombok works in compilation time almost always. Indeed, inclusion of lombok.jar is not necessary in all cases, a few ones doesn't need it.


Some examples from their web:

@Getter and @Setter, it is like using Eclipse "generate getter and setter automatically" but Lombok does even easier and cleaner:

import lombok.AccessLevel;  import lombok.Getter;  import lombok.Setter;    public class GetterSetterExample {    @Getter @Setter private int age = 10;    @Setter(AccessLevel.PROTECTED) private String name;        @Override public String toString() {      return String.format("%s (age: %d)", name, age);    }  }

turns in

public class GetterSetterExample {    private int age = 10;    private String name;        @Override public String toString() {      return String.format("%s (age: %d)", name, age);    }        public int getAge() {      return age;    }        public void setAge(int age) {      this.age = age;    }        protected void setName(String name) {      this.name = name;    }  }

Those changes are made in compiled bytecode, although you can decompile this changes (delombok) and see them in Java.

Another example, let's avoid the tricky and ugly closing file even inside and exception catch.

import lombok.Cleanup;  import java.io.*;    public class CleanupExample {    public static void main(String[] args) throws IOException {      @Cleanup InputStream in = new FileInputStream(args[0]);      @Cleanup OutputStream out = new FileOutputStream(args[1]);      byte[] b = new byte[10000];      while (true) {        int r = in.read(b);        if (r == -1) break;        out.write(b, 0, r);      }    }  }

turns in

import java.io.*;    public class CleanupExample {    public static void main(String[] args) throws IOException {      InputStream in = new FileInputStream(args[0]);      try {        OutputStream out = new FileOutputStream(args[1]);        try {          byte[] b = new byte[10000];          while (true) {            int r = in.read(b);            if (r == -1) break;            out.write(b, 0, r);          }        } finally {          if (out != null) {            out.close();          }        }      } finally {        if (in != null) {          in.close();        }      }    }  }

More: automatic log declaration

import lombok.extern.slf4j.Log;    @Log  public class LogExample {        public static void main(String... args) {      log.error("Something's wrong here");    }  }    @Log(java.util.List.class)  public class LogExampleOther {        public static void main(String... args) {      log.warn("Something might be wrong here");    }  }

turns in

public class LogExample {    private static final org.slf4j.Logger log = org.slf4j.LoggerFactory.getLogger(LogExample.class);        public static void main(String... args) {      log.error("Something's wrong here");    }  }    public class LogExampleOther {    private static final org.slf4j.Logger log = org.slf4j.LoggerFactory.getLogger(java.util.List.class);        public static void main(String... args) {      log.warn("Something might be wrong here");    }  }

(Several implementations of log are provided).

And so on and on... check the features list

http://projectlombok.org/features/index.html

This tool is also integrated with Eclipse (through installation) and Eclipse (only by including it as a dependency). Use it wisely, but use it :)

What if I want to deploy/upload several files to artifactory or nexus with Maven?

Regardless of the repository you are using to store generated artifacts, the default Maven configuration for, let's say, WAR file, is upload a WAR file.. interesting..

A common configuration commented here also uploads the source code as a jar file, and that is very recommendable for debugging.

Another uploaded or auto-generated file is a "pom.xml" project descriptor file for your artifact.

But we want to go further, we want to upload the sql file needed to generate our initial database, or the pdf with the documentation of this release.. how to get this?

<plugin>   
  <groupId>org.codehaus.mojo</groupId>   
  <artifactId>build-helper-maven-plugin</artifactId>
  <executions>
    <execution>
      <id>attach-artifacts</id>
      <phase>package</phase>
      <goals> 
        <goal>attach-artifact</goal>
      </goals> 
      <configuration>
        <artifacts> 
          <artifact> 
            <file>src\main\sql\initdb.sql</file>
            <type>sql</type>
            <classifier>create</classifier>
          </artifact>
          <artifact>
            <file>target\site\docs\document.pdf</file>   
            <type>pdf</type>
            <classifier>doc</classifier>
          </artifact>
        </artifacts>
      </configuration>
    </execution>
  </executions>
</plugin>


With this plugin, you are associating with the same artifact the "document.pdf" file, the "initdb.sql" file, the war, the sources jar, the pom.xml file... all them with the same version, and with the possibility of declaring any of them as a dependency.

It will be executed in package phase, beware because you may mistake the phases and attach something not yet created, for example, the pdf may be created in pre-site phase instead of .. compile, for example.

In the case of multiple configurations, you can configure it like this:


<plugin>   
  <groupId>org.codehaus.mojo</groupId>   
  <artifactId>build-helper-maven-plugin</artifactId>
  <executions>
    <execution>
      <id>attach-artifacts1</id>
      <phase>package</phase>
      <goals> 
        <goal>attach-artifact</goal>
      </goals> 
      <configuration>
        <artifacts> 
          <artifact> 
            <file>src\main\sql\initdb.sql</file>
            <type>sql</type>
            <classifier>create</classifier>
          </artifact>
        </artifacts>
      </configuration>
    </execution>

    <execution>
      <id>attach-artifacts2</id>
      <phase>site</phase>
      <goals> 
        <goal>attach-artifact</goal>
      </goals> 
      <configuration>
        <artifacts> 
          <artifact>
            <file>target\site\docs\document.pdf</file>   
            <type>pdf</type>
            <classifier>doc</classifier>
          </artifact>
        </artifacts>
      </configuration>
    </execution>

  </executions>
</plugin>

In this case, if we execute "mvn deploy", only "initdb.sql" will be attached, if we execute "mvn site-deploy", it will be only attached "document.pdf" (because it is supposed to be created in pre-site phase, but I am not sure if it will be deployed in "artifactory" or "nexus", because you don't pass through "deploy" phase, I would say it won't.

The best, if you execute "mvn site-deploy deploy", firstly "document.pdf" is attached in "site" phase, then "initbd.sql" is attached in "package" phase, and finally all them are deployed in "deploy" phase. Really useful!

This is the solution for a problem we wanted to solve in my job, the solution was taken here, thanks!

Roll your own Continuous Integration System (C.I.S.): Artifactory repositories configuration: snapshot, releases and security.

Roll your own Continuous Integration System (C.I.S.)

Content:

Abstract

Install Tomcat

Basic Tomcat configuration - Memory

Basic Tomcat configuration - JMX

Basic Tomcat configuration - Application Manager and permissions

Apache and uSVN

Installing Artifactory from WAR

Configure Artifactory and MySQL

Configuring Artifactory security and repositories

In order to ease the explanation about security and maven deploy, lets create the simplest Maven project possible.

Create a directory
Create a "pom.xml" file with the following content:

<project xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xmlns="http://maven.apache.org/POM/4.0.0" xsi:schemalocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd">
<modelversion>4.0.0</modelversion>

<groupid>com.mycompany</groupid>
<artifactid>mavenproject</artifactid>
<version>1.0-SNAPSHOT</version>
<packaging>pom</packaging>

</project>


Now that you have the out-of-the-box Artifactory installation and a Maven project, try to upload your project to the Artifactory repository manager.

Remember: You must have a Maven installation in order to run those examples. As far as you have followed this blog, you should have installed SpringSource Tool Suite, that brings with Maven. Maybe you need insert "mvn" executable in the PATH.

Try to execute "mvn deploy" in the same directory than "pom.xml".


[INFO] Scanning for projects...
[INFO]
[INFO] ------------------------------------------------------------------------
[INFO] Building mavenproject 1.0-SNAPSHOT
[INFO] ------------------------------------------------------------------------
[INFO]
[INFO] --- maven-install-plugin:2.3.1:install (default-install) @ mavenproject ---
[INFO] Installing /home/***/NetBeansProjects/mavenproject10/pom.xml to /home/***/.m2/repository/com/mycompany/mavenproject/1.0-SNAPSHOT/mavenproject-1.0-SNAPSHOT.pom
[INFO]
[INFO] --- maven-deploy-plugin:2.5:deploy (default-deploy) @ mavenproject ---
[INFO] ------------------------------------------------------------------------
[INFO] BUILD FAILURE
[INFO] ------------------------------------------------------------------------
[INFO] Total time: 0.586s
[INFO] Finished at: Tue May 03 22:36:23 CEST 2011
[INFO] Final Memory: 3M/74M
[INFO] ------------------------------------------------------------------------
[ERROR] Failed to execute goal org.apache.maven.plugins:maven-deploy-plugin:2.5:deploy (default-deploy) on project mavenproject: Deployment failed: repository element was not specified in the POM inside distributionManagement element or in -DaltDeploymentRepository=id::layout::url parameter -> [Help 1]
[ERROR]
[ERROR] To see the full stack trace of the errors, re-run Maven with the -e switch.
[ERROR] Re-run Maven using the -X switch to enable full debug logging.
[ERROR]
[ERROR] For more information about the errors and possible solutions, please read the following articles:
[ERROR] [Help 1] http://cwiki.apache.org/confluence/display/MAVEN/MojoExecutionException


Read the bold, it means that either in your project or in other possible settings file, you are not specifying where are you willing to deploy your project.

Note: not explained yet, but there is a lifecycle somewhere in Internet that says "deploying requires compiling first" for instance. So, when you ask Maven to do "deploy", you are also asking it to compile, testing and some other actions.

Note: The examples are based in basic configurations and no network troubles. Everything in the same host and so on. Take it as it is, an example.

Go to Artifactory, "Artifacts" tabs, and click in the second repository, "libs-snapshot-local"

By clicking the repository, its information is displayed at the right side. We want "Distribution Management" section. 

Copy it to your "pom.xml" (your project, remember?)


<project xmlns="http://maven.apache.org/POM/4.0.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd">
<modelVersion>4.0.0</modelVersion>

<groupId>com.mycompany</groupId>
<artifactId>mavenproject</artifactId>
<version>1.0-SNAPSHOT</version>
<packaging>pom</packaging>

<distributionManagement>
<repository>
<id>dhcppc24</id>
<name>dhcppc24-releases</name>
<url>http://localhost:8080/artifactory/libs-release-local</url>
</repository>
</distributionManagement>

</project>


Now execute again "mvn deploy", let's check the response: (Yes, it fails again, I do not like suspense)

[INFO] Scanning for projects...
[INFO]
[INFO] ------------------------------------------------------------------------
[INFO] Building mavenproject 1.0-SNAPSHOT
[INFO] ------------------------------------------------------------------------
[INFO]
[INFO] --- maven-install-plugin:2.3.1:install (default-install) @ mavenproject ---
[INFO] Installing /home/***/NetBeansProjects/mavenproject10/pom.xml to /home/***/.m2/repository/com/mycompany/mavenproject/1.0-SNAPSHOT/mavenproject-1.0-SNAPSHOT.pom
[INFO]
[INFO] --- maven-deploy-plugin:2.5:deploy (default-deploy) @ mavenproject ---
Downloading: http://localhost:8080/artifactory/libs-release-local/com/mycompany/mavenproject/1.0-SNAPSHOT/maven-metadata.xml
Uploading: http://localhost:8080/artifactory/libs-release-local/com/mycompany/mavenproject/1.0-SNAPSHOT/mavenproject-1.0-20110503.214739-1.pom
[INFO] ------------------------------------------------------------------------
[INFO] BUILD FAILURE
[INFO] ------------------------------------------------------------------------
[INFO] Total time: 0.660s
[INFO] Finished at: Tue May 03 23:47:39 CEST 2011
[INFO] Final Memory: 3M/74M
[INFO] ------------------------------------------------------------------------
[ERROR] Failed to execute goal org.apache.maven.plugins:maven-deploy-plugin:2.5:deploy (default-deploy) on project mavenproject: Failed to deploy artifacts: Could not transfer artifact com.mycompany:mavenproject:pom:1.0-20110503.214739-1 from/to dhcppc24 (http://localhost:8080/artifactory/libs-release-local): Failed to transfer file: http://localhost:8080/artifactory/libs-release-local/com/mycompany/mavenproject/1.0-SNAPSHOT/mavenproject-1.0-20110503.214739-1.pom. Return code is: 409 -> [Help 1]
[ERROR]
[ERROR] To see the full stack trace of the errors, re-run Maven with the -e switch.
[ERROR] Re-run Maven using the -X switch to enable full debug logging.
[ERROR]
[ERROR] For more information about the errors and possible solutions, please read the following articles:
[ERROR] [Help 1] http://cwiki.apache.org/confluence/display/MAVEN/MojoExecutionException


This error means that you are not allowed to do the upload. That is because you are trying to upload an SNAPSHOT in a RELEASE repository. WTF?

See your pom.xml, <version>1.0-SNAPSHOT</version>. That -SNAPSHOT is being read by Artifactory in order to determine whether it is a release or not.
Now travel to Artifactory. Log in (user "admin", password "password"), and go to Admin tab.

Inside Admin tab, Repositories menu and put your mouse over "libs-release-local", click "Edit".

There it is, the guts of the repository configuration. See how out-of-the-box "libs-release-local" repository can only handle with RELEASES, and you are trying to upload a SNAPSHOT. It is not possible!!.

First option, change <version>1.0-SNAPSHOT</version> for <version>1.0</version>. Come on! do it! it will fail xD. Why? You do not have permissions yet :P
It is not a good idea anyway. Just do not release anything until you know what are really doing.

Second option, in that window, you can tick "Handle Snapshots" and it will fail, because of permissions. It is a matter of time.

Third option, see above now to copy the "distribution management" information of a repository, and take the information from "libs-snapshot-local". This is the best option right now. Your "pom.xml" should look like:


<project xmlns="http://maven.apache.org/POM/4.0.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd">
<modelVersion>4.0.0</modelVersion>

<groupId>com.mycompany</groupId>
<artifactId>mavenproject</artifactId>
<version>1.0-SNAPSHOT</version>
<packaging>pom</packaging>

<distributionManagement>
<repository>
<id>dhcppc24</id>
<name>dhcppc24-releases</name>
<url>http://localhost:8080/artifactory/libs-release-local</url> </repository>
<snapshotRepository>
<id>dhcppc24</id>
<name>dhcppc24-snapshots</name>
<url>http://localhost:8080/artifactory/libs-snapshot-local</url>
</snapshotRepository>
</distributionManagement>

</project>

Now try "mvn deploy" in order to confirm that I was right, permissions are not set yet.


[INFO] Scanning for projects...
[INFO]
[INFO] ------------------------------------------------------------------------
[INFO] Building mavenproject 1.0-SNAPSHOT
[INFO] ------------------------------------------------------------------------
[INFO]
[INFO] --- maven-install-plugin:2.3.1:install (default-install) @ mavenproject ---
[INFO] Installing /home/***/NetBeansProjects/mavenproject10/pom.xml to /home/***/.m2/repository/com/mycompany/mavenproject/1.0-SNAPSHOT/mavenproject-1.0-SNAPSHOT.pom
[INFO]
[INFO] --- maven-deploy-plugin:2.5:deploy (default-deploy) @ mavenproject ---
Downloading: http://localhost:8080/artifactory/libs-snapshot-local/com/mycompany/mavenproject/1.0-SNAPSHOT/maven-metadata.xml
Downloaded: http://localhost:8080/artifactory/libs-snapshot-local/com/mycompany/mavenproject/1.0-SNAPSHOT/maven-metadata.xml (359 B at 7.5 KB/sec)
Uploading: http://localhost:8080/artifactory/libs-snapshot-local/com/mycompany/mavenproject/1.0-SNAPSHOT/mavenproject-1.0-20110505.213434-2.pom
[INFO] ------------------------------------------------------------------------
[INFO] BUILD FAILURE
[INFO] ------------------------------------------------------------------------
[INFO] Total time: 0.637s
[INFO] Finished at: Thu May 05 23:34:34 CEST 2011
[INFO] Final Memory: 3M/74M
[INFO] ------------------------------------------------------------------------
[ERROR] Failed to execute goal org.apache.maven.plugins:maven-deploy-plugin:2.5:deploy (default-deploy) on project mavenproject: Failed to deploy artifacts: Could not transfer artifact com.mycompany:mavenproject:pom:1.0-20110505.213434-2 from/to dhcppc24 (http://localhost:8080/artifactory/libs-snapshot-local): Failed to transfer file: http://localhost:8080/artifactory/libs-snapshot-local/com/mycompany/mavenproject/1.0-SNAPSHOT/mavenproject-1.0-20110505.213434-2.pom. Return code is: 401 -> [Help 1]
[ERROR]
[ERROR] To see the full stack trace of the errors, re-run Maven with the -e switch.
[ERROR] Re-run Maven using the -X switch to enable full debug logging.
[ERROR]
[ERROR] For more information about the errors and possible solutions, please read the following articles:
[ERROR] [Help 1] http://cwiki.apache.org/confluence/display/MAVEN/MojoExecutionException


Now let's set the damn permissions!

"Admin" tab, menu "Permissions"

New (in the right side)

We are going to configure the permissions for all our local repositories:

Name: Local repositories
Include Patterns: Any
Exclude Patterns: None
Repositories: Any Local Repository

Users tab: Anonymous users must be able to Deploy, automatically will be able to Annotate and Read.

And "Create" it, it is done.

Repeat the operation, "mvn deploy", and it should be able to compile and deploy your project to Artifactory repository :)


[INFO] Scanning for projects...
[INFO]
[INFO] ------------------------------------------------------------------------
[INFO] Building mavenproject 1.0-SNAPSHOT
[INFO] ------------------------------------------------------------------------
[INFO]
[INFO] --- maven-install-plugin:2.3.1:install (default-install) @ mavenproject ---
[INFO] Installing /home/***/NetBeansProjects/mavenproject10/pom.xml to /home/***/.m2/repository/com/mycompany/mavenproject/1.0-SNAPSHOT/mavenproject-1.0-SNAPSHOT.pom
[INFO]
[INFO] --- maven-deploy-plugin:2.5:deploy (default-deploy) @ mavenproject ---
Downloading: http://localhost:8080/artifactory/libs-snapshot-local/com/mycompany/mavenproject/1.0-SNAPSHOT/maven-metadata.xml
Downloaded: http://localhost:8080/artifactory/libs-snapshot-local/com/mycompany/mavenproject/1.0-SNAPSHOT/maven-metadata.xml (359 B at 5.7 KB/sec)
Uploading: http://localhost:8080/artifactory/libs-snapshot-local/com/mycompany/mavenproject/1.0-SNAPSHOT/mavenproject-1.0-20110505.214558-2.pom
Uploaded: http://localhost:8080/artifactory/libs-snapshot-local/com/mycompany/mavenproject/1.0-SNAPSHOT/mavenproject-1.0-20110505.214558-2.pom (823 B at 3.4 KB/sec)
Downloading: http://localhost:8080/artifactory/libs-snapshot-local/com/mycompany/mavenproject/maven-metadata.xml
Downloaded: http://localhost:8080/artifactory/libs-snapshot-local/com/mycompany/mavenproject/maven-metadata.xml (351 B at 8.6 KB/sec)
Uploading: http://localhost:8080/artifactory/libs-snapshot-local/com/mycompany/mavenproject/1.0-SNAPSHOT/maven-metadata.xml
Uploaded: http://localhost:8080/artifactory/libs-snapshot-local/com/mycompany/mavenproject/1.0-SNAPSHOT/maven-metadata.xml (598 B at 20.1 KB/sec)
Uploading: http://localhost:8080/artifactory/libs-snapshot-local/com/mycompany/mavenproject/maven-metadata.xml
Uploaded: http://localhost:8080/artifactory/libs-snapshot-local/com/mycompany/mavenproject/maven-metadata.xml (317 B at 4.1 KB/sec)
[INFO] ------------------------------------------------------------------------
[INFO] BUILD SUCCESS
[INFO] ------------------------------------------------------------------------
[INFO] Total time: 1.034s
[INFO] Finished at: Thu May 05 23:45:58 CEST 2011
[INFO] Final Memory: 3M/74M
[INFO] ------------------------------------------------------------------------




Last travel to Artifactory, let's find the uploaded artifact.

In the "Artifacts" tab, navigate through "libs-snapshot-local" and you will arrive to your project.

By clicking in the artifact itself, you will get the Maven code for declaring it as dependency for other projects.


<dependency>
<groupId>com.mycompany</groupId>
<artifactId>mavenproject</artifactId>
<version>1.0-20110505.214856-1</version>
<type>pom</type>
</dependency>



The end... quite long, isn't it?

Other useful Maven plugins, Reporting and Site (Part III)

FindBugs Maven Plugin

FindBugs looks for bugs in Java programs. It is based on the concept of bug patterns. A bug pattern is a code idiom that is often an error. Bug patterns arise for a variety of reasons:
Difficult language features
Misunderstood API methods
Misunderstood invariants when code is modified during maintenance
Garden variety mistakes: typos, use of the wrong boolean operator

FindBugs uses static analysis to inspect Java bytecode for occurrences of bug patterns. We have found that FindBugs finds real errors in most Java software. Because its analysis is sometimes imprecise, FindBugs can report false warnings, which are warnings that do not indicate real errors. In practice, the rate of false warnings reported by FindBugs is generally less than 50%.

Usage:
[<reporting>]
<plugins>
    <plugin>
        <groupId>org.codehaus.mojo</groupId>
        <artifactId>findbugs-maven-plugin</artifactId>
        <version>2.3.1</version>
        <configuration>
            <excludeFilterFile>${findBugs.excludeFilterFile}</excludeFilterFile>
        </configuration>
    </plugin>
</plugins>
[</reporting>]

Notice that you can make ignore some classes in its analysis, by putting some value on excludeFilterFile property.

Source: http://mojo.codehaus.org/findbugs-maven-plugin/index.html


Maven CheckStyle Plugin

The Checkstyle Plugin generates a report regarding the code style used by the developers. For more information about Checkstyle, see http://checkstyle.sourceforge.net/. This version of the plugin uses Checkstyle 5.0.

The plugin can be configured in the project's POM. Predefined rulesets are included with the plugin, these are: sun_checks.xml, turbine_checks.xml, avalon_checks.xml and maven_checks.xml. You can also use a custom ruleset by specifying it in the plugin configuration.

Usage:
[<reporting>]
<plugins>
    <plugin>
        <groupId>org.apache.maven.plugins</groupId>

        <artifactId>maven-checkstyle-plugin</artifactId>

        <version>2.6</version>

        <configuration>

            <configLocation>PUT HERE YOUR CONFIGURATION</configLocation>

            <consoleOutput>true</consoleOutput>

        </configuration>

    </plugin>
</plugins>
[</reporting>]

Related: http://looking4q.blogspot.com/2011/01/checkstyle-configuration-distribution.html
Source: http://maven.apache.org/plugins/maven-checkstyle-plugin/index.html


Maven PMD Plugin

The PMD plugin allows you to automatically run the PMD code analysis tool on your project's source code and generate a site report with its results. It also supports the separate Copy/Paste Detector tool (or CPD) distributed with PMD.

The plugin accepts configuration parameters that can be used to customize the execution of the PMD tool.

Usage:
[<reporting>]
<plugins>
    <plugin>
        <groupId>org.apache.maven.plugins</groupId>

        <artifactId>maven-pmd-plugin</artifactId>

        <version>2.5</version>

        <configuration>

            <targetJdk>${java-version}</targetJdk>

            <excludes>

                <exclude>${pmd.excludes}</exclude>

            </excludes>

        </configuration>

    </plugin>
</plugins>
[</reporting>]

Notice you can make PMD plugin to ignore some classes.

Source: http://maven.apache.org/plugins/maven-pmd-plugin/

Maven Javadoc Plugin

The Javadoc Plugin uses the Javadoc tool to generate javadocs for the specified project. For more information about the standard Javadoc tool, please refer to Reference Guide.

Usage:
[<reporting>]
<plugins>
    <plugin>
        <groupId>org.apache.maven.plugins</groupId>

        <artifactId>maven-javadoc-plugin</artifactId>

        <version>2.7</version>

    </plugin>
</plugins>
[</reporting>]

Source: http://maven.apache.org/plugins/maven-javadoc-plugin/

JMR Maven Plugin

The JXR Plugin produces a cross-reference of the project's sources. The generated reports make it easier for the user to reference or find specific lines of code. It is also handy when used with the PMD Plugin for referencing errors found in the code.

Usage:
[<reporting>]
<plugins>
    <plugin>
        <groupId>org.codehaus.mojo</groupId>

        <artifactId>jxr-maven-plugin</artifactId>

        <version>2.0-beta-1</version>

    </plugin>
</plugins>
[</reporting>]

Source: http://maven.apache.org/plugins/maven-jxr-plugin/

Taglist Maven Plugin

The Taglist Maven Plugin generates a report on various tags found in the code, like @todo or //TODO tags.

Usage:
[<reporting>]
<plugins>
    <plugin>
        <groupId>org.codehaus.mojo</groupId>

        <artifactId>taglist-maven-plugin</artifactId>

        <version>2.4</version>

    </plugin>
</plugins>
[</reporting>]

Source: http://mojo.codehaus.org/taglist-maven-plugin/

Maven Site Plugin

All the previous plugins are meant to provide data to three possible targets. Humans, Hudson/Jenkins or Maven Site

The Site Plugin is used to generate a site for the project. The generated site also includes the project's reports that were configured in the <reporting> section of the POM.

Once you execute "mvn site", a html site is created in your target/site directory, where all or most of previous reports are presented, together with project information like developers, scm or issues.

Usage:
[<reporting>]
<plugins>
    <plugin>
        <groupId>org.apache.maven.plugins</groupId>

        <artifactId>maven-site-plugin</artifactId>

        <version>2.0-beta-6</version>

        <configuration>

            <locales>en</locales>

            <outputEncoding>${project.reporting.outputEncoding}</outputEncoding>

        </configuration>

    </plugin>
</plugins>
[</reporting>]

Source: http://maven.apache.org/plugins/maven-site-plugin/

Managing Maven dependencies with an external dependencies repository

Content:

Maven abstract.

Tunning your Maven proyect

Maven standard folders

Managing dependencies with Maven

Adding a nature in Eclipse

Maven profiles inheritance

Managing Maven dependencies with an external dependencies repository

I have been talking a lot about Maven dependencies for a local machine. It is high time to start talking about remote dependencies.

When you define a dependency in your pom.xml file, that dependency is sought in repo1.maven.org. But your libraries and internal dependencies are not there. Certainly they are in your local repository, but not in your workmate's own.

I do not find necessary to speak in favour of a code repository, the reasons are widely known. Reasons for a dependency repository might be less known because code is always used, but Maven is not (always).
Basic scenario is easy to find nevertheless:

• You create a library for a project with Maven.
• You design that library as a dependency for that project.
• You might want your workmate to compile the project in his/her computer, but...
• Binaries should never be uploaded to code repository!
• Shared folders are system dependant and difficult to automatize, trace and export.
• We are looking for zero specific configuration in a compilation.

If you could only upload to a private repo1.maven.org repository in order to make accessible an internal dependency for your company... You can!

There are two main open-source implementation for this task: Nexus and Artifactory. Both seem alike, but I chose to know deeply the last one, maybe I could find some time for comparing both in installation, configuration and performance. 

If you had in your pom.xml a configuration like this:

    <repositories>

        <repository>

            <snapshots />

            <id>snapshots</id>

            <name>libs-internal</name>

            <url>http://localhost:8080/artifactory/libs-internal</url>

        </repository>

    </repositories>

With this entry in pom.xml, we are asking Maven to search in this URL for any dependency we might need, and, for being in pom.xml, that configuration is project-scoped, being shared with the project itself in code repository.

All we have to keep in mind is that the server MUST be accessible by any of your partners who you are expecting to compile the project. In this example, our partners will find that they do not have Artifactory installed in their localhost, and you will find lots of complains for creating a local-dependant configuration for a department project.

Your second try made it better:

        <repository>

            <snapshots />

            <id>snapshots</id>

            <name>libs-internal</name>

            <url>http://srvmachine:8080/artifactory/libs-internal</url>

        </repository>

and now, if that machine is accessible for all your department, and permissions are well configured, so firewall is, and half a dozen more configurations, your dependencies would be able to download broadly.

However, as you find comfortable to download dependencies, you will find awful to upload your libraries manually through web interface. It is possible as well to improve this.

With the correct Artifactory and Nexus (trying to be neutral by now) settings, you can automatize your system for uploading a library once it is compiled by Maven:

<distributionManagement>
    <repository>
        <id>linux-wdx0</id>
        <name>linux-wdx0-releases</name>
        <url>http://srvmachine:8080/artifactory/libs-release-local</url>
    </repository>
</distributionManagement>


by typing "mvn deploy" instead of "mvn install" in your Maven console or IDE compilation button.

Now you can enjoy of an easy way to share and download department-scoped libraries within your department in a new fashion way of making things work. Your objetive: your project should compile and run only by downloading it in any machine with zero configuration.

Other useful Maven plugins (Part II)

BuildNumber Maven Plugin

This mojo is designed to get a unique build number for each time you build your project. So while your version may remain constant at 1.0-SNAPSHOT for many iterations until release, you will have a build number that can uniquely identify each build during that time. The build number is obtained from scm, and in particular, at this time, from svn. You can then place that build number in metadata, which can be accessed from your app, if desired.

The mojo also has a couple of extra functions to ensure you get the proper build number. First, your local repository is checked to make sure it is up to date. Second, your local repository is automatically updated, so that you get the latest build number. Both these functions can be suppressed, if desired.

Optionally, you can configure this mojo to produce a revision based on a timestamp, or on a sequence, without requiring any interaction with an SCM system. Note that currently, the only supported SCM is subversion.

[<plugins>]
<plugin>
    <groupId>org.codehaus.mojo</groupId>
    <artifactId>buildnumber-maven-plugin</artifactId>
    <version>1.0-beta-4</version>
    <executions>
        <execution>
            <phase>validate</phase>
            <goals>
                <goal>create</goal>
            </goals>
        </execution>
    </executions>
    <configuration>
        <useLastCommittedRevision>true</useLastCommittedRevision>
        <doCheck>false</doCheck>
        <doUpdate>false</doUpdate>
        <providerImplementations>
            <svn>javasvn</svn>
        </providerImplementations>
    </configuration>
</plugin>
[</plugins>]


I use this plugin for a highly detailed file name and, in a interface variable, showing to user the build number of this compilation, in order to trace back any problem.


<build>
    <finalName>${project.artifactId}-${project.version} r${hudson-env}</finalName>
</build>




Maven Release Plugin


This plugin is used to release a project with Maven, saving a lot of repetitive, manual work. Releasing a project is made in two steps: prepare and perform.

I usually use this plugin manually for creating Tags, Branches and commit releases from command line. As far I have investigated, tagBase is project dependent, so they are scm parameters.

[<plugins>]
 <plugin>
     <groupId>org.apache.maven.plugins</groupId>
     <artifactId>maven-release-plugin</artifactId>
     <version>2.1</version>
     <configuration>
         <providerImplementations>
             <svn>javasvn</svn>
         </providerImplementations>
         <tagBase>http://desenvolupament/svn/projectes/root/tags</tagBase>
     </configuration>
     <dependencies>
         <dependency>
             <groupId>com.google.code.maven-scm-provider-svnjava</groupId>
             <artifactId>maven-scm-provider-svnjava</artifactId>
             <version>1.10</version>
         </dependency>
     </dependencies>
</plugin>
[</plugins>]


<scm>
<developerConnection>scm:svn:http://desenvolupament/svn/projectes/projects/scpr/trunk</developerConnection>
<connection>scm:svn:http://desenvolupament/svn/projectes/projects/scpr/trunk</connection>
<url>http://desenvolupament/svn/projectes/projects/scpr/trunk</url>
</scm>

Other useful Maven plugins (Part I)

Maven Compiler plugin 

The Compiler Plugin is used to compile the sources of your project. The default compiler is javac and is used to compile Java sources. Also note that at present the default source setting is 1.5 and the default target setting is 1.5, independently of the JDK you run Maven with.

Usage:
[<plugins>]
<plugin>
    <groupId>org.apache.maven.plugins</groupId>
    <artifactId>maven-compiler-plugin</artifactId>
    <version>2.3.2</version>
    <configuration>
        <compilerVersion>${java-version}</compilerVersion>
        <source>${java-version}</source>
        <target>${java-version}</target>
        <encoding>${project.build.sourceEncoding}</encoding>
    </configuration>
</plugin>
[</plugins>]


Maven compiler settings:

compilerVersion: Version of the compiler to use, ex. "1.3", "1.5", if fork is set to true.
encoding: The -encoding argument for the Java compiler.
source: The -source argument for the Java compiler.
target: The -target argument for the Java compiler.

If source and target are equals, no room for error in what version should the compiler user to interpret your sources (source) and to create the binary (target). More information about -source and -target configuration over here.

More options about this plugin is over here.


Maven Resources plugin

The Resources Plugin handles the copying of project resources to the output directory. There are two different kinds of resources: main resources and test resources. The difference is that the main resources are the resources associated to the main source code while the test resources are associated to the test source code.

Thus, this allows the separation of resources for the main source code and its unit tests.

Starting with version 2.3 this plugin uses the Maven Filtering shared component for filtering resources.

[<plugins>]
<plugin>
    <artifactId>maven-resources-plugin</artifactId>
    <version>2.2</version>
    <configuration>
        <encoding>${project.build.sourceEncoding}</encoding>
     </configuration>
</plugin>
[</plugins>]

encoding: The character encoding scheme to be applied when filtering resources.


Maven Source plugin

The Source Plugin creates a jar archive of the source files of the current project. The jar file is, by default, created in the project's target directory.

They will be uploaded to your maven dependencies repository in deploy phase, and will be downloaded if necessary afterward.

[<plugins>]
<plugin>
    <groupId>org.apache.maven.plugins</groupId>
    <artifactId>maven-source-plugin</artifactId>
    <version>2.1.2</version>
    <executions>
        <execution>
            <goals>
                <goal>jar</goal>
            </goals>
         </execution>
    </executions>
</plugin>
[<plugins>]



Maven  Eclipse plugin

The Eclipse Plugin is used to generate Eclipse IDE files (*.classpath, *.wtpmodules and the .settings folder) for use with a project.

[<plugins>]
<plugin>
    <groupId>org.apache.maven.plugins</groupId>
    <artifactId>maven-eclipse-plugin</artifactId>
    <version>2.8</version>
    <configuration>
        <downloadSources>true</downloadSources>
        <downloadJavadocs>true</downloadJavadocs>
    </configuration>
</plugin>
[<plugins>]

Maven Eclipse settings:

downloadJavadocs: Enables/disables the downloading of javadoc attachments. Defaults to false. When this flag is true remote repositories are checked for javadocs: in order to avoid repeated check for unavailable javadoc archives, a status cache is mantained. With versions 2.6+ of the plugin to reset this cache run mvn eclipse:remove-cache, or use the forceRecheck option with versions. With older versions delete the file mvn-eclipse-cache.properties in the target directory.

downloadSources: Enables/disables the downloading of source attachments. Defaults to false. When this flag is true remote repositories are checked for sources: in order to avoid repeated check for unavailable source archives, a status cache is mantained. With versions 2.6+ of the plugin to reset this cache run mvn eclipse:remove-cache, or use the forceRecheck option with versions. With older versions delete the file mvn-eclipse-cache.properties in the target directory.


These sources were uploaded by Maven Source Plugin ;)



There will be more soon, then we could talk about how to organize them all.

Maven profiles inheritance

Content:

Maven abstract.

Tunning your Maven proyect

Maven standard folders

Managing dependencies with Maven

Adding a nature in Eclipse

Maven profiles inheritance

DISCLAMER: THIS ENTRY STAYS HERE FOR HISTORICAL REASONS, BUT ITS CONTENT IS NOT ACCURATE (MILD WAY OF SAYING IT'S WRONG).

APOLOGIES FOR ANYBODY AFFECTED, IT WAS HONESTLY WRITTEN AND BADLY MAINTAINED.

As far as I have investigated, Maven profiles are not inherited, but there is a mechanism alike.

Let's build an example:

Parent pom.xml

<profiles>
    <profile>
        <id>profileA</id>
        <activation>whatever</activation>
        <build>[whatever]</build>
    </profile>

    <profile>
         <id>profileB</id>
        <activation>whatever</activation>
        <build>[whatever]</build>
    </profile>
</profiles>



Let's assume there's a child project, without specific profiles configuration.

It isn't a crazy idea imagine that if a project inherits from a parent pom like above, when activation conditions are met, child project should enjoy the build profiled configuration, but IT DOESN'T HAPPEN.

The only way to get something similar is configuring "profiles" and "activation" in child project, and the same "profiles" with same "id", with "resources" and "configuration" associated to that profile in parent pom.
This way, when a child project is compiling, profiles are activated and, with a profile active, it is actived in parent pom.


Example 2:

Parent pom.xml
<profiles>
    <profile>
        <id>profileA</id>
        <build>[whatever]</build>
     </profile>
     <profile>
        <id>profileB</id>
        <build>[whatever]</build>
    </profile>
</profiles>

Child  pom.xml (assume a parent pom configuration in this project)
<profiles>
    <profile>
        <id>profileA</id>
        <activation>whatever</activation>
    </profile>

    <profile>
        <id>profileB</id>
        <activation>whatever</activation>
    </profile> </profiles>
</profiles>


Notice that parent pom does not have activation conditions (they would not work either), and child pom does not have any configuration, because you wanted a parent pom for a common configuration.

This way, you have got something similar to Profiles inheritance.


Alberto Navarro.
@anavarro_prof
looking4q.blogspot.com