Roll your own Continuous Integration System (C.I.S.)
Content:
Understand that it is necessary, you need it, you deserve it, you want Subversion to control your code revisions. But you are clumsy enough (just as me) to be able to build your own repository at home with svn and Apache.
Thank God I found some brilliant tutorials and products, all of them, of couse, open source. I was intended to use only Tomcat6 and war packaged software, but both Apache and uSvn really deserve some Megas in the hard drive of your heart (and in the Continuous Integration Server's one).
I liked it too much because, mainly, because when I read about an easy tutorial, it takes me some days or weeks to accomplish its goal, ufff, I am not exaggerating, I am often silly. But it didn't happen this time with this uSVN tutorial and product.
By the way, what is uSVN?
"User-Friendly USVN is a web interface written in PHP used to configure Subversion repositories.It's goal is to ease the creation of new projects without having to use the command line interface, therefor, maybe without privileged access to the server. USVN will then generate the list of users allowed to access your source code. This enable the delegation of rights to administrate Subversion repositories."
From its official web:
https://trac.usvn.info/wiki/Documentation/Install
Another point of view from one of my favourites and abandoned blogs:
http://killerconfigurations.blogspot.com/2009/08/easy-svn-web-administration.html
Enjoy :)
lunes, 21 de febrero de 2011
domingo, 20 de febrero de 2011
Roll your own Continuous Integration System (C.I.S.): Basic Tomcat configuration - Basic application manager and user permissions
Roll your own Continuous Integration System (C.I.S.)
Content:
Our tomcat installation provided an application manager, with all basic function to administrate the life cycle of our applications.
Top advantages:
1) No need to leave applications in a folder to deploy them, this application will copy there for us.
2) Manage start, stop, redeploy and undeploy of every application comfortably.
3) Secured, only users with "manager" or "manager" role can reach this application.
4) Web and REST interface.
As said in the third point, this application is secured, using basic tomcat configuration. Take a look to your conf/tomcat-users.xml, read the useful comments inside <tomcat-users> tags:
<!--
NOTE: By default, no user is included in the "manager" role required
to operate the "/manager" web application. If you wish to use this app,
you must define such a user - the username and password are arbitrary.
-->
<!--
NOTE: The sample user and role entries below are wrapped in a comment
and thus are ignored when reading this file. Do not forget to remove
<!.. ..> that surrounds them.
-->
Now uncomment the next block of xml user declarations, although I rather keeping only one line like this:
<user username="tomcat" password="tomcat" roles="manager"/>
and I usually erase the others, in order to keep it clean.
Once done this, restart the server (service tomcat6 restart) and try to enter to your server:port/manager/html (usually localhost:8080/manager/html). type your user and password for a manager user (in my example, "tomcat" and "tomcat") and discover one of the most useful tools in tomcat administration.
The first half of the page is devoted to list your application and their status.
Scroll down a bit, you will find two ways to deploy an application, by local address in tomcat's computer, or by uploading a war file from your computer, that might not be always the same.
Now you are ready to deploy easily all the applications for your Continuous Integration System :)
pd: A stopped application will start if Tomcat is rebooted. That is important and not very intuitive.
Content:
Our tomcat installation provided an application manager, with all basic function to administrate the life cycle of our applications.
Top advantages:
1) No need to leave applications in a folder to deploy them, this application will copy there for us.
2) Manage start, stop, redeploy and undeploy of every application comfortably.
3) Secured, only users with "manager" or "manager" role can reach this application.
4) Web and REST interface.
As said in the third point, this application is secured, using basic tomcat configuration. Take a look to your conf/tomcat-users.xml, read the useful comments inside <tomcat-users> tags:
<!--
NOTE: By default, no user is included in the "manager" role required
to operate the "/manager" web application. If you wish to use this app,
you must define such a user - the username and password are arbitrary.
-->
<!--
NOTE: The sample user and role entries below are wrapped in a comment
and thus are ignored when reading this file. Do not forget to remove
<!.. ..> that surrounds them.
-->
Now uncomment the next block of xml user declarations, although I rather keeping only one line like this:
<user username="tomcat" password="tomcat" roles="manager"/>
and I usually erase the others, in order to keep it clean.
Once done this, restart the server (service tomcat6 restart) and try to enter to your server:port/manager/html (usually localhost:8080/manager/html). type your user and password for a manager user (in my example, "tomcat" and "tomcat") and discover one of the most useful tools in tomcat administration.
The first half of the page is devoted to list your application and their status.
Scroll down a bit, you will find two ways to deploy an application, by local address in tomcat's computer, or by uploading a war file from your computer, that might not be always the same.
Now you are ready to deploy easily all the applications for your Continuous Integration System :)
pd: A stopped application will start if Tomcat is rebooted. That is important and not very intuitive.
viernes, 18 de febrero de 2011
Roll your own Continuous Integration System (C.I.S.): Basic Tomcat configuration - JMX
Roll your own Continuous Integration System (C.I.S.)
Content:
About the content of jmxremote.password, a simply plain text with pairs of name password like this:
monitorRole tomcat
controlRole tomcat
About the content of jmxremote.access, a simply plain text with pairs of name policy like this:
monitorRole readonly
controlRole readwrite
Now you are able to see Tomcat with codename Catalina in your JConsole panel if you are running it locally, or throw remote connection.
Sorry for the delay, it really got me on my nerves to find the way of populating the jmx service!
Some help for Windows:
http://www.askproductions.no/blogg/2009/02/add-jmx-support-to-tomcat-6-on-windows/
Content:
I continue with this tutorial, this time I will assume you are working with Linux, in order to simplify the installation and configuration explanations. I do not have any license of Windows (no, Verbatim is not a Windows licenser).
JMX is one of those fabulous tools+protocol that almost anybody knows and everybody should know.
JMX is one of those fabulous tools+protocol that almost anybody knows and everybody should know.
It will help you to monitor any application since Java 1.5, it is based on reflexive calls and that guaranties uniform calling to unknown classes.
"The Java Management Extensions (JMX) API is a standard API for management and monitoring of resources such as applications, devices, services, and the Java virtual machine. The JMX technology was developed through the Java Community Process (JCP) as Java Specification Request (JSR) 3, Java Management Extensions, and JSR 160, JMX Remote API.
Typical uses of the JMX technology include:
Consulting and changing application configuration
Accumulating statistics about application behavior and making them available
Notifying of state changes and erroneous conditions.
The JMX API includes remote access, so a remote management program can interact with a running application for these purposes."
(from here.)
There's an easy-to-use tool for newbies called JConsole (jconsole.exe in Windows or jconsole in Linux), it is brought by JDK >1.5, you will find it in the same folder than java.exe or javac.exe. Find it, execute it, right now, do not wait one more minute.
The first you discover is the connection panel, first half for local applications, second half for remote ones. Local connections will never be empty due to the JConsole application itself is JMX-monitorizable. A monitorizable monitor!! Yehaaa!!!
Close to a dream, data, data, dataaaaaaaa :)
All about JConsole, it shows cpu consumption, memory usage, vm parameters, and MBeans...
MBeans are some special entities in java allowed to be exported like statistics and methods to be run. Indeed, cpu and memory statistic data are extracted from some well-known MBeans, and JConsole takes advance of knowing those MBeans for plotting some graphs. But it could could any metric you define.
Some day, in near future, I will describe how to populate application metrics and methods for being executed from JConsole or your own JMX protocol consumer.
Do not be shy, press that button, you will provoke a Garbage Collector call, and some memory will be freed. Change to memory tab to confirm it.
Summarizing, I find it quite important to monitor our Tomcat server in order to see memory consumption, CPU, and our application exported beans.
How to get it? It is not difficult.
http://tomcat.apache.org/tomcat-6.0-doc/monitoring.html#Enabling_JMX_Remote
It has taught me longer than expected to make it run, and the cause is that a line is missing for empty Tomcat6 installation.
The final configuration for Tomcat6 JMX enabled and simply secured is:
It has taught me longer than expected to make it run, and the cause is that a line is missing for empty Tomcat6 installation.
The final configuration for Tomcat6 JMX enabled and simply secured is:
# You can pass some parameters to java here if you wish toJAVA_OPTS="$JAVA_OPTS -Djava.awt.headless=true -Dfile.encoding=UTF-8 -Xms1536m -Xmx1536m -XX:NewSize=256m -XX:MaxNewSize=256m -XX:PermSize=256m -XX:MaxPermSize=256m -XX:+DisableExplicitGC"
# Enabling JMX CATALINA_OPTS="$CATALINA_OPTS -Dcom.sun.management.jmxremote.port=8091 -Dcom.sun.management.jmxremote.password.file=$CATALINA_BASE/conf/jmxremote.password -Dcom.sun.management.jmxremote.access.file=$CATALINA_BASE/conf/jmxremote.access -Dcom.sun.management.jmxremote.ssl=false -Djava.rmi.server.hostname=192.168.1.106"
About the content of jmxremote.password, a simply plain text with pairs of name password like this:
monitorRole tomcat
controlRole tomcat
About the content of jmxremote.access, a simply plain text with pairs of name policy like this:
monitorRole readonly
controlRole readwrite
Note some changes:
1) JAVA_OPTS is redefined with $JAVA_OPTS, in order to preserve some possible variable preset. The same for CATALINA_OPTS.
2) -Djava.rmi.server.hostname is defined in CATALINA_OPTS, with the server address. If not present, JMX module will only handle local calls, even though you set com.sun.management.jmxremote.port.
3) "com.sun.management.jmxremote" is not needed if "com.sun.management.jmxremote.port is defined."
4) Every variable set in a single line.
5) jmxremote.password and jmxremote.access should be only readable (and only readable) by tomcat user.
Of course, you can apply these instructions to the tomcat6.conf file in Linux, or Start->All programs->Apache Tomcat 6.0->Configure Tomcat. "Java" tab, for Windows, as we did before for memory configuration.
1) JAVA_OPTS is redefined with $JAVA_OPTS, in order to preserve some possible variable preset. The same for CATALINA_OPTS.
2) -Djava.rmi.server.hostname is defined in CATALINA_OPTS, with the server address. If not present, JMX module will only handle local calls, even though you set com.sun.management.jmxremote.port.
3) "com.sun.management.jmxremote" is not needed if "com.sun.management.jmxremote.port is defined."
4) Every variable set in a single line.
5) jmxremote.password and jmxremote.access should be only readable (and only readable) by tomcat user.
Of course, you can apply these instructions to the tomcat6.conf file in Linux, or Start->All programs->Apache Tomcat 6.0->Configure Tomcat. "Java" tab, for Windows, as we did before for memory configuration.
Remember, open Firewall for your JMX port if necessary!
Now you are able to see Tomcat with codename Catalina in your JConsole panel if you are running it locally, or throw remote connection.
Sorry for the delay, it really got me on my nerves to find the way of populating the jmx service!
Some help for Windows:
http://www.askproductions.no/blogg/2009/02/add-jmx-support-to-tomcat-6-on-windows/
lunes, 14 de febrero de 2011
Half a Thousand Thanks
Two month after beginning this blog, I have a modest visit rate, fair enough to feed my ego and make me very happy. Thanks!
By the way, Who are you? :)
pd: Javier del Palacio was here (to correct my English).
domingo, 6 de febrero de 2011
Roll your own Continuous Integration System (C.I.S.): Basic Tomcat configuration - Memory
Roll your own Continuous Integration System (C.I.S.)
Content:
In previous post, you installed Tomcat and let it running in your computer. Now it is time for a first configuration. We are going to focus in three usual needs.
1) We need more memory, always.
2) We need to monitor our system, we will use built-in JMX server.
3) We need permissions to get the manager application run. That manager application will make it easier the deploy task in near future.
Enlarge your memory
In spite of the amount of memory your system has, Java limits some thresholds for its applications. Default size for Sun's JVM PermGen space are 64M, and this is little memory for Artifactory, Hudson or Sonar. I do not remember which gives the problem :P
Objectives:
- Avoid PermGen exceptions by classloaders or application leaks, at least, in development environment.
- Increase available memory, some of the applications we will install are really hungry. Our Continuous Integration System will do some compilations which take much memory and we will foresee that need.
What have I added?:
All these are system parameters, they do not do anything by themselves, but that information is available for all applications and JVM. This way, when an application or the JVM itself is running, they can read system parameters and change their behavior.
java.awt.headless=true:
Headless mode is a system configuration in which the display device, keyboard, or mouse is lacking. Sounds unexpected, but actually you can perform different operations in this mode, even with graphic data.
http://java.sun.com/developer/technicalArticles/J2SE/Desktop/headless/
-Dfile.encoding=UTF-8:
You are specifying the default encoding for this JVM. From now on, if anything different is said, your applications will read and write and String will be built using that internal encoding.
By specifying this, you do not have to speculate with customer OS-dependent configuration.
-XX:PermSize=256m and -XX:MaxPermSize=256m modifies structure and behavior of JVM memory. I owe you a better explanation.
More options:
http://www.oracle.com/technetwork/java/javase/tech/vmoptions-jsp-140102.html
Tomcat6 performance guide:
http://tomcat.apache.org/articles/performance.pdf
Content:
In previous post, you installed Tomcat and let it running in your computer. Now it is time for a first configuration. We are going to focus in three usual needs.
1) We need more memory, always.
2) We need to monitor our system, we will use built-in JMX server.
3) We need permissions to get the manager application run. That manager application will make it easier the deploy task in near future.
Enlarge your memory
In spite of the amount of memory your system has, Java limits some thresholds for its applications. Default size for Sun's JVM PermGen space are 64M, and this is little memory for Artifactory, Hudson or Sonar. I do not remember which gives the problem :P
Objectives:
- Avoid PermGen exceptions by classloaders or application leaks, at least, in development environment.
- Increase available memory, some of the applications we will install are really hungry. Our Continuous Integration System will do some compilations which take much memory and we will foresee that need.
- For any Linux Tomcat6: Open /usr/share/tomcat6/conf/tomcat6.conf (or $TOMCAT_HOME/conf/tomcat6.conf if you installed it in other folder).
- Check that JAVA_OPTS is not set, I mean, there is a "#" in front of every JAVA_OPTS= line or any JAVA_OPTS line exists. If it is really not set, add this line:
JAVA_OPTS="-Djava.awt.headless=true -Dfile.encoding=UTF-8 -XX:PermSize=256m -XX:MaxPermSize=256m"
else, just merge available with new parameters, but if the system is just installed, it shouldn't exist.
- For Windows Tomcat6: Easier to find, Start->All programs->Apache Tomcat 6.0->Configure Tomcat. Then click "Java" tab.
- Check if any of next parameters are set, I mean, if that line exists. No "#" possibility. If the parameter exists, replace with new value, else simply add.
-Djava.awt.headless=true-Dfile.encoding=UTF-8-XX:PermSize=256m-XX:MaxPermSize=256m
What have I added?:
All these are system parameters, they do not do anything by themselves, but that information is available for all applications and JVM. This way, when an application or the JVM itself is running, they can read system parameters and change their behavior.
java.awt.headless=true:
Headless mode is a system configuration in which the display device, keyboard, or mouse is lacking. Sounds unexpected, but actually you can perform different operations in this mode, even with graphic data.
http://java.sun.com/developer/technicalArticles/J2SE/Desktop/headless/
-Dfile.encoding=UTF-8:
You are specifying the default encoding for this JVM. From now on, if anything different is said, your applications will read and write and String will be built using that internal encoding.
By specifying this, you do not have to speculate with customer OS-dependent configuration.
-XX:PermSize=256m and -XX:MaxPermSize=256m modifies structure and behavior of JVM memory. I owe you a better explanation.
More options:
http://www.oracle.com/technetwork/java/javase/tech/vmoptions-jsp-140102.html
Tomcat6 performance guide:
http://tomcat.apache.org/articles/performance.pdf
lunes, 31 de enero de 2011
Maven profiles inheritance
Content:
DISCLAMER: THIS ENTRY STAYS HERE FOR HISTORICAL REASONS, BUT ITS CONTENT IS NOT ACCURATE (MILD WAY OF SAYING IT'S WRONG).
APOLOGIES FOR ANYBODY AFFECTED, IT WAS HONESTLY WRITTEN AND BADLY MAINTAINED.
As far as I have investigated, Maven profiles are not inherited, but there is a mechanism alike.
Let's build an example:
Parent pom.xml
Let's assume there's a child project, without specific profiles configuration.
It isn't a crazy idea imagine that if a project inherits from a parent pom like above, when activation conditions are met, child project should enjoy the build profiled configuration, but IT DOESN'T HAPPEN.
The only way to get something similar is configuring "profiles" and "activation" in child project, and the same "profiles" with same "id", with "resources" and "configuration" associated to that profile in parent pom.
This way, when a child project is compiling, profiles are activated and, with a profile active, it is actived in parent pom.
Example 2:
Parent pom.xml
Child pom.xml (assume a parent pom configuration in this project)
Notice that parent pom does not have activation conditions (they would not work either), and child pom does not have any configuration, because you wanted a parent pom for a common configuration.
This way, you have got something similar to Profiles inheritance.
Alberto Navarro.
@anavarro_prof
looking4q.blogspot.com
| Maven abstract. |
| Tunning your Maven proyect |
| Maven standard folders |
| Managing dependencies with Maven |
| Adding a nature in Eclipse |
| Maven profiles inheritance |
DISCLAMER: THIS ENTRY STAYS HERE FOR HISTORICAL REASONS, BUT ITS CONTENT IS NOT ACCURATE (MILD WAY OF SAYING IT'S WRONG).
APOLOGIES FOR ANYBODY AFFECTED, IT WAS HONESTLY WRITTEN AND BADLY MAINTAINED.
As far as I have investigated, Maven profiles are not inherited, but there is a mechanism alike.
Let's build an example:
Parent pom.xml
<profiles>
<profile>
<id>profileA</id>
<activation>whatever</activation>
<build>[whatever]</build>
</profile>
<profile>
<id>profileB</id>
<activation>whatever</activation>
<build>[whatever]</build>
</profile>
</profiles>
Let's assume there's a child project, without specific profiles configuration.
It isn't a crazy idea imagine that if a project inherits from a parent pom like above, when activation conditions are met, child project should enjoy the build profiled configuration, but IT DOESN'T HAPPEN.
The only way to get something similar is configuring "profiles" and "activation" in child project, and the same "profiles" with same "id", with "resources" and "configuration" associated to that profile in parent pom.
This way, when a child project is compiling, profiles are activated and, with a profile active, it is actived in parent pom.
Example 2:
Parent pom.xml
<profiles>
<profile>
<id>profileA</id> <build>[whatever]</build>
</profile> <profile>
<id>profileB</id> <build>[whatever]</build>
</profile>
</profiles>Child pom.xml (assume a parent pom configuration in this project)
<profiles>
<profile>
<id>profileA</id>
<activation>whatever</activation> </profile>
<profile>
<id>profileB</id>
<activation>whatever</activation>
</profile> </profiles> </profiles> Notice that parent pom does not have activation conditions (they would not work either), and child pom does not have any configuration, because you wanted a parent pom for a common configuration.
This way, you have got something similar to Profiles inheritance.
Alberto Navarro.
@anavarro_prof
looking4q.blogspot.com
jueves, 27 de enero de 2011
Software versioning, Apache strategy
In the last few weeks, my job is to make my mates job easier. I have been installing a simple Continuous Integration System, Maven plugins and Eclipse add-ons.
I can't share a lot of this information until I have learned it properly. I would want to show that with good examples and error-solving snippets.
What I can tell you right now is the versioning system of apache, I liked it too much because it is simply, easy to learn and use, easy to learn, teach, memorize..
http://apr.apache.org/versioning.html
This system is used by lots of other software companies, let's explain in few words:
Let's take a random version: 3.2.1, for example.
You resolve a bug, and make a patch for your clients, increment the third number.
You create a feature, but you don't change any of previously existent functionality, increment the second number.
You change a functionality, breaking the contract of your previous versions. increment the first number.
The result might be the following.
Previous version: 3.2.1
You upgrade to 3.2.2
It is supposed to be fully compatible
Previous version: 3.2.1
You downgrade to 3.2.0
It is supposed to be fully compatible. Obviously, it could work worse, there is an absent patch.
Previous version: 3.2.1
You upgrade to 3.3.1 (or 3.3.0)
It is supposed to be fully compatible with the same configuration.
Previous version: 3.2.1
You downgrade to 3.1.1 (or 3.1.x)
Compatibility with prior minor versions is not guaranteed. *
Previous version: 3.2.1
You upgrade to 4.2.1 (or 4.x.x)
Compatibility with different major versions is not guaranteed.
Previous version: 3.2.1
You downgrade to 2.2.1 (or 2.x.x)
Compatibility with different major versions is not guaranteed. *
* The best explanation for this phenomenon is, lets say, 3.2.1 introduced two lines in configuration files, which are parsed with a certain pattern. It could be possible that previous version was not able to parse your configuration with those unknown new lines.
A good coherent versioning system will give free information to your team and customers about how compatible your versions are between them. Avoid marketing formulas for your version, at least, inside your I.T. department. You could also use standard tools for autoincrement versions, like Maven (sooooon).
See you at @anavarro_prof twitter ;)
I can't share a lot of this information until I have learned it properly. I would want to show that with good examples and error-solving snippets.
What I can tell you right now is the versioning system of apache, I liked it too much because it is simply, easy to learn and use, easy to learn, teach, memorize..
http://apr.apache.org/versioning.html
This system is used by lots of other software companies, let's explain in few words:
Let's take a random version: 3.2.1, for example.
You resolve a bug, and make a patch for your clients, increment the third number.
You create a feature, but you don't change any of previously existent functionality, increment the second number.
You change a functionality, breaking the contract of your previous versions. increment the first number.
The result might be the following.
Previous version: 3.2.1
You upgrade to 3.2.2
It is supposed to be fully compatible
Previous version: 3.2.1
You downgrade to 3.2.0
It is supposed to be fully compatible. Obviously, it could work worse, there is an absent patch.
Previous version: 3.2.1
You upgrade to 3.3.1 (or 3.3.0)
It is supposed to be fully compatible with the same configuration.
Previous version: 3.2.1
You downgrade to 3.1.1 (or 3.1.x)
Compatibility with prior minor versions is not guaranteed. *
Previous version: 3.2.1
You upgrade to 4.2.1 (or 4.x.x)
Compatibility with different major versions is not guaranteed.
Previous version: 3.2.1
You downgrade to 2.2.1 (or 2.x.x)
Compatibility with different major versions is not guaranteed. *
* The best explanation for this phenomenon is, lets say, 3.2.1 introduced two lines in configuration files, which are parsed with a certain pattern. It could be possible that previous version was not able to parse your configuration with those unknown new lines.
A good coherent versioning system will give free information to your team and customers about how compatible your versions are between them. Avoid marketing formulas for your version, at least, inside your I.T. department. You could also use standard tools for autoincrement versions, like Maven (sooooon).
See you at @anavarro_prof twitter ;)
Suscribirse a:
Entradas (Atom)