martes, 8 de diciembre de 2015

Low cost software organization for open source projects: Allocation and provision.

This is a quite difficult topic to explain, as this problem is solved differently if you are in a medium sized company with a decent IT department, or you're sitting alone in your sofa and all your infrastructure is a microserver at the corridor. And I'm writing this from my sofa for tiny companies or personal projects.

To be honest, there's little point to analyse Allocation (of vms) and Provision (of software) separately, as we will find solutions that brings both aspects integrated or at least highly oriented to one another.

Allocation 

Requirements for allocation:

  • It can deal with both cloud and on-premise infrastructure.
  • Command line interface available. 
  • Low effort in installation and maintenance.
  • Low overhead, complex solution will require several boxes running the orchestration.
Non requirements, but worth to highlight that they are not.
  • Autoscaling is not required.

Candidates:

  • AWS ECS (Elastic Container Service) or EB (Elastic Beanstalk)
    •  Obviously this doesn't fulfil the main requirement, if we used this solution, we wouldn't be able to reuse it on-premise.
  • Docker
    •  Docker has many facets, and among them, a remote-api call might simulate allocation of resources. Not sure if same interface works from AWS.
  • Vagrant
    •  Perfect tool (or at least the most suitable I know) for the job, as it offers allocation with more than a handful of backends, among them, AWS, libvirt, and docker itself. It also offers a plugins system for hooking different provisioners. 
  • Manual
    • It's an option, isn't it? You could allocate resources manually from both cloud and on-premise hardware.
  • Terraform
    • I haven't used this tool, but I think you should if you're still facing the same problem that I am. It support several frontends, but I haven't really found if libvirt is supported.

 

Provision

Requirements for provision:

  • It must protect sensitive information, it can't be exposed in an open-source repository.
  • Easy to manage / change / expand.
  • Linux platform is the only one required.
  • Not incompatible with allocation system.

Candidates:

  • Chef
    • I'm clearly biased towards this provisioner, as it offers free in-cloud service for small number of users (not only chef-solo), it's easily integrated to Vagrant and you end up working in a programming language.
    • Secrets are managed in data bags and encrypted data bags, stored in your computer (chef-solo) or in the cloud (chef-server).
    • Ruby based.
  • Ansible
    •  I have less experience with Ansible than with Chef, bear that in mind.
    • Serverless, like chef-solo
    • Yml oriented, it can be extended with Python.
    • Secrets are encrypted and stored with the rest of the configuration data, somewhere in your computer.
  • Puppet
    • Even less experience with puppet, but for some reason, I don't know anybody that uses it.
    • I can only recommend some reading if you are unfamiliar with Chef, Ansible and Puppet, just in case Puppet is better, but I cannot recommend the unknown.
  • Docker
    • What's docker doing here? Well, it might perfectly be consider a provisioning engine as long as you're actually automating a manual installation of software in a virtual environment.
    • Secrets need, however, to be managed externally, or be sure you're pushing to your private repository.
  • Docker-compos
    • A even more beautiful way of relating software in our boxes, specially if, like me, you think that consolidating different microservices in one box is going to make you save money.
    • Again secrets needs to be managed externally.
  • Manual
    • No.

 

Best options:

The one chosen by me (examples coming soon):

  • Vagrant + Chef (cloud) + Docker
    • Vagrant + libvirt for my microserver (allocation < 30 seconds)
    • Same Vagrant + aws for the cloud (not yet explored)
    • Chef installs docker, pulls the right image, and configures it to start the containers on start.
    • Docker contains the software with some placeholders for environment variables containing passwords and sensitive data (therefore docker images are public).

Adopt (other options I'd use):

  • Vagrant + Chef or Puppet
    • WebUI management of nodes, configuration and secrets
    • Push and pull modes available
  • Vagrant + Ansible
    • Less help, more manual

Assess (Investigate before adopting):

  • Terraform +  Chef or Puppet or Ansible
  •  Docker as platform + Chef or Puppet or Ansible

Hold (Don't bother):

  • Anything + Manual
    • You'll fill tired soon

Not considered for being too big or complex for small projects:

  • Cloudfoundry
  • Kubernetes
  • Mesos

domingo, 6 de diciembre de 2015

Low cost software organization for open source projects: Source Code Version Control System

There's no way you can think of developing any product and not use any Control Version System at all, too many benefits at a really low cost.

Requirements for our CVS would be as follow:
  • High Availability
  • Backup
  • Public repositories for our open source projects.
  • Private repositories for (maybe) some private configuration data.
  • Accessible 
  • Git, it's the industry standard right now and it'll cover your software requirements 99.99% of times.
And the possible options come in two different buckets
  • Self-Hosted:
    • Seriously do you think a tiny company can support the complexity (and maybe cost if you use the cloud) of backing up a hosted repository and keeping it open to anywhere else?
As a service:
  • Github:
    • Popular and ticks almost all the requirements, except for free private repositories. I'm using github for my opensource project, yes.
  • BitBucket:
    • Ticks all the boxes as well, plus a number of free private repositories, enough for some sensitive information we might want to store. I'm using bitbucket as well for my Jenkins configuration auto-backup.
So, as first step in our new low cost organization for open source projects, I'd recommend BitBucket with the information I have today.

viernes, 4 de diciembre de 2015

Low cost software organization for open source projects

This is an effort I've been willing to do for long time, but it's now when I think I have something like a solution.

I'll be creating an entry per topic to cover if you want ideas about how to create not your application, as I assume you know your stuff, but everything around your project that makes it a solution.

My own game will be my example throughout these stages, its name is "Cabo Trafalgar" and I've talked about it enough already, there's a link above if you're interested on a 3D sailing simulation made in Java.

The topics I expect to cover are as follow:
  1. Code repository
  2. Allocation and provision
  3. Continuous integration
  4. Installation of software / Configuration management
  5. Deployment and platforms
  6. Continuous delivery
  7. Logging
  8. Monitoring

And the criteria we're going to prioritize
  1. Low price, free when possible.
  2. Secure, configuration values safe and far from code.
  3. Stateless / Easy to recover / Easy to reproduce.
As a result for this process, you should be able to have an enterprise-quality / almost production ready solution running for really little money for your small project.

jueves, 13 de agosto de 2015

Nifty-flow explanation

My first open-source collaboration!


When I started "Cabo Trafalgar", it took me three tries to find the technology that made it possible. I started with Irrlicht, and I also took a look to another library I cannot recall anymore. Only when I discovered that the language that paid my bills was really able to render native 3d and really nice water I adventured on JMonkeyEngine with Java.

Once you start with JME, you don't have many options for creating rich graphic interfaces, and the one that highlights is, needless to say, nifty-gui. Easy to use (I find easier to make examples using JME3 than raw nifty-gui libraries, that easy!) and good documentation and examples. But no doubt it was my webby background I absolutely missed some features:
  •  When you're defining a screen, you're pretty much defining its connections, in my words, "flow of screens". That actually reduces by a lot the possible re-usability of screens in other parts of your game, or other games whatsoever.
  • You can create screens by static xml, or by dynamic java, but web developers have long gone over these restrictions, and we have like a dozen template languages to mix code and static text. At the end of the day, xml for screen is nicer and easier to interpret than crude java code (no offense).
  • Remember, when enumerate, always use 3, 7 or 10 bullet dots, force it if necessary :P
At this point, I had little options, I could have dropped my project, but I didn't feel like doing it again. I could have renegotiated my use cases to make it more cumbersome to manage repeated screens in different parts of the game, or... I could actually build what I need on top of nifty-gui.

For first time I was actually solving my own problems instead of waiting for somebody else to make that for me, and it didn't take me long to realize that it was actually possible.

The use case:

For my game, I needed the user to walk one or two screens, one after the other, until find the "hall", a screen that contains every mode the player can play. In my case, I have a "CounterClock" game and a "Windtunnel" game, both accessible from the same "Menu" screen.

Once you choose "CounterClock", user must "Select profile", then "Select ship", then "Select map" and finally "Select controls", then play.
Once you choose "Windtunnel", user must "Select ship", then "Select controls", then play.

It's easy to spot that I really wish to reuse some screens, and the flow of screens will be linear most of the times in as many examples I can think.

Implementation:

Just take the base of Struts, you have a central controller that will receive every interaction with the user. This controller has, somewhere, the capacity to decide what are the user's options given the current status, and using the current input, calculate the next screen to render.

That exact idea made me build the RedirectorScreenController, all it does is redirecting the user to the next calculated screen, from the ScreenFlowManager. My screens will always onNext and onPrevious actually direct to this RedirectorScreenController and it's associated empty screen.

Next, as I'm talking about reusability, I need to differentiate between "screen definition" and "screen instantiation", and bear in mind that the relationship will be 1:N.
My implementation defines a "screen definition" (See ScreenDefinition) as:
  1. a unique name
  2. a way of getting the controller live instance (unique in the system, so far)
  3. a way of getting the screen constructor (a class that actually knows whether the screen is xml or java and executes it)
My implementation defines a "screen instantiation" (See Screen) as:
  1. the flow it belongs to
  2. a name, unique within the flow
  3. the live instance of the controller associated to this screen.
  4. the live instance of the generator associated to this screen.
Every screen has a uniqueScreenId, made from the flow name and the screen name.
In case the flow name is not set, we can safely assume we need to search that screen in our current flow (local search).

Finally, this theoretical system was actually becoming alive, we just need to define flows.
  1. A name for the flow, unique
  2. A sequence of screen definition names
  3. An optional screenUniqueId parent
We need an initial flow, that's the flow without a parent, we can only have one, the ball is rolling!

If we declare more flows, we need to "hang" them from a concrete existing screen, either from the root flow, or another flow.

We can always query the ScreenFlowManager (See ScreenFlowManagerImpl for implementation details) for our options, it'll tell us whether we can continue forward, and if other flows are available from this screen.

Internally, a JGrapht (in-memory graph database) instance have all the information to know where are you, what are your options and what's your next state from current state and input.

User input

Maybe it's a bit annoying, but there is at least another element to cover. How do we communicate the ScreenFlowManager what's our next move?

I only managed to solve this problem by "injecting" the same ScreenFlowManager into the controllers (not even automagically) and allow a "setNextScreenHint" method to tell the flow manager what's your intention.

Instance resolution

I gently let this for the end.
Do you remember I mentioned "a way of getting the controller live instance" when I was talking about ScreenDefinition?. I was ambiguous because this library is quite technology agnostic, so much, that you have to provide a way of telling me how to get your live instances.

The easier example, LiveInstanceResolutor, it's an object you feed with "instance name" and "instance", you give it to me, and every time you mention the instance name, nifty-flow will take the instance from the resolutor. It's a f*****g map.

StaticScreenGeneratorResolutor will take your static xml directly, that work is done already.

Nifty-gui provides, however, other resolutor, DefaultInstanceResolutor, nothing more than a "resolutor of resolutors" so you can use several at the same time if you wanted. Find in the example below how you can assign a prefix to later use in your Screen Definitions.

The origin of this mechanism is because I cannot live without Spring, so every time I needed an instance, I'm actually invoking a resolutor that is digging inside the BeanFactory for the right bean, autowired, resourced, initialised and ready to use. This implementation is not provided to avoid Spring dependencies, I'm sorry ;)

How's that looking so far?

public interface ScreenFlowManager {
    String NEXT = "next";
    String PREV = "prev";
    String POP = "pop";

    void addScreenDefinition(ScreenDefinition screenDefinition) throws InstanceResolutionException;

    void addFlowDefinition(String flowName, final Optional<String> screenNameFrom, List<String> flowDefinition);

    String nextScreen();

    void setNextScreenHint(String nextScreenHint);

    Collection<String> getChildren();
}

I wouldn't complain much, 5 method, all them explained above. Easy to use!

Working example (please find the entire code here)

public void simpleInitApp() {
    NiftyJmeDisplay niftyDisplay = new NiftyJmeDisplay(
            assetManager, inputManager, audioRenderer, guiViewPort);
    Nifty nifty = niftyDisplay.getNifty();
    guiViewPort.addProcessor(niftyDisplay);
    flyCam.setDragToRotate(true);

    nifty.loadStyleFile("nifty-default-styles.xml");
    nifty.loadControlFile("nifty-default-controls.xml");

    DefaultInstanceResolutor defaultInstanceResolutor = new DefaultInstanceResolutor();
    ScreenFlowManager screenFlowManager = new ScreenFlowManagerImpl(nifty, defaultInstanceResolutor);

    LiveInstanceResolutor liveInstanceResolutor = new LiveInstanceResolutor();
    defaultInstanceResolutor.addResolutor("static", new StaticScreenGeneratorResolutor(nifty));
    defaultInstanceResolutor.addResolutor("live", liveInstanceResolutor);

    RootScreenController screenController = new RootScreenController().setScreenFlowManager(screenFlowManager).setApplication(this);
    ScreenController screenController2 = new Controller2(screenFlowManager);
    ScreenController screenController4 = new Controller4(screenFlowManager);
    liveInstanceResolutor.addController("root", screenController);
    liveInstanceResolutor.addGenerator("root", new RootScreenGenerator(nifty, screenController, screenFlowManager));
    liveInstanceResolutor.addController("controller1", new Controller1(screenFlowManager));
    liveInstanceResolutor.addController("controller2", screenController2);
    liveInstanceResolutor.addController("controller3", new Controller3(screenFlowManager));
    liveInstanceResolutor.addController("controller4", new Controller4(screenFlowManager));

    liveInstanceResolutor.addGenerator("generator2", new Generator2(nifty, screenController2));
    liveInstanceResolutor.addGenerator("generator4", new Generator4(nifty, screenController4));

    try {
        screenFlowManager.addScreenDefinition(new ScreenDefinition("root", "live:root", "live:root"));
        screenFlowManager.addScreenDefinition(new ScreenDefinition("screen1", "live:controller1", "static:/screen.xml"));
        screenFlowManager.addScreenDefinition(new ScreenDefinition("screen2", "live:controller2", "live:generator2"));
        screenFlowManager.addScreenDefinition(new ScreenDefinition("screen3", "live:controller1", "static:/screen.xml"));
        screenFlowManager.addScreenDefinition(new ScreenDefinition("screen4", "live:controller4", "live:generator4"));

        screenFlowManager.addFlowDefinition("root", Optional.<String>absent(), newArrayList("root")); 
        screenFlowManager.addFlowDefinition("screenFlow1", of("root:root"), newArrayList("screen1", "screen2", "screen3", "screen4"));
        screenFlowManager.addFlowDefinition("screenFlow2", of("root:root"), newArrayList("screen1", "screen4"));

        nifty.addScreen("redirector", new ScreenBuilder("start", new RedirectorScreenController().setScreenFlowManager(screenFlowManager)).build(nifty));
        nifty.gotoScreen("redirector");

    } catch (InstanceResolutionException e) {
        e.printStackTrace();
    }


}

In few words:
  1. Create and feed your resolutors with the information you'll need from the screen definitions.
  2. Create your Screen Definitions, I have defined 5 screens, two of them from the same XML and other three generated from java code. Here lays the potential of nifty-flow, the static screens would be reusable even across projects, without knowing what screen to link forwards and backwards.
  3. Create your Screen Flows, I have a root one with one screen, then from that screen, another two reusing some screens.
  4. Add redirector (with that name) as "start" and let the magic happen.
 Final words

There's some more work to do. I implicitely create NEXT and PREV links between screens, but I'd rather have implemented a mechanism to push every movement to a stack, and a POP would always return to the previous screen, this would allow us to jump to the middle of another flow and being able to come back (PREV is "statically" linked to the same screen, while POP could be any invoker).

From the features I was missing in nifty-gui, I still keep missing some help templating XML files, instead of making complex screen from java only, that's something I'd thrill to implement, but god knows I don't have the time now.

Hope you enjoy of this library

Alber

sábado, 2 de mayo de 2015

Some lessons learned: Gradle



After a year dealing with Gradle in different companies, I think it's high time to start collecting some patterns and suggestions.

1) Avoid redundant folder declaration.

Remember, apply plugin java already sets up some defaults folder, you don't need to declare them if you don't plan to change them.

Recommended
apply plugin: 'java'

Redundant
apply plugin: 'java' 

sourceSets {    
    main {        
        java {            
            srcDir 'src/main/java'        
        }        
        resources {            
            srcDir 'src/main/resources'        
        }    
    }     

    test {        
        java {            
            srcDir 'src/test/java'        
        }        
        resources {            
            srcDir 'src/test/resources'        
        }    
    }
}

Groovy plugin and scala plugin also declares other folders, just read some documentation before start using them.


2) Choose your syntax for dependency management.

Gradle allows you two syntax for dependency declaration.
  • Extended (group: 'ws.lazylogin', name: 'lazylogin-core', version: '1.0.0')
  • Compact ('ws.lazylogin:lazylogin-core:1.0.0').
Be consistent, your users and colleagues will be pleased.

I prefer the compact one :)


3) Your users don't need to know project details to compile it.

Any project should be able to clean + build from the code and a JVM in place. Ideally nothing else should be required.

Although some projects will inevitably break this suggestion, try not to.

Some examples of build breaking customs.
  1. Usage of variables declared in ${user.home}/.gradle/gradle.properties that  don't have default value in build.gradle or exist in project's gradle.properties.
    • This would fail for anyone not owning / not knowing what variables to include / contaminate hi/er user gradle properties file. No no no no no.
  2.  Assumption of executables in the PATH or some predefined location.
Try your harder to make your project easy to build.


4) Prevent incorrect dependency versioning

Declare your dependencies and version in a single place (that was a good idea in Maven's universe) and reuse those dependencies by its variable name.

The risk of not following this advice is the likelihood you'll end up using different versions for the same library in different modules in the same project, and you don't want that, neither do I.

Solution would look like this:

Dependency declaration (they can be declared in a different file)
ext{   
    //VERSIONS
    v = [           
        spring: '3.2.0.RELEASE',
        jme3: '3.1.0-snapshot-github'
    ]   

    deps = [           
        //SPRING           
        spring_core: 'org.springframework:spring-core:'+v.spring,
        spring_beans: 'org.springframework:spring-beans:'+v.spring,
        spring_context: 'org.springframework:spring-context:'+v.spring,
        //JME3           
        jme3_core: 'com.jme3:jme3-core:' + v.jme3,           
        jme3_effects: 'com.jme3:jme3-effects:' + v.jme3
    ]
}

Dependency usage for any subproject
dependencies {   
    compile (           
        project(':mod-api'),      
        deps.jme3_core,           
        deps.jme3_effects,           
        deps.spring_beans,           
        deps.spring_core )
}

Instead of the more error prone way:

Dependency usage for a given subproject A
dependencies {   
    compile (           
        project(':mod-api'),                        
        'com.jme3:jme3-core:3.1.0-snapshot-github',           
        'com.jme3:jme3-effects:3.1.0-snapshot-github',           
        'org.springframework:spring-beans:3.2.0.RELEASE',           
        'org.springframework:spring-core:3.2.0.RELEASE' )
}

Dependency usage for a given subproject B
dependencies {   
    compile (           
        project(':mod-api'),   
        'com.jme3:jme3-core:3.1.0-snapshot-github',           
        'com.jme3:jme3-effects:3.1.0-snapshot-github',           
        'org.springframework:spring-beans:3.1.0.RELEASE',           
        'org.springframework:spring-core:3.2.0.RELEASE' )
}


5) Group your dependency by dependency configuration.

It's nicer :)

Dependencies grouped by configurations
compile (           
    project(':mod-api'),           
    libs.lazylogin_common_context,           
    libs.nifty,           
    libs.jme3_core,           
    libs.jme3_effects,           
    libs.spring_beans,           
    libs.spring_core,           
    libs.jackson_dataformat_yaml,           
    libs.jackson_databind
)
runtime (
    libs.nifty_default_controls,           
    libs.eventbus,           
    libs.auto_value))    

Dependencies ungrouped (and disorganized)
compile project(':mod-api')          
compile libs.lazylogin_common_context,           
compile libs.nifty,           
compile libs.jme3_core,           
compile libs.jme3_effects,           
compile libs.spring_beans,           
compile libs.spring_core,     
runtime libs.nifty_default_controls,      
compile libs.jackson_dataformat_yaml,           
compile libs.jackson_databind,           
runtime libs.eventbus,           
runtime libs.auto_value)

(Not possible when you want to exclude dependencies from dependencies).


martes, 13 de enero de 2015

Cabo Trafalgar v0.9 Released!!

Changes:
Connected with LazyLogin and RecordServer services to provide online services.
Profile manager screen.
Control selection to be ready for more than keyboard, and key selection screens.
New Ship introduced, with weight balance control apart from MainSail and Rudder.
Map selection screen now with descriptions and graphic maps for a better understanding.
Introduced tidal sea movement (sea currents) as a possible element in the maps.
Win installer.
Enhanced dynamics, more realistic.
A thousand bugs and aesthetical details fixed. Probably another two thousands created.
And an exit button!!

More information:
http://cabo-trafalgar.blogspot.co.uk/2015/01/cabo-trafalgar-v09-released.html