Low cost software organization for open source projects: Allocation and provision.

This is a quite difficult topic to explain, as this problem is solved differently if you are in a medium sized company with a decent IT department, or you're sitting alone in your sofa and all your infrastructure is a microserver at the corridor. And I'm writing this from my sofa for tiny companies or personal projects.

To be honest, there's little point to analyse Allocation (of vms) and Provision (of software) separately, as we will find solutions that brings both aspects integrated or at least highly oriented to one another.

Allocation 

Requirements for allocation:

• It can deal with both cloud and on-premise infrastructure.
• Command line interface available. 
• Low effort in installation and maintenance.
• Low overhead, complex solution will require several boxes running the orchestration.

Non requirements, but worth to highlight that they are not.

• Autoscaling is not required.

Candidates:

• AWS ECS (Elastic Container Service) or EB (Elastic Beanstalk)
•  Obviously this doesn't fulfil the main requirement, if we used this solution, we wouldn't be able to reuse it on-premise.
• Docker
•  Docker has many facets, and among them, a remote-api call might simulate allocation of resources. Not sure if same interface works from AWS.
• Vagrant
•  Perfect tool (or at least the most suitable I know) for the job, as it offers allocation with more than a handful of backends, among them, AWS, libvirt, and docker itself. It also offers a plugins system for hooking different provisioners. 
• Manual
• It's an option, isn't it? You could allocate resources manually from both cloud and on-premise hardware.
• Terraform
• I haven't used this tool, but I think you should if you're still facing the same problem that I am. It support several frontends, but I haven't really found if libvirt is supported.

 

Provision

Requirements for provision:

• It must protect sensitive information, it can't be exposed in an open-source repository.
• Easy to manage / change / expand.
• Linux platform is the only one required.
• Not incompatible with allocation system.

Candidates:

• Chef
• I'm clearly biased towards this provisioner, as it offers free in-cloud service for small number of users (not only chef-solo), it's easily integrated to Vagrant and you end up working in a programming language.
• Secrets are managed in data bags and encrypted data bags, stored in your computer (chef-solo) or in the cloud (chef-server).
• Ruby based.
• Ansible
•  I have less experience with Ansible than with Chef, bear that in mind.
• Serverless, like chef-solo
• Yml oriented, it can be extended with Python.
• Secrets are encrypted and stored with the rest of the configuration data, somewhere in your computer.
• Puppet
• Even less experience with puppet, but for some reason, I don't know anybody that uses it.
• I can only recommend some reading if you are unfamiliar with Chef, Ansible and Puppet, just in case Puppet is better, but I cannot recommend the unknown.
• Docker
• What's docker doing here? Well, it might perfectly be consider a provisioning engine as long as you're actually automating a manual installation of software in a virtual environment.
• Secrets need, however, to be managed externally, or be sure you're pushing to your private repository.
• Docker-compos
• A even more beautiful way of relating software in our boxes, specially if, like me, you think that consolidating different microservices in one box is going to make you save money.
• Again secrets needs to be managed externally.
• Manual
• No.

 

Best options:

The one chosen by me (examples coming soon):

• Vagrant + Chef (cloud) + Docker
• Vagrant + libvirt for my microserver (allocation < 30 seconds)
• Same Vagrant + aws for the cloud (not yet explored)
• Chef installs docker, pulls the right image, and configures it to start the containers on start.
• Docker contains the software with some placeholders for environment variables containing passwords and sensitive data (therefore docker images are public).

Adopt (other options I'd use):

• Vagrant + Chef or Puppet
  
  • WebUI management of nodes, configuration and secrets
  • Push and pull modes available
• Vagrant + Ansible
• Less help, more manual

Assess (Investigate before adopting):

• Terraform +  Chef or Puppet or Ansible
  
•  Docker as platform + Chef or Puppet or Ansible

Hold (Don't bother):

• Anything + Manual
• You'll fill tired soon

Not considered for being too big or complex for small projects:

• Cloudfoundry
• Kubernetes
• Mesos

Low cost software organization for open source projects: Source Code Version Control System

There's no way you can think of developing any product and not use any Control Version System at all, too many benefits at a really low cost.

Requirements for our CVS would be as follow:

• High Availability
• Backup
• Public repositories for our open source projects.
• Private repositories for (maybe) some private configuration data.
• Accessible 
• Git, it's the industry standard right now and it'll cover your software requirements 99.99% of times.

And the possible options come in two different buckets

• Self-Hosted:
• Seriously do you think a tiny company can support the complexity (and maybe cost if you use the cloud) of backing up a hosted repository and keeping it open to anywhere else?

As a service:

• Github:
• Popular and ticks almost all the requirements, except for free private repositories. I'm using github for my opensource project, yes.
• BitBucket:
• Ticks all the boxes as well, plus a number of free private repositories, enough for some sensitive information we might want to store. I'm using bitbucket as well for my Jenkins configuration auto-backup.

So, as first step in our new low cost organization for open source projects, I'd recommend BitBucket with the information I have today.

Low cost software organization for open source projects

This is an effort I've been willing to do for long time, but it's now when I think I have something like a solution.

I'll be creating an entry per topic to cover if you want ideas about how to create not your application, as I assume you know your stuff, but everything around your project that makes it a solution.

My own game will be my example throughout these stages, its name is "Cabo Trafalgar" and I've talked about it enough already, there's a link above if you're interested on a 3D sailing simulation made in Java.

The topics I expect to cover are as follow:

1. Code repository
2. Allocation and provision
3. Continuous integration
4. Installation of software / Configuration management
5. Deployment and platforms
6. Continuous delivery
7. Logging
8. Monitoring

And the criteria we're going to prioritize

1. Low price, free when possible.
2. Secure, configuration values safe and far from code.
3. Stateless / Easy to recover / Easy to reproduce.

As a result for this process, you should be able to have an enterprise-quality / almost production ready solution running for really little money for your small project.