Tag Archives: Cloud Foundry

What is Concourse CI?

This is my third blog in my “What is” series about different products that are part of the Cloud Foundry ecosystem. I discussed Cloud Foundry and BOSH earlier and now it’s time to for he next: What is Concourse CI?

So what is it?

The github tagline for the concourse project is “Continuous thing doer”. Which is quite accurate. Some would call it a Continuous Integration tool. It serves the same purpose as the well known tool Jenkins. It works quite different though. you can find a comparison between Concourse and and other CI tools here so I won’t go into details right now.

What is interesting to know though is that Concourse was born at Pivotal and is considered the standard CI tool for Cloud Foundry and related projects for a while now. The product was born out of necessity. Other tools just couldn’t deliver what the CF development teams needed. And what may even be more important: Other tools don’t follow the design principles all Pivotal and CF software is following. One of the most important ones being: “No snowflakes”.

Snowflake?

As you may know, each snowflake is different from other snowflakes. It’s unique. And that’s fine when we’re talking about real snowflakes. I’s not so fine when it concerns servers, especially if you’re running hundreds of them. If every server is special you have to run a backup for each one of them regularly, you need instructions on how to configure the server when it needs to be rebuild or DR’d. Troubleshooting becomes difficult because you don’t know how it needs to be configured. After all it’s different from all other servers so you have no reference.

In order to avoid snowflakes CF, BOSH and Concourse use text files to store configuration for Apps, Servers and Pipelines. If a server or app fails you can just blow it away and reload from the config file. Done.

If you are using Jenkins for your CI you probably did a lot of specific configuration on the Jenkins server. If you lost the server you would need to spent a lot of time re-configuring it or restore it from a backup. It’s different for Concourse Though. In concourse everything is stored in yml files. Concourse server is gone? Build a new one from scratch and reload your pipelines from yml files. You already know that works fine. After all that’s how the config got there in the first place.

Concourse concepts

Pipelines are first class citizen in Concourse CI. A CI pipeline is basically all the steps that need to be taken to get application code from the code repository all the way to production servers or at least a production release. Steps could be: download the code, build the code, run unit tests, run integration tests, deploy to Cloud Foundry.

concourse pipelines consist of resources and tasks. Jobs are used to compose resources and tasks. The pipline is describes in a yml file. Tasks can be describe in the same yml but are often described in external files. Since all this is stored in the same repo as the application code versioning tasks and pipelines becomes really easy.  For an example take a look at the ci folder in my demo app here. Below is a screenshot of what that pipeline looks like in the Concourse GUI

Screenshot from 2017-04-13 15-23-29

The online documentation for Concourse CI is excellent so I’ll be lazy and give you the link to the description of the concepts here in case you want to know more :).

Try it yourself

Before you run off and try it yourself let me tell you how to interact with Concourse. I already showed you the GUI. But know that the GUI is only intended to give a visual presentation of your pipelines. It is great so show on a big monitor in you dev teams office.

Creating the pipelines and some other configuration things are done through the fly cli. Which is nice, I hate taking my hands off the keyboard :).

If you want to try Concourse out for yourself then running the dockerized version is probably the fastest way to get going. If you read my blog post about BOSH and gave that a go yourself you might want to try to deploy Concourse using BOSH. To help you get started I shared my BOSH manifest below. I couldn’t get the HTTPS part working so I left that out for know.

 

 

What is BOSH?

In a previous post I went into what Cloud Foundry is and why you’d want to use it. What I didn’t go into was some of the magic behind the scenes. For infra minded people like myself this part might be even more exciting than the platform itself.  The thing that makes Cloud Foundry so robust and portal is called BOSH. So What is BOSH?

BOSH is a recursive acronym for “BOSH Outer Shell”. But that doesn’t tell you much about what it does. The bosh.io website explains: “BOSH is an open source tool for release engineering, deployment, lifecycle management, and monitoring of distributed systems.”

What does BOSH do?

It’s kinda hard to put BOSH in a certain box like “cloud management platform” or “software deployment tool”. BOSH does a lot of things: It deploys virtual machines, but it’s not strictly a virtual machine deployment tool. It deploys software but it’s not just a software deployment tool and last but not least it also monitors but it’s definitely not a monitoring tool.

It’s something better. BOSH deploys versioned software into a running infrastructure. The software needs a VM to run on so BOSH also deploys a VM. Once software is deployed it’s important that it keeps running. So BOSH also monitors the software and automatically heals the application when needed. If you accidentally delete a VM that’s part of a software deployment, BOSH will automatically redeploy the VM, install the software and rejoin the cluster.

BOSH components and concepts

A BOSH installation consists of the following componenst:

  • BOSH Director: This is what you could call the “BOSH Server”. It is the main part of the software that is responsible for orchestrating deployments and acting on health events.
  • BOSH Agent: This is a piece of software that runs on every VM deployed by BOSH. It is responsible for all the tasks that happen inside the VM.
  • CPI: The Cloud Provider Interface is a component that implements an API which enables BOSH to communicate with different types of infrastructure. There are CPIs for vSphere, vCloud, Google cloud, AWS and even for rackHD if you want to deploy to phyisical hardware. The CPI basically translates what BOSH wants to do to the specific cloud platform you want to deploy to.

When working with BOSH you’ll use the following constructs:

  • Stemcell: This is a bare bones virtual machine image that includes a BOSH agent. It’s a zip file with some descriptor fields and a machine image. Stemcells are platform specific. So there are stemcells for AWS, vSphere and so on. In the case of a vSphere stemcell you’ll simply find a VMDK packaged in a zip. You can download publicly available stemcells but you can also build your own if you want to.
  • Release: A BOSH release a a bundle of everything that is needed to deploy a specific application excluding the virtual machine templates. So it includes all runtimes, shared libraries and scripts that are needed to get the application running on a stemcell. There are public releases for a lot of opensource software including Cloud foundry.
  • Manifest: This is a YAML file that describes how stemcells and releases will be combined into a deployment. It describes the desired state. If you’re familiar with vRealize Automation, this is basically a blueprint.
  • Deployment: A deployment is basically the execution of a manifest. A deployment can contain many machines. When deploying, BOSH uses the manifest to determine the desired state. If you running the deployment BOSH will determine the current state and will do what is necessary to get to the desired state. This is contrary to what vRealize Automation does. When you change a vRA blueprint, that does not change any of the deployments. But if you change a BOSH manifest and run deploy again for that manifest BOSH will implement whatever changes you made to the desired state.

Can I try it?

Start out with bosh.io The documentation is quite good but the learning curve can be a bit steep. I hope to give you some pointers on how to get it running in another blog post soon.

NLVMUG UserCon Session: The Why, What and How of Automation

On March 16th the Dutch VMUG UserCon took place. Again a big event with around 1000 attendees. And again I had the honor to fill one of the breakout sessions. This year I presented with my co-worker Ruurd Keizer. Our session was titled: “The Why, What and How of Automation”.

In this session we talked about digitization, the differences between power tools and factories, containers, Cloud Foundry and more.

The recording of our session is now available. It’s in Dutch, no subtitles. But the Demos are towards the end so feel free to skip the first part if you just want to watch the awesomeness 🙂

This presentation also inspired a whitepaper which you can find here.

What is Cloud foundry?

I recently did a talk at the Dutch VMUG UserCon in which showed how it it is to deploy software by using Cloud foundry. I recently mentioned it in a blog post as well. I also published a whitepaper on automation in which I mention Cloud Foundry. But I realized that in the VMware community Cloud Foundry might no
t be very well known and understood. Instead of telling you all to “just google it”, in this post I’ll try to answer the question “what is Cloud Foundry?”

So what is Cloud Foundry?

Let me start out with a quote from the Cloud foundry website: “Cloud Foundry is the industry standard cloud application platform that abstracts away infrastructure so you can focus on app innovation”.  So Cloud Foundry is a platform that can run your cloud apps for you.

What does that mean? It means Cloud Foundry (CF for short) is a collection of software that together forms a platform on top of your infrastructure. Developers can use the platform to deploy their software using simple command line tools or an API. The infrastructure and platform admins don’t have have knowledge about the software being deployed nor do they have to be involved in the deployment of the software.

What does it do for you?

The CF platform takes care of everything: A command line utility takes the source code of an app or even the compiled binary, uploads it to the platform. Then the platform will compile the code as needed. Then it will create a so called “droplet” which is conceptually similar to a Docker image. Everything that is needed to run the app is included in the droplet. So if you upload a war file it will automatically include a tomcat server, if you upload node.js code it wil include the node.js runtime. When the droplet is ready to run CF will start it in it’s “elastic runtime”.

But it doesn’t stop there. When it’s deployed CF will monitor the app and when it crashes it will automatically restart it. It can monitor the load on the application and automatically scale it out or in as needed. Developers can also manually scale their application with a few keystrokes.

Opensource

Cloud Foundry is completely opensource and has a very active community. There are commercial distributions out there but nothing stops you from running the opensource version. For free :).

CF support multiple platforms: vSphere, vCloud, AWS, Google Cloud and even Bare metal through RackHD!

But why?

So Why would you want to use Cloud Foundry?

  1. It makes the life of infrastructure operators (cloud Ops) really easy. When you’re using VMware’s vRealize Automation you’re probably used to writing tons of scripts and workflows just to make automated software deployment work. When you use Cloud Foundry you don’t have to do any of that.
  2. It makes the life of your developers very easy. They don’t have to tell infra operators how to deploy their software. They can simply do it themselves by running “cf push”.
  3. It enables a clear separation between DevOps and Cloud Ops. Developers can now deploy, run and operate their own app without bothering the infra team. The infra team on the other hand doesn’t have to spent time understanding and deploying the developers apps. So with CF it’s possible to have a Cloud Ops team that is responsible for the platform and the developer teams are not only responsible for developing their app but cat employ true DevOps. CF Gives them the tools to manage their own app in production.

Show me!

Want to see it for yourself? You can use a public Cloud Foundry platform if you want to try to deploy some code on it: run.pivotal.io

If you’re more interested in the platform itself checkout the Pivotal website. Pivotal provides a distribution of Cloud Foundry which makes it really easy to get going. Fair warning: you need a decent amount of disk and memory resources to run the whole platform.

The Why, What and how of Automation

Today my first ever whitepaper was published. It’s titled: The why, What and how of Automation. Here is the teaser:

The current digitization wave puts an ever increasing load on enterprise IT departments. At the same time the business is expecting shorter delivery times for IT services just to stay ahead of the competition. To keep delivering the right services on time enterprise IT needs a high degree of automation.

The whitepaper explains why automation is so important, what you need to automate and how this can be done. Those who attended my NLVMUG session might notice that this whitepaper has the same title as my presentation. That’s obviously not a coincidence. If you missed the session make sure to download and read the whitepaper here: http://itq.nl/the-why-what-and-how-of-automation/

I’ll be posting a few more blogs on some of the topics in the whitepaper as well so stay tuned :).

The right tool for the job

I work with vRealize Automation and vRealize Orchestrator on a daily basis. And I really enjoy doing so, especially the custom code development part. vRO gives a lot of flexibility and it’s not often that I’m unable to build what my customers need. Whetever the request I usually find a way to emply vRA and vRO in such a way that if fulfills the customers need. But more and more often do I wonder if we’re using the right tool for the job.

Today I presented a break-out session during the annual NLVMUG UserCon. In the presentation we emphasized the importance of using the right tool for the job. After all, you don’t drive a nail in the wall with a power drill. you can do so if you really want to but you’ll probably spent more time than needed putting up your new painting and likely destroy you power drill in the process. It’s similar in enterprise IT: You can use a customizable tool like vRA/vRO for nearly anything. But that doesn’t mean you should.

But if you can make it work anyway then why not? First of all: If you’re using a product to do something that it wasn’t originally intended to do you’ll spent a lot of time and money to make it do what you actually want.  But getting the product to do that is only the beginning. Now you need to maintain the product customizations. chances are something will break at the next product upgrade. So you postpone the upgrade, then postpone again and in the end the upgrade never happens because the risk is just too high.

Let me give an example: Let’s say you’re trying to deploy in-house developed code through different life cycle stages. You could argue that everything needs to run on a virtual machine so you start out by automating virtual machine deployment. You’ll probably use vRA or something similar to do that for you. After this first step you realize that the code does not run on an bare OS, you may need IIS or .NET or Java or a bunch of shared libraries. So you decide to automate the deployment of middleware software as well. But that still isn’t enough to run the code. You also need a database, a load balancer, an SSL certificate and last but not least: you need a way to deploy the code to your machines and configure the way it’s running.  Oh and of course all this needs to be triggered by the code repository and be completely self service. By the time you have implemented all this you’ll have written tons of custom installation scripts and integration workflows.

Automating code deployment can be tricky to say the least. And in my opinion all this difficulty stems from the fact that we’re starting with the VM as the unit of deployment. The actual unit of deployment is the code/application your developers are writing. By using the wrong data as input for the tool selection you ended up with the wrong tool.

Luckily there are tools designed for application deployment. One of them is called Cloud Foundry. If you use the Pivotal distribution you can set it up in a day or so. And then your developers can just run cf push and their code is running. In the cloud. Sounds a lot better than writing countless installation scripts and custom integrations doesn’t it? Also, the Cloud Foundry platform gives you loads of options you wouldn’t have out of the box with tools like vRA: auto-scaling, easy manual scaling, application health monitoring, service bindings, application statistics, centralized logging, custom logging endpoints and lots more.

There is one major “drawback” however: your applications need to be cloud native or 12factor apps. But you’ll have to transform your apps into cloud native apps at some point in the future anyways so why not start now?