Tag Archives: vRO

HOWTO: Speed up vRA template deployments

[ Since flowgrab is no longer amongst us you can get it from dropbox now ]

I recently did some scalability and load testing on a vRA deployment. One of the problems I encountered was that the template deployments,which seemed reasonably fast suddenly became a bottleneck. So obviously I created a workflow to fix that. So here is howto: speed up vRA template deployments.

TL;DR: if you’re too lazy to read an just want the workflow click here.

Let me explain the scenario first: Usually when you deploy a virtualmachine from a vSphere template it is reasonably quick. After all it just copies a file from disk to disk. If you happen to have an all flash array (my customer does) it is probably pretty fast.  Especially when your array is VAAI compatible. Because that would offload the whole copy action to the array.A typical VAAI accelerated template deployment takes around 4 seconds on an EMC X-IO.

But what happens if your template is in one cluster and needs to be deployed to another cluster which has a separate storage array. Then VAAI can’t  do anything for you and the template will be copied over the network. Which could be rather slow compared to a VAAI accelerated deployment. More in the region of 2 minutes instead of 4 seconds.

So before we get to the solution let me answer a few questions:

1: Why do you have the template on another cluster? Well… this customer has more than 10 different clusters with their own storage. Why that is is a completely different sotry. Anyhow, I wasn’t going to copy the template to each of them manually. Keeping them up to date would be a nightmare.  Second reason is that that strategy would require another blueprint for each cluster. Which is another nightmare to maintain.

2: Why don’t you use a template LUN shared between all clusters? This was actually not physically possible and also otherwise unwanted. It also wouldn’t fix the VAAI issue since it would require copying from one storage bo to another. It would be faster then copying over the network, that’s for sure.

3: 2 minutes isn’t that long. Why bother? Actually, consider the fact that vRA by default only does 2 deployments in parallel. That means that if you kick-off 50 deployments it takes at least 48 minutes before it even start deploying the last request. That is an unacceptable  long delay and even causes time-outs in vRA. I tried bumping up the number of parallel deployments but that slowed the deployments so much they never finished within vRA time-outs.

So…. I didn’t want a separate blueprint for each target cluster and I didn’t want to copy the template manually. The solution that remains is having the template copied by a workflow and then overwrite the template that the blueprint is going to use. Can we do that? You we can!  :). Turns out there is a custom propery called __clonefrom which contains the template name. If you overwrite this property during the buildingMachine state it will just use that machine to clone from.

to automate this process I created a workflow that:

  1. gets the template name from the __clonefrom property
  2. gets the cluster name of the cluster where vRA is going to deploy the machine
  3. add the cluster name to the tempate name and checks if such template exists
  4. If it doesn’t exist it will clone the template that is configured in the blueprint to the target cluster and adds the cluster name to the template name so we can find it next time we deploy something to the same cluster.
  5. overwrite the __cloneid property and then let vRA do it’s jobs

selectTemplate workflow

That’s it. This will make sure you have VAAI enabled deployments on each cluster. In my case it decreased the template deployment time from around 2 minutes to 4 seconds. This is so fast tha vsphere deployment is done before vRA can kick-off the next one.

you can download the workflow from dropbox. Use at you own risk! It’s tested wit vRA 6.2.1 and vSphere 5.5. Should work on any vRA 6.x version or even 5.x but I didn’t test that. Not sure about vRA 7. I’ll let you know when I had a change to test that.

vRealize Orchestrator 7 Release

As you’re probably aware by now VMware released vRealize Automation 7 today. I already discussed vRA 7 before and there are a lot of blogs out there writing about it.

What I didn’t blog about before is that with vRA7 VMware also released vRealize Orchestrator 7.  Since vRO is the magic sauce that makes vRA work anyways I’m far more interested in the vRO release. So I quickly downloaded the ova (yep, no windows installer anymore), imported the ova into VMware workstation and connected to it. All in under 7 minutes :).

I read on another blog that there now is an HTML 5 client. But nope…. only the control center is HTML5.  The client used to develop workflows is still the same Java contraption. But it supports reconnecting now. Which is very nice because that means you don’t loose any work when a vRO server stops responding. Other than that I couldn’t find any improvements in the client. Still no decent IDE like environment to write your scripts in, no improvements to the api explorer as far as I can see. I’ll digg deeper in the coming week and let you know what I find.

Other new features:

  • Centralized Server administration.
  • Easy cluster configuration.
  • Easy workflow troubleshooting and runtime metrics.
  • Enhanced log monitoring, log persistency and added ability to export logs for a particular workflow run.
  • Direct correlation of system properties and workflow performance through the embedded JMX console integration.
  • Significant Orchestrator client improvements, including a workflow tagging UI, client reconnect options and enhanced search capabilities.

The Magic Button

On March 19th we used The Magic Button ( a.k.a “The What Does This Button Do Button”) in our demo’s at the Dutch VMUG UserCon. It magically  made a CoreOS cluster appear out of nowhere, Launched our demo app and then it scaled it out so all people in the room could open the page. Of course you want to build your own now. Here is how.



The button itself is just a regular emergency stop button I got off e-bay (6$). Inside there is enough space for a battery holder with 2x AA batteries. These batteries power an ESP8266-01 board. The ESP8266 is a Wifi SoC, has a 80Mhz processor, wifi connection, 96KBytes of RAM, a couple of GPIOs, comes with SPI flash on the board, costs around 5$ and looks like this:


The chip has a UART and was originally intended to function as a serial to wifi module. Out of the box it comes with an awkward AT firmware (hello 1990!). But thanks to a very active community we can now build ourfirmware for this neat little chip. I don’t have the time or the knowledge to write my own firmware in C++ but luckily someone created a firmware for this chip that let’s you run Lua code on it! I didn’t know any Lua before but it turns out to be rather easy. Since it’s an event driven interpreted language it has some commonalities with Javascript, which I am very familiar with.

Here is how I connected the board:

  • Connect the button between GPIO0 and Ground
  • Connect the LED between GPIO2 and Ground. I used a 100Ohm resistor to limit the current through the LED
  • Put a 1K pullup resistor between VCC and CH_PD
  • Batteries are directly connected to VCC and GND. No Caps or regulators.

The magic button internals

When everything is connected you can squeeze it all into the case. It actually doesn’t really fit. When I close the case the battery gets damaged a bit. But whatever, it works….

The MAgic Button Squeezed


So How did I turn this wifi board into a magic button? The button simply does an HTTP POST to my webEye application. This application forwards the posted body to an AMQP bus where it get’s picked up by vRealize Orchestrator. vRO in turn runs a workflows which actually performs the magic. To enable your board to do the same, follow these steps:

  • Setup webEye or another webhook receiver to test the button
  • Flash this firmware on the chip: nodeMCU
  • Use ESPlorer or another tool to load these two Lua files: https://github.com/vChrisR/TheMagicButton.
  • Please edit the variable at the top of the files before copying to your ESP
  • Emergency stop buttons are normally closed. So make sure the button is pressed (open) when you power up the ESP. If you don’t do that it will keep GPIO0 low which makes it boot into bootloader (flash) mode.

Now build a cool workflow which you can trigger with this button. Share your creations in the comments or find me on twitter.

webEye – The webhook receiver

When building out the demo environment which I was going to use for our NLVMUG UserCon presentation I came accross a problem: I wanted to notify my private cloud whenever a new Docker image was build on Docker Hub. This proofed impossible with the existing VMware software so I created my own solution. And here it is: webEye – The webhook receiver. It will simply forward all received webhooks to an AMQP bus after checking that it’s a valid webhook message. You can pick up the message with your favourite orchestration tool and act on them.



Every hook needs an eye to hook into. That’s why my little app is called webEye 🙂


webEye is written in JavaScript and runs on node.js. It is designed to run in a docker container. However, it already evolved fom something that was originally intended to just receive docker hub webhooks. Currently it also has support for my “Magic Button” and even for vRealize Operations. Other web hook senders might follow.

Getting started

As I said, webEye is developed to run in a docker container so this “getting started” will only cover how to start the app in a docker environment.

  • All received hook are forwarded to an AMQP bus. So let’s start an AMQP Server: docker run –name rabbit -p 5672:5672 -p 15672:15672 dockerfile/rabbitmq
  • Now start webEye: docker run -p 80:80 -p 443:443 -e “DHKEY=12345” -e “MBKEY=12345” –name webEye –link rabbitmq:rabbit -t vchrisr/webeye
  • The DHKEY in the line above sets the API key that you need to send with the request. This adds a bit of security. Make sure to put in a random string instead of “12345”.  tip: random.org
  • Now make sure port 80 on your webEye server is mapped to a public ip address
  • Open the now running webEye page in a browser to to get it running. This first visit actually triggers phusion passenger in the container to start the node.js app. This in turn creates a persistent exchange on the rabbitMQ server.
  • Create a webhook on your docker hub repository to http://{your public ip}:{public port}/dockerhub?apikey=12345
  • Connect your consumer to the rabbitMQ server
  • Create a new Q to receive your messages
  • Create a binding which routes messages with routing key webeye.docker.hub to your Q
  • Create a suscription for the Q you created
  • If you’re using vRO you can now create a policy which triggers a workflow when a message appears in the subscription.
  • Create a workflow that does whatever you want when a docker hub hook is received.

Testing webEye

If you were able to make webEye available on the public internet and you’ve configured a webhook on your docker repo you can

  • now simply click “test” on the webhook configuration page.
  • To test offline I usually use the Firefox RESTClient plugin.Select “POST” as the method.
  • Enter this url: http://<ip of webEye machine>:<port>/dockerhub?apikey=<apikey>
  • Add this header: Content-Type: application/json
  • For the body you need some actual content. webEye will check the presence of some specific fields in the json to make sure it’s a Docker Hub webhook. I usually use the json from the Docker Hub Documentation:


Orchestrator JavaScript speed test: .map()

A while ago I wrote a blog post in which I showed the performance difference between the array prototype function .indexOf() and a for each loop. Now it’s time for the second part of the series: Orchestrator Javascript speed test: .map()

Test setup

The test setup is identical to the setup I described in the previous post. Same machine, same vCO appliance. I did change the script that generates the test array slightly. Instead of a string I now store an object in each array elemen. Here is the script:

Both tests are done in the same workflow so they are as close together as possible to get the same circumstances of the tests.

map vs for each workflow

 Mapping an Array

Mapping an array into an other array means running some action on every element of the array and returning the value of that action into a new array. What I will do in this test is taking one attribute of the object that is stored in each array element and create a new array that only consists of that one attribute. This makes it easier to  search for the right index number later on using .indexOf()

  • So here the content of one array element: { number: “1”, value: value=1 }
  • And what we want as an end result is an array where each element just contains “value=1” for example.

There are basically two ways to do this. You can either use the prototype function .map() or create your own loop. Let’s try the prototype function first.  map() takes takes a function as an argument. Whatever the function returns is stored in the active element of the target array.

Below is the result of this test:

So the map action took 94 milliseconds. But over a couple of test runs I did get different results. Ranging from 119 to 86mS. Now let’s try a for each loop to see how long that takes:

And here are the results:

So this particular run took 106 milliseconds. But again as with the .map I don’t get consistent results. I’ve seen values everywhere between 82 and 139 mS.

I run both test sequentially in the same workflow. And even the difference between both methods is not the same. Sometimes the map() is faster, sometimes the loop is faster.


I cannot definitively say which method is faster. The only thing I can say for sure that they are about the same speed. But if you ask me which method I prefer the answer is: .map()! Why?  Because if I read somebody elses code and I see the .map being used I know something is being mapped. But if I see a pof each loop I have to go through the whole loop to understand what’s going on. In the example above the loop might be simple but in real life it can get complicated pretty quick.


Automating vRA (vCAC) using vRO – Split Brain

Recently I have done some work on automating vRA (vCAC) using vRO (vCO). This meant I had to dive into the vCAC APIs. The bad news is that this felt like diving into a pool of dark muddy water. The good news is that I’m still alive, my headache is gone and I’ll try to capture some of the things I learned on this blog.

Split brain

In this post I’ll start out with an introduction to the vCAC APIs. Yes, plural. Not just one API.

vCAC ahem… VRA is actually not just one product, it’s two products which are loosely coupled and sold as one. The first product is the vRA Appliance also known as CAFE. This is a new product that was introduced with vCAC verion 6.0. It is developed in Java (springsource), runs on linux, uses postgres as a data persistence layer, seems to use a micro services architecture , supports multi-tenancy and provides a REST API.

But there also is the old product that was originally developed at CreditSuise, spun off as DynamicOps and then acquired by VMware. It was sold as vCAC 5.x, is developed in .net, uses an MS SQL back-end, runs .net workflows, has no notion of multi-tenancy and provides an OData API. This part is usually called the Iaas Part.

The two products are also reflected in two separate vCO ahem… vRO Plugins. Although you download and install just one package there are really two plugins installed. One is called VCAC and has the description “vCloud Automation Center Infrastructure Administration plug-in for vCenter Orchestrator” the other one is called CAFE and is described as “vCloud Automation Center plug-in for vCenter Orchestrator”.

Confusing. Right? So let’s clear things up:

CAFE is the virtual appliance. All new features are developed in CAFE. So anything that was added since 6.0 runs on the appliance and can be used from the REST API. On top of that some functionality was moved to the appliance. Functionality running in CAFE in version 6.1 includes:

  • Business Groups and Tenants
  • Advanced Service Designer
  • The Catalog
  • Resource Actions
  • Approval policies
  • Notifications

So if you want to automate anything regarding any of these features you’ll need the CAFE plugin which talks to the REST API running on the virtual appliance.

IaaS is the name of everything that’s not on the appliance. It is the reason you need a windows server to run vRA, not just the appliance. This windows server (or multiple servers) runs the old DynamicOps Software with some modifications. Features provided by this part of vRA include:

  • Virtual Machine Blueprints
  • Machine Prefixes
  • Provisioning Groups (Maps to Business Groups in CAFE, GUI only knows Business Groups in the current version)
  • Reservations
  • VirtualMachines (vCAC VM objects which map to vSphere/vCloud VMs or even physical machines)

If you want to automate any of the above you’ll need to use the vCAC plugin or the Odata API. If you’re note familiar with Odata APIs there is something you should know: It’s not an actual API. It’s just a representation of the database. There is no application logic behind it, just database constraints. This means that creating new things (called entities) is rather difficult. You have to figure out all the links between different database tables yourself. I’ll try to dive into this deeper in another blog post.

There another peculiarity I want to point out: there is no multi-tenency in the IaaS part. This means that a lot of items from the IaaS part (for example: machine prefixes) are shown to all tenants!


The fact that vRA basically has a split brain provides some challenges when automation things in vRA. For Example: You’ll have to create a blueprint in the IaaS part but when you want to publish it you have to create a catalog item in the CAFE part of the product. Which brings me to the last part of this post.

As I said before the two product are loosely coupled. The actual touchpoints are not documented. Or at least I couldn’t find anything. But after spending a lot of hour trying to find out how to autmate the publishing of blueprint I found these touchpoints between both APIs:

  • The Business Group ID in CAFE is identical to the Provisioning Group ID in IaaS. If you create a Business Group in the REST API then vRA also creates the ProvisioningGroup in IaaS for you.
  • The catalog actually consists of three catalogs. More on this later. One of the catalogs is the provider catalog. Each provider manages its own provider catalog. IaaS is on of the providers. Somehow CAFE knows where to find certain provides IDs. Not sure where to find or set that mapping.
  • Every Catalog Item has a providerBinding attribute. This contains the bindingId. This binding ID is the blueprint ID (virtualMachineTemplateID) from the IaaS Part. This is how vRA figures out which blueprint to deploy when you request a catalog Item.
  • A Resource Operation has bindingId which maps the CAFE action to the IaaS action (like powerOn a VM for example)

Orchestrator Javascript speed test: IndexOf()

As you might know Javscript is the scripting language used in vRealize Orchestrator. So while I’m not a web developer  I  use a lot of Javascript. When handling arrays in my scripts I tend to use a lot of prototype functions like .map, .forEach, indexOf and a couple others. But when I go through the library workflows I see a lot of of for each loop with some ifs and a break instead of the prototype functions being used. I have some opinions on this which I will share later. For now I was just wondering which method is faster, using the prototype functions or using your own loops. To answer this question I decided to do some speeds tests. This is the first post about these tests: the Orchestrator Javascript speed test: indexOf()

Setting up the test

To be able to measure a difference in performance I needed a significantly large array. I settled on an array with 100.000 elements as this seemed to take enough time to loop through to see some actual performance difference between different methods. I executed the tests on a vCO 5.5.2 Virtual Appliance running on my laptop. So if you run the appliance on a faster machine you might need a bigger array.

I used this script to create the array:

The actual speed tests are in the same workflow as the array generation script. This way both tests are ran as close together as possible to ensure the same circumstances for both tests.


Finding the index of a value

Imagine you have an array and you want to figure out in which element a certain value is stored. There are two ways to do this. The easiest is using the .indexOf() prototype method. Alternatively you could use a for each (..) loop. To find out which method is the fasted I generated an array with 100k elements. the value in each element is the string representation of the index number. On the array I executed the code below:

This piece of code searches for the value “99999” in the array elements. We already know that is the very last element of the array so this measures how long the function takes to loop through the whole array while still validating that the actually works correct.

Below is the result of this script.

So the total time elapsed for the indexOf() method is 52 milliseconds.

Let’s compare this to a for each loop.

And here are the results:

So this run took 166mS. Which is more than 3 times slower than the .indexOf() prototype method.


Not only did I have to write more code to achieve the same result, the execution of the code also takes more than 3 times longer to execute the code. Obviously if you hit the target earlier in the array or use a smaller array the difference would be smaller. Still, It doesn’t make sense to write more code that is slower, harder to understand and not maintained by the software vendor.

So please: use the .indexOf array prototype method when searching the index for a specific value.


vRealize Orchestrator 6.0 New Features

With the release of vRealize Automation 6.2 VMware also released vRealize Orchestrator 6.0. In this post I’ll explain the new features.

You’ll find vRO 6.0 on the vRA appliance. There is no stand-alone virtual appliance or installable version for vRO 6.0 at this point in time. Sources tell me this will be released with vRO version 6.0.1. vRA 6.2 is build to work with vRO 5.5.2 so if you want to use an external vRO server that’s the version you’ll be using. Unfortunately that means you’ll miss out on these new features of vRO 6.0:


If you’re familiar with javascript or any other scripting language you have probably used the switch case statement before. It selects a code block based in the value of a variable. Orchestrator already supported this inside scripts but now there is a switch element you can drag into a workflow. This way you can fork your workflow into different flows based on the value of a variable.

This is what it looks like in a workflow schema:


The picture below shows the configuration of the Switch element:

ou can add rules to the switch element by clicking in the green plus icon. For each rule you can select a variable (in this case “someVariable”), the matching operator (Equels, contains, match and Defined) and for some operators the value to match to.

These switch rules work like firewall rules, only the first match is used. That’s why you can move the rules up and down to change the order of the rules.

Default error handler

Another new element is called “Default error handler” . When you drag it into the schema it looks like this:

It is actually not connected to anything in you schema. It will be executed when an unhandled exception occurs in the workflow. Added to the workflow from the previous example it looks like this:


As you can see in the schema the default error handler allows you to run certain actions whenever an error occurs in the workflow that is not handled by any other error handler. So this gets rid of all the red lines to one error handler. Neat!

vRealize 6.2 releases

VMware released version 6.2 of a lot of the vRealize familiy products, vRA is one of them. I’m not going to cover all new features here, you can find them in the release notes. One feature I really like is this one:

  • Ability to edit custom properties for published applications in the service catalog.

This feature means we can now have user input for IaaS Custom properties on Application blueprint catalog items. I used to create some fairly complicated ASD workflows to work around this so this new feature makes life a lot easier for those customers using Application Services.

A couple other features worth mentioning or those:

  • Schedule date and time for the reconfigure operation.
  • Supports configurable email templates.

VMware also released a new product: vRealize Code Stream. This is VMwares take on a Continuous delivery tool. Its is connecting existing tools an repositories and provides the use the ability to create pipelines containing tasks. Interaction with external systems is handled by vRO (former vCO).



VMworld and the future of Orchestrator

Last week I attended VMworld Europe in Barcelona. I had a great time, eating tapas, drinking Rioja and learning something new in between. I already wrote about elasticity achieved using project fargo and docker on my company blog. Since this blog is more automation focussed I wanted to highlight some automation news. OR actually it is more about the future of Orchestrator.

The first thing that stood out to me was the lack of vCenter Orchestrator uuhh vRealize Orchestrator break-out sessions. I think there were two or three session explicitly about Orchestrator. A couple others went a little bit into orchestrator but were focused on vRealize Automation (vCAC). Last year there were quite a couple of sessions about Orchestrator. Telling us it was the best kept secret or the best VMware product never released and we should really go and use this awesome tool. And of course they were right to say so. And seeing where VMware is going with Orchestrator I was really surprised they didn’t give it more attention during Vmworld.

Which brings me to my second point. It is clear by now that Orchestrator will be used as the back-end for vRealize Automation. We can already see this in the current versions: The integration with NSX is completely implemented using Orchestrator. vCAC ugh… vRA has no interaction with NSX whatsoever, everything is handled via Orchestrator.

The same goes for what VMware calls Anything as a Service. Which is delivered using the Advanced Services Designer. Yeah that’s a lot of buzzwords in one sentence. In reality it is just a forms designer which you can use to design user front-ends for Orchestrator workflows. The objects created by the workflow can then be managed by vRealize Automation.

I already see that the adoption of Orchestrator is mainly driven by the use of vCAC. But there is more to come. VMware told in one of the Orchestrator sessions that Orchestrator will be used as a DEM replacement for vRealize Automation (but any information given in such presentation may change at any time). For who isn’t familiar with vCAC/vRA; The DEM is the Distributed Execution Manager. It is basically the component which does the actual work in a vCAC deployment. Currently it is 100% .net code and runs MS .NET workflow foundation workflows. So it makes total sense to replace that workflow engine with VMwares own workflow engine. The result will be that some day we can get rid of the windows components in vCAC and end up with just a virtual appliance which is easy to deploy and configure. That day will be a good day.

To be able to use orchestrator on the scale that vRA requires there will be some changes to the product in the future. For example, better permission management, multi geographical deployment models, integration with DevOps solutions and a lot more.

So although Orchestrator didn’t get a lot of attention during VMworld it seems it is going to play a crucial role in VMware’s automation strategy. Nice 🙂