May 5, 2016| Dan Young
The Cloud Foundry ecosystem has given rise to some amazing things. This series of posts is about two of them - BOSH and Concourse. Together with CF, they form an awesome triangle of tools that will change the way you think about how to get things done when delivering software.
BOSH is the distributed systems powerhouse that makes Cloud Foundry possible. It answers the impossible question: How do you continuously deliver not just an app, but an entire platform of distributed virtual infrastructure?
Concourse is a intuitive, cloud-native CI tool that has a powerful simplicity to it. Actually, I’d go so far as to say it’s more than just an automation tool, it’s a medium for artistic expression! Just have a look at some of the vast Pivotal pipelines (e.g. the Buildpacks team) and marvel at their aesthetic value. If you haven’t used Concourse yet you should visit their site and try the Vagrant box locally - you’ll be up and running in just a few minutes. If you’re not familiar with BOSH, stop right now and read about why you need it. If you want to try out BOSH locally and have a fast laptop with 8-16GB RAM, you should play with bosh-lite. I can also highly recommend this brilliant intro tutorial by Maria Shaldibina at Pivotal.
If you follow this tutorial and deploy a BOSH Director on IaaS, you can then use it to deploy any other available release. Be careful though - once you’re wielding your new weapon you may want to just go crazy and deploy your whole life using BOSH. You should be aware that there are ‘7 stages of BOSH’ you must endure to achieve mastery, as humorously explained by Daniel Jones (EngineerBetter) and Chris Hedley (Pivotal) at CFSummit Berlin 2015.
The Concourse Vagrant box works like a dream for testing - being able to test your whole pipeline or just individual jobs locally is a killer feature - but eventually you’ll need to take things to the cloud. The aim of this tutorial is to get you up and running with a BOSH director and a fresh deployment of Concourse on AWS. Please remember, the examples provided here are for demo purposes and should not be considered production-ready.
In Part 1 we will do the following:
In Part 2 we will:
First, clone the GitHub repo accompanying this tutorial:
$ git clone https://github.com/EngineerBetter/bosh-concourse-setup
We’re going to use Terraform to express our desired AWS environment as code, so we can put this in source control along with everything else. The terraform configuration we’re using will create the network and security resources for BOSH, but also an elastic load balancer for Concourse with an SSL listener. However, this does assume that you already have the following:
You will also need an EC2 SSH key pair before you start.
Once you have these, it’s time to install some tools! You’ll need to use either Linux or OSX with Ruby installed for the bosh tools. Terraform will work on Windows.
The Terraform config needs to know about some parameters that are specific to you and your AWS environment. These are defined in
variables.tf and we set the corresponding values either in environment variables, or a file. You can use the example tfvars file:
$ cd terraform $ cp terraform.tfvars.example terraform.tfvars
variables.tfvars to provide your own values. Set your desired AWS region in
variables.tf. Ensure terraform is in your path, then apply the configuration:
$ terraform apply
Terraform will execute the plan and save the state of your environment locally. If you need to make any changes, just reapply and it will converge towards your new desired state. A nice feature of terraform is being able to output a dependency graph of your resources. If you have Graphviz installed, you can do this:
$ terraform graph | dot -Tpng > graph.png
Which produces this:
You should check the terraform output values and make a note of the elastic IP we’ll use for the BOSH director using
Now we should have an EIP, VPC, Subnet and Security Groups ready for BOSH.
We’re going to create a manifest for
bosh-init. First you should set the following environment variables:
$AWS_ACCESS_KEY_ID $AWS_SECRET_ACCESS_KEY $AWS_REGION $AWS_AZ $BOSH_PASSWORD $AWS_KEYPAIR_KEY_NAME $PRIVATE_KEY_PATH
Then you can go ahead and create the
Once this is done, you are ready to deploy the BOSH Director. Are you ready?
$ bosh-init deploy bosh-director.yml
Now go and make yourself a cup of tea - this deployment will take a while. While you’re drinking your tea, why not read more about BOSH?
Back in your terminal, you should eventually see something like this:
Once the director is deployed, you can target it using the bosh cli and log in using the admin account and your chosen password.
$ bosh target <your elastic ip address>
‘BOSH 2.0’ now uses a concept called cloud-config which separates IaaS-specific config like IP addressing from the deployment manifests and makes it an operator rather than user concern. Set your chosen AWS AZ and your subnet_id in
terraform output subnet_id to get your subnet_id, or just run the script provided in
bin/make_cloud_config.sh to output
Then you can tell BOSH to update the config:
$ bosh update cloud-config aws-cloud.yml
You can use
bosh cloud-config to output your current configuration at any time.
Congratulations, you have a working BOSH Director!
Once you’ve finished playing and want to tear down your whole environment, first tell bosh-init and to delete the deployment:
$ bosh-init delete bosh-director.yml
Then use terraform to destroy your configuration in AWS:
$ terraform destroy
In the 2nd part of this post, we will look at using BOSH to deploy Concourse.