Simpler Machines

Should you use Concourse?

Nat Bennett -

Hey there! This is Nat Bennett, from Simpler Machines. We're not going back to regular publishing yet but I wrote this up and figured, might as well send it to ya'll.

Occasionally people will come into the Concourse Discord asking "is this project still active?" Usually because they're evaluating whether to deploy it at their company. I thought I'd write something that could be useful for folks who are trying to understand the current state of the Concourse project.

My real goal here of course is to get someone who actually knows what's going on with Concourse to write something explaining why I'm wrong.

Full disclosure: I am a daily user of Concourse in my day job. We don't have plans to switch off any time soon. We know Concourse, it does its basic job well, we prefer it to all known alternatives.

Some facts about Concourse

v10 is dead

I don't think it's quite fair to say that the project is "dead," since it does appear that VMware is still adding some features that their teams need. Concourse is also critical for actually delivering Cloud Foundry. I can describe this in more detail in the future if folks are interested but the short version is, the Platform Automation system runs on Concourse, and Platform Automation is basically how VMware customers download new Cloud Foundry bits and deploy them across their installations.

I do think that the the open contribution process that the home page describes is no longer functioning. If you need a new feature from Concourse or you want to contribute to it you will probably need to fork it.

One thing I'm very confident about is that the public-facing documentation about the project – what's on the roadmap, how to contribute to it – is wrong. The project has substantially less staff than it did when the v10 roadmap was laid out. It hasn't been updated since that staffing change. So there are things that are on there that aren't going to happen.

I'm much less confident about what exactly won't happen. Again, because the public facing roadmap hasn't been updated, it's hard to understand what the team is working on.

But I think it's a reasonable guess that the collection of features described as "v10" aren't going to happen, because VMware doesn't need them to continue to use Concourse to deliver Cloud Foundry.

Should you use Concourse today?

If you're using it now, you've built processes around it, and you understand how to operate it? Yes.

The current CI/CD landscape is, honestly, a disaster, especially for handling large pipelines of the type that Concourse is best at handling. There is no good alternative to Concourse for managing large distributed systems deployed to VMs.

But if you're not already using Concourse? And you're not experienced with it or with BOSH? And you're not managing a large distributed system deployed to VMs?

In that case it is almost certainly not worth the operational overhead. Which is substantial. The current CI/CD landscape is a disaster for a reason. The domain is deceptively hard, especially once you get outside the most basic cases.

(This is probably its whole own post but a big part of the problem is that there's continuous integration/deployment/delivery and then there's "workflow automation." These domains intersect but they're very different when you get into the gnarly details. What makes a great abstract workflow automation tool is pretty different from what makes a great CI/CD tool, but since "workflow automation" isn't really a category yet, both products and teams tend to end up there accidentally. So you end up with good workflow automation tools trying to awkwardly wedge themselves into handling CI/CD cases and vice versa.)

My request of the Concourse team

I would love to be proven wrong about all of this. I know ya'll are working on things that haven't been reflected in the public facing documentation. Please, please, update your website and README, remove the dead plans and processes from the project, and help me (and other Concourse users) understand what you're actually working on.