Manage Cattle of Projects With Projen: Intro and Problem Statement

As an Ops engineer (SRE, DevOps whatever it called these days) I try to make developers life easier, so they can experiment more, do not be afraid of changes and move fast with everything they build. This is not easy to achieve and we have a lot of tools to help us with and some aspects of the process is not always obvious.

Everyone like to talk about the solutions and tools, but I really like to start from problem first to avoid XY problem and, if I’m lucky, fix the right problem.
So, what is expected to solve at the first place?

Let’s start from very simple thing — one project. We want to deliver code from developer’s computer to the server to distribute the result.

Simplified picture We want changes there to be predictable and tracked. So we can VCS to control the versions of the code. Great, now few developers can work on the same code and all changes are tracked.

with VCS Since many ppl are working on project, we don’t want to argue about style and formatting, so we can add lint and formatter to deal with it automatically.

lint and formatter Working with VCS is also requires some branching strategy, so everyone understands project lifecycle and avoid conflicts.

branching strategy We may need to deliver not the code itself, but built artifact and deliver the result (as we usually do), so we can add build process to the project.

build Great, but we don’t want just deliver the code, we want to deliver safe and quality code. So we adding tests and security checks to the mix.

test and security checks Now we don’t really want to run all of these steps manually, and to make sure that no one will forget something, we can add Continuous integration tool so all of the steps will be run automatically on every commit or merge.

Continues Integration We also don’t wont to deliver the result version manually. For simple case it could be done by CI itself, of course, but what if we have multiple places to deploy?

Multiple deployment targets We don’t wont to accidentally deploy non-working code to the production, so we introduce other environments — like developer environment, QA environment etc.
Before code reach the production it will pass intermediate deployment stages based on our branching and QA strategies. We also may have some environments in other geographical locations. For this part of the job we added Continuous Delivery tools.

Continues Delivery Wonderful, we have all processes covered with automation, right? We can just repeat all of steps above for every project we will have. Only one question remains, and it became more and more pressing with every repository added and with every team created in the company.

Who will watch the watchers?

All tools from above need it’s own configuration files, scripts, dependencies etc. and someone and something need to take care of. It’s a lot of code with it’s own lifecycle. In reality it will be more like this:

Basically, now we have 2 things in the project we need to deal with:

- Project code itself
- Project maintenance code

When company has hundreds of repositories this is became real pain for engineering teams to maintain project maintenance code, so we should consider some strategies to deal with it.

There are few strategies to consider and I will talk about each in the next chapter.

Originally published on Medium.