Resolving a CI + GitOps Deadlock!!

Hey Every one!!
Let’s 1st have a background about this issue:

Recently we have migrated our dev and stage environment from AWS to Oracle and during that migration we have stopped the AWS based runners and added a single runner for dev and stage environment.

Yesterday night, Our one of the developer have been tried to deploy our 2 services on new environment using Github Actions. And he have been faced this error nad github actions failed a job.
Error Screen :

The problem showed regarding the build failure. At the surface level it seems a simple problem and Initially I assumed it as It is failed due to the lack of resources availble to deploy / do this job.

Then Finally I dig and got a different problem as It is a dealock occureb between the CI process and GitOps.

To occure the dead lock mainly 4 conditions have been contributed :
( The Coffman conditions)

Mutual Exlusion : As the Ci holds a single runner then that can’t be used by GitOps.
Hold and Wait : CI holds a Runner 1 and Wait for GitOps to do it’s process
Circular Wait: CI and GitOps wait for each other to complete.
No Preemption : We don’t want to forcefully stop the CI process and Retrigger it.. Becuase even though after doing this we will be entered in the same situation..

Brute Force Approach:

To Solve this problem the brute force approce is to introduce the 3 SelfHosted Runners with same tags in the Resource pool. Which will increse the availability of resources and I expected that this problem will be resolved.

But, I was Wrong..
Due to Having the same tags on all 3 resources those resources are beign acessed by CI workflows + GitOps as well. As the all the CI will be triggered 1st and they are occupying the all the available runners and when it’s GitOps turn it doesn’t have any runner to execute it’s process.

That’s why, when a developer triggers the multiple CI workflows together then It again cuases a DeadLock!!!

So I have to Think on a another Approach…

Better Approach:

I Decided to reserve a single Runner for GitOps workflow and 2 for the CI workflows.

Now All CI workflows will only able to occupy the 2 runners at max and If a developer introdued a 3rd process then it will be added in the queue..
and wait for CI Runner to get Free/ Available.

Fig : 3 runners with separate Tags for CI and Gitops

This is how now No any CI and GitOps will fight for the resources.

While Implementing these changes I have considered to install the 3 Runners on a single VM while increasing it’s configurations. And the reservation of runner is archived by attaching a Workflow oriented tag to it like ‘ci-runner’ and ‘git-ops-runner’.

This is How a Simple error has become a simple case study and a Fun Experiment for Me!!
Thank You for Attention!!

Make sure to follow me for Such Interesting Dev Stories.

Socials :
X
Portfolio
Github
LinkedIn

Resolving a CI + GitOps Deadlock!! was originally published in DevOps.dev on Medium, where people are continuing the conversation by highlighting and responding to this story.