Ep.114 CI/CD Pipelines Have Become Complicated

By Published On: March 18, 2021Categories: Blog, Podcasts, The CTO Studio

Are you a technology professional looking to connect with like minded people?

We have a thriving community of CTOs discussing these episodes and more.

Click HERE to set up a call and learn more about becoming a member.

About The Speaker:

Matt Ferguson is the CTO at Galley Solutions.

Episode Resources:

Check out https://7ctos.com/

Episode Transcription:

Nickolai Walker: [00:00:15] Welcome back to the studio, I am your host, Nicholai Walker. We are in the middle of Episode 114. This is our final episode with Matt Ferguson, so insert sad face here. Matt is the CTO at Galley Solutions and he is a fellow tinkerer, just like myself. And I usually ask the first question, but I’m going to toss it over to Etienne because they’re beginning the conversation about CI/CD pipelines and I’d rather have it and take the charge and ask the question so that we get the right answer. So here we go.


Etienne de Bruin: [00:00:48] So just to touch on that briefly, with CI/CD pipelines it is kind of shocking how complex one can get and one thing I’ve noticed is that there’s an exponential correlation between complexity and developers fears to touch the CI/CD pipeline. I mean, they get afraid. Really afraid. Even worse than that crazy recursive piece of code that no one wants to touch. If the pipeline is complex then no one wants to do a production push and everyone’s afraid of a rollback. So Argo solves that?


[00:01:31] Argo does a very good job of solving that, more so than I’ve seen in other platforms. We now have one click rollbacks,  so, you know, that was one of the things we asked  when we were investigating it. For example, well, we don’t ever want to do a roll back, but if we have to, who how hard is it? You know, you can always deploy that image if you have to, but what are you gonna do? So, yeah, we now have one click rollbacks. We have one click to increase the Kubernetes pod count for a particular service. Every piece of configuration is in a repo, so you’re managing configuration and repos in GitHub. And that’s a beautiful thing to say the number of pods is in a repo and if I merge a change to the number of pods, that’s going to trigger Argo to change that configuration. So everything is a git ops. And that’s fascinating and so simple. And it’s separating the complexity of doing it versus the where I saw what I want. And so as a developer, if I can separate those two things and not commingle the intent with the implementation. And the intent is a configuration file in environment variables and the implementation of how to get it done is over here. Then I can teach the entire team where to make the change and feel comfortable. Because there’s also visualizations and a cute little UI and it makes it just much more accessible.


Etienne de Bruin: [00:03:22] So can you give a quick overview of where Argo fits in a typical pipeline? Am I confusing it with terraform and tools like that? Is it in the same category?


Matthew Ferguson: [00:03:41] Yeah. Those are complementary technologies. So TerraForm is a wonderful scripting tool for managing Kubernets though there are others.  Excuse me, AWS, among other things. AWS and your actual cloud platform, you know, and then you’ve got things like Kubernetes with KUBECTL that are managing Kubernetes potentially if you’re using Kubernetes as your container strategy, which there’s also tools that can sit on top of that that Argo can talk to in a more sophisticated way. But what Argo is doing is it’s just the orchestrator, right? It’s just keeping track and saying, I’m going to watch all these repos. I can watch a repo for your code. And if that code changes and Jenkins goes and builds it, then I will take it and deploy it for you. When that image gets built, I have a repo for images and when you put a new image there, I know what to do with that, right? So somehow there’s a new image and I know what to do. Or the number of pods that you want. Now, Argo doesn’t know how to actually change the pod count, but it knows if it sees that the pod count is changed, it can go tell KUBECTL to run this command with KUBECTL? So Argo is the glue, the build and the deploy and the configuration changes. It’s doesn’t do the build. It doesn’t execute the configuration change, but it can call all the tools all due to GitHub triggers.


Etienne de Bruin: [00:05:51] Ok, and so what do you use for the build?


Matthew Ferguson: [00:05:58] We’re a Jenkins’ shop.


Etienne de Bruin: [00:06:00] Oh, Jenkins. Okay.  So CircleCI or any kind of… So Aagot lives in harmony with the build tools, the pipeline tools, but Argo monitors repositories and then trains your developer culture to be one of committing config changes, committing images. So then do you tool Jenkins then to commit to the image then to a repo?


Matthew Ferguson: [00:06:31] Exactly, so as part of your Jenkins pipeline, when you build that image, you put it in a repo, right? You’re updating repo.


Etienne de Bruin: [00:06:42] That’s a delicious separation of complexity because now you’re not having Jenkins run some doofus deploy script.


Matthew Ferguson: [00:06:52] Yeah, it really centralizes down the Jenkins responsibility, right? Yeah, you’re also massively de-complexing things. I think the separation of environment variables from code, we all say we should do that, but now there’s a repo for your environment variables and that’s a whole other world, right?


Etienne de Bruin: [00:07:19] That is a whole other world. I mean, configuring environment variables in CircleCI is..


Matthew Ferguson: [00:07:25] Yeah, I mean, when you build that first project as a developer and you see how I should build, how I should test, how I should deploy. Let’s say you’re using node and your environment variables are buried in there and maybe your test scripts are buried in your code and your deployment environment variables are buried in there. I mean, I look at that and I’m like, no, I want my code over here. I want my environment variables over here because these might change, but my code might not. And as soon as you commingle them, you end up with the complexity, which is why everyone is scared to touch their CI/CD pipeline. If you can separate those things and say, hey, code is code. Environment variables over here. Configuration over here, I mean, that really simplifies the solution.


Etienne de Bruin: [00:08:16] But does that mean that you’re committing secrets to a repo?


Matthew Ferguson: [00:08:23] I mean, you have to manage secrets with a secret strategy, never in a repo. No, you don’t have to do that. You have to inject secrets just like you do with Jenkins’ or with any of the tools. You’ve got to manage that in a special way.


Etienne de Bruin: [00:08:37] So then what do you mean by environment variables being in a repo?


Matthew Ferguson: [00:08:43] Well, you might have environment variables like I just mentioned, like how many pods.


Etienne de Bruin: [00:08:47] Config variables or..


Matthew Ferguson: [00:08:50] Yeah, yeah. Secrets are a whole other animal for sure.


Etienne de Bruin: [00:08:54] Yeah. Because the problem is when we configuration CircleCI, you can’t see it after you’ve configured it so it’s gone. This is why you have your, I don’t know what AWS has, as some secrets manager or something right?


Matthew Ferguson: [00:09:14] They have a secrets manager, yeah.


Etienne de Bruin: [00:09:16] So let’s talk a little bit about.. I almost got sucked into talking about coding with your kids. Are you teaching them how to code?


Matthew Ferguson: [00:09:41] I have backed off a bit, but I have been involving my kids in my projects.


Etienne de Bruin: [00:09:47] Oh, you mentioned that. Talk to me about that,


Matthew Ferguson: [00:09:49] It has been really fun. So both of my middle daughters have gotten involved in different projects. So when I can find tasks that I think I could teach them, instead of teaching them how to code, I teach them just enough to accomplish the goal because there’s so much of what we do in the software development space. It’s like, oh man, I have to do this 10 times or 100 times, or this is the same pattern over and over. So there’s a lot of opportunities I’ve been finding, especially with working from home now and the kids are schooling from home that I’m able to say, hey, I will pay you to do this job and I need this data transformation done and we’re just going to brute force it. I need you to do this. I needed to change. You know, we’re not going to make this like some object oriented craziness. I just need you to put this pattern in play. You think you could do that? Oh, yeah. And so that’s how I’ve been getting them involved, but with actual work projects. We have a we have a GraphQL API for how we run Galley and so we’ve exposed that to our customers as a way for them to get to our data.


Matthew Ferguson: [00:11:08] And we’ve also created a templating or we’re using a templating engine. So you can upload a Microsoft Word document and you can embed your GraphQL query and then, if you want to build recipe cards or you want to build a menu or if you want to download the cost of your recipes, whatever. But those templates have to be built. They’re a great opportunity for a kid to first go explore the GraphQL and spend time not writing code per say, but just writing GraphQL queries in a sandbox and then convince me that, yeah, I know how to get to all the data dad. Show me how to build a template and all of a sudden they’re able to write a template for a customer and it’s just a fun project. So we’ve done that. We’ve had my other daughter help with just data transformations. Some of my other projects where we just need this data converted. Can you do this? And we teach her how to do it. So I’m really having fun with that and they’re enjoying it and they’re making, you know, sneaker money, or ballet shoe money in some cases.


Etienne de Bruin: [00:12:22] What do you do when they get stuck?


Matthew Ferguson: [00:12:24] You know, they’re usually right here beside me. And so I reach over and say, oh, you do that. So, yeah, they’re kids, right? So you don’t expect them to do it fast. And if they get tired after two hours, that’s about it. It’s not slave labor so if it doesn’t get done, then there’s other people.


Nickolai Walker: [00:13:06] Thanks again for joining us here in the studio and a massive thank you to Matt Ferguson, who gave us some really great, great, or at least to me, some really fascinating information about the ecosystems, the backyard pond that he’s built. The fact that that water is drinkable, et cetera, et cetera. The interviews were great and I’m really glad that we had him on and we