Ep.113 IOT, Micro Services And Data Visualization

By Published On: March 16, 2021Categories: Blog, Podcasts, The CTO Studio

Are you a technology professional looking to connect with like minded people?

We have a thriving community of CTOs discussing these episodes and more.

Click HERE to set up a call and learn more about becoming a member.

About The Speaker:

Matt Ferguson is the CTO at Galley Solutions.

Episode Resources:

Check out https://7ctos.com/

Episode Transcription:

Nickolai Walker: [00:00:08] Hello, hello and welcome back to the studio, I am, of course, your host, Nickolai Walker. We are joined again today in studio by Matt Ferguson, who is the CTO at Galley Solutions and a fellow tinkerer, as you guys have discovered through previous shows. And we are going to talk today a little bit to him about IoT micro services and data visualization. So let me just start off with the first question, picking up from last week’s conversation about Matt’s backyard ecosystem… So then you took the material ecosystem that you made and added high tech to it, right?

 

Matthew Ferguson: [00:00:43] Well, the Iot that I’m doing is around water recycling for a company here in San Diego. It’s not around my pond. And that has been a neat project. I have a background working in the water industry for about eight years and in the utility space, as a software provider and hardware provider to utilities. So it’s been kind of neat to rekindle those roots and work with a company here in San Diego that is building a water reclamation system for use in, well, everything from breweries and food operations and food factories to the military who need to recycle water into some maybe not potable or potentially potable after that with a few more steps, but that can go back in the environment at that point. It’s not sewage. And so the company Aqua Cycle is doing a great job, fantastic folks who are the founders of the technology. And we’re helping with building out the gateway that monitors each of these platforms. So all of the work they’re doing and that they’ve pioneered needs to be monitored when it’s on site. And so monitoring over 600 reactors that are organically cleaning water through biotechnology and measuring the voltage, but also watching the hydraulics. And what was fascinating about getting involved in this project was realizing that the team needs to be able to access these sites and needs to be able to interact with that data and tweak things that are going on.

 

Matthew Ferguson: [00:02:44] You know, it’s a living system and you still want to be able to massage how the system’s working and react to things. So we really spent some time thinking about how to design this and we decided to go with some technologies that would enable them to, as operators, not have to ask for an engineer to recode something. So what are the building block technologies, the Legos, that would allow them to really interact with what’s going on? And obviously, that, I think, can be a lower cost solution if you’re able to build from off the shelf technologies, which is great, but also a more flexible solution in some ways. If you’re focusing on the capture of data, the storage of data and then the visualization of data. And if you can design that, each of those things, the ability to see what’s been stored is available to an operator and the ability to visualize is something an operator can reconfigure. So the IoT space is pretty interesting. There’s a lot of work that’s been covered there over many years, but how to do that, right, so that you could take a scientist or an operator who isn’t a software engineer and give them control over all that data and the visualization.

 

Matthew Ferguson: [00:04:20] We settled on some cool technologies. I’m actually watching it on my screen here, because the test harness is firing away. And we’re using some tools, stream sheets from Sundalo, which is a German company that has developed these micro services that you can just configure like a spreadsheet to transform data and push that data into off the shelf tools like an influx TV and then wire that up to Grafana for visualization. So we were able to architect a solution for them that doesn’t require a software engineer now or in the future. It’s a lot of configuration to get it off the ground to design it, right? But our vision is that, as they roll out system after system, this is going to be very flexible and something that they’re going to say, oh, I need a chart for this. Oh, let me go find that. And they can create the visualizations, the alarms and everything without writing code. It’s the ultimate in being able to cut out that middle layer of complexity and get the operator immediately involved in the data.

 

Etienne de Bruin: [00:05:54] The concern I feel when I hear that is that the development team has a lot of fun putting it building blocks in place, but the end user can’t bridge the gap to actually do what you just described. So do you just train them or do you have to have a special kind of user that’s willing to jump in and work on this unknown system or what? Just because there’s no code doesn’t mean that it’s still scary system to the end user, right?

 

Matthew Ferguson: [00:06:31] Well, there’s the benefit in the system that, you know, there’s a human machine interface, you know, very LCD.  Its sitting over here for, you know, if somebody is at the system and they don’t have any understanding of how to enter the data and they just want to push the red button or they just want to see what’s going on. But the operators who need to dial in are more sophisticated and understand how the system is connected. So there’s a little bit of both, right? And I think that we’re taking advantage of the fact that the folks that are dialing in or need to work with the system are a super user, if you will.  And they are truly someone who understands when when the system is cycling like this and we’re getting this many millivolts out of the bacteria that’s processing through the water, then, you know, it has a meaning. And obviously that user is a sophisticated user. But it’s part of why you would be in the back end of the system, right? Now, the thing is, as soon as you open up the world up of flexible reporting, you always run into that situation where I can build something I don’t understand. It’s a double edged sword, like what you’re saying. If you expose it all, then are we really enabling someone to do something that shouldn’t be done? In our case we’re not actually creating a system that allows them to manipulate, just create reports, dashboards and alerts.

 

Etienne de Bruin: [00:08:21] Which, in your case, just in the low code or no code approach, enabled you to do it quickly without buggy software that has to be patched. And you were able to just build it in sort of a low code environment for your own maintenance.

 

Matthew Ferguson: [00:08:42] Absolutely. Yeah. Stand up and focus more on building test harnesses and focus on making it very, very solid and less on the intricacies. So taking a systems approach instead of a, “I’m a software engineer approach.” This is the tool I know. Therefore, I should always use that hammer, which I could talk about for the next hour and a half, if you’d like. I’ve just seen it too much in the past little while, you know, that when we are masters of one tool, that’s the tool we’re going to use. And I don’t think code is always the answer. Even as a software engineer and a CTO, when we are reaching out for third parties, we’re already saying, hey, there’s somebody whose already done this, how do I wire it up? And a lot of our time is now spent in writing those wiring it up code, and so how can we do that? As Software gets more advanced and software starts writing itself, I think that’s what we will spend more time doing. I don’t think you’ll ever see that we don’t need software engineers, but more and more of our time will be in that wiring up part of the solution.

 

Etienne de Bruin: [00:10:13] Well yeah, isn’t that what we do as software developers anyway? I mean, we were using open source packaging. I mean, we’re wiring up packages or libraries anyway. And so that little thin layer of business logic that we’re used to coating. And we call it software development when really we’re just finding packages and we’re using whatever’s out there to add that thin layer of business logic, hopefully in a small service that you have multiple of. We’re just saying that the low code approach is really just doing it at the application level, maybe. Same thing. That’s why I think tooling is so important in our role as CTO. We have to be aware of the tools that are out there, man. How do you stay plugged in to the tools? Like how did you find the tools that you used to do this water monitoring thing?

 

Matthew Ferguson: [00:11:27] Well, it’s interesting, you know, I’ve built an IoT platform, I guess, now 10 years ago that reads, you know, all the meters in entire parts of the country. And so I’ve been through the days where we had to write everything ourselves. And so when I was envisioning this much smaller platform in this micro system. Really, what I was thinking about was that I don’t want to reinvent the wheel. I think that’s an important thing for all of us to ask ourselves as soon as we start a project. Break it down into the component pieces: data, movement of data, display of data. And how do you solve those three problems based in the domain you’re in? Are you a data company? What is your core competency?  What’s the business you’re in? So the tools, you can’t stay up on all the tools but I think what you can do is say I have a framework for finding and investigating this domain when I need it. And if it’s every five years, that’s fine, you know. But do you have a process that says I’m at least going to go spend half an afternoon coming up with candidates and then I can go narrow that down to, you know, either with my team or with some other criteria to say, oh, these are interesting.

 

Matthew Ferguson: [00:13:03] And so that’s what I did. I didn’t believe that this thing I found out there from Sundalo was truly going to be what my final solution was. I assumed I was going to end up writing a bunch of code or my team was. And in this case, I started to build a proof of concept around it. And I was kind of blown away. I was like, well, maybe this will work. And so I think proof of concept is an important part of the process. Whether you’re investigating kafka or messaging or, in this case, we’re using MQTT. But when you’re in that domain, go build these simple proof of concepts. Prove to yourself it’s possible. Ask your team to do that same thing. I don’t think you can read a book, though, or take others recommendations. It’s not good enough. There’s a lot of false prophets out there.

 

Etienne de Bruin: [00:14:02] Yes. Yes. And it’s also very hard to measure whether someone actually understood the problem. And I think with our engineers, if we just show them that we want to collaborate with them on the prototype and that it’s not a delegation to grind them down or make them unhappy, but that it’s a collaborative experience like, hey, let’s see if we can build this quick prototype and I’m here to help. And one plus you versus, you know, it’s your job to go and figure it out. And by the way, you can’t use these tools and you’ve got to go find these tools. You know, I think in our roles as leaders, it is important to show that we aren’t really interested in that technical solution.

 

Matthew Ferguson: [00:14:52] There’s a lot of good examples of that. I think right now it’s probably the biggest part of our jobs, right, is to do that coaching. I mean, I think I’ve seen this technology stack that I’m in is unique to this solution. But also at you know, at Galley, we just reinvented our CI/CD pipeline. And I’ve always been a big believer in git ops and I’ve implemented git ops sort of hand woven previously and with lots of good technology, but still like, oh, man, there’s too much code here or too much of our own secret sauce making this work. And so we investigated and looked at a bunch of technologies and and realized that, you know, Argo was the solution we wanted to go with. We did a proof of concept and were like, oh my gosh, this really solves a bunch of things that we were going to have to do. We knew what we wanted to accomplish. But this does it without us reinventing a lot of these wheels. So there’s always these great solutions and packages like Argo that have big corporate sponsors who are fantastic. Fortunately, we live in a space that does have corporate sponsorship for these open source solutions because the CEOs of these organizations realize that’s not my core competency. It would be better to hand that over to the open source community. If we look at kafka, where did kafka come from, right? LinkedIn. And when solving a LinkedIn problem, they realize that’s not our core competency. That technology is better served in the wild with a team that’s going to service that team. Argo, same thing. You can just see it all over the place. Supporting technologies that have been put out into the wild after being incubated somewhere. It’s a great, beautiful way to go.

 

Nickolai Walker: [00:17:11] Thanks again for joining us here in the studio. We have one more interview scheduled with Mr. Ferguson. And we’re going to talk about CI/CD pipelines and how it’s all become complicated. So stay tuned for that. Now, if you are enjoying what you’re listening to, please go to the iTunes and subscribe to the podcast so you can get them fresh every week. And then also, if you would check out Matt Ferguson’s LinkedIn, it would be appreciated. And please go check out 7CTOs.com. We will see you around these parts again next time.

Click Here for Itunes

Please rate and review our podcast!

Click Here for Itunes