From Signal Processing to Open Source Security: Sal Kimmich on Linux Kernel Defense
Intro Music + Promotional Clip/Sound Byte (00:00)
“Here's where that gets beautiful though. Security is in the real world. And that highest bar of cybersecurity around privacy means that for the first time in my life, we are able to share data for anti-human trafficking globally, because that is the most sensitive data. And we are making substantial gains on one of the biggest problems in the world because security has changed the bar and allowed us to move on that. I think that's really beautiful. That's really important work. And for OpenSSF, I'll be here till we don't have cybersecurity problems.”
Sally Cooper (00:27)
Hello, hello, and welcome to What's in the SOSS? An OpenSSF podcast where we get to talk to amazing people. And these are the people that make up the open source security community, the maintainers, the open source professionals, architects, engineers, and leaders that are making our world more secure today. And I have a very special guest.
Sal Kimich, a security architect and author and a mentor to many in the OpenSSF community. So for those of you out there who don't know, and I'm sure there's very few who don't, Sal, please introduce yourself.
Sal Kimmich (01:08)
Hi I’m Sal. Yeah, so yeah, I work in cybersecurity because I've worked in and across computers my whole life. I think for me, I’ve uniquely been in open source from every single consumer perspective in academics, in national defense, and as a consumer in a corporation ranging from like banks to healthcare. And I I enjoy working around the Linux kernel because it allows me to take – what I call “trout strategies” when I need to, which is go upstream and fix the problem where it needs to be fixed in the kernel. And I enjoy it.
Sally Cooper (01:54)
That's great. I know you do enjoy it and you do a great job. Sal, okay, a hot topic right now. We are in the age of AI and I know everyone right now is trying to figure out what actually matters. Do you? Well, I know you have a specific answer to that, but can you share it with people here?
Sal Kimmich (02:14)
Yeah, so I think if you're building or using MCPs, you're in a situation where you are just expanding the data that can be utilized for an AI system. And the problem isn't really with AI or with accessing data, but it's the kind of leakage that you can get with object orientation. And I think that MCP is one of those beautiful use cases where like we're finally outside of like financial services or healthcare where you cannot
disclose some data. And so those best practices around security and privacy are finally coming out into public. Almost every problem that you're ever going to see in cybersecurity comes from the ability to have pointers that change mutable data. That means that you've built a system that has things anywhere from a policy all the way down to the configuration of your workloads that can be changed.
if there's a clever enough cybersecurity attack. And what's really cool about that is that it brings the focus of the threat model around the kernel to protecting it at the kernel level. And there's projects right now that are sandboxing that. So as a consumer or a builder of MCP, think about building it in a way that protects your data, but also consider the fact that it's one of those problems that needs a trout strategy, has to be fixed at the kernel level so that everyone can build.
Sally Cooper (03:43)
that trout strategy. It's great. Good takeaway. Okay. So you came from a signal processing PhD and that's sort of an unusual origin for someone working in open source security. How does the PhD connect to some of the things you just named?
Sal Kimmich (04:02)
Yeah, well, I will say my PhD was so different for most people's PhDs, and it's probably the reason why I thrive in open source. So as an undergraduate in the United States, I had my undergrad funded by National Institutes of Health, and they funded my research. And then from there, I was accepted into an accelerated PhD program. So it was only three years, no classes, just research. And I could do that research between
National Institutes of Health headquarters outside of DC in Maryland and at University of College London. What's really cool about that is that they accepted me into the PhD program and then they told me I could choose any lab that I wanted to work in at all. So I chose a really cool project which was working over neurodebian. So this is a distribution.
that works over the Linux kernel and it allows us to do extremely parallel processed massive computation when we're trying to look at how the brain behaves in as close to real time as possible. And I was building algorithms that allowed us to actually see the brain networks in real time instead of having to process them afterwards. And so I was working in open source and it's always interesting because like NeuroDebian, I'll go to a conference for that. There's 35,000 people.
Like, the extensions and massive, massive impacts of the Linux kernel. You just can't really imagine until you actually see it. And yeah, so I jumped into that. I fixed up a couple of Python scripts and people were really grateful and that's how open source started for me. But what really...
changed about that for me. What was interesting is I was working on really computational projects early on and now AI is all about AI/ML advanced computation. All of this requires really strong concurrency models and it used to be that if I wanted to play with something that I'd be playing with today on either hugging face or even like a clawed model that I would have to run it through an IRB, ask an institution and then get funding to do this open source work.
Now I can do the open source work in a day and it's kind of beautiful. But yeah, I think the signal processing was interesting because what I was working on later was figuring out how to look at large distributed systems of human communication. And that's where my attention really came to the Linux kernels.
And I enjoy being part of the only large distributed system where as a civilian I can have an opinion. And I think that's really cool. That's what I try to teach people they can do. If you've got something you really need from Linux kernel, go ask.
Sally Cooper (07:07)
That's beautiful. Okay, you've written about nono.sh as an example of what gets missed by teams focused on Kubernetes security. What is actually happening there?
Sal Kimmich (07:20)
Yeah, so I think this is a cool moment in time because a lot of the attention around cybersecurity is in Kubernetes, which is fundamentally an application layer of security. Now, if you want to open up your systems to something like agentic or MCP, then you have to be really, careful about the layers over the chip. So the chip in itself is an inanimate object.
You can bootstrap a kernel onto it, then you have to be in the relational space of the logic that it provides. And unfortunately, with security, if you loosen things up at a higher layer, like application, you may lose security at a lower layer and never be able to observe it. That's really concerning because someone else has complete control over your chip.
So Nono is very, very, very cool. And I'll hold this in the perspective of in the last 48 hours, if you're in cybersecurity, I'm so sorry. You've probably been experiencing copy fail, which is just the latest breach over the Linux kernel. It hits the cryptographic libraries, the cryptographic subsystem of the Linux kernel, really major. And it's been present for nine years.
It takes a little over 700 lines in Python to execute. So if you look at the application layer and the kind of security that people are doing there, would you have been able to capture this in your S-bombs traditionally? No. Would you be able to capture it now? Yes, because there's a project out there that if you have an open source project, you should be running darnit. It's very good. It comes out of the open SSF suite and it builds on what we've been doing. You should run darnit.
make sure that you have that first layer of if there's known vulnerabilities fully set, you'll be able to have those covered. Second to that, I'm gonna make sure that everything that I bring into that system, everything I check in the supply chain is cryptographically observed in a way that cannot be tampered so that I can trust my S-bomb. That's just about bringing non-rotten fruits into my fruit basket, right? That's pretty simple, but...
Sal Kimmich (09:25.844)
There is the ability to have protections at the kernel level with specifically Nono. This is something that we have to put in place, something that has not been built before and that there's no, it's one of those beautiful solutions where I don't think there's ever going to be a competitor because it's a very simple thing. You must make sure that the kernel itself is sandboxed if you're going to use agentic systems because if the agentic system breaks out,
of what it is intended to do in a malicious manner, it's going to point down, not up, because it doesn't want you to see it. No, no. If you have that installed, you would not have been caught by copy fail, because it would not be possible. So the trout strategy is for those things. Upstream, get it, do the kernel level hardening. When it's going to fix the threat model for my banks,
my hospitals, and myself.
Sally Cooper (10:50)
That cross layer thinking clearly extends beyond the technical stack. And I know you've spent years working across foundations, CCC, CNCF, OpenSSF. How did operating in those environments shape how you think about this work?
Sal Kimmich (11:08)
Yeah, all the acronyms of the Linux Foundation. So I work more broadly in open source than the Linux Foundation, but I do most of my public discourse around that kernel because it's effective. And that's because unlike, I'll take the FreeBSD kernel, for example, it's going to exist, it back propagates, it has pretty good security principles. But the Linux kernel is this concentric circle.
And at the center of that circle are critical services. Those are all the things that I rely on if I want to live in society. If I want to use a hospital, if I want to use a bank, if I want to use an energy grid. And all of the cybersecurity attention from all of those sectors is pointed at the center of that. And I hope that's one thing I wish people really understood about the Linux kernel is if you were to remove it, from society today, society would break down.
And so it's going to exist. And you have to ensure that like those critical services are good. But the outer layers of that, what are the secondary communities, tertiary communities? So maybe I'm a researcher in a hospital instead of the cybersecurity administrator. Do I still get those patches in time? Those are interesting conversations. And
What I enjoy about the Linux Foundation space and I move strategically just where the threat models are. So CNCF, that's the largest exposure for critical infrastructure resources. CCC in the Linux Foundation, which is the Confidential Computing Foundation, that's where you go if you cannot afford to have your data disclosed end of story. And I'll tell you just one quick thing about the CCC. It's the reason why...
I went to do that work and then we can move on. But I think it's so interesting to think that cybersecurity sometimes is so good that it's above and beyond what regulation expects. So here's an example. Confidential computing. Chase Bank has been running a confidential computing version of its full workloads for several years. I would love to put it in production because it's a higher quality standard than the security you currently have in financial services.
And we can't because banking regulation has specific language that doesn't even recognize the ability to not tamper or access data. Here's where that gets beautiful though. Security is in the real world. And that highest bar of cybersecurity around privacy means that for the first time in my life, we are able to share data for anti-human trafficking globally, because that is the most sensitive data.
You're not allowed to copy it. You're not allowed to share it unless it's over a confidential computing server. And we are making substantial gains on one of the biggest problems in the world because security has changed the bar and allowed us to move on that. I think that's really beautiful. That's really important work. And for OpenSSF, I'll just be here till we don't have cybersecurity problems.
Sally Cooper (14:37)
Sal, got chills. So the intra-foundation work, I'll switch it up on you, just like the security slam, Alpha-Omega, you've had, you know, years with working with different environments. What would be like a takeaway from that? What did it actually teach you from being involved in multiple foundations work?
Sal Kimmich (15:02)
Yeah, well, working across several sub-foundations in the Linux Foundation really requires you to understand that it is a formal antitrust. think, thankfully for me, the part of my education I didn't bring up was as soon as NIH was willing to fund me as an undergraduate, I asked, is this unlimited credits? They said yes, and so they paid for a second degree. I have a whole degree in public law, and I was fascinated, fascinated by antitrusts.
and one of the largest antitrust in the world is the Linux Foundation. Why does that matter? It's the reason why all of this is open, right? So once you're in a specific portfolio of the antitrust, it's because it's trying to develop an ecosystem level of coherence and non-wasted developer time. In cybersecurity terms, it means everyone is aligned around the same threat model, which is super important. And it's the reason why I stick around, because it's
really rational. So I think what I found was really interesting is in order to keep that coherence, you have to be careful within those foundations to maintain those boundaries. That's where and why Alpha-Omega is very cool. So Alpha-Omega is a slightly independent fund that allows you to move very quickly and insert cybersecurity interventions when it's absolutely needed.
So I wanted to do a security slam over CNCF projects several years ago, and it's still running, which is great. But when I wanted to run that, I was representing OpenSSF to do that, but it was putting that work into CNCF. You must work with Alpha-Omega to do that. And I have to state that on this call because that's one of the most common conversations I have with young people that are jumping into cybersecurity and trying to understand where and how things move.
There's fast and slow cybersecurity. Fast cybersecurity is going to be alpha and omega. It's going to move quickly and quietly. But if you want to stick around the things, and you should for cybersecurity that are slow moving, good, and going to remain useful, you look at public repositories. You look at openSSF.
Sally Cooper (17:21)
Love it. Okay, I've heard you coined something anti-OSPO. What does that mean? Can you describe?
Sal Kimmich (17:28)
Yeah, yeah, it's interesting. It's how I explain to people the way that I am personally working through the Linux Foundation. Because an OSPO is something that you will run into all the time, right? It's a centralized team representing a company in order to be able to move and understand their own intellectual property inside of a very large antitrust system.
Opposite approach to that is what you have to do if you're coming in as just a person. So as a person, I've got a couple of hours a week. Otherwise, I've got my attention off of the kernel. I just want to remain up to date. And I have built within Gadfly the ability to bring in experts from massively different fields covering massively different portfolios across the Linux kernel and other kernels.
But an anti-OSPO works like this. Instead of saying, what did you do for the company this week? Everybody gets on a call and just says, hey, what did you learn? How did this move your work? And uniquely, because we allow for people to show up for their own personal projects and work and movement, I get so much better information. And I don't have to sit in the calls around embedded security.
because I have someone sitting in that call for their own education already. And that's where my mentorship comes in. When you see Gadfly or you see someone representing Gadfly, they're basically doing the world's most intense independent study. I am asking them to go in and evaluate the entire system. I'm asking them to evaluate every working group. I'm asking them to evaluate the entire foundation and how will they move? What do they need from?
and what can it provide? Because the beauty of the Linux kernel is that you can participate no matter what hat you're wearing, and if the only hat that you're wearing is that you're somebody that has a hobby project and you really want to get it right, you should still show up.
Sally Cooper (19:41)
Yeah, the mentorship. I know you've done a lot with that. Specifically thinking of Ejiro, because she is one of those people. Can you tell me a little more about her and the remarkable work bringing cybersecurity awareness to Africa that you've witnessed through your mentorship program with her about the work and what you saw in her specifically?
Sal Kimmich (20:03)
Yeah, Ejiro is great. So for Gadfly, if you're gonna like stick in and you've got something you really want to work on, I have a couple of questions that I ask. Because at first, Ejiro approached me and just said, Hey, I want to be working professionally and openly and open source security. And I said, that's great. I'm going to ask you a few other questions. And I said, what is the one problem that you want to make sure that you solve in your lifetime? What is that?
That's why the Gadfly system is called Gadfly. It's a Socrates idea. The second question I ask is, who is one person living or dead that you would love to describe the solution to after you get it done? And more often than not, it's a family member or a close friend. That's good because I need them to remember that when open source gets hard. And then number three, I ask them, is this a problem that can be solved with open source or not?
So those three questions are my first barriers to entry before I'll even start to talk to someone. And by the time Ajiro got through those questions, they had switched up their answers and they had understood what they were looking for. Africa is a massive continent that if you're in the United States could fit several United States inside of it. It has very different cultures. It has massively different cybersecurity needs depending on the country.
Ejiro is based out of Nigeria, but is likely moving to the UK at this time. So I want her to be really, really expert in being able to pick up technology and communicate technology. So I brought her to OpenSSF and I thought that was going to be it. Here's what she's done. She has decided that what she wants to do is bring cybersecurity to Africa.
For the last two years, she's been doing a bunch of work that you don't have awareness of. She's been working with code roots. She's been putting out beautiful, really interesting talks. And we've been engaging with rural communities in Africa. We've been engaging with universities and getting the answer to what does it look like to provide cybersecurity education and hardening to this country because for the continent that is moving the fastest in tech - we do not have enough communication awareness out there.
So she stepped up and she's really grown into being someone that just last week she gave a keynote talk in her country representing OpenSSF. And it's not just that she's good. I think once that talk comes out, everyone should see it because I'm a pretty level person, but her talk and her ability to communicate why it matters made me cry. She inspires me.
So pay attention to the work that she does. I'm paying attention to the work that she does and I invest my time in mentoring people that have the ability to make massive social movement. And she's one of those people.
Sally Cooper (23:18)
She sure is. I love her 100 days of cybersecurity posts. I follow along. That's how I started getting awareness of her. I think she's on the podcast we've recorded, but that episode will come out. If we have it out before this, we'll link to it and link to Ejiro. Cool. Okay. Last question. And I want to make it a practical one for the practitioners and organizations listening. Where do they start?
Sal Kimmich (23:43)
Well, depends. What are you? If you're a maintainer today, I really want you to be playing with Dangit or Darnit is the new repository. I really like it. It's useful. And there's definitely more nuance that could be built into it with a little more attention. So go test it. If you're a consumer of open source, it depends on where you're jurisdictionally based. But
Whether you are in the US or the EU or otherwise, I would pay attention to your SBOMs. That was a very good time to get your SBOMs right, but I want you to be paying attention and asking the question in your organization, can we guarantee that our SBOMs are not tampered? If you answer that question, you're going have a really strong cybersecurity profile. So do an audit.
of that. Go have fun. I'm tell people to go do a cybersecurity audit and have a great time. But that's really it. It's worth it. And get in touch with me if you want some help with anything along the way or if you want to get engaged with OpenSSF. It is a community. You can reach out.
Sally Cooper (24:59)
Amazing. Okay, now it's time for our rapid, rapid fire questions. So don't think too much about these first answer that comes to your mind. Favorite sport?
Sal Kimmich (25:10)
Dressage.
Sally Cooper (25:12)
If you had a security themed band, what would it be called?
Sal Kimmich (25:17)
acid burn.
Sally Cooper (25:18)
Good. Favorite open source mascot.
Sal Kimmich (25:23)
You know what? I'll just say honk for now, because there's so many iterations of honk and all the stickers are great.
Sally Cooper (25:31)
There are a lot of iterations, so that was a hard question. Tea or coffee?
Sal Kimmich (25:35)
Coffee every time.
Sally Cooper (25:37)
Nice. Favorite city for an open source event.
Sal Kimmich (25:40)
Ooh, Prague.
Sally Cooper (25:42)
Good job. All right. Well, thank you, Sal, so much for joining us. This was a really insightful discussion. We all learned a lot. And to everyone listening, thank you for tuning in to What's in the SOSS. Happy open sourcing, and that's a wrap.
Clips from this episode
You May Also Like

Hell Money
Hell Money

Machine Learning: News on AI, OpenAI, ChatGPT, Artificial Intelligence, AI Models
Machine Learning: News on AI, OpenAI, ChatGPT, Artificial Intelligence, AI Models

ITSPmagazine
ITSPmagazine, Sean Martin, Marco Ciappelli

The Paul Barron Crypto Show
paulbarronnetwork.com

The New Stack Podcast
The New Stack

MacVoices
Chuck Joiner

Bitcoin Magazine Podcast
BTC Inc

Jack Hopkins
Jack Hopkins

Day Two DevOps
Packet Pushers

The Fat Pipe - Most Popular Packet Pushers Pods
Packet Pushers
Community discussion