How to Recover from the Log4j Supply Chain Attack with Ilkka Turunen

In this episode of the HackerNoon Podcast, Amy Tom sits down with Ilkka Turunen to talk about Supply Chain Security. They go over the Log4J incident that made a lot of apps built-in Java vulnerable to exploitation, what it means to be a field CTO, how companies can place themselves to collect user feedback, and a lot more! Ilkka Turunen is the Field CTO of (Sonatype). https://www.sonatype.com/ On this episode of the HackerNoon Podcast, Amy Tom and Ilkka Turunen chat about: What is a field CTO anyways? 🤔 (01:20) How do you stay in the loop on customer needs and feedback? ➿ (05:19) How has Ikka’s job as a field CTO changed since the pandemic started? 😷 (07:30) Supply chain attacks have increased since the pandemic started. How have Sonatype’s customers and the business changed over this period? 🧰 (08:53) Breaking down how the executive order by Biden’s administration regarding supply chains is affecting the software industry ⚙️ (10:06) What is the best way to mitigate supply chain risk? ⚠️ (11:49) Getting into vendor due diligence as mitigation of supply chain risk 🚩(17:22) Learnings from the Log4J incident 📝 (22:44) Why are 40% of Log4J downloads still the old vulnerable versions? ☢️ (25:47) Log4J vulnerability resource center: https://www.sonatype.com/resources/log4j-vulnerability-resource-center Find Ilkka Turunen online: https://twitter.com/llkkaT Learn more about HackerNoon: To read HackerNoon stories, check out hackernoon.com To apply for a position, check out https://careers.hackernoon.com/ To participate in HackerNoon writing contests, check out https://hackernoon.com/u/hackernooncontests Podcast Transcript (Machine Generated, Excuse the errors) [00:00:00] Amy Tom: Hello hackers. This is the HackerNoon podcast. And my name is Amy Tom, your podcast host, and very best friend. I've got a little PSA for you about the times that we're living in and technology. I just started reading this very average romance, crime novel. It's called the Bitcoin Widower. And it's about a woman who meets a man on Tinder who owns a Bitcoin exchange, and he goes missing. And then she has to deal with all of his like Bitcoin assets. So that's the society that we're living in. Now, there are now romance novels about Bitcoin. So just a PSA for everybody that that's where we're at in 2022 now. But anyways, aside from that, I am very excited to chat today with Ilkka Turunen, who is the field CTO of Sonatype welcome to the podcast Ilkka. [00:01:02] Ilkka Turunen: Happy to be here. And didn't that bitcoin thing actually happen in like 2017. [00:01:08] Amy Tom: Probably. I wouldn't be surprised. There are probably definitely real Bitcoin widowers out there. [00:01:14] Ilkka Turunen: Um, I'm pretty sure someone actually faked their death to get away with bitcoin. So yeah, that's the world we live in. [00:01:20] Amy Tom: Great. Well, tell me Ilkka, I would love to know more about you and specifically about your title as field CTO, because really this is kind of the first time that I've come across the specific designation of field CTO. So tell me what that. [00:01:38] Ilkka Turunen: Well, you know, I looked into the wonderful world of tech titles, you know, we'll, we'll make one every year, you know, for ourselves I'm, Ilkka, I'm the field CTO at sonatype, which is a company that deals with software supply chain security and, managing open source and, all things related to that sort of thing. And, field CTO is a bit of a mouthful. But the closing, the title, field and CTO, you know, the kind of business that, us as tech companies often are, especially when we're talking about developing tooling means that, there's a component of needing an engineer to help you figure out what the tool is, how it kind of fits within your specific software delivery infrastructure, how it kind of sits there. and so what I'm responsible really is. running the teams that actually engage with, people thinking about our tools and, uh, you know, implementing them and just help them, make sure that they, put it over there, get it built correctly, you know, get it designed correctly. They're buying the right thing so that they like put a hundred grand down and, then buy completely the wrong tools and never use it again and regret it horribly. And the other aspect of the job really is the CTO side, which means that I work with our CTO office. to represent whatever we hear from customers like, Hey, wouldn't it be cool if your product had this feature or would it be cool? if you guys did this or I don't really like how that works, we collect all that information and make sure that it gets, taken in by engineering and product management. So they have accurate data about what's actually being asked, what's kind of going on in the field. So I looked after solutions, architects. I look after developer relations. I look after all they sort of cool things. that kind of happened around the company that, has folks talking to other people that don't work for the company. So, yeah, that's a, that's a very long answer. The short answer is when the CTO who was in America is asleep. I usually pick up the phone calls. So in that, [00:03:26] Amy Tom: right, how many people are in the CTO suite? [00:03:30] Ilkka Turunen: Yeah, there's just two, there's Brian Fox who our co-founder and CTO, he's the actual CTO and an amazing tech God. And then there's me. Who goes, all right. That's uh, that's how you say to the world, like here's how we act, what he actually means to the people that have to implement this thing. So that's basically the two of us. [00:03:48] Amy Tom: That makes so much sense because I think talking to customers and being on the ground and understanding the technical aspects of your customer's environment is such an important part of being a CTO, but it often gets so overlooked because there's so much other stuff to do. So that's very cool and exciting that you guys have segmented it that way. [00:04:06] Ilkka Turunen: Well, it's also, a reality in the field. often as a tech company, We're thinking about the future, how things should be and how should you structure yourself for your development in the future? You know, isn't exactly an exciting thought for most people's everyday life. So there's a long way of getting folks to understand. Hey, okay, you're here today. Here's where you could be, but here's step one and two that you need to take today or tomorrow in order to get there. so it really helps having that sort of tight visibility of, Hey, here's what people are actually struggling with in their real, everyday working lives. And here's how we can help them. when we were thinking about what products should we build next, or what features should we implement next? It's always an eternity problem with most tech companies is how do you like accurately do that? Cause everybody's got a gut feeling and everybody's got a vision, but that might be completely. misguided, compared to what the customers are actually telling you, or even people that aren't your customers yet, who are really the harder crowd to attract, having that sort of tight, one foot in, in the outside world at all times, and being able to talk about. It meaningfully, when we're making these product and investment decisions really just helps, the company do the right moves and, built the right thing so that, six months down the line, the customers don't scratch their heads and go what's this, then once you build it. [00:05:19] Amy Tom: Okay. I need to get into more of the, how of having your other foot in your customer's door and like in your customer feedback. So how do you do. [00:05:29] Ilkka Turunen: Oh, yeah. I mean, uh, he, no days like the other it's very much a day to day. so, one of the kind of key things that all the teams that I work with, do, and all my colleagues in, in, in software, solution architecture and in developer relations, they're really out there every day, either as a part of a ongoing engagements for our customers or evaluating our tools or trying out new technology, or going out to conferences, going out to communities and working with them, just listening what's going on. And so, as a part of it I might do any of those things and, all the rest. And sometimes it's about just listening. you might tell me that, Hey, something's going on in my company. I really just want to solve this problem. I just want to turn on a feature and forget about it. But actually you might not be thinking about it at the proper scale. So we're helping you to figure that one out. And as, as we have these conversations, like, no, company's the same. it feels like everybody thinks about the same things and thinks about things in the same way. But actually when you get into the specifics of every single company, like absolutely no companies, exactly the same, there's always like somebody invented some new way of doing things. Somebody's DIY to some part of their, production, set. So having that sort of sense of, Hey, here's what, how other companies do this thing? Here's how you can gain some extra advantage by doing that thing. Maybe, in a custom way for your thing kind of helped you in having that sort of sense of what's going on with every customer hearing all those stories come back and. Sometimes pretty awful feedback as well. coming back and kind of taking that back in and kind of making sure that we've got a process that, associates that with the customer, their value, their, adoption helps us then when we're looking at things in the cold, hard light of day, we've only got so much time in the world and only so many teams, what can we do and what will make the most impact to most customers. So w we try and take kind of a. Data oriented way of making that decision. obviously mixed in with our good senses as well. So really it's, it's all in all in everything, one day I might just be, doing a white boarding session next day, I might be talking to executives about how to measure a medical. [00:07:30] Amy Tom: Yeah. And how has your job changed since the pandemic has started? Because I imagine like, as a field CTO going out on the field, you would have had a lot of in-person things. And now maybe more like some zoom time. So what's going on there [00:07:47] Ilkka Turunen: a lot more time. I'm sure. I mean, that's the beauty of the software industry in general, right? Do you know by definition, we're all very asynchronous. You know, the funny thing is when the pandemic struck, when everybody kind of went remote working, I don't think I really saw any differences. So for teams, other than, Hey, I can just turn on my, the slack at home instead of, uh, anywhere else. so we were already very used to talking to customers, and companies just remotely, and actually in that sort of sense, if anything, it just made it easier to talk to them, directly because you don't have. Ceremony of having to book a room and figuring out who gets the coffees in and, and things like that. on the other hand, yeah. It, it certainly did it change because, you sometimes hear the most honest feedback after the meeting in the whole, on the way out. and, those sorts of moments you now have to think about, you know, how do you make sure that, you know, we're, we're running the meeting in a way that allows people to express that. and yeah, it's certainly been a change for sure, but I think, Overall, the, actually like the actual specifics of it haven't really changed that much. It's just a, you can stack more of them in half an hour increments now. [00:08:53] Amy Tom: And what about how your customers and how the business has changed in general? Since the pandemic, with regards to like the increase of supply chain attacks. [00:09:02] Ilkka Turunen: Well, that's been a big phenomena, for us. So, we, have seen quite a lot of increase in customers thinking about supply chain attacks, just generally companies that we don't really hear from before really getting quite serious about it. You know, it's kind of follows over. The last few years, feels even weird to say, but we'd go to events like Coco and supply, uh, Soloway and, uh, obviously log for J happened over Chris it's Fremont anniversary tomorrow, actually. So, it feels like it was so long ago, but it was literally just, yeah, pretty much less than three months ago today. So, customers are starting to pay attention to it. and they speed and also, uh, added onto, by the fact that governments have started issuing, decrease, you know, the Biden administration putting out. The executive order about dealing with the supply chain. So it is certainly, increased activity, quite a lot. And the conversation is no longer about a change agents. You know, someone who was really enlightened with DevSecOps, really wanting to change how companies are working is literally companies taking kind of a board view of, Hey, we could have a mitigation to this. Cause we going to get asked about it, which is definitely a chance. [00:10:06] Amy Tom: So are there now under the Biden administration, are there now compliance regulations with regards to monitoring your supply chain? [00:10:15] Ilkka Turunen: Yeah. So there's, there's literally a November, a executive order that came out and it's called love Americans. The goal was go great titles for these it's the, uh, executive order on the nations, uh, cyber security. And in there, there's an entire segment about a software supply chain security. And one of the things that they say. Is that, they will come out with minimal sulfur bill of materials standards, and they will require every company that sells software to the U S government to provide a list of ingredients, so to speak. And at least have every piece of open source that's been built into the software that's provided and, assurances that it. The pieces of source or the pieces of the supply chain goes wrong, that the company has a process to find them mitigate it. and that that's a big change because, historically. When you talk to companies and you say, Hey, did you know that 90% of your software is actually external? Like you didn't write it yourself. It came from some person somewhere. and you're just adult that it, cause it was you, it was useful. most companies would have go, go on a complete blank. So for them to then say, Hey, we're going to require this from all of our suppliers. It's a very big change and a very significant shift in standards. Now what's happening now is there's actually. Supply chain security standard that's being worked on, by, uh, NIST, which is like the national standards organizations, like really exciting stuff. I'm sure. but basically that will have a big effect on, you know, just what the baseline of good security should look like. And I think, you know, supply chain is going to increasingly get, get pulled into that. [00:11:49] Amy Tom: Yeah. Would you say that the best way to mitigate supply chain risk would be vendor consolidate? [00:11:55] Ilkka Turunen: Yeah to a certain degree. I mean, it, it depends on how deep you want to go, but the actual fundamentals of this come from the sixties. Because in the sixties car manufacturers, when they invented something called lean manufacturing, which is really the core of how we build software today, you know, lean and agile essentially, essentially have that the person that invented that manufacturing, style was a guy called w Edwards Deming. and he spent a lot of time in Japan in kind of the fifties and sixties rebuilding the industry. And one of the things that he realized, as a part of like figuring out how to get Toyota, to be the biggest car maker in the world, Was that they were using load of third party suppliers. Like, they had a local Foundry that would manage their nuts and bolts for them, like would just literally cast them on the spot. So no two cars really had the exact same parts and he realized that, uh, actually. understanding what actually went into your car. Like, you know, if, if there's a problem in any part of a car down to a bolt, they'll issue a recall, right. You know, you'll have to drive your car and they'll do something, you know, change the airbags or whatever. If I asked anyone writing software today, could you do. Well after log4J, I think more companies will say yes, but they really, even six months ago I wrote, it was like, what do you even mean? Like what what's that? and he, he realized that actually reducing the amount of externally built solve for parts in those cars. It's really, really useful, but also just like basic stuff. Like, Hey, have you looked at three different bolts and decided which ones are the best? you'd be surprised at how often did the answer is now. I just took the first one that looked right. and all of those lessons that he made, you know, reduce the amount of suppliers, reduce the complexity of your supply chain, you know, uh, understand what parts are in your car, et cetera. All of those principles are actually really applicable to software when you follow all of them. Um, it turns out, you know, that's, that's what it takes to start managing this whole. [00:13:45] Amy Tom: Right. I, and, but like when you refer that to software, right. And you're you think about like, as an organization in security and operations, all of the different applications that I need to have connected to my network, and. All of the different, open source code and everything. even if vendors were to provide that information to me as a business owner, as a security person, how am I supposed to say oh, okay, great. This person is using this open source code. cool. Onto the next, how many is too many? how do I manage my environment? [00:14:19] Ilkka Turunen: It's a really good question. It starts with, you have to do a little bit of self seeking, and figure out what is your tolerance. Right. You know, we use the bark, but it's policy that helps you and helps you define that. So in most cases, it's not just the amount of open source that's, that's meaningful, you know, some projects like today, when you look at. The piece of, um, you know, Java, for example, easy to see 150 300 pieces of what a third party components, uh, in their JavaScript. You can probably draw zero at the end. You know, if you've ever heard of dependency help, that's what they do. And that sort of, uh, that sort of sphere. Um, the amount of open source is less meaningful. What's really important though, is understanding exactly what you do have, like in every shape meant what's the complete list of ingredients. Like not, and not just the open source packages, but like the Docker container that you run it in, or, the network of packages that are installed in your operating system. it's a tough ask. Most companies can answer this today, but then in order first to manage anything, you have to understand what you have, But do you have in release 1.0, what are the pieces of open source? Second question then is do you understand if there's any existing security bundle of tests, like most hacks? Although we talk a lot about zero days and you know, kind of cool new attacks coming out the most exploits, the actual breaches of companies that happen. On these new, new fad things. They're on vulnerabilities that were discovered five years ago that were kind of briefly in the minds, hearts and minds of security teams and developers, and then probably for golden light. And now when you think about load for J today, most companies that I talk to, they're kind of getting to that sort of hangover phase off while we had a big. Fire braille. We kind of figured out to mitigate it, but there's this long tail of stuff that probably has it, you know, he didn't even, you know, as a part of another component or something like that. And, they're unable to find it and that leaves a lot of risks on the table. So kind of similarly with open source when you're, when you're accepting degree, it might just start with, Hey company, you're sending us software, we're accepting it. Like just asking that very basic question sometimes gets a lot of our heads spinning going like, well, actually, I don't know. What do you mean what's in it, like not accepting complete black box deliveries can be useful. The second part of it though is just very basic. Does it have any known security benefits? If so, are any of them really critical? in software engineering, we've got this saying. what's the fastest and cheapest way of dealing with technical debts or bugs. Well, the earlier you find it, the faster and easier it is to fix similarly with security vulnerabilities, the earlier you find it the faster and easier it is to fix. And the fast system cheapest way of fixing a security vulnerability is not having one at all. So if somebody tries to deliver you a software, they've got a software bill of materials. You're looking through the list and saying, Hey, wait a minute. That has a known vulnerability. It's like a 10 hour. Don't don't accept it. Just like leave it at the door. Cause that's like a ton of work off of your desk, like immediately. [00:17:22] Amy Tom: Yeah. Uh, the idea of the vendor due diligence is like so overwhelming to me because there's so much going on. [00:17:33] Ilkka Turunen: It's not something that, you know, has a, has a, really a bearing on every day, developers per se. Cause it. It feels like it's like a big, hairy beast for someone else, but actually we as Debs do all of that every single day, like everybody who installs a package, like you were just writing a code and you're like, Hey, I should probably use express to, do this, quick and dirty web server. Well, congratulations. You've just literally downloaded a hundred packages. Hey, I should, I don't know if the frontend should be React. Well, there you go. That's a 200 more packages in your node modules. Hey, I should run this in a Docker container. Well, that's the four layers and just asking the question of, Hey, where did this all come from? Sometimes lead you down a hellish rabbit hole. And most people go, you know what, as long as it works and I don't get breached, that's probably okay. and, that's the kind of track that a lot of people fall into. It's like, learning to wash your hands before surgery. it feels like a completely disconnected thing. I'm just washing my hands, like nothing to do with cutting people, open healing, they're in it. But turns out if you've got bacteria in your hands, if you do that by one second thing, it can save a lot of pain down the line. and similarly with software stopping and thinking for a second and going like, wait a minute. Yeah. why did I pull this in, in the ad? Like, do I really need, this can kind of help you a lot down. [00:18:51] Amy Tom: Yeah, supply chain attack scare me so much too, because like, you can feel so secure. Like you've done all of the things that you need to do all of your due diligence, but then like it's a third party that's gonna like screw you over at the end and you have no control, the lack of control, [00:19:12] Ilkka Turunen: that's the typical, symptom of, having to come up with a new, discipline. I think we're starting to realize, right. Is that it isn't just like, Hey, I should be mining about everything. I think it's really a new kind of muscle that every software company has to, has to develop because you're absolutely right. If you can do everything right yourself, but if you'd have in mind, And 90% that you didn't write yourself, but that forms your software. That can be a problem, A really, really big problem. And it's not just, the extent the supply chain it could mean your operating systems and things like that. Very quickly, you go down a rabbit hole wall. Well, it's too big to even think about icon, do anything about it. So I shouldn't do about it. And that's a very. Quick trapped that we all fall into, like the first thing to managing anything is to understand it. And it is all of it's hairy goodness. And it doesn't mean that you need to take very big steps to start, like making a very big dent in it, just by running one of, one of the many security scanners, out there, can help you a lot with that sort of thing. it's not just you that's feeling that dreads. we're also seeing, actual, bad actors, ransomware gangs and hackers and script kiddies, and everybody in between kind of going like, well, wait a minute, instead of hacking you specifically, if I hack, the OpStream and I just put a fake package out there, that might get spread to. Thousands, if not millions of people, and even if it's off for just a day or two, that's enough of a downstream effect. That's unfortunately the reality of the way we write software today. And it certainly something that that's causing a little bit of a heartache and then, [00:20:44] Amy Tom: you know, like how you'll occasionally run into these CTOs where they're like, my security is. Perfect cat. No one can get in, no, you're lying. If you have even one, like other third party in your network, you have a security vulnerability. [00:21:02] Ilkka Turunen: It's like just, a game of broken telephone. If you're borrowing something for someone else, you don't understand it fully, that leaves things open. It. There's no such thing as a hundred. Secure security. Like, there's just absolutely. If somebody tells you that, just walk away and go somewhere else, but that's a complete lie. Um, yeah, yeah, exactly. You know what? I'll take my business elsewhere, but the secret to that is saying, you know, why. It's like running a marathon, you can't run it off cold, you know, and no matter how hard you try, but if you keep at it every single day, uh, and it keep going, going at it and keep trading and keep running a little bit more, it turns out you get pretty good at just running in general. Right. It doesn't mean that you need to start very big to make quite a very big impact on it. And I think. Companies that are resistant to it at the moment, because they're like, well, Hey, you know, it's the dabs job. I don't really want to think about it. It's like, well, wait a minute. If your devs use open source and that license had the license that they, adult that has a very strong, co-chairing clause, then you need to share your like entire code base with the world. Are you, are you willing to do that? And that often gets folks scratching heads. So it's, it's not even just about security. Always. It can just be like, well, You've got 10 teams and every team uses a different front framework. Why is that? Like just asking sort of very fundamental questions and people go, well, my devs choose their own tools. Like. That's absolutely what should happen, but like, do you really need like four different frameworks for the same thing? Like, they're all doing the same buttons soon day, like be sharing components. It's like very basic stuff and people go, oh, you mean like, like I can just borrow code from that team. Like yeah. That that's, that's kind of the. Yeah. [00:22:44] Amy Tom: Yeah. There's, there's just so many aspects of everything to think about insecurity. So I want to ask you more about your thoughts on log for J , and just what you think the industry has learned from that. [00:22:59] Ilkka Turunen: I think the industry has certainly learnt that a fire grills, just before, the holiday season is always fantastic. So look for J a was described actually by the director of Seesaw, the cyber information security, center in the U S, which is their top agency for dealing with this sort of incidents as the worst security vulnerability, in her entire career, which is saying something, cause that's, [00:23:23] Amy Tom: that's like, wow, [00:23:25] Ilkka Turunen: there you go. Very, very hairy beast. And the reason why. Is two things. First of all, look for Jay. He's a really popular open source component. My employer Sonatype we actually run one of the custodians of one of the largest, registries of open source, something called Maven central. Basically every time somebody's. Java code or Android code or anything that's related to Java like Scala, typically the third-party stuff comes from us. We run the servers and the CDN that, the provides them with the dependencies. And when look for Jay happened, we actually ran the numbers. Look for Jay Is in the top 0.03% most popular open source component. There's over 8 million open source components and central it's like a lot. And since, look for J the vulnerability was announced since. Up to last hour average, you got the dashboard here. There's been 30 million downloads over 30 million downloads of float for Jay since. So it tells you a little bit of the volume of it. Just to say, look, what Jay is very popular is spread everywhere. And as an attacker, it's a very simple assumption to make. If I find a piece of Java software, it probably has looked for J in it, because look for Jay deals with logging, writing. Right to your soul for us logs down for you. So it's a very, very ubiquitous thing. Everybody needs to do it. So everybody probably has it. And the security vulnerability that was discovered is very easy to export it. I can train anyone in the space of fight. To run attack code on log for J. So when you combine something that's very widespread with something that's very, very easy to run. Like any, anyone can do it. That's where the danger. So it happens. So, when Jake came out there were like people put punching it in, in their Teslas, the attack string and getting like responses back, knowing that, Hey I could have done this. And that's why actually most governments tilt their at teams that they have until about Christmas. To actually find and mitigate it. It was so big and so widespread. And even today, when I look at the download counts, we publish a dashboard to the world about how the weld is fixing for Jake, because we can see the, download volume off different versions to this day. 40% of all Donald's for what? For Jay are still the old vulnerable versions, not to the new fixed ones. [00:25:47] Amy Tom: That's crazy such a big, widespread attack that got so much media attention. Why do you think that's the case? [00:25:53] Ilkka Turunen: Yeah, it's because it really is so easy to exploit. It's pretty much anything that runs Java. We'll probably have it and because it's so easy to exploit, it's very easy for bad actors then to go out, just literally spray every IP address on the. And see which one's thick because you don't really have to have finesse about it. You just need to like knock on the door and see which ones are vulnerable. And then you can come back later to do it. So [00:26:18] Amy Tom: kind of [00:26:18] Ilkka Turunen: attack. Indeed. It's like later just walking down the street, trying every door and every window and seeing which ones are open and then going all right. I'll come back to this later. Exactly. The same strategy actually. Yeah. That's the dangerous, because it's so widespread and so widely adopted you can make an assumption that maybe 1 0 1 or two of those companies will probably forget one of the older pieces software that they have. and that's what made it so scary and widespread it's because it's not just the code that the software, the companies wrote themselves. It's any piece of Java software that they've ever bought in for Gordon to patch runnings in some server somewhere that might have Nick broker access, that's the liable to that attack. And that's why it was such a big deal. And why it really broke into the mainstream news, which is something that we've not seen before for a vulnerability like this. It's just the fact that it's like somebody telling you that the concrete in your house turns out every piece of concrete poured in the last 30 years was bad. And now everybody has to fix it. It's that magnitude of an issue for Delta programs. And that's the way the. The odd reason why it blew up. And I hate to say, but we were in the long tail now where most companies probably ran a ton of fire drills because often when stuff like this happens, it's not the actual original vulnerability. That's the problem. It's because people start paying attention to the technology like that. So as soon as that happened, Every single security researcher in the world start looking at every other logging framework. So within the space of like three weeks, we saw five other security vulnerabilities come out in, look for J itself in other logging frameworks and other things. So all of a sudden everybody's minds are focused on. That single method of attack. And, it's like somebody finding a very fundamental flaw in the logic of how that code is written. It, it spreads out and that's really the next dangerous, like, it wasn't actually just that thing. It's now we're at the, like what six releases from that original vulnerable versus there's been like back ports. There's been buying fixes fixes for fakes, his biggest thing, introduced new vulnerabilities. And, the project's been very good at providing them. They're like, they turn them out like beasts, absolutely. I'm astonishing how good they're dealing with it. But as a receiver, there comes a point where you'll just like, your inbox is filled with, Hey, update this update that eventually you go, eh, you know what, can't be bothered anymore. I think like the biggest thing is over now and that's really the dangerous lie that we usually tell ourselves. So I'll get to it tomorrow, I'll, I'll look at it again and, you know, So that's kind of the face that we're in with what, for J now, like the acute big, uh, sort of, you know, skinnies trying every single server face is probably over, but just yesterday or a couple of days ago, there was a news that like actual ransomware gangs are now doing multimodal attacks using what J as the kind of first entry point level, like getting in through the door, using it. So it's, the danger is still very much out there. And I think. Just the mathematics, you know, 40% of every new bill still has an old version of log for J you know, that there's a lot of a surface up or for them. [00:29:19] Amy Tom: Yeah. Okay. So that 40%, like, is that basically saying that for like, people are not like there's so much open source and there's so much with vendor due diligence that. Even still 40% is falling through the cracks, even though we know about this vulnerability. [00:29:41] Ilkka Turunen: Totally. And you know, we not just know, we're probably bored of even talking about it at this stage. Right. Um, yeah. That's, that's exactly what it means. it's like somewhere out there, companies still don't quite understand the full chain of third-parties that they've got a full chain of open source that they've got in there. So. And it's not just your software. It's also the logging framework that you pulled in that also has look for J in it and the older monitoring software that's running on the server, that's running on top of what the JVM that probably has looked for Jane too. So it's, it's, it's all of that. plus companies probably just don't really even know how their software is actually being built, what software they got in. Like, just honestly, that, that leaves us in this sort of situation. Like it's, it's, it's a very real data, uh, data that we look at every day and we disseminate to the world and yeah, it is a little bit depressing going, like we've done all of these campaigns and all these conversations about it. And yet still to this day, it's it's. The kind of state of aligning though, there's various in countries like generally English speaking countries tend to do a little bit better. I think it's because all the tech companies are, from those countries and there's a lot of PSA's on them, but there are countries still in the world. Where are the average. Fixed version download ratio is about 30%, like 70% of all of the downloads are still to those older fundable versions. So that's the danger is the message doesn't mean get evenly dispersed across the industry. [00:31:09] Amy Tom: So what can companies do then to mitigate the risk of, uh, supply chain attacks? [00:31:14] Ilkka Turunen: Find out a way of building a software bill of materials. there's dinner, haters out there standards like psycho and DX are really, really good at that. Like you run as a part of your build a plugin. And, what it does is just literally lists, Hey, here's all the open source that went into this file, that away somewhere. Like, even if you do just. You can then go back to it later with a security scanner, or a vulnerability scanner or a tool like what we do at Sonatype to analyze it later and go, Hey, here's the known risk about it, right? It's the number one thing to managing anything is understanding what you. The second question then is, does it have any really, really bad known security button abilities? Like even if it's 10 years old, you should probably get rid of it, or you find a way of operating it or you find a mate mitigating it. It's understanding just like the task at hand is a really important thing. So it takes two seconds per build and it makes a huge dent further down the line, for you the second step then is to start thinking about, across teams or, if I just talked to my colleague in the next team over, or, talk to my colleagues in, in, in America, do they even use the same libraries that I do if they don't? Why? Are they so different? Can we make things more uniform? Because then that means that we can fix the things as well. And then just, I think the third little thing that anyone can do, like right now, it's just take 15 minutes every day or 10 minutes or five minutes every day. And just run a dependency tree or an MPM list command, and just take a look at those and ask yourself have I really thought of managing things because, open source also just releases like these packages have new releases, right? On average there's about 20,000 releases of new versions every single day. So am I staying up to date, am I keeping these updated? And it turns out the best way of me to getting a lot of this stuff is just stay relatively new. Can you make sure that you keep upgrading your versions? It is do that sort of stuff that helps you to talk like that just takes a ton of the sort of danger off the table as well. So really it's all about that, understand what you have and then just set aside a little bit of time every day managing your dependencies. If you do that, that's going to go a long way to actually just mitigating a lot of these sort of longer tail risks, that might happen. And then when something new happens, like often what we like to ask customers is, Hey, if I told you there's another log for J and it's going to come up tomorrow, what would you do? And the amount of blank stare. So you get in a room and they're like, well, uh, I don't know. It's like, well, you just went through it. What didn't you just do what you did then? Would you do something a little bit better? Just. Take a moment pause. Cause it's very, I have to pause in life, but just like take a moment and pause, like, okay, what if it did happen tomorrow again? What would I do differently compared to the last time just having that thought exercise like five minutes a day. Again, I guarantee that's going to go a long way to making sure that it's going to be a lot easier next [00:34:13] Amy Tom: time. Right? Definitely. All right. Well thank you so much for joining the podcast. It'll ELCA. Oh, where can we find you and what you're working on online? If our listeners want. [00:34:24] Ilkka Turunen: Oh, fantastic. Well, you can find me on Twitter. Like I'll handle this L L a K K a T M and a, if you want to find out these, look for J numbers, we've got a resource center up, at Sonatype, sonatype.com/resources/log4j-vulnerability-resource-center. So, um, Matt, you can actually take a look at these numbers yourself, is something that we brought in, you know, on the hour, every hour. And, what was interesting to kind of see how the squiggles, evolved. So, so yeah, take a look [00:34:53] Amy Tom: three. Okay. Thank you very much. And just so you know, if you can find hacker noon online, anywhere you look at. Twitter, Twitter, LinkedIn, Facebook, Instagram, or wherever, and where whenever it's hacker noon, you can visit hacker noon.com to read stories about technology and as always stay weird. And I'll see you on the internet. Try not to become a Bitcoin widower in the meantime. Goodbye.