Security Confidential S2 E9 Chris Gerritz

Manoj Tandon: Welcome to the webinar, everybody. We really appreciate all of you taking some time here after lunch. In the Northeast, it is proving to be a very cold and blustery winter storm day. We’re going to talk a little bit about the prevention paradox, and we’ll get right into this topic. But before we do, I want to introduce my co-panelists with me. Tyler Smith from Dark Rhino joins us. Tyler, do you want to give a little bit of your background real quick?

Tyler Smith: Sure. I am a military veteran; I served in the Marine Corps as an infantryman. I got into technology, started working at Apple, and worked in security as an analyst for a while. Then I started doing consulting and started Dark Rhino Security after doing that for several years. That’s my background in a nutshell.

Manoj Tandon: Chris Gerrtiz, I think there’s a little bit of overlap there, other than the Apple part. Please go ahead.

Chris Gerritz: Yeah, definitely. We didn’t run into each other in the military, but I did spend 10 years with the Air Force. I did everything from system administration when I was younger to leading the incident response team and the hunt team at the Air Force for a while. I founded Infocyte with a couple of those folks back in 2014. Right now, we basically focus on incident response and hunting for our partners and our customers.

Manoj Tandon: Well, thank you very much for joining us, Chris. I appreciate it. So, let’s give everybody in the audience just a very quick 30-second intro to both of our respective companies. On the Dark Rhino side, Dark Rhino is a purpose-built cybersecurity consultancy and managed services provider. That is all we do. We don’t do IT infrastructure or any other ancillary services; we’re purely dedicated to cybersecurity. Our difference is in the value innovation that we bring to our offerings so that companies of all sizes can be enabled with leading cybersecurity protection. We have a pretty comprehensive portfolio that covers most of the critical elements of the SANS sliding scale, which we’ll talk about here in a second. Chris, if you want to just quickly introduce Infocyte, that’d be great.

Chris Gerritz: Yeah, besides being a Texas-headquartered company, we are a product vendor. We build technology and a toolset platform to be able to hunt and to do incident response. There are other detection vendors out there in the world, but Infocyte really focuses on that advanced hunting use case and that advanced incident response and investigation use case. We arm partners like Dark Rhino to be able to do that kind of work.

Manoj Tandon: Fantastic. Well, with that, let’s dive into this a little bit. I think it would be really good for all the attendees if we could set a little bit of context. Just one quick note before we get into the nitty-gritty of this presentation is that everyone should hang around until the end. It is the holiday season, and there is a wine raffle at the end to bring a little joy in these times. I would ask you folks to hang out because you’ve got to be present to win it. Now with that out of the way, let’s get into this in terms of setting context. I think a lot of folks may be familiar with the SANS sliding scale of cybersecurity. Basically, the ideation behind this is that you should consider making your investments from left to right on this. We see a lot of folks follow this, but that is not necessarily always the best thing to do. It is not necessarily what needs to be done today because of advances in the way proactive monitoring, threat hunting, and cyber intel—which are later parts of this scale—have evolved. Chris, do you have any thoughts on this?

Chris Gerritz: Yeah, I think the general gist of this entire webinar is about how organizations view their maturity in this. Every organization is going to be at a different level of maturity, whether they just have antivirus and a firewall at the very left end and they’re figuring out patching, or they have an advanced SOC and they’re trying to augment and figure out what else they’re missing in their environment. Regardless of what that is, the intention here is to take the common knowledge of this kind of material from SANS and from NIST and then think about it in a different way. We’re going to think about it with this paradoxical thing we call the prevention paradox that really changes how we think about this particular scale.

Tyler Smith: No, I mean, that pretty much nailed it. Like anything else that’s out there, it’s a good guide for people trying to get a handle on security. You don’t have to; it’s not a rule, just a guide that helps people understand where their money is best invested. We’re going to use this, like Chris said, to talk about the problems that can arise from focusing solely on these kinds of things.

Manoj Tandon: With that, a lot of cybersecurity revolves around prevention, detection, and response. Those are three pillars of cybersecurity, and the NIST framework has an extensive focus on all three. Chris, if you want to jump in here and talk a little bit about the framework and how it relates.

Chris Gerritz: I think we talk about five functions of cybersecurity operations when we approach NIST: everything from identifying the assets you need to protect to protecting them and responding when things go wrong. That really boils down to two different efforts that folks make. That first effort is really figuring out what you want to defend and stop. Stop the attackers from ever getting in; prevent them from ever getting in. Put your walls up, put your technical controls in place, and protect those assets. That’s number one. Number two is, in the case of failure, detect and respond to any incident that occurs. If someone does get in, if data is lost—whether that’s maliciously or some insider thing—we need to be able to detect that failure and then respond to it and recover quickly. This really boils down to: protect the assets, and if you can’t protect them, detect the failure and respond to it.

Manoj Tandon: Do you think, Chris, there is maybe a little bit of an over-emphasis on the prevention side? At least anecdotally, I personally see that, but what’s your experience with it? I’d like to get Tyler’s take on that too.

Chris Gerritz: From my perspective, obviously, we focus on hunting and incident response, and those are failure cases through and through. Our job is to detect the failure cases and respond to them quickly. That’s what we do today. We find that most organizations are completely ill-prepared to do that, so they don’t have investments in the ability to respond. They generally have a retainer with someone to go in and do an incident response engagement, which ends up being very costly and a lot of times late. We find that most investments are in technical controls: build the wall, build the moat, and walk away.

Tyler Smith: I agree with what Chris said. It’s easier to build that stuff because it is a known. You know you need a firewall, you need AV or EDR tools on your endpoints, and you need a SIEM. These are all known factors. The problem is they only get you so far, and as soon as someone introduces something that walks past all of those, that’s where you run into issues. Especially if you’re like the drunk guy looking for his keys under the streetlight because that’s where the light’s at. You’re never going to find them there. They’re already gone through your defenses and doing whatever they’re doing. But that’s the whole point of this webinar.

Manoj Tandon: Yeah, well, it kind of brings up this pyramid of pain, which some folks may be familiar with. You have at the base level things like hash values and IP addresses—known knowns that we all know how to automate, address, and deal with in a very repetitious fashion. As you move up this continuum, it gets exceedingly difficult to automate, and at the very top, you very much need that human intervention. It leaves the domain of the vendor and really becomes a thing of human intelligence to deal with some of these TTPs. Chris, you made a comment on the last slide about it being a very reactive process that a lot of companies face where there is an incident and then they’re reacting to it. I think part of this is to shift and turn that around and look at a very proactive response: proactive hunting, proactive detection, and really not revealing our own TTPs to those bad guys. Any thoughts on that you might want to share?

Chris Gerritz: Yeah, I think for the pyramid of pain, it really has two perspectives. One is how hard it is to do these things for a defender. The sophistication of the defender is going to go up when they go up the pyramid of pain. It’s going to be harder and require higher skill sets. A lot of people may have to make choices about the investments they make there. Can they afford a full-time hunter? Can they afford a full-time monitor that has the skill set to do the top things? Oftentimes, no. I like Dark Rhino’s approach when it comes to this kind of stuff because not everyone is looking for a SOC full of people who can do the entire tip of this thing all the time, 24/7. That’s what some people offer and they charge for that. In other cases, not everyone needs that 24/7. Some people need a risk-based approach, which you guys have, on how to apply the harder things and how often, and whether that is effective. What we found, and what Dark Rhino is doing, is effective at a lower cost.

Manoj Tandon: Chris, we’ll talk a little bit more about that towards the end of this presentation: how what we are presenting here with the prevention paradox and addressing it really can be done by a company of any size, even with limited resources and budgets, so that they are far more secure than what traditional techniques might offer them. So with that, let’s dive into this thing that is the prevention paradox. Give us a little history on this. This was actually something that came out of the U.S. Air Force, right?

Chris Gerritz: It is. It’s not well-documented outside the Air Force. I haven’t seen very much. A lot of people nod their heads whenever I talk about it, and they didn’t realize it had a name. It’s never been presented at anything, but I first heard this in 2011 from a friend of mine who’s still in and doing crazy stuff. They haven’t brought it outside the military because they’ve stayed in the military. I started talking about it and blogged about it a few times, and people really found it super interesting. That’s why we’re here. It’s the idea that as we make our investments in prevention more and more, we do get more secure because we’ve increased our prevention. But if anything defeats our prevention, we can’t see that. The prevention paradox is this interesting case from 2011 when we were looking at our investments because we had a layer of prevention, a layer of detection, and a layer of response and hunting for those unknowns. You can prevent what you can see and what you are confident in. When you’re less confident, you can monitor passively, and then when you don’t know what’s in there, you’ve got to proactively go look for it. What ends up happening is, if you increase your prevention to where you’re saying, “Hey, if I can detect it, why don’t we just automatically stop it?” there’s just this idea that there’s this problem that occurs. We found we took all of our detection signatures that we could make confident and we turned them into an auto-block mode, and we lost visibility. There was this paradox where we did increase our prevention capability, but we actually lost visibility on everything that was going to get through in that case. This was a study done to show that if you do try to prevent every attack that attacks you, yeah, you’re going to have a lot of stuff stopped. But when it does fail, you will have no idea unless you are proactively hunting or someone’s looking at your network. So, we basically reduced our monitoring and reduced our effort in all those other cases because we felt like we were more automated and were stopping everything automatically. What else do we need to do? We ended up being less secure, and that was the prevention paradox. We became more blind. The reason we found is because the attacker is knocking on our door, knocking on our defenses, and getting feedback from that. They’re going to keep trying stuff and go up that ladder, that pyramid of pain, until they find something that’s more sophisticated than our prevention and they’re going to get in. When they get in, they get to stay in there.

Manoj Tandon: Is that, Chris or Tyler, a reason why we see dwell times in environments being so large—200-plus days on average, depending on what industry magazine you’re reading? It’s generally 200-plus days.

Tyler Smith: Yeah, I think that contributes to it heavily. A lot of the other part of this is the human element—the way people get used to the security controls they have. They get to trust them, and it’s like a security blanket almost. It’s not really protecting them from anything beyond a certain point. There is a lack of protection. If you’re confident that your prevention methods are stopping everything, then why would you be looking for compromise? The more you prevent, there’s a chance that you become less safe. If you don’t know where the edges of that prevention begin and you’re not looking beyond those, then yeah.

Manoj Tandon: Let’s talk about the role of detection and response. What do you see?

Chris Gerritz: We see that there should be a distribution of resources. Most organizations can’t afford a full-time SOC, and that’s why they go to folks like Dark Rhino, because you guys can provide a cost-effective way of doing this. All secure organizations are going to devote time to prevention and to stopping the attacks from recurring in the first place. That’s important. They’re also going to layer on monitoring of their network to make sure they’re seeing the failure cases. When our prevention controls don’t stop the attack, can we see it and stop it quickly? That’s detection and response, and that’s really where the managed detection and response market came from. It’s because of this need to layer on that detection and hunting to find the things that get through the defenses and the things that might still be working in there—those unknowns. If you have those layers, you’re able to, as the attacker iterates against you, stop them. A lot of them are going to give up because your defenses are great and your prevention stopped them. Some of them won’t. They’re just going to keep trying different things until they get in. Hopefully, when they get in, you have a layer of monitoring and are able to respond so you can stop them even if they are able to get through.

Manoj Tandon: I think one thing I want to add on that is, for that persistent attacker that is trying to get in, all organizations inherently must really truly look at risk to their critical assets. When I say assets, not just the computers, but the data. What is that critical data that someone really wants that can be monetized? By and large, most of these attacks, unless they’re led by a nation-state, are monetarily driven. It’s very critical that organizations look at that risk and understand it so that when this is being deployed—even when we’re looking at monitoring or proactive threat hunting—we are really focused on which haystacks you’re looking at, right?

Manoj Tandon: Right. Let’s talk a little bit about this prevention versus detection and response. The common wisdom is always: is an ounce of prevention worth a pound of cure? I think everyone is taught that, and I feel it is definitely true in most cases. You do need some kind of prevention layer. But what we found is that many in the cybersecurity industry are investing in prevention before they invest in anything else. To use an analogy that I’ve told before: I’ve heard folks say, “I can’t afford hunting, I can’t afford MDR, I can’t afford to monitor my network. What is the point if I don’t even have a fully functioning defense in place to stop the attacks in the first place? It’s just going to be playing whack-a-mole.” That’s the common wisdom I keep hearing. But if you put that analogy in the physical space, it doesn’t work. “I can’t hire a security guard until my fence is completely surrounding my organization.” No. The reason you have a security guard is because your fence isn’t complete; something is going to get through. You don’t wait to hire that security guard because your fence isn’t perfect or isn’t tall enough. Most organizations don’t have a perfect defense, and they still need to monitor their network. In fact, it’s probably more important to monitor and have hunting services given to them because they can’t stop everything. This is a little backwards from where the SANS scale might guide you initially, but technology has changed and services have changed, so this is now a tractable scenario for most organizations. We’d urge them to take a look at it.

Manoj Tandon: Let’s introduce a potential solution here. When we look at the Lockheed Martin Cyber Kill Chain, which I’m sure everyone here is very familiar with, we’re talking about addressing everything post-weaponization. What we have put together as a potential solution is what we at Dark Rhino term “iPionR.” We intentionally chose not to use the words MDR because MDR is a very reactive thing. Response means something has happened. What we’re saying is that we’re going into a very proactive mode, and we’re going to do it on an analytics basis so we can evaluate several hypotheses rapidly and not go through them one by one. For us, iPionR is incident prevention, incident identification, and finally, response. The Infocyte tools and technology that you co-founded are critical to this as an enabling factor. Is there anything that you would like the audience to know on Infocyte capabilities here, or should we go on?

Chris Gerritz: It’s interesting—the SolarWinds breach that happened, or the vulnerability with that back door everyone’s dealing with right now. Some people ask, “Why didn’t we detect this if it’s been in place for six months and was delivered by SolarWinds?” This is why you need eyeballs on, why you need someone like Dark Rhino with the skill set and the tooling to be able to go do proactive looks and monitor for activity that wouldn’t normally flag on your prevention stuff. On one hand, every antivirus in the world labels some SolarWinds tools as malware. They have a tool called Dameware that they distribute; it flags on everyone’s anti-malware. If you call them and verify this is indeed Dameware from SolarWinds, they will tell you to whitelist that. Even though it’s detected as malware, we whitelist it because we need to use that tool and it looks like malware. Dameware is a remote access tool, and there’s no way to not detect it as malware—it’s doing what malware does. But people want to use it and it’s authorized, so we authorize it. If you get this tool and it’s flagging, a lot of people are going to look at that and say, “Is this from a vendor? If it is, whitelist it,” because it’s a false positive. That wasn’t the case here, but that’s why no one really detected it until someone really started looking at the behavior of these apps. Number two: now that we do know about the vulnerability, there are 30,000 organizations that have this software. How many of them actually got compromised and had stuff stolen? How fast can an organization search for that and see which systems are affected? It turns out, not very well. We’ve had a lot of folks talk to us about how we published an extension for our product because we can detect that Orion is installed natively and we can detect the use of things like Cobalt Strike natively in memory and monitor that. Dark Rhino’s team is going to know about those things. But in the case of these specific things that could have been whitelisted because it’s a vendor product, you’ve got to go look for that manually. We ended up publishing an extension to go look for that activity in particular, on top of the normal hunting we do, which is very broad. When you’re doing hunting that’s very specific, that’s where you really need the expertise of a team like this. I believe your iPionR service offers that to your customers.

Manoj Tandon: Chris, I would like you to briefly touch base on something you and I had a sidebar about: analytics-based versus hypothesis-based hunting. When you go to Black Hat and you hear these talks on hunting, a lot of times it is people who have a large SOC and a large set of data. They’re approaching threat hunting from a perspective of: “I have a hypothesis based on the attackers we’re seeing or that are targeting us. I’m going to go look for that particular activity because we might have it.” We’re being targeted by an organization that uses one very specific tool or happens to attack SolarWinds, so I’m going to use that hypothesis to look for that very specific tool. That hypothesis-based hunting is a very advanced technique. It is the one most talked about, and it requires a significant amount of maturity to even do. Infocyte does not play in the top 100 Fortune companies that have the resources to do that. Our approach is very different from hypothesis-based, which requires existing maturity. Ours is a very broad approach that says, “I don’t trust anything on your network trying to detect this stuff. I’m going to go look for it.” We’re going to do a very broad stroke and uncover every rock. Then, when we want to go on specifics like this SolarWinds thing, we can add an extension and go look for that particular activity once we’ve covered our other bases. The approach that we take is that broad look: what are all the MITRE techniques, all the different techniques that could possibly be there? Look at every process that’s activated and verify that it’s not known malware as a first brush. Then look for those TTPs as a second brush, and then look for specific activity based off a hypothesis like, “I might have SolarWinds breaches.” That’s the approach we take and the one we recommend and train for our partners like Dark Rhino.

Manoj Tandon: Let’s use a healthcare example because healthcare has been under significant attack lately, especially with ransomware. We’ve worked with organizations where they believed they had excellent prevention controls in place. They had next-generation firewalls, endpoint detection and response, SIEMs, and all the things you’d expect. But when we actually went in and started proactively hunting, we discovered persistence mechanisms, unauthorized PowerShell activity, suspicious scheduled tasks, and credential misuse that had been sitting there for quite a long time. None of those things had triggered the preventative controls because the attacker wasn’t necessarily dropping a known malware file. They were “living off the land.”

Tyler Smith: Yeah, and that’s the thing. Once somebody understands your environment well enough, they stop acting like malware and start acting like users. They use built-in tools. They use legitimate credentials. They use PowerShell, WMI, remote administration tools, and scheduled tasks. They don’t necessarily have to bring in malware anymore. So if all you’re doing is waiting for your antivirus or EDR to tell you there’s malware, you’re going to miss a lot. That’s where hunting and proactive monitoring become critical because now you’re looking at behavior instead of just signatures.

Chris Gerritz: Exactly. The industry talks a lot about “fileless malware” or “living off the land,” but really what we’re talking about is attackers using the exact same tools administrators use every day. PowerShell is not malicious by itself. WMI is not malicious by itself. PsExec is not malicious by itself. The question becomes: who is using it, how are they using it, and why? That’s where behavioral analysis and hunting become incredibly important because prevention technologies struggle with context.

Manoj Tandon: One of the things we found interesting is that when companies invest in hunting and monitoring, they actually start improving their prevention posture as a side effect. Because now they’re learning where their weaknesses truly are instead of guessing. They’re identifying systems that shouldn’t exist anymore, stale accounts, forgotten assets, poor segmentation, weak access controls—all these things that don’t necessarily show up in a vulnerability scan but absolutely matter from a security standpoint.

Tyler Smith: Right. Hunting gives you visibility, and visibility changes everything. If you don’t know what’s happening in your environment, you’re basically operating blind. Once you start looking, you realize there are all kinds of things going on that nobody was paying attention to. Some of them are benign, some of them are dangerous, but the point is you now have awareness.

Chris Gerritz: We’ve seen organizations where they thought they had 2,000 endpoints and after running hunts they realized they had 3,000. Or they had systems that hadn’t been patched in years because they were forgotten. Or they had credentials embedded in scripts. Hunting often uncovers operational issues far beyond malware. It uncovers hygiene problems.

Manoj Tandon: That goes back to something Tyler said earlier about the drunk guy looking for his keys under the streetlight because that’s where the light is. Traditional prevention is often focused only on the areas that are easy to see and automate. Hunting intentionally goes into the dark corners of the environment where people aren’t normally looking.

Tyler Smith: And attackers know that. They know organizations focus heavily on perimeter defenses and prevention controls because that’s where the budgets traditionally went. So attackers adapted. They learned how to blend in, remain persistent, move laterally, and avoid triggering alarms.

Manoj Tandon: Chris, I think this is a good point to talk about dwell time again because one of the statistics that always shocks executives is how long attackers can remain in an environment before discovery.

Chris Gerritz: Absolutely. In many cases, attackers are discovered not because the organization found them, but because law enforcement or a third party notified them. Think about that for a second. Somebody else had to tell you your environment was compromised. That means your prevention controls didn’t stop it, your monitoring didn’t detect it, and your response capabilities didn’t identify it. That’s why we emphasize proactive hunting so heavily. You cannot assume prevention alone is enough.

Manoj Tandon: And that’s really the core of the prevention paradox. The more confidence you place exclusively in prevention, the less visibility you may actually have into what’s happening behind those controls.

Tyler Smith: Exactly. Prevention is necessary, but it’s not sufficient. That’s the key takeaway.

Manoj Tandon: So let’s talk a little bit about how organizations of different sizes can approach this because one of the biggest objections we hear is cost. Companies assume hunting and proactive monitoring are only available to Fortune 500 organizations with massive SOCs and huge budgets.

Chris Gerritz: That’s definitely an old way of thinking. Historically, that was true because the tooling was expensive and the expertise required was highly specialized. But that’s changed dramatically over the past several years. The combination of cloud infrastructure, automation, analytics-driven platforms, and service providers like Dark Rhino has made these capabilities much more accessible to smaller organizations.

Tyler Smith: It also comes down to risk prioritization. Not every company needs a 24/7 SOC with hundreds of analysts staring at screens. But almost every company benefits from periodic hunting, proactive validation, and monitoring focused around their critical assets and highest risks.

Manoj Tandon: Exactly. We’re not suggesting every organization needs to operate like a three-letter intelligence agency. What we are saying is that organizations need visibility and proactive validation that their controls are actually working. That can absolutely be scaled to fit the size, budget, and risk profile of the company.

Chris Gerritz: And honestly, from our perspective, some of the smaller and mid-sized organizations are actually in greater need because they’re often targeted precisely because attackers assume they lack those capabilities.

Manoj Tandon: Before we wrap up, I want to ask both of you for one key takeaway you’d want the audience to remember from this discussion. Tyler, let’s start with you.

Tyler Smith: Don’t assume prevention equals security. Prevention is important, but if you aren’t validating and monitoring your environment, you don’t actually know how secure you are.

Chris Gerritz: Mine would be that attackers adapt. If your security strategy never evolves beyond prevention, eventually somebody is going to figure out how to bypass it. You need layered security, visibility, and proactive hunting capabilities.

Manoj Tandon: And for me, it’s this: cybersecurity ultimately comes down to understanding risk. What are you protecting? Why are you protecting it? And how quickly can you identify and respond when something goes wrong? If organizations begin thinking that way instead of simply buying more tools, they’ll be in a far stronger position.

Manoj Tandon: We really appreciate everybody joining us today. Thank you, Chris. Thank you, Tyler. We hope this discussion around the prevention paradox was valuable and maybe challenged some conventional thinking around cybersecurity strategy. And for everyone attending, thank you again for taking time out of your day to join us.