Security Confidential S5 E10 Manoj Tandon and Tyler Smith

Manoj Tandon: Welcome to Security Confidential by Dark Rhino Security, where we get together and talk about all things information security. Let’s talk about incident response then. I mean, that to me has always been an interesting topic, right? When I say “interesting,” it’s been a dark topic because I personally am not a security practitioner, and I never understood it. I always thought it’s a room full of really special people that are trying to track down the bad guys and figure out how to undo things that they’ve done. But it seems like there may be an element of that, but it’s not exactly that. How about it, right?

Tyler Smith: Yeah, there’s more to it than that. A lot of it is preparation, to be honest. You know the old adage: “An ounce of prevention is worth a pound of cure.” When it comes to incident response, I think it has many meanings that are applicable. The big thing is people really underestimate what the experience is going to be like when they go through it.

Manoj Tandon: Where do you think these companies—most companies—envision their biggest problem areas to be if a breach were to occur?

Tyler Smith: You know, I think that changes drastically from industry to industry. But as far as where people see the issues, I think a lot of people are starting to focus on the Internet of Things and “bring your own device” policies that are gaining traction, especially with people working from home more nowadays. But I suppose it runs the gamut. If we surveyed a hundred people, they’d probably come up with all kinds of answers to that question.

Manoj Tandon: And then how does the fact that, depending on what rag you read—so I’ll just paraphrase a statistic because I haven’t seen a firm number or a consistent number from anyone—but it’s generally reported that a threat lies dormant inside your network for 200-plus days. So now, depending on the rag you read, it could be 220 days or 260 days, but I’ll just round it down and say 200-plus days.

Tyler Smith: Mhm.

Manoj Tandon: So that means that an incident is already in the works. It’s usually already in play and, as you were alluding to with preparation, one of the steps that NIST says is containment and eradication, but it seems to be out of sync with the fact that these threats are already in there. So your baseline is already, in many instances, a threat-poisoned landscape, if you will. So if you restore an old system, you’re really restoring the threat that came with it. Or am I not thinking about this correctly?

Tyler Smith: No, I think you’ve got a very good grasp of it. And a lot of people espouse the idea of the post-exploit environment instead of looking at the environment as though it were pristine and clean and trustworthy. There are many who’ve espoused the idea that the environment is already compromised. You’ve just not detected it yet.

And, you know, I think that is not necessarily a bad way to look at it, but I think it can also breed some pretty unhealthy attitudes within an organization, especially from a security standpoint, because I’ve worked in organizations where security didn’t really enable the business very well, and that is a problem. If you don’t have business, you generally don’t have a reason for security, right?

And a lot of practitioners use that: “Well, we’re probably already breached, so we can’t take chances here or there.” However, they are predisposed to look at it as individual practitioners, but I think it has merit. You know, assuming that you’re breached has some merit. There’s a military connotation where, when you’re in training, they teach you to always assume that you’re under observation and behave like you’re under observation by the enemy.

And so that means, you know, don’t light your cigarettes in the open. Don’t do stupid things because the enemy is watching. And I think that may be a more healthy approach to that “assuming you’re compromised” way of looking at things, where organizations can look at the landscape and say, “Okay, we can’t take things for granted.” Like deploying an edge device that is not configured with a strong password or hasn’t been hardened for deployment is an example of that, where we don’t look at the world through rose-colored glasses. We assume bad things are going to happen.

Manoj Tandon: Right. And if you look at the research, it does support that. Researchers have put devices on the internet, accessible to the internet, and I’ve read of some where in less than 15 minutes, the device has been compromised and taken over by an attacker.

So, yeah, I guess, you know, for that to happen, there must be something of value, unless attackers are just going out there for the heck of it. But it seems like more often than not, the cause is money, right? They want cash and they want to monetize whatever it is that they’re stealing in some way, shape, or form.

Tyler Smith: Correct. They want to eke some Bitcoin out of it somehow.

Manoj Tandon: That’s right. They want to get that out of it. So with that being the case, I would think that in the preparation phase, which you were earlier on in this podcast saying is one of the most important phases, it might be a great idea to take stock of what the real assets of the company are. And when I say “assets,” I’m not saying tables and chairs and computers, but truly the intellectual property that makes the company the company, right? And what a loss of that would mean, because that would probably really outline risks and it might outline strategies of how to best protect that intellectual property.

Tyler Smith: Right. Exactly. And that’s the thing. It is a different conversation when you remind people that assets are not limited to things you can put a little sticker with a barcode on.

Once you have that conversation—and it doesn’t matter what company it is—you have something of value. Maybe it’s your company credit card number. Maybe it’s your checking account number for the company, or client lists, or even maybe it’s just employee data. Maybe you’re like, “Hey, you know, I just have an ice cream shop. I scoop ice cream. What are they coming after here?” Well, if you employ people, you’ve got Social Security numbers, right?

So in that regard—and then the other thing is—everybody thinks, “Oh, well, we’re good people.” Or it’s some semblance of this. Everybody that hasn’t really thought about this thoroughly thinks in this way, and they think either, “Oh, we’re good people, we do good things, why would someone attack us?” or, “Oh, we don’t have that much money, why would anybody care about us?” or, “We don’t really do anything important, why are they going to come after us?” And to a point, who am I to argue on the overall value of what it is your company deals in?

But think about maybe your clients. For example, maybe you just own a small shop, but maybe you service a very small portion of a much larger company. There was an HVAC company that was the avenue of breach for a much larger company, right? And I can very easily picture in my mind’s eye the guy that owned that company at some point in time going, “Why do I need to worry about this? We just do HVAC. We don’t do anything special.” But your client does something special.

Manoj Tandon: Right. Exactly. And that’s why it’s important to really not look at this as just our problem, and only going to be our problem, because if you’re breached, you could become a problem for your clients, which has consequences most people don’t even want to consider, right?

And it’s not that any of this stuff—I mean, like we talked about, if you go out and you look for an incident response plan, or documents about it, or blog posts about it—almost all of them have a six-step approach. A lot of that, I think—and I don’t know the primary source of that—but I’m guessing most of them can be traced back to the SANS Institute. And in their Incident Handlers Handbook—

Tyler Smith: Yep.

Manoj Tandon: They broke it down into six steps. Which, you know what I mean? Six steps, 12 steps—whatever. The idea is to make it something you can go after. You can set goals and check boxes, but really, preparation is really, really important. And as you mentioned, it does go back to identifying what it is you initially have to protect and really wrapping a written security policy about how you’re going to achieve that goal of protecting those systems or that data.

So I think one key point here is for all the listeners to understand that there’s no such thing as absolute security, right?

Tyler Smith: Correct.

Manoj Tandon: Because someone’s going to be asking why not prevent an intrusion in the first place? Well, the problem is that all of these systems are designed by human beings, and anything designed by a human being is going to have some flaws to it. It’s just the nature of it, and there will always be someone who will find those flaws. If nothing else, you have human beings who are intimately a part of the security process. And those are not people who are part of your security organization, but they are your everyday employees who may click on an Excel file, a PDF link, or something to that effect that instantiates a compromise. Right.

Tyler Smith: Right. Exactly. There’s no such thing as, “Well, why don’t we just spend $60 billion and deploy every hardened system?” Fundamentally, if somebody internally clicks on something and lets the bad guys in the door, then all those controls were for naught, right?

Manoj Tandon: Exactly. And it happens. So I think all the listening audience should take that to heart and understand that there’s no such thing as absolute security. And I guess, from the way you’re describing it, we’re basically playing zone defense, if you were playing football, right? You want to minimize—let the wide receiver complete the catch, but minimize the yardage loss, right?

Tyler Smith: Right.

Manoj Tandon: Contain it.

Tyler Smith: Exactly. And that is very much pretty much what we would be playing on an incident response team: knowing that we have hardened the systems enough, we’ve cataloged our assets well enough, and we know what’s at stake. The trick is: can we find it fast enough and then act on it equally fast so that there is damage, but it’s very minimal at best?

And that’s the second widely held step: identification—which is answering the question ahead of time. Again, preparing and answering the question ahead of time: what is going to trigger a formal incident to be declared? And it’s going to depend on the organization. But usually, it’s going to be a group that’s going to be triggering that for you. So maybe it’s your security analysts, maybe it’s the guy that monitors the antivirus for the environment, or maybe it’s a group of people. But somehow defining, “Hey, this is what triggers our plan of action that we’ve prepared,” should be set up so that, when that thing happens, things start falling into place. People are doing what’s expected of them in order to bring the situation back under control.

Manoj Tandon: So what would you say are the critical practical things that need to be done from a process and technology perspective to have rapid incident response?

Tyler Smith: Well, that’s a question that probably has more answers than we have time for, but sure. I mean, big things are, again, ahead of time: identifying where your risks are, making sure that you’re addressing those risks appropriately with the appropriate people, process, and technology.

And then identifying normality to the degree possible. And then being able to identify things that are attacks, and in a way that allows you to differentiate between just your regular daily alerts about China trying to brute force their way into everything they can touch and differentiating between that and something that’s truly abnormal. That’s going to be the critical part.

But as far as the technology side goes, it’s making sure you’ve got the basic blocking and tackling: your network is protected appropriately. You have a network that’s segmented, meaning that people in the company are placed into different subnets, if you will, or portions, zones on the network, and that there are controls in place keeping them from talking to other parts of the network that they don’t need to speak with. And that’s going to help contain the spread—the lateral movement and spread—of anyone that attacks you. So once they get in, they may get a hold of someone’s laptop in the service department, but they’re not going to be able to make it easily to someone in accounting or a server. So that’s one part: the basics of security. Make sure you have those covered. Make sure you have a good system set up to discover vulnerabilities and patch systems in a timely fashion.

Manoj Tandon: But incident response—Tyler, if you look at the industry—it is the most expensive exercise that’s out there.

Tyler Smith: Yes.

Manoj Tandon: Right. Yes. And it costs a lot of money. A lot of companies have incident response firms on retainer, whether they use them or not, right? They’re paying that retainer fee. Others use some combination of technologies and a managed service. But is there, in your mind, being an industry practitioner, some things like IT hygiene—I get it, that’s very important: keep your systems patched, keep your endpoints protected—those things are fundamental. But when you’re seeking a best incident response solution—whether that be internally developed or outsourced externally to another company—do you have any thoughts, recommendations, or guidance as to how a company can actually mechanically pursue this and come to what might be the best for them?

Tyler Smith: Yeah. I think if you look at the elements of any incident, you can, in my opinion, boil them down to trying to answer a couple of questions. And the first question that seems to always come up—imagine you’re the analyst. You’re sitting at a computer. You’ve just realized that something very weird is happening, and you’ve confirmed it. Even though you’ve confirmed it, you’ve looked and you’ve said, “Yes, this machine reached out. It did these things, and then these other things happened.”

The very first question usually in an incident response situation is: is this real? Is this really a bad thing? Is it real? Is it really happening? And if we’ve established that yes, it is really happening, the next question is: how bad is it? Is it super-super evil stuff, or is it trivial, like commodity malware that’s trying to use my system to pretend to click on a bunch of advertisements? Or is it literally multiplying laterally across the network and taking machines over and exfiltrating data? Which is the next question: how bad is this?

So if you think about an incident response wanting to answer, at a very basic level, those questions—“Is this real? Is it really a bad thing that’s happening, and how bad is it?”—and you kind of look at what’s going to get me from not knowing the answer to those questions to knowing the answer to those questions the fastest.

And I think one other that I would throw in there, that I forgot to add, was: what do we do to stop this, and can we stop this? Or how do we stop this, and how do we stop it immediately and in the long term from happening again? So if you can answer those questions, and then you can answer them very, very quickly, you’re going to be in a better position when an incident happens inside your environment.

And I would love to say that it’s never going to happen—wouldn’t it be great?—but the chances that you will experience an incident are pretty solid. It’s not a question of “if,” it should be a question of “when.”

Manoj Tandon: I would think that, especially if you’re a smaller firm, it might be better to find folks that can manage this process for you at a reasonable level, because most cyber insurance policies are going to cover the forensics, the analysis, and the restoration of these systems. But it’s a little late to take on that exercise when the incident has already happened. You should do that—and I guess that’s part of preparation—you should really look for those kinds of solution providers upfront, right? And form a good relationship with them and have them on call.

Tyler Smith: Mhm. Yep. Exactly. Because you don’t want to be doing comparison shopping while your place is burning down. You want to already know who you’re going to call, when they’re going to be there, and what they’re going to do when they get there. And these are all things that you should really know ahead of time.

And in the few incident responses I’ve been involved in, the people that were in leadership positions didn’t necessarily have the answers to those questions. As far as the who, what, when, where, why, and how—they at least had somebody on retainer, but they did not know any of those. There wasn’t like a binder they could open up and go, “Okay, we call this number and then we talk to these people, and based on these criteria, they’ll either be here in 4 hours, a day, or 3 days to get things going.”

One of the ones that I did, the company had another company based several hours away and experienced the beginning of an incident. Picked up the phone, called those people, and went through the initial question: “Is this real?” Yes. “Yes, it is real. Yes, it is really happening. Is it really, really bad?” Yes, it is really bad. At least it looks really bad. “Well, how bad is it?” “Well, we don’t know. That’s what we need you guys here for.”

And they were like, “All right, well, just image a machine and upload the image, and then we’ll download it and analyze it.” I just remember sitting there with the client and going, “You pay these guys how much money, and they’re not willing to get on a plane and come here in person for this, or even jump on a remote session with us?” And then we couldn’t even answer the question: how are we stopping this right now?

In that instance, it turned out, thankfully and horribly, it was on a Friday, so we unplugged the company’s internet, basically, and just shut the network down until we could get a better grasp on it. And by Monday morning, we were able to turn things back on because we had deployed a solution rapidly from a different company, just because the first company wasn’t working out.

And again, it was because nobody had those answers. There was no binder that said, “Hey, look, this is our agreement. This is how this whole process is going to work.” Nobody had those answers. So when the green light turned red, we were kind of left going, “Well, what are you guys doing? How are you helping us?” “Oh, well, we’re going to get a forensic image of one of the affected machines, and then we’ll analyze it, and then we’ll get back to you.” And we’re just kind of left hanging in the breeze in the meantime.

So really knowing what the response is going to be from that company you partner with, having that relationship established ahead of time—and then the other thing that a lot of people fail to do is they fail to practice this. Why do we have fire drills? Because we know that under stress, the human brain performs poorly. If we practice ahead of time and we get people used to a set of steps, which are actions, we can have a better outcome when that stressful incident occurs. And this is no different. Did I lose you?

Manoj Tandon: Oh, I’ve been talking to myself. There’s a fellow cutting the grass outside, and so I’ve been muting this so it doesn’t record into our session here. This is some of the trials of working from home.

Tyler Smith: Yeah.

Manoj Tandon: Right. So I’ve been muting it, and while you were talking, I was on mute and I was commenting. But yeah, you’re right. Comparison shopping at the 11th hour: bad idea. Get this in place now. If you’re a smaller company, this is a specialized area. Incident response is something that you should really look at farming out. Heck, most large companies also farm out incident response, right? Because you really need an objective outsider that is not biased by your setup inside the corporation or biased by the policies and tribal knowledge of that company—the way the company works—in doing a formal response to an incident. Right.

Tyler Smith: Right. Right. This is one—

Manoj Tandon: It might be the company’s policy—

Tyler Smith: Yeah. You want to get it right.

Manoj Tandon: You know, it might be the company’s policy: “Well, we never shut down our point-of-sales system. There’s no reason for that to ever happen.” You could argue that at a retailer, if that POS is compromised and your incident response firm picks that up, you should let them shut that down. Will it cause a temporary inconvenience? Absolutely. Will it cause upset customers? Yes. But it will cost you a lot more in reputational, legal, and financial losses if they get away with all their credit card data and all the specific customer identity specifics that can allow them to replicate the identity, right?

So you need—I guess the reason I say that again: you need an outside party that is not bound by the handcuffs of the organization in responding to an incident. That is absolutely critical.

Tyler Smith: Yeah. And really, I think one of—if we were looking at the metallurgy of those handcuffs, a big part of that would be self-interest. In these instances, the organization is interested in continuing to do things that generate revenue, which is not a bad thing. I think every company should do that or they won’t exist very long.

The problem is, in an incident response, one of the most damning things you can have happen is somebody who’s a decision maker going, “No, everything’s fine. We have the bestest incident response ever.” I don’t know. No. Like, “We found it. This isn’t a big deal. We found all the bad things. It’s fine. Everything’s fine.” Because again, that self-interest: they want to get back to normality. And that decision may be a bit on the reckless side if it’s left purely in the hands of people that are praying for normality when there is none.

It’s best that you go through the process. I mean, an incident response doesn’t have to be organizationally crippling if you prepare for it correctly and take appropriate steps ahead of time. You’ve tested your approach in tabletop exercises.

Manoj Tandon: That way, everybody—

Tyler Smith: Red team and blue team.

Manoj Tandon: Right, exactly. Role playing and tabletop exercises are great ways to make sure that people understand their role and they can move the ball forward in an expedited manner when a bad thing happens, so you aren’t sitting there going, “Okay, now what?” Because every one of those moments where you’re stalled out like that, you don’t know what to do next.

Let’s assume that the attack was bad enough you had to shut your company’s network connection down or your website down. Every time you stall or you hit a wall and you don’t know what to do next—because you haven’t planned for it and you haven’t tested that plan to the point that people know it—every time that happens, that’s more revenue lost. That’s an increased chance that the attacker gains a stronger foothold in the environment, or more data is lost or destroyed.

So the more well-planned out your response is—and again, it doesn’t have to be an overly wordy thing. It’s just: identify what tasks need to be taken, assign those to different roles within the organization. I tell people to do roles and not people. The reason being is it can really throw off a plan when it’s tied to individuals in the organization and then there’s a reshuffling or something like that happens within an organization. And now your plan that was great—you’re all sitting there and you’re doing your next tabletop exercise, hopefully—and you uncover, “Okay, and now Bob is going to call the board and let them know what’s going—Wait, where is Bob?” And somebody’s like, “Oh, Bob is not with the company anymore.”

Manoj Tandon: Right. But whoever’s responsible for that role must do it.

Tyler Smith: Exactly. And just having alternates is a big thing, too. If the person that’s in charge of communication isn’t here, then the person who’s in charge of recording the timeline and history of the incident will take over. Something like that. But yeah, having a plan ahead of time, then testing that plan, and role-playing that plan regularly is the best bet for best outcomes of incident response.

Manoj Tandon: I think, based on that, we have come to our time limit here, Tyler.

Tyler Smith: Yeah. And I guess we are planning on talking about this a little bit more in-depth.

Manoj Tandon: So we’ll probably—we’ll have a series on this. This really was more about the importance of incident response and about preparation, but I think this is a very deep topic, and we’ll do a series on this and explore it quite a bit in-depth.

Tyler Smith: Yeah. Absolutely. Look forward to it.

Manoj Tandon: Sounds great, Tyler. Take care.

Tyler Smith: You too. Thanks, man. Bye.

Manoj Tandon: Bye. Thanks, everyone. Please subscribe and share this, and we will talk to you all next time. Thank you.