Security Confidential S11 E3 George Kamide

Manoj Tandon: Language models are getting very, very advanced.

George Kamide: Yes.

Manoj Tandon: And they’re going to continue to get more and more advanced. You mentioned something about using them for processing. I guess before we get there, how do they change the vulnerability landscape?

George Kamide: This is a fantastic question. I think it pulls together a lot of things that we’ve talked about.

The anthropologist in me is going to say that language is the underpinning of culture, democracy, and society. It is not overstating that. We are primates who transmit culture and knowledge to the next generation through language and written text. That is rather unique to us—that we can sustain that level of transfer over generations.

So when you get to a place where you can synthesize language in what is largely plausible—because an LLM is designed to predict with very good accuracy the next possible word in a permutation of words—you enter this age of synthetic reality.

We have voice cloning software, we have text generation software, and very soon we will have more realistic video generation software. In the words of colleague and friend David Mahdi, the risks become the volume, the velocity, and the variety of attacks.

A Wall Street Journal reporter just cloned her voice and got past her own bank’s voice verification. Basically, we have all content verification out the window. We have a compounding of existing vulnerabilities.

I don’t know if anyone remembers, but last year we had what were called thread hijacking or conversation hijacking email attacks, where someone gets in and sits inside of a conversation, watches the back and forth, and then jumps in as a participant in that conversation. It’s not like a single inbound phishing email; they look like a legitimate person within that conversation.

Look how easy that is, if you are a foreign language adversary, to mimic native English usage. You could jump in right away. You could also process the information in that conversation for intelligence gathering at massive scale.

So, I think it really changes the landscape, but I’m also hopeful that it changes for defenders too. When you look at what Microsoft is doing with Copilot or what Google is trying to do with the Cyber Security Workbench, ideally, the toolset is also moving in the direction of natural language commands.

What if we could reduce the time to build queries to run scripts? You could simply ask your environment, “Can you reverse engineer this file?” That could save us a lot of time on the defender end.

So, I think it brings great risks, but I’m hoping it brings a lot of utility as well.

Manoj Tandon: Well, yeah. Just like you described with the bank—she got past the controls. I could think of a whole bunch of controls you could get past if you’re really good with natural language. Everything that someone has set up, you could just walk right past it pretty much.

George Kamide: Yeah. Very recently, there was this terrible story in the state of Washington about a cybercriminal calling parents and being able to mimic their children’s voice in a panicked state using a voice clone.

Manoj Tandon: I did not hear about that.

George Kamide: It’s a voice cloner trained off of social media videos, which is the ultimate OSINT resource. You and I have no hope; we have recorded so much so far.

Manoj Tandon: Well, my identity has been stolen. People have created accounts, and I’m sure they’ve used my voice and everything else because it’s all out there.

George Kamide: Yeah. To date, it’s been maybe spoofed email addresses or posing as you, and people have that fishy feeling like, “Oh, why did he misspell that?” or “Why is he asking for Amazon gift cards?”

But now they could call up Emily, our producer, and say in your voice that you need something. The human mind is evolutionary hardwired for collaboration and trust. That is the fabric at which we are talking about vulnerability—not just systems, but actual human trust.

That is where things get very tricky very quickly. The processes inside of defenders’ systems—not just the technological processes, but how they talk to one another and how they verify—are going to have to change pretty radically.

Manoj Tandon: I am so glad you said that. On this show, we’ve often proposed that cybersecurity is not necessarily a technology problem; it’s a business problem and a people problem.

If we build the awareness properly and people get involved—since people are the biggest cybersecurity asset—then we could probably make a big dent in the number of successful compromises.

George Kamide: Absolutely.

Manoj Tandon: It’s interesting to think of where this is all going. I guess one thing would be: how does a machine generate context? That is such a human thing. I know you, George; you and I spoke, and I now have context.

If I was going to reach out to you on LinkedIn, I have a way of reaching out to you. A machine may not have that same context. I don’t think they’ve gotten there yet. I don’t know what your experience is.

George Kamide: Wow, yeah, that’s a good point. On the one hand, we have “context-less” attacks, if you want to call it that. North Korea has been like the “OG fishers” on LinkedIn. They find somebody who’s been in a job for three years and offer them the next step up.

They’ve been running that fake recruiting scam for a long time. The context is your professional life, your self-worth, and your financial compensation.

But in terms of these new AI systems, AI is a process and a means, not an end. It’s a way to get to something.

The thing that scares me more than a ChatGPT—which is a tool that waits for input and can be used for good or ill—is the Auto-GPTs. These are agents that can run tasks and create their own task lists.

They can do that in an automated fashion. You could set up a series of Auto-GPT agents to run OSINT reconnaissance on a target, or scrape everything they’ve said or written so you can mimic it.

You could essentially send out an army of robots to bring that context to the human operator, or they could conceivably launch it in an automated fashion.

Manoj Tandon: Now, on the defender side, do you think threat intel is catching up and identifying the sources of these bots?

George Kamide: That is a good question. The way bots have been created in the past regarding influence operations on social media is somewhat different than the way these agents are being created.

I don’t know if those technologies are watermarking or if there’s an indicator yet. That remains to be seen in terms of how they can be detected.

Manoj Tandon: Yeah, we would have to. When this whole thing went down with Twitter and Elon Musk regarding the bot issue, they wouldn’t declare it.

One thing might be: why was it so hard to know how many bots are out there? There should be some indication. Even if you just look at the volume of posts—a real human may not post that much.

There might be some metrics that would lead you to believe that this is pretty much a bot of some kind.

George Kamide: Posting at all hours or an inhuman number of times… you physically could not type messages that fast. It seems like there should have been indicators that were fairly easy.

I can’t speak to the backend Twitter architecture or why that may have been difficult. But we have seen troll and bot farms where they have banks of iPhones or Android devices plugged in, running scripts against them.

Now imagine you don’t actually need the hardware; you can just run agents through the internet. That becomes a much more magnified problem.

Manoj Tandon: That was a little understated, but you are absolutely correct.

[Laughter]

Putting your “Future Vision” cap on, what job functions do you think are going to go away or become redundant?

George Kamide: I want to be very careful here because we’ve lived through a few hype cycles. Anyone listening will remember the breathless hype around blockchain—”In the future, we won’t have banks,” or “We’ll know the origin of every component in every product.”

I think there are a few things to consider. One is the assumption that all AI systems will be incredibly accurate, so you won’t need humans. We’ve seen repeatedly that’s not the case. With skin cancer detection algorithms that were really accurate until we discovered they were just looking for the ruler in the photo.

The other is that despite space-age leaps in technology—if we were talking in 1990 and I said you would have a black square in your pocket that is a supercomputer attached to all human knowledge—you would have thought everything would be so easy that we’d never work again. But actual economic productivity levels have remained very level.

I’m very cautious about any promise that would just replace technologies wholesale. I will say I’m more comfortable saying that job functions will change dramatically.

Poor med students pay a ton of money to become diagnostic computers. They take in information and provide a diagnosis. If we can take any of that off their shoulders—and not just doctors, think of SOC analysts—if we can take log data analysis off their plate, we can free those humans up to do more important work.

In the case of doctors, maybe it’s actually engaging with patients in care plans. In the case of security professionals, maybe it’s taking that data and understanding how better to manage risk across different departments rather than hours of analysis and running scripts.

That would be the game-changer for us. I think work functions become redundant, not necessarily the roles. We’ve seen promises to replace entire groups of people, but even in car manufacturing, we still have robots in cages working alongside humans. There are things a robot would find difficult or very expensive to do.

Manoj Tandon: You mentioned a great example with medicine. In the field of radiology, this change has already started. That’s a group of physicians that doesn’t interact with patients as much as others; they are in a back room looking at scans.

You could probably train a pretty accurate data model to recognize those.

George Kamide: We should reckon with inertia and culture. Currently, insurance companies get paid when the radiologist looks at it, not when software looks at it.

How much does that have to change before the job function changes? Who says one radiologist now doesn’t look at a hundred thousand scans?

Medicine is very tricky because it calls into question whether the training data is representative of all different phenotypes and diagnoses. That is where trust becomes really big. I think that’s the frontier.

As we rush into employing AI, things like the NIST AI Risk Management Framework will become extremely important. Providers will want to show a stamp of approval. We don’t have baselines yet, but they will want to show they’ve audited for bias.

Companies taking on those technologies are in new territory regarding the transference of risk.

If I take on a new HR technology to replace back-office operations and those algorithms make risky choices, who owns that risk?

Manoj Tandon: Well, we’re already there. Look at driverless vehicles. If they run over a cat, whose fault is it?

I don’t want to think about humans getting hurt here. If a drone shoots at a building and hits the wrong target, whose fault is it?

We’re already there, and these questions will get answered in due time. I hate to say the term, but “acceptable losses” has generally applied throughout history.

George Kamide: When you think about our first contact with machine learning—the algorithms used to drive engagement—there was no conniving person in a back room thinking, “This will divide society.”

Those were unintended consequences of objective-driven machine learning. That learning curve was very steep.

The problem now is that if technologies are generative, it’s different than just processing data. It’s also about “arms and legs”—APIs connecting processes to other things.

It’s one thing to have a drone; it’s another if an autonomous decision-making algorithm can connect to it. Connections between systems are the most risky.

Manoj Tandon: The genie is out of the bottle. It’s just a matter of time.

Hopefully, as you mentioned, the role may not go away because there’s an oversight function for a real human in all these instances.

Take autonomous lawnmowers—that’s a big industry coming up. But it’s fraught with issues.

If that thing hits a rock and shoots it at a kid, there’s a liability. If you are mowing the highway and cause an obstruction where a motorcyclist dies, there are real ramifications.

You would really hope there’s an OverWatch function that is uncompromising, and we don’t just let AI say, “Well, it’s a pattern recognition engine and it recognized this pattern, so it must be right.”

George Kamide: To go back to the idea that jobs change—when we were dealing with de-industrialization, people said we’d teach coal miners to be coders. Now we actually don’t need that many coders because generative AI will do it for us.

But there will be a need for humans in the loop, whether it’s OverWatch functions for an autonomous semi-truck or security functions built around that software.

Jobs change; I don’t know that they all vanish. It’s not like a mass die-off of work.

Manoj Tandon: To me, this is just cyclical. If you look at human history, these changes have happened since time immemorial—the invention of fire, the wheel, or the Boilermakers Union in the early 1900s. How many boilermakers do we have today?

George Kamide: In Victorian England, they had dreams of steam-powered robots replacing workers. There are a couple of science fiction books on that topic—very ripe for steampunk novels.

Manoj Tandon: It will be a gradual change. Maybe it’s accelerated because technology is accelerating, but it’s the same cyclical process which brings us to trust.

Do you think we can really 100% trust something? There’s always “trust but verify.”

George Kamide: I would hope not. We came of age before the internet and social media, but the digital natives growing up are contending with things we took for granted, like data brokerages.

Snapchat rolled out their AI app to make the platform “stickier,” thinking the younger user base would be more engaged, but they got a tidal wave of rejection.

Users said, “I don’t want this in my feed,” or “You didn’t ask me if you could pin this to my chats.”

We took for granted that people would want to talk with a robot. We also see kids buying flip phones because they are reckoning with the level of distraction in their lives.

We are of the generation in power, but we’ve got to keep an eye on the ones coming up behind us because they have a different understanding of how to use technology and what trust entails.

We could stand to learn from them and have them involved as stakeholders.

Manoj Tandon: That’s a very illuminating response. I haven’t had anyone pose it that way—that there is an opportunity to learn from them.

You’re absolutely right because our generation did not grow up in that era. So for the Millennial crowd listening, we’ve got to get Generation Z and the ones after them involved.

George Kamide: Absolutely.

Manoj Tandon: Before we run out of time, I wanted to touch on work from home. What do you think the ramifications are there?

George Kamide: I worked in an office pre-pandemic, and then we all went home. In July 2020, while balancing my laptop on books, I realized we’re never going back.

Work from home is what we make of it. Commercial real estate brokers hate it, but it has allowed accessibility for employees who identify as neurodivergent or have disabilities. Being able to customize their setup has been life-changing.

I miss the “water cooler” moments and I get Zoom fatigue, but I probably wouldn’t have met as many people all over the world. I share ideas with people in England and text a group regularly that I never would have connected with otherwise.

I have the privilege of meeting these people in person at conferences, and that’s been additive to my career and the ideas I engage with.

Manoj Tandon: I imagine some of these changes are permanent. Work from home might take us back to a more human-interactive time because we’ll have more chances to spend time with loved ones without a three-hour commute.

People have gotten a taste of the good life; they don’t want to go back.

George Kamide: I remember in 2021 when big banks said they were going back to the office, and then rival banks said, “We’ll pay you 10% more and you can work from home.”

I don’t know how you win that argument.

Manoj Tandon: Facilities are a huge cost in any business. Google and Microsoft have probably adjusted algorithms as well because this changes the way you market to people and what you display to them.

We’re at the hour. I wanted to give you a chance to plug anything you’d like. The floor is yours.

George Kamide: Regarding the vendor-customer divide, Danny Wolf of Audience First and I are doing an exciting gap analysis.

We issued surveys to the buyer side—from Directors of Infosec to CISOs—asking what they want out of events and what messaging resonates.

We also analyzed all the booth messaging at RSA. We are looking at the contrast between those two samples to show the divergence.

We hope this gives people the data they need to make the case that the approach has to change. If marketers are going in one direction and buyers in another, it doesn’t leave anyone safer.

Manoj Tandon: When do you plan on publishing these results?

George Kamide: We are in talks with a couple of media organizations. We are in the final throes of it and hope to have something soon.

People will be able to download it and see the appendix with the survey questions. We tried to make it as systematic as possible.

Manoj Tandon: Fantastic. Shoot us a note when you’re ready to launch and we’ll let our listeners know.

I think it’s a great project. If you can help shut down junk calls and emails, that’s definitely worth a lunch.

George, it’s been a pleasure. Thank you so much for taking the time.

George Kamide: It’s been a blast. Thank you for having me.

Manoj Tandon: Take care.