Security Confidential S15 E6 Jake Moshenko

Manoj Tandon (00:02.098)
Hello everyone, this is your host, Manoj Tandon. Welcome to another episode of Dark Rhiino Security, Security Confidential. Today we have another fantastic guest joining us. It’s Jake Moshenko. For those of you who don’t know Jake, he has been in the industry for a long time. He’s been in the cloud native ecosystem for over 15 years. He’s an innovator. He has worked at Amazon. He’s worked at Google. He is the founder of Quay. Which was the first private Docker registry, which has been acquired. He’s also been a lead engineer at CoOS, which was acquired by Red Hat. And we’re going to actually talk about some of these things. And now he’s the co -founder and CEO of OZZ, not OZZ, a little semantic play there, but it’s OZZ. And they are commercialized in SpiceDB. So there’s a lot to unpack here. Jake, thank you so much for joining us and spending an afternoon.

Jake Moshenko (01:03.15)
Of course, thank you for having me.

Manoj Tandon (01:05.17)
So I gotta tell you, you’re an entrepreneur and that the first question is the origin story. How did you go from a secure role at Google or Amazon making, I’m sure a significant amount of money and say, know what, I’m gonna do something insane like quit all that and I’m gonna start my own company. What happens to short circuit the mind to do that kind of thing?

Jake Moshenko (01:33.036)
Yeah, it’s sort of an interesting story. Both of my parents are actually business owners or were business owners. My dad was a small business owner. He had a financial consulting practice. And then my mom actually owned an ice cream store. So just growing up, that was kind of very standard for me to own a business. But then they were like, you know, when I was young and they saw how I played with Legos and I was constantly building things, they’re like, maybe you should be an engineer.

And so I was like, all right, I’ll try the whole engineer thing. So I went to school. I went to university of Michigan, got a degree in engineering, and then started down that corporate path that you were talking about earlier. Boeing was my first job, super fun, got my feet wet in the corporate world, technology, things like that. But then I moved over to Amazon and Google, which is really where I cut my teeth in cloud and distributed systems and all these things that are sort of important to what I do today.

While I was at Amazon, I actually created an iPhone app, like at the very beginning of the whole iPhone ecosystem. It was a game and I put the game into the store and hundreds of thousands of people downloaded it. And I was like, this entrepreneurship thing is easy. Why doesn’t everybody just like build a thing and throw it out there and it’ll get adopted, right? Like everyone should just do that. So then kind of, you know, that entrepreneurship came back from my, you know, seeing what my parents did, it came back and

re -entered my life, I’ll say. And then I kind of sort of carried that entrepreneurship chip on my shoulder, that itch, I carried that with me through my other roles. So

Manoj Tandon (03:09.81)
So there was never any doubt in your mind to give up a secure, what could have been a retirement role likely and jump into the.

Jake Moshenko (03:20.494)
Of course, of course there was doubt, right? 100%. But like the flip side, it’s kind of ironic. I don’t know if you’ve ever heard Jeff Bezos talk about his regret minimization framework, where it was like.

Manoj Tandon (03:33.806)
I have not actually. I’ll look it up after this though.

Jake Moshenko (03:37.228)
Yeah, so his whole thing was you don’t want to look back at your life and your career at the end and think about what if, right? Like, what if I had tried that thing that I really wanted to do? You know, what if I had taken that leap? What if I had gone and done that thing? You’re probably more likely to regret the things that you didn’t do than the things that you did do. Is that me?

Manoj Tandon (03:58.406)
No, that’s me. I have to, I have to, I don’t have my phone with me, but it’s ringing on my, I hate Apple. Yeah, it rings at once. It’s like, and I can’t find that device to just say, do not disturb.

Jake Moshenko (04:02.464)
Okay.

Jake Moshenko (04:11.416)
Yeah, your whole life rings all at once,

Manoj Tandon (04:23.196)
So sorry, go ahead, I apologize. Emily, you’re gonna have to edit that. So getting back to the Jeff Bezos regret framework, let’s go back to that.

Jake Moshenko (04:26.178)
Yeah, no problem. You just reminded me.

Jake Moshenko (04:34.988)
Yeah, so Jeff Bezos has this concept of his regret minimization framework, where it’s basically like, you know, when you take stock of your life and your career and all the choices you’ve made and the things that you’ve done, you’re more likely to regret the things that you didn’t do than the things that you attempted and that you did do. So, ironically, you know, he, at the time, owned one of the largest companies in the world. And, you know, he actually encouraged me to leave his organization in an indirect way, right? So, you know,

Manoj Tandon (05:02.159)
Really?

Jake Moshenko (05:03.148)
Yeah, that just resonates with me that, you you should always be trying new things. You should always be trying to build. You should always be pushing yourself out of your comfort zone, things like that. yes, of course, you know, I was like, why am I leaving this very stable job at what was the time, like the hottest company in tech for this, you know, lottery ticket essentially, or this chance at entrepreneurship. But looking back, you know, it was the right move for me for sure.

Manoj Tandon (05:32.63)
that anyone that is looking to make this kind of decision, would there be some advice you’d give to a young person that is just getting out of CMU, maybe has their job at Microsoft and is thinking, you know what, I really should.

Jake Moshenko (05:53.174)
Yeah, there’s a phrase that I like to quote and the exact numbers aren’t important, but it’s like six years later, they were an overnight success, right? The way we view companies, the way we view startups, the way we view these entities is we start, we notice them, we’re first made aware of them when they’re already on their hockey stick growth, right? So when you first start something, it’s a grind, right? It’s you just working.

crazy hours, working crazy hard, and it seems like it’s not going anywhere. And everybody has to go through that phase, right? So if you start a business, you start entrepreneurship, and it’s not immediate hockey stick growth, that’s the norm, right? That’s not the exception. So if that doesn’t sound like it’s for you, then maybe you should avoid it, entrepreneurship. But if that does sound okay to you, or you’re willing to stick it out and invest in that grind, then there can be a light at the end of the tunnel.

Manoj Tandon (06:52.048)
What about failure? The fear of failure or failure? I mean, we’re not, at least I can tell you personally, I have not been successful at everything I’ve done. That’s just, and you know, five companies, some went well, some went not so good,

Manoj Tandon (07:10.704)
that fear of failure. Do you have any thoughts on that or on failure in general?

Jake Moshenko (07:16.034)
Yeah, I mean, fear of failure.

Do we fear failure or do we fear how people perceive our failure? Right? That’s one thing that you need to come to terms with. Right? Like, do I care that I failed or didn’t fail? Or do I care how people see that in the context of a career and career trajectory and things like that? So that’d be one thing that you’d want to ask yourself and think about. On actual failure, we’ve done it a ton. So when we left Google, we didn’t set out to build Quay and sell that to CoreOS, right? We set out to build

a web IDE, we called it DevTable. And I don’t know if we were ahead of our time or behind the times or there was no product there or whatever, but it was just an absolute slog and it never went anywhere. So that one, like that was a failure. That was an abject failure, right? But as part of that, as part of building that, we learned and we realized this other thing that we needed, which was Quay, this private Docker registry. We had all these private Docker images and nowhere to put them.

And so he said, well, hey, if we have this problem, other people have this problem too. Let’s just scratch our own itch and see if we can turn that into a business. And Quay was, you know, it was very successful, right? Like a lot of people need it. You can still go and buy it from Red Hat today. That’s how much this product was needed. And that’s how much the market needed an answer to Docker incorporated controlling this whole ecosystem. you know, failure is an opportunity to learn. And what you want to be doing is you want to be minimizing

this cycle, the amount of time that it takes you to learn. in that way, think like fail fast is one of the best possible outcomes of an entrepreneurship exam, like a journey.

Manoj Tandon (08:53.584)
Yes.

Manoj Tandon (08:59.228)
Jake, those are some very, very wise words and they come from experience. I’m glad to hear you say it, because nobody listens to me, but hopefully they’ll listen to you. That’s very, very true of what you’ve said. anyone, any of our audience that’s listening, think about what Jake said. And if it’s something that is in your DNA to do this, absolutely take the chance.

Jake Moshenko (09:04.546)
Haha.

Manoj Tandon (09:28.87)
The worst is never so bad, actually, from what I’ve seen. It can be bad, but it’s never so bad. So let’s talk about going to distributed systems. Well, before we get there, I’m gonna ask you a couple of stupid questions and maybe you can enlighten us. When we think of Red Hat open source model, Kubernetes open source,

How do you pay salaries on an open source system when you build off of it, utilize it, or when you’re providing your technology as an open source platform? How does that work?

Jake Moshenko (10:08.972)
Yeah, well, at Auth said we’re also open source with SpicedDB. So there’s a few different business models, but the standard business models are charged for support or charged for management of the service. Sorry, of the service. Or the third one is usually to charge for ancillary software and services. So first one’s really obvious. You put a thing out there.

People adopt it and then they need some kind of support. lot of times banks, for example, will require that they have a support contract from a vendor for all of the software that they bring in. So that one’s really straightforward. That was kind of the first open source monetization strategy. Then like you can take the software and you can run it as a SaaS. That’s something that we do at AuthZed. We run SpicedDB as a service.

Manoj Tandon (10:46.546)
Okay.

Jake Moshenko (10:59.754)
And that takes the burden of running and managing the care and feeding that takes it off of the customer and moves it on to you. And that’s a thing that they’re, they’re, you know, they’re, familiar with paying for. Right. So like I would go and have to requisition this hardware from a cloud vendor and I would have to get SREs to manage this thing. And I would have to update it and go through the whole software life cycle management of the service. So I might as well pay someone else to do that. Right. That one’s the other really standard model.

And then the third model is to sell ancillary services and software. So if you have an open core, then you can decorate that with other things that make it more enticing for businesses. And then they can pay you for those things and run those in the periphery of the open core. And you can make your money

Manoj Tandon (11:46.182)
There you go, folks. There’s the mystery that is solved. So you guys are open source.

Jake Moshenko (11:52.098)
We are,

Manoj Tandon (11:54.406)
So, you’re built on SpicedDB?

Jake Moshenko (11:57.506)
We built it. So SpicedDB is just an open source project that we created. AuthZed are the maintainers of that project. We build it that way so that we can build a multi -company coalition behind the idea of SpicedDB. It doesn’t just have to be an AuthZed project. And in that way, sort of like how Google created Kubernetes but launched it with a multi -company coalition.

Right, like that way it can become something that people can adopt without worry about vendor lock -in, things like that. We’re still the primary maintainers, so that hasn’t come to fruition yet, but that’s sort of a rationale for why we built it and launched it the way we did.

Manoj Tandon (12:37.658)
So let’s talk about the basics of a distributed architecture. So, and then we’ll get into more details on what AuthSet is doing and how you guys function. But in a distributed environment, if you could do a compare and contrast to maybe a more traditional environment like an Oracle or a Microsoft SQL relational database, what’s going on?

in a distributed database in simple terms for non -rocket scientists. What’s happening?

Jake Moshenko (13:14.774)
Yeah, so the first question is why? Why do we create distributed systems at all? The reason we create distributed systems is because we’ve exceeded one of the limitations of running something on a single server, on a single box, in a single process. And those limitations, there’s some really obvious ones, right? Like the CPU isn’t fast enough to do all the work that we need it to do.

The biggest CPUs that you can buy right now have on the order of hundreds of cores. But if you have 1 ,000 cores worth of work, you can’t run that on even the biggest server. Another limitation would be memory. That’s an obvious one. But other limitations that we try to work around are things like failure domains. So in a single server running in a single data center with a single internet connection and a single power provider, we have all these different failure domains like

that these things that could interrupt our ability to maintain our service. So if you’re running out of one co -location in Newark, New Jersey, and that colo loses power, loses internet, your service is offline. So distributed systems let us exceed the capability of one box, like one particular server, as well as decouple our services uptime from all of these various overlapping failure domains of running a single box in a single server and a single data center.

So that’s sort of the rationale of why we do distributed systems at all. When you contrast it with something like an Oracle, like let’s take Google Spanner or Spanner likes like CockroachDB. When we take those and we consider them against something like a Postgres, like a single Postgres server or an Oracle SQL database, what we’re really doing there, like the Oracle SQL database, its high availability comes from running an exact copy of that one box.

Manoj Tandon (14:50.289)
Okay.

Jake Moshenko (15:06.382)
So you have one box, then you have an exact copy. If you lose the primary, you flip over to the secondary and you go from there. What happens if you lose the secondary? Or what happens if they’re too close together and they’re in the same data center? So there’s all kinds of things that can still take you out even when you’re running a quote unquote highly available single server database. So the distributed databases try to work around that, but of course there’s a price to pay. When we’re writing to a distributed database, we’re usually storing multiple copies of the data.

We’re usually requiring that multiple different servers acknowledge that they’ve accepted and stored those copies of the data before they accept the write and say that the write is quote unquote committed. We’re usually trying to coordinate with multiple servers in order to service a query. So if someone says, give me all the data back, right? Like give me all the rows in the database. In a traditional database, that’s just a single scan through a table. But in a distributed database,

you’re potentially talking to every single node in the distributed database in order to gather all of that data and put it together and give it back to the user. Was that like a good 10 ,000 foot view?

Manoj Tandon (16:14.866)
That is a great 10 ,000 foot. you actually used the word of the commit of the transactions. And that’s something that’s fairly interesting here is how does that work in a distributed model? I remember, and it’s been years since I have actually done work on an Oracle database, but in a high transaction space, we would wait to commit to the transactions much later in the day when there was availability and you’d hold off.

you had really fancy transaction management going on to manage that entire process. When you got a distributed environment, how do you, do you commit those transactions immediately to something? And then how do all the other servers pick that up and synchronize themselves?

Jake Moshenko (17:09.132)
Yeah, so this is getting out of my direct field of expertise. There’s a couple of different papers that I would recommend if you’re interested in durable rights to distributed systems. The first one is the Dynamo paper that came out of Amazon. And that was basically saying, if you have seven servers and they all need to agree on something, if you can get seven, you

Manoj Tandon (17:15.762)
It’s okay.

Manoj Tandon (17:27.964)
Okay.

Jake Moshenko (17:36.852)
more than half, so four would be one more than half, right? If you can get four servers to say, I’ve received the write and I agree that this write, you know, like I agree that this write has happened, then all you have to do now is read from four servers. And if they all agree, then you know that you’re looking at the latest copy of the data. That’s how you can get your consistency, right? So they call that quorum write and quorum read.

You can relax the consistency requirements. So you can just do quorum writes to make sure that things are progressing on a commit timeline. But if you don’t do quorum reads, you’ll get potentially stale copies of the data. The other paper that’s super interesting is a paper called the Spanner paper that came out of Google. And the Spanner paper talks about how you take these primitives, how you take essentially Paxos, which is a distributed decision -making algorithm.

Manoj Tandon (18:18.866)
Okay.

Jake Moshenko (18:30.402)
how you take Paxos and you turn that into an asset database with transactions. But fundamentally, it still relies on the same idea. There’s a leader election process where those four out of seven servers, for example, have to agree on who the leader is. And then the leader is the one who’s coordinating all of the reads and writes unless the leader goes away for some reason. But then as you have transactions going in, there’s a weird thing that can happen. Whereas if transactions are interleaved, one transaction can preempt another.

Manoj Tandon (18:58.267)
Okay.

Jake Moshenko (19:00.078)
in this distributed system. And then you need compensation mechanisms for what happens. What do you do when that happens? Right? Sometimes you can just reorder them if they’re not overlapping. Or if they are overlapping, you potentially have to reject them. So there’s a whole bunch of very deep, very technical mechanics for how this gets done. And those two papers are sort of like the, the holy grails or the Bibles of how this, how this system is progressing.

Manoj Tandon (19:24.898)
thanks for sharing that. oftentimes we have folks that talk about these topics and they’re talking about it at a level where some of the folks don’t know the foundational basics. And you just described what the basics are here, how this is working. And thanks for sharing. I will try and link those up in the show notes too, those papers.

and let anybody who wants to get into more of the details get into it. But that’s absolutely a fascinating space. Now, do you have to have all your database on one hosting provider? So do you have to be on AWS for everything in a distributed system? Or Microsoft or whatever it is? How does…

Jake Moshenko (20:16.49)
No. So the short answer is no. You can have your data on multiple different providers. In the Dynamo example that I gave earlier, the downside to doing that is if you’ve got data from far -flung providers or providers that are sort of far away from a network latency or speed of light distance.

You have to wait for all of those nodes to acknowledge that something happened before the system can make progress. And that takes longer, right? Like the farther away the servers are that are acknowledging things, the longer that takes.

Manoj Tandon (20:48.335)
How are you now at Aught Zed, you’re going to face that issue with the work that you folks are doing.

Jake Moshenko (20:56.982)
Yeah, mean, we don’t have customers who want to run in multiple cloud providers with a single database right now. From a durability perspective, they might want to have backups in different cloud providers. But it’s not something that we’re dealing with right now. But it is common to actually have, under the Spanner model, it is common to have your database spread out in multiple geographic regions or in multiple, at least, availability zones.

Manoj Tandon (21:07.932)
Okay.

Jake Moshenko (21:24.94)
And you still suffer from some of those problems. But one of the things that really helps is within a single cloud provider, they may have a single clock that they’re tracking, a single version of UTC. And the closer the clocks are and the closer that you can prove the clocks are together, the less compensation you have to do when those transactions are overlapping. Because one of the hard parts about distributed systems is even figuring out when things are

Because you can’t trust the clock in the computer.

Manoj Tandon (21:55.986)
Absolutely not.

Jake Moshenko (21:59.214)
So this thing says it started at noon and this thing started at 1201. Well, what if your clocks are off by two minutes, right? You don’t know what order those things actually happened. So within a single cloud provider, the clocks are usually closer together than they will be across cloud providers.

Manoj Tandon (22:13.606)
Yeah, there’s no, unless you had an atomic clock coordinating every single and everybody was referencing that, but that sounds like an expensive, expensive.

Jake Moshenko (22:25.102)
I don’t know if you’re teeing me up or what’s going on here, but the Spanner paper from Google actually talks about using atomic clocks in order to set a very tight bound on how far their clocks can skew so that they have to do as little compensation as possible in order to make sure that their database is progressing.

Manoj Tandon (22:27.142)
Go ahead.

Manoj Tandon (22:45.486)
Yeah, and that would be an expensive solution unless, well, the U .S. Naval Observatory is handling our national time, unless they’re willing to provide that service as a gratis.

Jake Moshenko (23:00.536)
Well, the hard part isn’t getting time from an atomic clock, right? Like anyone can go and do that. Even like GPS, GPS is just satellites with atomic clocks in them. The hard part is how far that atomic clock is away from the call to what time is it right now that happens on the Linux kernel. So for example, if I have a server that’s running in a data center somewhere,

And it did an NTP request to get the time from an atomic clock and it was synchronized for a very short amount of time. That’s fine. But if I make a syscall to get that time and I get interrupted, or if I make a syscall to get that time in it, you my VM gets put to sleep or there can be all kinds of things that can cause those, like even those very high quality time signals to degrade. And so then the work becomes, how do we give like a hardware path or like a guaranteed path to call and ask for the time?

such that we have a high degree of confidence that the time that we think it is, is still the time that it is when we, you know, attach that to our transaction or

Manoj Tandon (24:02.992)
Now, when you guys have that sorted out, let me know, because I have no idea how that’s gonna take place, and I have no issue in admitting that I have no idea how that’s gonna work. But all joking aside, it’s a very serious problem. Like when you’re doing role -based access control, anything with RBAC, Latency is gonna become a big deal, I would imagine.

Right? Because you’re controlling all the identities in one place and you might have geographically distributed users everywhere. And any queries to that are gonna, any latency is gonna, it has to flow back or no. Am I wrong?

Jake Moshenko (24:42.454)
No, you’re absolutely right. It’s critical that authorization be fast because authorization is usually in the line of fire for every user request. Right? If I go and I try to send an email, I happen to have my email open on my desktop. If I go to send an email, right? Am I allowed to do that? Am I authenticated as the user? Right? Like, am I sending it to a contact that I’m allowed? You know, that’s just one silly example, but if I open a document, am I allowed to make changes to it? Am I allowed to view it?

So all of those things are directly just turning into latency and reduced user experience for the user if you don’t have fast authorization checks. And one of the reasons that we use distributed databases to back our authorization service is so that we can have copies of that data very close to the users who are requesting that data. And we also want to make sure that we’re giving a consistent view of the authorization world globally, because we don’t want like someone revoked something on a different continent

and it just hasn’t trickled to us yet, we really don’t want to be giving you access to something that you’ve already had your access revoked. So we try to use a highly consistent, globally distributed data store to back our authorization decisions for those reasons.

Manoj Tandon (25:41.745)
Yeah.

Manoj Tandon (25:55.67)
That was going to be one of the cybersecurity questions that I was going to ask and you already answered it. So now don’t have to ask that, but in.

Manoj Tandon (26:08.068)
When if everything is on in this kind of a distributed approach and you mentioned like you folks don’t have anyone that’s really on multiple hosting providers. So the concern would be while there’s a lot of there may be a lot of resilience in a distributed architecture. If the underlying platform itself isn’t distributed, then that becomes the weakest link and it can become a single source of failure. there’s right nothing that says

that Amazon can’t be taken down. I mean, they’re going to make all kinds of claims, but in theory, nothing is 100 % that’s hack proof. There’s nothing that is totally safe. So if you take AWS house as an example, and I’m just picking on them because they’re popular, not because they have anything wrong with them, it doesn’t matter that you had a distributed platform.

How do you mitigate these risks or you have any thoughts on

Jake Moshenko (27:06.274)
Yeah, I

Jake Moshenko (27:11.386)
There’s two parts. One is that our customers themselves are also running on AWS. So if AWS gets taken out, like their app is down, they’re not super concerned with their authorization provider being up if the app itself is down. The other thing is that AWS is architected quite cleverly, right? You have individual buildings that house servers, right? Those are data centers.

And then multiple data centers, right? Like physically distinct buildings, multiple data centers are aggregated up into the idea of one availability zone. So people don’t know that. Some people think that you can have multiple availability zones within a single physical building. That’s not true at all, or even that they’re one -to -one correlated. In AWS, an availability zone is usually multiple different physical buildings, data centers. Then on top of that, you have this concept of a region, right? So you have,

US East 1, for example, would be a region. And then you have US East 1, A through F. Those are your availability zones. So you have like seven availability zones contributing to a region. You try to spread your work all around the availability zone. And then the region is geographically co -located. So let’s say US East 1 is in Northern Virginia somewhere.

Manoj Tandon (28:22.994)
Okay.

Jake Moshenko (28:24.94)
Now you also have multiple geographic regions. So you have Northern Virginia, you have Iowa, you have Ohio, you have Northern California, you have Oregon, you have Seattle. So you can also spread your data and your application around into multiple geographic regions. And that’s kind of the highest level that most people worry about when they think about high availability is what if I’m in multiple geographic regions?

then you can survive sort of some of these sort of disaster scenarios like an asteroid strike or a nuclear bomb or things like that. And that’s kind of the highest level that you think about, but no one, like literally no one can ever guarantee 100 % uptime, right? Like I think you remember back, what was it? 2001 or 2002, something like that, the whole Northeast of the United States all the way through the Midwest lost power, right? Like.

Manoj Tandon (29:07.514)
Absolutely.

Jake Moshenko (29:21.922)
That’s one of those Black Swan events where you’re like, okay, well I was in Ohio and I was in New York or Newark or whatever. Like that should have been good enough, but this once in a century kind of outage happened. you know, when those kinds of things happen, like it’s okay for us to have a little bit of downtime. Does that make sense?

Manoj Tandon (29:38.834)
It makes sense. It does make sense. And the key part is that there’s no one that can guarantee 100 % availability. And I think when folks are thinking about this and decision makers are thinking about these things, they should know that. The Titanic was not unsinkable. Right?

Jake Moshenko (30:01.518)
Yeah. We usually talk about these things in terms of nines. Have you heard this term before? How many nines of availability you have? Yeah. So two nines would be like 99%. Three nines would be 99 .9. Four nines would be 99 .99, et cetera. Five nines is sort of what like the telecom industry uses to try to measure their availability. Like that’s their availability target. And that’s considered like a very high availability. I think it’s like a few minutes of downtime.

Manoj Tandon (30:05.946)
Yes, I have, but go ahead.

Manoj Tandon (30:13.488)
Yep.

Jake Moshenko (30:30.702)
per year or something like that. But even they don’t say 100%.

Manoj Tandon (30:32.378)
Although we saw with the AT &T issue recently, they completely violated that.

Jake Moshenko (30:39.63)
Yeah, well that’s, that’s a different story. But like I think anybody who’s out there claiming that they have 100 % uptime, what you really want to be looking for are what are the things that they’re going to give me when they eventually violate that? Because nobody is up 100 % of the time. Just period.

Manoj Tandon (30:58.044)
So let’s talk about Ozzed. I wanna make sure we get to that and dive into us. Tell us a little bit about the company and what you guys are focused on towards delivering.

Jake Moshenko (31:12.556)
Yeah, so we’ve talked about it a little bit already, but we’re an authorization platform. So if you’re building an application, let’s say you’re trying to build the next Facebook or the next Google Docs or the next Slack or whatever, you’re building an application and you get to this part where you need to decide, can this user perform this action? It sounds like a very trivial problem at first. All right, I’ll just store a little bit of data in the database and say this user’s admin.

And then I’ll write some code that says, admins are allowed to do x, y, alpha, beta, gamma. It seems very simple. But what ultimately ends up happening are you get more traffic than you were anticipating. And you end up spending a lot of time loading that data out of the database and processing it and running it through that decision -making source code. You expand geographically. And so now you’ve got

Manoj Tandon (31:42.769)
Yep.

Jake Moshenko (32:04.3)
Two places where you need to make authorization decisions and you’ve only got the database in one place, right? So there’s where the distributed database comes from. Your users start asking for things that you never anticipated, right? Like you built the next Facebook, but then you’re trying to sell Facebook to enterprises and they have a much richer view of what authorization should look like than you anticipated when you first built the app. So you built something very crystallized and not very flexible. And now you’re sort of stuck with it.

So ultimately what all these things sort of point to is having a separate authorization service that’s very good at making these decisions. That’s very flexible. That’s very scalable in both the traffic and geographical and feature requests along all of those dimensions. And having that external service that can make these computations for you really takes all that kind of gross security, like very risky code that you have to write. It kind of takes it all off of your plate.

You get to delegate it out to someone else who’s an

Manoj Tandon (33:04.912)
Now, again, a stupid question I’m gonna ask here. Are you not treading onto or have overlap with a lot of the IAM vendors that are out there?

Jake Moshenko (33:18.904)
It’s an interesting question. So IAM for anybody who’s listening doesn’t know what it is. IAM is identity and access management. IAM as a field traditionally refers to like a company’s first party employees, right? Like a company’s direct employees. You know, I’ve got Jake over in accounting and Jake is allowed to touch the spreadsheet or make these updates in QuickBooks or whatever.

What we’re really focusing on is more of a new field, and that would be C IAM, like customer -facing IAM. And we’re not even doing the identity part at all. We want you to go pick your identity provider of choice. We support all of them. So we’re really just focusing on customer access management. And there aren’t a lot of vendors in this space yet, but certainly we have competitors. Any legitimate business is going to have competitors because there’s value there. There’s value to be had.

So there are a few competitors there, but they’re not the traditional IAM vendors that you would think of.

Manoj Tandon (34:22.022)
That’s interesting. Because I thought, you know, the Siam folks would, might overlap. So you’re relying on somebody else to verify the identity. You’re picking it up once you’re saying, okay, Minoji is verified. Now question is what controls does he have or not have at that point? Once they’re to you,

Jake Moshenko (34:38.05)
Mm -hmm.

Jake Moshenko (34:47.02)
Yep, exactly.

Manoj Tandon (34:51.716)
You don’t care if that’s fake, legit, whatever it is. At that point, an identity provider said, this is good, so you’re going to run with

Jake Moshenko (35:05.292)
Yeah, well obviously we hope that you’re using an identity provider that you trust and isn’t giving you fake or illegitimate identities in your requests.

Manoj Tandon (35:10.834)
Well, of course, but nothing’s 100%, as we’ve already discussed. And there’s been umpteen cases in recent times where…

Jake Moshenko (35:21.676)
Yeah, absolutely. One of the neat things though is that you can sort of work with the controls that the IAM provider gives you. For example, sometimes they’ll tell you, well, this person logged in within the last hour, or this person logged in using two factor authentication, or this person, we verified their email or not. You can use some of those attributes in your authorization decisions. So for example, Minoj logged in with like assurance level of one.

where he just had the password, for example, might be allowed to do different things than Minoj with assurance level five, where he did two factor within the last minute and all these additional things, these additional heuristics that you can bake into the identity decision. You can use those in your authorization decisions to say, you can do the admin things if you’re logged in with a high level of assurance, but you can only browse around if you’re logged in with a low level of assurance.

Manoj Tandon (36:15.811)
Now, in the AuthZ environment, are you storing a lot of attribution information against the identity or is it only against the role itself? That’s it. So they don’t know it’s me. we’re sorry, again, we’re a cybersecurity podcast.

Jake Moshenko (36:33.08)
So that, yeah, that’s a great question.

Yeah, that’s a great question. We actually store everything from the resource all the way to the identity. So we don’t stop at roles. Rollback, RBAC, role -based access control, is only one of the authorization models that you can use with AuthZED. You can also use ABAC, attribute -based access control. And it’s all built on top of this thing called REBAC, relationship -based access control. So what we’re actually storing are relationships. So Minoj is an admin.

of this organization, right? That’s one relationship that we would store. And then we would interpret that relationship in order to make access control decisions. But some of the more interesting ones are, you know, Jake is a member of the Eng team, which is a member of the Employees US team. And the Employees US team has view access to these policies, right? So you can build up this really interesting convoluted graph of how somebody has access to a piece of data.

And our whole game is interpreting that graph and making access control decisions based on the relationships that are there, as well as a set of rules that you’ve given us for how to interpret those relationships and make decisions. So we’re really a very flexible, neutral platform that will help kind of apps of any shape and size add authorization to their application.

Manoj Tandon (37:55.922)
So now as an outside idea, is it possible to actually anonymize the name but still provide the right access control? So instead of Minow, G .I .U. verify my identity is correct. Now from that point forward, I’m anonymized. So if any of the distributed servers are

They don’t know who is actually in that control.

Jake Moshenko (38:28.11)
Yeah, it’s really funny that you asked that because I only use Minoj as an example. 100 % of the time in production, what these relationships actually look like are UUID 1437628 belongs to group 734562153, right? Like it’s completely unintelligible to even us as the provider of the authorization service. And it’s just giving a relationship between

these pseudo -anonymized or anonymized identities that exist on the identity provider or on the application side. A Google Doc, for example, has this 20 -digit long alphanumeric identifier, and that’s what you’re actually giving access to. You’re not giving access to a document named the secret notes or atomic secrets. So the relationships that end up in AuthZet are universally anonymized by

Manoj Tandon (39:14.79)
Right.

Manoj Tandon (39:23.472)
That’s very cool. And that means that as far as compliance goes, you guys are compliant all over the world, automatically.

Jake Moshenko (39:34.354)
I wish it were that simple. I wish it were that simple, but it certainly gives us an extra level of assurance when it comes to data privacy agreements and data protection acts. We don’t know what we’re managing necessarily, but we’re still going to do our utmost. We’re still going to treat it as if it’s the nuclear secrets, but the things that are actually in there are mostly useless if they leaked.

Manoj Tandon (40:01.606)
That’s very interesting. And that’s also really cool. That’s a really neat implementation of it. Because if you can get at the name, now you got another critical piece of data. now you can, you know, thinking from a hacker’s perspective, now you know who the admins are. Now you know who you want to spend money compromising.

Jake Moshenko (40:26.126)
Yeah, well, that’s really interesting point. I don’t know if you remember this thing that happened a long time ago now, probably 15, 20 years, but there was a leak of all of the Verizon metadata. Like, who made calls, what the area codes were, how long the calls were, and using that metadata, people could identify almost down to the individual what was actually going on, right?

Manoj Tandon (40:38.226)
Yeah.

Jake Moshenko (40:56.406)
And so that’s why like, even though this stuff is mostly anonymized, we’re still going to guard it and treat it like it’s nuclear secrets because we don’t want, you your business’s privacy and security is of the utmost important to us. We understand

Manoj Tandon (41:09.618)
And you guys have built it from the ground up. do you see yourself getting acquired here? mean, could, a Microsoft, an Okta, somebody could come along and say, you know what, this is a damn good idea. All the guys who come up with the really coolest ideas are never standalone for a very long time. what…

Jake Moshenko (41:36.686)
I, unfortunately, I don’t have a crystal ball. I can’t predict the future. I think if you look at venture funded startups and the outcome distribution, you can, you you could sort of assign an expected value to where we end up. But, you know, we’re, building this thing, you know, we’re, all second time entrepreneurs. We’re building this thing because we want, we want it to

a big independent company that can sort of solve the problem regardless of whether you’re a Microsoft or an Amazon customer. So I won’t venture to say where we’ll be in five or 10 years, but it’s not the goal. We’re not trying to build this thing to flip it.

Manoj Tandon (42:15.098)
Yeah, that is fabulous to hear. And most of the big innovators, the guys who are passionate about it aren’t. They’re set out to solve a problem. A very real, tangible problem.

Jake Moshenko (42:28.876)
Right, exactly.

Jake Moshenko (42:33.164)
And it’s a problem that we’ve lived. So we know how important it is to solve application authorization in a flexible way.

Manoj Tandon (42:41.043)
What will AI do in your world?

Jake Moshenko (42:47.362)
You know, I get this question a lot.

Jake Moshenko (42:53.88)
I think if you sit down and think about it from first principles, like what would you want AI to do in this world?

Jake Moshenko (43:03.382)
You could imagine, let’s use like a physical access control metaphor as an example, right? Like a lot of people have passcode or like key cards that they use to buzz into their apartment building or their office or whatever. And that’s being enforced by computers and it’s very precise. And if we think about what that replaced, it replaced like the front desk person.

Manoj Tandon (43:14.705)
Yep.

Jake Moshenko (43:30.016)
or like a physical security guard. And the security guard could be fooled, right? Like somebody may have been let go, but the security guard didn’t get the update, didn’t know. Someone could be, you know, we could go mission impossible with it and they could be wearing like a really clever mask, right? So it’s like, if we think about the failure modes of AI, AI has a propensity to make things up sometimes.

Manoj Tandon (43:56.572)
Sure.

Jake Moshenko (43:58.54)
And we don’t really want our access control subject to that failure mode. So I don’t think we want to put AI in the loop in terms of making access control decisions anytime soon. The place where I see AI giving us the ability to work faster or innovate would be in places where we can have AI, where we can double check the work that AI is doing. Right? So.

Manoj Tandon (44:01.137)
Yeah.

Jake Moshenko (44:25.024)
I have some ideas for how I want my permissions model to work in my application. Maybe I can describe it to an AI and it will generate the policies and everything for me. And maybe it’ll suggest how I could integrate that with my source code.

Manoj Tandon (44:33.189)
Okay.

Jake Moshenko (44:38.968)
But I can still double check those things. I can look at the policy and I can say, all right, did it make the right decision here about what an admin should be able to do? And if it didn’t, I can go and as a human, I can fix it. So it’s just saving me from typing and maybe helping me think in some avenues that I wouldn’t have thought ahead of time. But it’s not in the loop, right? I’m not taking the output of the AI and immediately using that to make decisions without double checking it. So that’s sort of where I see our ability to use AI.

Manoj Tandon (45:05.614)
And I would agree with you, the human is still very much in the loop. So you’re describing a scenario where as an engine of innovation, mean, the innovation really still stays with the human, with a thinking conscious person. AI per se is not going to innovate for you. It can work on scenarios, but…

You have to come up with the scenario ideas. You have to come up with, you know.

Jake Moshenko (45:41.184)
Yeah, yeah, I think that’s a great succinct way to put it, right? Like the AI is not going to innovate for you. When the AI does start innovating for us, you know, I think we’re all going to have to go out and find new jobs because…

Manoj Tandon (45:52.934)
I think a lot of things. Well, now we’ve got the Terminator story. Because at that point, why does it need us?

Jake Moshenko (46:01.706)
Right, right, right, right. The AI is going to be like, I don’t know why I’d put a human in the loop. That just seems dangerous. They’re, they’re so fallible.

Manoj Tandon (46:05.752)
Yeah. Yeah, there’s no measuring of nines in their reliability. So

Jake Moshenko (46:13.75)
Right. Did you know any of them could disappear at any moment? That’s horrifying. Yeah. Yeah. Exactly.

Manoj Tandon (46:16.75)
Yeah, yeah, it’s not reliable. To heck with them. So, Jake, in the last couple of minutes here, anything you want to plug, anything you got up and coming, want to let our listeners know about,

Floor is yours.

Jake Moshenko (46:35.884)
Yeah, I don’t have anything. We have some talks and things that we’re always doing around the internet. But without knowing the specific timing, I don’t want to plug any of them. Just check out our YouTube channel. We tend to repost things there. Our blog, if you enjoyed some of the things that we talked about with authorization or distributed systems, things like that, our blog is a great place to go and learn more about those things.

generally very technical blog. We’re not afraid to dive into some of these issues and some of the tricky bits of what makes this problem hard and what makes it interesting to solve. yeah, that and then other than that, if any of this stuff sounds interesting or relevant to you as a business, I’d be happy to talk about what you’re doing for authorization. You know, no strings attached.

Feel free to reach out. have a calendar on our website where you can reach out and book a meeting with us and kind of just talk about what we’re doing and we can kind of help you from a sort of neutral stance to see if there’s anything we can help with or if we can point you in a different direction.

Manoj Tandon (47:48.115)
Well, that’s awesome. If there’s any links you want us to put in the show notes, just send them on through and we’ll make sure that those are connected up as well. But this has been a very cool conversation, Jake. And yeah.

Jake Moshenko (48:06.36)
Cool, I’m glad you enjoyed it. I enjoyed it too.

Manoj Tandon (48:07.334)
I think our audience is gonna like it. There’s a lot that was discussed and we’ve only just scratched the surface of what are very, very deep topics and their own individual rights of the many things that were talked about. But we’re honored to have you here, Jake, and thanks for doing this.

Jake Moshenko (48:28.652)
Yeah, and thanks again for having me.

Manoj Tandon (48:29.39)
Absolutely, you’re welcome back anytime.