Cheap Prediction, Expensive Change

All right, so we're really kind of in the thick of a breakout moment for predictive technology, thanks to OpenAI's large language models. And I was curious if these developments have changed some of your perceptions about predictive technology or general thoughts about how it might continue to grow in the coming months and years.

Okay, it's been an exciting six months and it's been the changes have happened faster than I thought they would. The technologies still are prediction technologies. And so what CHAT GPT is doing or GPT-4, what all these generative models are doing is taking the information they can gather from wherever around the world, the internet, wherever else, and then filling in missing information to help with some task. technology, machines are increasingly doing prediction. We're starting to realize a whole bunch of things that we may not have thought of as prediction, like writing, our prediction, and that's leading to a bunch of opportunities for us humans. In what can we do better now that we can do these various types of predictions at scale? It came faster than I anticipated, but it hasn't really changed my thinking about what the opportunity is and where the underlying challenges are.

So as an economist, being that this form of predictive technology is kind of generative AI that's doing things that seem to be surprising a lot of people, is that disrupting decision theory or kind of the way you think about economics?

Um No, I'm trying to just have a longer line than just no. So look, during the dot com boom, there were all these people who said we needed, there was a new economy and we needed a new economics. And a handful of prominent economists, Halvary and Khrushpiro and some others said, no, we don't need a new economics. The principles of economics are very powerful. We understand that when the price of something falls, we do it more.

we do it a lot more. And so once we understand what's changed, then we can map out the consequences. I don't see anything different here. So when we had the rise of the internet, we thought through, well, what changed with the rise of the internet? Search got cheap, copying got cheap, and communication got cheap. And some of the consequences of those were pretty straightforward. When search is cheap, you're going to do more search. copyright, it's going to be a little bit harder. Some of them were a little less intuitive, but understandable once you thought through what a downward-sloping demand curve means. For example, once copying is cheap, privacy is suddenly going to be a much bigger deal because when anything you say to anyone is now instantly broadcast around the world for free, you're going to pay a little more attention to what you say. But I do think we need to understand what's cheaper, what exactly has gotten better, how does that fit into our current workflows, and then perhaps most importantly, what are the compliments to that thing that's gotten cheaper? What becomes more valuable as prediction gets cheap? To the extent that the manifestations of the prediction have to do with writing or coding, or all the various capabilities of generative AI, that remain human skills. And one of the compliments that are assets that companies can invest in.

one thing that Rob and I have discussed a lot is that, and it's interesting that in order to really move forward with AI and integrate it into a business, you really need to invest in your people. It becomes an investment in employees and training them on how to implement AI and making them an active part of actually kind of building solutions because they often know firsthand what kind of processes can and should be automated and confined. ways of doing that, that surpass kind of what they were able to do on their own or even in groups. So,

For sure, for sure. with any new tool and any technological change, you need to make sure that people, working in your organization can use it and understand how it applies to their particular workflow, their particular context, and how it might create new opportunities for them to create value to your end customers to various other stakeholders in the orchestra.

Yeah, and you mentioned also that economics can be valuable for mapping consequences. I wonder if some of that thinking could be useful, not just as businesses are searching for ways to cut costs and make more money, but also thinking about their designs and kind of unintended consequences that might show up three or four years down the line that could not only negatively impact their business, but also could have kind of perilous society.

There's lots to worry about. There's lots to be excited about, but there's also lots to worry about.

When we look at a technology, an emerging new tool, it's always easy to think through, what are the current processes that we do that we're not gonna be able to do anymore? So which humans are gonna get replaced, which aspects of my job are gonna get replaced, and what things that we worry about is this new tool not really gonna fix? For example, if prediction machines are able to code and code effectively, there's a whole bunch of people whose jobs involve coding who may be at risk. If the machine is able to write, there's a whole bunch of people who make a living by parsing obscure grammatical rules in English, and that's going to create a threat. If the machines are using data based on human inputs and data, and we know that humans are all the time, then we should worry about machines discriminating just like humans do. All the things to worry about, we can jump at them, we can identify them very easily. The harder challenge, but maybe the more important one, or at least the equally important one, is to think through, what can we do differently? What's better about using a machine for this process than the old way of doing things? new product lines and maybe even hire lots more people because we can now write or code at scale. How can we use these tools to reduce discrimination and bias? Because we know humans are both extremely biased and not really auditable. Where a machine, when you have a biased machine, you can audit it and to the extent of the person who owns it or controls it, wants to improve and reduce discrimination, they can So, thinking through what those new opportunities look like, for example, There's a handful of people who make a living writing, for sure. There's a lot more people whose job opportunities are constrained because they don't write well. And what I mean by their job opportunities are constrained is writing is a compliment to some other skill they have. So if you run a small business, writing letters to your customers or introductory emails or things you're selling as part of your small business. And millions of people's job opportunities are constrained by the inability to write. If now anybody can write effectively at scale by putting in some notes into a generative AI and getting a well-parced paragraph or sentence or article back, then that creates incredible opportunities, we call it upscaling, for all those people. Now, I don't want to belittle the fact that those of us who do make a living writing might might be worse off. That matters. But there's a lot fewer of those people than there are, we think, we haven't done the detailed research on this, but we think there's a lot more fewer of those people than there are people whose job opportunities are constrained by writing.

Yeah, there was a time when you were lazy because you used spellchecker, and then you became lazy because you didn't use spellchecker. And you wonder if we're on that same trajectory, like you're lazy because you used, you know, And then all of a sudden somebody sees something that's, you know, grammatically incorrect and misspelled and they say You're lazy because you didn't use it Same sort of idea there

Absolutely. So when it was clear how easy to use chatGPT was, it was right at the beginning of this semester. And one of my students put up their hand and said, okay, so what's the deal? Are we allowed to use this for our homeworks and assignments and whatnot? And I said, of course, of course you are. But for those of you who don't, you no longer have an excuse know that half of you, your first language is in English, and so I've always been a little understanding about these grammatical errors or whatnot. But now going forward, there's no excuse. You can put anything in chat GPT and have it and say write it neatly, and it will write it neatly and clearly. And you can copy that and paste it in and tante your homework. No more excuses for such mistakes.

So I wanted to kind of center on that a little bit. I don't know when I was introduced to this idea, but at some point I came across a concept, I don't know if I was reading an article or a book or something and they talked about how, formality in our writing, formality in our communication really biases the kinds of ideas that get shared. towards fear of being scrutinized for bad grammar or poor spelling causes a swath of the population to not feel comfortable sharing ideas formally. And so there's a certain amount of, let's say, intrinsic censorship that goes on. That when we see things like rap, we see ways that ideas can kind of make its way to a broader And so it almost seems like now it sort of opens the door to a, yes, a lot of noise, but also the ability to get more diversity in the sharing of ideas. And I don't know if the noise is going to kind of diminish the value of the diversity.

So it sounds familiar. What do I mean by that? Most adults in the early 1990s didn't write or wrote very, very rarely. We read. We consumed information. But actually writing stuff was unusual. With the rise of the internet, particularly early social media, like Facebook, which is largely written, there was a adults actually wrote things. And that clearly led to a lot of noise, but it also clearly led to a much wider sharing of ideas. Now, the political economy of that, whether that's been good or bad, reasonable people can disagree. Lots of people say it's led to massive polarization. Lots of people say it's led to increased political participation. That's your point of view. This feels like a movie we've seen before. And what I would expect going forward is what we've seen in the past, but even more so. So if you think the changes that we've seen as more and more adults have been able to write and communicate their ideas publicly over the past 30 years, if you think that's good, then you're going to think it's more good. are not going to be covered.

So is it fair to sort of say that possibly some of what we're seeing today isn't a changing of ideas or polarization of ideas, but that people that otherwise were muted are now speaking and heard and that's that ability to speak publicly and be heard is creating them to organize better than they were able to before as a result of just being limited in the ways of communicating. So we haven't changed our minds. These are values that have been already seeped in society just quietly.

Absolutely. And an important aspect of this is the intermediaries have changed. So it used to be if you wanted to write something that would be widely read as a media piece, you would need one of the major news organizations to approve it and print it. That's no longer the case. You can have whatever thought you want. You can write it down in 140 characters or less, maybe more. around the world to whoever might be interested. Often nobody's interested, but sometimes those ideas take off. And Joel Waldfogel, who's an economist at the University of Minnesota, has done a bunch of work on not so much media per se, but how the change in these intermediaries has changed all sorts of cultural products. So in the book industry, publishers And they had a particular view until about 15 years ago about what people might want to read. And then self-publishing came along and effectively an entirely new category of books that is by some accounts the most popular category of books arose, which is the, we might call the romance genre, that prior to self-publishing was tiny. It was not something publishers really focused on. It turns out there's a huge audience for that. And then that self-publishing led to the publishers even printing some of those books. We've seen something very similar in music with the rise of independence and the rise of people just posting their own songs and maybe people listen. And frankly, that's what podcasting is about too. You're no longer constrained by the airwaves and who wants to put what on the radio.

I want to switch a little bit over to, I don't know if you guys have followed what Stanford recently released, but their ability to create a model, a large language model that they're saying compares quite, quite evenly with GPT 3.5. They, they trained it for $600 and that's massive difference considering a $5 million training. But they used, they used GPT 3.5, I think, to train it. So it's this idea that now this technology just went from $5 million to train. You need investment to $600. The implications to open AI is that they invested $5 million and now somebody can, in a sense, that competes with them. And I think, nobody expected the economics of that to change so rapidly, so quickly. You know, I think people thought maybe years from now, this will get down to thousands of dollars. But nobody thought weeks, five weeks later, $600. And they haven't run it on GBD4 yet. And so, you know, so there's a lot of room for improvement on this. This commoditization of sophisticated technology like this and the speed in which this is happening is staggering. What do you think about the concept? I know in your book, which I've used it a million times, probably since I read it, this idea of the sort of incremental improvement on the tools we have using AI versus the systemic changes that are coming and where the real value is. And that, in compared to the speed in which this technology is It just makes me wonder, who are the companies, who are the organizations that, what qualities do they possess, the ones that are going to adopt and really win? Because it feels a bit like a race.

Okay, that's a great question. So first, it's still a bit of an open question whether this $600 AI is really through a bust machine that what open AI has built? That's an open question, but let's say it is. Let's just take the claim of face value and say, okay, we now have, you can spend billions of dollars building an AI and someone could copy it for, for effectively free, if you think about what the opportunity is. What does that mean for the business model? So the first take is, well, it means you're not making money Okay, so, open AI, we might have thought of its business model, we might have had one as selling those predictions. So you write something, it comes back, and it's selling you that prediction. But if anyone could copy that for very, very little, then your incentives to build predictions around a business that sells predictions goes away. And similarly to the extent that open AI and

or effectively doing the same thing, even if it's not $1,000 per, but it's $5 billion, but if there's five or six or seven competitors, then it's going to be hard to make money on selling those predictions. So where do we look for the business opportunity? We look to the compliments. So what is a key compliment, or in this case, an input, into those predictions? It's going to be compute. are are fine if someone else can mimic your algorithm because you're still going to have to do the compute. There's clearly a business opportunity for Microsoft, for Amazon, for Google, for others who sell cloud services and who sell compute. Even in the world, to be clear, I'm not sure this is happening, where the predictions become a commodity. There's distinctive or unique data that can allow you to build a particular kind of prediction. Then you could selectively sell that data and that can create opportunities. That data would have to have a short shelf life because selling an archive is going to work once and then it's done. Another opportunity is going to be around selling the compliments to those predictions. in medicine, diagnosed really well. So, I can provide a prediction of what's wrong with you. Well, if you have a business built on the compliment to that, say, the treatment for a particular diagnosis, and now more and more people are diagnosed, then that's gonna be a huge business opportunity. The way that, to me, the way to think about the challenge of, well, if predictions are commoditized, what happens? So version one is then no one's incentivized to build predictions at all. But I think the more likely scenario.

Right, right. It's like drug companies that want to, invent drugs or develop drugs that are for rare diseases.

Right, exactly. And so they'll have incentives, people who sell the compute have incentives, people have unique data have incentives to invest, and people whose business models can get better with a commodity prediction will have incentives to build and invest in those

Yeah, we talked about this idea in our book. We use the example of Photoshop, Photoshop being a bundled suite of tools. But if you're using a chat interface and you have a photo you want to edit, if you can just ask it, bring up the, I don't want to lighten the shadows in this photo, can you bring up that tool? And all you get is just that tool in a window. And you can do the work. You don't care that it's part of the Photoshop suite. you're going to be fine with doing that. So if we kind of pull at that thread, as things kind of become more decentralized, the idea of like what a business is or what a government is kind of starts to get blurry. it's tempting to think that, yeah, that's a long way off, but then it's also, we're seeing how quickly this technology is developing, which kind of makes me think maybe it's going to be sooner rather than later. And I wonder how disruptive that might be unfold.

The way most of us are, or most press discussions around AI have been so far is you look at an existing workflow, you think about the things that the machine could do, you take out those, you take out the human processes there, you drop in the machine, you leave the workflow the same. get our heads around because we already imagined the workflow. It's easier to implement because you don't have to mess with the workflow. So you only have to mess with a little piece of the overall production process for the organization. And you do what you were doing, but a little bit better. This is what we call point solutions. It's not really fundamental to the strategy and no one really cares. So yeah, you say 5% on something that's great, good for you, that's not going to get you promoted too much, that's you doing your job. Where the big opportunity lies is in the hard part, which is the hard part you were just describing, which is thinking through how does this technology, the amazing things that it can do, has it allowed us to do things completely differently? operates to deliver a new kind of value to our customers, to reframe and rethink the way our supply chain works, to change the way our HR operates, whatever it might be, to totally transform what we do. Well, that's what we call a system solution. That's where that's what's exciting. But that's where the disruption happens. When you have disruption, you have people who lose. And when you have people who lose, So a lot of organizations are going to see the potential say you know what if we had better predictions We could do all sorts of things differently. That'd be amazing We can take advantage of writing at scale coding at scale whatever it might be But then some vested interest within the company will say but wait that that reduces the power I have within the organization That makes my job less important my department less important and they'll resist and so we different kind of organization or totally different kind of regulatory environment, etc. Where that plays out is if there's enough of that resistance, but the potential for the technology is big enough, then we're going to have startups or maybe nonprofits or others try to figure out how to get around the old established way and create new opportunities. about disruption, right? We think about disruption in the tech world as this great thing because it leads to technological advance and it leads to economic growth, but it also is a threat to existing incumbent organizations.

Yeah, I mean, you said, you know, there's, there's people that essentially benefit from the status quo and, and the status quo economically is they have resources typically, right? They have, you know, they're benefiting, therefore they have resources. preserve the status quo. What are some of the tools you think in this particular case, given that it's very hard to get your hands around, looking at a Google that didn't even see their search so imminently being disrupted or at least potentially being disrupted. you know, their new search engine. And I think most of us would have thought that Google search still had search way in the bag, right? No threat on the horizon. And then bam. So what are some of the tools you think that folks will use that, you know, let's say part of the status quo club to try to contain this?

As a whole, okay. So if you're a large dominant incumbent in an industry, you can use your other products that you dominate to force people to stick with whatever you were doing before. So that's an antitrust concept called tying. is at risk of disruption, you make sure that all of your customers, you force to the extent possible your customers to keep using that because if they want to keep using the other aspects of your business that are not disrupted. So that's piece one. Resistance can also come from within the organization. people who've sort of run it historically. So one of the examples we talk about in our new book, Power and Predictions, is the insurance industry. And the insurance industry, underwriting is at the center of the insurance industry. So the power center in insurance, and most insurance companies, is the underwriting department figuring out how to price risk. Because that's what they're in the business of doing, right, the insurance company. What they do is they price risk. And so underwriting is where the power center center often the most talented people are there, etc. Now, insurance companies often have a mission that's not really about pricing risk though. It's about, you know, say, peace of mind against catastrophic loss or something like that. That's not about pricing risk. That's not really the service they sell. The service they sell is if something catastrophic happens, we'll give you some more. Now, so I think about, you know, in home insurance, if something catastrophic happens to your home, they'll give you some money and they'll say, that's giving you peace of mind against catastrophic loss. Okay. But that was true 30 years ago, but with better sensors and satellite images and all sorts of other things, they can do a lot more than that. They don't just have to, you know, price the risk, you know, predict the overall house, but they can predict it hazard by hazard or sub-hazard by sub-hazard. have that the reason your house has expensive insurance is because your wiring creates a real risk of electrical fire. And the reason someone else's house is at risk of insurance is because their pipes are at risk of leaking. And so that's underwriting. Now that creates a different business opportunity. The different business opportunities instead of just pricing that risk they can help you reduce it. They can say you know what your risk is about leaky pipes. And in your home to detect when your pipes are about to leak. And if your pipes are about to leak or they start leaking a little bit, we'll just shut off your water and we'll send a plumber right away to fix that so that you don't have that insurance. They're reducing risk, they're not just pricing it. Now, why is that complicated? Because once we get to risk reduction and not risk pricing, it's not so much about the underwriting side of the business anymore. We're going to have to figure out how to change customer behavior and how to convince them that this new way of doing insurance makes sense. So it's about behavior change, not just pricing. Behavior change is often since in marketing, but marketing hasn't been a power center in most insurance companies historically. And so it becomes complicated. And that means within a company, even though they may all see this huge opportunity, There's a bunch of organizational challenges. Underwriting might resist because it's giving away their role at the center of the organization. Their marketing team may not be designed for this kind of thing. This requires really creative, the best marketers in the world. The insurance company didn't have any of that in the past, and so maybe they don't have that. And then you're going to have to combine that with some aspect of how claims work and a bunch of other things. So when we think through these challenges on resistance to change, some of them are, okay, well, if you want to use that product, you also have to use this one, and it can create resistance. I've been running the company in the past and I'm not going to change that. And there, to the extent it's a competitive industry, maybe a startup can come along. But that's tricky. Startups have to be trusted in those kinds of industries like insurance. And there's also all sorts of regulatory barriers to making that happen too. And the insurance companies are going to be lobbying the regulators to say, well, you know what, this is not a writing business.

What do you think about a company like Lemonade then who have come in technology first and caused some disruption and they have grown, right? I think it started as rental insurance and now they have Pet and Auto and they've moved into the European market. They might not be a direct threat now, but I wonder too, if like some of these old-guard insurance companies, they probably have a lot of customer base that's almost like grandfathered in. They're just used to using these crappy digital interfaces and tools. resistance within a big company. If you have Lemonade offering like this really streamlined, almost personalized digital experience and then you have you know younger customers coming up that are going to be drawn to that more than they will be to like downloading an app or you know that sort of thing.

100% I agree. So I think Lemonade is a super exciting company. And whether it's them or a competitor there's or another startup, they're leveraging what prediction technology can do and what digitization generally can do in order to deliver a better kind of insurance product. And so exactly what the right. This is the economist in me very much. I don't know what particular company is gonna figure this out and do it right. Maybe it'll be Lemonade, maybe it'll be somebody else. But there is clearly an opportunity here and the kinds of services and product Lemonade is offering are the kinds of services and products that we can expect from companies that leverage AI the existing power structures that there are.

Yeah. When they're coming at it from like that technology first mindset, it's almost like they decided first that we're going to be, we're going to have technology at the core and then we'll figure out what we want to sell.

So yeah, so is there an angle here where it's that just companies just need to disrupt themselves, looking at Microsoft investing so much into open AI versus doing it in-house? I wouldn't call it a new model. Is there a popularizing this concept of organization, investing in their own disruption.

Um It can be. But every company can't be sort of trying to disrupt themselves because, you know, their old business models have value too. And where this gets really challenging is, look, I'm pretty confident that the changes we've seen in AI are going to come for most industries and most companies. I'm not confident whether that's going to be within one, three, five, 10, 20 or 50 years. And so if you're focused on, if you're spending 10% of your

your revenue or some real number trying to disrupt yourself. And you happen to be in an industry where the disruption is coming 20 to 50 years from now. That's not going to be so good. And so the trade-off here, this is going back to Christensen and my co-author, Joshua Gans work on disruption, is a real trade-off, which is, yes, you recognize what new technology hedging against that risk, building Skunkworks, trying to effectively disrupt yourself as you put it. But for many organizations, it's not those investments are going to fail. Oh, I should say they can fail in two ways. So one way I've described those kinds of investments failing is they thought something was going to come in a year and it didn't come for 20. But there's related to what we were just talking about with respect to lemonade. So yes, there's exciting stuff happening in insurance, and there's likely disruption on the way in the insurance industry. Which particular bet is the right one is really hard to say. And any company can only make one or two of those disrupt themselves bets, where the set of people who aren't in the company, the entire VC industry, and all their competitors And so the the risk of saying, okay, you know, I see the change coming and try to start to start myself is Even if you succeed even if that change is coming You might not make the right bet and so there's Often reasons for companies that see the challenge is coming out and they see that they're coming soon. They still almost rationally maybe even rationally say course or my profits while I can and when things start to go sour try to buy my way out of it.

it seems like a lot of them are maybe in this world of kind of your point to can you iterate? I guess this is more of a question. Point solution. Let's maybe think of point solution as incremental change. incremental change at a rapid rate that would ultimately evolve into systemic change.

Under particular circumstances, sure. In one of those particular circumstances, you need patient leadership. And ideally, you need a vision about where you want to go. So where we've seen a lot of this, to the extent that we've seen success through technological change step by step, it's the company or the company leadership 10, 20 years from now, and then they can sort of map out all the little steps needed to get there. And what I mean by that, ultimately every system solution has a point solution at its center. It's just a point solution in a different system. And often where we see success in moving to that system solution is someone figured out how to, the point solution that fits both in the old system and the new system, they old system in order to prep themselves to really get the, you know, prep themselves for that big change coming down the road.

Yeah, well, there's this issue too, I guess, where like a lot of point solutions are you're on someone else's development cycle. So what we talk about in our book a lot is like setting yourself up so that you own your own production cycle, which is again, like a monumental risk. And if you have businesses that are hesitant to this change for a whole variety of reasons, the prospect of making a big financial investment where you know that your first attempt is going to fail, possibly somewhat But that's the only way you're going to be able to build on experience and start kind of building your own automations and creating an ecosystem where you can be doing that quickly and regularly. That poses a whole set of unique challenges as well.

to me.

delivery.

As this technology rolls out, it's hard to predict exactly the demand that's going to ensue for people who understand and have experience applying it. In terms of the job market, for those who will gain experience over the next, let's or already have a few years of experience under their belt. How much do you believe that the expertise is going to throttle back the rate of change? Like, do you think that's gonna be one of the gating factors or that people will just accelerate their learning?

Tchau. Yeah, I actually think I'm very hopeful that we're going to see the opposite with these technologies, particularly with generative AI. What do I mean by that? So, for the past 50 years, technological change has been what's been called skill biased. By skill biased, I mean that the people who are relatively well educated and who are particularly from computing and from the internet. There's reasons to be hopeful that generative AI is going to be the opposite. Which a lot of the things that generative AI does well are the things that the people at the top of the input and distribution do well. Whether it's diagnosis or writing or coding, those are all things that for well-educated people at the top fear, the worry is, oh, that's going to just make it even harder, and you have to be even more skilled in order to make a contribution. That's possible. But another possibility that I'm using the word hopeful as opposed to confident on purpose, but another possibility is that this is going to upscale everybody. So I already talked about writing and everybody being able to write, but if the barriers to of an undergraduate degree to be able to do some pretty simple things, but useful things. That's going to be great for massive amounts of the population, leading to where skill is maybe the very top skill still a barrier if you're the one actually building the machines. But for doing most uh, you know, lever your skills in order to make a good living. I'm, I'm hopeful that this is going to reduce those barriers, not increase

That makes sense. You said that you triggered something for me, which was, again, back to your book on incremental versus systemic change. And being a coder myself for most of my career, I look at how we're using Codex with open AI what I used to call disposable code, but one-time use applications, right? Or ephemeral applications is probably a better way to put it. This is where an application is developed just for a single transaction and then, and gone. And so we see that people are using GPT to write code so they can put it in their app so they can go through their development process upload to the App Store and add a new version of their software. And so there's a copy, paste, test, blah, blah, blah. But then you look at being able to write code in a second, you start to get down to, like, we can create an application that's unique and on the fly. or software programs being created for one-time use. So it would be this snapshot where I created a report in a moment and an application that didn't persist beyond that conversation. And it has some phenomenal implications. I think so many that I can't even sort of think through the implications of that. given much thought to that.

Yes, sorry, here's how to sort of figure out where it fits and the way we've been thinking about this. So one of the big opportunities with AI is around personalization. Right, that we can. So we've had a sense of this for a few years that what prediction technology does, it allows personalized recommendations which can allow personalized decisions, personalized actions.

No.

is that we don't have to do the same thing all the time. We don't have to follow our standard operating procedures and just do what's on average best. We can look at any particular situation, figure out what the right code is, what the real need is and build around that. And that's an incredible opportunity. It's also quite disruptive because a lot of what companies do, a lot of the secret sauce they have

And if you're going to disrupt standard operating procedures, that's going to create problems for incumbents. It's been great talking to you.

Yeah, yeah, it's been fantastic. Appreciate you joining us.

Yeah, thanks, Avi. We appreciate it. Yeah, definitely.