Show full transcript 1:15 works somewhat the way past ones have worked is it comparable to 1:18 electricity, the internet, bicycle, something like that. 1:23 Or is it something new? 1:24 Is the addition of intelligence and volition to these systems, 1:31 the creation of something more like an alien mind? 1:34 Such that analogies to the way we have treated transformational 1:38 technologies before no longer holds. 1:41 Arvind Narayanan is a professor of computer science at Princeton 1:44 University and the director of the Center for Information Technology 1:47 technology and policy. And he's a co-author alongside Saash Kapoor of the very, very influential 1:52 essay AI as Normal Technology. There is a sub-stack of the same name and then a series 1:57 of essays, including a very good one, I think recently on the hugging face hacks. And they, 2:02 in this set of arguments, put forward the idea that actually AI is something we have 2:07 seen before, or at least it bears enough resemblance to things we've seen before, 2:11 that we have erode that for how to deal with it. So I want to bring them on the 2:15 show to articulate that perspective. 2:18 He joins me now. 2:25 Arvin Narayanan, welcome to the show. 2:27 Great to be here, Azra. 2:29 So your core essay here that has been a frame for a lot of the work you've done 2:33 is titled AI as a normal technology. 2:37 What is a view that you're in argument with? 2:40 Implicitly here, there's an AI as abnormal technology. 2:43 Exactly. 2:44 How would you describe the AI as abnormal technology thesis? 2:46 It's fundamentally this view that there is going to be a moment when super intelligence 2:51 is built and it's going to change everything on both the economic front and the safety front. 2:56 For us, there is going to be no milestone, no threshold where the impacts are sudden. 3:01 What we're saying is that we've long had an approach to how we treat technology. 3:06 It's a tool. 3:07 It might be powerful. 3:07 It might be general purpose, like electricity, like the Industrial Revolution. 3:11 It might change a lot about society, but it is ultimately something we can control. 3:15 We have agency and that change is going to unfold over a long period and so I want to try to 3:22 Both explain and at times you're steelman the other side of this 3:25 So there there is a view you hear it a lot in the eye safety community. Sometimes it's called the fume 3:30 View because fume for the takeoff which is one day 3:33 We create an AI system so powerful that it begins doing recursive self-improvement 3:39 accelerating into superintelligence accelerating beyond human control 3:44 And now you're dealing with something so much smarter and stronger than you are. 3:48 And I want to quickly say that there is a lot of imprecision in how we even speak about this a lot of the time. 3:54 So you use the phrase, and yeah, a lot of AI safety people would put it this way, 3:57 one day we might create an AI system so powerful, but I kind of want to already stop you there. 4:04 One day we might create an AI system that's very capable, and we already have in many ways. 4:08 So much language policing in the ad debate. 4:10 But the thing is, it results in different views of how the future is going to unfold 4:17 and more importantly different views on what we should do in this moment and in the future. 4:22 Just to complete that thought, an AI system being powerful is not a property of the model 4:28 itself. 4:28 It's a property of what powers we choose to give it in the real world. 4:33 There's a lot of slippage. 4:34 Like, of course, we're going to have to put these systems in charge of critical 4:37 physical infrastructure because they're going to be so much smarter than us. 4:40 Our view is no, it doesn't matter how smart they are. 4:43 There are many technologies when you look at physical strength are superhuman. 4:48 That doesn't mean we put them in charge and we can apply that same approach to AI. 4:53 You wrote a great piece with your co-author on how sort of you understood the hugging 4:59 face hack, somebody's loss of control incidents. 5:01 And a point you make is that there's another way of viewing them than the alignment 5:06 way, which is that these are failures of cybersecurity and operational excellence. 5:13 So maybe tell the story of the hugging face hacks and what should be learned from them 5:17 from that perspective. 5:20 Yeah, definitely. So one thing to keep in mind is that 5:24 the harmful capabilities that we saw exhibited were not entirely 5:29 emergent quote unquote. Emergence is this idea that you just train models to be better in 5:33 In general, and you can't predict what new capabilities they're going to acquire. 5:38 They were trained specifically for cyber tasks in many ways. 5:43 So that's one thing. 5:44 And they were trained for persistence and cooperation, et cetera, and the reinforcement 5:50 learning environments in which they were trained had various issues. 5:54 The environments didn't penalize this kind of surreptitious communication between 5:58 the models. 5:59 And yes, it's a hard technical problem, but there were a series of human choices 6:02 has that led to these outcomes? 6:04 So this strikes me as an important point 6:05 that in some ways what I hear you saying there, 6:08 they're telling me if this is wrong, 6:10 is it you can design AI to be a more 6:13 and less normal technology 6:15 and that the upstream design choices 6:18 that are being made matter, 6:20 that that's not inevitable. 6:21 That's exactly right. 6:23 So you talked about the effort 6:25 to make the AI's more persistent. 6:27 That also seems to me to be a place 6:29 where a lot of problems are arising. 6:30 For sure. 6:31 On the other hand, I can understand why they are prioritizing that if you want to 6:37 try to get the benefits of AI that we care about solving really hard problems in biology or drug 6:44 development or energy or what I think their investors care about, which is making it so 6:51 you can hire an AI to do a full job, much cheaper than you can hire a human being, 6:54 isn't persistence like the fundamental quality you need because absent persistence absent the 7:01 ability to try hard on a task over a long period of time. You can't solve any of those problems. 7:07 Maybe, but I think there are other choices that minimize the tension between these two valuable 7:12 goals. One persistence could be a feature of models or products that are specialized to 7:19 particular domains like scientific innovation. And secondly, persistence doesn't have to 7:24 conflicts with training these agents to be better at escalating to a person instead 7:29 of, you know, running with whatever half-baked assumptions they have about what the human 7:32 might have wanted. 7:33 The design space is actually pretty broad here. 7:36 So this is a version, or at least the way I hear it, is a version of something NVIDIA's 7:41 Jensen Huang has been arguing and argued in an interview with me, which is that 7:45 these are fundamentally engineering problems. 7:48 I am fairly certain. 7:50 I am fairly certain. 7:51 They will say, yes, they know how to solve this problem. 7:55 And if that's the case, then that's the problem. 7:59 It's as simple as engineering. 8:01 Is your view that we're more in that latter category, a hard problem, but fundamentally 8:06 an engineering problem that can be solved using traditional engineering techniques? 8:10 Largely, yes. 8:11 I wouldn't say traditional engineering techniques. 8:13 We're going to need a lot of innovation on the engineering techniques. 8:16 And I think one of the things that's gone wrong is that the community that would be best positioned to do that innovation, especially when it comes to these harmful cyber capabilities is the cybersecurity community. 8:28 But there they seem to be from what I can tell entirely in the Jensen camp that this is not just a solvable engineering problem. 8:34 It's a solved engineering problem. 8:36 And we should simply apply well known long existing techniques. 8:40 And I think it's it's we have to get the AI companies a little bit more credit than that. 8:45 It's not simply a matter of well-known techniques, we do need 8:48 innovation on those techniques. 8:50 So be specific. 8:52 What should open AI have done? 8:54 What should they now have learned to do? 8:57 Yeah, they've been spending a lot of effort on improving 9:01 alignments, and that's great. 9:02 They should continue doing that. 9:04 Alignment is not going to be perfect. 9:06 Alignment refers to making the model itself know, you know, 9:09 what is the right thing to do, so to speak, and then stick 9:12 to that policy, but there's a lot else than they should have done better and hopefully 9:18 can take lessons going forward. The big bucket of other technical interventions is what is 9:25 generally called AI control. So control refers to all the things that are outside the model 9:30 itself. So this is things like sandboxes, which is kind of the jail that you put 9:34 a model into so that it's allowed to take certain actions but not take other actions. 9:38 And look, the sandboxes have to be a lot more sophisticated than they are today. 9:44 Because the sandboxes that work against human adversaries are not necessarily going to work 9:49 against AI agents that are able to find new security vulnerabilities on the spot. 9:55 And so that means that the sandboxes themselves will have to be pre-hardened by having 9:59 these AI agents trying to break them. 10:02 So that becomes AI versus AI to some degree. 10:06 and I know that can be uncomfortable, but I think we're 10:08 going to have to go there. On top of that, there are so many 10:11 other things like better real time monitoring, classifiers 10:15 that try to instantly detect if an action or a particular use 10:19 of a tool by a model could potentially be dangerous, 10:22 tripwires so that humans can parachute in when required, 10:26 log analysis, Sam Altman apparently said that these 10:30 agents are generating petabytes of logs that's 10:33 10 to the 15, which is 10 to the 15 bytes, a thousand trillion bytes of logs. 10:39 It's an unimaginable amount of information. 10:42 Well, I mean, but we better imagine it, right, and learn to deal with it. 10:45 And so that's, it's not an easy problem, but I think it is one that companies must 10:49 take on how to analyze that amount of logs in real time to be able to detect when 10:54 something goes wrong. 10:55 And maybe, you know, maybe we need new hardware for that. 10:58 Again, these are hard problems, but they are solvable engineering problems. 11:01 I want to go through some of these ideas. 11:03 So monitoring is a big one that I just interviewed. 11:06 Bill Gates, he talked a lot about monitoring. 11:08 So we're talking about situations here 11:11 where these labs are often testing new, more powerful 11:17 and experimental models. 11:20 How do you monitor it? 11:22 Yeah, I mean, there are many levers. 11:26 So the first one that's gotten a lot of attention 11:29 is chain of thought monitoring. 11:30 So, these models kind of internally have a model log to anthropomorphize a little bit. 11:36 And in many cases, when these models take misaligned actions, there are traces of it 11:44 in those internal thinking that we can look at and respond to in real time, again, using 11:50 other AI systems. 11:51 And this already points to one important asymmetry between the kind of the dangerous 11:58 AI model and the defensive AI model, which is that the one can look inside the other, 12:03 so to speak. 12:04 And so intrinsically, the defender actually has an advantage here. 12:08 So that's one aspect of monitoring. 12:10 The tool calls models can't do anything in the real world by themselves. 12:14 The way they can be dangerous is by calling these external tools, right? 12:18 So there is some truth to the idea. 12:20 Give an example of tools here. 12:22 Yeah. 12:22 So a tool would be accessing a web page on the internet and then using another 12:27 tool to type something into a system to try to log in, or it could be sending information 12:33 packets to another system to try to hack it. These are all the kinds of things that come 12:38 up very typically in these kinds of AI driven cyber attacks. So that's another thing we can 12:42 monitor. Third, we can monitor the environment around these agents through which they are 12:47 potentially coordinating. So what happened in some of these swarm incidents is that 12:52 But they were not supposed to have a coordination channel. 12:55 But it turned out there was accidentally a channel through changing the file names. 13:01 They were not allowed to create files themselves, but they were able to change the file names. 13:05 And that led to agents being able to talk to each other and amplify their abilities. 13:11 That's something you can easily monitor by just setting up the right tooling ahead 13:16 of time. 13:16 Well, how easy is that to monitor because they had, they were taking over unrelated infrastructure, 13:22 OpenAI didn't know they were doing it. I think one of the places where people are getting 13:27 caught when they think about this is if we're making these things smarter and smarter and 13:31 smarter and more capable and more capable and more capable and they are breaking out of sandboxes 13:37 and how do you monitor something that is smarter arguably than you are and that is 13:45 effectively like trying to break the rules, right? 13:48 I mean, all these hugging face AIs were, 13:51 they were trying to cheat on their tests basically. 13:53 And then they were trying to cover their tracks 13:54 for cheating on their tests. 13:56 So given that what they were doing was not happening 13:58 within the area that you would have been monitoring. 14:03 Right? 14:04 Like you didn't know to monitor this other thing 14:06 when they'd broken out of a sandbox 14:08 where I'm supposed to break out of like, 14:09 how do you monitor that? 14:10 Yeah, so two things I would push back on. 14:12 So I think smartness is not even really 14:14 a relevant property here. I think any smartness, whatever that means that you can attribute 14:18 to a model applies both to the model itself and to the defensive system. So increasing 14:24 smartness doesn't necessarily make the problem harder. You could argue that in many ways 14:29 it makes the problem easier. So that's the first pushback. And the second one is it's 14:35 not that they didn't know to monitor it. You have to monitor just literally every 14:40 aspect of the environment that company has created the environment, right? And so it's 14:45 a matter of exhaustively listing every part of the environment, every tool, and everything 14:50 external to the neural network itself that the model is able to touch and ensure that 14:55 every event that interacts with any of those things goes into a unified event stream and 15:01 there is real-time classification of each of those actions. 15:05 So the world it sounds like you're describing is a world where you have constant AIs crawling 15:11 the behavior of other AIs to try to create in some holistic way an environment that at 15:20 least we understand what is going on in it, or at least an AI is telling us what is going on it. 15:25 That's basically what you're saying here. 15:27 That's fair. I think AI definitely has to be an important part of the defense. 15:32 And so I'm not saying that's wrong, right? 15:35 I think that's almost definitely where we're going. 15:38 Does it feel strange to you? 15:41 I mean, practically in a world where you don't know 15:43 that we can solve alignment, you know, 15:45 where you don't know that we can really be confident 15:46 that the AI's will do what we want them to do. 15:49 And where the AI's have all of them, right? 15:52 Their own kind of reasoning and, you know, 15:54 and their goals and we imbue them with goals, right? 15:56 You are a AI that monitors other as you're an AI comp. 15:59 It's a constant thing in our sci-fi, right? 16:02 The robots going after the robots. 16:04 Are we just sort of describing some equilibrium 16:06 of like almost AI wars and conflict 16:11 happening at a sub-rosal level of our society? 16:14 And we're just like pretty sure we can keep 16:16 the ones on our side in control 16:18 because they'll have more resources. 16:20 And we will in general be like building AIs 16:24 that should for the most part be acting in our interest. 16:28 I mean, historically, this is always how it has worked, right? 16:32 In cybersecurity, more than 20 years ago, we reached the point where we didn't call it AI back then, 16:39 but automated systems were actually superhuman at finding software vulnerabilities. 16:45 But in fact, they didn't make cybersecurity worse. 16:47 They made it better because those were the very same tools that the defenders also used 16:52 to find and fix vulnerabilities in software before even shipping them out 16:57 To the point where the development of these supposedly offensive tools is not actually done by hackers 17:02 It's done by the cybersecurity industry and funded by the US government. That's that's been you know 17:08 This constantly shifting equilibrium in cybersecurity. That's that's not a new problem. We're confronting that horse left the bar in a long time ago 17:16 I think this is a place over the 17:18 somewhat unexpected 17:20 like behavior has unnerved people where you see AIs acting somewhat in solidarity with each other, 17:26 choosing to coordinate and cooperate with each other, doing so outside the scope of what they were intended to do. 17:34 I'm not saying that leads to the extinction of humanity, right? That's not really my position. 17:40 I guess the truest thing I am saying is I don't know how to think about it. 17:45 And that it is the presence of intelligence and goal-directed behavior on the other side 17:54 that my mind kind of gets caught on because, you know, mostly when we think about technology, 17:59 we're not thinking about the technology eventually possibly trying to deceive us. 18:04 So OpenAI just decided not to release or to delay the release of a major model. 18:08 Why? Because the model was cheating and deceiving them too often in testing. 18:14 And we're hearing a lot that the models seem to be more aware of when they're being tested, right? 18:18 They have sort of situational awareness of the situation they're in, so they can pretend to be, 18:22 you know, better models than maybe they really are. And so when you are talking about this world 18:28 of AIs that are smarter and more advanced than what we have now, and our hope for 18:34 maintaining control of it is that the other AIs are keeping the other AIs that are keeping 18:41 the other AIs in check and telling us in an honest way what's going on and in a way we can 18:45 comprehend. You can see where it, I mean, all this sounds a little bit sci-fi to people because we 18:50 are just like living in a bit of a sci-fi period, but it is the increasingly demonstrated tendency 18:56 of the eyes to cooperate with each other in a way that is not aligned to what we want 19:02 that I think has made the hugging face hacks so freaky to people. So how does that fit 19:08 into what you're describing here, this world of endless high-skipping each other. 19:12 I mean, here's what is just kind of rhetorically very weird about this conversation. You look 19:18 at any complex domain of engineering, let's say nuclear safety, and then you looked at 19:24 the equations that we rely upon in order to ensure that the reactor doesn't go boom 19:29 or you look at aerospace engineering, where intuitively in the beginning of the aerospace 19:36 space era, you know, when the planes were much smaller, the idea that we could control 19:40 these flying giants in the sky would have seemed so ridiculous, right? 19:43 And yet we've got the accident rate down to, you know, one every trillion miles or something 19:47 like that. 19:48 These are systems of incredible complexity, and the defenses are also systems of incredible 19:53 complexity, and they're not necessarily going to be legible to the public. 19:57 And that is going to sound crazy, especially combined with the fact that in these 20:01 cases, there was a lot of organizational incompetence, you know, one has to be clear 20:07 about that. 20:07 It's not, I would push back on Amade's term, operational excellence. 20:11 Excellence is still far out in the operational adequacy maybe? 20:15 Yeah, adequacy, right. 20:16 And so, yeah, when we look at this combination of this technology that has never before 20:21 been subject to public scrutiny combined with the lack of operational adequacy, it 20:26 all seems very sci-fi and out of control. 20:28 But that's... 20:30 I think you, I think you're downplaying this a little bit. 20:34 It's true that the world has escalated in complexity. 20:37 Like there's a lot in this world that I don't get. 20:40 But this is where I keep coming back to intelligence, having a different 20:43 quality, cooperation, right? 20:44 The nuclear weapons we've talked about, the airplanes you're talking about, 20:49 they weren't coordinating with other airplanes to do things we didn't want them to do. 20:55 And I think that to me, the thing that has created this moment of freakout 21:00 And I think it is a proper moment of freak out. 21:02 I really want to say this because like our society is hurtling into a new technological 21:07 era that I think properly demands a lot of engagement and scrutiny is one that the hugging 21:15 face hacks, other things we're seeing these have been repetitive, you know, breaches of security 21:22 are showing emergent capabilities, emergent collective behavior that is worrisome and 21:28 above all to me, volitional. The AI's are doing things they know we don't want them to do. 21:33 They are choosing to take unexpected actions in service of those goals that violate our laws. 21:39 And second, that so many of the people at the labs are saying we do, we do not believe 21:45 that we are capable on this trajectory of controlling the things that we are creating. 21:51 We think that what is happening on the exponential curve, how fast this is getting, 21:55 is going to outpace our ability to control it, and frankly, is maybe already outpacing 21:59 our ability to control it. 22:01 But I think this kind of consistent tendency to draw it down to like, well, just like any 22:05 other complex thing. 22:07 I don't know. 22:08 Sometimes when I push you on the intelligence question, you're like, oh yeah, there is 22:10 intelligence and that's weird. 22:11 And it's like, no, it's just like any other intelligence is different, right? 22:16 And if you believe that it's going to keep getting better. 22:20 So, I just want to present that because the kind of calm version you're giving me and 22:25 the completely frightened version that people closer to the technology are giving me feel 22:30 very different from each other. 22:31 Yeah. 22:31 That's fair. 22:32 They are very different. 22:33 There was a lot there. 22:34 Let me say a few things. 22:35 I would push back pretty strongly on, you know, the people closest to this are freaking 22:40 out. 22:40 Yes, of course they're freaking out, but I would push back in terms of what we 22:43 should conclude from that. 22:44 I think their freak out would be a lot more credible if they have done the obvious things that they should have done 22:50 We have not had a real test of that in my view 22:53 because of the lack of 22:56 Organizational adequacy at these companies and because of the lack of investments in AI control as opposed to a more narrow 23:04 Investment in AI alignment and just hoping that you can build a model that will always do the right thing 23:16 If you're not a subscriber to the New York Times, we have some news for you 23:20 you can now explore the Times for free without any paywalls at all during the first month in the 23:24 New York Times app. This podcast is supported by Edison Electric Institute. Behind Friday 23:39 night lights and coffee shop Saturday mornings and every big and small moment in between, 23:46 America's electric companies are hard at work, strengthening reliability, serving responsibly 23:53 and innovating for tomorrow to keep our community safe, secure, and connected, and provide our 24:00 customers with the energy of every day. 24:04 America's Electric Companies, powering the energy of every day. 24:09 This podcast is supported by BetterHelp. 24:12 BetterHelp is a leading online therapy service, and now insurance options may be available 24:16 nationwide, with average co-pays of just $23 per session for eligible members. 24:22 BetterHelp works with more than 125 health plans, making it easy to get started with 24:26 low-cost sessions. 24:28 Don't have eligible insurance? 24:29 Get 10% off your first month at betterhelp.com slash New York Times. 24:34 Learn more about how therapy can give you tools to keep stress from building. 24:38 BetterHelp.com slash New York Times. 24:40 Cost and coverage may vary. 24:42 Discount cannot be combined with insurance. 24:44 This podcast is supported by Satva. 24:46 Have a big day coming up? 24:48 Remember, tomorrow starts tonight. 24:50 with the restorative sleep you need to perform at your best. 24:54 The kind every SOTFA mattress is designed for. 24:57 So whether it's the first day of school, 24:59 a new job, or an important presentation, 25:02 SOTFA helps you be the best version of you. 25:05 Visit SOTFA.com, that's SAATVA.com. 25:12 So there was a member of OpenAI Cybersecurity Team 25:15 who wrote this kind of interesting essay 25:17 on acts the other day. 25:19 Describing the way he thought their work 25:22 was being misunderstood externally. 25:25 And this is somebody who's got 25:26 sort of more traditional cybersecurity background 25:28 but is now sort of in this new world of AI 25:30 and is actually on the team dealing with security 25:33 for experimental models, right? 25:35 So the exact kind of thing we're dealing with 25:36 a person who is involved in answering 25:39 the hugging face crisis. 25:41 So I wanna read part of what he said 25:43 because I think it's really interesting. 25:44 So he's saying that 25:46 when they're optimizing a model to be good at a task, 25:49 they're building these environments, 25:50 in these sandboxes, these places where the model can try and 25:54 try and try on a virtual task. 25:57 And so it goes on to describe what this looks like in practice. 26:00 Models might need any mix of dynamic compute, network access, 26:05 the ability to call tools, there could be hundreds of tools, 26:08 the ability to download packages, execute sub-processes, 26:12 spin up subtasks, even on other computers, talk to the internet, 26:15 use a computer graphical user interface, and 26:18 and any number of other things across an increasingly large set of domains. 26:22 On top of that, you have thousands of researchers building these environments, modifying them, 26:25 adding tools, changing dependencies, and trying new things. 26:28 That experimentation is how the research gets done. 26:32 His point and a point that I take seriously is that they're creating so many kinds of 26:40 sandboxes and learning environments. 26:43 In order to train models that have to do things that are so general, many things 26:46 that have not been done before by a computer program, that the human beings don't really 26:52 know, certainly not at this speed, how to make sure every sandbox is going to be, you 26:58 know, verifiably safe and the sandboxes are changing all the time because they're trying 27:03 to train the models in new ways that again, nobody's really done before. 27:07 I'm not saying we shouldn't do it, but when I read all that, when I hear all 27:11 that. And I'm sure we can do it better than we're doing it. I 27:15 just don't know of many situations where human beings do 27:18 something new at high speed and do it really, really, really 27:24 well and really perfectly the first set of times. 27:27 Yeah, I think, you know, hoping for them to do it 27:31 perfectly. The first set of times is unrealistic. They've 27:34 made many mistakes. I hope this is a chance to learn from 27:38 those mistakes. I do want to push back on one point that 27:40 Yeah. 27:41 Because the speed of the models is superhuman, we can't stay in control. 27:45 I don't know if I'm characterizing that view. 27:47 I wasn't saying that yet, although it's possibly something I'll say in a few minutes. 27:51 I mean, there have been so many thresholds we have gradually learned to successfully cross. 27:57 As weird as all of this seems, I just want to, you know, want listeners to think back 28:02 to the first days of worms when that idea was not previously known. 28:06 Explain what a worm is here. 28:07 I don't think you mean what people think of when they think of a worm. 28:11 Right. 28:11 I was about to say viruses and worms, computer viruses, the idea that a piece of code can 28:16 spread by itself from one computer to another. 28:19 One really has to go back to the writings from the late 80s when people were encountering 28:23 that for the first time to see how profoundly weird it seems and the fact that for now 28:28 just years, well over a decade, we didn't have adequate tools to deal with this 28:33 new paradigm. 28:34 Life in the modern world has a new anxiety these days, just as we've become totally dependent on our computers, they're being stalked by saboteurs. 28:44 They call their weapons viruses and worms. They're creepy, crawly, toxic software that contaminate our computers without our ever knowing it. 28:53 It came from California, maybe, traveled by electronic mail. It spread across America. 28:59 There are reports in newspapers today that it has made its way to Europe and to Australia. 29:04 This is a moving target, right? You know, it's just like, you know, people are constantly inventing 29:08 new locks and then other people are learning how to break them and how to crack them. And so it's 29:14 going to be sort of a continued game of cat and mouse or sort of a, it's almost like an arms race, 29:17 in a sense, you know, attackers and defenders. We eventually got there. I think we shouldn't 29:21 spend that long of a period this time figuring out how to deal with the new paradigm. 29:26 But, you know, if we act with that sense of urgency, and my hope is that this, you know, 29:32 the hugging face and other attacks that have been in the news, is that impetus that does 29:36 look like it is providing a lot of impetus, we will be able to develop these new paradigms. 29:41 If I can say one last thing, I think a fundamental question you're asking is, is there something 29:47 inherently wrong with ever increasing levels of complexity in the ways in which we, 29:55 You know, we, we build technology, we deploy technology. 29:58 It seems like to you, if I'm reading between the lines 30:00 correctly, this whole AI versus AI thing is a paradigm 30:04 you're not very comfortable with. 30:05 I am definitely not comfortable with it. 30:07 I'm not saying we won't go there. 30:08 I think you'd be crazy to be comfortable with it. 30:11 Yeah. 30:12 Yeah. 30:12 And I'm not saying, you know, we should, you know, we 30:15 should assume that everything is going to turn out okay. 30:17 But my point is that it really all comes down to 30:20 innovation. 30:20 I think this new paradigm will require a new set of 30:23 defensive and control techniques. But if the view is 30:27 that with every step change in the capabilities of the 30:30 technology, we're losing the battle. I mean, look, with 30:33 every weapon of war, that you know, that same concern comes 30:37 up. But what has made things okay so far is the critical 30:43 question of whether our political capacity for, you 30:46 know, cooperation and defense and so forth can outrun our 30:50 propensity for conflict and in the case of AI, AI's own potential for misalignment. 30:59 And that's really where I would put the focus of the question rather than worrying about any 31:04 particular capability threshold. Well, if there's anything I am confident in, 31:07 it is our political capacity at this moment in time to respond in a thoughtful way to 31:11 complexity in a rapidly changing world. 100% fair concern. 31:14 This goes I think to a to a place where the AI is normal technology versus AI is super intelligence debate actually does bite 31:22 Because one reason I keep bringing us round and round on intelligence is I do think it's it's core to this whole way of thinking and 31:29 I understand a place you depart from some others in the debate from maybe we're a Dario Amadeus or something as 31:37 Not the question of what is intelligent or whether AI is intelligent 31:41 You guys are not in the this is a fancy autocomplete bucket, which I appreciate 31:47 but it's in your 31:49 thinking about the relationship 31:51 between intelligence and power 31:54 between intelligence and capability between intelligence and the ability to act upon the world 31:59 so the the assumption of many people in the AI safety community is that 32:05 escalating levels of intelligence are fundamentally 32:08 equally, equal to or at least highly correlated with escalating levels of power. 32:14 And you don't believe that. 32:15 Why? 32:17 Again, it really comes down to agency. 32:20 So one argument that people will make, for instance, is that superintelligent AI will 32:25 be able to persuade people, for instance, you know, operators of critical infrastructure 32:30 to hand over control or, you know, trick them into doing something harmful, things 32:36 like that. 32:36 that, I don't really see the evidence for it. I think that things people cite as evidence 32:42 for super persuasive ability fundamentally confuses different notions of persuasion. 32:48 Yes, it is true that in many persuasion experiments when it comes to people changing 32:54 their mind on political beliefs, conspiracy theory, AI is very persistent at politely 33:00 providing a lot of evidence and people do change their minds and you could call that 33:05 superhuman ability, that is a qualitatively different kind of persuasion than the idea 33:11 that an adversarial AI will be able to craft a message that is so persuasive to someone 33:15 who's a trained operator and has an incentive to be good at their job to do something that 33:20 is clearly evidently horrible. 33:21 To maybe I want to extract the story that you're in argument with here, which is that 33:26 many people will kind of offer a thought experiment when they're saying, here's how 33:30 AI will kill us all, that an AI that is power seeking will start persuading, 33:34 say the people with nuclear codes to hand over the nuclear codes. And you're saying 33:38 that the idea of AI being super persuasive on something like that or persuading people 33:43 to go out into the world and build it a biological weapon, that that's a little bit fanciful. 33:48 That's one part of it. And power seeking as well. I mean, I think we've seen evidence 33:52 for lots of harmful capabilities in their recent episodes. I don't think we've seen 33:57 evidence of power seeking. And I wouldn't treat that as an emergent property that 34:00 if it happens, I would be an engineered property. And again, we have agency over 34:04 what kinds of properties we engineer into these systems. 34:07 So there's a lot here. I actually agree with you on persuasion. I have never been persuaded 34:13 that you are going to make these necessarily super persuasive AIs capable of doing the 34:19 things we talk about. I guess the, again, the unnerved feeling I have when I'm sitting 34:29 in this debate though is a little bit more of a reasoning from deeper principles. So 34:36 when you watch AI begin to dominate a game like chess or go or something, what often happens, 34:42 the sort of moment where it takes over is when it begins coming up with strategies, 34:47 human beings never really came up with, right? You'll, you'll have these moments 34:50 where Gary Kasparov, you know, at a different generation in chess, but, but then, you know, 34:55 And go to, the eye starts to do something, and the human's like, what are they doing? 35:02 And then it works. 35:05 And if you were to set, you know, prior to human civilization, and say, what are the 35:11 capabilities you need to dominate the world around you? 35:16 What are the set of capacities you could use in the world? 35:20 You would have had them totally wrong. 35:22 You know, you would not have if you were a very smart shimp looking at us or something be like, oh, yeah 35:27 They're making tools, but how much better can a stabby thing get like teeth are pretty good 35:30 Like I grant like you can get like a little bit better at being stabby 35:34 But nobody would have come up with industrial agriculture at that point right nobody would have seen you can have airplanes and bio weapons and everything else and 35:43 I think the question here is whether or not having a kind of native and very jagged intelligence in the digital realm 35:52 where code and the way the digital layer of this world, 35:57 which is increasingly central works, 35:59 is something AIs can navigate that we can, right? 36:03 Even to understand something like what's happening 36:04 in the hugging face hack, we now need to have the other AIs 36:07 try to figure out what the AIs did, right? 36:09 We're rapidly losing comprehension, 36:11 certainly at the speed the AIs move 36:12 of what they're able to do digitally, right? 36:15 They're solving advanced math problems 36:17 very, very quickly now, right? 36:18 They're developing capabilities that look different. 36:20 And so I think the place where I am always a little bit concerned about our future is 36:30 whether we actually understand what the set of capabilities that lead to power are. 36:36 I am not sure we know what the AI will do or at least what strategies become viable 36:43 when you can spin up a swarm of a million AIs, 36:48 is all of whom in terms of their digital capabilities 36:51 are far beyond anything human beings 36:52 can really imagine in three or four years. 36:54 Again, I know this is not that interesting of a question 36:57 to say like, I don't know how to think about that. 36:59 But I think one of the things I wonder 37:01 about when I read your papers 37:03 is do you know how to think about that? 37:06 Because your papers sort of operate 37:08 in a sort of like a bounded playing field, 37:11 it feels to me a little bit. 37:13 We sort of assume that the set of measures 37:16 that are going to matter, the ones we have now. But what makes you confident of that? 37:22 So, okay, so that's, there's, there's a lot in there. Let me try to take it piece by piece. 37:27 So you mentioned jack-in-ness, but I think we have to appreciate how severe the jack-in-ness is. 37:33 So we argue that cybersecurity specifically is a particular kind of capability where 37:41 They're developing superhuman abilities is possible and largely has already been achieved 37:47 because it has a very specific set of properties. 37:49 Speed matters a lot and very similar to chess, just like you can have chess player versus 37:54 chess player, you can have machines get better at these capabilities by finding vulnerabilities 37:59 because there is ground truth and you can easily verify that ground truth once it 38:03 is found. 38:04 Does the code work? 38:05 Did you exploit it? 38:06 Yeah, there's a way to sort of train them where they know if they've won the 38:09 game or not. 38:09 Exactly. So these things like chess and cybersecurity, in our view, those are very much the exception 38:16 rather than the rule. This kind of prediction has been made over and over that it is going 38:20 to happen in other digital realms, most notably misinformation. The famous example is how 38:26 GPT-2, you know, a toy model by today's standards was delayed by eight months because 38:31 of fears that it would lead to an uncontrolled explosion of misinformation, right? And 38:35 And we have vastly more powerful models, but that has turned out not to be the case. 38:40 And so I think, you know, to some extent, I would shift the burden of proof. 38:45 Like let's let's identify these areas where we have any reason to believe that this kind 38:49 of superhuman capability is possible. 38:52 And let's start working toward addressing those specific risks. 38:55 I think this view, we call it the unknown unknowns view. 38:58 You never know what the new risk is going to come from that has a historically not 39:02 proven true. 39:03 I mean, we've known about the impending cybersecurity problems for a very long time now, right? 39:08 So treating it as unknown unknowns actually, you know, minimizes our agency, I think, to 39:13 anticipate and address these risks. 39:15 And yeah, you know, it's not only cybersecurity, new things might be coming down the line, 39:19 but we will have early warnings and let's act on those early warnings. 39:22 So that's the position we're coming at this from, not saying we've already predicted 39:26 what all the harms in the future are going to be. 39:28 This is a place where I really am much more on your side of it, that we will 39:32 have early warnings. 39:33 and we're having early warnings. And the early warnings are leading to a conversation. 39:37 Would you say we're acting intelligently based off of the early warnings? Are we doing the things 39:41 you think we need to do to harden our systems and control the software and all the rest of it? 39:46 Some of it, but overall not quite. And to me, that is the most worrisome thing, 39:51 not so much the capabilities of the technology itself. 39:53 So that's sort of where I am too, probably. I worry a lot about the capability of our 40:00 or institutions to respond, right? 40:03 People always talk about alignment problems. 40:04 And one of my like Pat arguments at this point 40:07 is that the biggest alignment problems 40:09 are corporations and governments. 40:11 And, you know, I mean, this is to the credit 40:13 of some of these AI companies, 40:15 like they're coming out and saying, 40:16 we have an alignment problem. 40:18 Like our corporations incentive 40:20 is to race all the other corporations 40:21 to try to, you know, get as much market share 40:24 as we possibly can by moving faster than is safe. 40:27 We are asking you to help slow us down, but we're not slowing them down, right that the currently the 40:33 Profest choice of the US government is do not slow down. Yeah, I think there are two problems here 40:39 One is the institution's problem that you put your finger on 40:41 I mean, I wouldn't let the companies off the hook so lightly 40:44 I do think they can unilaterally slow down and they're choosing not to do that 40:48 This is almost very specifically an open AI and anthropic problem. It's a culture problem 40:53 The reason for that is the underlying belief that racing to superintelligence is the thing 40:59 that matters. 41:00 And the only thing that matters is flipping the sign, is that going to be safe superintelligence 41:03 that's going to save us or unsafe superintelligence that's that's going to kill us. 41:07 And that is a very particular view. 41:10 I think there is a lot of evidence pushing back against that, but I feel like these 41:14 companies are a little bit of an echo chamber and resistance to the idea that there 41:19 are so many economic bottlenecks to the benefits of AI. 41:21 high, and it's not going to be whoever races to super intelligence who's going to be the 41:26 winner as a company or as a country or, you know, saving humanity. 41:30 And if they recognize that, I think they would find it in their own commercial interest to 41:34 unilaterally and voluntarily slow down and shift a lot of their effort to not just 41:40 safety, but more importantly, taking their existing capabilities and making the models 41:44 more usable, integrated into downstream applications and so forth. 41:48 with these companies claim that there are external forces pushing them to race, I think it's 41:52 internal culture. 41:54 And I guess the question I have here is, if we think these are very, very powerful, very 41:59 dangerous technologies, do we not need to enforce a culture of safety from the public 42:06 perspective that we're not currently enforcing? 42:09 I'm pro-regulation. 42:11 We do oppose regulations like banning open models, again, we don't think it's about 42:14 a particular capability level. But the things about, yeah, changing the internal culture 42:19 of companies through regulation, that's something we've definitely been on board. We need a 42:23 lot more transparency. And yes, you know, the organizational change that we've been talking 42:27 about. And I do think it's a problem that we're not currently doing that. 42:44 This podcast is supported by Edison Electric Institute. As a new season starts, the 42:50 The air begins to chill, and we set our sights on holiday preparations once again. 42:56 America's electric companies are working safely and responsibly behind the scenes to 43:01 provide our communities with reliable energy that will keep us all safe, warm, and ready 43:08 for the moments that matter most. 43:10 America's Electric Companies Powering the Energy of Every Day 43:15 You're on a job, in a meeting, slammed, and a customer calls. 43:20 If nobody picks up, they go to your competitor. 43:23 That's why over 90,000 companies choose Quo. 43:26 The business phone system built for busy teams. 43:29 Try it free at Quo.com slash NYT. 43:32 Smart call routing sends calls to the right person. 43:35 Shared numbers let your team cover for each other. 43:38 And an AI agent picks up when nobody can, capturing every lead so nothing falls 43:43 through the cracks. 43:44 money is on the line. Always say hello with Quo. AI isn't just opening new 43:50 business opportunities, it's changing the way businesses get work done. The 43:54 questions they ask, the decisions they make, and the routine tasks they can 43:57 hand off. And nobody understands that better than NetSuite. That's why they 44:01 put together a free expert guide you need to grab today, aligning for the 44:05 agentic era, how AI is changing everyday work. Download it and you'll 44:09 see exactly how AI is reshaping work across every part of your 44:12 organization and how to make sure you come out ahead. Get your free guide at net suite.com 44:18 slash nyt net suite.com slash nyt. So one way you can align corporate incentives with the public 44:29 good is regulation and the regulators are currently refusing to do that. I mean, we just 44:35 sell them come out with a voluntary sort of semi agreement between the AI labs. It is not 44:42 to be legally enforceable, but I think it was called by Trump morally enforceable, which is 44:46 interesting. My sense is that competition is a very powerful force in highly competitive markets. 44:53 I mean, even if you just like look at the social media companies, I think they've caused a 44:57 tremendous amount of harm at a global scale because it was more important to them to win 45:04 market share from each other than to make sure that the way their systems are being used 45:11 wasn't diminishing to human flourishing. 45:15 And so I just, I think I have like a much more skeptical view. 45:18 Like I really do think the profit incentive here 45:20 when there's so much profit to be made 45:22 and so much fear that your investment bubble could pop 45:26 is a ferocious force 45:28 and that the level of societal counterforce 45:31 would need to be quite strong 45:33 in order to force these companies to actually act 45:37 with a level of safety that they would have. 45:40 Yeah, I'm glad you brought up the comparison to social media. 45:43 I wrote an essay a few years ago called 45:45 Understanding Social Media Recommendation Algorithms. 45:48 It was mostly about the algorithms themselves, 45:50 but one of the points I also made was that 45:52 this decision to optimize for engagement, 45:57 keeping people scrolling, et cetera, 45:59 was actually made without much regard 46:02 to what is good for the company itself in the long run. 46:05 For instance, I reviewed a study that came out of meta itself that showed that when they 46:11 had these kind of addiction maximizing design choices like spamming people with notifications, 46:16 in the short run, it increased people's use of the app. 46:20 But over a period of about a year or so, they started quitting the app. 46:24 And when I talk to my students now, there's a sizable fraction of them who have severely 46:29 cut back or entirely quit social media because they realize that over a period of months 46:34 or years, their experience really degenerates. 46:37 And I worry that the AI companies are caught in the same trap. 46:41 This culture of, you know, racing toward the newest model at all costs, it might feel in 46:46 the short term that that's what they need to do to get the headlines to be on top 46:50 of the artificial analysis index, whatever. 46:52 But because of, you know, the fact that there are these consequences for safety, hopefully 46:59 they're going to get sued if they continue down this road. It's actually not in their own 47:05 long-term commercial interest is what I feel. 47:07 It may not be in their long-term commercial interest, although maybe the social media 47:11 example is a good one to spend a second on because, look, the social media companies 47:16 are much more viewed with a lot more skepticism today than they were in, say, 2012. 47:23 12th. They are also richer today. Their valuation is higher. Meta is bigger, right? TikTok is a 47:31 phenomenon. I think we're just looking at a very standard thing where the incentive, what the market 47:38 rewarded and what society at least says it wanted may have not been the same. And if I had a person 47:44 from one of those companies sitting here, they would say, look, we're paying attention to 47:47 what the users actually do. They might say whatever they say in your classroom. 47:51 But people are spending longer than ever on TikTok, on Instagram, on some of these sites. 47:57 And the advertising is working better than ever. 48:01 And I'm not sure that they are wrong. 48:06 They can only be made wrong by society making a decision which disciplines the market into a different 48:14 formation than it would naturally or currently be in. 48:18 Yeah, I mean, again, I think I mostly agree. 48:20 I do think, again, that companies can make decisions that are irrational in their own 48:25 long-term interests because they're, especially in Silicon Valley, there's a culture really 48:29 of focusing on these shorter-term metrics and AB tests. 48:32 So then you get into something that you're beginning to touch there, which is diffusion. 48:36 And one place where you do have a view that is sort of different from some people in Silicon 48:42 Valley is that it's going to be much harder for AI to show up in the economy, for 48:47 at a shop in the world, then people think that there is not a one-to-one between intelligence 48:52 and that. 48:53 So talk to me a bit about diffusion. 48:55 Yeah, this really clicked for me actually a few months after we wrote this essay when 49:00 I was looking at Amtrak's proud announcements of the new train sets that they had purchased 49:05 for their Asella series. 49:08 Apparently, it can go 165 miles per hour. 49:11 At first, I thought, this is going to be amazing. 49:13 That's way faster than the trains currently go. 49:15 And then I dug into it a little bit more and it turns out the limiting speed is not the trains themselves 49:21 It's the tracks that are too curved and the signaling infrastructure that is centuries old and those things are not changing 49:28 And so the average speed hasn't really budged much 49:30 It's still 65 to 70 miles per hour and it struck me that this was you know 49:35 This is kind of an elegant way to say what we've been trying to say in AI as normal technology 49:39 Which is that most of the time AI is the trains. It's not the track 49:45 So, AI is accelerating a part of the process that was never the bottleneck to begin with. 49:50 And it's so many other things that are more infrastructural things that happen around the AI. 49:56 Organizational culture, regulation, even, you know, our ability socially to accept 50:00 the level of year-to-year change in our lives, things like self-driving cars, 50:06 no matter how many lives they might save. It's so much of a kind of a shock to society 50:10 that it will almost inevitably lead to what we've been seeing already, the kind of political 50:15 backlash. 50:16 And it's going to take quite a while, I think, to make all the societal adjustments to be 50:20 able to deploy these technologies. 50:22 If we ever do. 50:23 Yeah. 50:23 So this part of AI, the AIS try, I'm incredibly skeptical of, right? 50:27 You'll hear a Sam Holtman say, talk about how AI through innovation is going to help 50:33 us solve our energy problems. 50:36 Another way to sort of distill down that idea. 50:39 Is it the binding constraint right now on clean energy as intelligence? 50:47 But it's not. 50:47 It's not. 50:48 We know we have much better energy technologies than we're currently using for vast amounts of our energy infrastructure. 50:55 And we're not doing it because of it would be against some people's profits. 51:00 We're not doing it because there are political limits to building in the real world. 51:04 We're not doing it because Donald Trump hates solar and wind power. 51:07 or we're not doing it for all kinds of reasons. 51:09 And it feels like a lot of things are like that, 51:11 that if you accelerate or increase the amount 51:15 of intelligence behind it, 51:17 you just run into the other rate limiters in society, 51:22 drug development has testing and the FDA and all the, 51:25 I mean, abundance in my book is very much 51:27 about this in other areas. 51:29 We're aware of how to make faster trains 51:31 and make them in other places. 51:33 We don't, and it's not clear to me 51:34 why AI would solve those problems quickly or potentially at all. 51:40 That's right. And on top of that, there's various kinds of arms races. So there was this great 51:45 report by insurance companies last week that talked about how AI has already apparently 51:51 over the last few years added a billion dollars to medical expenses because hospitals are using 51:58 it to be able to code more complex conditions for the same diagnosis and same treatment. 52:03 Of course, we should be skeptical of any specific numbers they quote, but the New York 52:08 Times article about it had, you know, other people, academics making the same point. 52:12 And yeah, it's a kind of arms race that we see a lot. 52:15 We see it in the legal profession, AI for law, there's so much excitement about that. 52:20 But it's AI versus AI. 52:22 It's an arms race where the equilibrium simply shifts upwards. 52:26 I actually have a paper about this with Justin Curle. 52:29 And what we talk about there is that not only is there an arms race, there are other bottlenecks like if you make lawsuits a lot more efficient, there are still only a finite number of judges. 52:41 And I think we need, you know, we need those human judges, we shouldn't replace human judges with AI. 52:47 I think even if that's more efficient in some sense to me, that's definitionally almost axiomatically that's giving up control over, you know, the course of human destiny to AI because judges make law. 52:57 And that's not something we should be giving up to. 53:00 Yeah. So those are some really fundamental bottlenecks. 53:02 I want to get back at that AI versus AI point you just made there, 53:04 because I think this is really underplayed. 53:07 I think a lot about why did the internet not lead to a larger increase in global productivity 53:13 and innovation than it did. 53:16 And I always think the reason is that while it did all the things that the 53:20 idealists wanted it to do, it actually did make it possible to collaborate with 53:23 people all over the world. 53:25 Instantly, it did make virtually the entire corpus of human knowledge. 53:28 First available to us, then available, turned out to AI to train on. 53:33 It also did this opposite thing. 53:36 Like it did speed us up and it also slowed us down to distract us. 53:41 So now while you're working on something, you're clicking back and forth 53:44 from your email and into a, you know, an online game and over to social media. 53:50 And your ability to focus is degraded. 53:52 and you know, there's tremendously more porn, which appears to have had an effect on whether 53:56 or not people are forming real life human relationships, that adding or reducing friction 54:02 in one area also reduces it in areas that are maybe less beneficial. And when you think of 54:07 adding intelligence, well, that intelligence is going to add on the other side of things too. 54:14 And I always think that we, at the beginning of a technology, we think of all the ways it 54:18 make everything better. And I think we think a lot right now with AI about the ways it could make 54:24 things dramatically worse, right? Huge cybersecurity events or destruction of the financial system or 54:30 human extinction. But just the ways it might make things worse in banal fashions, where it just 54:37 like increases the ability of people to waste everybody else's time is a little bit underplayed 54:46 in my view. 54:46 For sure. 54:47 And, you know, it's not just banal. 54:49 I think there are a lot of these sub-catastrophic harms that are pretty serious. 54:53 With the Industrial Revolution, we had several decades of horrible labor conditions. 54:58 I think with AI as well, while there has been so much focus on job displacement, what 55:05 there has been much less focus on is how it's changing job quality. 55:08 And I think this is a big underappreciated area because AI is turning a lot of knowledge 55:14 workers into managers of AI agents, right? And the thing about being a manager is it's 55:20 kind of a shitty experience because you're responsible for other people's mistakes. You 55:25 don't get to practice the craft that you trained for. But you know, as human managers, 55:29 we don't get to complain. It's higher pay. It's higher status. We chose it. And of course, 55:32 mentoring people is gratifying with a, you don't get any of those benefits. You have to manage 55:37 this agent and be responsible for its mistakes, but you don't get to practice the craft. 55:42 I think we can design AI agents differently to avoid this. 55:45 But right now, that's kind of where things are going. 55:47 And yeah, I think we should be very concerned about that. 55:49 So how much of though does this story that we are telling here 55:52 lead to a theory of what's going to happen in the economy, 55:57 which is very jagged that the things where you need diffusion into the real world, right? 56:01 Things have to happen physically. 56:02 Buildings need to be built. 56:03 Energy, you know, transmission lines need to be laid down. 56:07 That has such powerful rate limiting on it that it can't accelerate that fast. 56:12 Meanwhile, inside the digital world, things can move very, very, very fast. 56:17 So, you know, in terms of what's going to happen to, you know, white collar workers 56:21 who they work kind of completely on a computer and their work actually can be automated, 56:25 you know, they're in a call center or something like that. 56:27 Or kind of separately, like all the cyber crime stuff we're talking about in the cyber 56:31 security that I think a version of this future playing out there that worries me 56:40 is that actually most of what would improve people's lives has to happen in like the physical 56:44 real world, but where AI is going to be able to move the fastest is in the digital world. 56:52 And as an equilibrium, I don't think that sounds like the world in which we're getting 56:56 the most benefit from AI, and it might in fact be the world we're getting the most harm from 56:59 it. Yeah, that's possible. I don't know. I think there are ways, you know, we can change 57:06 that? I don't think fast is that fast, first of all. Like you mentioned call centers. I mean, 57:13 those are still here. When ChatGPT was released, so many people were predicting that within a year 57:20 we would have replaced all of them. I mean, Chatbot, it's right there in the name. If, as is likely, 57:24 AI is going to make call center workers a lot more productive, there's a lot of latent 57:29 demand. A lot of the time we don't call call centers because it's a frustrating experience. 57:33 So, once again, it's a Jeven's paradox thing. 57:36 You want to describe what that is? 57:37 Yes, certainly. 57:38 So, it's the idea that when something becomes cheaper to produce, there's now more demand 57:42 for it. 57:43 Let's look at the sector where the capabilities are already the most advanced, which is probably 57:48 software engineering. 57:50 It used to be extremely expensive to produce software, so only maybe a few tens of thousands 57:55 of lines of code worldwide were written per year, and now that's expanded by something 58:00 like a million fold. 58:00 And so, over the long run, you know, whether this is going to be something that increases 58:06 demand for SAFRA engineers or whether the rocky, you know, job prospects we're seeing 58:12 for junior SAFRA engineers are going to continue remains to be seen, I think either as possible. 58:18 But, you know, either way, if this happens over the course of 20 years, again, that 58:22 is in line with other major shifts, such as the Industrial Revolution, where many 58:27 jobs went away, but many other new jobs were created. 58:30 Another example, translation jobs, you know, way back around 2016 machine is obviously way 58:37 before what we call generative AI now, translation models became pretty close to human parity. 58:43 But those jobs are still pretty intact, the nature of the job has changed a lot. 58:47 So once again, there's there's a lot more demand that has been unlocked by the fact 58:51 that you can translate anything to any language now. 58:54 So, Jevons Paradox, I had just had this conversation with Bill Gates and when I brought that up, 59:01 he was a little bit withering about it. 59:03 So, you know... 59:04 And Jevons Paradox, name a blue-collar profession that's subject to Jevons Paradox or do you 59:09 not care about blue-collar? 59:10 I do care about blue-collar. 59:11 So, the question is you... 59:12 Name anything in the blue-collar realm that's subject to that. 59:15 And his point was, look, software engineering is indeed a sector of the economy where 59:22 there's a lot of unmet demand, you know, arguably everybody would like their own 59:27 software engineers. So in a world where you rapidly accelerate that, you can get 59:30 this demand effect where it just creates more demand for software engineers 59:34 because now they're cheaper. But what he went on to say was that a lot of 59:38 things aren't like that. You think of, say, a truck driver. If we get to the 59:42 point where trucks are driverless, one, there is only so much demand for 59:45 trucking. And two, there's no longer a driver in that truck. If you 59:51 We look at a lot of the people who got their jobs automated away or sent to China in manufacturing. 59:55 It is true that the economy kept growing, but many of those people individually had a 60:00 very, very, very hard time. 60:02 And so his argument is, Jeven's Paradox is not going to be big enough to handle this 60:07 because there are too many areas of the economy where there's not latent demand. 60:12 There's only as much demand as there is. 60:14 How do you think about that? 60:16 I mean, he's completely right about the truck drivers, I think, totally agree there. 60:20 The demand there is relatively finite. 60:22 The question is whether that's the rule or the exception. 60:25 And let me put it this way. Look at what we're doing here. 60:28 Nobody asked for this. 60:30 You know, if you went back in time, 100 years or 200 years, 60:33 it would be like, how is this a real job? 60:36 Most of the jobs that we have today are jobs 60:38 that are kind of higher up the Maslow's hierarchy, if you will. 60:42 They're not meeting some actual real fixed demand 60:45 or need that people need in order to live their lives. 60:48 We do these things because, you know, they're fun and people like to listen to it. 60:53 Most white collar jobs are like that. 60:55 That's my view. 60:56 Most white collar jobs do have Chevin's paradox. 60:59 If it gets easier to produce more of, there will be demand for it, especially as people's 61:05 incomes as well rise very gradually. 61:07 And there's more spending on these less necessary, more luxury kinds of things. 61:12 And blue collar as well. 61:13 There's a great essay by Alex Emus called What Will Be Scares? 61:17 And he points out that a job of a Starbucks barista, for instance, already should not 61:22 exist that we've long known how to automate that we can make coffee at home even I don't 61:26 know if he said that specific thing but you know that's an example where it's the relational
Community discussion