Show full transcript 0:24 especially for my team at Flight Story. 0:26 As over the last year, 0:27 we've moved out of startup mode into scale up mode. 0:30 And that's the stage where we found it mission critical 0:32 to use NetSuite. 0:33 We no longer have to juggle multiple systems 0:35 and having everything together has reduced 0:37 the number of manual tasks 0:38 and of course the number of errors. 0:40 Over 42,000 businesses have chosen 0:42 to future-proof their business with NetSuite. 0:44 So if you'd like to learn how it can help your business, 0:47 head to netsuite.com slash Bartlett 0:49 and free download the CFO's guide 0:51 to AI and machine learning. 0:53 That's netsuite.com slash Bartlett. 1:00 What I find really interesting is to look at how people's quotes evolve over time with their 1:04 incentives. So I was looking at all of the things they've said on the record on podcasts 1:08 in their blog posts to see how it's evolved over time. And Dario, who was the former VP 1:12 of Research, OpenAI, and has now moved on to Anthropic, who are taking a slightly different 1:17 approach to developing AI, said back in 2017, while he was still at OpenAI, that this is a 1:24 quote, I think at the extreme end is the Nick Bostrom style of fear that an AGI could 1:28 destroy humanity. I can't see any reason in principle why that couldn't happen. My chance 1:35 that something goes really quite catastrophically wrong on the scale of human civilization might 1:40 be somewhere between 10% and 25%. And also you mentioned Ilya, who was a co-founder 1:47 of OpenAI, and then left. I guess the first question I'd ask is, why did Ilya leave? 1:52 That's a great question. 1:55 So he was instrumental in trying to get Sam Altman fired. 1:59 And he's another one of the people who over time began to feel like he was being manipulated 2:05 by Altman towards contributing something that he didn't believe in. 2:10 How'd you know? 2:11 Because I interviewed a lot of people. 2:13 Ilya, in particular, had two pillars that he cared about deeply. 2:19 one is making sure we get to so-called AGI and the other is making sure that we 2:26 get to it safely. And he felt that Altman was actively undermining both 2:31 things. He felt that Altman was creating a very chaotic environment within the 2:36 company where he was pitting teams against each other, where he was telling 2:41 different things to different people. Have you ever spoken to him? I have so 2:44 So I interviewed him in 2019 for a profile that I did of Open AI for MIT Technology Review. 2:52 And back in 2019, he has a quote where he says, 2:55 the future is going to be good for AIs regardless. 2:57 It would be nice if it was also good for humans as well. 3:00 It's not that it's going to actively hate humans or want to harm them, 3:03 but it's just going to be so powerful. 3:05 And I think a good analogy would be the way that humans treat animals. 3:08 It's not that we hate animals. 3:09 I think humans love animals and have a lot of affection for them. 3:12 But when the time comes to build a highway between two cities, we are not asking the animals 3:16 for permission. 3:17 We just do it because it's important to us. 3:20 And I think by default, that's the kind of relationship that's going to be between 3:23 us and AI, which are truly autonomous and operating on their own behalf. 3:30 That was in 2019, the year that you interviewed him. 3:32 One of the things that I feel like we should take a step back to examine is going 3:36 back to this idea of what even is artificial intelligence and what do we mean by intelligence. 3:40 intelligence, and a huge part of the views of the different people and the quotes that 3:45 you're reading derives from a specific belief that they each have in this question of what 3:52 is intelligence, what constitutes intelligence. 3:55 For Ilya, he has throughout his research career felt that ultimately our brains are 4:04 giant statistical models. 4:06 This is not something that we actually know, but this is his own hypothesis, also the hypothesis of his mentor Jeffrey Hinton who also was on this podcast. 4:17 This is why they have such a strong conviction in the idea of building AI systems that are statistical models and that this particular approach is going to lead to intelligent systems as we are intelligent. 4:30 It's a hypothesis that they have. 4:32 It's not one that has been proven by science and some people vehemently disagree with them on this particular thing 4:39 but if you step into their shoes and take on that hypothesis and 4:44 assume that it's true that our brains are in fact statistical engines and 4:50 that these 4:51 systems that they're building are also statistical engines that they're making bigger and bigger and bigger until they become the size of the human brain 4:58 that's why they say that 5:01 that making this comparison where the system will become 5:05 equal to human intelligence 5:06 and then maybe exceed human intelligence 5:09 is relevant in their framework. 5:12 And Ilya gave a talk at one point 5:14 at this really prominent AI research conference 5:17 that happens every year 5:18 called Neural Information Processing Systems. 5:21 It's mouthful. 5:22 But he gave this keynote where he shows this chart 5:26 of the size of brains and the intelligence of a species. 5:32 And it's roughly linear, the bigger the size of the brain, 5:36 the more intelligent the species. 5:38 And so for him, he thinks he's building a digital brain 5:42 because he thinks brains are just statistical engines. 5:46 So from that logic, it's like, okay, 5:49 if we then build a bigger statistical engine 5:51 than the human brain, then based on this chart, 5:55 where it will be more intelligent and then we will be subjected to the same treatment 6:00 that we've subjected animals. 6:02 But it's really important to understand that these are scientific hypotheses of specific 6:06 individuals within the AI research community. 6:10 And there's a lot, a lot of debate about whether this is in fact the case. 6:15 And some of the biggest critics say it's very reductive to think of our brains as 6:20 simply just statistical engines. 6:21 Why does it matter to know the mechanism? 6:27 Is it not just important to know the outcome, which is that it's going to be able to do 6:32 make a video for me, or agents are going to be able to do the work that I do? 6:36 Does it really, really matter for us to know the mechanism behind it? 6:40 Yes or no. 6:41 So it matters because these companies, they are driving their future actions based 6:49 based on this hypothesis. 6:51 So they have decided we think that this hypothesis is true. 6:57 Like we should just continue building 6:58 larger and larger statistical models 7:00 in the pursuit of artificial general intelligence. 7:03 And that's then having global consequences. 7:06 Like in order to continue doing that, 7:08 they're hoovering up more and more data. 7:10 They're building more and more data centers. 7:13 They are exploiting more and more labor 7:16 in order to continue on this path. 7:19 Here's a question that I think is important to ask, 7:22 is why are we trying to build AI systems 7:25 that are duplicative of humans? 7:27 We're kind of having this conversation right now 7:28 where we've just taken the premise of this industry 7:32 as a good thing. 7:34 Like they said that we should be building AGI, 7:37 so we say that we should be building AGI. 7:38 But I would like to ask, like why are we doing that? 7:42 Why is it that we are building a technology 7:45 technology that is ultimately designed to replace and automate people away. 7:49 That is not the enterprise of technology, like we should be building technology and the purpose 7:57 of technology throughout history has been to improve human flourishing, not to replace 8:03 people. 8:04 And so this is like a critical part of my critique of these companies and the scientists 8:11 is that I've just adopted this goal 8:13 and have relentlessly pursued it 8:15 and have had enormous capital 8:16 and enormous resources to pursue it, 8:18 is this the right goal? 8:20 Like, why are we doing this? 8:22 Why can't we just build AI systems 8:25 that do things like accelerate drug discovery 8:28 and improve people's healthcare outcomes, 8:31 which are systems that have nothing to do 8:33 with the statistical engines that they're trying to build 8:35 to duplicate the human brain? 8:37 So why are they doing it? 8:38 I mean, you've interviewed all these people. 8:40 I think it's 300 people in total, 80 or 90 of them from OpenAI, the maker of ChatGBC. 8:46 Why do you think they're doing it? 8:48 I think it's because they're driven by an imperial agenda. 8:51 And that is why I call these companies empires of AI. 8:53 What do you mean by an imperial agenda? 8:56 What does that term mean? 8:57 Empire is the only metaphor that I've ever found to fully encapsulate all of the 9:03 dimensions of what these companies do and the scale that they operate and what 9:09 but motivates them to do what they do. 9:11 And there are many parallels that you see 9:14 between what I call the empires of AI 9:15 and the empires of old. 9:17 They lay claim to resources that are not their own 9:20 in the pursuit of training these models. 9:22 That's the data of individuals, 9:23 the intellectual property of artists, writers, 9:25 and creators, their land grabbing 9:27 in order to build these supercomputer facilities 9:30 for training the next generation models. 9:32 Second, they exploit an extraordinary amount of labor. 9:34 They contract hundreds of thousands of workers 9:38 all around the world, including in the US, to ultimately make these technologies. 9:44 We can talk about that more. 9:46 And they also designed their tools to be labor-automating so that when the technologies are deployed, 9:52 it also affects labor rights because it erodes away labor rights. 9:57 And this is a political choice that they have. 10:00 Third, they monopolize knowledge production, so they project this idea that they're 10:04 the only ones that really understand how the technology works. 10:07 and so if the public doesn't like it, it's because they don't actually know enough about this technology. 10:11 They do this to the public, they do this to policymakers, 10:14 and they've also captured the majority of the scientists that are working on understanding the limitations and capabilities of AI. 10:22 You think they're gaslighting the public, in a way? 10:24 They are, yeah. So if most of the climate scientists in the world were bankrolled by fossil fuel companies, 10:31 do you think we would get an accurate picture of the climate crisis? 10:35 No. 10:35 And in the same way, the AI industry employs and bankrolls most of the AI researchers in the world. 10:44 So they set the agenda on AI research in soft ways simply by funneling money to their priorities 10:50 so that only certain types of AI research are produced. 10:54 But they also will censor researchers when they do not like what the researcher has found. 11:00 And so I talk about the case of Dr. Timnit Gebrou in my book who was the ethical AI team 11:07 co-lead at Google when she was literally hired to critique the types of AI systems that Google 11:14 was building. She then co-wrote a critical research paper that was showing how large language 11:20 models specifically were leading to certain types of harmful outcomes. And in an attempt 11:26 to try and stop this research from being published, Google ended up firing Gevru and then fired 11:33 her other co-lead, Margaret Mitchell. 11:36 And so they control and quash the research that is inconvenient to the Empire's agenda. 11:46 Did you have an example where this is happening to journalists as well that are asking questions 11:50 of their team members? 11:52 I think I was watching a video of yours where there was a young man that was saying he had 11:57 someone show up at his door, knocked on his door and asked for information, emails, text 12:02 messages, and this person was from one of the big AI companies. 12:05 This was opening, I started subpoenaing some of its critics, yeah, as part of what appears 12:14 to be a campaign of intimidation, but also what appeared to be a campaign of phishing 12:17 for more information to figure out, to map out the network of critics further. 12:23 But this was a man who runs a small watchdog nonprofit, and they had been doing a lot of 12:30 work during that time to try and ask questions about OpenAI's attempt to convert from a 12:37 nonprofit to a for-profit. 12:38 Ultimately, OpenAI was successful in that conversion, but during the period where 12:42 where it was sort of existential for open AI 12:45 to complete this conversion. 12:48 There were a lot of civil society groups 12:50 and watchdog groups like Midas 12:51 who were trying to prevent the process 12:56 from happening in the dead of night. 12:58 They were trying to get more transparency. 13:00 They were trying to have more public debate about this 13:03 because it's unprecedented. 13:05 And it was then that there was a knock on his door 13:09 and he was served papers. 13:11 What do the papers say? 13:13 The papers asked him to reproduce every single piece of communication that he had had that 13:19 might have involved Musk. 13:20 So this was like the strange paranoia that opening I had that Musk was somehow funding 13:25 these people to block the conversion. 13:27 None of them were actually funded by Musk. 13:30 So in this particular case, their request, he simply was just answered, you know, I 13:34 don't have any documents because this doesn't exist. 13:37 So going back to this point of empires, you were saying that one of the factors 13:40 of an empire is land grab. And then the next one was... 13:44 Was labor exploitation. 13:45 Labor exploitation. 13:47 The third one, controlling knowledge, production. 13:51 And one of the other ones that's really important to understand about the AI empires in particular 13:58 is empires always have this narrative that they say to the public like, 14:04 we're the good empire and we need to be an empire in the first place because 14:09 because they're also bad empires in the world. 14:11 And if you allow us to take all the resources 14:15 and use all of the labor, 14:17 then we promise we will bring you progress 14:19 and modernity for everyone. 14:22 We will bring you to this utopic state akin to an AI heaven. 14:27 But if the evil empire does it first, 14:29 we will descend into a hell. 14:31 And the evil empire being in this case? 14:33 In this case, most often it's China. 14:36 But actually in the early days, 14:38 open AI evoked Google as the evil empire. 14:42 So all of their decisions were about we need to do it first 14:44 because otherwise Google, this evil corporation 14:48 that's driven by profit, us as a benevolent nonprofit, 14:52 like this is a critical contest of who wins. 14:58 Do you think the people building these AI companies 15:01 believe that the outcome is gonna be all good now? 15:05 Do you think they think that it's gonna be, 15:07 it's going to serve everyone, it's going to be the age of abundance, everything's going to go 15:11 well. What do you think they believe? What do you think Sam believes? 15:14 So this is so funny, is such a core part of the mythology that they create around the 15:20 AI industry includes the belief that it could go very badly. It goes hand in hand. Like, 15:28 they need that part of the myth in order to then say, and that's why we need to be in 15:34 control of the technology, because that's the only way that it's going to go really, 15:36 really well. And Altman has said publicly, you know, the worst case lights out for everyone, 15:43 but best case, we cure cancer, we solve climate change, and there's abundance. And Dario Amade, 15:49 same kind of rhetoric, who's like, worst case, catastrophic or existential harm for humanity, 15:57 best case, mass human flourishing. So this is like two sides of the same coin, like they have to 16:02 use both of these narratives in order to continue justifying an extremely anti-democratic approach 16:11 to AI development, where there should not be broad participation in developing this technology. 16:16 They must be the ones controlling it at every sip of the way. 16:20 Sam Otman did a tweet saying, 16:22 there are some books coming out about Open AI and me. We only participated in two of them. 16:26 One by Keshe Hagee, Keech Hagee, focused on me, and one by Ashley Vance on OpenAI. 16:35 He went on to say, no book will get everything right, especially when some people are so intent 16:40 on twisting things, but these two authors are trying to. 16:45 You quote retweeted that tweet from Sam Altman, and you said, the unnamed book Empire 16:51 of AI is mine. 16:54 Do you believe that tweet from Sam Altman was in reference to your book? 16:58 A hundred percent because there's only three books coming out about him. 17:02 And he'd caught wind that your book was coming out and... 17:04 He knew my book was coming out because I had contacted OpenAI from the very beginning 17:08 of my process and said, I'm working on a book now, will you participate in it? 17:12 And actually initially they said yes, even though, so my history with OpenAI, I profiled 17:17 the company for MIT Technology Review. 17:19 I embedded within the office for three days in 2019. 17:23 My profile comes out in 2020. 17:25 The leadership are very unhappy. 17:28 And in my book, I actually quote an email 17:31 that I received that Sam Altman sent to the company 17:35 about my profile saying, yeah, this is not great. 17:39 And from then on, the company's stance to me 17:45 was we are not going to participate 17:49 and anything that you do, we are not going to respond to any questions that you receive. 17:54 And this was things that they explicitly articulated. It wasn't like me inferring. 18:01 So I had a colleague at MIT Technology Review that also covered AI. And at one point, 18:07 OpenAI sent him this press release, being like, we would love for you to cover this story. 18:10 And he was like, I'm really busy. Will you send it to Karen? And they were like, 18:15 like, oh no, we have a history you understand. 18:19 And so for three years, they refused to talk to me. 18:23 But then I ended up at the Wall Street Journal where if they felt a bit compelled because 18:30 it was the journal to reopen the lines of communication. 18:34 And so I started having more dialogue with them. 18:39 Every time I wrote a piece, I would always send them, here's my request for comment. 18:42 I would always ask them, like, will you sit for interviews? 18:44 And we did get to a more productive relationship. 18:48 And then I embarked on the book, 18:50 so I left the journal to focus on the book full time. 18:53 And I told them right away, I'm working on this book. 18:56 I want to continue this productive conversation 19:00 where I make sure I reflect Open AI's perspective 19:04 in the book. 19:05 And so they were like, we can arrange interviews for you, 19:09 you can come back to the office, 19:10 this, we'll set up some conversations. 19:14 And then as we were going back and forth on this, the board fired Sam Altman. 19:20 And that's when things started going kind of south, because the company started becoming 19:25 very sensitive to scrutiny. 19:28 And so then they started pushing, kicking the can down the road, down the road, down 19:31 the road. 19:32 And I kept saying, hey, when are we rescheduling this? 19:34 What's going on? 19:35 And then I get an email saying, we are not going to participate at all. 19:39 you are not coming to the office, you're not doing interviews. And I had actually already 19:43 booked my tickets, so I was already going to fly to San Francisco to have the interviews. 19:50 And so then I told them, I was like, that's fine. I will still engage in the process, 19:56 well give you extensive requests for comment. I'll ask through my reporting, I'll keep 20:01 you updated on all the things that I'm finding so that you can choose to still 20:05 comment. I gave them 40 pages of requests for comment. And I gave them over a month to respond 20:12 to all of that. So this was when the tweet came out was we were doing all this back and forth 20:17 trying to and that's when Altman tweeted this. And they never responded to a single one of the 20:27 one of the 40 pages. Sam Altman does a lot of interviews. Yeah. You know, 20:32 He's doing a lot of interviews all the time. He's done every podcast. I've seen him on 20:36 everything from Tucker Carlson to I think he's done Theo von Joe Rogan. 20:41 Podcasts all over the world. I wonder why he won't do mine. 20:48 Well, maybe. I don't know why. I think I'm fair with everyone. I just ask questions I genuinely 20:54 care about. I don't come in with huge preconceptions or at least meet people for the 20:58 first time, but I've heard through the grapevine that he doesn't want to do mine. 21:04 I mean, going back to what you were saying earlier that with the way that OpenAI and these 21:10 companies control research, you asked, do they also do this with journalists? 21:15 I mean, yes, the answer is yes. 21:18 And apparently they also do it with anyone who has a broad mass communications platform. 21:24 It's not just about the conversation that you're going to have with them. 21:28 It's about who you also choose to platform. 21:31 And there's this huge problem in technology journalism where companies know that a really 21:39 big carrot that they can give to technology journalists is access. 21:43 And they will withhold that access at the drop of a hat if they catch wind that you're speaking 21:51 to someone that they didn't want you to speak to. 21:53 This is so true. 21:54 And I don't think the average person really truly understands this. 21:57 So this kind of sounds like theory as you say it, but I'm not going to name names here 22:02 because I don't think it's important. 22:04 But there is a particular person in AI whose team have basically dangled the carrot of them 22:12 coming here for like 18 months and I'm like, you don't have to dangle the carrot. 22:16 I'm going to speak to whoever I want to regardless of the carrot or not. 22:19 And when this person comes if they want to come, I'll give them a fair shot. 22:22 I'll ask them all genuinely curious questions about what they're doing, their incentives. 22:26 I won't gotcha them I don't have a history of ever gotchering anybody even if I just like even if I have a different of opinion 22:31 I'll ask the question. Yeah, but they dangle carrots and they say well, if you know, he might he's thinking about it 22:37 Let's think about a day and what the what the strategy is and I don't think they they think both people don't understand is if we 22:42 Just dangle it for long enough then they will 22:46 Perform in the way that we want them to do and they'll be 22:50 They'll be pleasant about us. They won't be critical. They won't give a give a 22:56 Critics. 22:56 Critics. 22:57 And I think a lot of their game is just dangle the carrot forever. 23:00 Yes. 23:01 Yeah. 23:01 That's like the optimal outcome, is that if we just dangle it, if we just tell them, 23:04 yeah, no, we're just trying to look at the schedule, it just doesn't work. 23:07 I think in the modern world, you just have to go there and give your opinion and allow 23:10 the clash of ideas in the public forum, let the viewers decide for themselves what 23:15 they think. 23:16 Yeah. 23:16 But this is, yeah, this is such a huge part of their machinery is the way that 23:22 they use these tactics to massage the public image of these companies and make sure that 23:27 information that they don't want out and even opinions that they don't want out there go 23:31 out there. 23:33 And so this is, you know, I feel very lucky now that opening I shut the door early on 23:40 me. 23:41 At the time I didn't feel lucky. 23:43 I felt like I had screwed myself over. 23:46 I was like, should I have been nicer to them and the profile so that I could maintain 23:49 access, which is a horrible question to ask as a journalist, right? Like, you're supposed to 23:53 report the truth and you're always supposed to report in the interest of the public. Like, 23:58 that is the point of journalism. And in that moment, I was like, relatively junior in my career, 24:04 I was like, did I misunderstand what journalism is about? Like, should I have actually been
Community discussion