Malcolm Got in a Heated Argument with Eliezer Yudkowsky at a Party (Recounting an AI Safety Debate)

Malcolm recounts a heated debate with AI theorist Eliezer Yudkowsky on AI safety. He explains his belief that subsystems in an advanced AI would converge on the same utility function, while Yudkowsky insists no AI would subdivide that way.

Simone notes Yudkowsky's surprising lack of knowledge in physics and neuroscience given his confidence. They express concern his ideas ruin youth's outlooks and discuss hypothetical clapbacks. Overall they conclude that while well-intended, Yudkowsky's certainty without humility on AI risks is dangerous.

Simone: [00:00:00] What's

Malcolm: really interesting is that he actually conceded that if this was the way that an AI structured itself, that yes, you would have terminal convergence, but that AIs above a certain level of intelligence would never structure themselves this way.

Malcolm: So this was very interesting to me because it wasn't the argument I thought he would take. And that would be true. I, I will agree that if the AI maintained itself as a single hierarchy, it would be much less likely for its utility function to change. But the problem is... Is essentially no government structure ever created and has functioned that way.

Malcolm: Essentially no program ever created by humans has run that way. Nothing ever encoded by evolution has run that way. i. e. the human brain, any brain, any neural structure we know of. There are none that are coded that way.

Malcolm: So it is very surprising. So I said, okay, gauntlet thrown. Are you willing to be disproven?

Malcolm: , because we will get some more understanding into AI interpretability, into how AIs. Think in the near future. If it turns out [00:01:00] that the AI's that exist right now are actually structuring themselves that way, will you concede that you are wrong about the way that you tackle AI apocalypticism?

Malcolm: And then he said, and this is really interesting to me. He's like, no, I won't

Malcolm: I was also like, yeah, also, we could run experiments where we do a bunch of basically unbounded A. I. S. and see if they start to show terminal convergence.

Malcolm: Do they start to converge on similar utility functions? You know what they're trying to optimize for again? He was like, well, even if we saw that, that wouldn't change my views on anything, right? Like his views are religious in nature, which was very disappointing to me. Like, I thought that maybe he had more of like a logical or rational perspective on things.

Malcolm: That. And it was, it was really sad.

Malcolm: You know, we don't talk negatively about people on this channel very frequently, but I do think that he destroys a lot of people's lives. And I do think that he makes the risk of AI killing all humans dramatically higher than it would be in a world where he didn't exist.

Would you like to know more?[00:02:00]

Simone: Hello, Malcolm.

Malcolm: Hello. So we just got back from this wonderful conference thing we were at called Manifest. So we had gone out to SF to host a few pronatalist focused dinner parties and randomly we got looped in to something called Manifest, which was a conference for people who are interested in prediction markets.

Malcolm: But interestingly, we ended up meeting a bunch of people who we had known through like online stuff. Some were absolutely fantastic, like, Scott Alexander, absolutely. I never met him before in person. We'd communicated on a few issues, really cool guy. Would you say so Simone?

Simone: Yeah. Like super awesome.

Malcolm: Richard Hedania, a really nice guy as well. Robin Hanson, who we, we, we'd actually met him before. But. And of course Ayla, we, we, we're old friends, you know, she's been on this channel before but we did get in a fight with someone there and I am very excited to tell you guys this [00:03:00] tale, because it was Eliezer Ukoski, but before we go further on that, I want to talk about a secret that we, we had a mystery.

Malcolm: The Pronatalist

Simone: Foundation had a mystery. Oh, I can tell this story. Yeah. So for the past few months, maybe closer to a year, we've received the odd random donation from someone. And it was the same person in the same amount each time, but it was very random timing. I could never predict when these would come in.

Simone: And it's very unusual for someone to donate. Multiple times like frequently like that. So we were always like very flattered and pleased. We didn't know this person We didn't recognize their name, but we're like, this is amazing. Like, thank you so much It means a lot to us and it really does and then we actually Met that person recently and Randomly

Malcolm: at the conference, you were talking to her and she mentioned she was the And

Simone: she mentioned that, yeah, that, that she was the mystery donor and that the reason why she donates turns out to be the coolest reason for [00:04:00] donating that I've ever heard before.

Simone: And I think it's the only way we should ever receive donations in the future. So she has a group of friends who she likes very much. And, and she enjoys spending time with them, but politically they are very Very different from her. So occasionally she has to just keep her mouth shut when they start going off on politics, because otherwise she will lose this group of friends because their politics is such that they will probably just.

Simone: You know, deep six, anyone who doesn't agree with him politically, and so instead of, you know, dealing with her anger by speaking out in the moment with her friends, she'll go home and she will revenge donate to whoever that most recent conversation that made her angry would be. The perfect like thorn in the side of these people.

Simone: So every time we've received a donation, it is, it is a

Malcolm: donation. But, and here I would actually say this for people watching who might not know this because they know us as like the internet, we have a nonprofit. It's a 501 C three. If you [00:05:00] are interested in like giving money, cause sometimes we get like super chats and stuff here and stuff like that.

Malcolm: You know, Google gets a big cut of those. And I don't think that any of us want to be giving Google any more money. So if you wanted to, you could always go directly to the foundation through through the donation link and also none of the money goes to us. Like, we don't we don't use it to pay our salaries or something, you know, as I said in the news, like, we spent over 40 percent of our salary last year.

Malcolm: On donations to this foundation, but it does go to something that we care about that much in terms of trying to fix the educational system. But yeah, and some other

Simone: donate with hatred. Donate when you are angry. Donate when you want it. Twist the

Malcolm: knife. Yeah. Donate with hatred. That's the type of donation we want.

Malcolm: We don't want people, we want your,

Simone: yes, we

Malcolm: want you to be biting other people. When you donate. That we want. And, and, we actually had a, a big donation recently who might push us down a different path to creating a nation state, which is something that we have been an idea we've been toying with. I'm excited about that, but let's [00:06:00] get to the topic of this video, the fi