Based Camp: Spencer Greenberg on Trying to Fix Science

In this insightful discussion, Spencer Greenberg delves into the replication crisis plaguing academic psychology research. He discusses projects aiming to improve reliability through replications and details warning signs like questionable statistics. Spencer advocates raising scientific standards to restore public trust. He also champions "renegade science" by independent researchers and highlights tools enabling robust studies outside academia. Overall, Spencer makes a thoughtful case for multiplying skill with truth-seeking to unlock discoveries that benefit society.

Links:

https://www.positly.com/

https://www.guidedtrack.com/

Simone: [00:00:00] Okay, here we go. Hi everyone. We have a very, very special guest today who we have known actually almost as long as we've known each other. We met Spencer Greenberg back in like around 2015 when he was first working on some of his projects that are now pervasively used.

Simone: Which is really, really cool. He is someone that we've profoundly respected for many years. He has been running Clearer Thinking for a ton of time, but more recently he launched the Clearer Thinking podcast, which is a series of interviews with incredible people that we really enjoy. I'm addicted to it personally.

Simone: So please check it out.

Malcolm: I'll just summarize the important point is he's probably one of, if not the most respected social figure in the E. A. And rationalist movement in the New York area, which is a very big thing because it's one of the major hubs of the

Simone: movement. Yeah. And Spencer, could you tell us what your top projects are right now?

Spencer: Yeah. Well, thanks for having me on. So [00:01:00] one of the projects I run is called clear thinking, which you mentioned at clear thing dot org. And what we do is we take interesting ideas from psychology, economics, math, and so on, that people might learn in passing, maybe they'll learn in blog posts or reading books, but they don't generally apply them to their lives.

Spencer: And so our goal is to make it really easy to apply these ideas to your life to try to achieve the things that you want to achieve. So we have these interactive modules, we have over 70 of them right now, and you can use them all for free. And I also do the Clear Thinking Podcast, as you mentioned. In addition to that, we have a bunch of other projects for accelerating social science.

Spencer: So our goal is to try to help. Psychological research go faster, be more robust, be more reliable and help unlock important ideas about human nature that can be of benefit to society. Speaking of that,

Malcolm: what we wanted to focus on this podcast is you recently did some research into the replication crisis, how bad it's gotten, and I think you have some theories on how it could be fixed.

Malcolm: So I'd love you to just dive into that first, explaining what the replication crisis is, its scope and your [00:02:00] research, and then going through potential

Spencer: solutions. Sure, yeah, it's a topic I think about a lot. So basically, there are many really interesting findings in psychology that have unfortunately failed to replicate, which means that basically when people try to redo the same study, collect a new sample of study participants, they just don't get the original answer.

Spencer: And that's been very disturbing. A bunch of findings that were in textbooks and that are really famously known just don't work, it seems. So some examples of this would be from the social priming literature where they do things like have someone hold a warm cup of coffee. And then people would find that there would be rated as more warm, or they'd rate things as more warm because we make these psychological metaphors.

Spencer: Well, it's a really cool sounding idea, but doesn't necessarily replicate. Or another example, when you prime people with words that are related to being older, the people then walk slower. Well, again, a really cool concept, but it didn't replicate when people tried it again. And so the question is, why are so many findings not replicating?

Spencer: And how pervasive a problem is [00:03:00] this. And so looking at the many different replication studies that have occurred, my best guess is that from top journals and very top journals, probably about 40% of the results don't replicate. Cool. Enormous. Wow. Yeah, it's pretty, it's pretty it's pretty shocking. Some people, their response is, Well, science is hard, human nature is complicated.

Spencer: What do we really expect? But my view is that, no, 40% not replicating is way, way too high. It should be something on the order of 5 to 10%. I think that might be reasonable. Yes. But the problem with 40% is it's almost a coin flip. If you read a paper, will this, will this result hold up, right?

Malcolm: Yeah.

Malcolm: So do you think that this is one, one thing that might be fun to go into for the audience is how does this system work? Like the, the scientific system that gets things in these journals and where do you believe it's failing?

Spencer: Yeah, so it's an interesting question why this is happening, because you might think isn't this what peer review is designed to prevent, right?

Spencer: People are submitting to these [00:04:00] top journals, experts in their fields are reviewing the papers. But I think the fundamental problem there, it's not that peer reviewers want to let in garbage papers. They don't really have an... Reason to let in garbage papers. They don't get paid more if they let in bad papers.

Spencer: They don't get more prestige if they let in bad papers. No, the reviewers are reading the papers and trying to let in the problem is that, that whether something replicates or not is not generally visible from reading the paper. Oh,

Simone: because I was going to ask are there warning signs?

Spencer: Yeah. Well, we've, in recent years, there's been a lot of interesting things learned about the warning signs, but people didn't really know what the warning signs were necessarily.

Spencer: In the past, and it's very difficult for people to tell what is a legitimate paper and what's not. I can tell you about some of the warning signs. Please. One is about p values, which are a rather technical subject and something that is confusing people. But the basic gist of a p value is when you're testing a hypothesis using statistics, let's say you say, well, if I give people this psychological treatment, then they'll have less anxiety [00:05:00] at the end of the study on average than if I don't give them the psychological treatment, right?

Spencer: And then you want to know, well, did the result that I got exceed what you'd expect by chance, right? So let's say you find that people who, who got the psychological treatment had An average anxiety rating of 10, whereas those that didn't get it had an average anxiety rating of 7 out of 10. So that looks good.

Spencer: It looks like the psychological treatment worked. But the question is, well, it's 5 compared to 7. Is that really good enough that we can conclude the treatment works? And so what you do is you compute what's known as the p value. And the p value is basically saying, what's the chance I would get a result that's this different?

Spencer: So, a difference this large or larger, if in fact there was no effect. And so if the p value is re