AI Fundamentals: Benchmarks 101

We’re trying a new format, inspired by Acquired.fm! No guests, no news, just highly prepared, in-depth conversation on one topic that will level up your understanding. We aren’t experts, we are learning in public. Please let us know what we got wrong and what you think of this new format!

When you ask someone to break down the basic ingredients of a Large Language Model, you’ll often hear a few things: You need lots of data. You need lots of compute. You need models with billions of parameters.

Trust the Bitter Lesson, more more more, scale is all you need. Right?

Nobody ever mentions the subtle influence of great benchmarking.

LLM Benchmarks mark our progress in building artificial intelligences, progressing from

* knowing what words go with others (1985 WordNet)

* recognizing names and entities (2004 Enron Emails)

* and image of numbers, letters, and clothes (1998-2017 MNIST)

* language translation (2002 BLEU → 2020 XTREME)

* more and more images (2009 ImageNet, CIFAR)

* reasoning in sentences (2016 LAMBADA) and paragraphs (2019 AI2RC, DROP)

* stringing together whole sentences (2018 GLUE and SuperGLUE)

* question answering (2019 CoQA)

* having common sense (2018 Swag and HellaSwag, 2019 WinoGrande)

* knowledge of all human tasks and professional exams (2021 MMLU)

* knowing everything (2022 BIG-Bench)

People who make benchmarks are the unsung heroes of LLM research, because they dream up ever harder tests that last ever shorter periods of time.

In our first AI Fundamentals episode, we take a trek through history to try to explain what we have learned about LLM Benchmarking, and what issues we have discovered with them.

There are way, way too many links and references to include in this email. You can follow along the work we did for our show prep in this podcast’s accompanying repo, with all papers and selected tests pulled out.

Enjoy and please let us know what other fundamentals topics you’d like us to cover!

Timestamps

* [00:00:21] Benchmarking Questions

* [00:03:08] Why AI Benchmarks matter

* [00:06:02] Introducing Benchmark Metrics

* [00:08:14] Benchmarking Methodology

* [00:09:45] 1985-1989: WordNet and Entailment

* [00:12:44] 1998-2004 Enron Emails and MNIST

* [00:14:35] 2009-14: ImageNet, CIFAR and the AlexNet Moment for Deep Learning

* [00:17:42] 2018-19: GLUE and SuperGLUE - Single Sentence, Similarity and Paraphrase, Inference

* [00:23:21] 2018-19: Swag and HellaSwag - Common Sense Inference

* [00:26:07] Aside: How to Design Benchmarks

* [00:26:51] 2021: MMLU - Human level Professional Knowledge

* [00:29:39] 2021: HumanEval - Code Generation

* [00:31:51] 2020: XTREME - Multilingual Benchmarks

* [00:35:14] 2022: BIG-Bench - The Biggest of the Benches

* [00:37:40] EDIT: Why BIG-Bench is missing from GPT4 Results

* [00:38:25] Issue: GPT4 vs the mystery of the AMC10/12

* [00:40:28] Issue: Data Contamination

* [00:42:13] Other Issues: Benchmark Data Quality and the Iris data set

* [00:45:44] Tradeoffs of Latency, Inference Cost, Throughput

* [00:49:45] Conclusion

Transcript

[00:00:00] Hey everyone. Welcome to the Latent Space Podcast. This is Alessio, partner and CTO and residence at Decibel Partners, and I'm joined by my co-host, swyx writer and editor of Latent Space.

[00:00:21] Benchmarking Questions

[00:00:21] Up until today, we never verified that we're actually humans to you guys. So we'd have one good thing to do today would be run ourselves through some AI benchmarks and see if we are humans.

[00:00:31] Indeed. So, since I got you here, Sean, I'll start with one of the classic benchmark questions, which is what movie does this emoji describe? The emoji set is little Kid Bluefish yellow, bluefish orange Puffer fish. One movie does that. I think if you added an octopus, it would be slightly easier. But I prepped this question so I know it's finding Nemo.

[00:00:57] You are so far a human. Second one of these emoji questions instead, depicts a superhero man, a superwoman, three little kids, one of them, which is a toddler. So you got this one too? Yeah. It's one of my favorite movies ever. It's the Incredibles. Uh, second one was kind of a letdown, but the first is a.

[00:01:17] Awesome. Okay, I'm gonna ramp it up a little bit. So let's ask something that involves a little bit of world knowledge. So when you drop a ball from rest, it accelerates downward at 9.8 meters per second if you throw it downward instead, assuming no air resistance, so you're throwing it down instead of dropping it, it's acceleration immediately after leaving your hand is a 9.8 meters per second.

[00:01:38] B, more than 9.8 meters per second. C less than 9.8 meters per second. D cannot say unless the speed of the throw is. I would say B, you know, I started as a physics major and then I changed, but I think I, I got enough from my first year. That is B Yeah. Even proven that you're human cuz you got it wrong.

[00:01:56] Whereas the AI got it right is 9.8 meters per second. The gravitational constant, uh, because you are no longer accelerating after you leave the hand. The question says if you throw it downward after leaving your hand, what is the. It is, it goes back to the gravitational constant, which is 9.8 meters per, I thought you said you were a physics major.

[00:02:17] That's why I changed. So I'm a human. I'm a human. You're human. You're human. But you, you got them all right. So I can't ramp it up. I can't ramp it up. So, Assuming, uh, the AI got all of that right, you would think that AI will get this one wrong. Mm-hmm. Because it's just predicting the next token, right?

[00:02:31] Right. In the complex Z plane, the set of points satisfying the equation. Z squared equals modulars. Z squared is A, a pair points B circle, C, a half line D, online D square. The processing is, this is going on in your head. You got minus three. A line. This is hard. Yes, that is. That is a line. Okay. What's funny is that I think if, if an AI was doing this, it would take the same exact amount of time to answer this as it would every single other word.

[00:03:05] Cuz it's computationally the same to them. Right.

[00:03:08] Why AI Benchmarks matter

[00:03:08] Um, so anyway, if you haven't caught on today, we're doing our first, uh, AI fundamentals episode, which just the two of us, no guess because we wanted to go deep on one topic and the topic. AI benchmarks. So why are we focusing on AI benchmarks? So, GPT4 just came out last week and every time a new model comes out, All we hear about is it's so much better than the previous model on benchmark X, on benchmark Y.

[00:03:33] It performs better on this, better on that. But most people don't actually know what actually goes on under these benchmarks. So we thought it would be helpful for people to put these things in context. And also benchmarks evolved. Like the more the models improve, the harder the benchmarks get. Like I couldn't even get one of the questions right.

[00:03:52] So obviously they're working and you'll see that. From the 1990s where some of the first ones came out to day, the, the difficulty of them is truly skyrocketed. So we wanna give a, a brief history of that and leave you with a mental model on, okay, what does it really mean to do well at X benchmark versus Y benchmark?

[00:04:13] Um, so ex