97% Cheaper, Faster, Better, Correct AI — with Varun Mohan of Codeium

OpenAI just rollicked the AI world yet again yesterday — while releasing the long awaited ChatGPT API, they also priced it at $2 per million tokens generated, which is 90% cheaper than the text-davinci-003 pricing of the “GPT3.5” family. Their blogpost on how they did it is vague: Through a series of system-wide optimizations, we’ve achieved 90% cost reduction for ChatGPT since December; we’re now passing through those savings to API users.

We were fortunate enough to record Episode 2 of our podcast with someone who routinely creates 90%+ improvements for their customers, and in fact have started productizing their own infra skills with Codeium, the rapidly growing free-forever Copilot alternative (see What Building “Copilot for X” Really Takes). Varun Mohan is CEO of Exafunction/Codeium, and he indulged us in diving deep into AI infrastructure, compute-optimal training vs inference tradeoffs, and why he loves suffering.

Recorded in-person at the beautiful StudioPod studios in San Francisco.

Full transcript is below the fold.

Timestamps

* 00:00: Intro to Varun and Exafunction

* 03:06: GPU Efficiency, Model Flop Utilization, Dynamic Multiplexing

* 05:30: Should companies own their ML infrastructure?

* 07:00: The two kinds of LLM Applications

* 08:30: Codeium

* 14:50: “Our growth is 4-5% day over day”

* 16:30: Latency, Quality, and Correctability

* 20:30: Acceleration mode vs Exploration mode

* 22:00: Copilot for X - Harvey AI’s deal with Allen & Overy

* 25:00: Scaling Laws (Chinchilla)

* 28:45: “The compute-optimal model might not be easy to serve”

* 30:00: Smaller models

* 32:30: Deepmind Retro can retrieve external infromation

* 34:30: Implications for embedding databases

* 37:10: LLMOps - Eval, Data Cleaning

* 39:45: Testing/User feedback

* 41:00: “Users Is All You Need”

* 42:45: General Intelligence + Domain Specific Dataset

* 43:15: The God Nvidia computer

* 46:00: Lightning round

Show notes

* Varun Mohan Linkedin

* Exafunction

* Blogpost: Are GPUs Worth it for ML

* Codeium

* Copilot statistics

* Eleuther’s The Pile and The Stack

* What Building “Copilot for X” Really Takes

* Copilot for X

* Harvey, Copilot for Law - deal with Allen & Overy

* Scaling Laws

* Training Compute-Optimal Large Language Models - arXiv (Chinchilla paper)

* chinchilla's wild implications (LessWrong)

* UL2 20B: An Open Source Unified Language Learner (20B)

* Paper - Deepmind Retro

* “Does it make your beer taste better”

* HumanEval benchmark/dataset

* Reverse Engineering Copilot internals

* Quora Poe

* Prasanna Sankar notes on FLOPs and Bandwidth

* NVIDIA H100 specs - 3TB/s GPU memory, 900GB/s NVLink Interconnect

* Optimizer state is 14x size of model - 175B params => 2.5TB to store state → needs at least 30 H100 machines with 80GB each

* Connor Leahy on The Gradient Podcast

Lightning Rounds

* Favorite AI Product: Midjourney

* Favorite AI Community: Eleuther and GPT-J

* One year prediction: Better models, more creative usecases

* Request for Startup: Superathlete Fitness Assistant

* Takeaway: Continue to tinker!

Transcript

[00:00:00] Alessio Fanelli: Hey everyone. Welcome to the Latent Space podcast. This is Alessio, partner and CTO in residence at Decibel Partners. I'm joined by my cohost, swyx, writer, editor of L Space Diaries.

[00:00:20] swyx: Hey, and today we have Varun Mohan from Codeium / Exafunction on. I should introduce you a little bit because I like to get the LinkedIn background out of the way.

[00:00:30] So you did CS at MIT and then you spent a few years at Nuro where you were ultimately tech lead manager for autonomy. And that's an interesting dive. Self-driving cars in AI and then you went straight into Exafunction with a few of your coworkers and that's where I met some of them and started knowing about Exafunction.

[00:00:51] And then from out of nowhere you cloned GitHub Copilot. That's a lot of progress in a very short amount of time. So anyway, welcome .

[00:00:59] Varun Mohan: That's high praise.

[00:01:00] swyx: What's one thing about you that doesn't appear on LinkedIn that is a big part of what people should know?

[00:01:05] Varun Mohan: I actually really like endurance sports actually.

[00:01:09] Like I, I've done multiple triathlons. I've actually biked from San Francisco to LA. I like things that are like suffering. I like to suffer while I, while I do sports. Yeah.

[00:01:19] swyx: Do you think a lot about like code and tech while you're doing those endurance sports or are you just,

[00:01:24] Varun Mohan: your mind is just focused?

[00:01:26] I think it's maybe a little bit of both. One of the nice things about, I guess, endurance athletics, It's one of the few things you can do where you're not thinking about, you can't really think about much beyond suffering. Like you're climbing up a hill on a bike and you see like, uh, you see how many more feet you need to climb, and at that point you're just struggling.

[00:01:45] That's your only job. Mm-hmm. . Yeah. The only thing you can think of is, uh, pedaling one more pedal. So it's actually like a nice, a nice way to not thi