Managing AI Coding Costs at Scale

Over the past couple of years, companies raced to get their employees using AI, often encouraging them to lean on it as much as possible. However, as AI coding tools spread through engineering teams, many companies watched their AI bills explode. That kind of runaway cost growth can threaten a business, so the new challenge is capturing the productivity gains from AI without the spending spiraling out of control.

Databricks is a data and AI company that hit this problem early, running AI tools across thousands of its own engineers. Rather than clamp down on usage, the team built a set of practical ways to manage the cost, from taking advantage of steadily falling model prices to automatically routing each request to the cheapest model that can still do the job well.

Patrick Wendell is a co-founder of Databricks, where he leads engineering teams and has driven much of the company’s internal AI adoption. He recently wrote a viral blog post on managing AI coding costs that reached the top of Hacker News. In this episode, Patrick joins Kevin Ball to discuss why AI costs suddenly exploded, the main levers companies have to bring them under control, the trade-offs between managing costs on the server versus in the developer’s tools, and why the next big challenge is giving AI the right context about how a company actually works.

Sponsorship inquiries:
[email protected]

The post Managing AI Coding Costs at Scale appeared first on Software Engineering Daily.