0:00 That was probably one of the first big magical moments for me with using agent harnesses. This is Adios Mani. 0:07 He's a director at Google Cloud AI, famous for his work on Google Chrome and more recently focused on AI agents in software engineering. I'll delegate any technical task to my harness and and see what it does. 0:20 I can give it a goal and it will be able to figure out a path to accomplishing that goal. Record it live at Google IO. 0:27 We talk about agents, harnesses and the back on software engineers. Good. 0:32 And bad. We can also start to lose our muscle memory around how things get done. 0:38 Cognitive surrender. And I think that what you need is. Have you experimented with like Open Claw and Hermes? Oh yeah, yeah, yeah. 0:48 What's your thoughts so far? I think that in the early days, I had a lot of hope and I got, I felt I felt productive. 0:56 But after a while, having to babysit the whole setup became a little bit more than I was expecting. 1:02 Yeah. But I feel like being able to have it connect to so many different third party services is like a big on luck because you can begin to like delegate things you'd usually do yourself. 1:13 And it's easier if you're just a single person trying to do this stuff, because if you're a team or a business, that's very, very, very different, you know? Yeah. 1:21 This term babysitting agents is something I hear from people that are kind of taking a dab into working with agents in a gentic. What would be the biggest thing they need to do to no longer feel like a babysitter? 1:32 I think that there's probably two or three things. 1:35 So the first is if you are building a personal agent and you're giving it access to, you know, your credentials or you're sharing tokens or anything like that, you, you probably have a level of comfort about things going wrong. 1:48 Maybe you don't, you know, there are people who've done this where they don't really know what they're doing. 1:52 If you were trying to graduate from babysitting to something more serious, you kind of need to have identity around your agents. So if you've got lots of sub agents going off and doing different things, you kind of know well what is doing what. You kind of want governance of 2:08 some sort around them as well so that, you know, if agents can do anything, you want to limit the blast radius because we have seen lots of cases of them going off and doing things that you didn't really expect. And you know, I've, I've heard 2:20 people telling me examples of like, well, I don't, I, I give it access to, you know, Google CL is, is it going to delete my Gmail or is it going to delete my customer data or anything like that? So you really just want to think 2:30 about how they can help you be effective while also mitigating any of the crazy things that can happen. 2:36 Whenever I hear kind of people working with agents and they're against it, they give the biggest example or someone give it access to production and it deleted XY and Z catastrophic example. Absolutely. 2:46 And I think the challenge is that, you know, there's people feel a lot of FOMO right now, right? 2:51 You, you see so many people experimenting with their own personal agent set up and they'll say like, Oh yeah, I've never been more productive. I'm having it do like half of the stuff I'd usually do myself. And that sounds very impressive 3:03 on the surface. But then somebody who doesn't really know what they're doing, you know, goes and does that and they could easily leak API keys. They could end up with a very large, you know, bill at the end of it, those types of things. 3:14 And so I just think people need to exercise care while also experimenting, you know? From a tooling perspective, do you think it's the tool's job to kind of lower the floor a little bit? Easier barrier of entry? 3:26 Better agent experience. I guess absolutely. 3:29 There's a great quote by Grady Buch, who is the creator of UML, and he says that the history of software engineering is of rising abstractions. And we have that moment every time new technology comes out. So, you know, we we have 3:46 JavaScript as a language. Well, then we have frameworks and libraries that sit on top, make it easier for people to to build on top of it. Now we have agents and harnesses and models and tool calling and all this stuff. 3:58 And so I think we're going to see increasing like more abstractions that sit on top that just make it easier for people to not shoot themselves in the foot and ideally follow those best practices as well as they can. 4:08 I I know people are experimenting with different harnesses. Last year for me was the experimentation year for agents and this year it's more harnesses specifically. What have you seen that in harnesses work really effectively if people master 4:21 that from a tool perspective? So I, I, I'm, I'm always happy to tell people what I use. I'm, I'm a very big fan. 4:28 I'm a very big fan of lots of different harnesses, including the work that Google is doing with anti gravity and a lot of personal projects. I also love using Claude code. 4:35 Shout out to Anthropic and the team there. 4:38 I think that one of the things I love about a good harness is that I can give it a goal, and regardless of how specific my instructions may or may not have been, it will be able to figure out a path to accomplishing that goal. 4:54 That was probably one of the first big magical moments for me with using agent harnesses. So for example, a lot of people when they think about agent harnesses, they think about the eugenic coding use cases, right? Like we're programmers. 5:07 That of course, is one of the first things we think about, but increasingly for me over the last couple of months, I'll delegate any technical task to my harness and, and see what it does. So for example, if I have a 5:21 directory of, you know, I, I, I do EDM music on the side, right. 5:26 So if I have a directory full of WAV files and I know that well, I didn't actually add a nice fade out at the end, but I want it to be very natural at the point when you decide to do the fade out. I could spend a long time doing that manually. I decided to give that task to 5:44 Claude code. It figured out what tools to download. It figured out how to determine what is the correct point in each track to begin the fade out, and then it installed the tools necessary to actually process and make those changes. That's a small thing. 6:00 It's a very small thing, but the fact that you can now delegate any of these tasks to an agent and have them actually complete them, kind of amazing, right? I'm not telling it what tools to use. I'm not telling it where to get these tools. I'm not installing any extra 6:15 skills for these kinds of tasks, but they still figure them out. 6:17 Something amazing about that. Yeah, fantastic. 6:20 From a harness perspective, the thing we build around agents or kind of the capabilities we have in place for agents to thrive, it's very comparable how I see it in practice with people falling in love with their first IDE. 6:33 This harness is my favorite, right? 6:35 I've seen this is the most effective marketing wise or community wise. This is what people are using either in their work or in their personal projects. 6:42 Would you recommend people experimenting with different harnesses and what would the gain be? 6:45 Absolutely. I think that if you take a look at every major model lab or developer tooling company, if they are vending an agentic engineering solution, an agentic coding solution, they're probably offering you up some version of a harness. And behind that harness is the 7:04 model, right? So an agent is the model and the harness together. And so very often, you know your perspective is going to be, well, if I'm using the harness that comes from that company. Surely the combination of them having those teams all working together, it being optimized, 7:21 the model being optimized for the harness, all of those things will lead to the very best possible outcome. 7:26 Maybe that statement holds true, but it's also possible, especially if you can use that model, whether it's your subscription or your API keys with a different harness, then you might get even better outcomes. 7:37 And and so for example, open code or there there are a list of other open source harnesses people can try out. 7:44 But I think that the value there is really just understanding what is possible. Even if you decide ultimately to stick with a harness that you know the main company is giving you because that will give you some idea. 7:54 Well, maybe they will go in that direction in the future. 7:57 And if they don't, I can always go try that harness out myself to get that extra power. Yeah. 8:03 For me, the the industry moves so fast that for people that say, well, I can't really keep up with what is in in this from an industry perspective and I already have my hands full with kind of day-to-day job and I don't have bandwidth to experiment. I feel like it's OK to just 8:17 focus on one harness because I also see harnesses learning from what the other capabilities are doing. 8:21 Absolutely. We had a panel discussion earlier where we were talking about how a lot of us feel FOMO about our tools today, right? Like every other week you'll see some update where you're like, oh man, you know, I, I just switched to cloud code, or I just switched to codecs or I 8:38 just switched to anti grad and I have to go and switch over to another tool. And what I was telling people I think applies to harnesses as well. 8:46 Think about the time that you take to invest in trying in another tool as having an innovation token or an innovation budget, right? You don't have all the time in the world. So be very explicit about where you spend that experimentation time. 9:03 If there's a harness you're using or tool you're using that is actually delivering you good value right now, stick with it. 9:11 Similar to how orchestration U XS are all kind of converging on the same look and feel. I think we see that harnesses are also in places converging. And so even if you know, you feel FOMO about, oh hey, I heard good things about that other 9:27 project or that other harness the patterns that you're building up the muscle memory you're building up with your current one will probably still apply if you decide to switch at a later point. So I'm always looking for not 9:38 what is converging, but what is like the thing that people are trying out that is different to everybody else, right? 9:45 Like is somebody trying out something in a completely different way, even if it looks crazy, that'll at least let me understand. Well, is there value to that other idea that's like left field, or I'm actually right to just stick with my current tools? 9:58 Yeah. This feeling of FOMO really resonates with me because a year back I had a role as product person. I always wanted to do that. 10:05 It's very close to engineering. I wanted to be responsible, make decisions, drive a vision, etcetera. 10:10 And then specifically kind of Q3, a lot of agentic capabilities happened last year and I felt exactly that FOMO. 10:17 I went back to a hands on role because I felt like things are evolving so quickly that if I am not hands on like on a day-to-day, I feel like I'm really going to be lacking and I'm not going to be able to resonate and have conversations about what is kind of cutting edge. 10:29 I really felt that. Yeah, I think that's, that's a feeling that's difficult to shake because very often the industry will paint things in these broad strokes. 10:40 So if you go on X for example, right, you can feel FOMO very heavily. You scroll through your feet and it's like, Oh my gosh, there's like 2 new harnesses this week. 10:50 There's new models, new open source models, new patterns, like new skills. There's so many things that you could potentially try out. And I think it's important for people to just like calm down it calm down just a little bit. 11:03 And also realize that there's a big difference between, you know, the risk that a person who's, for example, a solo founder or a three person company can take trying out like all these new things, especially if it's in production and what a 11:19 slightly more established business would maybe be able to do because they need more guarantees in place and security and privacy and all those other things need to be thought about. 11:28 But at the same time, I think the way I look at it is I always try to stay on top. I always try to read, even if I'm not trying out every single thing. 11:35 I try to read other people's experiences about it because it might be a leading indicator of what's going to come. 11:40 So you can still feel your way through the FOMO a little bit even if you're not trying out everything, I think. 11:46 One of the biggest kind of accelerators in my career was working in collaboration and a big component of that was literally just pair programming. Sitting next together next to someone physically or when COVID happened way more remotely, 11:59 watching someone do their thing, hands on keyboard or just from an ID perspective, navigated things, how they solve problems, thinking before acting, etcetera, etcetera. 12:08 I feel like we are a little bit losing that because people are way more pairing with agents, right? 12:14 They put their ideas into a terminal, answers come out, they vibe like that way more. I also do that, but I feel like then that's maybe a miss because especially if things are accelerating and learnings are to be had everywhere. 12:26 I feel like as soon as I see someone do that I can immediately ask questions and get a feeling for what someone is doing. There there are a few topics around this that I've I've talked about recently. 12:37 One of them is cognitive debt. And so especially as we get used to the fact that code is getting generated so quickly right now, we can also start to lose our muscle memory around how things get done right. Like, it can erode our memory, 12:53 it can erode our skills because we now feel comfortable having an AI like always available there to help us. 13:00 And for a lot of things that can be OK, but we're also not being very proactive about, well, how am I learning everyday? 13:07 How am I staying a little bit closer to what's getting generated so that if I have to, I can roll up my sleeves and go in there and make the changes I need to just in case the agent isn't able to, to help out with that. 13:20 And the other end of that spectrum is what we're seeing more and more of these days, which is cognitive surrender. 13:26 So you, you, you just stop having critical thinking at all and you just blindly accept, well, whatever the, the agents generated is what I'm going to commit. 13:34 It's what I'm going to ship, and if anything goes wrong, well, I don't need to worry about it because I'm just going to ask the agent to fix it. And that's an OK mindset. 13:44 But there's a challenge there. How are you going to tell whether the agent actually got those fixes right or not right? 13:51 So generation is now easy. Verification is a big thing that I think that we as engineers need to keep a close eye on. 13:58 Verification. If we solve that problem, we also solve. I feel like a current bottleneck of kind of manually. Reviewing code. 14:06 Yeah, yeah, yeah. That I feel like is the current bottleneck that we are creating. Yeah, I would like to have the confidence that I have so much built in place static code analysis to make sure my code is simple, easy to change. 14:18 Few other, I think more in depth analysis and kind of dependencies and yeah, more of the harness, maybe even orchestration around a harness specifically that I have the confidence that I don't even have to yes. 14:30 Absolutely. I was talking to a a founder earlier. It's like a basically a one person company. They were telling me, you know, I used to do a lot of coding myself and then I started using clawed for a lot of stuff. And now I have to do all of the 14:45 things that a business owner has to do. 14:47 So I don't have enough time to spend actually reviewing the code. I'm terrified of, you know, the agent doing something wrong and then that just getting, you know, having a domino effect over time, right? 14:59 And so they're asking, well, what, what should I do in that situation? And I think that what you need is to codify what good and what right means into your system. 15:09 So that can be everything from your user journeys to your tests, to your visual regression testing, to, to anything that helps ground itself in what right and wrong is. 15:22 So that even if you're just using the agent to keep doing these turns and changing the code, it at least has something to compare against. Because otherwise, like, what's it going to do? Maybe it will fire open a browser and load up the app and then click around and see, OK, 15:37 well, I guess I'm able to register. 15:39 I guess I'm able to do these actions, but that's not the same as being able to verify that the behavior hasn't changed from one release to another. So I think that really what we're talking about is just keeping good engineering hygiene 15:53 in place as we're using agents. Yeah, absolutely. 15:56 This core ingredient that is what does good behavior look like. I think it's pretty funny that you can do that in a specific agent that you build. 16:03 You can do it in the skills. You can build around the harness specifically. You can assume some stuff from a harness perspective is already in there. 16:09 We now have a spectriven development. 16:11 You can do spectriven development, but tests 1st and do behaviour driven tests with so many options. 16:17 And I feel like the innovation budget that you talked about, it's incredibly important that you spend your personal time in there. But I also think it's important that companies need to invest in the teams they have and also 16:26 allow them to innovate because right now we need to experiment. 16:30 Yes, I completely agree with that. 16:32 And I think that what we often see is a lot of developers saying, well, I feel I feel this pressure right now around velocity just need to ship faster, ship more speed is of the essence. And so they worry that they're not going to have as many opportunities to to learn on the 16:51 job, to experiment, to get better at what they do. 16:55 Running multiple agents does not mean that there is more of you to go around because you have cognitive bandwidth that doesn't parallelize. And so I think what's a very important is for people to figure out what is your comfort 17:13 level with using agents, with parallelizing your work and don't push yourself too far beyond that. 17:20 For me, I'll usually split my work into two buckets. 17:23 There's usually things that are isolated tasks I'm very comfortable delegating and maybe I'll throw those to cloud agents, background agents. And then there are tasks where I really want to use my cognitive bandwidth to pay closer attention. So that can include everything 17:38 from just working more closely with the local agent to looking at let the code to just paying more attention to the details and the quality of the outcomes. And so just be very intentional I think is one piece of advice I would say resonates maybe 17:51 people. Thank you so much for coming on and sharing. My pleasure, thanks for the chat.
Community discussion