New models and agents

From model choices to Valhalla

New models and agents

Sebastian: It performs at the level of Claude Fable 5.1 on most work and costs 40% less than Opus 5. Right? So you basically have Fable-level intelligence at cheaper than Opus level.

Not even close to a prototype of what I was trying to achieve. And then Sonnet 5.5 comes out, and over basically two hours of cranking at it, I was able to create a fully playable game.

If you're using Claude for most work, day to day, I would actually use Sonnet 5.5, which is so crazy to say because Sonnet 5 was one of the worst models I've ever played with in my life. Sonnet 5.5 is just fantastic.

Have you been playing around with what's new? Because it's been a dramatic couple of weeks.

What changed

It's the first time I'm recommending Claude models for several months. I'm spending a lot of time in Claude now.

Basically, what I want to do is two parts for today. One is jumping into all the changes that have happened: what's new, looking at the different models and what they mean and how good they are. And then looking at some of the new agents, because agents are now here in a big way, and there's a bunch of different tools. They all came at the same time. I've been playing around with the different ones, so talking about what that means.

Then we'll try and contextualize what this step means by looking at something I built, playing around with it, and talking through how I built it.

To spoil the ending, Fernanda, if I remember correctly, you were talking about how easy it is to get to 80%, but those final bits are really, really hard. I think if I had to summarize what this moment was with these new models, it's that now, without dropping F-bombs and all this really hard friction, they can just build the thing that you're trying to build. I'll hopefully illustrate it with what I built, and we can play around with it and look under the hood.

Astra

All right, so Astra. Astra came out. It was a pretty seismic thing.

Astra was OpenAI's answer to Fable. It is a new class of model, right? You have the heavyweight frontier models. Astra is one of these. It was comparable with Fable, but I think its strength that is the most compelling is its ability to use your computer and use tools.

What Astra has really enabled is some of the crazier things within 3D. This was before Claude made some announcements. Being able to use your computer to manage video editing software, using it to use Blender and 3D software, and just being able to control your computer in a way that is pretty remarkable. It opens up whole new avenues of different things you can do with it.

Astra was a really big deal and created all kinds of new opportunities. I think that was the thing that was a little bit hard to understand about the model, because in some ways it's not as good as Fable, but Fable wasn't able to unlock these new paradigms.

For folks who didn't watch the video that we shared around the Kingston talk, I think my thesis around some of these new models is: yes, they're super intelligent, but the thing that they unlock is the ability to play in new technical domains. Coding is the one that we're all playing around with. That's their technical domain. But now it's like, I can play around with 3D.

We had talked about what happens when everyday people can use 3D. Now we're starting to see that people are creating interactive explainers, using it for education, for architecture, designing their interiors, for sales and marketing. All sorts of new applications that 3D designers or CG artists never used to do. All these new technical domains are possible now, and that's really exciting. Astra was a big breakthrough there.

Opus 5.5 and Sonnet 5.5

The news kept coming. Then we had Opus 5.5. We had Fable, this unbelievable model. Probably the best model at coding, just the smartest model we have, really a workhorse, but also almost unusable.

Unusable because it was so expensive that essentially you can have a $200-a-month plan and pretty much burn through all of your credits in about an hour or two. You're like, wow, this is amazing, and I can't use it to do anything real. You could use it for planning, but really you couldn't use it for anything.

I'm not gonna go through the whole thing here, but this is what I wanna call out. You essentially have Fable, the best model in the world, come out, and then within weeks they release Opus 5.5, the first model in our new Claude 5.5 family. It performs at the level of Claude Fable 5.1 on most work and costs 40% less than Opus 5. Right? So you basically have Fable-level intelligence at cheaper than Opus level. That was just like, wait, what? How is that even possible? How are they basically cannibalizing their own model?

For Fernanda, I would actually recommend that you almost never use Fable because I don't think it makes sense. Opus 5.5 is pretty much as good as Fable and dramatically cheaper. You're not going to see dramatic increases unless you're doing the absolute most complex work. I would say the only thing would maybe be planning or a really difficult bug thing, but really Opus 5.5 is it.

They do that, and then I think two weeks later, or even one week later, they release Sonnet 5.5. I thought I had Sonnet. I maybe don't have it up, so I'll just speak to it.

Sonnet is just as crazy because Sonnet is actually also at Fable-level quality, just a little bit less. But on many tasks, it's actually better at some things than Fable. I was reading this guy looking at it, but basically for 3D, doing these crazy 3D worlds, Sonnet is actually better than Opus and Fable at doing it.

It's kind of confusing, but the reason we're seeing these dramatic upticks in capability is that they're probably training these new models with their latest unreleased model. It's probably a Mythos 5.5 or something like that. It allows them to release really capable, cheap models very quickly.

They brought out Sonnet, and now you basically have Fable-level intelligence with Sonnet at staggeringly cheap prices. I've basically been running Sonnet nonstop on hard engineering tasks for two weeks straight, and I can almost not get through my usage limits. I can run it all day, all night, and it just keeps going. I still haven't been able to get through my $100-a-month plan. The cost reduction is staggering.

It allows us to do real engineering work, really hard things, and dream bigger without having to be so worried about cost. So where does that land us after a lot of words?

I would say that if you're using Claude for most work, day to day, I would actually use Sonnet 5.5, which is so crazy to say because Sonnet 5 was one of the worst models I've ever played with in my life. Sonnet 5.5 is just fantastic. Only with the hardest things do I then go to Opus.

The reason I like Sonnet 5.5 is because I get the same kind of quality, but it's wicked fast too. I don't have to worry about cost. It performs really, really fast, and it's just a great model. I love it for all engineering work.

That being said, for my day to day I'm still using Codex. My preference there comes from the harness or the tool. Everything feels organized. I know where stuff is. I'm happy there. For most of my knowledge work and my day-to-day work, it's fine. But when I'm engineering, I always jump into Claude.

Mary: It's amazing too. We were working with a client, and truly on a Tuesday or whenever it was, we were saying, "Don't use Sonnet because it's not powerful enough." Someone truly was like, "When I use Sonnet, it loses track of my context. It doesn't follow a thread."

Then the next day we had to go into their Slack and be like, "Just kidding, forget everything we told you yesterday. Sonnet 5.5 is exactly where you want to be working."

Sebastian: Yeah, it's pretty crazy for corporate clients because you essentially build a relationship around what AI can and cannot do. You just wake up next morning and have access to models this much more powerful, and they have to reevaluate everything they know about them.

We have extended relationships with these tools, so we know things shift. But that's not a normal thing to wrap your head around. Wait, I wake up next morning and what I can do is infinitely bigger and better. It's quite cool to come into a workshop and be like, "Hey, have you tried it? Maybe give it a shot. You'll be able to do some cool things."

Here's Sonnet 5.5. I did have it. Introducing Claude Sonnet 5.5, the second model in the Claude 5.5 family. It's a clear upgrade over Sonnet 5, runs 30% faster and costs up to 30% less for most work. It does hit about Fable-level intelligence. That happened. That was pretty crazy.

Participant: Sorry, while you're at it, that's amazing news. Super excited about it, and definitely been noticing how this is awesome, especially for coding. What I'm trying to start doing a little bit more of, and it's not my area of expertise at all, and I'm finding the quality of the work a lot worse, is videos.

Sebastian: Editing videos?

Participant: It doesn't QA the video. It's very bad. It doesn't know how to watch your video. It plans it, but it seems to plan it conceptually, and then the execution turns into something a little nonsensical.

The imagery is like, I gave it feedback, "Looks like you made it out of Paint," like '90s software. It's really, really bad. All these things that I'm not seeing in coding. I'm like, this is a rocket ship. And then for videos, part of that is me because I have no idea about the video world, but what have you found that has been working well for you?

Sebastian: Just so I'm clear, what kind of videos are you trying to make? Is it editing of videos or is it motion design type stuff?

Participant: It's more motion design style stuff. It's not editing. It's definitely creating a few things for Instagram, but where I will need to go next is educational videos, content. I have a bunch of content written, and I will need to explain those things through video.

Sebastian: A use case like that is the major breakthrough with the Opus and Sonnet models, and I would actually recommend using Sonnet for this. It should be unbelievable at it. I can send a couple of examples of the kind of video explainers that it's able to do, but it's pretty wild.

A couple of things that you would want to make sure: I'd recommend that you have HyperFrames and give it the HyperFrames tool. HyperFrames is a plugin that I've talked about a little bit before. It's basically a video editor for agents. It gives it a little bit more scaffolding around being able to preview videos, jump between frames, compile things, and also for you to work with it and give it feedback in video format. HyperFrames is something to take a peek at.

Then also, probably a thing that I'll be demonstrating a little bit later on, is finding some open-source motion examples and pointing it to them. That might be the unlock where it's like, okay, this is how I make a good motion video. Let me go with that.

GPT-6.1 Sol

A couple of days later, we have GPT-6.1 Sol. For folks who are ChatGPT users, this is a very interesting model. This is kind of ChatGPT's answer to maybe Opus 5.5. It is essentially a model that is almost at Astra level, but its price is even cheaper than Sonnet. Its price is just ridiculous. It's really, really crazy.

You've basically gone to a world where they release these massive models that are super expensive and really capable. Then within weeks they're releasing models that are cheaper than any model we've ever used before, but at the same capability as these frontier models. So 6.1 is a phenomenal everyday model.

It's kind of crazy because we spend a lot of time learning how to route to different models. For this kind of task, let's go to GPT Terra medium, and then for this task, Luna Max. This model is so cheap that you can pretty much do it for your hardest tasks and your simplest tasks. For almost anything, if you're in the chat universe, make sure you're using ChatGPT 6.1. It's just a great model and wildly, wildly cheap.

Consumer agents

Let's jump into the dawn of consumer agents. The first one that came about was Grok Bot. This is by the xAI team, and actually it's by the people who did Cursor. Cursor is a really great product, so they have a really good design team.

The thing that was a real breakthrough and what made this so popular overnight was that it is agents, but easy. There's almost no complexity in there. You open it up, you create a little agent and assign it a task, be it check my emails, check my Slack, or help me with planning my kids' schedules. It learns over time. It hides a lot of the complexity of building agents and skills and all of that. It just does it, and it does it well.

This spread like wildfire in the community, and people are still using it a decent amount. It's something to check out. In this environment, if you are a ChatGPT user, I'd probably recommend playing around with dots. You don't need another service. But if you are a Claude user, this is an interesting alternative. It's very good.

Shortly thereafter, Meta released Muse. Muse is their answer to this. We don't have this in the UK, but it is very similar to Grok Bot. An agent that works on your behalf.

I think the big paradigm shift with agents is proactivity. The way we've been working with AI is you assign a task, it does it, you have to check back on it. You leave your computer, it doesn't do a thing unless you've had it work. Agents are now proactive and working on your behalf.

The thing that's crazy about Muse is that it's by Meta, which means it's gonna be a consumer AI for hundreds of millions of people and probably one of the dominant consumer, non-corporate AI agents that people use, just because Meta is so pervasive.

The thing that's surprising to the industry is how good it is. It's actually really well done. They've had a really amazing design team and product team on it, and the details are really well executed. It's a really good product, and people have had really good experiences with it. We don't have it in the UK, so I haven't tried it, but from folks I trust, they're like, "This is legit."

That being said, Kevin Roose is a guy I like. He's a journalist and author, and his take is, "I'm sorry, it's a fun product, but if you hand all your data to Muse and you didn't expect it to be used in the most invasive things imaginable, you're Charlie Brown with the football and you deserve whatever happens to you."

This is asking Meta's Muse, "What happens to my data?" Me and Mary have worked at both Google and Meta. They make their money through data and have really pioneered that economics. In a world where we are giving and offloading so much information about ourselves to our AIs and our agents, you want to think about the business models that these model providers are engaged in. Anthropic and OpenAI have a different business model, though I wouldn't say they're saints. Something to take into account when you're using Muse or Gemini.

Participant: That was a concern. I wanted to test Muse and Grok, and what held me back was exactly that. I already have a ChatGPT account, I have a Claude account, I have a Gemini account. For them to be useful, they need context and information. But if I don't hand over my data, I won't be able to test those tools at all.

Sebastian: I think it's for all of us to decide. This is how I would think through it, right? So much of my data is already in the Google ecosystem because of Gmail, because of work apps. I'm not as worried about using Gemini because it's part of an ecosystem that already has so much information about me.

OpenAI and Claude, the utility is so dramatic that I've now built these deep relationships that are high risk, but there's a payoff, and that's fine. Moving into another tool that I haven't built that trust with, whose companies I'm not necessarily aligned with as far as their leadership, it's a decision one would have to make.

For me at this point, I will play around with these tools, but with Grok Bot and Muse, I'm gonna be more conservative because I wanna reduce what they call the blast radius.

dots

Now OpenAI is playing in the world of agents, and that brings us on to dots. I think there are some things that make OpenAI really well suited to have a really good solution to agents. Ultimately, so much of my life is already there. It's not a big jump to extend it into their solution.

Several months ago they hired Peter Steinberger, who pioneered OpenClaw. He's obviously been working in the background, and a really strong team has been working in the background to think this through and get it right. Granted, everybody in the space has done a really good job. I think they've been thinking about it really hard because it is a hard space. It's a hard problem. But I think they've done some really good initial approaches.

The thing that I really like about dots is, one, it's already in your ecosystem, where your work happens. Two, the problem with Muse and Grok Bot is that the models underneath the hood are not that great. They have a great tool. It's really user-friendly and beautifully designed. But when you need a little bit more intelligence, it's just not quite there.

The thing that I think is really compelling about dots too is that it's a hybrid between your work life and your personal life. The way I've been setting it up is it's active on all of my projects. A session back, we talked about these unithreads: basically in Codex, having one chat thread where you do everything, and it will then shoot off tasks to other threads to do the work.

dots is really that paradigm. You start moving into one chat, which then delegates to different tasks. You can either jump into that task, or it just comes back with what you need. You can do it for your personal life. You can do it for your work life. Under the hood they're using Astra, which is their absolute frontier best model.

Their experience was that you needed a certain level of intelligence to actually do useful things in the real world. If I'm trying to buy theater tickets with an agent, it needs to be able to use browser use. It needs to figure out what's going on on the internet, understand all these different connectors. What's your calendar? Have you got a conflict with something? What was the email thread where you were talking with your friends to organize?

It's actually quite complicated, even simple things. They need this really amazing model. dots compared to the other ones is using a way, way more capable model than any of the other ones. I actually think dots is a very compelling offering and something definitely to check out. Unless you absolutely hate dots, I wouldn't see why you would need to jump onto the other ones at this point.

If you click that link there in chat and then click "Create your dot," you should be able to do it. But right now you do need, I believe, a Pro account. Right now in the UK you can't access it at all other than with a Business Premium account. Otherwise nobody can access it.

Building Valhalla

What I'm gonna show next, in the last twelve minutes, is to explain what this moment means through one of my vibe coding projects.

I started a project. I had an idea a while back, maybe two months ago, where I'd seen somebody make this really cool 3D holo card. There's two very popular games on the iPad. One is called Hearthstone, and another one is called Marvel Snap. I thought, a lot of people are making games. It could be fun to play around with making a game.

I tried with Fable, and I think I tried with a bunch of models, and wanted to basically recreate the game Marvel Snap. Let's show you really quick what that looks like.

This is Marvel Snap. The idea is it's kind of a card game like Magic: The Gathering or something like that. It's based on a German board game where you have three bits of land and various cards. They have cool cards, lots of cool little illustrations and really fun little effects when things happen, weird zappy things or whatever. I was like, yeah, it'd be cool to do.

When I was first working on it, I couldn't even get it to do a 3D card turnaround. I had to scope it back all the way to really basics. Then when Sonnet 5.5 came out, one morning I was like, "Hey, you know what? I'm just gonna revisit some of my vibe coding projects."

What I will show here is about a morning and then an AI psychosis-infused twelve-hour shift where I was working on it on a Saturday from twelve to twelve, and I don't think I went to the bathroom.

I was thinking about Fernanda, your thing around it's easy to get to 80%, and then the last 20%. I was at 80% after an hour or two, letting it run in the morning. Then I spent that twelve hours getting it to where I was happy with it. The design, the functionality, the details. That is where the hardest bit is, right? The last 20% is 80% of the work.

I'll do a quick demo, and actually if you guys want to play it, I'm not gonna force you to play it, but here it is. I'll just demo it.

Essentially you start, you can choose one of three cards in the deck. I will choose Thunder of Asgard, and I'll do it easy because it's kind of hard. Level one, you get one bit of money to buy something, and then next level you get the other. You see all the artwork, all of the VFX. This astonished me.

Actually, I'll show you one other thing. I was able to build these twenty-five cards, and then when you click them you have 3D effects like that. It kind of blew my mind because I was like, wait, a week ago or the day before, I wasn't able to get even close to a prototype of what I was trying to achieve. Then Sonnet 5.5 comes out, and over basically two hours of cranking at it, I was able to create a fully playable game.

If I really have to coin it, it's like we've moved into this world where you were able to get to these results, you'd be able to build these tools, apps, services, and websites. But to get it fully functional, to actually be something that you can ship, with these new models I think we've transitioned to a moment where you can pretty much build almost anything you can imagine and do it relatively affordably.

This example is a use case in what this moment is. It's like, okay, my ideas can actually become reality now. I think that's a massive shift. I'll talk a little bit about some of the things that were involved because designing games is a lot more complicated than regular software. There's so much involved.

How it was built

I made a slide deck. I don't know if it's any good because it just finished before, but it was to talk a little bit about the journey.

This is starting off where it started. A lot of it was about trying to create concept works. I'll tell you a little bit about the idea. Marvel Snap is about the Marvel universe first. I really love history. I was like, what if Thor and his universe of Norse mythology was actually rooted in historical accuracy?

There's this great book called The Children of Ash and Elm, which is about the history of the Norse and the Vikings. It talks about their clothing, their mythology, how they lived, and all of that. I was like, what if you made a game that would make a historian proud?

A lot of it is background research. All the characters, all the different sections, all the mythology are based on history. How they dressed, what they believed, and all of this.

This is a rough cut of all of the assets that were created. There's tons and tons. That's the thing that's also crazy about it: there's hundreds and hundreds of artifacts. Each card has frame artwork, it has its own artwork, it has background artwork. It's over 100 jobs on Comfy Cloud using fal.ai.

The way I was working with the image generation was you have your APIs. They're connected to either Comfy Cloud or fal.ai, which are the generative video and image model providers. What that allows you to do is create assets in natural language and create assets at scale.

There's a reason games take years to create. In the real world, every one of those assets would have to be illustrated, and then you'd have to do massive amounts of production work on getting the artwork formatted for gaming. Being able to use your agent to call your image APIs, be like, "Hey, give me back a bunch of different versions. Let me see what ones I like, what ones I don't."

Then have it compile it and do all the production work, getting all the assets the right size, the right format, the right layering. It's really complex and tedious work that an agent can just do for you.

This is how it all started, 8th of August. "Would you be able to create a Marvel Snap..." I wanted to say "clone," but it said "close." "But instead of Marvel, it's Norse gods and Valkyrie."

In the background, you can see a faded thing of creating some concept boards using ChatGPT image gen. "Hey, can you create an illustration of what the game could look like?"

These were the decisions, right? Marvel Snap, Children of Ash and Elm, desktop and tablet, some early artwork. Every card: you have the concept art, you have the name tag, that's its own thing. You then isolate the image from the background using another model that's provided by fal.ai. Then you have the background asset, and you compile it all into a holo build like that.

Here's some different artwork. The different name tags. Then some issues with cutting out characters, so playing around with that. See what else we got. This isn't a hallucination. Actually, this horse did have seven legs, I believe.

Then complexity around building the grid and framework for how the cards are laid out and how to play around with breaking the mold. That took a lot of back and forth.

One other thing in the last minute that I wanted to show was what's really helpful with this too: the models themselves can do a lot of powerful work, but often they don't know everything. What can be really helpful is to have an open-source project that you can point it at. Two that I did point at was this one here. I believe another person made an open-source project like that. I pointed it at that open-source repository on GitHub and was like, "Hey, could you look how they did that? Can you do something close to that?"

There's another open-source repository, this one here, which is a VFX sandbox built with Three.js for crazy spell casting and effects. For a lot of the game card effects, I pointed it at this one. I was like, "Hey, could you look how they do this?"

Because what it was trying to do, some of its VFX were truly horrific. I did have to drop some F-bombs there. I was like, "What on earth? Do you have eyes? Did you look at what this is? This is so bad." This was decent, so I pointed it at that, and that really helped with effects.

The other thing I did was point it at a video. I gave it this video, told it exactly where those VFX happened, and then had it take frame-by-frame images of every single effect. Then say that's the quality bar. I wanted it to look at how this was built and how good this was.

This is a ripoff of an existing game. That's something I wouldn't go out and sell. But similar to artists, what we'll often do is copy the greats and understand how something was painted and really analyze it. You learn so much from deconstructing something that is world class.

That's the time we've got. Basically, we're in a whole new paradigm. What you can build is night and day from what you were able to build a couple of weeks ago. I call this out a lot, but dream bigger and try some really ambitious projects.