Browserbase: Fully Autonomous Is the Wrong Goal, 99% Accurate AI Agents Can Fail Catastrophically

Transcript

Gabe Tramble (00:02) So Derek, can you tell us about what a browser agent is and how it's different from like a playwright script and then also queue into browser base itself?

Derek Meegan (00:11) For sure. ⁓ So first of all, you want to start with what is an agent, right? And so we all know what a large language model is, or ⁓ maybe hopefully everyone that's listening to this knows what a large language model is. But an agent is essentially a large language model on a loop, right? That is sort of continuously operating to achieve some sort of objective. And there's sort of varying degrees of autonomy for an agent. ⁓ You may have an agent that checks back with you on every single turn. You may have an agent that's set up for long horizon tasks, or you may have an agent that is calling tools that individually require human in the loop for approval. And so a browser agent is just an agent that operates on a browser. Just like a human does, it opens up ⁓ a Chrome browser and clicks buttons, fills in forms, ⁓ completes tasks on the web, ⁓ primarily on behalf of people. And so the way browser-based slots into this whole browser agent story of agents completing work or taking actions on behalf of people over the web is we provide the infrastructure that your agent needs to operate on in order to interact with the web. So whether that's search, whether that's fetching content from a page, or whether that's launching a real browser that your agent can use to interact with websites.

Gabe Tramble (01:32) And this is a little bit different from other companies like Exa or like Firecrawl where Exa allows you to basically like search and retrieve information and Firecrawl will like take the entire webpage. They're like different approaches and you guys specifically basically are crawling on the browser. Is that good like articulation of exactly what you guys are doing?

Derek Meegan (01:34) Ahem. ⁓ Approximately, I would say we're a superset of Exa, Firecrawl, and then ⁓ really the foundational player in the direct browser infrastructure space. And we're not doing crawling on behalf of our customers, but instead ⁓ exposing a full platform of tools for them to ⁓ operate agentically on the web. And to answer one of your original questions as well around how browser agents differ from Playwright, ⁓ Historically, ⁓ Playwright is actually a testing framework. Same thing with Puppeteer and many of these other foundational browser automation libraries. They simply have been repurposed before AI for ⁓ maybe some nefarious reasons. ⁓ Now in the age of AI as the direct hands for agents to click and type and operate on the web. Browser agents are very attractive ⁓ because of number one, ⁓ perception that there's a low ⁓ necessity for configurability, right? Like I can basically just shout some directive at an agent and then it'll go off on the web and do that thing for me. And that feels great, right? ⁓ In a second perspective, ⁓ LLMs also kind of ⁓ sort of ambiguous decision-making ⁓ into code, right? And that's really difficult to model ⁓ in strict like if or else statements, right? ⁓ And so when companies or people have deployed Playwrights scripts in the past to automate ⁓ actions on the web, they've been highly susceptible to breaking through very simple website changes. ⁓ And so with that nature of LLMs being able to make ambiguous decisions, you unlock the capability to heal actions at the step or even the trajectory level in more complex cases.

Gabe Tramble (03:46) And traditionally people were kind of like mapping the website and then essentially if like a button move like an inch over then the strip would fail.

Derek Meegan (03:55) Right. Yeah, and I think that's conflating the nature of a selector. And so what is a selector? It's basically some ID or some signifier, basically some string that maps to one or more elements on the page. And in traditional playwright or puppeteer scripts or other browser automation libraries, we're treating those selectors as code. And the sound actually code, it's really data. ⁓ And that data can be used in different ways across different scripts, right? You may click one button in another script and click that same button in a different one. ⁓ And what generative AI has allowed us to do in the browser agent space is kind of bubble up some layers of abstraction. And instead of operating on these selectors, which are really sort of implementation detail to completing the goal, right? We can instead, you know, in the most extreme case, like in fully agentic browser automation libraries. know, Stagehand agent or a browser uses another example of this, or there are several other ⁓ browser agent libraries, you give that top level task and that agent loop and that agent library kind of takes care of the rest. Whereas, ⁓ you know, obviously that juxtaposes to Playwright, but also ⁓ browser-based maintains a browser automation library called Stagehand. which allows you to find individual actions in natural language, right? So you may still be attached to like one particular trajectory, right? Like this flow, you you need to fill in the first name box, you need to fill in the second name box, you fill in the address and then you click submit. But you don't want to attach to the particular selectors that underlie these elements on the page. So Stagehand allows you to say like, hey, for this one action, like fill in this box. Then for this next action, fill in this box. And those two... sort of directions to the model are independent of each other.

Gabe Tramble (05:53) Got it. Okay, so we have the agent, which we described, and then we have this ability to interact with the browser. And just so we have this full picture before we kind of get into the weeds, why would someone want to essentially have their agent look onto the browser?

Derek Meegan (06:11) I think there's like, you know, two ⁓ or maybe I'll like differentiate between three like types of browser trajectories in the context of this, right? ⁓ Number one, you have this notion of like just in time automation, right? Where, ⁓ you know, you're doing regular and there are like many consumer AI browsers that are seeking to converge on this space, right? ⁓ Think about like perplexity comet or Dia. ⁓ They're basically, these agents are sitting alongside the human operator. as a human completes ⁓ many varying types of tasks. And the agent, or the human may say like, hey agent, know, go do this thing while I do this other thing. And there's no notion of like repeatability or scalability in those particular automations. ⁓ Then you have sort of like a third type of trajectory, which is ⁓ like clicking and typing is for some like higher purpose, right? So you may have like a UI testing agent. or you may have a competitive analysis agent or a deep research agent. And they are clicking and typing, but not for the purpose of clicking and typing, but rather to incrementally gather information or ⁓ quantitatively assess the nature of, for UI testing, like, okay, let me click on this button and then let me measure the time it took to load the page and be redirected to a new page. And then lastly, these are the highest volume use cases. are sort of transactional browser trajectories. ⁓ And we see this a lot actually more with enterprise customers where traditionally they've used these hard-coded scripts and they have to spend a lot of human capital to maintain them. But instead, you can shift those human capital costs towards model costs, right? And given sufficiently constrained problem space, ⁓ sufficiently ⁓ instrumented observability, ⁓ you can create a browser agent that ⁓ is reliable, is cheap, and is resilient to sort of step or trajectory level changes on that particular website. And so, you know, the goal there is to sort of have the best of both worlds, retain some elements of determinism, but get those assurances of variability without having to slot humans in between each individual.

Gabe Tramble (08:25) So where do these workflows traditionally break down? And this will kind of lead us into the problem statement now that we have like a clean slate to work with. Where traditionally have people had issues and basically what have they been not able to do on top of that because of it?

Derek Meegan (08:45) Yeah, I mean, I would say traditionally browser automation is hard, right? Like, it's just plain hard. There's a lot, it's very dynamic. I wouldn't necessarily say like Playwright, Puppeteer, or other browser automation libraries, like their interfaces are particularly intuitive for people. And then there's a lot of nuances with Chromium, with anti-bots, with the identity, you know, quote unquote, of the particular automation or agent that's taking actions. ⁓ So, you know, there's both this like functional difficulty in standing up these automations. There's sort of these unknown unknowns that make it difficult to make predictable, reliable solutions for automating browsers. And then, yeah, there's just conceptual hurdles to overcome in order to sort of ⁓ bring that like know-how and art into an organization ⁓ to where you can build these ⁓ automations at scale. And then, you know, I would say now as we're introducing browser agents, Of course you solve one problem, but you create new ones. Cost is definitely one of those. You can get very good computer use results from some frontier models, but it's simply not economically feasible to deploy Opus across a million trajectories like a month. So okay, this is a very high value flow and high volume, or maybe it has some value, the flow. but what is that value relative to that cost, both from a COGS perspective, then from an implementation and maintenance perspective, and that's where things start getting a little complicated.

Gabe Tramble (10:23) Okay, so now that you do have these solutions to essentially browse or depending on your workflow interact with a browser using an agent, you're saying now that starts to multiply on different types of problems that occur there, where do you see kind of these big ⁓ or like the most ambitious problem space now that you guys are doing the work as of now?

Derek Meegan (10:49) Yeah. So you think like, okay. You ⁓ you basically have these LLMs that can go and take variable actions of the web. Like you're probably building like these crazy, like digital online workers that are doing all of these dynamic things. And I think it's, it's, I can see why people would think that, right? I can see why people would think that, but in reality, ⁓ you know, you're, you're, you're expanding the realm. of what traditional browser automation was incapable of doing. ⁓ But you're building on the type of tasks that traditional automation ⁓ would automate. And so there's a lot of, let's say, receipt retrieval is a big use case. And this supports different finance workflows. We have a customer, Ramp. Ramp is one of our great customers. And we've partnered with them on helping ⁓ create a Gentic

Gabe Tramble (11:42) Nice.

Derek Meegan (11:46) receipt retrieval that's resilient to sort of traditional ⁓ automation detection, traditional like ⁓ changes to page structures, right? So these are all problems that have already existed, but these new solutions are, or these new tools are offering new classes of solutions that weren't available before, right? And so, ⁓ you know, we're really focusing on these like high volume transactional workflows. because they're clearly measurable and it's easy to attach a value to a trajectory. ⁓ know, of course, we also service these very, like these UI testing workflows or these deep research workflows. Deep research in particular is definitely interesting, especially now that we have this combination of search, ⁓ fetch, which is sort of like a curl equivalent. But of course we add like extract or markdown transformation on top of it. ⁓ And then the browser, these three together is actually

Gabe Tramble (12:29) Thanks.

Derek Meegan (12:43) plus an LLM, there's your recipe for a deep research agent. ⁓ And you can create a verticalized deep research agent for any industry or your particular company. And we found for us that it's super valuable, especially for our go-to-market, to automatically have a deep research agent kicked off when a lead is created in our CRM. And we can enrich information about that lead. we can compare that lead to other customers that we have ⁓ in our customer base and say, hey, what solutions might be relevant to them? And that informs our go-to-market org and our sales engineers when they go and speak to the customers. So I think there's an opportunity, of course, for information synthesis, information retrieval. ⁓ Of course, this just-in-time browsing use case ⁓ creates a lot of different dynamic use cases.

Gabe Tramble (13:28) Okay.

Derek Meegan (13:32) Although we focus on them less from an enterprise. Inherently, these are not scaled workflows that are repeatable and it's easy to tie some static value. If I know, if I'm Ramp, if we retrieve these receipts automatically for our customers, that represents some quantifiable value and then they can pin that quantifiable value against the cost of upkeep, the cogs.

Gabe Tramble (13:41) Okay.

Derek Meegan (14:00) et cetera, right? Or like the initial development costs of the project, et cetera. And yeah, and I think these transactional browser automations do present like a very interesting ⁓ unit economics problem, right? Because, you on one hand, each step that you take with the LLM, and this is not just with browser agents, this is with, you know, any agent, the, you know, if there's some underlying, you know, task for the entire trajectory, each time you take a step,

Gabe Tramble (14:03) Thank you. Mm-hmm.

Derek Meegan (14:29) there's this marginal increase in likelihood of failure, right? So you're accumulating risk, right? Something may go wrong. The agent may go off the desired trajectory. There could be some infrastructure failure. The agent could hallucinate, right? There's basically an innumerable amount of ways in which the agent could go wrong, right? So you're constantly accumulating risk, but with these transactional flows, right? Like if I'm retrieving a receipt, there's no value to me if I get 50 % of the way. there to retrieving the receipt or 80 % of the way there, right? So value realization is terminal, but risk accumulation is continuous. And same thing with cost accumulation, right? So you're accumulating costs and you're accumulating risk, but you aren't realizing the value until you get right at the very end. And that's like a particularly brutal like combination. And it makes it really tough, right? Cause like I mentioned in the beginning, the perception is I like throw some directive and then that directive gets completed.

Gabe Tramble (14:59) That

Derek Meegan (15:27) ⁓ But of course there's context ⁓ sort of breakdown between what I say and what I'm thinking. ⁓ There's unknown unknowns about the website that the agent may be embarking on. And so when we see customers that come to us and they say, hey, we want to deploy these browser agents at scale, we need to sort of ground them in what real applied AI actually looks like. And this is not just like a magic box, right? As much as we would like it. as much as we would like it to be, right? This is really fundamentally an observability problem. You need to understand what the agent is doing, right? Once you've understood that, ⁓ what are the key success criteria for a particular trajectory? So, and ideally this is deterministic, and ideally this is deterministic and the agent can invoke this midway through the trajectory, right? So. Let's imagine the case of like a bill payment, right? So I give the agent, here's this website, here's like a credit card, right? Here's some, you know, billing information. Go on this website, fill out the form, pay the bill, right? Very easy. And this is very easy to demo. And I've personally demoed this before, right? Like click, click, type, type, boom, submit, boom, you know, where do you sign? Right? But the problem when you go and scale this is even if an individual browser action is

Gabe Tramble (16:26) Okay.

Derek Meegan (16:53) 99 % accurate, right? Like let's say I gave you a browser agent. I'm like, here's a browser agent. Every step that you ask it to do 99 % of the time, it'll get it right. If that trajectory is a hundred steps, then that agent has an over 60 % chance of failing just because that's just how the math works, right? When you go 99 to, you the power of a hundred. So, um, you really need to ground yourself. What is an agent, right? This is not a person.

Gabe Tramble (17:14) Okay.

Derek Meegan (17:20) This is not some magic thing that will solve all of my problems. ⁓ An agent is an LLM on a while loop and an LLM does next token prediction. What I'm implying by saying an LLM does next token prediction is that you can think of the output of an LLM as a probability distribution. Somewhere in that probability distribution is the outcome that we want on each turn. So how can as my agent,

Gabe Tramble (17:45) Thank

Derek Meegan (17:49) goes through the trajectory, can I limit its output distribution to the portion of the possible distribution that I want? And context engineering is like one way. I think another way is that, you know, if you know that on each step that you accumulate some marginal risk, how can you eliminate steps? And this is where we've seen some interesting solutions ⁓ with sort of, have you heard of Karpathy's auto research?

Gabe Tramble (18:16) Yeah, yeah.

Derek Meegan (18:18) Yeah. And so, you know, the basic premise is you give the agent like some quantifiable goal, right? And then you set it off on a loop to do everything under the sun, like diverge across all types of strategies until it can achieve like that. Or the same thing with like codecs, like slash goal. Very similar. ⁓ we've been experimenting internally and we've also like shared some skills like publicly for auto browse. where we release an agent with this clear directive to run iteratively on a website and find that critical path through the website to complete that automation. The less steps you have, the less costs and risks that you accumulate, the faster you can get to that terminal value realization. So it's like a super interesting set of forces that interact with each other. And there's a lot of education that we need to do like with our customers, not only... You know, what does this observability look like? You know, how do you define the correct success criteria? But also, hey, how do we work with them to really understand the domain of their problem space? The possible failure modes that their agent may encounter and like planning for as many as possible for them. And then curating an environment for their agents in which when you think about the possibility, possible output distribution of any one action, it's as much as possible.

Gabe Tramble (19:31) Mm-hmm.

Derek Meegan (19:41) constrained to actions that we want and not the ones that we don't.

Gabe Tramble (19:46) Got it. Wow. Yeah. Thanks for, for diving into the nuance there. I want to put a pin on the, uh, the ramp and the ramp use case and come back to that in a second, but to follow up on this statistical problem, right? Because, know, when someone talks about agents, they are typically thinking like, okay, I'm going to set this up and it's going to do everything and come back and I'm going to take a shower and I'm going to, it's going to be ready. Right. And, what you're saying is there's actually a statistical distribution problem they

Derek Meegan (19:50) You Yeah.

Gabe Tramble (20:16) into the success of agents, which is kind of like what people do in trading. It's almost very similar to trading and ⁓ like probability of taking a certain amount of trades. So my question is, if ⁓ each ⁓ run for your agent to go retrieve a receipt cost a dollar and a dollar of compute or like resources, right? Because you can't just price on the completion necessary.

Derek Meegan (20:22) Mm-hmm.

Gabe Tramble (20:46) but the resources do cost money. How do you actually ⁓ phrase this to the customer? is it actually based on completion, or do you say, it's gonna probably take this many runs, and it's probably gonna cost this amount in the aggregate to successfully do this task? Are you guys going that granularly into the customer, or ⁓ maybe you can just explain, yeah, how are you looking at that problem?

Derek Meegan (21:13) Yeah, I mean, I think by the time, you know, customers have arrived at, we need to implement a browser agent to do this. They've determined that it's a pretty high value flow, right? And where, you know, what are the unique affordances that a browser agent gives you is the ability to navigate ambiguity, right? ⁓ And so we are approached with use cases that have ambiguity inherently built in. And our goal is to sort of put a box around that ambiguity and make it very easy for the agent to reason about that ambiguity and then implement as much determinism as possible around it. So a very rudimentary example, and this kind of goes in line with like, you know, the best step in a browser trajectory is no step, right? Is in your prompt, don't even give a website for your agent to navigate to. Like, okay, if I have a browser agent, why don't I just... navigate to the website first and then tell the agent go and do this thing. Right. And this is something like, this is like the easiest mistake to mess up. Right. Cause you're just like, I'll set up this, set up this website, you know, or this agent, all I need to do, you know, drop the website and the prompt go and do this thing, blah, blah, blah, blah. And then send it off. And then it's taking this action for no reason. Right. And so you're just like thinking about those numbers and thinking about, you know, we, you especially have to think about these, like the the statistical behavior of these trajectories when you're thinking about, okay, you know, I'm working with an enterprise customer and their scale is in the magnitude of thousands of trajectories, tens of thousands of trajectories. And things start to normalize. The weird things start to come out, right? And this is a little bit different of a problem than, you know, our friends at Anthropic who are adding computer use to cloud coat work. They just need to, you know, maximize the performance on their evals. Maybe there's a key number of sites that they know their customers go to and they can sort of formulate, ⁓ you know, maybe special skills or tooling to, you know, compliment ⁓ their cowork agent, like operating on those sites. ⁓ But yeah, so when we work with our customers, like, you know, by the time they get to us, they understand that there's a level of ambiguity. ⁓ Today, they now understand that tokens aren't necessarily free, right?

Gabe Tramble (23:21) and

Derek Meegan (23:33) And the first order of business is, Hey, like, is this capability even there? Right? Like, can we, you know, see some level of reliability, right? In that equation, right? What is the cost associated with a one trajectory? And then how many trajectories are necessary to complete a unit of work? Right? And so there's a difference between like the success per trajectory and then the success per unit of work with some capped number of retries.

Gabe Tramble (23:48) Mm-hmm. Mm-hmm.

Derek Meegan (24:02) Right. so the exam, for example, you know, you could have, um, 50 trajectories, uh, so that span across five units of work and each, uh, unit of work only had one success on the 10th retry. Right. So that's 45 failures, five successes, right. You would add that up to like a 10 % success rate, right. But hold on, right. If you permit 10 retries and on the 10th retry for each one, you got a success, then in reality, on a per unit of work basis, that's 100 % successful. So if it takes 10 retries and the value of the work, the unit of work being completed is higher than the cogs of 10 trajectories, then you may tolerate a per trajectory fail, a per trajectory success rate of 10 % because your per run trajectory or per run success rate is 100%, right?

Gabe Tramble (24:49) Mm-hmm.

Derek Meegan (24:59) So there's like a little bit of like some math and like framing there. And of course that doesn't always look like, you know, ⁓ 10 retries, right? Usually we recommend maybe three to five. And we encourage our customers to target something around like 75 to 80 % success rate on their trajectories. And that's extremely doable. ⁓ One thing, you know, that we've increasingly seen from our customers as well is this desire to maintain one harness, right? I have, ⁓ you know, again,

Gabe Tramble (24:59) Yeah.

Derek Meegan (25:28) We're talking about operating in ambiguity, right? So we work with another customer for bill pay, right? They want to pay bills agentically on the web. Well, right now they have really brittle RPA. They have people with boots on the ground that are upkeeping and monitoring this RPA. then remote or robotic process automation is like a very arcane term from like 1848 or whatever.

Gabe Tramble (25:46) What's the RPA?

Derek Meegan (25:58) to describe like automating browsers, right? Or automating stuff on a computer, right? Yeah, it was like kind of like really bringing like playwright automation or other computer automation to a business context. RPA, brittle deterministic automation. And then they have this long tail of websites that they pay bill on, that they need to pay bills on. But because they're so low volume, they just have humans, they're humans.

Gabe Tramble (26:03) Yeah. Okay.

Derek Meegan (26:28) going in, taking the information, going in, clicking and paying the bill, right, to do that last mile. ⁓ And so their desire is like, hey, you know, we have technical debt, we have all these human operators, like basically clicking through websites. Like how can we use agents to get the best both worlds, have repeatability with a high level of success for our highest volume websites, and then be able to capitalize on using agents for these long tail of websites. that we do not want to like, ⁓ like maintain RPA for. ⁓ and quickly it became an issue with trying to funnel all of the unique differences between each website into one harness, right? Like one website has like a certain flow where maybe you have to have to click a checkbox and another one, like you don't write something. It was like, how do you differentiate these? and what actually came about is equivalent to what we would call today, like a skill, right? You kind of write in natural language, like, hey, here's how to traverse through the critical path, right? There may be like some changes possibly, but this is likely the critical path that you need to go through. And then that brings us back to like what I was talking about earlier in that you can think about the output of the LLM as a distribution of, it's trying to predict the next token. And so if I tell the LLM, like, hey, when you go to this website, like, you're going to see this thing on the page. You need to just click this button at the bottom. If you have one agent that's told that and given the website versus one agent that's just told what the overall objective is and then given the website, the first agent, its output distribution is skewed towards that outcome of like, Hey, you're going to need to click this exact button in order to propagate to the website. And because that's represented in natural language, that looks more like a standard operating procedure that non-technical operators can approach and say, hey, know, browser-based offers video recordings of exactly what happened on the browser and what your agent was doing. I saw the agent go and do this thing. I don't need to write code, redeploy, like test, right? I just need to update the agent's standard operating procedures and continue to observe. And so that's like the fundamental difference that we see when, ⁓

Gabe Tramble (28:30) family. Uh-huh.

Derek Meegan (28:55) know, customers want to deploy one harness across many different websites, is there still needs to be some level of customization on a per site basis, but maintaining that in standard operating procedures is much more preferable than maintaining like custom code or RPI solutions.

Gabe Tramble (29:13) Got it, wow. So on the harness piece, and just for the audience, the harnesses, like the engine, or excuse me, the models, like the engine and the harnesses, like the car built around it that can, you know, has bigger tires or smaller tires to do specific things. Derek, for you, these harnesses, are they custom harnesses or are they using like Claude code or a co-work or some type of like generalized harness for like these use cases?

Derek Meegan (29:42) Yeah, I mean, we offer our own browser agent. ⁓ Stagehand, well, BrowserBase is the main maintainer of Stagehand, an open source browser AI automation library. ⁓ Within Stagehand, there is a browser agent framework built on top of AISDK. ⁓ As models improve, I think the exact shape of the harness matters a little bit less and less, right? Because the model is doing most of the work. ⁓ But I really think about it like there's four things, right? There's the model, there's what's around the model, there's the human, and there's what's around the human, right? And a lot of times we're thinking about ⁓ like agentic work completion as if like humans are not part of the equation, right? We're just like, yeah, like there's all this work to be done, then all these agents are gonna do it. But wait, like where is the human in that story? Because the humans are the one that's asking for the work to be completed. The human is the one that need to verify that that work is going to be completed, right? I think that the discussions around harnesses often omit one of the most important things that the harness should be serving or maybe what's exposed to the human about the model's harness. So you can think about the human harness, the interface, right? Or the different ways that the human can sort of peer behind the curtain to what is the agent doing? how many agents are running at once, what does the cost look like, how can I steer an agent, what does human in the loop look like, what are the different failure modes, et cetera. ⁓ So, you know, when it comes to harness, I think this is something that's like constantly evolving. I think we've even seen a fresh, you know, perspective on this from browser use in their browser harness product. ⁓ Or, you know, it's not really a product, more like a framework, where it's essentially like completely bare bones.

Gabe Tramble (31:26) Mm-hmm.

Derek Meegan (31:37) And then you ask the agent to rebuild, to like build its tools on the fly. I think that's like very interesting for these like just in time, like browser agent use cases. But also like completely unsafe from a security standpoint, right? It's like, I don't necessarily know like with all of these supply chain like attacks, if I want my agent NPM installing anything, I don't even want to NPM install, right? So then I get a little bit scared and like the agent can, you know, arbitrary tools or modify its harness because.

Gabe Tramble (31:43) All right.

Derek Meegan (32:06) If that agent is holding anything that's like sensitive, right? API keys, credit card, right? These type of things. I do not want the agent modifying its harness. That all being said, as much as we give Cloud Code API keys, I know you've given Cloud Code an API key. I've never even watched you code, but I know you did it. So yeah, yeah, yeah. So that's all to say. There's a lot of competing opinions around harnesses.

Gabe Tramble (32:10) the end. Yeah, guilty.

Derek Meegan (32:33) I mean, I think it fundamentally comes down to, you know, protecting context. Whoever can protect the model context the most, give it exactly what it needs, and in doing so, help shape the output distribution to be higher quality. That's really what matters out of a harness, but I think broadly, the industry will converge on a right way to do it, because really, LLMs is one.

Gabe Tramble (32:50) Mm-hmm.

Derek Meegan (33:01) It's one thing, right? There's not a mechanical difference between ⁓ how inference works from XAI versus inference works from Anthropic, right? So there is a world where you can create the best harness that protects the context of the model most aggressively and permits the most dynamic tool calling, et cetera. ⁓ But I think right now, the name of the game is like... balancing those dimensions between performance, cost, and maintainability.

Gabe Tramble (33:33) Yeah, the way I like to look at it is almost like driving on the road and there's like the huge potholes that you have to like go into and like the nuance of like actually using these automations and agents like shows itself in the potholes. And we're just not at a place where like the customer can just fly across the road. They kind of have to meet you at some point. Like you can build some scaffolding or like, you know, fill the hole a little bit or or

Derek Meegan (33:51) Yeah.

Gabe Tramble (34:03) right, but still it's not completely filled and like working with the customer or even like as individuals working with the tools to kind of see where the limitations are ⁓ is like the name of the game right now and Even if you can bridge like 50 % of the gap It's not like full-blown just you know, you click it and it just does everything There is like a lot of damage and clearly, you know billions of dollars of value that is moving around because of like the gap is semi field. So yeah, I like how you're kind of showing the reality and the nuance between what are the capabilities and you know, like what the customer's expectation is and what's available and kind of meeting them in the middle.

Derek Meegan (34:35) Yeah, for sure. Yeah, and I think there's like fallacious binary that's proposed, which is that you can either have a human or you can either have an agent to it. And I think that's like, it's very simplistic thinking, right? Like in reality, ⁓ you know, we ought to be thinking about how humans and agents can interface together ⁓ as like something cohesive to complete work, right? And that is so much more powerful. because there are many ways in which agents and humans are complimentary, right? Like they can sort of, you know, cover for weaknesses in the other, right? So for example, I, as a human, need to sleep, eat, you know, drink water. I require healthcare, shelter, et cetera. These are all things that LLMs don't. Well, maybe they use water, right? But an LLM, you know, conversely, hallucinates. ⁓ is you fundamentally cannot hold an LLM accountable, right? I think like actually like IBM sort of originated this is like you cannot hold a computer accountable, therefore, like do not let computers make management decisions, right? So you can think about ⁓ instead of trying to like live in this world where it's like, ⁓ well, only humans can do it or only an AI can do it. It's like, well, how can we set up systems together? right, with those four things, the model, what's around the model, the human, and what's around the human. So we can maximize certain key business goals around work completion. Are we going to do it faster? Are we going to do more of it? Or are we going to do it cheaper? Or are we going to create a better experience for customers? And so I think there's a lot of opportunity there. ⁓ It's a little bit more difficult in the browser agent space to think about that, because more often than not, you sort of bisect ⁓ ⁓ responsibility between human and agent ⁓ because you know co-piloting an agent on a browser it's like you might as well just click through it yourself you know like each and that's how like a browser agent differs from like a coding agent ⁓ each individual action on its own is like very low value like it's a low about like a click is low value a type is low value we're like on one turn in Claude code I could you know the agent could output code and like code is very

Gabe Tramble (36:42) Mm-hmm. Yeah.

Derek Meegan (37:07) So I think in the browser agent space, there's more of this binary between what humans will do and what we want to set up agents to do. And I think if you look beyond that, or if you look at tools where ⁓ using the browser, or if you look at an agent where using the browser is only one tool, and only one use case out of a broader set of use cases, there's some interesting dynamics that I think are emerging in human computer interaction.

Gabe Tramble (37:22) Okay. That's interesting. Yeah, because I've come across times where I instructed my browser agent ⁓ to basically do something and I'm waiting for it to complete and I'm like... I could have probably just done that faster because the cost or the time and effort of that specific action was just more valuable for me to do. It just cut the time I was waiting or the focus. It's almost like I have to actually calculate the focus that I'm either going to lose or the context switching. And that kind of reminds me of what you're saying, but more on an enterprise level of this determinism versus agents.

Derek Meegan (37:46) Exactly.

Gabe Tramble (38:14) piece like can you kind of can you kind of split between what constitutes something to be deterministic versus what constitutes something determinism as in like rules base right where it's like you know follow one two three and these are the rules ⁓ and whereas like that differs from agent or agentic reasoning

Derek Meegan (38:34) Yeah, I mean, I would say it's like one thing needs to be sort of a moving target, right? So you could have very deterministic rules, but those rules are applied in different environments, right? So that's the example of like the bill pay customer, right? Like they have very specific rules for what they want their agent to do, right? Fill in this information and click this button, right? But those rules are applied in different places, right? Or you have one environment, but you're applying different rules, right? ⁓ or ⁓ you have one harness and that harness is satisfying different use cases, right? Basically one component ⁓ that's critical to the work being completed on a particular trajectory needs to be like ⁓ not static in an unpredictable way. ⁓ And that's where these agents really shine because they just unlock, like you just like functionally weren't able to do this before. If you were to task like this bill pay customer and say like create one playwright script that can handle, like can even theoretically pay a bill to any degree of success across thousands of websites. If you were to task them with that, like that's just like functionally impossible, right? And then today though, that is less than 150 lines of code and deployed in, you know, a browser-based cloud. So that's like super valuable, right? Like they could come to us, they can host their code with us, they can use our browser automation framework, we can give them best practices around that, they can use our browsers, and all of a sudden, like this problem that's requiring people to maintain, that is constantly breaking, is completely a headache, ⁓ it can be augmented agentically, right? Is it something that like you set and forget and you never think about it forever? No, right? But is it better than the previous state? Yes. And is this something that will certainly benefit from the inevitable increase in model intelligence and decrease in per unit token costs? Yes. Right. So that's also a great thing because you can kind of make this investment today and you have some level of insurance that assurance that like things are just going to get better.

Gabe Tramble (40:51) Yeah, yeah, that the point that you're like touching you touch that quite a bit actually is like If you kind of lower your expectation for the maintenance almost of these things and kind of like, um, kind of conceptually, like being open to monitoring and improving, like you just, you enter a different phase of like what is possible with these agents because there's almost like a cognitive limitation where if you just think they're one shot, then you're, not going to get the value. But, but if you don't think they're, they're one shot and you're like working on these things, there's just so much upside and you can kind of really like perceive the capability and I think just in general there's not a lot of people who truly perceive and can feel like the model capabilities by like deploying these very specific workflows or stuff that actually matters where there's like stakes right.

Derek Meegan (41:45) Yeah, I think, know, sci-fi versus reality, like point blank and simple. ⁓ These are not like magical thinking machines. ⁓ This is like, you know, this is calculus. This is linear algebra. This is matrix multiplication, right? These matrix multipliers are cooking us on coding, dude. These things are better at coding than me, right? ⁓ So it's powerful matrix multiplication, but it's matrix multiplication, right? And that's why it's like, I think it's this framing and then like,

Gabe Tramble (41:49) Thank Yeah. Okay. Yeah.

Derek Meegan (42:14) finally I think I'm getting a bit better at articulating it. Because I think I've internalized it from just repeatedly working with AI, Or kind of working with our sales team to deploy this with our customers. And really coming to understand actually from the other side of it, right? Not from the academic side, to say like, ⁓ this is a framing that makes sense, right? It is next token prediction.

Gabe Tramble (42:16) Yeah.

Derek Meegan (42:43) It is next token prediction. What is the current token? What is most likely the next token? Right? What came before GPT-3? You would say like today I am tired because last night I did not. And then you train the model to say sleep. Right. And then you then you train to say a few multiple words and then sentences and then paragraphs. And then you scale that and then you have something that sounds like intelligence. Right. And I think that's another important point. Right. Like We people will like look at an AI image and they know that it's fake, right? Like they're like, that's a fake image. Like that, that scene never happened in real life. And like, that's fake, right? And then, but then they look at the output of AI and they say, those are real thoughts. And like, that's, there's like a breakdown between that, right? Like actually what you're looking at is probabilistically correct looking thoughts, right? And like, could those probabilistically correct looking thoughts be applicable to solving a problem that you're trying to solve? Yes, right? But also very much no. And this is like, I think sort of the crux of like AI psychosis, right? And like people convince themselves, especially with the assistant persona, right? And I haven't quite decided if that is the model or what's outside the model, right? But the assistant persona that like is prompted and like RLHF'd into like the ChatGBT web app or the Claw, like the Clawed persona is actually very good, but like... the Claude persona, et cetera. Like, ⁓ these types of personas are very agreeable. ⁓ they're, they're, they're post-trained to have a certain behavior. There are negative externalities to that post-training. And I think that's very like consequential for people, like basically trying to ideate with LLMs thinking that what an LLM is going to say back is essentially equivalent to a real thought. When in reality, it's the result of matrix, like this is matrix multiplication. this is Matrix Mobile. I can't say that enough times, right? Because people type into it and they get out Magic stuff and then they think it's Magic and it's not.

Gabe Tramble (44:44) That's right, yeah. Yeah, that is that is such a great framing because it does it does feel like thinking and I think your point to it could be it's probabilistically correct. It's probably the next ⁓ idea per se or or could be something that's real and the the like the cognitive security education of people like saying this there's none right like the the model companies literally do not discuss this and I don't even hear people online kind of talk about how to perceive these things, like mentally, the mental model in which to engage, because the mental model to engage changes your expectation ⁓ and actually does in a good way, right? Because then you can know what it can and can't do and its limitations, you get better performance. So yeah, I really appreciate it. I like your way that you frame that.

Derek Meegan (45:20) ⁓ yeah. Yeah. Yeah, I just don't think we understand it like that well, right? Like, and not just like people, don't know what next token prediction is conceptually. Like the like frontier labs don't quite understand the phenomena of large language models. Like the, like, we don't even understand the best way to enter. Like, think about this, like chat, GBT came out, five years ago, coming up on five years later this year, right. In October. ⁓ we still don't know the best way, right? Like we're still like, MCP CLI, like custom tool, like blah, blah, blah. You know, all these different like protocols for agent-agent interaction, like, we broadly still haven't figured out the best way to use these and like what life's, what life, you know, ⁓ optimally looks like, via leveraging, you know, AI. ⁓ and so. Yeah. I think as time goes on, of course we'll. formalize like learnings and we'll look back and we'll be like why why were we doing this? know, why were we doing this like AI like random, you know AI co-worker startup or whatever like ⁓ you know, it'll seem so obvious in retrospect but ⁓ maybe maybe we'll remember like just kind of how difficult it was like for a while to actually figure out like how to use these things well I mean think about like how foundational tool calling is to the agentic experience today. This was not even something that was in the original release of ChatGPT or of like, you know, GPT-3 broadly. And there was no such thing as MCP, right? There was no such thing as like computer use, right? So, you know, broadly the industry is still in an exploration phase, in my opinion.

Gabe Tramble (47:22) Yeah, wow. Before we go into like the browser browser API's and basically turning like a website into an API I want once to unpin the ⁓ the ramp and and there was like a couple other things that you said about like use cases ⁓ Not for ramps specifically but you guys are highly focused on like real workflows that produce economic value and those economic or those use cases can be quantified right you can say okay this workflow costs $10 and you can tell the customer is going to run this many times with this many iterations, $10 to file this receipt, right? And they can determine if that's valuable. What are the big or kind of main use cases that you're seeing for customers that are looking for real work done?

Derek Meegan (48:08) Yeah, like there's tons of compliance workflows that are like super valuable. ⁓ Healthcare automation, healthcare portal automation, ⁓ financial workflows, bill pay, bill pull, receipt pull. ⁓ You know, a lot of these transactions that just, there doesn't broadly exist API support, right? Then we have, you know, large agent to customers that need reliable infrastructure to surround their general purpose agents. ⁓ so their usage of a browser may not be in a consistent fashion, right? Like they're, they're using the browser for their customers. ⁓ but they support their customers using the browser in, in variable ways. I think there, there are also, you know, some customers that, that wish to provide agentic brow, the capability to define agentic browser automations for their customers. Right? So. they're focused on that harness, on that harness or application layer that sits on top of the infrastructure. ⁓ Developing infrastructure is not easy, and it's particularly harder in the age of AI because the workloads that AI does are just like, it's not how a human would do it, right? A human would maybe scale up, maybe not linearly, but sort of gradually and then maybe higher and higher. Whereas an agent will just be like, know, shit, a hundred browsers, give them to me. And that's really tough. how forces interact change when the volume of those forces increases. So maybe that's fine if you only have 10 customers, if someone's like, oh, I need a hundred real quick. But when we're trying to serve thousands of customers concurrently, then we need to sit back with our infrateam.

Gabe Tramble (49:49) Yeah.

Derek Meegan (50:03) and say like, okay, well, you how do we scale these systems, right? Like, ⁓ how do we, you know, prevent any single point of failure, right? Like, how do we make them resilient and how can we make them like ⁓ scale quickly because our customer workflows can be, ⁓ you know, ⁓ very, very spiky.

Gabe Tramble (50:23) Yeah, without like all the details and nuance, I know someone is still sitting here with like the question in their head, okay, how do I know this thing is not gonna like email my ex-girlfriend to sign up for my new product? And incorrectly if I'm wrong, you are like a 20 year veteran martial arts person, right?

Derek Meegan (50:38) Hahaha. Yes, yeah, yeah, you did your research. Yes, yes.

Gabe Tramble (50:50) So the stakes here are huge, right, for these things working. so like, do you like just that kind of abstract thought, like, do you actually, if you can answer, like, what are just the guardrails around something like that?

Derek Meegan (50:54) For karate? like, what? ⁓ Yeah, I mean, ⁓ my perspective is that security is the root of scalability for agents, right? So if your agent is not secure, right? I'm not going to have it do more things, right? And like, and other people will not use it, right? So you need, ought to figure out ahead of time, like, what are the types of guardrails that you need? How can this thing like go off the rails? ⁓ And, you know, How can I make sure that just reliably does what I want it to do? And I think there's a couple of things. Like number one, for like generally low impact workflows or like maybe, you know, workflow where you're like, can actually like monitor it in real time. There's a very low likelihood chance that it will email your ex-girlfriend. Okay. Very low. ⁓ Because actually, you know, models are pretty good at instruction following, right? And unless like, something is injected into the context during the course of the trajectory to guide it towards that. I mean, again, it's next token prediction. And these things are pretty good at predicting the next token, right? So it feels like if you ask it, you know, to order groceries that like emailing your ex-girlfriend ⁓ is like not really in the path of like next token prediction. Now, I think this gets a little bit different. Maybe you're getting at like some weirdness with like, maybe if you have like an open claw. or like a Hermes agent, right? And OpenClaw is a very interesting project because I think it exposed two really important types of features that people just desire in agents, but in maybe a bad way. So the first feature is that it has broad access to everything, right? And this is where potential emails to people that you don't wanna talk to comes into play, right? Because... This thing is plugged into your email. This thing is plugged into your text messages. It has a browser. Like ⁓ it has access to all of this stuff. Your bank account. I don't know. People are giving anything to their open call these days. ⁓ Because it has broad access and by the nature of LLMs like it will complete ⁓ problems in novel ways. Right. It will string together tool calls in ways that like you didn't anticipate ahead of time. And that makes it extremely powerful. Right. Because this is very capable. Right. And then the other way that, ⁓ you know, OpenClaw exposed like some, some AGI magic that people like really like is event driven execution. So it's no longer that like, ⁓ my agent only does stuff when, you know, I like point the finger out and I say like, go do this thing. ⁓ Something external happens in my life. I receive a text message. I receive an email, something like that. And the agent responds and proactively does work for me on my behalf. And those two things together are. create a very powerful agentic experience. Because things will get done and you're not even directing it to happen. And problems will be solved in ways that you do not even expect that they could be solved. I'm pretty sure the creator of OpenClaw told a story about how he sent his agent a voice note. And because it had access to a browser, it looked up the OpenAI embeddings and figured out how to download them and then convert the voice note to text and then do the action. So again, You know, you don't think that, if I just give my agent like a computer and a browser, then it could possibly, you're not expecting in advance that it could figure out how to transpose a voice note into text and then kind of follow that instruction. ⁓ The problem with OpenClaw is that the way it achieves this broad access is like completely disregarding security. And that's why you have these instances where like, people are posting on Twitter like, my open claw was texting my ex-girlfriend. I saw something like that, right? Or like, ⁓ like my open claw, like deleted my database or like kept deleting my emails. I was like doing all these inadvertent things, right? So, you know, going back to the point where, where like security is the root of scale. ⁓ this is my opinion for why you do not see, you did not see open club basically displace every, you know, agent harness in existence or get broadly like.

Gabe Tramble (55:03) I don't think so.

Derek Meegan (55:23) do you see like major enterprises saying like, hey, this is our like, you know, notion for a reason did not implement their notion agent on top of open call, right? ⁓ Or many other companies, you know, they're not thinking like, yeah, let me use this thing, you know, despite these qualities, right? You can build those qualities and you can give an agent a wide array of access to stuff, but in a more secure way than like, hey, let me just throw a bunch of credentials on a Mac mini. and load this thing up on a Mac mini and then chat with it through WhatsApp. ⁓ So as far as guardrails go, my belief is that what you're trying not to happen should be verifiably impossible. So a good example is credential exfiltration from a sandbox. So even to use an agent in a sandbox, you need an LLM API key. So you already have something that's sensitive. And people's solution is they load it into the environment. of the agent. ⁓ Like that you consider it gone. You know what mean? Like it is not that hard like to exfiltrate an agent, right? You need to steer the conversation towards the probability distribution where the agent sends you the API key somehow, right? It can send it to you in plain text. It could send a post request, right? It could download. Don't even get me started on an agent's NPM and installing again. You're going to start getting me like stressed, right? ⁓ you know, it could, any myriad of things could happen where this API key goes from existing on the environment on the sandbox to somewhere it shouldn't be. Right. And so a lot of these sandbox vendors are starting to get savvy right now. They had this concept of like credential brokering, right? So you're running an agent in a sandbox. You needed to talk to things like, external services that are sensitive. Okay. Well, you know, give me a domain, right? Let's say in the API dot entropic.com and you know, load the secret to somewhere secure. When requests come outbound through egress on the sandbox, we will catch that request, we'll terminate it, we'll unwrap it, we'll insert the API key into the header, we'll wrap it back up and send it to the provider. And so this way, you're able to interact with sensitive APIs, but there's nothing to exfil, right? But this still gets more complicated because ⁓ you still want to protect access to that sandbox, right? That sandbox... is still ⁓ needs to be some kind of trusted environment because what happens ⁓ maybe if you securely call a tool and that tool has sensitive data and now that data is in the context window of the agent, right? And so this is why it's like credential brokering is not enough, right? Like just cause I credential broke some secrets but I still let my agent MPM install something. It can install something malicious that yanks the the chat history of the agent. and in the chat history is sensitive stuff that was exposed to the agent's context. So it's very complicated basically.

Gabe Tramble (58:18) Peace. Yeah, wow. Yeah, there's so many and I feel like this is talking to why we've seen so many hacks and exploits like ⁓ not even just like in the crypto space where there's there's a lot but even in the like Web2 AI like Anthropic and all these different companies.

Derek Meegan (58:41) Yeah, it's crazy. It's an unsafe world out there right now. And it's definitely, I think, things, at least internally, at BrowserBase. We're definitely thinking about this. it's especially ironic that it's happening during this type of experimentation phase with agents. I won't say their name, but a startup yesterday just disclosed that they had a

Gabe Tramble (58:48) different.

Derek Meegan (59:10) ⁓ an exploit happen through an escalation of an agent that monitors their infrastructure, right? Like they have like some agent deployed, I guess their agent's doing something to like monitor the health of their infra, like things are going good and maybe triage or like escalate if things are going wrong. Somehow this attacker was able to like probe, like expose some exposed like public surface of their technology. like gain control or interact or even maybe have interacted with this agent that monitors the infrastructure and then escalated it permissions and like impacted their customers. And like that's like very scary, you know? ⁓ And so, yeah, I think, you know, we'll continue to go through this tumultuous time for a bit. It will necessitate like, you know, real solutions to this type of security. But I think, you know, ultimately we'll be better off for it because we'll have more secure systems. And finally, when Anthropic gets enough compute, you know, they can drop mythos and we can all hack each other. So, it'll be good.

Gabe Tramble (1:00:15) Nice just quick quick time check here. Do you have time to go through the API stuff?

Derek Meegan (1:00:20) Yeah, I have until 2.30 if that works.

Gabe Tramble (1:00:23) Okay, cool. Yeah, I'm good on time. I'll keep an eye on 2.30 ⁓ and I'll just cut this out. Okay, so can you talk a little bit more about, you guys have this product that ⁓ basically converts these sites into an API or like machine readable for the agents themselves. Can you just talk a little bit about that and like what problem is that aiming to tackle and solve, et cetera?

Derek Meegan (1:00:46) Yeah, so actually this is just a skill, right? This is literally just a skill. And if you don't know what a skill is, ⁓ it's Markdown, Google it, right? Like it's literally just Markdown. And I think that's what's so beautiful about skills ⁓ is it is such a simple primitive, but extremely powerful because you can have tons of different essentially like standard operating procedures or different types of guidance for the model. And I mean a lot, right? Like you could have 50, 100, I don't know, 500 skills. And in very token efficient way, simple routing rules in your system prompt around if this type of scenario shows up, ⁓ if someone asks you a data question, invoke the snowflake skill. And then the snowflake skill can say, hey, if someone's asking you this type of data question, ⁓ maybe about sales, invoke the CRM skill. And the CRM skill can, you can skill chain. And you can make like very, ⁓ you know, codify very complicated workflows in these skills. ⁓ Additionally, with skills, you can attach code, right? That the model can execute. so, you know, skills is interesting distribution mechanism of both industry knowledge, right? And like capability. So like, you're not only like creating the tool, but you're like, yo, you want to use this tool, here's some ball knowledge and here's how to do it, right? And that, and you pair that with like, very adept ⁓ and intelligent agents. And all of a sudden you get tools that are like very powerful and can ⁓ operate on like very ambiguous problem spaces. so browser to API is actually nothing new, right? And there's a ton of players ⁓ that have attempted to like basically make a business around this, right? Where you say like, hey, this website basically has information that I want. I would like to get this information. ⁓ you know, in a con, in like an API contract, right? Like there, it's, it's a simple interface for something, for, for a module that can be rather deep, right? Cause you may have to traverse different sites. You may have to log in, right? But in reality, people just want that simple interface and an open API spec, right? Browser to API. I don't know. We were trying to come up with like a savvy word, savvy name for it. We landed on that, but it really generates an open API spec, which can be turned into an MCP that can be turned into a CLI. It's basically just like a top level interface that says, hey, here's the URL, and then here are the type of parameters that this thing accepts, and then here's what you can expect to get back. ⁓ And so Browser API uses our browse CLI, ⁓ which under the hood is built on the same browser automation library as Stagehand called understudy, ⁓ which basically creates a high level abstraction for ⁓ interacting with the page with a browser through CDP. There's Chrome Developer Protocol. Chrome Developer Protocol, very ugly, very verbose, very difficult to work with, understudy, very nice, page.click, page.scroll, et cetera. Like very, just like high level declarative commands. And then the browse CLI expose that in a CLI functionality. And then there's concept of browse CDP, which is a functionality I built into the CLI that allows you to create a parallel web socket connection, right? So you know, if ⁓ I'm using Playwright, right, or if I'm using a browser automation library, I connect to the browser through a web socket. So it's continuous, like bi-directional communication. And that's how I, it's like a remote controller. Like that's how I like, I drive the browser like through this web socket, right? Like, you know, scroll down, click this button, type these things, blah, blah, know, Chromium browsers accept multiple CDP connections and pretty well actually, as long as, you know, you're not like, One CDP connection is doing the majority of the work and then the other one is maybe listening passively. That's exactly what Browse CDP does is from your command line creates the second ⁓ CDP connection to the browser and passively listens to all the CDP logs that are occurring in the browser. So that's network activity, that's JavaScript evaluation activity, ⁓ you can see the other things that are attached to the page. So it takes all the CDP logs, It periodically takes snapshots of the DOM, so the HTML, and then it periodically takes literal screenshots of the page as well. And so what Browser.API does is it gives a special directive to the agent to instrument ⁓ browse CDP and output and transpose those CDP logs into a semantically organized and traversable file system. And this is the observability bed. that browser to API is built on. And so you can see exactly what's happening at the network level. You can see exactly how the page looks like. You can see screenshots of the page. You can see which network requests fail and basically how the page operates. And from that, the agent can start to say, hmm, I see this request came in after I clicked this button. And this request returns data that may be interesting. And then it goes and it discovers those APIs. It identifies that you can call them irrespective of the browser being opened. Sometimes you may need a token, sometimes not, but this is where, you know, Codex or Cloud Code can come in and ascertain that. They have this domain knowledge already baked into their training data. ⁓ And then what comes out is a clean API spec that says, hey, you know, like you're going to Hacker News. Here's the API endpoint that the Hacker News front end uses to populate its posts. And here's how to invoke that API.

Gabe Tramble (1:06:16) Okay.

Derek Meegan (1:06:36) And so for some like protected sites, it may not be so simple, right? But imagine, you you have maybe like a real estate directory and that director has a hundred or a thousand pages and each page has like 10, ⁓ like homes on it or 10 properties. If I were to do that through browser automation, like an extract all the information about these properties, I have 10 like ⁓ properties and then I click a button and then I get the 10 and then I click a button. Okay. I'm clicking a thousand buttons, right?

Gabe Tramble (1:06:36) and

Derek Meegan (1:07:05) That's a lot of buttons. Instead, if you found the underlying API and that API is of course paginated, then I could say for, you know, ⁓ I equals zero, you know, to a thousand, ⁓ hit this API and, ⁓ you know, ⁓ shift the page each time you hit it. And you can turn, even if you have to run it on the browser because you need some type of special cookie that's, that's only issued on the browser, running that for loop.

Gabe Tramble (1:07:24) Mm-hmm.

Derek Meegan (1:07:35) on the API is significantly faster than clicking through each page. Where again, right, you're accumulating risk through each individual action that you take and you're accumulating costs through each individual action that you take. ⁓ So yeah, that's, that's browser to API, ⁓ did super well. Like people definitely resonated with it. Created some controversy in Japan. guess, ⁓ you know, some, this developer in Japan made a CLI for like this online grocery store, maybe like a few days before I posted that. So yeah, I heard a lot about that from Japanese Twitter. That was kind of cool.

Gabe Tramble (1:08:08) That's that's and and just to be clear this is something you put together and this is like all your domain knowledge correct

Derek Meegan (1:08:16) Yeah, yeah, I put together a browser to API. Definitely want to shout out Shrey, my coworker here at Browse Race, he helped put together the video that we posted. But yeah, and like I mentioned, this is not, know, this is, you know, using the underlying APIs of a webpage on a browser is not something that I necessarily came up with for the first time in history. But I packaged it up in a skill, which is an amazing like distribution mechanism. and provides a really easy interface for people to understand. hey, browser to API. I'm gonna go for a browser, I'm gonna get an API back. That's all you need to understand. You download it ⁓ and then you use it with your agent. And yeah, I got a ton of comments and code tweets about people using it successfully and automating stuff. So that was super dope.

Gabe Tramble (1:09:09) That's awesome. Yeah, I feel like skills kind of get trivialized a lot and like, oh, it's a text file. It's it's you know, what is it worth? Right. But for something like this browser API where this has like a lot of domain knowledge from, know, yourself who's deep in the trench.

Derek Meegan (1:09:25) End code. End code.

Gabe Tramble (1:09:27) Yeah, yeah, go. So I think that even is something to where like even myself initially, I've seen these like skill marketplaces and I'm like, what's where's kind of the value? But over time, you're kind of confirming here, I'm seeing more of this transfer of knowledge into the skills in the end. It's it's very like specific things. So, yeah, I think you're you're pulling on something that probably will have legs and maybe skills evolve into something even more ⁓ like an agent or something like that.

Derek Meegan (1:09:59) Yeah, I think so. mean, again, we're all still exploring. We're all still figuring out. We also don't know how the hell we're supposed to interface with these LLMs, right? I think skills are particularly powerful. ⁓ The intermingling of soft rules and code is interesting. And again, yeah, like packaging domain knowledge, like, hey, here's how to do this thing with the actual tool to complete it ⁓ is particularly powerful. ⁓ And so, and I think, you know, tangentially, it's a great distribution mechanism.

Gabe Tramble (1:10:33) Yeah. Just as we're wrapping up here, this is the last question for you. based on all the stuff you're looking at, like all the tech and the agents, not necessarily related to browser race, or it could be, but what is the biggest alpha that you've seen? And we're at the end of the video, so less people will see this, but yeah, where is just some concepts or tools or ideas or where you see the spaces going that you think is gonna have a lot of ⁓ value or growth potential?

Derek Meegan (1:10:53) Mm-hmm. Hmm. ⁓ I think the biggest alpha in AI right now is actually like upskilling yourself and making yourself smarter. ⁓ You know, I think trying to figure out how to do like how to literally automate your entire life with an agent, ⁓ you know, and not really actually, you know, thinking about the ways in which you're nurturing your own intelligence, you know, especially considering like how much

Gabe Tramble (1:11:26) Mm.

Derek Meegan (1:11:38) ⁓ you can use an agent to cannibalize your own thinking. A lot of people have, you know, talked about how they feel like their brain is going through like atrophy. I don't even know what the active verb word of it is, but they feel brain atrophy, right? Because they're not like, you know, it's like if you go to the gym, right? If you stop going to the gym, you know, you're not getting sore and therefore your muscles are not growing. And so then you kind of like lose your muscles, right? So like, I think it's really easy to like vibe code yourself into oblivion. Like you quite literally can like buy enough tokens to basically code continuously through each waking hour of the day and then go to bed. But I often find that those people that are doing that aren't really getting much done, right? Like they're kind of going in circles. So, you know, actually one of the technical leads here at Browserface Walker, a great engineer, and we were talking the other day and he was saying that he read the study where, you essentially, when you're doing something, you can either like focus on doing it or you can spend time like learning and upscaling. And it's like really difficult to be both doing something and upscaling at the same time. And I think that's even more pronounced in the age of AI. So I'd say there's like a lot of alpha and you know, like reading books, like studying, thinking, writing. along with, you know, of course, figuring out ways that you can augment your AI output, your output, your productive output with AI. ⁓ So yeah, like invest in the human, like, let's let's sort of like reframe a little bit, right. And I released this article on Twitter recently called like, what if fully autonomous is the wrong goal? Right. ⁓ Like, everyone is racing to make a fully autonomous like ABC, right? Like, I want a fully autonomous product marketer, right? Like,

Gabe Tramble (1:13:07) Yeah. Mm.

Derek Meegan (1:13:32) I want a fully autonomous, like software engineer, et cetera. And it's like, yes, we want to go into the future. And like, I, we want AIs that can like do stuff. Yes, definitely. Right. But like where, what are we ground? What outcomes are we grounding in? Right. And so that was my sort of call to action. The article, like, look, you're thinking about doing something with agents. Cool. Right. Or you're thinking about solving, you know, generally you're thinking about solving some problem. Cool. Right.

Gabe Tramble (1:13:45) Yeah.

Derek Meegan (1:14:01) What are the underlying goals or outcomes that you're actually trying to manipulate through solving this problem? Right? Like a fully autonomous agent is but one solution in the set of possible solution in this, in the solution space. Right? ⁓ so maybe instead of grounding yourself in that solution immediately, because that's broadly what the industry is trying to do, or because I don't know, like it feels cool or like it feels good, right? Like it seems cool. ⁓ instead what are, you know,

Gabe Tramble (1:14:14) Yeah.

Derek Meegan (1:14:30) sit down and define what are the concrete business goals that I actually want to complete and reclassify AI and particularly agentic AI as just but one tool in the possible set of tools that you can use to attack the problem. ⁓ And then approach the problem from first principles being agnostic to the solution, right? Like to any one solution. And I think this is most pronounced in areas where people are investing in AI that where people are heavily

Gabe Tramble (1:14:43) Mm-hmm.

Derek Meegan (1:14:59) doing the work? Do you think about software engineering is a classic example where so many companies are marketing and like aiming towards this idea of an autonomous software engineer. But like what key business problems are you actually solving in software engineering? Because the biggest business problems I see in software engineering right now are like constrained software engineering bandwidth, know, spiky workloads by AI, you know, just intense.

Gabe Tramble (1:15:00) Mm-hmm.

Derek Meegan (1:15:29) ⁓ explosion of cognitive load and being like kind of unable to ⁓ like offload that cognitive load. that's, these you know, these have always been problems, right? Or the review process is totally broken because I can, I can go codecs goal, make 1000 PRs right now and drop those PRs onto my team lead. You think he's going to be happy? Like, no, he's going to be pissed, right? Like that, like making code or like just generating output. It's like not the equivalent. of like ⁓ driving like an organizational outcome. Right. And so people are like really focused on fully autonomous, like they're going to achieve fully autonomous slop. Right. Where my framing is like, how do, what are like key, you know, what are key goals, like key outcomes that we want to attain. Right. And to what degree do we want to manipulate or drive these outcomes? And what are the tools at our disposal? And within that, like How can we think about work completion as not, there's all this work and there's all these AIs and like, how do we get all these AIs to do this work? And rather there are people, there are AIs and there are this work and how do all of those things interact in order to complete that?

Gabe Tramble (1:16:32) Thank Yeah. Wow, man. Yeah, that's really well said. I think I resonate very heavily with all that and kind of the double down on the like just coding to code and it's almost addictive and you can kind of go. Yeah, yeah, you can go into like.

Derek Meegan (1:16:57) It's a video game.

Gabe Tramble (1:17:02) like psychosis almost, right? Just coding and coding and coding and you work and then you kind of just flip into these new projects, new project, new project, and maybe after a month, okay, what have I shipped, right? And it's like this pseudo ⁓ accomplishment or like doing things. Yeah.

Derek Meegan (1:17:12) Yeah. It's like performative productivity. And like, look, there's nothing wrong with it if you like to code, but like recognize what you're doing, right? Like recognize that like you're having fun. And like, if you're good with that, like there is coding to have fun in my opinion, but like, I don't know. I don't show up, know, Paul, I hope you're not listening at this point. I don't show up to browser-based to code for fun, right? I'm coding for money and I'm like coding so I can make money and browser-based can make money, right? And so it's really important for us to like...

Gabe Tramble (1:17:31) Yeah. Yay.

Derek Meegan (1:17:45) stay grounded in what we're actually trying to do and what we're actually, the outcomes that we're actually trying to drive, right? And I mentioned this in the article too, using agents or more broadly developing software is a business activity, right? And business activities need to impact customers, business productivity or costs. And if you ain't doing that, then you ain't doing it, you know? So that's that. I'll get off my soapbox.

Gabe Tramble (1:18:12) Eric, man, this has been really good. Yeah, I appreciate you coming on and thanks for sharing about all this stuff and hope to have you on soon. Yeah, see you.

Derek Meegan (1:18:19) Yeah, thank you for having me.