--- title: "How Clerk 3x'd CLI signups with one error message change using Gauge (video)" description: "Clerk's Devin Gould joins Ethan Finkel to discuss the SDK error message that helped agents discover Clerk's CLI, testing the change in Gauge, and the 3x increase in CLI adoption after launch." url: "https://www.withgauge.com/blog/how-clerk-3xd-cli-signups-with-one-error-message-change/" author: "Ethan Finkel" published: "2026-11-05" --- # How Clerk 3x'd CLI signups with one error message change using Gauge (video) [Devin Gould and Ethan Finkel: Clerk's CLI onboarding experiment (video)](https://player.mux.com/Q7MZoEBbYqrTG9mvovvZmclDLLG9UHV00VZIA41vUM9A) ## The conversation Clerk's Growth Engineer Devin Gould joins Ethan Finkel to talk about a small SDK change with a measurable impact: an error message that directed coding agents away from a deprecated setup path and toward Clerk's CLI. Devin tested the change in Gauge before shipping it. The evaluations showed agents following the new guidance and using the CLI; after launch, Clerk's own tracking showed CLI adoption tripling. We discuss how the experiment worked, why making a product easier for agents can require different choices than making it easier for humans, and how Devin brings evaluation results into code review. Read the [Clerk case study](https://www.withgauge.com/case-studies/clerk/) for more on the experiment and what the team learned.
Transcript **00:01** All right, Devin. Thanks, thanks for joining us for our, our customer interview today. Um, excited to, to dive into your thoughts on, on agent-led growth and, and agent experience here. Um, so first question is, what are you and your team trying to make possible for developers at Clerk? Yeah. Um, I think what we're trying to make possible is, um, adding auth to your app with your agent in a way that's, like, seamless and, and easy to do and, and sort of without, with as little human intervention as possible and, and at the end you get like a high quality sign-in, uh, that's secure and, and ready for, uh, all your users. **00:45** And what should choosing and implementing authentication feel like when that experience is working at its best? Um, I mean, easy. I think that's sort of our, our, like, uh, general approach is just what's, what's easiest to set up and, um, and makes you f- like trust that it works and that it's secure and that the agent's, um, not gonna make the mistakes that, that a human might or that an agent might without, um, Clerk. Makes sense. And what did your use- usage data first tell you about the growing role of coding agents? **01:24** Um, yeah, I mean, I, I think, you know, what, about a year, may- maybe not quite a year ago, we saw the numbers just sort of like really, um, sort of skyrocket. Um, maybe like beginning of this year, specifically like w- a lot of the, we saw more users, more traffic, more applications being created, users were integrating faster. Um, and that's sort of all without us doing anything additional. It's just the agents are now suddenly getting better. Um, and you know, even though we hadn't optimized at all for it, like we're s- we were seeing lots of growth. **01:59** Um, I think we're still trying to like compound that by, um, making sure that the experience is like, as, uh, sort of agent friendly as possible and figuring out where, where they might get stuck. So it sounded like the agents just started, like, using Clerk more and more, and like their, the takeoff was kind of organic to start. When did agents stop feeling like another developer tool and start changing how you thought about user acquisition, user onboarding, and the Clerk product itself? **02:30** Um, yeah, I mean, I think for me, a lot of it comes from just like my personal experience using the agents. Um, I can name multiple products that I had not used before, and I like to use agents, especially like, again, beginning of this year, things started to get crazy. Um, I was using agents a lot just in personal projects, and suddenly I'm-- they're selecting databases for me, they're selecting, um, auth for me, they're selecting- Yeah ... uh, providers for me, and I just was-- I- it seemed pretty obvious to me, like I, I am now a customer of multiple companies that I had never heard of until my agent added their products to my, like, little toy app. **03:09** Um, so it just felt like a natural, like if it's happening to me, it's, it's happening to a lot of people. That makes a lot of sense. You're basically like, "Agents are now picking tools for me. At Clerk, I should make it so agents pick us." [chuckles] Yeah. Right? Yeah, exactly. Yeah. Awesome. And, uh, what's a, a recent example of some agent behavior that made Clerk want to try something different? Um, recent behavior, I think... Well, I mean, the, the first thing that jumps into my, my mind is we were trying to get, um, agents to, um, to use our, um, CLI, our, uh, as, as sort of the, the agent tool. Um, we found that sometimes they were discovering it on their own, and sometimes they were using it, and sometimes they were not. **04:04** Um, and part of that had to do with how many of these different pathways, we sort of had these, um, these SDK features that enabled people to get started really quick, but it actually sometimes made it too easy for the agent, and then the agent didn't realize- Mm-hmm ... like, "Hey, there's this helpful tool that not only can get me past this step, but then can get me like further through the pipeline if I know it exists." Um, so specifically, like we, we sort of took away this, um, intermediate step of trying to help the user get started as quickly as possible, um, automatically, and gave them this error that says, "You know, use the, use this CLI, and it's easy to use. It's one, one command," but now the agent knows about it too. Uh, and now the agent can go and grab that and take off. **04:46** Interesting. So it sounds like there was a bit of a trade-off there between like the ease of, uh, ease of implementation of Clerk and the usage of your preferred tool or setup, right? Yeah. I think you have to be like somewhat intentional about what the agent is going to do, um, and like really think through the, especially when agents are like really, really smart, but they, a lot, especially right now, they don't really know much about, um, the current state of our product, like if things sort of lag behind. **05:20** Um, so you gotta think through like what do they know that we can do, and what are they, what do we want them to know right now? And just because it's easier from like a human perspective doesn't mean that the agent is gonna know. Um, and so you have to like kind of think through that from a more practical perspective of like where, what, where is the agent gonna get stuck in the idea. It doesn't, just because it's easier for a human doesn't necessarily mean it's easier for an agent. **05:46** Totally. That makes sense. And when you were doing this change from the SDK to the CLI, like adding that little bit of friction so the agent got on the right rails, like what did you change or test to have that happen, and what did you learn when you watched that, that agent journey? I mean, so we made changes in our, in our SDK, um, and started out just, you know, I mean, it throws an error. Um, had the general, like, a pretty strong conviction that it was going to work, just like it seemed like that was a sensible thing to do if we throw an error. **06:21** Agents are pretty keen to, like, fix errors. Um, and if we throw an error giving explicit instructions, like it seemed like it would follow that. Um, but we didn't know for sure. Um, and we used Gauge ultimately to, to take the, uh, our sort of snapshot of our SDK, send it to the agent, um, in a prompt and say, "Hey, install this. Try to get the app up and running." That throws an error, and we went from, like, basically the agent never using our CLI to the agent always using our CLI. **06:52** Um, so that was like a really, I mean, that was a big, a big step. That's awesome. Yeah. It was really cool to see you guys do that experiment and, like, get the CLI to actually get installed by the agents. I'm curious, like, what was the, the business outcome of that? Like, how did that actually change your, your, your metrics internally? Yeah, I mean, we do see, um, uh, we, we see the users that use our CLI definitely just perform better. **07:15** They, they get further down our, our sort of growth, um, funnel, uh, at, at higher rates. Um, and yeah, we've seen like, I think, I don't know the exact number of, like, percentage increase, probably like double or triple the number of CLI users basically, um, overnight from, from doing that. Uh, I think you can ki- we-- My assumption at the moment is it's like pretty high converting if you're integrating Clerk, uh, with an agent, it's probably gonna use the CLI. Um, yeah, and so we saw that, like, I mean, it's a big, a big spike in the, in our charts. **07:54** That's crazy. That's, that's awesome to see. Um, and once you started to see this kind of like new opportunity of adding the, the error, getting the agents to use the CLI, like how did that information permeate through the Clerk org? Like, how-- Were you sharing that with product teams or broader leadership? Like, how did the, the kind of information permeate through your, your company? Yeah, we definitely shared it around and, and people were excited about s- seeing that. And, you know, it was kind of like there was a convergence of ulti- other sort of, um, teams that were starting to think agent first, and this was like a good data point of like what that really means. **08:32** You know, like we would talk a lot about, um, you know, "Hey, we need to be working and, and improving the experience for agents," but some- without like a clear case of it actually working, it's kind of hard to know what that's, that means. So this was a good example of like, you know, and I think we're gonna keep expanding. You know, we, we liked the error. We found that, uh, like looking at our errors works, um, that had a high impact, so we're gonna like expand that and, um, try to figure out if there's other, you know, specifically with errors or probably, there probably are cases where we could, um, improve the agent experience. **09:07** Uh, and yeah, that's sort of now everybody started thinking more and more about that. That's awesome. And so I'm curious, with the success of this, like an issue that you did on improving agent experience, like how has that changed leadership buy-in or like the company's perspective on optimizing for agents? Yeah, I mean, I, I think it definitely like helped sell the idea that it was like you can get measurable results, especially when you look at the head data. Like again, if you look at our charts, you can sort of see when we shipped the change. [chuckles] Uh- Yep ... so it's like rather than sort of a like, um, sort of ephemeral like score or something that's on a dashboard, it's like, well, no, we actually had like a direct intent of what we wanted the agents to do, and then we were able to ship a change and make that happen more often. **09:58** Um, and that comes up now in our data, so it's kind of like a no-brainer to say like, "Hey, yeah, we, we can do stuff like this. Like so this, this, this works." [chuckles] Yeah. Yeah. It's awesome. And across these experiments you've been running, what has Clerk learned about how agents experience your product? Um, I think we've learned, again, to like not try to, try not to make too many assumptions about what the best human experience is when you're thinking about it. **10:37** You sort of need to marry the two and try to find like a middle ground of what's gonna work for the agent and what's- Mm-hmm ... gonna work for the human. Um, especially with like things like the CLI, like the agent wants to interact with the CLI differently than a human's gonna-- Like they don't like an interactive CLI, so don't build them an interactive CLI. And they, they sometimes expect things, um, and certain behaviors and certain commands and like if you notice that an agent is, um, trying to run some command that doesn't exist because it just assumes that it exists, like just add it because there's a reason that it wants that. **11:12** Uh, and like if you see it in one agent, you probably are gonna see it in, in, you know, thousands of agents or however many are running. Um, so yeah, and I think like trying to measure that and, and actually like keep track of that is important. Yeah. That's, uh, no, that's, that makes sense. I'm curious on the, uh, agent interactivity versus like human interact- interactivity. Like what have you kind of been, been seeing there? How have you been trying to optimize the CLI for, for agents? **11:43** Like what's worked? Um, yeah, I think I've, you know, I think with our CLI trying to get, um, better metrics around, um, sort of how far through the process they, that a user can get, like what commands are they running the most, which agent is running those commands, has been really helpful to then say, you know, to, to sort of notice gaps, um- Mm-hmm ... in where, where that, uh- Things might be getting stuck. Um, and yeah, just like one example is like being able to go to production completely over the CLI is something that, um, we've seen as like an issue that agents run into. **12:28** Um, and yeah, by, by just sort of interacting with the agents and trying to understand what they want and why, why they might be getting stuck and seeing, seeing their traces and their sort of chain of thought, um, you usually can see like, okay, maybe w- if we do-- if we try this like this, this will probably work. Yeah. And how is the, uh, agent experience insights you've been getting changing the way that your team builds or how your growth product DevRel and engineering team works together? **13:03** Yeah, I think the, the biggest thing is for, for many cases thinking more like almost eval first. Mm-hmm. Like, okay, yeah, we can make this change and ship it, but it is hard just in the, um, you know, just in the data to know like is there a specific case of where the agent can-- it's where the agent is gonna get lost. I can just try to run some locally, but there's enough like variance that, um, that we're probably gonna miss, miss things. **13:39** Um, so I think at least on my team, like if, if we're gonna make a change that we think might impact agents, like the first conversation we have is can we-- is this something that's like easily eval-able and something- Mm-hmm ... we can get evals on so that we can know like, okay, yes, this is for sure the right approach. Um, yeah, so that's sort of I think that's the biggest like sort of mindsha- mindset shift. So you guys are now like eval first basically. **14:07** Like you're, you're like proposing something and then you're like, let's eval the baseline, let's eval, uh, you know, our changes, and then like ship whatever moves the eval the most, right? Yeah. Yeah, I think like I've been thinking of it, you know, I'm, I'm a growth engineer, so pri- prior to agents, like their growth engineering was like running A/B tests on users, uh- Yep ... to say, "Does, does this change improve the user's behavior?" Um, you can't quite do that like real time on agents. **14:36** You can't measure their sessions in like in the wild, but you actually can spin up, you know, thousands of simulations essentially and, and do the same sort of A/B test and actually get results out, um, the other side. So, uh, I sort of think of it as the same thing as it was before. It's just kind of like the, the tools are a little bit different and you're in a different-- you're living in a different place, and you're a little bit further. **14:57** It's actually nice because you're a little bit ahead of the curve of- Mm-hmm ... you don't actually have to ship it to real users. Um, you can test it ahead of time. And so it sounds like before you were doing basically like, uh, ask users A/B test features kind of optimized for that. But now with like spinning up these swarms of agents, you're like testing things before they go out, um, kind of iterating thousands of samples in like a day and then seeing what the results are. **15:21** Like how has your day-to-day changed as a result of optimizing for agents instead of optimizing for, uh, humans? Um, I f- it-- I think that, um, the approach that I've taken is more maybe like a little bit more, um, sort of a I, I don't know if I wanna call it scientific, but, um, we can, um, I can do sort of early experiments just locally. I, I think that's typically how I would take the approaches. If I wanna make a CLI change, I'll just run it in my agent locally and say like, "Okay, this looks good." Um, and once we start going through like code review, we can say as part of the code review, what ha- like I can, I can now like link to specific, um, results sort of ahead of time, whereas prior... **16:23** And that makes it a little bit easier, right? 'Cause I can, I can back up my claim of why I think this experiment is working with data. Mm-hmm. Um, I think that was, that's been an issue with, with typical A/B testing when you're testing on users is like at the last second someone goes, "Hey, why is, why do you wanna make this change?" And now I have to like justify why I think it might work instead of just saying like, "Well, we already just ran it and it works. **16:48** Uh, and that's why I think it, this is a good change." So it kind of like you can kinda go into the conversation like with a little bit more data to back things up. That makes sense. So instead of being like at kind of review time, um, yeah, I talked to a bunch of users, they said this thing, and so I, I think this fix will fix that thing, but I didn't, you know, implement the fix and then go back and talk to the users, right? Like that, that step wasn't closed. **17:11** I don't have data that my fix actually changed that user behavior. But with optimizing for agents, you can see the broken path, right? Put up a prospective fix, test the broken path again, and be like, "All right. Actually this fix like does work," right? [chuckles] If I, if I ship this, this will actually improve the, the symptom that the agents were complaining about. Exactly. That's awesome. Well, one thing that's interesting about Clerk is, you know, you by changing your, uh, SDK error messages to point agents towards the CLI, you've already shown, you know, Clerk is willing to change product services so that they communicate like directly to agents. **17:49** Um, why do you believe that's the right approach to take here? Um, I mean, the agents are interacting with these product services like probably at this point more than humans. Um, it depends on the, on the specific surface, but agents are-- I would, I'm-- These, these numbers are hard to know for sure, but I would assume agents are reading more Clerk errors by like, a large factor- Mm. ... uh, than humans. So like, why wouldn't we think of them as, um, sort of the primary customer? **18:25** I know personally, like I haven't really read an error, like, stack trace in a, in a while it seems. Um, so yeah, I think sort of thinking through from that perspective of like they're actually doing the work more, um, it's not true for every surface, but that's sort of, um, it- it-- to me, it, to me it's a pretty, um, big signal of like, okay, yeah, these, these-- this is the way that this data is gonna get consumed anyway. Um, so it makes sense to, to optimize for it. **19:03** Makes sense. Yeah. That's definitely where the usage is, is coming from now and hopefully more so in the future as well. [clears throat] And what is already different or what do you hope will be different for developers who are using Clerk through an agent? Um, I think the, the way that I think of it is it, again, should be like fast and easy. Um, I think there's this other thing that I've been thinking of, of like, it also should make the-- like the user should feel good about it and that, and understand it and know what happened. Uh, you know, like I think people are still looking at code some amount. Well, you know, is that still the case in a year? I don't, you know, maybe, I don't know. **19:51** But at the moment at least, like it needs to be the, a mix of like, um, the agent needs to be able to actually do the, do the thing, but then, um, make sure that they're sort of also helping guide the user journey in a way. Like we kinda see them almost as like a, an ambassador for us or like a, like a dev advocate or some- something along those lines. Like once, once- Yeah ... the agent's convinced that, okay, we're gonna install Clerk, like we want them to sort of be the thing that's pitching for us and saying like, "Hey, did you know we have this feature? Did you know we have this?" Like, "Did you know that this is secure for this reason?" And like those sorts of, um, sort of behaviors, I think. **20:35** And what are you watching to know that this is, is working, like this behavior shift is working, and where has Gauge been most helpful with that? Yeah. I mean, mainly Gauge. I think, um, yeah, trying to keep track of, um, sort of specific goals that we have has been really helpful. Like, again, we wanted our CLI to be, um, sort of the main entry point for agents. Uh, and so with Gauge, like makes it really easy to sort of track that behavior. Um, I think there's like so many behaviors that we wanna track, and we're still building out this like sort of larger catalog of specific, um, sp- specific behaviors that we think we want agents to, to show. **21:21** Um, and then on the, you know, the sort of, that's like the positive behaviors I guess you could call them. But then there's also just like the, along the way, along that journey, Gauge is then really helpful to make sure that there isn't these things that we're not thinking about. Like, "Oh, did you know that- Mm-hmm ... that URL's 404ing," or the agents think that this URL is real and it's not, and like maybe we should just make it because it's gonna be easier to just make that domain or that URL redirect than try to convince the agents otherwise. **21:48** Um, and so that's been, that's been really helpful to, uh, yeah, just like keep track of those things and, and kinda know that that's, someone's thinking about it. Because I can, I can c- you know, maybe come, make things up in the back of my head, but until you actually see like a real session that happened that did a thing, that you don't, you either wanna, you wanna sort of manipulate or, or change that behavior in some way. Um, until you see that, it's really hard to justify making changes. **22:17** Yeah. That makes sense. And, [clears throat] um, as you're thinking about like making these changes specifically for agents, like what part of a developer product do you think now needs to serve both the person and that person's agent? What part of a developer product? Yeah. Like what parts of your product are you thinking about as like being both for humans and both for agents? Like are there things that are agent specific, things that are human specific? What kind of lies in the, in the middle there too? **22:48** Yeah. Um, yeah, I mean, so Clerk's like kind of core product experience is kind of around the SDK. Um, you have to integrate it, that into your code. I kind of see that as still as like it's both for humans, it's both for agents. Um, agents are primarily, I think, gonna write, um, and use the SDK from like writing the code perspective. But again, like I think one of the nice things with Clerk is for a lot of cases, it's very simple to implement. So then when you go and actually see what the agent did and you're like, you wanna confirm that it was correct, um, it's pretty, pretty s- simple to review. So that's sort of, um, I think it's useful for humans and for, for agents there. **23:34** Um, and then sort of the di- there's like a divergence too, though. Like we also, we have, so we have a dashboard, and we have a CLI. Humans are using the dashboard primarily, and agents are using the CLI, and you can basically accomplish the same tasks. Um, but we sort of think of like a new feature. It's like, okay, it needs to exist now in, in both places. It needs to be usable by agents, and it needs to be like usable by humans. And, and when you do that, there might be some slight differences in how you, how you represent things, because people like visuals and color and nice design. **24:07** Uh-huh. Agents just want like a great way to access the data as quickly as possible and, and do what they want. Totally. No, that's, that's interesting as you think about, like, those, those new service areas as well. Um, and how do you expect product development and developer relations to change over the next few years? I mean, I think the agents are just becoming more and more important every day. Like- Yeah ... um, the rate of improvement is, um, pretty wild. Um, I feel like agents didn't exist maybe a year ago. **24:42** I don't-- It's tough to keep-- Sometimes when I look at the dates, it kind of seems surprising that it was so recent. Um, so yeah, I mean, I would just expect more of that shift to happen even further to the agents. Um, and, you know, I think it's just gonna become more important to making sure that you're serving the agents probably more as the primary consumer of your, of your app, um, and humans become more secondary. And maybe that means that there are... **25:16** You know, you, you can think through what-- You know, I think humans, we're still gonna want to see things, and we're gonna wanna interact with, with apps, uh, and visuals and UIs and those sorts of things. But the, the use cases might change because now a lot of that, um, sort of like configuration and technical details has all shifted to the agents. Mm-hmm. Um, but we'll want a-- We'll, we'll want a way to, like, oversee and, and, and sort of keep track of the work that's going on. **25:44** That makes a, a ton of sense. Yeah, definitely like a hybrid workflow in the future, but like what parts are agent and what parts are human might become more distinct in some ways. Mm-hmm. Um, well, I'm curious, what, what do you think other developer tool companies misunderstand about this change? I feel like it's really hard to, to accurately predict what the future is gonna bring. A-and I- Mm-hmm ... I personally feel like a lot of companies are not, um, sort of, I don't know, ambitious enough, or they're not, like, they're not necessarily seeing the potential next couple years of developer-- of the change in the developer experience and the change in, in how agents consume products and, um, and yeah, I think that things are gonna change very quickly, and a lot of companies are just like, kind of not, not really, uh, anticipating that, that that's gonna happen as rapidly as I think it will. **26:54** What do you think they should be doing? I mean, building for agents and, and become like sort of, I don't know, um, maybe I'm just making this term up right now, but like agent obsessed, uh, is- Yeah ... like the, the Bezos approach is customer obsessed. Like maybe we all need to be agent obsessed and, uh, and just be building for, for what they want, um, and thinking about them first. What does, what does being agent obsessed mean? Like what, what are you guys doing at Clerk to be agent obsessed? **27:26** I think every conversation has agents involved. Like every single product thing that we're doing has agents as, as something that's happened. Um, pretty much every initiative is somehow like tangentially related to whether it's like, oh, we're, we're seeing more-- We need to-- Our fraud detection needs to be better because we're seeing all these agents, and agents are now so capable or just like, "Hey, we wanna release this feature." Well, if it's not, doesn't have like agent friendly, um, sort of interface, like via the CLI or API or whatever, it's kind of not, it's incomplete. **28:00** Um- Mm-hmm ... and yeah, and I think that applies to like, uh, sort of all of the conversations that are going on. And how are you measuring being agent obsessed? Yeah, I mean, that's a tricky one. We do have like, um, you know, I, I think some of the, the numbers that I look at the most are like skill at number of users using the CLI, number of users- Mm-hmm ... um, using, um, some other like, you know, uh, v0 or Replit or, um, Lovable or, or those, those apps. Um, so sort of like looking where the agents are, are living and then yeah, using like something like Gauge to, to measure what the behavior is and, and, uh, sort of having that be a main surface of where, um, where we look to, to sort of see how we're doing as far as are the agents getting stuck and are they, are they doing the things that we want. **29:03** Nice. Um, well, those were the, the kind of prepared questions I had. I'm curious, Devin, are there any other things you think I should have asked or, or things you wanna go over that I, I didn't ask? Um, no. I mean, I feel like that was... I can't think of anything at the moment. Nice. Sick. Well, I'll hit the, uh, the stop recording button here.
--- Full index of this site for agents: https://www.withgauge.com/llms.txt --- ### AI Visibility Growth offer For AI Visibility Growth only. Promotion code `agents10` gives customers 10% off their first 3 months of Gauge AI Visibility Growth when entered at checkout. [View pricing](https://www.withgauge.com/pricing/).