Neo4j has spent years refining the web interface developers use to work with its graph database. A developer using a coding agent may never see it. The agent researches the product, picks an implementation, provisions it, and starts building through the command line and APIs.
It’s not going to go into the browser, look at the tooltips, look at how we’ve laid out the information architecture.
For Yaru Lin, Lead Product Manager at Neo4j, that is a new product surface. “It’s going to depend on a whole different set of other material,” Yaru says. “It might be doing research, but it’s going to be looking at docs, and it’s not going to read the whole docs. It’s going to look for things that it thinks it needs.” The result is “a whole different field of user/agent experience.”
Yaru uses Gauge to evaluate those agent journeys, inspect the documentation agents actually read, and test what changes their decisions. The findings have overturned internal assumptions, set onboarding priorities across Neo4j’s local and Aura cloud experiences, and become the foundation of Yaru practice of Agent Led Growth.
Learning agent experience by living it
Yaru’s own path through Gauge was eye-opening. She started in the Gauge UI to understand what was available, then installed the CLI. Once a template evaluation looked right in the browser, with the prompt, persona, and codebase all visible, she could hand it to a coding agent and ask for seven more variations. When a run’s judgment was not the one she wanted, she could pull the data through the CLI and rejudge it herself.
I’m spending less time in the Gauge UI now because of this CLI. This is very much a learning experience that I need to translate to how people are using Neo4j.
That experience produced an insight Yaru is still working through. Humans do not stay in one surface. When an agent dumps a complex result into text, a person needs a visual to make the judgment call. Shortly after, they might go back to the terminal. “We need some sort of handover between the two experiences that’s a lot more frictionless than it is today,” Yaru says. “Most of the onboarding I have seen from other products assumes you’re solely going to use the terminal or solely going to use the UI. But it might actually cut in and out quite a bit.”
The experiment that exploded
Yaru began the way a product manager would: map the customer lifecycle, list the tasks inside it, test the corresponding flows. On paper it looked like a checklist.
When that user flow doesn’t go quite as you expect, you begin to tweak variables here and there and this suddenly explodes into so many different variations.
One investigation looked at how agents recommend a Neo4j implementation: local for a proof of concept, self-managed, or Aura in the cloud. Yaru could vary the user, the company, and the application being built. Then, the pages that the agents were reading lead to a second set of questions. Would the recommendation change if that information were laid out differently? What if the prompt told the agent to do first-party research before anything else?
“Right now it feels like there’s an endless number of rabbit holes,” Yaru says.
Testing the docs agents actually read
Agents often read far more pages at once than a developer. A single run might fetch ten documents, which raises the question of which one to change. Gauge’s documentation rewrite feature lets Yaru intercept what an agent obtains from a page and test a change to one document, then a second, then a combination, to see whether behavior actually shifts.
It’s the sheer volume of variations you have to try that becomes overwhelming. Having a platform that can help with that is super helpful, instead of managing all of this through a directory.
The same logic applies to launches. Yaru now prototypes an onboarding idea and evaluates it against agents “before you even decide to go to engineering.” Alongside Gauge, she runs some of those tests locally with harnesses such as Pi and Codex, mocking the responses an agent would receive. Test before you ship, in Yaru’s phrasing, is becoming the default.
What changed inside Neo4j
Yaru shares the results with marketing and product marketing, the documentation team, the CLI and API tooling teams, and the product-led growth team that owns the human user experience. Watching real agent behavior has been the most persuasive input.
There are a lot of assumptions that we make that are just constantly being pushed over.
Those findings fed a concrete prioritization decision: where developers are actually entering the product, whether through Aura or the local experience, and which onboarding to build first. They are shaping what a first-class onboarding flow in the Aura console should look like when a developer is using a coding agent. And they have pushed the team to treat docs as something agents consume, not just people. “Even in the days before agents, nobody used docs,” Yaru says. “Now it becomes asking your agent to help you find certain pages of relevant information. How do we make sure it lands for the agent?”
The hard part is translation. Running evaluations continuously, with a coding agent on top producing analysis, generates an enormous amount of output. Colleagues do not need “my 120 different HTML files.” They need the interesting things learned about agents, the experiments and outcomes, and the decision chain: because A happened, we investigated B, and here is where we are today.
The new practice: Agent Led Growth
Agent Led Growth is the idea that optimizing for agents ultimately drives visibility for the product and consumption of it. Yaru believes that the whole user lifecycle could be mediated by agents. Today, discovery and agent experience are already critical. Soon, we may be upselling to agents, retaining agents, and collecting feedback from agents.
It is a multidisciplinary practice by necessity. Marketing is adding AEO alongside SEO. The product-led growth engineering team is applying its onboarding expertise to a new kind of user. “It’s still one onboarding experience into your product,” Yaru says. “It’s just now across multiple surfaces and possibly multiple personas.”
The new operating reality
Several habits are emerging from the work.
Agent flows next to user flows. Every feature should be considered for agent consumption, not just humans. Yaru is already considering whether PRDs should include agent flows alongside user flows, tested in Gauge before they go live.
Change management instead of regression testing. Agents behave differently from users. “You realize just how bold the agents are at extrapolating what it thinks you need,” Yaru says, “and then just going and doing it.” Their behavior also shifts between model and harness versions in a way human behavior never does.
It’s not even called regression testing anymore. This is just called change management with regard to harnesses and models.
Dogfood before you design. “It is hard to think about the agentic experience for your users if you yourself don’t use enough of it,” Yaru says. Neo4j is a context engine, so its internal agent workflows should run on a Neo4j context graph, a digital twin of what customers are building.
Make the framework repeatable. Yaru’s next goal is an evaluation framework that any engineering pillar can follow when given the mandate to ship for agents as well as people. Teams need to know which tools to use and how to use them step by step, or have an agent walk them through it.
What other product leaders can take from this
- Use agents heavily yourself. You cannot design an agent experience from the outside. Yaru’s CLI habit produced the handover insight now shaping Neo4j onboarding.
- Start from a lifecycle task, then expect it to branch. Choose one journey, like selecting an implementation, define success, and budget for the variations that appear once the first run surprises you.
- Inspect what agents read, not what they fetch. Change one document at a time and confirm the behavior moved before changing the next.
- Ship the decision chain, not the run logs. Bring colleagues what you learned, what you tried next, and the priority it informs.
- Know when to pull back. An interesting result is not the same as the product question you came in with.
Yaru describes the relationship with Gauge as one of growing together. “I’m really hoping that we could grow with each other as we figure out what exactly this agentic experimentation looks like.” For Neo4j, the payoff is already a more grounded practice of Agent Led Growth.
