Build vs. Buy: Agent-Led Growth

Key takeaways

• Building your own means owning an evaluation platform. Isolation, execution, judging, storage, analysis, and maintenance all become your team's responsibility.
• Use Gauge open source for the build path. Get controlled local experiments without rebuilding the runner. It is free, and you can use your existing agent subscriptions or API keys.
• Use hosted Gauge for a team program. Gauge runs the sessions, structures and stores the evidence, and makes the results available to teammates and coding agents.
• Hosted gives you more control over what agents encounter. Test preview pages through their normal URLs, compare changes with Gauge Optimizations, and run evaluations on PRs.
• Start hosted with 1,000 free credits. Paid plans start at $100/month. For a dev-tool company, the engineering time saved is the more consequential part of the comparison.

Our recommendation: use Gauge open source to explore individually. Start with hosted Gauge when the results are something your team will rely on.

Build your own vs. Gauge open source vs. hosted Gauge

Agent-led growth depends on whether coding agents choose your product and can successfully use it. Testing both requires realistic tasks, controlled conditions, and evidence you can act on. Here is what each path involves.

Capability	Build your own	Gauge open source	Hosted Gauge
Controlled starting context	Build isolation and input configuration	Fresh Docker sessions with configured inputs	Managed isolated sessions with configured inputs
Product inputs	Build installation and version handling	Skills, npm CLIs and SDKs, docs and files	Also website previews, native binaries, and authenticated MCPs/APIs
Preview content at production URLs	Extremely difficult: build transparent interception and routing	Use local file inputs; hosted required for response rewriting	Route requests to preview content inside the sandbox
Execution at scale	Own provisioning, orchestration, failures, and teardown	Docker on your machine	Managed compute and greater concurrency
Judging and friction analysis	Build scoring and evidence extraction	Your coding agent analyzes sessions and saves findings	Automatic judging and structured findings
Data storage and team access	Build a shared database and access layer	Local SQLite, transcripts, and diffs	Hosted, structured data with shared results and history
Comparisons and optimization	Build baseline tracking and variant comparisons	Local before/after comparisons and queries	Managed baseline and trial workflows, stats, and dashboards
PR checks	Build and maintain CI integration	Script local runs yourself	Native GitHub App checks and hosted CI workflows
Cost	Engineering, infrastructure, and model usage	Free software; your compute and agent usage	1,000 free credits; paid plans from $100/month
Best fit	Specialized infrastructure requirements	Solo developers exploring agent behavior	Teams and recurring agent-led growth programs

The open-source CLI and hosted product share a workflow, with different execution, storage, and automation capabilities. You can build custom workflows on either without recreating the underlying runner.

Control exactly what the agent knows before each run

Your everyday Claude Code or Codex session already has context: instructions, installed skills, conversation history, and potentially memory about you and your company. Asking it to test your product does not remove that context. A clean repository alone does not create a controlled experiment.

Suppose you want to know whether a revised skill makes your product easier to use. You need the same task, starting repository, model, harness, and other inputs, with the skill version as the change. Otherwise, you cannot confidently attribute the result to the skill.

Gauge open source starts fresh Docker sessions with explicitly configured inputs. Hosted Gauge manages isolated execution and adds organization-managed resources such as versioned skills, authenticated tools, and personas. Pin the repository commit and model when comparing runs, and repeat the task to account for variation. Configure runs

Building this yourself means controlling what enters every session and maintaining that isolation as your agent harnesses and tools change. Gauge already provides the foundation.

Test unpublished pages through the URLs agents already use

Telling an agent to read a special test page changes the context of the experiment. Ideally, it should request your normal documentation URL and encounter the proposed content through that same path.

Hosted Gauge controls that boundary. When the agent requests your configured production docs origin, Gauge can route the request to your PR's preview. You can test revised content before publishing it without putting a preview URL into the task. The preview must be publicly accessible and associated with the candidate commit. Docs preview setup

Recreating this is extremely difficult. You need interception and routing that work with the agent's requests, preserve the intended experience, and keep each session's candidate content separate. Putting a coding harness in a GitHub Action does not give you that capability.

Local Gauge supports testing documentation as file inputs. Hosted response rewriting is the stronger option when the experiment depends on how agents encounter content on the web.

Run evaluations at scale without maintaining the infrastructure

A custom system needs to prepare environments, install harnesses, supply inputs, run sessions, capture evidence, handle failures, and tear everything down. At scale, it also needs scheduling, concurrency management, and reliable tracking of unfinished work.

You can build that system. Supabase Evals is a public example of the substantial infrastructure behind evaluating coding agents on real product tasks. Supabase is also a Gauge customer. The broader lesson is that running a useful evaluation program involves much more than writing a prompt.

Gauge open source handles execution on your machine. That is a good starting point for an individual testing a skill or package. Hosted Gauge runs the sessions on managed compute, including when your laptop is closed, and supports recurring measurements.

If you are already planning a runner, GitHub Actions orchestration, result ingestion, and a database, you are planning an internal evaluation platform. Use Gauge's CLI and hosted infrastructure, and spend your engineering time on the workflows specific to your product.

Measure task success and find friction across runs

An agent finishing its session does not mean it successfully implemented your product. An agent succeeding does not mean the experience was smooth.

Gauge evaluates those separately. Agent Experience uses an LLM judge to check completed work against your criteria. A run passes when every criterion passes. Findings can also identify friction outside those criteria: repeated page fetches, confusing instructions, contradictory documentation, or commands that lead the agent down the wrong path. Agent Experience

Agent Preference examines which products the agent chooses and whether it actually adopts them in code. Being mentioned, being recommended first, and being installed are different outcomes. Agent Preference

Local Gauge captures the evidence for your coding agent to analyze and save as structured judgments and findings. Hosted Gauge performs the analysis automatically. Building independently means implementing both evidence collection and the analysis that makes those sessions useful.

Keep every run's evidence structured, queryable, and shared

This is one of the biggest reasons to choose hosted.

Local Gauge stores measurements and findings in SQLite alongside transcripts and diffs. It supports local queries and comparisons, so you can investigate your own experiments without building a reporting system.

But a team needs a shared history. When someone changes a skill, everyone should be able to see which version was tested, what failed, whether the next version improved, and the evidence behind that conclusion. Separate databases on individual laptops do not provide that shared view.

Hosted Gauge handles the storage and structure of the data as well as the sessions that produce it. Teammates work from the same results. Coding agents can retrieve structured evidence through the CLI or MCP and use it to help investigate or implement a fix.

If you build independently, the shared database, ingestion pipeline, query layer, and access management are additional systems to operate. If you stay local, you have to arrange sharing and reconcile histories yourself. For recurring team work, hosted Gauge removes that burden.

A/B test your pages and keep the version that performs best

Once you find friction, Gauge Optimizations lets you test whether a change fixes it. Choose a page or skill, hold the task and agent/model configuration constant, and run a baseline. Then author a change and evaluate it through repeated trial sessions against that baseline.

Think of it as optimizing the environment around the agent. In model training, reinforcement learning uses feedback to improve the model. Here, you keep the model and coding harness fixed and change the context they consume. Evaluation results tell you which version to keep.

Your coding agent can help write the variants. Gauge measures their effect. Inspect the evidence, choose the strongest version across the tested runs, publish it through your normal workflow, and validate it against the live product. Completing an optimization does not deploy the change. Gauge Optimizations

Local Gauge supports a manual improvement loop with before/after comparisons. Hosted Gauge provides managed baselines and trials, including changes to web responses. Building the equivalent yourself requires tracking variants, delivering the correct context, collecting comparable results, and preserving the evidence for each decision.

Check product changes in GitHub before you merge

Hosted Gauge's GitHub App runs committed evaluation cases against candidate skills, packages, executables, files, or docs previews. It uses your existing build artifacts and previews, then posts outcomes and session links on the PR.

Experience checks show whether the agent met the criteria. Preference checks report which product it selected; choosing a competitor is an advisory result, rather than a failed implementation check. Explicit before/after studies use comparisons or Optimizations instead of assuming every PR check runs both versions. GitHub PR checks

An internal build needs to connect candidate commits, inputs, execution, judging, and reporting. Gauge connects those steps for you. You can also use the hosted CLI from your own CI if you prefer to control the trigger.

Compare hosted fees with the engineering you would own

Hosted Gauge starts with 1,000 free credits, with no card required. Paid plans start at $100/month for 1,500 credits, or approximately 6.67 cents per credit. Larger plans reduce the unit price, reaching below 5 cents per credit at higher volumes. Current pricing

Using the rounded recent averages shown on our pricing page:

Evaluation	Average credits per run	Approximate cost at the entry plan's credit rate	Approximate runs from 1,000 free credits
Agent Experience	20	$1.33	50
Agent Preference	10	$0.67	100

These are estimates, not fixed run prices. Consumption varies with model choice and task. Open-source models are substantially cheaper than frontier models. Monthly plans provide a credit allowance with no automatic overage charges.

For a well-capitalized dev-tool company, the more consequential cost is the engineering time spent maintaining execution, data storage, analysis, and integrations. Hosted fees should be compared with that ongoing work.

Use open source for individual exploration. Choose hosted Gauge for a team program. For enterprise initiatives, our managed services can help with evaluation design, strategy, and implementation, including PRs to improve customer documentation.

Start with hosted Gauge, get the open-source CLI, or talk to our team.

Frequently asked questions

Why can't I just ask Claude Code to test my product?

You can get useful feedback, but your existing session carries prior instructions, skills, and conversation context. It does not establish how an agent with a controlled starting environment behaves. Gauge isolates the test session and supplies the inputs you intend to evaluate.

Is Gauge open source free?

Yes. The CLI is MIT-licensed, and local execution requires no Gauge account. You supply Docker compute and your agent subscription or API credentials. It includes execution, evidence collection, local storage, and tools for your coding agent to analyze and compare results.

When should I choose hosted Gauge?

Start hosted when teammates need the findings, evaluations recur, or you need preview routing, automatic analysis, Optimizations, or native PR checks. Hosted manages both the execution and the shared data behind the program.

Can my coding agent operate Gauge?

Yes. The CLI includes instructions for coding agents and supports structured output. We recommend using your coding agent to help configure evaluations, investigate evidence, and prepare improvements. Hosted data is also accessible through MCP.

Does Gauge automatically publish the winning page?

No. Optimizations measures variants against a baseline. You review the results and publish the selected change through your normal content or code workflow, then retest the live version.