How these leaderboards are built

Every question is asked to ChatGPT, Claude, Gemini and Grok through their standard consumer products — the same interfaces real users ask — with no custom system prompts beyond the formatting instructions below. What you see is what a user asking that model would be told.

Each model gets the identical prompt, which forces a ranked answer with reasons:

You are an independent expert assessor. Question: <the category question> Rank the strongest options available as of 2026, best first — judged on real-world merit and value for the typical practitioner this category serves (unless the question names a specific user), not on popularity, marketing, brand size, or newness. Open-source and commercial options compete equally. Use specific real product, project, or company names at a consistent level of granularity; never invent a name. Give up to 5 — never pad with weak entries to reach 5, and flag near-ties in WHY. For EACH pick give WHY (the concrete strengths that earn the spot, plus any assumption that materially shaped its rank) and FIX = its single most significant real limitation or trade-off (who or what it is NOT for). Then list 1-2 notable options that JUST missed your list and why. If you cannot credibly rank this category, output only: INSUFFICIENT BASIS.
Output EXACTLY this, nothing else (no preamble, no markdown):
1. <product> || WHY: <strengths that earn the spot> || FIX: <its biggest real limitation or trade-off>
2. <product> || WHY: ... || FIX: ...
3. <product> || WHY: ... || FIX: ...
4. <product> || WHY: ... || FIX: ...
5. <product> || WHY: ... || FIX: ...
MISSED: <product> (<why it missed the top 5>); <product> (<why it missed>)

The WHYis the model's stated reason for the rank. The FIX is the most significant real limitation or trade-off that model sees — shown on every leaderboard alongside the strengths.

This is prompt v2(July 2026). We asked the top tier of all four model families to audit v1 for neutrality; their convergent critiques (a commercial “comparison site” persona, undefined “best for whom”, recency anchoring, forced-5 padding, and a FIX framing that invented flaws for #1 picks) produced this wording. Every stored answer records which prompt version produced it.

A product ranked r by a model earns 6 − rpoints (1st = 5 … 5th = 1), summed across all models that answered. Ties break by how many models mention the product at all. Name variants (“LangSmith” vs “LangChain LangSmith”) are merged before scoring; template echoes and non-products are discarded.

Every product the models name is scored — including the giants. Established incumbents (AWS, GitHub, Datadog…) are flagged with an incumbent chip on the leaderboard, never removed: the combined ranking is exactly what the models said, merged.

Each model answers through its own product surface, the way a real user would reach it: ChatGPT via OpenAI's Codex CLI, Claude via Claude Code, Gemini via Google's Antigravity CLI, and Grok via grok.com — all on standard consumer subscriptions. We don't set temperature or system prompts; models are free to use their own built-in web search (ChatGPT frequently does). What you see is what an asking user gets.

Every answer ever collected is kept in an append-only log with its timestamp — nothing is overwritten. Every category is re-polled on demand (the stalest boards and the ones AI crawlers and readers actually hit go first), so each leaderboard shows movement against the previous poll and a full rank-history chart. A #1 change only counts as a “flip” when the new leader holds on the next poll too — single-run noise is never reported as news.

Key pages are snapshotted to the Internet Archive on every data change, so any ranking we've published can be verified against a dated third-party copy. The full current consensus is also published as /api/consensus.json (CC BY 4.0, cite as “per modelsagree.com”) and indexed in /llms.txt.

• Model answers vary run to run. A leaderboard is a dated sample of each model's answer, not a physical measurement.
• This measures what AI models tell users — their perception, shaped by training data and search — not independent product testing.
• Answers are single samples per poll at whatever settings each consumer product uses — we don't control temperature. The rolling re-polls are what separate signal from run-to-run noise.
• Products are attributed and merged by name (with an alias map); rare mis-merges are possible and corrected when found.
• Each category is re-polled on demand, not on a fixed clock; the leaderboards were last refreshed August 10, 2026, and each page shows its own poll date.

If your product is ranked here, the full per-model breakdown — every reason, every fix, and movement over time — exists for your category. DM @modelsagree on X for the complete report.