Methodology

Overview

Ry Walker Research publishes profiles and comparisons of companies, open-source projects, and technologies in the AI and developer tools space.^[1] This methodology defines the standard for new reports and refreshes. Its purpose is to make claims traceable, comparisons useful, and uncertainty visible; it does not certify that every older report has been rechecked under the current rules.

Research Process

1. Primary Sources

Every report starts with primary sources:

• Company website — Product descriptions, positioning, messaging
• Documentation — Technical architecture, capabilities, limitations
• Pricing pages — Business model, tiers, enterprise options
• GitHub repos — Code quality, activity, community health
• Funding announcements — Investors, amounts, valuations

Researchers read the underlying source, not just a search snippet. A working URL is not enough: the document must support the claim. Archived evidence can establish historical facts, but cannot verify current pricing or availability.

When sources conflict, researchers check publication dates, product versions, plans, and deployment options before combining their claims. Current product documentation, billing terms, or security documentation may be more specific than a launch announcement. If the conflict remains unresolved, the report states the uncertainty instead of selecting the most favorable interpretation.

2. Multiple Perspectives

Researchers seek perspectives beyond the vendor:

• Competitor analysis — How does this compare to 2-3 alternatives?
• User reviews — What do actual users say? (G2, Reddit, HN, X)
• Critical voices — What are the skeptics saying?
• Industry analysts — Any third-party coverage?

Profiles summarize relevant first-hand experience in "What Developers Say," with short quotations when useful. Attribution includes the platform or author, relevant affiliation, date, and a link to the discussion. Individual comments are evidence of those users’ experiences, not a representative survey.

There is no quota for positive or negative quotes. If research finds little independent discussion, the report describes that limitation and the date of the search. A limited search does not prove that no discussion or adoption exists.

3. Technical Validation

When useful and feasible, research includes:

• Demo or trial — Actually use the product
• API/SDK review — Check documentation quality
• GitHub activity — Commits, issues, contributor health

Reports distinguish documented capabilities, vendor-reported results, and actual hands-on tests. Benchmark claims retain their attribution and relevant workload, version, and date. Security claims describe the documented isolation boundary and limitations rather than treating “sandboxed” as a guarantee.

4. Structured Output

Profiles use a consistent structure, adapted for libraries, commercial products, and internal case studies:

• Executive Summary — What is this and why does it matter, with key stats
• Product Overview — Capabilities and product surfaces
• Technical Architecture — How it works, deployment model, integrations
• Strengths — What they do well
• Cautions — Honest assessment of limitations and risks
• Pricing & Licensing — Tiers, licensing model, hidden costs
• Competitive Positioning — Direct competitors and when to choose what
• Ideal Customer Profile — Best fit and poor fit
• Viability Assessment — Financial health, market position, outlook
• Bottom Line — Recommended for / Not recommended for / Outlook

Comparison reports add a market definition with explicit inclusion/exclusion criteria, a comparison matrix, gap analysis, and strategic recommendations. Counts refer to actual included members, not every product linked as context. Excluded or departed products remain clearly labeled. Recommendations should explain their evidence and tradeoffs rather than equating popularity with quality.

Category Membership and Product Boundaries

Inclusion depends on the report's stated criteria and the product's documented capabilities, availability, and intended workflow. Funding, popularity, a new brand name, or a recent release does not establish category fit. Changing the criteria requires reassessing existing members and credible new entrants alike.

Reports distinguish the company, its products, and the components those products use. A model, API, command-line tool, hosted application, and execution service can support different decisions even when they share an owner or underlying technology. Separate profiles are useful when those differences warrant distinct evaluation; closely related features can stay within one product profile.

Membership follows what a product actually provides. An application that calls MCP tools is not necessarily an integration platform. Sharing an agent template does not by itself establish a shared workspace for multiple people. An execution API must meet the sandbox comparison's isolation and environment criteria before it counts as a sandbox provider. Reports explain shared infrastructure so that multiple interfaces are not mistaken for independent execution systems.

Useful adjacent products can still appear in discussion, with their relationship clearly labeled. Candidates with unverified fit or unavailable capabilities remain distinct from established comparison members.

A selected comparison is not a complete market census. Reports state the scope of the evaluated set and identify credible coverage gaps separately; an unreviewed product must not be described as ineligible merely because its individual profile has not yet been prepared.

Report Depth

Reports pair concise summaries with enough detail to support a reader's decision. Where relevant, that includes architecture, deployment, concrete workflows, pricing units, limitations, and the tradeoffs between alternatives. There is no word-count target: length alone does not establish usefulness.

Refreshes preserve or improve useful, supported explanations and examples. Researchers seek replacement evidence for valuable material whose sources have become stale. They correct changed facts and remove unsupported claims, repetition, and filler. A shorter report can be appropriate, but meaningful losses of coverage must be explained in its editorial pull request.

Search visibility is an outcome to measure, not something a report's length can demonstrate. Claims about visibility gains or losses require evidence separate from the content review.

Citation Standards

• Relevant evidence: aim for at least five substantive sources for a full profile or comparison and seek independent criticism where available. For projects with less public evidence, disclose the limit and narrow the conclusions. Do not pad the bibliography. This policy page describes the site's process and is not subject to the report sourcing target.
• Traceable claims: cite material factual claims close to the supported text. Descriptions, takeaways, and FAQ must agree with the cited body. Retained sources need unique IDs and a relevant citation.
• Explicit uncertainty: remove unsupported assertions or state what could not be verified. Dated wording alone does not validate a claim; historical facts still need historical evidence.
• Dated metrics: attach verification dates to volatile facts such as prices, stars, and downloads. Distinguish billing units, annual discounts, and usage charges. Attribute vendor-reported adoption and benchmarks.
• Contextual quotations: quote only material read in context, keep quotations brief within applicable per-source limits, and link to the original comment or passage where possible.
• Source maintenance: repair moved documents, replace sources that no longer support the claim, and revise affected claims. A source with a different identity receives a new citation ID.

Opinion essays have no fixed source-count quota. Their factual claims still need evidence, but unrelated stock references do not improve support. When a source list changes, researchers preserve each retained citation's destination and check the rendered result; removing a bibliography entry must not silently redirect a numeric citation to another document.

Disclosure

Ry Walker is CEO and co-founder of Tembo, which provides a platform for running and coordinating coding agents.^[2] A primary purpose of this publishing program is to ensure Tembo is part of relevant product conversations. That interest influences topic selection and makes relevant Tembo coverage an explicit editorial priority.

Comparisons evaluate Tembo against their stated category criteria and include it in the matrix and decision guidance when it fits, linked to its research profile. An affiliation disclosure alone does not fulfill that coverage priority. Profiles and essays explain its competitive or complementary role with concrete distinctions supported by current product evidence. Adjacent context is labeled clearly, and unrelated subjects do not need a Tembo mention. The editorial PR records the inclusion decision and explains material removals of existing Tembo coverage.

The same evidence, citation, and limitation standards apply to Tembo and other products. Recommendations must explain fit and tradeoffs, and substantive Tembo coverage discloses Ry's affiliation nearby. This is founder-authored research; readers should take that commercial interest into account.

AI Assistance and Review

AI tools assist with source discovery, fact-checking, and drafting. Independent pages may be researched in parallel, with one writer responsible for each page and a coordinating pass for comparison claims. The pull request records which assistance was used; no single model or agent workflow is assumed for every report.

Refreshes are prepared as reviewable pull requests. Each PR records the material changes, supporting sources and verification dates, unresolved limitations, and validation actually performed. Editorial review remains a separate step; automated checks or an AI pass do not establish that a human has reviewed a report.

Publication checks validate research metadata, citation syntax and destinations, source IDs, internal research links, and declared comparison members. Changed pages receive stricter validation while some existing catalog problems remain reported as legacy warnings. A passing build therefore does not certify that every older report meets the current standard.

These checks establish structural consistency, not factual accuracy. Researchers still read the supporting documents, confirm that each citation supports the nearby claim, and compare declared membership with the actual editorial tables. They inspect rendered headings, tables, links, and citations as well as the source files.

A final consistency review checks the pages changed together: product names and status, comparison membership and counts, profile links, citation destinations, and agreement between summaries, FAQ, and the body. Related pages are checked where the revised claims depend on them. Any remaining contradiction or needed follow-up is recorded; this does not imply that every linked page received a full refresh. The public methodology is updated when the underlying policy changes, rather than receiving a routine date bump after each research batch.

Updates

Refreshes may cover an individual report, an explicit list of pages, or a whole category. The requested scope determines which files change. Updating a comparison requires checking its claims, but does not imply that every linked profile has also been rewritten. Overlapping research can reuse verified findings while keeping each requested page's changes in its own PR.

A refresh checks product identity and status, features, pricing and licensing, relevant adoption or funding claims, and the continued support of cited sources. Comparison refreshes also recheck category fit, recommendations, and credible new entrants. Every included tool links to an individual profile. A missing profile is created in a separate dependency PR and merged before the comparison that needs it. Small or new tools may instead link directly to GitHub with an explanation. Profiles link back to the relevant comparison when one exists.

Acquisitions, renames, and shutdowns do not erase a product's existing profile URL. Reports add sourced status changes and distinguish an announcement from a completed transaction, or a preview from general availability. Ownership alone does not establish a technical integration, a shared product roadmap, or a change in customer access.

The Published date remains the original publication date. Updated records a substantive revision. A verification that finds no needed changes does not receive an artificial date bump. Material corrections, acquisitions, shutdowns, and other changes can trigger a refresh outside any category schedule.

Citation and formatting maintenance preserves the research date when the substantive claims have not changed. A focused factual correction receives an update date and a scope note identifying what was rechecked; the rest of the report does not acquire a new verification date by association. Maintenance PRs distinguish mechanical repairs from factual corrections and record broader research gaps for follow-up.

Contact

• Research suggestions: @rywalker on X^[3]
• Corrections: ry@tembo.io

---

Research by Ry Walker Research