Review Methodology

Start with the decision

Each review begins with the reader's real decision, then selects tasks that can answer it. We do not apply one generic benchmark to every product category.

Minimum test record

• Test date plus product, model, and tool versions.
• Machine, operating system, runtime, network region, and material configuration.
• Test repository or public input, common prompt, task description, and failure criteria.
• Raw output, elapsed time, tokens, estimated cost, and human interventions.
• Success, first-pass completion, and repair success shown as numerator and denominator.

Scoring

Scores must be calculated from published weights and raw results. If data is incomplete or candidates cannot be tested fairly, we publish factual differences without a composite score. Cost estimates cite the official price and its verification date.

Evidence

• Real screenshots retain useful version context while redacting accounts, keys, and personal data.
• Terminal logs, Git diffs, failures, and request logs stay with the conclusion.
• Architecture diagrams are labelled as illustrations; charts are generated from the published dataset.
• If paid functionality is inaccessible, it is marked not tested rather than represented by a vendor demo.