The decision to test
Run the same difficult cases with each native thinking configuration. Do not compare a non-thinking Qwen probe with a full DeepSeek reasoning response as if they were the same workload.
Three criteria for this comparison
- Final-answer correctness with a fixed rubric
- Reasoning and visible output tokens together
- Time until the final usable answer
Run a fair comparison
Use the same representative inputs and a written answer rubric. Pin the model version and record the endpoint, thinking configuration, quantization when known, context length, output allowance and concurrency. Preserve native request differences in an adapter rather than silently dropping fields.
Measure full request time, error rate and reported usage. For tools, validate arguments and score the result after execution. For images, preserve resolution and preprocessing. A valid response format alone does not establish a correct answer.
Compare cost per accepted result
Calculate input cost plus all billed output, including reasoning where the provider counts it. Add retries and discarded answers. Divide the total by the number of results that pass your quality criteria. Keep API usage cost separate from deployment and engineering cost.
Estimate token costs with the API calculator →