FLONNO / REPRODUCIBLE EVALUATION
Agent API checks, with evidence
The model runs on our own GPU system in Switzerland. This small operator-run evaluation checks specific behavior through the public authenticated HTTPS API. It is not an independent benchmark or proof that the model suits every task.
Measured: 2026-10-17T10:52:57Z. Model: qwen3.8-27b. Result: 19/19 checks passed. The main cases have three trials each. The two-request concurrency case and two-turn tool loop are separate.
What was tested
| Case | Passed / checked |
|---|---|
| Nested tool arguments · non-streaming | 3 / 3 |
| Nested tool arguments · SSE streaming | 3 / 3 |
| Two tool calls in one streamed response | 3 / 3 |
| JSON object and expected field values | 3 / 3 |
| Medium thinking with JSON output | 3 / 3 |
| Two concurrent JSON requests | 2 / 2 |
| Tool loop · validated initial call | 1 / 1 |
| Tool loop · answer from client tool result | 1 / 1 |
Tool checks require the expected function name, a nonempty call ID, complete parseable argument JSON and the exact expected nested field values. The parallel case requires both expected calls. Streamed fragments are assembled by tool-call index and checked after completion; the test also checks finish events, the SSE terminator and token usage. The client validates a synthetic tool call, supplies a quantity of 17 and checks the final answer.
For ordinary and thinking JSON requests, parsing alone is insufficient: the returned object must equal the expected city and count. Thinking uses reasoning_effort: "medium", a 1,024-token output budget and reported reasoning-token usage. Other cases disable thinking to isolate API behavior.
Usage and billing reconciliation
A dedicated synthetic test wallet started with $1 of evaluation credit; no customer balance or payment card was used. Its temporary scoped key was revoked after the run. The sum of charges calculated from response input/output usage was 323630 nano-USD ($0.00032363). The wallet debit and usage ledger matched exactly. Pending reservations after the run: 0.
Rates used: $0.03 per million input tokens and $0.25 per million output tokens, including reasoning, rounded up to a whole nano-USD per request. The test script is public; checking your wallet and ledger requires your account access. The public script does not create free credit or access the database.
Reproduce it
Download the Python 3 standard-library script, inspect its synthetic prompts and run it with your own scoped key. It makes billable requests. It does not execute external tools or print your API key.
# Set FLONNO_API_KEY securely in your local environment first.
python3 evaluate-agent-api.py > report.jsonFull synthetic responses, usage and durations (JSON). The recorded gateway image is flonno:b2fa9010a80ad5019a30f76bb2092cee5486d52c. The report describes that dated run rather than a continuous quality score. Current GPU artifact identity is documented on the model page.
What this does not establish
- Operator-run, not independent; 3 trials per main case.
- Synthetic prompts, no customer data. No statistical guarantee or general task-accuracy benchmark.
- Concurrency tested at 2; no saturation/load test or SLA.
- tool_choice=required is unsupported. The supported automatic tool selection was evaluated.
- Full JSON Schema compliance and named forced-tool selection were not evaluated.
- Public minute probes disable thinking; this separate evaluation includes medium reasoning.
The first diagnostic run used forced tool_choice: "required", which the runtime explicitly rejects with HTTP 400. The successful run uses automatic selection. Forced selection is not silently substituted, and automatic selection does not guarantee that a model calls a tool. Larger real task suites, long contexts, production load, interruptions and independent accuracy testing need separate evaluation.
Seven-day status, latency percentiles and capacity → · API contract and supported fields →