Task quality, without the command line

Review a task before it ships.

Upload a task package and run the same Oracle, quality, and trajectory checks used by the team. You’ll get a clear pass/fail summary and a detailed report.

Step 1

Connect your access key

Connected as —

Step 2

Choose a task package

ZIP · max server limit applies

Drop your task ZIP hereor click to choose it from your computer

Select the checks to run

Oracle testRuns the reference solution and no-op baseline in Docker. Free.Quality reviewChecks task design against 25 QC criteria. Uses model budget.Trajectory reviewReviews agent attempts included in a Data OS export.Comprehensive reviewAn agent judges the whole task with the oracle and QC results: valid or needs revision, with fixes. About $1–5 and 5–20 minutes per task.

Maximum spend (USD)

Recent activity

Your review jobs

Connect your key to see recent jobs.

Developer access: API and MCP

Agents can use the same private service over streamable HTTP. Add this MCP endpoint with your bearer key:

https://task-checker-1.stem.turing.com/mcp
Authorization: Bearer oops_...

MCP tools

start_upload

Step 1 of checking a task. Returns a one-time upload link (valid 30 minutes) and the exact shell command that zips a task folder and sends it there. Run that command yourself, replacing TASK_FOLDER with the folder to check, then call submit_task with the upload_id.

submit_task

Step 2: queue the uploaded task(s) for checking. kinds is any of oracle, qc, at, comprehensive (comma-separated; oracle is free, the others use the key's money). Returns the job id.

job_status

Whether a job is queued, running, done, failed or cancelled, with its latest progress lines. Oracle runs can take up to an hour: wait a few minutes between checks.

job_results

A job's results per task: oracle and nop rewards, QC and AT verdicts, and every failed criterion with the reviewer's reason. Use this to explain the results or fix the task.

list_jobs

This key's 20 most recent jobs, newest first: id, status, kinds, upload name, cost.

my_budget

This key's spending cap, what it has spent, what queued jobs hold, and what is left.

cancel_job

Cancel a job that is still queued (a running job cannot be cancelled).

HTTP endpoints

GET/v1/health

Is the server up.

GET/v1/me

Your key: name, cap, spent, reserved and remaining $.

POST/v1/jobs

Send a task. Multipart: file (a .zip of one task folder or several), kinds (default oracle,qc,at,comprehensive), budget_usd (held against your cap while the job runs). Returns the job, 202.

GET/v1/jobs

Your jobs, newest first.

GET/v1/jobs/<id>

One job: status, plan, progress lines, summary.

GET/v1/jobs/<id>/results

Every review's full result (JSON).

GET/v1/jobs/<id>/report

The job's HTML report.

POST/v1/jobs/<id>/cancel

Cancel a job that is still queued.

POST/v1/uploads/<link>

A one-time upload link from the MCP tool start_upload (valid 30 minutes; the link is the credential).

POST/mcp

The MCP server: agents connect here and get the tools below.