← Capability matrixMethod & data on GitHub

End-to-end runs

Assistants doing concrete Google Docs tasks on synthetic documents, graded by inspecting the document afterwards against a snapshot taken before the run. pass: every requested effect and reported fact is right and nothing that must survive was lost. partial: the change happened but something was lost or misreported. fail: not done, wrong, or declined. The surface check runs the same 30 tasks through a web app and its command-line twin to see whether the CLI stands in for the app.

By category

By task

Click a task for failed checks and notes.