Surge’s new tax benchmark asks whether an AI agent notices the work it was not told to do

Fifty assignments, not fifty quiz questions

Surge AI released DAYJOB: Tax on October 27, describing 50 professional-style environments spanning eight areas of tax practice. Agents receive a request and supporting records, then produce work such as a return, workpapers, a technical review or advice. The benchmark is organised around completing an assignment, not answering a list of detached tax questions.

The publisher says the workspaces contain 1,789 files and that tasks are created and reviewed by tax experts. It describes a three-layer expert review for correctness, completeness and solvability. Those are methodology statements from Surge, not an independent audit conducted by Foundra. The environments are benchmark assignments, not evidence that this newsroom observed agents serving real tax clients.

The distinction changes how the result should be read. A tax question can explicitly identify what must be calculated. A professional-style assignment can require discovering which issue matters before doing the calculation. A score intended to assess the completed work therefore speaks to a different object from a score on isolated factual answers.

The file count has a distribution

Surge reports a median of 30 files per assignment and as many as 123, alongside the total of 1,789 across 50 environments. Dividing the total by the number of environments gives a calculated mean of 35.78 files. That derived average is not the median and does not describe the contents of every assignment.

The difference between the mean and median helps explain why one generic description of a workload can be misleading. The total describes the whole collection; the median describes its midpoint by file count. Neither says how difficult a particular record is to interpret. File quantity is an input measure, not a direct measure of professional complexity.

The publisher lists PDFs, spreadsheets, documents and other records among the inputs. That matters to the design because a finished answer can depend on reconciling evidence presented in different forms. It should not be enlarged into a claim that every model accurately reads every file type, or that increasing the number of files by itself makes an assignment more realistic.

A threshold governs the headline rate

Surge reports pass@1 at a 95% rubric threshold: an attempt passes if it satisfies at least 95% of the task criteria. Its published leading result is 10.8% for Claude Sonnet 5.5. That is an attributed result under the stated evaluation rule, not a finding that the model knows only 10.8% of tax facts or that a randomly chosen AI-written return has that probability of being correct. Surge says the criteria include calculations, supporting evidence and consequential issues. That scope explains why the rate cannot be read solely as a calculation test.

An assignment can contain correct intermediate work and still miss the threshold. The result is sensitive to what the rubric counts as part of the completed task. That does not make the threshold meaningless; it makes the grading definition essential. A low whole-task pass rate and a useful supervised subtask capability are not logically incompatible.

Nor should the result be inverted into a broad claim that every narrower tax tool is unsafe. A document extractor, a draft assistant and a system expected to complete professional advice have different scopes. The benchmark is most informative when its scope remains attached to the score, rather than being stretched to every product that uses AI in finance.

The release’s distinctive contribution is that it puts records, omitted issues and the final deliverable inside one evaluation. Its numbers describe that particular test, with the publisher’s task and grading choices still in view. The meaningful comparison is between the claimed job and the evaluated job—not between a familiar percentage and an imagined universal measure of professional competence.