This page is in English.
In short
BenefitStep's text reader was run on 26 fictional test files: 25 different documents and one exact copy. It filled in 68 values, and 66 matched the document (97%). All 17 dollar amounts were correct. On layouts it did not recognise, it left fields blank instead of guessing.
One of the 66 was attached to the wrong document. In a file that holds two documents, the reader took the issuer from the first and the name and amount from the second. Counting that as an error too gives 65 of 68 (96%).
What the 97% means. It is the share of the values the reader filled in that were right. It is not the share of an application's fields that BenefitStep completes. When the reader does not recognise a layout it fills in nothing, and those blanks do not lower the figure. How much it filled in is reported separately below, as coverage.
This is a small test on documents written for the project. It shows that the reader does what it was built to do. It is not a measure of accuracy on real people's paperwork.
What was tested
- The local text reader in BenefitStep 0.3.11. This is the part that finds names, dates and amounts in documents that contain text, which covers most PDFs. It is the default way documents are read.
- With on-device AI off. The optional AI that reads photos and scans is not part of this figure.
- On the text of each PDF, as the extension's PDF library extracts it.
Not tested here: whether a value ends up in the right box of the PDF form, and how real applicants get on with the tool.
The test documents
There are 26 files in two sets. All are fictional and were written for this project, because real applicants' documents cannot be used for testing.
- Household set, 8 documents. One invented household's paperwork: two pay statements, a lease, a rent receipt, a utility bill, a child care invoice, a bank statement and a county request. They use the kind of labelled layout the reader was built for.
- Hard set, 18 files. Written to trip the reader up:
- 12 documents with unfamiliar layouts, or that only look like proof of income: a pay table in a grid, a two-page earnings letter, a vendor invoice full of payroll words, a Spanish receipt, an annual tax summary, bank deposits, a job offer, a utility bill with a credit balance, a damaged pay statement with the gross pay missing, a county request, a payroll teaching handout, and a lease that is not a receipt;
- 2 image-only scans, which a text reader cannot read;
- 1 copy with simulated OCR errors;
- 2 files that each hold two different documents;
- 1 exact duplicate.
Reference answers were written when the documents were authored, before the reader was run, and are never shown to it: 27 fields the reader should find, and 2 it must leave blank.
What was measured
Two things, reported separately.
- Accuracy of filled-in values. Every value the reader produced was compared with the document. A value counts as a match only if it is what the file says for that field. The headline figure is matching values divided by all values produced. A stricter count is given beside it: there, a matching value that the reader attached to the wrong document inside a file counts as an error.
- Coverage. Of the reference fields, how many the reader found, how many it left blank, and how many it got wrong.
They are kept apart because they fail differently. A blank field is safe: the person fills it in at the Review step. A wrong value is the failure that matters.
Results
| Household set | Hard set | Both | |
|---|---|---|---|
| Files | 8 | 18 | 26 |
| Files where the reader filled in anything | 7 | 4 | 11 |
| Values filled in | 56 | 12 | 68 |
| Values that match the document | 55 (98%) | 11 (92%) | 66 (97%) |
| Stricter: match, and belong to the right document | 55 (98%) | 10 (83%) | 65 (96%) |
| Dollar amounts correct | 14 of 14 | 3 of 3 | 17 of 17 |
| Reference fields found | 16 of 16 | 2 of 11 | 18 of 27 |
| Reference fields left blank | 0 | 9 | 9 |
| Reference fields wrong | 0 | 0 | 0 |
| Fields that must stay blank, left blank | none in this set | 2 of 2 | 2 of 2 |
The hard set shows the reader's limit. It found only 2 of the 11 reference fields there and left the other 9 blank. It did not put a wrong value in any of those 11 fields, and it did not turn a visible take-home amount into the missing gross pay.
Across both sets it found 18 of the 27 reference fields. The 12 values it did produce on the hard set include one that was wrong and one that was misplaced, described next.
Two values did not match the document:
- Utility bill, household set. The street-address field also took the city, state and ZIP code, because the bill separates them with a dot.
- Bank statement, hard set. The account holder's name came out with account text attached.
One value matched but was attached to the wrong document:
- Two documents in one file, hard set. The issuer is the one printed on the pay statement, but the reader kept it with the name and amount from the utility bill behind it. This is the difference between 97% and 96%.
The reader's own guess at the type of document was also wrong on three look-alikes in the hard set (an invoice, a bank statement and a teaching handout were taken for pay statements) and on the two-document file. It filled in no pay amounts from any of them.
Limitations
Only the values the reader filled in are scored for correctness. The 97% says how often a filled-in value was right. It says nothing about fields the reader left blank, and it is not a rate of completed application fields. Coverage is in the results table: 18 of 27 reference fields found, and only 2 of 11 on the hard set.
The documents are made up. They were written by the project, not collected from real households. Real paperwork is messier.
The sample is small. 26 files and 68 values. One more wrong value would move the figure by about one and a half points.
It is not a held-out test. The household set was written with labels the reader already supported, and both sets were looked at while the reader was being improved. A fair accuracy claim needs new documents the reader has never been adjusted to.
The values were graded once. Nobody has repeated the grading independently. The graded list is below and in the repository, so anyone can check it.
On-device AI is not included, and it is less accurate. In development runs on the hard set with version 0.3.7, reading with on-device AI got 103 of 160 values right over three runs (64%) and 35 of 39 dollar amounts. Most of the errors were placeholder text such as “unknown” accepted as an address; version 0.3.8 rejects those. The current version has not been measured again, and AI results change from run to run.
Other checks
Separately from this evaluation, each release is checked by automated tests (358 for version 0.3.11) and by browser runs of the whole workflow in each interface language. Those show that the software behaves as written. They are not accuracy measurements.
Whatever the reader fills in, BenefitStep shows each value next to the document it came from, and the person confirms or corrects it before it reaches a form.
Repeat it
The test files, the reference answers and the graded values are in demo/evaluation/ in the repository, and the script is scripts/evaluate-reader.mjs. From the repository root, run node scripts/evaluate-reader.mjs. It needs Node.js 22 or later and nothing else. It prints the figures in the table above and stops if the reader produces a value that has not been graded.
Every value and how it was graded (68)
| Set | File | Field | Value the reader filled in | Result |
|---|---|---|---|---|
| Household | 01_JuniperMarket_Pay_2026-09-15.pdf | issuer | Juniper Market | Matches |
| Household | 01_JuniperMarket_Pay_2026-09-15.pdf | person | Jordan Rivera | Matches |
| Household | 01_JuniperMarket_Pay_2026-09-15.pdf | period_start | 2026-09-01 | Matches |
| Household | 01_JuniperMarket_Pay_2026-09-15.pdf | period_end | 2026-09-15 | Matches |
| Household | 01_JuniperMarket_Pay_2026-09-15.pdf | pay_date | 2026-09-15 | Matches |
| Household | 01_JuniperMarket_Pay_2026-09-15.pdf | pay_frequency | semimonthly | Matches |
| Household | 01_JuniperMarket_Pay_2026-09-15.pdf | gross_pay | 1250.00 | Matches |
| Household | 01_JuniperMarket_Pay_2026-09-15.pdf | net_pay | 1054.38 | Matches |
| Household | 01_JuniperMarket_Pay_2026-09-15.pdf | ytd_gross | 22500.00 | Matches |
| Household | 02_JuniperMarket_Pay_2026-09-30.pdf | issuer | Juniper Market | Matches |
| Household | 02_JuniperMarket_Pay_2026-09-30.pdf | person | Jordan Rivera | Matches |
| Household | 02_JuniperMarket_Pay_2026-09-30.pdf | period_start | 2026-09-16 | Matches |
| Household | 02_JuniperMarket_Pay_2026-09-30.pdf | period_end | 2026-09-30 | Matches |
| Household | 02_JuniperMarket_Pay_2026-09-30.pdf | pay_date | 2026-09-30 | Matches |
| Household | 02_JuniperMarket_Pay_2026-09-30.pdf | pay_frequency | semimonthly | Matches |
| Household | 02_JuniperMarket_Pay_2026-09-30.pdf | gross_pay | 1300.00 | Matches |
| Household | 02_JuniperMarket_Pay_2026-09-30.pdf | net_pay | 1096.55 | Matches |
| Household | 02_JuniperMarket_Pay_2026-09-30.pdf | ytd_gross | 23800.00 | Matches |
| Household | 03_WillowCourt_Lease.pdf | issuer | Willow Court Homes | Matches |
| Household | 03_WillowCourt_Lease.pdf | person | Jordan Rivera | Matches |
| Household | 03_WillowCourt_Lease.pdf | period_start | 2026-09-01 | Matches |
| Household | 03_WillowCourt_Lease.pdf | period_end | 2027-08-31 | Matches |
| Household | 03_WillowCourt_Lease.pdf | rent_amount | 1400.00 | Matches |
| Household | 04_WillowCourt_Rent_2026-09.pdf | person | Jordan Rivera | Matches |
| Household | 04_WillowCourt_Rent_2026-09.pdf | issuer | Willow Court Homes | Matches |
| Household | 04_WillowCourt_Rent_2026-09.pdf | document_date | 2026-09-03 | Matches |
| Household | 04_WillowCourt_Rent_2026-09.pdf | period_start | 2026-09-01 | Matches |
| Household | 04_WillowCourt_Rent_2026-09.pdf | period_end | 2026-09-30 | Matches |
| Household | 04_WillowCourt_Rent_2026-09.pdf | rent_amount | 1400.00 | Matches |
| Household | 04_WillowCourt_Rent_2026-09.pdf | amount_paid | 1400.00 | Matches |
| Household | 04_WillowCourt_Rent_2026-09.pdf | expense_frequency | monthly | Matches |
| Household | 05_JuniperEnergy_2026-09.pdf | person | Jordan Rivera | Matches |
| Household | 05_JuniperEnergy_2026-09.pdf | issuer | Juniper Energy | Matches |
| Household | 05_JuniperEnergy_2026-09.pdf | home_address | 2148 Willowmere Lane, Apt 3B · Sacramento, CA 95814 | Wrong. The street field also holds the city, state and ZIP code. |
| Household | 05_JuniperEnergy_2026-09.pdf | document_date | 2026-10-01 | Matches |
| Household | 05_JuniperEnergy_2026-09.pdf | period_start | 2026-09-01 | Matches |
| Household | 05_JuniperEnergy_2026-09.pdf | period_end | 2026-09-30 | Matches |
| Household | 05_JuniperEnergy_2026-09.pdf | prior_balance | 800.00 | Matches |
| Household | 05_JuniperEnergy_2026-09.pdf | current_charges | 140.00 | Matches |
| Household | 05_JuniperEnergy_2026-09.pdf | amount_due | 940.00 | Matches |
| Household | 06_LittleLantern_Care_2026-09.pdf | person | Jordan Rivera | Matches |
| Household | 06_LittleLantern_Care_2026-09.pdf | issuer | Little Lantern Learning | Matches |
| Household | 06_LittleLantern_Care_2026-09.pdf | period_start | 2026-09-01 | Matches |
| Household | 06_LittleLantern_Care_2026-09.pdf | period_end | 2026-09-30 | Matches |
| Household | 06_LittleLantern_Care_2026-09.pdf | document_date | 2026-09-30 | Matches |
| Household | 06_LittleLantern_Care_2026-09.pdf | amount_billed | 300.00 | Matches |
| Household | 06_LittleLantern_Care_2026-09.pdf | amount_paid | 200.00 | Matches |
| Household | 06_LittleLantern_Care_2026-09.pdf | expense_frequency | monthly | Matches |
| Household | 08_ValleyCounty_Request_2026-10-02.pdf | person | Jordan Rivera | Matches |
| Household | 08_ValleyCounty_Request_2026-10-02.pdf | issuer | Valley County Services | Matches |
| Household | 08_ValleyCounty_Request_2026-10-02.pdf | program_stated | CalFresh | Matches |
| Household | 08_ValleyCounty_Request_2026-10-02.pdf | document_date | 2026-10-02 | Matches |
| Household | 08_ValleyCounty_Request_2026-10-02.pdf | request_item | Income received during September 2026 and current rent expense | Matches |
| Household | 08_ValleyCounty_Request_2026-10-02.pdf | period_start | 2026-09-01 | Matches |
| Household | 08_ValleyCounty_Request_2026-10-02.pdf | period_end | 2026-09-30 | Matches |
| Household | 08_ValleyCounty_Request_2026-10-02.pdf | stated_deadline | October 16, 2026 | Matches |
| Hard | document-06.pdf | person | Drew Marin • Account ••DEMO | Wrong. Account text is attached to the name; the document says Drew Marin. |
| Hard | document-06.pdf | issuer | Alder Grove Credit Union | Matches |
| Hard | document-08.pdf | person | Tessa Brook | Matches |
| Hard | document-08.pdf | amount_due | 0.00 | Matches |
| Hard | document-08.pdf | issuer | Pine Estuary Energy | Matches |
| Hard | document-09.pdf | person | Remy Lane | Matches |
| Hard | document-09.pdf | issuer | Birch Transit Collective | Matches |
| Hard | document-09.pdf | pay_date | 2026-09-25 | Matches |
| Hard | document-09.pdf | net_pay | 1470.26 | Matches |
| Hard | document-16.pdf | person | Tessa Brook | Matches |
| Hard | document-16.pdf | amount_due | 0.00 | Matches |
| Hard | document-16.pdf | issuer | North Alder Fabrication | Matches, wrong document. The file holds two documents. This is the issuer of the pay statement in it; the name and amount the reader kept come from the utility bill, whose issuer is Pine Estuary Energy. |