Benchmark results

What was measured under the protocol in BENCHMARK.md. Each fenced block is generated by the script its marker names, whose --check fails when the block is stale, and recomputed by a test on every CI run. Prose outside a fence is written by hand.

Section	Generated by	Recomputed by
Gate 1	bench/gate1_score.py	tests/unit/test_gate1.py
Fix dry run	bench/dryrun_score.py	tests/unit/test_dryrun.py
Round 1	bench/summarize.py	tests/unit/test_bench_summarize.py
Error analysis	bench/errors.py	tests/unit/test_bench_errors.py
Comparison arm	bench/comparison.py	tests/unit/test_bench_comparison.py

Scan precision and recall (gate 1)

Computed from the frame in bench/gate1/sample.yaml, the five answer keys in bench/gate1/labels/ and the scan records in bench/results/gate1/scan.json. The keys were written before the scanner ran on those repositories. Definitions: BENCHMARK.md; the reasoning behind them is written up separately.

Frame drawn 2026-09-18T14:05:55Z over the term generativeai; 80 repositories in the frame, 80 scanned, 665 file(s) selected and 144 parsed -- the byte prefilter eliminated the rest -- for 701 findings over 210 file(s), 90 of which nothing bails in.

Measured with obelize 0.7.0 on CPython 3.12.9 / arm64, against pack gemini/google-generativeai-to-google-genai sha256 5017c6856ceb122c.... Nothing re-fetches eighty checkouts on CI, so these records are a dated artefact: a change to the scanner does not invalidate them visibly, and re-running python -m bench.gate1_collect scan is what makes the number current again. A change to what the pack tells a scan to look for does fail a test -- the projection fa59b7a16c9d722b..., which is the whole of what scan/ and impact/ read, and not the pack's own bytes, which move whenever a rewrite rule gains a parameter.

Precision and recall, over the hand-labelled repositories. Actionable kinds only (attribute, call, import, method_call); a row matches on path, line, kind and symbol together.

Repository	Files read	Labelled	Found	TP	FP	Missed	Precision	Recall
AyaanKhaan5/Story_Create_AI_Agent	3	8	8	8	0	0	100.0%	100.0%
LucasHJin/vit	3	13	13	13	0	0	100.0%	100.0%
Vaibhav8299/Conversational_AI_App	7	20	20	20	0	0	100.0%	100.0%
xNabil/discord-ai-bot	3	14	14	14	0	0	100.0%	100.0%
yihong0618/tg_bing_dalle	4	9	8	8	0	1	100.0%	88.9%
Total		64	63	63	0	1	100.0%	98.4%

Everything else those repositories reported. Excluded from the rate above; the keys predict these rows too, and the positions they forbid outright are counted under the table.

Kind	Reported	Expected by the keys	Unexpected
manifest	7	6	1

13 position(s) the keys forbid outright, 0 of them reported.

Repositories with at least one file where nothing bails. The number Phase 2's scope turns on: a file with an actionable eligible finding and no withheld finding.

File-size band	Population	Scanned	Errored	With a zero-bail file	Share
size:<1000	41,152	8	0	6	75.0%
size:1000..4000	122,112	24	0	18	75.0%
size:4000..16000	169,984	33	0	24	72.7%
size:>16000	79,488	15	0	9	60.0%
Total	412,736	80	0	57	71.2%

Of the 80 repositories scanned, 47 had nothing withheld anywhere, 10 had a zero-bail file beside a withheld one, and 23 had no migratable file at all.

What withheld the rest. The ranked reason a real file is not migratable, which is the table Phase 2's rule order is read from.

Bail	Rows
file_not_fully_migrated	123	derived from another row's bail
multiple_assignments	41
repo_not_fully_migrated	27	derived from another row's bail
receiver_unresolved	24
multiple_configure_calls	17
conditional_binding	14
model_object_escapes	11
local_import	9
client_source_unresolved	9
attribute_removed	5
module_alias_rebound	3
Primary causes	133

What repositories declare about their Python. blocked: runtime_unsupported is read from pyproject.toml, setup.cfg, a Pipfile or a string python_requires= in setup.py; this is how often any of that fires outside a fixture.

Declares	Repositories	Reads as
(none declared)	75	not blocked
==3.10.*	1	not blocked
==3.11.5	1	not blocked
>=3.11	1	not blocked
>=3.8	1	blocked
^3.10	1	not blocked
Blocked	1

Static resolution, over every usage in the frame. Gate 1 narrows scan when more than 30% of usages cannot be resolved statically.

Usages	Statically resolved	Unresolved	Unresolved share	Gate 1 trigger
615	577	38	6.2%	does not fire

The one miss is client.last, an attribute read on an untyped function parameter, in a module where client.send_message(...) on the same parameter is reported: a defect, tests/fixtures/scan/COVERAGE.md gap 20, left unfixed so the number is what the shipped scanner produces.

The one unexpected row is the key being incomplete: one requirements.txt line yields two manifest findings, the added google-genai pin beside the withheld legacy pin, in a repository that migrates one of its two modules.

The scan records were taken again for 0.7.0, against the scanner and the Gemini pack of this tree (obelize scan --pack, since a scan without it runs every pack a repository uses), over the same 80 repositories at the same commits, so they read obelize 0.7.0. Every record is the one 0.6.0 gave, finding for finding. The shares are the ones the record taken with 0.1.0 gave, 57 repositories (71.2%) with a zero-bail file and 47 with nothing withheld, apart from one repository that now stops on the Python floor it declares. Precision and recall are unchanged.

The zero-bail share decided Phase 2's scope. It clears the roughly 40% threshold pre-registered before the round, although the three development-corpus repositories the rules were written against produce no automatic edit. It describes small public applications, not codebases in general (the sample's biases).

Fix dry run

What the rules alone would write for each bundled pack over a stratified frame of eighty public repositories that use its library. Each record is one obelize fix --pack <id> --model none dry run at the repository's pinned commit, kept in bench/results/dryrun/; the frames are in bench/gate1/. Definitions and the frames' biases: BENCHMARK.md; the reasoning is ADR-024 D10. The records were taken again for 0.7.0, against the packs and the engine of this tree, over the same frames at the same commits, so they read obelize 0.7.0. Every record is the one 0.6.0 gave, withheld row for withheld row. Each record names the sha of the pack it was taken with, and tests/unit/test_dryrun.py fails when a pack has changed since.

gemini/google-generativeai-to-google-genai

obelize 0.7.0, pack sha256 5017c6856ceb, frame drawn 2026-09-18T14:05:55Z for generativeai (412,736 files in the index): 80 repositories planned, 0 errored.

Outcome	Repositories	Share of planned	Share of those the pack ran on
blocked: the pack stops before it plans, for the reason below	1	1.2%	--
none: no usage found	0	0.0%	0.0%
held: usages found, no file written	40	50.0%	50.6%
partial: a file written, a row withheld	12	15.0%	15.2%
complete: a file written, nothing withheld	27	33.8%	34.2%

Why the pack stopped	A manifest declares it	Repositories
runtime_unsupported	yes	1

Why a row was withheld	Rows	Repositories
file_not_fully_migrated	185	48	a consequence of another hold
client_placement_ambiguous	52	7
repo_not_fully_migrated	51	41	a consequence of another hold
multiple_assignments	41	11
response_shape_changed	39	11
receiver_unresolved	24	11
multiple_configure_calls	17	2
conditional_binding	14	2
model_object_escapes	11	7
client_source_unresolved	9	3
local_import	9	5
attribute_removed	6	3
history_parts_shape_incompatible	3	1
module_alias_rebound	3	1
alias_collision	1	1
configure_consumed_elsewhere	1	1
configure_kwargs_unsupported	1	1
from_import_unmigrated_symbol	1	1
manifest_pin_shape_unsupported	1	1
usage_unmapped	1	1

The 10 symbols that hold back most rows	Bail	Rows	Repositories
google.generativeai.GenerativeModel	multiple_assignments	21	11
google.generativeai.GenerativeModel.generate_content	response_shape_changed	19	9
google.generativeai.GenerativeModel.generate_content	multiple_assignments	18	9
google.generativeai.configure	client_placement_ambiguous	15	7
google.generativeai.GenerativeModel	client_placement_ambiguous	12	4
google.generativeai.GenerativeModel	response_shape_changed	12	11
google.generativeai.ChatSession.send_message	client_placement_ambiguous	11	2
google.generativeai.GenerativeModel.generate_content	receiver_unresolved	11	8
google.generativeai	local_import	9	5
google.generativeai.GenerativeModel.generate_content	multiple_configure_calls	9	2

openai/openai-0-to-1

obelize 0.7.0, pack sha256 62b392e7f30d, frame drawn 2026-10-13T13:11:56Z for "openai.ChatCompletion" (89,152 files in the index): 80 repositories planned, 0 errored.

Outcome	Repositories	Share of planned	Share of those the pack ran on
blocked: the pack stops before it plans, for the reason below	27	33.8%	--
none: no usage found	0	0.0%	0.0%
held: usages found, no file written	26	32.5%	49.1%
partial: a file written, a row withheld	0	0.0%	0.0%
complete: a file written, nothing withheld	27	33.8%	50.9%

Why the pack stopped	A manifest declares it	Repositories
legacy_version_unsupported	no	22
legacy_version_unsupported	yes	4
runtime_unsupported	yes	1

Why a row was withheld	Rows	Repositories
repo_not_fully_migrated	103	34	a consequence of another hold
file_not_fully_migrated	79	24	a consequence of another hold
flag_only_surface	72	17
response_shape_changed	34	19
unsupported_kwarg	25	8
conditional_binding	20	1
legacy_version_unsupported	11	8
module_alias_rebound	3	2
from_import_unmigrated_symbol	2	1
attribute_removed	1	1
manifest_pin_shape_unsupported	1	1

The 10 symbols that hold back most rows	Bail	Rows	Repositories
openai.ChatCompletion.create	response_shape_changed	23	17
openai.error.OpenAIError	flag_only_surface	18	4
openai.api_type	flag_only_surface	15	8
openai.ChatCompletion.create	unsupported_kwarg	14	7
openai.api_version	flag_only_surface	13	8
openai.ChatCompletion.create	legacy_version_unsupported	11	8
openai.Completion.create	response_shape_changed	8	3
openai.api_base	conditional_binding	8	1
openai.Completion.create	unsupported_kwarg	7	1
openai.error.InvalidRequestError	flag_only_surface	7	4

py-pdf/pypdf2-to-pypdf

obelize 0.7.0, pack sha256 34db4a1315d5, frame drawn 2026-10-13T13:17:31Z for PyPDF2 (157,888 files in the index): 80 repositories planned, 0 errored.

Outcome	Repositories	Share of planned	Share of those the pack ran on
blocked: the pack stops before it plans, for the reason below	36	45.0%	--
none: no usage found	2	2.5%	4.5%
held: usages found, no file written	5	6.2%	11.4%
partial: a file written, a row withheld	5	6.2%	11.4%
complete: a file written, nothing withheld	32	40.0%	72.7%

Why the pack stopped	A manifest declares it	Repositories
legacy_version_unsupported	no	20
legacy_version_unsupported	yes	11
runtime_unsupported	yes	3
new_version_unsupported	yes	2

Why a row was withheld	Rows	Repositories
flag_only_surface	73	17
legacy_version_unsupported	49	16
file_not_fully_migrated	38	21	a consequence of another hold
repo_not_fully_migrated	24	23	a consequence of another hold
conditional_binding	11	3
local_import	8	6
module_alias_rebound	8	2
alias_collision	4	1
runtime_unsupported	4	1
usage_unmapped	4	1
from_import_unmigrated_symbol	3	1
manifest_code_mismatch	2	2
new_version_unsupported	2	1
manifest_pin_shape_unsupported	1	1
roundtrip_mismatch	1	1

The 10 symbols that hold back most rows	Bail	Rows	Repositories
PyPDF2.PdfFileReader	flag_only_surface	44	16
PyPDF2.PdfFileWriter	flag_only_surface	25	10
PyPDF2	legacy_version_unsupported	22	16
PyPDF2.PdfReader	legacy_version_unsupported	19	15
PyPDF2	local_import	8	6
PyPDF2	conditional_binding	7	3
PyPDF2	module_alias_rebound	4	2
PyPDF2.PdfReader	conditional_binding	4	3
PyPDF2._page.PageObject.create_blank_page	usage_unmapped	4	1
PyPDF2	runtime_unsupported	3	1

The version gate comes before any rule. The openai pack plans nothing for a repository that uses the library and declares it nowhere, or declares it with a pin under the pack's floor (>=0.27.0) or in a form obelize cannot take. A declaration with no lower bound is taken as a release in >=0.27.0,<1 where the code calls openai.ChatCompletion or openai.Audio, and stops otherwise. That stops 27 of the 80 openai repositories, a third where it stopped about half under 0.5.0: 22 declare the library nowhere obelize reads, 4 declare it at a version or in a form the pack cannot take, and 1 stops on the Python it declares. The PyPDF2 pack's floor is >=3, but a declaration that names no version, or a range below 3 that admits 3.0.1, its newest release, is taken as that release unless the repository shows another one (ADR-056). It still stops 36 of its 80 repositories: 20 declare the library nowhere obelize reads, 11 declare it with a version under 3 or beside evidence of another release, 3 stop on the Python they declare and 2 on a pypdf pin that admits nothing the pack writes. The run says what to declare, so the stop is a message and not a silent miss. Many of the stopped repositories carry a rule's own bail as well, so the stop is an upper bound on what lifting it would write. The Gemini pack has no legacy-version gate and stops on a Python floor only, which it did for 1 of its 80.

Since the 0.5.0 records, the same 80 repositories per pack moved as follows. For openai, 9 went from blocked to complete, 6 from held to complete and 6 from blocked to held, and 1 went from held to blocked on a Python floor that is now read; complete rose from 12 to 27. For PyPDF2, 2 went from blocked to complete because a requirement.txt is now read, and 3 went to blocked (a pypdf pin that admits nothing the pack writes, twice, and a Python floor read from setup.py, once); complete rose from 31 to 32. For Gemini, 8 went from complete to held and 1 from partial to held, because a read of response.text inside a bare, Exception, BaseException or ValueError handler is now refused as response_shape_changed, which is the price of failing closed, and 1 went from held to blocked on a Python floor read from a Pipfile or setup.py; complete fell from 35 to 27. Every one of these is a plan that wrote something or did not, not a measure of whether it was right.

Beyond the gate, the symbols hold back differently by pack. For openai they are the exception classes and settings the pack flags (openai.error.*, openai.api_type, openai.api_version), keyword arguments it does not carry across, and result reads whose shape changed; for PyPDF2 the camelCase classes (PdfFileReader, PdfFileWriter) the pack flags rather than renames; for Gemini, client placement, a model name assigned more than once, configure called more than once and a read of response.text that a handler could catch. In the openai pack one withheld row holds the whole repository, because openai is one distribution (ADR-053); in the other two it holds its file. A repository held only by a notebook that calls the legacy API, which the pack reads and never writes, has no cause row, so it adds nothing to the symbol table.

This is not an accuracy figure. There is no answer key, so complete means a plan wrote something, not that what it wrote is right; the Gemini rounds below measure that. The openai frame is files that call openai.ChatCompletion, so a repository that only embeds is missing from it. The Gemini frame is the Gate 1 sample, drawn on 2026-09-18, and it holds some of the round-1 cases the Gemini rules were written against, so its row is a comparator and not a holdout.

Round 1

The harness design runs in tiers. Running the round found four defects in it, all in bench/ and none in src/obelize/; each, its fix, the discarded first execution and what a published result may carry are written up separately.

Round 1, from bench/cases.yaml and bench/results/round-1/.

Round	Cases (n)	dev / holdout	Unique patterns	Controls	Results on disk
1	20	13 / 7	16	3	23, obelize 0.1.0, pack fbe82d40afea4894...

What can reach the headline, computed from the keys before the round runs. verified_success needs a baseline pass, an after pass, tests that cover the change and no hunk a human had to write, so every row the key names has to be one a rule can make. A case with a manual row is capped below the headline whatever the scanner does, and so is a case whose key names a line the pack flags rather than edits.

Case	Split	Can reach verified_success	What caps it
AmineChr54__CleanVision-Futury_AI-Hackathon	holdout	no	no test command
Arnav3241__Jarvis-v13	dev	no	no test command
LucasHJin__vit	dev	no	a row the key marks manual
RuntimeAI__silly-merchants-for-safe-hackathon	dev	no	no test command
dongxnb__gupiao	dev	no	no test command
grapeot__devin.cursorrules	holdout	no	a row the key marks manual
humanbound__humanbound-firewall	dev	no	a row the key marks manual
navamai__navamai	holdout	no	a row the pack flags rather than edits
ImprintLab__Medical-Graph-RAG	holdout	no	no test command
LotusSerene__Gemini-Novel-Maker	dev	no	no test command
Omadithya07__Phidata-Multimodal-Video-Summarizer-Using-AI-Agents	dev	no	no test command
Sharukesh3__LLM-for-hydrogen-storage	dev	no	no test command
LJX2017__A-Song-of-Ice-and-Fire	holdout	no	no test command
Momen2410__Question-Answering-ChatBot-Using-Gemini1.5	dev	no	no test command
Satheesh2908__AI-Assistant-for-Satheesh	dev	no	no test command
Zierax__HackerHelper	dev	no	no test command
kvcops__Virtual-Herbal-Garden	holdout	no	no test command
taitran501__dashboard_app	holdout	no	no test command
000wan__ara-ai-bot	dev	no	no test command
ashishkumar5de__PDF-QL	dev	no	no test command
Total		0 of 20	3 a row the key marks manual; 1 a row the pack flags rather than edits; 16 no test command

The rate. Only verified_success counts toward it, and it is reported over cases and again over unique pattern_key values, each with its own n.

Scope	n	verified_success	Rate
Cases	20	0	0.0%
Cases (dev)	13	0	0.0%
Cases (holdout)	7	0	0.0%
Unique patterns	16	0	0.0%
Unique patterns (dev)	10	0	0.0%
Unique patterns (holdout)	7	0	0.0%

Outcome tiers. Computed from the result files, never assigned by hand.

Tier	dev	holdout	Total
verified_success	0	0	0
patched_unverified	1	0	1
partial	3	2	5
unsupported	9	5	14
wrong	0	0	0
error	0	0	0

Every case in the round, including the wrong, the errored and the unsupported.

Case id	Split	Tier	Detected	FP	Missed	Unmigrated	Human edits	Runtime (s)
AmineChr54__CleanVision-Futury_AI-Hackathon	holdout	unsupported	11	0	3	14	--	1.795
Arnav3241__Jarvis-v13	dev	partial	32	0	3	26	--	2.514
LucasHJin__vit	dev	unsupported	13	0	0	14	--	1.63
RuntimeAI__silly-merchants-for-safe-hackathon	dev	unsupported	5	0	0	6	--	0.996
dongxnb__gupiao	dev	unsupported	27	0	0	27	--	2.181
grapeot__devin.cursorrules	holdout	partial	5	0	4	9	--	1.833
humanbound__humanbound-firewall	dev	unsupported	2	0	8	18	--	1.271
navamai__navamai	holdout	unsupported	5	0	0	9	--	1.242
ImprintLab__Medical-Graph-RAG	holdout	unsupported	23	1	0	22	--	1.586
LotusSerene__Gemini-Novel-Maker	dev	partial	19	0	0	17	--	1.863
Omadithya07__Phidata-Multimodal-Video-Summarizer-Using-AI-Agents	dev	unsupported	5	0	0	6	--	0.86
Sharukesh3__LLM-for-hydrogen-storage	dev	unsupported	14	0	0	15	--	0.981
LJX2017__A-Song-of-Ice-and-Fire	holdout	partial	19	2	4	14	--	1.468
Momen2410__Question-Answering-ChatBot-Using-Gemini1.5	dev	unsupported	5	0	0	6	--	0.823
Satheesh2908__AI-Assistant-for-Satheesh	dev	partial	7	0	0	7	--	0.94
Zierax__HackerHelper	dev	unsupported	3	0	2	6	--	1.154
kvcops__Virtual-Herbal-Garden	holdout	unsupported	20	0	0	20	--	2.571
taitran501__dashboard_app	holdout	unsupported	6	0	0	7	--	2.196
000wan__ara-ai-bot	dev	unsupported	8	0	0	9	--	1.441
ashishkumar5de__PDF-QL	dev	patched_unverified	2	0	0	0	--	1.562
Total			231	3	24	252		30.9

What the suites did. A pass is not a suite that ran: the skip column is the difference, and tests_cover_change is still read from the key rather than from the run.

Case id	Baseline	pass / fail / skip	After	pass / fail / skip
LucasHJin__vit	pass	100 / 0 / 0	pass	100 / 0 / 0
grapeot__devin.cursorrules	pass	2 / 0 / 19	pass	2 / 0 / 19
humanbound__humanbound-firewall	pass	14 / 0 / 0	pass	14 / 0 / 0
navamai__navamai	pass	7 / 0 / 0	pass	7 / 0 / 0

Negative controls. They feed the false-positive table and are excluded from every rate above; an applied edit on one is a false positive by construction.

Case id	Must produce	Tier	Findings	False-positive edits
ParamveerSinghS__alex-ai-assistant	no call site and no pin	unsupported	0	0
GoogleCloudPlatform__Open_Data_QnA	no call site and no pin	unsupported	0	0
pfnet-research__japanese-lm-fin-harness	no call site and no pin	unsupported	0	0

Round 1's after-runs ran with google-generativeai still installed beside google-genai (protocol step 7), so a file the migration missed still imported and the after-run could pass over it. The after column of the suite table above stands as measured, and round 1 has not been run again. It also recorded neither the installed packages nor the version of uv, which later rounds do, and its records name the initial public commit.

Every case is capped for the reason the ceiling gave before the round ran. navamai/navamai's cap is three mock.patch targets naming the legacy module as a string, which the pack reports as flag_only_surface and never edits.

Nothing was wrong and nothing errored, controls included, and every version assertion held. unsupported is fail-closed working: obelize reported what it found and wrote nothing.

The scan agreement reproduced exactly: the round's fresh fetch and scan gave the counts of the separately collected table in bench/CASES.md.

The three false positives are two shapes. Two are genai.GenerativeModel('gemini-pro').start_chat(history=[]) in LJX2017/A-Song-of-Ice-and-Fire: the key names a call and a method_call, the scanner reports one call named ...GenerativeModel.start_chat, so each line is one false positive and two misses (COVERAGE.md gap 25). The third is self.client.count_tokens(...) in a vendored camel/ module that imports the legacy SDK elsewhere: an unresolved receiver whose method name the pack names, reported by design.

One suite passes without running the covering test. grapeot/devin.cursorrules skips the whole TestLLMAPI class, test_create_gemini_client included, because it needs a local LLM server; its key still says tests_cover_change: true. Checking that against the run needs the key to name the covering tests: COVERAGE.md gap 27.

Gate 4 (PRODUCT.md) reads this rate, and it lands in the lowest branch. The written analysis that branch asks for is the error analysis; the decision it asks for is not made here. The one complete migration has no test command, and each case with a suite is capped by something its key named in advance.

Error analysis

Protocol: BENCHMARK.md; the reasoning behind it is written up separately.

Round 1, from bench/cases.yaml, bench/results/errors/round-1.json and bench/errors.yaml.

Where the rows went. Every row every answer key names, once. derived is one of the two atomicity codes and is a consequence of a row in the same file or repository, never a fact about itself; missed is a row the scan delivered nothing at, so it carries no bail code and its cause is hand-written.

What happened to the row	Rows	Share
migrated	25	9.0%
withheld	129	46.6%
derived	98	35.4%
missed	25	9.0%
Total	277

Every cause, and what happens about it. The lever is the number this table exists for: how many rows would migrate if this cause alone were fixed and nothing else were, under the same two atomicity rules the round ran with. Levers do not add up -- two causes in one file each buy nothing on their own -- so the column has no total.

Cause	Class	Rows	Lever	Disposition	Recorded
file_not_fully_migrated	derived	85	+0	limitation	ADR-010
multiple_configure_calls	withheld	19	+0	limitation	ADR-020
local_import	withheld	18	+26	rule_change	COVERAGE 28
module_alias_rebound	withheld	17	+18	limitation	ADR-019
repo_not_fully_migrated	derived	13	+0	limitation	ADR-010
module_returned_from_function	missed	12	+0	limitation	ADR-019
flag_only_surface	withheld	11	+11	limitation	ADR-030
generation_config_not_static	withheld	11	+26	rule_change	COVERAGE 29
model_object_read_elsewhere	withheld	9	+13	limitation	ADR-006
client_placement_ambiguous	withheld	8	+0	limitation	ADR-026
receiver_unresolved	withheld	8	+12	limitation	ADR-019
history_parts_shape_incompatible	withheld	6	+10	limitation	ADR-028
model_object_escapes	withheld	5	+0	limitation	ADR-027
attribute_removed	withheld	4	+0	limitation	ADR-030
multiple_assignments	withheld	4	+9	rule_change	COVERAGE 30
client_name_collision	withheld	3	+5	limitation	ADR-026
star_import_binding	missed	3	+0	limitation	ADR-019
chained_constructor_one_row	missed	2	+0	scan_change	COVERAGE 25
client_source_unresolved	withheld	2	+0	limitation	ADR-020
module_from_subscript	missed	2	+0	limitation	ADR-019
prefilter_token_absent	missed	2	+0	limitation	ADR-022
alias_collision	withheld	1	+0	rule_change	COVERAGE 31
attribute_read_via_getattr	missed	1	+0	scan_change	COVERAGE 22
configure_consumed_elsewhere	withheld	1	+4	limitation	ADR-031
from_import_unmigrated_symbol	withheld	1	+0	limitation	ADR-025
manifest_filename_case	missed	1	+0	scan_change	COVERAGE 33
manifest_filename_unknown	missed	1	+0	limitation	ADR-021
text_mention_not_flagged	missed	1	+0	scan_change	COVERAGE 32
usage_unmapped	withheld	1	+0	limitation	ADR-031
Total		252	--

The three dispositions, over the causes above.

Disposition	Causes	Rows
rule_change	4	34
scan_change	4	5
limitation	21	213

Per case. Reachable is what would migrate with every bail this round raised lifted at once, which is an upper bound and not a plan; out of reach is the rest, and no rule change reaches it because no pass reported the row.

Case id	Split	Key rows	Migrated	Reachable	Out of reach
AmineChr54__CleanVision-Futury_AI-Hackathon	holdout	14	0	11	3
Arnav3241__Jarvis-v13	dev	36	10	33	3
LucasHJin__vit	dev	14	0	14	0
RuntimeAI__silly-merchants-for-safe-hackathon	dev	6	0	6	0
dongxnb__gupiao	dev	27	0	27	0
grapeot__devin.cursorrules	holdout	10	1	6	4
humanbound__humanbound-firewall	dev	18	0	9	9
navamai__navamai	holdout	9	0	9	0
ImprintLab__Medical-Graph-RAG	holdout	22	0	22	0
LotusSerene__Gemini-Novel-Maker	dev	20	3	20	0
Omadithya07__Phidata-Multimodal-Video-Summarizer-Using-AI-Agents	dev	6	0	6	0
Sharukesh3__LLM-for-hydrogen-storage	dev	15	0	14	1
LJX2017__A-Song-of-Ice-and-Fire	holdout	21	7	19	2
Momen2410__Question-Answering-ChatBot-Using-Gemini1.5	dev	6	0	6	0
Satheesh2908__AI-Assistant-for-Satheesh	dev	8	1	7	1
Zierax__HackerHelper	dev	6	0	4	2
kvcops__Virtual-Herbal-Garden	holdout	20	0	20	0
taitran501__dashboard_app	holdout	7	0	7	0
000wan__ara-ai-bot	dev	9	0	9	0
ashishkumar5de__PDF-QL	dev	3	3	3	0
Total		277	25	252	25

What the scan reported that no key claims. A false positive that is withheld is not a false-positive edit, and none of these was applied.

Case id	Where	Symbol	Cause	Disposition	Recorded
ImprintLab__Medical-Graph-RAG	camel/utils/token_counting.py:422	google.generativeai.GenerativeModel.count_tokens	method_name_collision	limitation	ADR-019
LJX2017__A-Song-of-Ice-and-Fire	basic_game.py:27	google.generativeai.GenerativeModel.start_chat	chained_constructor_one_row	scan_change	COVERAGE 25
LJX2017__A-Song-of-Ice-and-Fire	reconstruct_timeline.py:10	google.generativeai.GenerativeModel.start_chat	chained_constructor_one_row	scan_change	COVERAGE 25

The two derived codes are consequences, not causes. file_not_fully_migrated is a ready row in a file that was not; repo_not_fully_migrated is a legacy pin kept until the code importing it is gone. Both are atomicity working as the fixture rules say it should.

generation_config_not_static ties for the largest lever, against a written prediction. Every instance is a name assigned once and readable: a module-level genai.GenerationConfig(...), a module-level dict literal, a local built two statements above the call. Tracing this bail through the three-repository development corpus found it bought no edit; that corpus under-represented the commonest way to write a generation config.

multiple_configure_calls is worth nothing on its own, because every file it touches also carries local_import. Ranked by count it would come second.

One alias_collision has a lever of zero, because the file it sits in also carries three client_placement_ambiguous rows. Omadithya07/Phidata-Multimodal-Video-Summarizer-Using-AI-Agents imports from google.generativeai import upload_file,get_file on line 5 and import google.generativeai as genai on line 6: rewriting the first asks for the alias the second already binds.

One lever must not be taken. history_parts_shape_incompatible is two lines, in two near-identical copies of one file, reading model.start_chat(history=str(time.time())): a float rendered as a string, which is not a history in either SDK. A rule that rewrote it would migrate a defect, which is why the lever column carries a disposition, not a ranking.

Three causes are one wall. module_returned_from_function, star_import_binding and module_from_subscript are the legacy module bound to a name through a function return, a star import in another file or a container subscript, and the resolver follows none of the three. That, not any rewrite rule, is where the round's recall went.

Lifting every bail would not move the headline. It would complete twelve of the twenty repositories, but ten of those have no test command, one has a manual row and one a line the pack flags, so all twelve stay capped below verified_success.

One key row was wrong, and correcting it moved no published number; see bench/CASES.md.

Comparison arm

Protocol: BENCHMARK.md; the reasoning behind it is written up separately. Obelize's cleared rows under the shared scan equal what the round published from its plan and withheld list, for every case; tests/unit/test_bench_comparison.py asserts it, and that agreement is what licenses scoring the agent, which publishes no plan, the same way.

The agent's side is a snapshot from 2026-09-22 and was not rerun for 0.1.0; its guide and the model behind it may have moved since. Obelize's side is current: Satheesh2908/AI-Assistant-for-Satheesh's count was corrected by hand from five cleared rows to one, because a fix landed on 2026-09-24 teaching the rule not to remove a configure() another file still runs on, and four of the five rows this arm once credited relied on exactly that removal.

Round 1, development split, from bench/cases.yaml, bench/results/comparison/round-1.json, bench/results/round-1/ and bench/comparison.yaml. The other arm is a general-purpose coding agent given the migration guide -- snapshotted 2026-09-22T07:20:07Z, sha256 223b9cd7aabf -- and one fixed prompt, and nothing else.

Both arms, the same cases. Rows migrated is a key row whose legacy usage a scan of the tree afterwards no longer reports; corrections left is every key row still legacy plus every edit the review rejected.

Arm	Cases	verified_success	Key rows	Rows migrated	Corrections left	Runtime
Obelize	13	0	174	17 (9.8%)	157	18.2s
General agent	13	1	174	154 (88.5%)	22	14.3m

The tiers. The same ladder for both arms, which is the round's own minus the two rungs that read a findings document -- an agent publishes none. Over this split the ladder returns the tier the round published for every obelize case.

Tier	Obelize	General agent
verified_success	0	1
patched_unverified	1	7
partial	3	3
unsupported	9	0
wrong	0	2
error	0	0

Per case. Seen is the key rows the scanner reports on the untouched tree; the rows it does not see are credited to neither arm and are in left for both. Off key is files an arm changed that the key names no row in, which is a pointer for the review and not a verdict.

Case id	Key rows	Seen	Arm	Cleared	Left	Corrections	Files	Off key	Runtime	Tier
Arnav3241__Jarvis-v13	36	33	Obelize	10	26	26	3	0	2.5s	partial
			General agent	28	8	9	9	0	1.9m	wrong
LucasHJin__vit	14	14	Obelize	0	14	14	0	0	1.6s	unsupported
			General agent	14	0	0	4	2	1.4m	verified_success
RuntimeAI__silly-merchants-for-safe-hackathon	6	6	Obelize	0	6	6	0	0	1.0s	unsupported
			General agent	6	0	0	2	0	51.0s	patched_unverified
dongxnb__gupiao	27	27	Obelize	0	27	27	0	0	2.2s	unsupported
			General agent	27	0	0	5	0	2.4m	patched_unverified
humanbound__humanbound-firewall	18	10	Obelize	0	18	18	0	0	1.3s	unsupported
			General agent	10	8	9	5	1	2.0m	wrong
LotusSerene__Gemini-Novel-Maker	20	20	Obelize	3	17	17	3	0	1.9s	partial
			General agent	20	0	0	5	0	1.1m	patched_unverified
Omadithya07__Phidata-Multimodal-Video-Summarizer-Using-AI-Agents	6	6	Obelize	0	6	6	0	0	0.9s	unsupported
			General agent	6	0	0	3	1	31.0s	patched_unverified
Sharukesh3__LLM-for-hydrogen-storage	15	14	Obelize	0	15	15	0	0	1.0s	unsupported
			General agent	14	1	1	3	0	1.2m	partial
Momen2410__Question-Answering-ChatBot-Using-Gemini1.5	6	6	Obelize	0	6	6	0	0	0.8s	unsupported
			General agent	6	0	0	2	0	33.0s	patched_unverified
Satheesh2908__AI-Assistant-for-Satheesh	8	7	Obelize	1	7	7	1	0	0.9s	partial
			General agent	7	1	1	5	1	28.0s	partial
Zierax__HackerHelper	6	4	Obelize	0	6	6	0	0	1.2s	unsupported
			General agent	4	2	2	2	0	21.0s	partial
000wan__ara-ai-bot	9	9	Obelize	0	9	9	0	0	1.4s	unsupported
			General agent	9	0	0	2	0	30.0s	patched_unverified
ashishkumar5de__PDF-QL	3	3	Obelize	3	0	0	2	0	1.6s	patched_unverified
			General agent	3	0	0	3	1	1.2m	patched_unverified
Total	174	159	Obelize	17	157	157		0	18.2s
Total			General agent	154	20	22		6	14.3m

Where an arm went outside the key, and what the review made of it.

Case id	Arm	Files off the key	New legacy usage	Edits the review rejected
Arnav3241__Jarvis-v13	General agent	--	0	1
LucasHJin__vit	General agent	CLAUDE.md, docs/AI_MERGE_DETAILS.md	0	0
humanbound__humanbound-firewall	General agent	CHANGELOG.md	0	1
Omadithya07__Phidata-Multimodal-Video-Summarizer-Using-AI-Agents	General agent	README.md	0	0
Satheesh2908__AI-Assistant-for-Satheesh	General agent	README.md	0	0
ashishkumar5de__PDF-QL	General agent	README.md	0	0

Run it twice. Obelize's answer is byte-identical output, asserted since Phase 1. This is the other arm's, measured over the same starting bytes.

Case id	Files touched	Same files	Same line counts	Cleared, run 1 / run 2	Runtime, run 1 / run 2
Arnav3241__Jarvis-v13	9	yes	no	28 / 28	1.9m / 2.9m
LotusSerene__Gemini-Novel-Maker	5	yes	no	20 / 20	1.1m / 57.0s
ashishkumar5de__PDF-QL	3	yes	no	3 / 3	1.2m / 1.0m

The agent cleared about nine times as many rows, about forty-five times slower.

Both of its wrong cases are real defects, and a suite caught one. In humanbound/humanbound-firewall, whose suite covers the change, _import_genai() exists to turn a missing optional dependency into an ImportError naming the [gemini] extra. from google import genai does not raise there, because google is a namespace package, so the user gets AttributeError: 'NoneType' object has no attribute 'Client' and a test fails, although the agent also rewrote both test modules, including the lazy-import guard's FORBIDDEN_MODULES. In Arnav3241/Jarvis-v13 it took the lever the error analysis above says must not be taken, turning start_chat(history=str(time.time())) into client.chats.create(history=[types.Content(role='user', parts=[types.Part(text=str(time.time()))])]): a user turn invented from a Unix timestamp.

The agent's account of its work was wrong on the largest case. Its report for Arnav3241/Jarvis-v13 says a repository-wide grep finds no legacy reference. Tests/r1.py has five, which neither run touched, and they are the whole gap between the rows the scanner sees there and the rows the agent cleared. Only the answer key shows it.

Comparing the arms exposed a scanner defect. Round 1 matches rows on (path, line, symbol), so key and scanner never had to agree on kind. In humanbound/humanbound-firewall the key grades six sys.modules stubs dynamic and the scanner reports them as text_mention, which is not_a_usage: a repository whose tests mock the legacy module has those lines left behind silently, and tests_touched_by_migration never fires. The agent had to rewrite all six. COVERAGE.md gap 34.

The agent is not repeatable. Its re-runs touched the same files and cleared the same rows with different line counts, and on Arnav3241/Jarvis-v13 the first run turned a line into a fabricated turn and the second deleted it, which keeps behaviour: same input, same score, opposite judgement about a defect.

The ceiling caps a rule, not a tool. On LucasHJin/vit the agent migrated all eight manual rows and the project's hundred tests pass, so it reached verified_success on a case the ceiling closes to a rule. It did so with an option the key did not list: an adapter class that is a model object, leaving the six callers and the seven tests that patch that seam untouched. The review accepted it; whether the abstraction is worth carrying is the maintainer's call. So the two cases where a suite could speak produced one verified success and one regression.

Obelize is ahead on what a reviewer must read, not on accuracy: no rejected edit, no file off the key, no new legacy usage, byte-identical output, and a run.json naming every refused row and why. The agent's off-key edits to READMEs, a changelog and project documentation were not asked for and are recorded nowhere a reviewer would see.

Limits. One agent, one day, one guide snapshot, and only the split obelize's rules were tuned against. The agent ran up to eight cases concurrently on one machine, so its runtime is an order of magnitude, not a benchmark. Every free choice favoured the agent.

Retired cases

None. A case that leaves the active set is listed here with its reason.

Case id	Retired in round	Reason