Applied research · production system

Research rigour is cheap in a notebook and expensive in production

Margin is a deployed LLM system that spends real money and holds a database of record. I use it as an instrument — and the constraints are the point, not the obstacle.

SettingWhy this system

The measurement problem is the engineering problem

Margin is a self-hosted GRE and IELTS writing coach — an LLM proxy and a system of record. Every model call is billed to a live API key. The database is the only copy of the study history. Registration is open, credentials are real, and there is no sandbox to catch a mistake.

Those stakes change what counts as a finding. A bug in the billing path costs money. A bug in the scoring path is worse: it sends someone into an exam prepared against a false signal. You cannot answer either with an offline benchmark, because the failure modes only exist once the thing is running, taking real input, and spending something.

So the question I actually work on is: how do you get research-grade answers out of a system you are also responsible for keeping correct? That is the intersection I want to build a career in — not measurement in a clean room, and not shipping without measurement.

Finding 01Reliability

Prompt instructions are not a control surface

Every guard in this codebase exists because a model violated a rule the prompt had explicitly stated. That is an empirical claim, not a design preference — each guard was written in response to an observed failure, one at a time, and the pattern only became visible once there were enough of them to count.

The operating rule that emerged: a constraint the output must satisfy is enforced in code, and may additionally be stated in the prompt — never only there. Model classifications are validated against closed taxonomies at the boundary. An unrecognised label is discarded, not displayed. A model that invents a category produces no label at all, rather than a confident wrong one.

Measured across two runs against a live provider:

RunCategory values returnedValid
Essayagreement ×33 / 3
Proofreadagreement, articleDeterminer, danglingModifier, wordChoice ×25 / 5

Zero invented labels

Full compliance — the validator was never needed as a backstop. Which is exactly why it stays: compliance observed twice is not compliance guaranteed, and the failure it prevents is a learner being taught a category that does not exist.

Transfers toAny team shipping an LLM feature where the model's output drives a downstream decision. The question is not whether the prompt asks for valid output — it is what your system does on the day it gets invalid output anyway.

Finding 02Falsified

The result I kept because it contradicted me

The examiner returns a 27-field assessment — per-criterion scores with comments, every error with a quote and a fix, a paragraph rewrite, a verdict. Its prompt capped the whole response at 1,250 tokens, roughly 900 words. I formed a hypothesis: that ceiling is truncating the explanations, and the per-criterion comments are thin because the model runs out of room.

I raised the cap 3.6× — 1,250 to 4,500 tokens — then ran a controlled essay through a live provider and measured what came back.

1,128 measured old ceiling 1,250 new ceiling 4,500 0 4,500 tokens Output never reached the ceiling it was assumed to be hitting.
One graded essay · gpt-5-mini · 4,512 characters returned ≈ 1,128 tokens · per-criterion comments 96, 112 and 66 characters

Hypothesis not supported

The response came back below the original ceiling. The cap had never been the binding constraint. What actually limited the explanations was the instruction's own opening words — "Be terse" — not the token allowance. I was one step from attributing an effect to the wrong cause, shipping a fix for it, and believing the problem solved.

The change stayed in on separate grounds: the ceiling was real, and a longer essay with more errors would meet it. But the negative result is written into the project's changelog rather than quietly dropped, because the next person wondering why explanations are short needs to know the budget has already been ruled out.

Transfers toCost and latency work on LLM products, where the intuitive lever is usually the wrong one. Raising a limit is easy to ship and easy to believe in. Measuring whether the limit was ever reached takes one experiment and prevents a class of confident, expensive mistakes.

Finding 03Tradeoff

Where generation is the wrong tool

The system contains 254 teaching entries across 11 modules — definitions, the grammatical mechanism behind each concept, a worked example, and the neighbouring concept it gets confused with. None is generated at runtime. Every one is hand-authored and ships in the bundle.

That was a deliberate build-versus-generate call, and the properties it buys are ones no sampling temperature provides: the content cannot hallucinate, it costs nothing per view, it is reviewable in version control like any other change, and it is invariant across runs — two learners meeting the same concept read the same account of it.

The generated alternative would have been faster to build and worse in every dimension that matters here. For explanatory content, reproducibility is not a refinement: a definition that varies per request is not a definition.

Transfers toProduct decisions about where an LLM belongs in the stack. Generation is a poor fit for content that must be stable, auditable and free at read time — which is more of a product surface than the default reach for it suggests.

PracticeUnder real stakes

What responsibility for the system taught me

These are engineering decisions, but they are the reason the measurements above are trustworthy at all — an instrument you cannot account for produces numbers you cannot use.

Billing is append-only

Every debit, refund and grant is a new row; a balance is never corrected in place. Any balance is reconstructable from history, which is what makes a spend anomaly diagnosable rather than merely visible.

An empty success is a failure

A provider returning HTTP 200 with nothing usable is escalated to a retry, never returned. That guard exists because users were once charged for empty completions — the exact class of bug that is invisible in testing and obvious on a bill.

Provider calls are transactional

Every call runs inside reserve → finalize or refund. A failed or empty completion refunds rather than bills. Any exception path between those points must release the reservation, so a crash cannot strand a charge.

The test suite cannot reach production

Every external call is intercepted by construction. The suite cannot touch a live provider, a production email account with no sandbox allowlist, or the study database. That is a safety property rather than a convenience: a test that spends money or sends mail on every run is a liability, not an assurance.

RecordWhat exists

The paper trail

192Commits
75Logged decisions
with rejected alternatives
46Session changelogs
254Authored teaching entries
18Automated tests
where none existed

The decision log is the artifact I would defend hardest, and it is the most academic habit in an otherwise industrial project. Each entry records the choice, the reasoning, and the alternatives rejected and why — because the most expensive failure in a long-running system is rediscovering an option that was already ruled out for a reason nobody wrote down.

LimitsWhat this is not

What this does not claim

  • These are findings from building, not results from a controlled study. Sample sizes are one run each, on one model.
  • None of it is novel to the literature on LLM evaluation. The value is that the claims are instrumented and checkable, not that they are new.
  • The system measures against published rubrics; it does not validate those rubrics.
  • The falsified hypothesis was mine, tested on my own system — the weakest experimental design available, and the strongest possible incentive to have believed it anyway.

Stating the limits is not modesty. An instrument whose error bars are unstated is not an instrument, and a portfolio that only reports what worked is reporting a filtered sample.

DirectionWhat I want next

Where this is going

I build measurement systems even for my own writing.

The habit this project demonstrates is narrower and more useful than the tool: forming a hypothesis about a system I am responsible for, measuring instead of assuming, and keeping the result when it contradicted me.

I am looking for doctoral work where the object of study is a system that actually runs — evaluation and reliability of deployed language systems, where the interesting constraints are real users, real cost, and real consequences for being wrong. Industry has the systems and the stakes; academia has the methodological seriousness. I would rather not choose between them, and this project is my argument that you do not have to.