Applied research · production system
Margin is a deployed LLM system that spends real money and holds a database of record. I use it as an instrument — and the constraints are the point, not the obstacle.
Margin is a self-hosted GRE and IELTS writing coach — an LLM proxy and a system of record. Every model call is billed to a live API key. The database is the only copy of the study history. Registration is open, credentials are real, and there is no sandbox to catch a mistake.
Those stakes change what counts as a finding. A bug in the billing path costs money. A bug in the scoring path is worse: it sends someone into an exam prepared against a false signal. You cannot answer either with an offline benchmark, because the failure modes only exist once the thing is running, taking real input, and spending something.
So the question I actually work on is: how do you get research-grade answers out of a system you are also responsible for keeping correct? That is the intersection I want to build a career in — not measurement in a clean room, and not shipping without measurement.
Every guard in this codebase exists because a model violated a rule the prompt had explicitly stated. That is an empirical claim, not a design preference — each guard was written in response to an observed failure, one at a time, and the pattern only became visible once there were enough of them to count.
The operating rule that emerged: a constraint the output must satisfy is enforced in code, and may additionally be stated in the prompt — never only there. Model classifications are validated against closed taxonomies at the boundary. An unrecognised label is discarded, not displayed. A model that invents a category produces no label at all, rather than a confident wrong one.
Measured across two runs against a live provider:
| Run | Category values returned | Valid |
|---|---|---|
| Essay | agreement ×3 | 3 / 3 |
| Proofread | agreement, articleDeterminer, danglingModifier, wordChoice ×2 | 5 / 5 |
Zero invented labels
Full compliance — the validator was never needed as a backstop. Which is exactly why it stays: compliance observed twice is not compliance guaranteed, and the failure it prevents is a learner being taught a category that does not exist.
Transfers toAny team shipping an LLM feature where the model's output drives a downstream decision. The question is not whether the prompt asks for valid output — it is what your system does on the day it gets invalid output anyway.
The examiner returns a 27-field assessment — per-criterion scores with comments, every error with a quote and a fix, a paragraph rewrite, a verdict. Its prompt capped the whole response at 1,250 tokens, roughly 900 words. I formed a hypothesis: that ceiling is truncating the explanations, and the per-criterion comments are thin because the model runs out of room.
I raised the cap 3.6× — 1,250 to 4,500 tokens — then ran a controlled essay through a live provider and measured what came back.
Hypothesis not supported
The response came back below the original ceiling. The cap had never been the binding constraint. What actually limited the explanations was the instruction's own opening words — "Be terse" — not the token allowance. I was one step from attributing an effect to the wrong cause, shipping a fix for it, and believing the problem solved.
The change stayed in on separate grounds: the ceiling was real, and a longer essay with more errors would meet it. But the negative result is written into the project's changelog rather than quietly dropped, because the next person wondering why explanations are short needs to know the budget has already been ruled out.
Transfers toCost and latency work on LLM products, where the intuitive lever is usually the wrong one. Raising a limit is easy to ship and easy to believe in. Measuring whether the limit was ever reached takes one experiment and prevents a class of confident, expensive mistakes.
The system contains 254 teaching entries across 11 modules — definitions, the grammatical mechanism behind each concept, a worked example, and the neighbouring concept it gets confused with. None is generated at runtime. Every one is hand-authored and ships in the bundle.
That was a deliberate build-versus-generate call, and the properties it buys are ones no sampling temperature provides: the content cannot hallucinate, it costs nothing per view, it is reviewable in version control like any other change, and it is invariant across runs — two learners meeting the same concept read the same account of it.
The generated alternative would have been faster to build and worse in every dimension that matters here. For explanatory content, reproducibility is not a refinement: a definition that varies per request is not a definition.
Transfers toProduct decisions about where an LLM belongs in the stack. Generation is a poor fit for content that must be stable, auditable and free at read time — which is more of a product surface than the default reach for it suggests.
These are engineering decisions, but they are the reason the measurements above are trustworthy at all — an instrument you cannot account for produces numbers you cannot use.
Every debit, refund and grant is a new row; a balance is never corrected in place. Any balance is reconstructable from history, which is what makes a spend anomaly diagnosable rather than merely visible.
A provider returning HTTP 200 with nothing usable is escalated to a retry, never returned. That guard exists because users were once charged for empty completions — the exact class of bug that is invisible in testing and obvious on a bill.
Every call runs inside reserve → finalize or refund. A failed or empty completion refunds rather than bills. Any exception path between those points must release the reservation, so a crash cannot strand a charge.
Every external call is intercepted by construction. The suite cannot touch a live provider, a production email account with no sandbox allowlist, or the study database. That is a safety property rather than a convenience: a test that spends money or sends mail on every run is a liability, not an assurance.
The decision log is the artifact I would defend hardest, and it is the most academic habit in an otherwise industrial project. Each entry records the choice, the reasoning, and the alternatives rejected and why — because the most expensive failure in a long-running system is rediscovering an option that was already ruled out for a reason nobody wrote down.
Stating the limits is not modesty. An instrument whose error bars are unstated is not an instrument, and a portfolio that only reports what worked is reporting a filtered sample.
I build measurement systems even for my own writing.
The habit this project demonstrates is narrower and more useful than the tool: forming a hypothesis about a system I am responsible for, measuring instead of assuming, and keeping the result when it contradicted me.
I am looking for doctoral work where the object of study is a system that actually runs — evaluation and reliability of deployed language systems, where the interesting constraints are real users, real cost, and real consequences for being wrong. Industry has the systems and the stakes; academia has the methodological seriousness. I would rather not choose between them, and this project is my argument that you do not have to.