Notes from inside the work.
50 pieces on what actually goes wrong in data, analytics and AI engagements, and what we do about it. No numbers we cannot stand behind, and no piece that could have been written without doing the work.
50 pieces
Data engineering
Numbers that arrive late or wrong, sources you do not control changing under you, and the bill for keeping it all running.
6 min readThe same customer under four names breaks every number downstreamDuplicate parties arrive faster than anyone can clean them. What survives contact with a real business is deterministic keys, a review queue
6 min readSome of your sources will never have an API and the design has to say soTwo or three systems in every estate will only ever send you a file. Designing for that means recording arrivals as facts, because this breaks by going quiet.
6 min readYour warehouse bill doubled and nobody changed the data volumeWarehouse spend almost never grows because of storage. It grows because a schedule, a cache or a refresh strategy changed quietly, and nobody connected the two.
6 min readStarting a warehouse migration by translating stored procedures will stall itTranslating a procedure estate in list order is how a migration loses a year. Start from the reports people genuinely read and work backwards.
6 min readA supplier changed a column and your pipeline found out in productionData contracts assume a producer you can hold to account. When the producer is a vendor or a partner, you need something else entirely.
6 min readA data test nobody acts on is worse than no test at allMost test suites fail quietly, by being read less each week. The useful cut is not which tests are correct but which ones have a named owner and a first move.
7 min readA pipeline is only as good as its worst backfillDaily runs prove almost nothing. The real test is the morning you find six weeks of wrong data and have to rebuild history while everyone is still reading it.
6 min readReading change data out of Postgres without slowing the application downLogical replication looks free until a slot stops being consumed and the write-ahead log quietly fills the disk on your primary.
6 min readMost real-time requirements are a deadline in disguiseA business asking for real-time data is usually asking for a number that is right before the 11am call. Those are different problems with very different bills.
Analytics and BI
Metrics that disagree, dashboards nobody opens, and reporting arguments that turn out to be definition arguments.
6 min readAttribution disagreements are an argument about the model, not the dataMarketing and finance rarely disagree about the events. They disagree about the credit rule, and a rule is something somebody chooses and owns.
6 min readPutting dashboards in front of customers turns reporting into a productThe charts are the easy part. What changes when customers log in is isolation, load shape and a support queue, and none of those is a dashboard setting.
6 min readA BI migration is the only time you will be allowed to delete reportsThe report list from the old tool is a record of every request ever granted, and rebuilding it faithfully is how a migration costs a year and fixes nothing.
7 min readReporting through a reorganisation without rewriting last yearSales territories get redrawn and last year's numbers move. Both answers are correct, and the model has to say out loud which one it is giving you.
5 min readHow to find out which of your dashboards anyone actually readsEvery BI estate has dashboards nobody opens. The usage logs to prove it are already sitting in your tool, and the audit is a day's work, not a project.
6 min readSelf-serve analytics is a staffing decision, not a licence purchaseEvery BI rollout ends with a few people building for everyone else. The mistake is treating that as a training failure rather than a fact to design around.
6 min readChurn is three different measurements wearing one wordLogo, revenue and activity churn can move in opposite directions in the same quarter. If your deck shows one and the room is thinking of another
6 min readWhen the spreadsheet is the real system of record, integrate itMigrating the finance model into the warehouse usually stalls. Landing the file as a governed source is faster, with a clear line for when it stops being enough.
6 min readTwo dashboards, one metric, two different answersTwo teams, two revenue figures, one warehouse. The tool is almost never at fault, and fixing it is less a modelling problem than an argument nobody wants to have.
AI evaluation
How to tell whether an AI feature is actually good enough, and how to keep knowing after it changes.
6 min readAn agent has to be graded on the steps it took, not only the answerA correct answer can hide a wrong tool call, a guessed parameter and a lucky input. Grading the steps costs more, and it is the part that predicts next month.
6 min readA larger context window did not make retrieval optionalA bigger window changed what retrieval has to be good at and left the reasons for having it alone. Here is the comparison worth running on your own questions.
6 min readAn LLM judge is a measuring instrument and it needs calibratingModel-graded evaluation scales the way human review never will, which is exactly why nobody checks whether the grader is reading true.
7 min readYour evals pass and users still say the thing is wrongOffline eval sets encode what you imagined users would do. Here is how to route live traffic back into the frozen set so the suite gets harder over time.
6 min readWhen two reviewers grade the same answer differently, the spec is the problemGrader disagreement is not noise to be averaged away. It is the clearest evidence you have that nobody ever wrote down what a good answer looks like.
6 min readA prompt change is a deployment, so gate it like onePrompt edits ship straight to production on the strength of one manual check. Here is what putting an eval suite in CI looks like, and what it honestly costs.
6 min readWhen a RAG system answers confidently and wrongly, look at retrieval firstRepeated prompt revisions and a model upgrade will not fix a system that never put the right passage in front of the model in the first place.
7 min readYou cannot choose a model without a graded set of your own examplesPublic leaderboards say nothing about your abbreviations, your scanned documents or your edge cases. A hundred graded examples of your own will.
Applications and automation
The work that lives in spreadsheets, inboxes and one person's head, and what it takes to give it a system.
6 min readMost of the work you want to automate does not need a modelA model earns its place on three conditions. A great deal of what gets scoped as an AI project fails all three, and a form, a rule and a lookup would do it better.
6 min readWhen the input to your pipeline is a PDF someone emailedInvoices arrive as email attachments and someone retypes them. Doing that well is less about extraction accuracy than about the documents the model gets wrong.
6 min readEvery automation needs a name attached to it before it goes liveInternal tooling rarely fails loudly. It fails by succeeding at nothing, for a fortnight, while the person who wrote it works somewhere else.
6 min readAutomation stalls at the human step, so design for the person on leaveYour workflow runs in a second and then waits four days on one approval. The fix is not another tool, it is an age, a queue, a deputy and a stated default.
6 min readThe moment an app can change data, it owes you an audit trailThe first write-back button is the cheapest moment to get identity, permissions, an append-only log and a reversal path right. Every later moment costs more.
6 min readBuilding around an ERP you are not allowed to changeThe customisation is refused, the upgrade path must stay clean, and the work still has to happen. What survives is a satellite, and a record of who owns each field.
6 min readSome questions need a button, not another chartA report can tell you fourteen orders are stuck. It cannot let anyone unstick them, which is why the work quietly moves into a spreadsheet you never see.
5 min readYou cannot automate a process nobody has written downThe flowchart you were handed describes the good day. The two days a week the job actually takes are spent on everything the flowchart leaves out.
Antvia
Lakehouse and BI practice, from whether you need one at all to what open formats and query-level governance are really for.
6 min readUpdates into a lakehouse are where the table design showsIncremental loads rarely fail on the night they are written. They fail in month five, when partitioning, file sizes and missing compaction finally add up.
6 min readData that is not allowed to leave the country changes the architecturePicking a region satisfies the questionnaire. It does not answer where the backups, the logs and the vendor's support tooling actually sit.
6 min readAccess rules that live in the BI tool are not access rulesA row filter configured in a dashboard protects the dashboard. It stops protecting anything the moment a notebook
6 min readOpen table formats matter on the day you want to leaveThe storage format is invisible until a renewal, a migration or an acquisition makes it the whole conversation. Test your exit while you are still happy.
6 min readBronze, silver and gold are a naming convention, not a designThree schemas called bronze, silver and gold are not an architecture. A layer earns its place by holding a guarantee you can test, and some estates need two.
6 min readHow to tell whether you need a lakehouse or just a bigger databaseMost teams reach for a lakehouse after a dashboard times out. Here is the written test to run first, and the four reasons that genuinely justify the move.
Uloborus
Running a process plant on one set of records: batch traceability, compliance paperwork and what a shop floor will tolerate.
6 min readThe shop floor will not fill in a form it gets nothing back fromCapture at the machine fails for physical and social reasons long before technical ones. The designs that survive ask for less and give the operator something back.
6 min readYield variance is usually a measurement problem before it is a process problemBefore funding an improvement project, check that input, output and the standard are still being captured the same way they were four months ago.
6 min readCompliance paperwork should fall out of the transaction, not be typed againThe invoice, the e-way bill and the goods that actually left the gate are one event recorded three times. That arithmetic is what month end is really paying for.
6 min readA recall is a query, and you find out in an hour whether you can run itA complaint arrives with a batch code printed on it. Whether you can trace that code in an hour was settled years earlier, by how you record consumption.
AI assurance
Deciding what an AI system may do before it ships, and being able to prove what it did afterwards.
6 min readLogging every model call is the cheapest governance you will buyWhen an AI feature goes wrong, only the call record can tell you what happened. The hard part is not logging it, it is deciding what to keep and what to mask.
6 min readThe model changed underneath you and nothing in your release notes says soBehaviour drifts without a deploy, and the first person to notice is a customer. What it actually takes to make provider-side change detectable.
6 min readWrite down what the AI feature is not allowed to do before it shipsAcceptance reviews argue about how good the answers are. The shorter, harder list is what the feature must refuse, and that is the one a reviewer asks to see.
Monitoring and alerts
Getting the one number that matters to the person who can act on it, without training everyone to ignore you.
6 min readThe people who need the number first are not sitting at a deskThe dashboards are correct and the news still arrives by phone call. What it takes to get one number into a pocket before somebody walks up the stairs.
6 min readFixed thresholds age badly, so alert on the change insteadA number chosen on a quiet Tuesday keeps asserting itself long after the business has moved on. Compare the metric against its own history instead.
6 min readAn alert nobody can act on is a subscription to worryAlerting estates grow by addition and are almost never pruned. One question decides what to keep, and everything that fails it is a report, not an alert.