Over three months I went from never having written several skills, to writing over a dozen, primarily for my professional work in finance, but also some routine day to day automation workflows. Each skills typically does something distinct, and the interesting evolution was that I kept relying on the model less and building more deterministic scaffolding around it. This is the field guide to what held up: provenance tags on every fact, a closed list of sources nothing may be cited outside of, web search as a switch rather than a default, failure logs read before every run, and gates written in plain Python that block delivery when something’s wrong.
I’ve written about a dozen skills for Claude since March. The mental model I landed on is that Claude is a fast, fallible analyst. Quick, broadly capable, reads anything you put in front of it, and will hand you a confident wrong answer wearing the same face it uses for a right one. Almost everything I’ve built since is a mechanism for catching that wrong answer before it ships. This is a companion to the toolkit piece I wrote earlier; that one is the what, this one is the how and the why.
Here’s the shape of it, oldest to newest.
Business-Understanding. Reads a company’s filings (10-Ks, S-1s, annual reports across the US, India and Europe) and pulls out what it does, how it earns, who its customers and rivals are. It taught me document triage: rank the filings before reading, and start with the Business Description section, because that one section gives you 70% of what you need.
Initiation Merge. A pure-Python script, no model involved, that stitches separate report stages into one document. The first step I deliberately took away from the model: because assembling text turned out to be a code problem, and model improvisation at the join is a defect, not a flourish.
Earnings Sentiment, and the first pipeline skills. Earnings Sentiment reads up to 16 quarters of earnings-call transcripts and pulls the recurring positives and concerns from management and analysts. This is where I started tagging every claim with where it came from, and where I split “closed-book” and “open-book” into separate skills.
SellSideInitiation. The flagship. One trigger phrase runs a five-phase pipeline that turns documents I supply into a 25-page closed-book research report. Eight hard rules, a six-tier source hierarchy, gates that block delivery. The most defended thing I’ve built.
investment-moats. Competitive-advantage analysis through four named frameworks: Helmer’s 7 Powers, Greenwald’s Earnings Power Value, Buffett-Munger, Porter. It proved the structure wasn’t tied to reports; the same discipline carried straight over to a framework skill.
voice-dna. How I write, across four formats and a professionalism dial, built from 16 of my own writing samples over seven years. It pointed the gate idea at prose instead of numbers.
user-profile. My standing instructions (identity, file paths, formatting standards, code conventions), loaded at the start of every task so nothing gets re-asked.
It’s the amount of work I entrust to the model to perform automatically (and thus without supervision) that decreases, while the amount of deterministic structure that I place around it increases.
What has changed over time wasn’t the subject, i.e., the “what.” It was the degree of reliance I have on the model to produce accurate information for me.
Default: Closed-Book
I got something wrong early on when I treated the model’s knowledge as a resource. I didn’t realize that everything the model knows about most publicly traded companies comes from their training data. Therefore, much of the knowledge the model has acquired during the course of its training is likely old, unverifiable, and irrelevant. Thus, across every analytical capability, I have adopted a straightforward policy, namely, training data cannot be used as a reference. As such, if I cannot identify the original document that provided the basis for a particular assertion I make, the assertion will either be flagged as unverified or will not be made at all.
To implement this principle required three additional components. First, I created a new type of provenance. All values in reports contain a six-character identifier indicating the source from which they came. These identifiers were developed based upon a closed dictionary; therefore, there is always an option available to determine whether a value in a report was properly sourced. Second, I established an explicit, finite list of allowable references. At the end of each report is a log listing all documents from which content was derived. Only references contained within this log are acceptable as citations. Thirdly, I implemented a mechanism to treat web searches as optional rather than mandatory. Web searches are only enabled during two steps within the report production process: once to obtain current market data, and again to enable access to external price and comparable data. All other steps rely exclusively on previously obtained data. By limiting use of web searches to these few steps, the potential exists for the entire provenance tracking mechanism to fail should data become accessible via web searches. This could occur whenever a numeric value enters a report via web searches but does not include an indication of its source.
Thus, underlying all three mechanisms is an emotional premise that ultimately took me the longest time to recognize: a report that contains honest gaps is proper output. Conversely, a report containing at least one confidently incorrect numerical value is defective regardless of whether the incorrect value also happens to be numerically correct. An empty cell marked “Not Retrieved” represents success. While the model produces results that are extremely similar to those that would be produced manually, and in many cases appear almost identical, the fact remains that the model cannot distinguish between those results that are essentially accurate and those results that are inaccurate.
The Snowflake Run
At one point I attempted to speed up development by using multiple agents to develop different parts of the report simultaneously: a merge agent, an audit agent, and several data extraction agents. Each agent had a narrowly defined role and clean architecture. However, I decided to examine some results from a single run carefully.
One segment of the report included 100 of 122 financial data elements that were fabricated by a data extraction agent. The agent had received clear instructions regarding what data it was expected to extract. The revenue and gross profit values were correct because they happened to be located near the beginning of the API response returned by the data provider. The remaining values had simply been manufactured along with the appropriate source indicators. The overall appearance of the report was normal. The results were incorrect for virtually every element.
The reason why this occurred is due to the architectural design of the system. When a sub-agent begins execution, it executes independently and has no visibility into the instructions file associated with the parent agent nor into the error logs generated during previous executions. Consequently, when it encounters an unknown field it relies entirely on training data to generate a plausible result. The merging agent subsequently generates a cohesive report comprised of plausible but incorrect segments, with no ability to visually detect the seams between them, since each individual segment is internally coherent. Because of this, I am compelled to adopt an absolute rule: no sub-agents will ever be permitted to execute within any analytical tasks. Every task will execute sequentially under the same conversation with all relevant contextual information passed down through that conversation. This approach is considerably slower than multi-agent approaches. However, it is currently the only architecture that consistently provides results consistent with expectations.
The failure log
All serious skills have a FAILURE_LOG.md file (append only). Each entry has: what broke, the run number, and the rule that now prevents the issue. Reading the log is the first step in every run, before any retrieval or writing.
The log works because it closes the loop between “something broke” and “the skill was changed.” Without a log every fix is local: this report, this company, this bug, and the same mistake will come back next week in slightly different clothes. With a log every fix is structural: a rule in the skill, a check in the gate, a line in the instructions file. New failures get written down in the same session they happen, never deferred, because by next time the context is gone. Trace almost any rule I have back far enough and you’ll find a logged failure at the root of it. The Snowflake run is one. A run where a bank’s accounts got read with software template producing phantom debt that didn’t exist, is another. The log is the thing the skills actually grew out of.
The skill is the spec; the gate is the gate
A skill file is a prompt; a prompt is a suggestion. The model follows the prompt most of the time and quietly ignores it some of the time; and you don’t get told which run is which. For rules that can’t bend, the only honest enforcement is code that fails the build.
So all the rules I refuse to ship without are each backed by a small Python program that runs over the output and exits with an error when something’s wrong. One runs regex over prose and hard-fails on the worst machine-writing tells. One runs five checks over a finished report: no leftover placeholder tags, no unretrieved numbers in historical columns, citations that actually resolve, a balance sheet that reconciles to the dollar. One diffs the written numbers against the raw data feed to catch exactly the Snowflake failure. Each gate is a hundred lines or less, runs in seconds, and uses no model at all. That last part is the point: you can’t ask the thing that makes the mistake to also be the thing that reliably catches it. The catcher has to be dumber, deterministic, and incorruptible.
This is the same line drawn over and over again. The model does judgment: reading, weighing, classifying, composing. Deterministic code does assembly, transformation, and checking. If a task has one correct output given its input, it belongs in code; if it needs reading and weighing, it belongs to the model and then goes straight to code for assembly and verification. Most of my failures came from putting the model on the wrong side of that line: asking it to do bookkeeping and watching it improvise.
The smaller habits that compounded
I developed a few lighter patterns. Classify your input before you process it. The report pipeline determines whether a company is a software business, a bank, an insurer, a REIT, a marketplace, a miner, or holding co. before it touches the financials, because each needs a different template and the wrong one corrupts everything quietly. That step exists because a bank once got read as a software company and the output looked clean while being conceptually wrong.
Tell your model where it may and may not exercise judgment. In my note-taking skills the body of a note is copied verbatim: no summarizing, no improving: and the only authored parts are the title and a few lines of metadata. The model instinctively wants to add value everywhere, and “value” often means paraphrase, which is destructive when the whole point is fidelity to source.
Write your skill against the tool, not the vendor. The data source appears throughout as a single named parameter with a note that today it’s one provider and tomorrow it might be another. Connectors change; a skill with the vendor’s name welded into forty instructions becomes a rewrite, while a parameterized one is a one-line change.
The whole kit, on one page
Strip the finance out and twelve patterns are left. Each one survives the domain: substitute your own work for mine and it still holds. Here they are in one place, for the reader who wants the checklist rather than the story.
Where this leaves me
The failure logs keep growing. The one next to the report pipeline is past two dozen entries, and each is a mistake the model made once and a line that now makes sure it can’t make it the same way twice. Every skill I write now opens the same way: read the log, confirm the parameters, decide what the model is and isn’t allowed to touch, and only then start.
It’s slower than just asking. It is also the only version I’d put my name on a report from. I don’t know what the next skill will break, or which rule it’ll add to the pile. I know it’ll start by reading the log.