In July, a batch of guest invites for the firm’s podcast looked finished. Every email was written. Before anything went out, the drafts were checked against their sources, and most of them could not be sent. Some were addressed to people the system had already marked as not a fit. One sat on a duplicate record with a guessed email address, and it repeated the person’s own deal terms back at them. None of the not-a-fit drafts went out, and neither did the one on the duplicate record.
I build AI systems at EG Capital, a finance firm, while I study business at the University of Miami. That batch is one example of why every system I build has a point where a person checks the work before anything goes out.
Confident and wrong
The hard thing about AI is not that it makes mistakes. People make mistakes too. The hard thing is that AI is wrong in the same calm, finished voice it uses when it is right. A draft to the wrong person reads as smoothly as a draft to the right one. A report with a wrong number in it looks exactly like a report with the right one.
In finance, that matters more than in most jobs. The people who read a report from a finance firm make decisions about their money with it. One wrong figure, stated with confidence, does more harm than a slow answer that is correct.
Regulators see it the same way. In June 2024, FINRA, the self-regulatory organization for broker-dealers in the United States, published Regulatory Notice 24-09. It reminds firms that its rules “continue to apply when member firms use Gen AI or similar technologies in the course of their businesses.” The tool can change while the responsibility stays where it was. If a firm sends something, the firm said it, whatever wrote the first draft.
Where the check goes
A check works best when it is part of the system itself, so nobody has to remember to add it at the end.
In the outreach system, a draft only reaches my Gmail drafts folder after the address has been confirmed as the person’s own, not guessed and not a shared inbox, and after every fact in it traces to a source. Anything that fails is held. The system never sends an email.
I had it do the research and the writing and park every draft in my Gmail, so I could look it over first. If a draft has a mistake, or reads like it came from AI, I won’t get an answer, and it comes across as disrespectful. A lot of these people are worth knowing, and one careless email is a burnt bridge.
People call this the 10-80-10 rule. I set the direction, the AI does the bulk of the work, and I do the last 10 percent. That last part is where the respect is.
The reporting system builds each report from the firm’s records and shows it in a portal behind a login. It sends nothing. Every report needs a person’s sign-off before it can publish, and every figure traces back to its record. When I change the code, a change that fails its tests doesn’t ship.
None of these checks is complicated. Each one answers a single question: what is the worst thing this system could send, and what stops it?
How checks build trust
A system people don’t trust doesn’t save anyone time, because they end up redoing its work by hand to be safe. A system with a visible check is different. The reader knows a person looked at it and can see where each number came from.
Tracing helps the person who builds the system, too. When every figure points back to a record, a wrong number can be followed to where it started. When a draft without a source stops on its own, the gap is caught before anyone outside the firm sees it.
The National Institute of Standards and Technology builds on the same idea in its AI Risk Management Framework, released in January 2023. The framework is meant to help organizations “incorporate trustworthiness considerations into the design, development, use, and evaluation” of AI. The July batch was caught because a person checked it before anything went out.
The rule I keep
AI is a fast first draft. At EG, the last word belongs to a person, and every system I build has a clear place for that person to give it. That check is what makes it safe to let AI do so much.
Sources
Next post Where AI helps at a finance firm, and where it doesn’t All posts