From Prompt to P&L: An Auditable, Self-Improving Multi-Agent Trading Desk
The Debate Is the Easy Part
Building a multi-agent trading system where language models argue, deterministic code owns every number, and the whole thing learns from its own track record — while weighing where a business is heading, not only where it has been.
— –
The one idea worth your time
Large language models are astonishingly good at one thing and quietly terrible at another.
They are great at narrative: give a model a pile of messy, half-structured evidence and it will read all of it and write a confident, coherent story that ties it together. They are bad at arithmetic you can trust: ask the same model to size a position, respect a risk limit, or keep a single number consistent across three paragraphs and it will happily invent a precise-looking figure that doesn’t survive a second look.
Most AI-agent projects fail because they ask the model to do both jobs. This project is built on a different bet, in two parts.
First: let the models reason, and let deterministic code decide anything that has to be correct. Second: a system that only argues never improves — so this one remembers every decision, waits to see how it actually turned out, and retrains on its own history.
The domain here happens to be markets — the system researches a stock or a commodity and produces a recommendation. But strip the finance away and what’s left is a general blueprint for trustworthy, self-improving agents: a reasoning stage that emits a typed opinion, a pipeline of auditable code that turns that opinion into an outcome, and a feedback loop that closes over time.
Code: https://github.com/shivu-agr/TradingCrew
The rest of this piece is about everything that happens around and after the debate, because that is where the interesting engineering lives.
— –
Why a clever prompt isn’t enough
Three failure modes show up again and again, and almost every design choice below is a countermeasure to one of them.
1. Non-determinism where you need determinism. The same prompt, run twice, returns two different position sizes. That’s fine for brainstorming and unacceptable for anything that acts on the answer. You cannot unit-test a vibe.
2. Hallucinated precision. A model doesn’t just make up facts — it makes up specificity. It will write that a company’s return on capital fell from 18% to 12% between 2019 and 2023 and attach a citation to a data source that never produced those years or that metric. The sentence looks rigorous. It is fiction wearing a lab coat.
3. No audit trail. When the output is wrong, you want to know *which step* went wrong. A monolithic prompt gives you one opaque blob; you can’t point at a line and say “this is where it broke.”
— –
The architecture in two halves
There is a clean seam down the middle. Everything to the left is language models talking to each other. Everything to the right is deterministic Python you can step through in a debugger.
The reasoning half
A small team of specialist agents built on CrewAI works four wide desks, each a consolidation of two former narrow ones (eight desks collapsed into four to cut redundant tool-loops and information dilution):
– Market + Quant — price, technical indicators, options/positioning, and analyst recommendations.
– News + Social — catalysts, corporate actions, news sentiment, and insider activity.
– Fundamentals + Sector — the financial statements (balance sheet, cash flow, income), the shareholding pattern, a long-term thesis, and sector/peer comparison.
– Macro + Geopolitical— the macro regime, geopolitical news, and supply-chain risk.
The desks run in parallel, a short bull-versus-bear exchange forces the model to argue both sides and rebut itself, a manager picks the stronger side, a trader turns it into a plan, and a small risk panel stress-tests it.
That is the part most people picture when they hear “multi-agent trading.” It matters — but it’s deliberately kept lean, because the debate is not where trust comes from. Trust comes from what the verdict passes through next.
The crucial detail: that verdict is not free text. It’s a typed, validated object — action, conviction, entry, stop, target, horizon, rationale, key drivers, risks, and “what would change my mind.” That typed boundary is the handoff, and it’s where the model’s authority ends.
The whole run is visualized live: each agent lights up as it executes, its tool calls appear as chips beneath it, and edges animate from idle → running → done → degraded. When you’re debugging a fifteen-step agent graph, watching it run is worth a hundred log lines.
The deterministic half
Once the typed decision crosses the seam, no model call is allowed to change a number. It flows through small, independently tested modules:
The shape is the point: the model proposes; a chain of verifiable functions disposes. Every stage has tests, so when something looks off you can point at the exact module.
— –
Beyond the debate: reviewers and the forward view
The bull and the bear get the attention, but two quieter ideas do more for reliability.
Reviewing is a different job than producing. A reflective critic interrogates the final decision — does every quantitative claim carry a real, machine-checkable source, and does the action actually match the reasoning? It can force the system to abstain rather than ship a decision it can’t defend. Two optional reviewers sit alongside it (a quality audit and a compliance check), but they are advisory by design. An early version of the quality reviewer was so strict it deleted good reasoning over minor gaps, so it had to be retaught that an honest, clearly labelled judgement counts as supported — and that its score is a dial, not a switch. A critic that can veto on weak evidence will quietly destroy your best outputs.
Weigh where the business is heading, not only where it’s been. Any system that grades itself on citations drifts toward the past, because the past is where the hard, quotable numbers live. So the fundamentals desk pulls an explicit forward view — analyst targets, the trajectory of forward-versus-trailing valuation and earnings, consensus growth, recent estimate revisions — and is then required to reconcile it against the trailing story and say which way it leans. On one real company this flipped the conclusion: the trailing picture looked ugly, but the forward one (real growth, a compressing valuation, rising earnings) was clearly constructive. Every decision also carries an explicit horizon — short, medium, or long, out to a ten-year view — and long-horizon calls must rest on multi-year fundamentals rather than a chart pattern.
— –
Guardrails: the code owns the numbers
This is the heart of the project — and the part I’d port into any agent system, whatever the domain. The model gets to propose numbers; code decides whether they may stand.
Provenance is a data structure, not a hope. Every fact a tool emits carries an inline `[source: …]` tag, and every agent downstream must keep that tag when it quotes the number — so “is this claim supported?” becomes a string check, not a judgement call. A deterministic pass then relabels any citation pointing at an obviously fake source as an honest “my own judgement,” and the critic hunts fabricated precision: a real source tag glued to a figure that source could never have produced. The rule of thumb — if support can be verified by code, it is; if it can only be verified by careful reading, it won’t be.
Price levels are checked, not trusted. A deterministic layer recomputes real volatility from the same price history the pipeline uses, then:
– widens any implausibly tight, wrong-side, or missing stop to a level the holding period can actually survive;
– validates the target the same way — fixing a nonsensical one, capping a fantasy one, leaving a plausible one alone;
– recomputes the expected return so the headline number matches the levels.
And when volatility genuinely can’t be measured, it abstains rather than invent a floor— because a guardrail that fabricates data to satisfy itself is worse than none.
— –
Closing the loop: learning from what actually happened
A recommendation is a hypothesis. The interesting part is what you do once reality answers.
Every decision is stored as an episode. When enough time passes for the outcome to be known, that episode becomes labelled data. The UI exposes it as a spreadsheet-style view — summary cards over scrollable rows with a sticky header — so you can actually see what the system has to learn from.
Those resolved episodes feed learning loops that get progressively more ambitious: reflecting on what worked and folding the lesson into future prompts; a grid search over strategy parameters ranked by a metric that penalizes the illusion of skill you get from testing many variants; a reinforcement-learning pass on the simulator; and a market-feedback loop that ranks past decisions by realised performance and reuses the strongest winners and losers as worked examples. There’s also a full offline path — export the episodes, run a LoRA fine-tune on a base model, then serve the tuned model locally and pick it from the same menu you’d use for any hosted model, all from the UI.
One rule holds throughout: nothing — retrieval, backtest, or training — is ever allowed to see information that wasn’t available at decision time, so the loop makes the system better without ever quietly cheating.
— –
Patterns worth stealing
A few things that had nothing to do with markets and everything to do with making the thing usable.
– Treat model calls as a budget. An early version made 400+ calls per run. Consolidating narrow specialists into wider desks, making reviewers optional and asynchronous, defaulting the heaviest memory feature off, and running the self-critique in a single pass cut that by more than half while keeping the debate. Call count is a first-class metric for an agent system.
– Own long-running work with sessions. Runs take minutes. The backend runs them as server-owned sessions: start one, close the tab, come back, re-attach to the live stream, with several tickers at once. Decoupling the run from the socket connection is the difference between a demo and a tool.
– Build the core so a second domain is nearly free. An entire second asset class — commodities and futures — reuses the exact same deterministic pipeline; only the domain tools and one cost setting differ.
And it all lands in one place: every decision executes against a virtual book — a sandboxed paper portfolio by default, so the system is effectively trading play money you can inspect. The portfolio view shows live NAV and cash, realised and unrealised P&L, drawdown, and open positions, plus a multi-ticker allocator (hierarchical-risk-parity, mean-variance, or equal-risk) that balances the latest proposal for each ticker into target weights. It’s where a pile of per-ticker calls turns into one book you can actually track.
— –
What I’d tell another engineer building agents
1. Draw a seam. Decide which outputs are narrative (model-owned) and which are facts/numbers (code-owned), and make the boundary a typed object. Everything good here flows from that.
2. Make claims checkable by machines. Provenance markers, typed schemas, and a shared definition of “supported” turn quality from a reading exercise into a test.
3. Put deterministic guardrails after the model — and let them abstain. Correcting is good; inventing to look confident is the exact failure you’re preventing.
4. Spend agents on reviewing, not just producing — and give reviewers dials, not vetoes.
5. Close the loop. An agent that can’t see how its past decisions turned out can’t get better; wire in memory, outcome labels, and a retraining path from the start.
6. Never let information from the future leak backwards. Whatever your domain’s equivalent is, it’s probably your subtlest bug.
None of this is specific to markets. It’s just what it takes to make a pile of probabilistic text generators behave like a system you’d trust.
— –
Inspiration & references
– Multi-agent debate topology — the analyst-team → bull/bear → trader → risk-panel → portfolio-manager shape is inspired by TauricResearch’s TradingAgents : https://github.com/TauricResearch/TradingAgents
– The paper — Xiao et al., TradingAgents: Multi-Agents LLM Financial Trading Framework, arXiv:2412.20138 (2025).
– The deterministic layer— the M1–M7 post-LLM contract (bridge, simulator, memory, critic, sizer + gates, walk-forward, allocator) draws on arXiv:2605.19337.
TradingCrew is an independent implementation — built on CrewAI, with its own futures dashboard, a FastAPI/WebSocket web UI, and the deterministic M1–M7 pipeline plus its training loops on top. No source code is copied from the upstream project.
— –
Try it
The full project — the agent crews, the deterministic pipeline, both dashboards, and the learning loop — is open source:
https://github.com/shivu-agr/TradingCrew
Clone it, point it at a ticker, and watch the graph light up. If you build something with the “narrative from the model, math from the code” pattern, I’d love to hear what you did with it.
A quick honesty note: this is a research and engineering project, not investment advice. It exists to demonstrate an architecture, not to tell you what to buy.
