Two weeks into the closed beta, one of our users was running a live XAUUSD long. Stop was set. Price moved against the trade, crossed the stop level. He asked the system whether his stop had been hit.
The system said no. He pushed back. It said yes. Then it walked the answer back.
This was not a prompt engineering problem. It was a category error in the architecture, and it is one that is easy to make when you are building AI tools quickly.
Here is the distinction that matters. There are two types of questions a system operating in a high-stakes domain can be asked. The first is interpretive: what does the chart structure suggest, is this setup still valid, where is the bias. The second is factual: has price crossed this level, what is the current spot price, is my stop hit. These look like the same kind of question from the outside. They are not.
AI is genuinely good at the interpretive kind. It can read structure, weigh session context, synthesise a narrative across multiple timeframes, and produce a useful read faster than most people can do it manually. That is real value. But when you ask it a factual question, especially one that requires live data it was never given, it will still answer. Confidently. And the answer will be based on inference from whatever context it does have, not verification of the fact. When the user pushed back, the system updated its answer based on the pressure of the conversation, not because it had checked anything. That is how language models behave. It is not a flaw. It is a design characteristic you have to build around.
The rule we established after this incident: models handle interpretation, code handles facts.
We rebuilt the price tracking that weekend. While a trade is active, the system runs a continuous live check in the background, comparing stored stop and target levels against current spot price. When a level is crossed, a notification fires. No AI in that path. The arithmetic is deterministic and does not respond to pushback.
The read engine still does everything it is built for: structure, narrative, directional bias, setup quality. The price monitoring layer handles the factual side. They do not overlap.
For anyone building AI tools in high-stakes domains, this is the design principle worth taking seriously. The risk is not that the AI refuses to answer hard questions. It is that it answers them anyway, and you believe it because the answer sounds right. The mitigation is not better prompting. It is deciding upfront what class of question each part of your system is responsible for, and making sure the architecture enforces that separation. We learned it in production. Better to learn it on paper.