AI · Product · Architecture

Adding an AI Assistant to Your Product: What Actually Works

code3 engineering · · 7 min read

Short answer: an AI assistant works when it explains facts your own code has already worked out, sees the user's own data rather than everything, understands the languages your users actually write in, and is cheap enough to run every day. A chatbot bolted onto the side of a product usually fails on all four.

We recently built the AI side of a trading journal platform: a coach that answers traders' questions about their own trading, reviews them against the rules they set for themselves, and turns recorded trading sessions into text it can learn from. Here's what we learned, and what it means for anyone planning to add AI to a product.

1. Let code produce the facts, and let the model explain them

Language models are excellent at explaining and surprisingly bad at arithmetic. Ask one to read a list of trades and tell you a win rate, and you'll get a confident, well-written number that may simply be wrong. In a product people use to make decisions, that's worse than no answer at all.

Two approaches. When the AI does the math, raw data goes straight to the language model and the answer sounds right but may be invented. When code does the math, your code computes the metrics, the language model only explains them, and the answer has the right numbers in clear words. AI does the math Your data raw records Language model reads and calculates “Your win rate is 62%” sounds right, may be invented Code does the math, AI explains Your data raw records Your code computes the metrics Language model explains them Answer right numbers, clear words
The model is the writer, not the calculator.

In our coach, around 60 trading metrics (win rates, best hours and days, risk/reward, position sizes, drawdowns, how emotional trades perform) are calculated in plain code. The model receives the results and turns them into advice a person can act on.

We learned this the hard way. At one point some answers came back with template placeholders where numbers should have been: the model was faithfully copying the format it was given, gaps included. A cleverer prompt wasn't the fix. Moving every calculation into code was. Anything that genuinely can't be worked out from the data is passed in as "not available", so the model says so instead of guessing.

2. Give it the user's own context, not everything

"When do I trade best?" is only a useful question if the assistant can see your trades. But sending a user's entire history to a model with every question is slow, expensive, and makes answers worse, because the important part gets buried.

The pattern that works: keep a private, searchable memory per user, and for each question pull in only the records that matter for it. Same model, far better answers, and one user's data never ends up in another user's conversation.

3. Your users don't all write in English

Our traders asked questions in English, Italian, Serbian and more. If your assistant only works well in English, a large part of your users quietly get a worse product.

What helped was deciding what kind of question it is before answering it: an analysis, an explanation, or a request for advice. A small, fast classifier trained on real examples in several languages does that reliably, and each type of question then gets the right data and the right instructions. It's cheaper and more predictable than hoping one big prompt handles everything.

4. Plan the running cost from day one

A demo that costs a few cents per question looks fine. At thousands of users asking every day, or hours of video to process, it becomes a line item someone will question.

Cost-aware routing: each request first goes through a quick, cheap triage step (a cache check and a small model). Most requests are handled by a small, cheap model; only those that need it go to a large model or GPU. Both lead to the result. Request question or file Quick triage done before? what kind? Small, cheap model most of the work Large model / GPU only when it's needed Result same quality
Use the expensive option only for the work that needs it.

Transcribing recorded sessions is a good example. Running every video through the largest speech model would be accurate and very expensive. Instead, a small model first detects the language, and only the recordings that need a bigger model or a GPU get one. Videos that were already processed are skipped with a single quick check. Same quality where it matters, at a fraction of the cost.

The general rule: check first, use the smallest model that's good enough, and save the big one for the cases that need it.

5. Small things that make it feel good

Is AI the right tool for your feature?

  1. Is the core of the feature explaining or summarising something, rather than calculating it? (Good sign.)
  2. Can the facts it needs be computed reliably in code first? (Then do that.)
  3. Does it need each user's own data to be useful? (Plan a per-user memory.)
  4. Do your users write in more than one language?
  5. Do you know roughly what it costs per user per month at your expected usage?
  6. Would a wrong answer be embarrassing, or would it cost someone money? (Then facts must come from code, not the model.)

If the honest answer to the first question is "no, it's mostly calculating", you may not need AI at all. A good report or dashboard can be the better product.

Adding AI to your product?

Tell us what you want it to do. We'll tell you where AI helps, where plain code is better, and what it will cost to run.

Talk to us