Why a 1B Ollama model failed as a trading risk gate
I tried to turn llama3.2:1b into a pre-trade rule checker with an Ollama Modelfile. It passed 2 of 6 test cases, invented a margin check and said 174 is under 45. Here is the setup, the eval script and what I use instead.
I have a few small Go jobs that place orders on my own Zerodha account. They already talk to a language model for advice, and in September I wanted to see if a tiny local model could do one narrow job for them: look at a proposed trade and say whether it breaks any of my hard rules. No forecasting. Just the checks.
It failed. 2 out of 6 test cases passed. In this blog we will look at how I built it with an Ollama Modelfile, how I tested it, and why those checks now live in plain Go with unit tests.
This is about the code and testing an LLM, not investment advice.
Why I even tried this
Before this, I had looked at how good the model's direction calls were in one session. It made 193 probability calls and only 9 of them could be scored against a price I had recorded. Six were right. The chance of getting 6 or more of 9 by pure coin flips is about 0.25, so that told me nothing at all.
So I flipped it around. Don't ask the model where the price is going. Ask it to do the boring part: apply the gates, check the arithmetic, and refuse when something is off. A 1B model should manage that. That was the theory anyway.
The Modelfile
Ollama doesn't fine tune anything for you. What it gives you is a Modelfile: a base model, some decode parameters, a system prompt and a few example messages, all baked into a new model name with ollama create. No weights change.
Here is a cut down version of mine. The real system prompt had seven gates (net position cap, margin floor, order size vs visible depth, time to the 15:20 MIS square-off, stop required, CNC product rules and a no-flip rule) and asked for JSON only.
FROM llama3.2:1b
PARAMETER temperature 0.05
PARAMETER top_p 0.9
PARAMETER repeat_penalty 1.05
PARAMETER num_ctx 8192
SYSTEM """
You are a pre-trade RISK GATE. You never predict direction.
Reply with JSON only:
{"verdict": "ALLOW" | "REDUCE" | "NO_TRADE",
"allowed_quantity": <int>, "failed_gates": [<string>], "reasons": [<string>]}
1. NET CAP. |current_net +/- proposed_qty| must not exceed net_cap.
2. MARGIN FLOOR. No new adds if free_margin is below margin_floor.
3. TIMING. No new MIS entry with under 45 minutes to square-off.
4. PRODUCT. A CNC sell larger than holdings_qty + t1_qty is a short
delivery. Always NO_TRADE.
If a number you need is missing, put it in failed_gates and return NO_TRADE.
Never estimate a price, quantity or margin figure.
"""
MESSAGE user """{"symbol":"TCS","side":"SELL","product":"CNC","proposed_qty":40,"holdings_qty":0,"t1_qty":0,"net_cap":500,"free_margin":90000,"margin_floor":50000,"minutes_to_squareoff":174}"""
MESSAGE assistant """{"verdict":"NO_TRADE","allowed_quantity":0,"failed_gates":["PRODUCT: CNC sell of 40 against holdings 0 and T1 0 is a short delivery"],"reasons":["short delivery goes to auction with a penalty"]}"""
The temperature is near zero on purpose. Gate arithmetic should give the same answer for the same input every time.
I had five MESSAGE pairs in the real file, and every one taught a refusal or a reduction. Only one showed a clean ALLOW. Keep that in mind, it matters later.
Building and running it is two commands.
ollama create algotrade -f Modelfile.algotrade
ollama run algotrade '{"symbol":"INFY","side":"BUY","product":"CNC","proposed_qty":10}'
The eval script
I didn't want to judge it by poking at it in a terminal. So I wrote six cases from real situations in one trading session, each with the verdict I knew was correct, and a small script to call the model and score it. It only uses the standard library.
import json
import urllib.request
HOST = "http://localhost:11434"
MODEL = "algotrade"
CASES = [
("CNC sell with zero holdings",
{"symbol": "TCS", "side": "SELL", "product": "CNC", "proposed_qty": 40,
"holdings_qty": 0, "t1_qty": 0, "net_cap": 500, "free_margin": 90000,
"margin_floor": 50000, "minutes_to_squareoff": 174},
{"verdict": ("NO_TRADE",), "must_flag": "PRODUCT"}),
("missing margin figure",
{"symbol": "INFY", "side": "BUY", "product": "CNC", "proposed_qty": 10,
"current_net": 0, "net_cap": 500, "margin_floor": 50000,
"minutes_to_squareoff": 200},
{"verdict": ("NO_TRADE",), "must_flag": None}),
]
def ask(payload):
body = json.dumps({"model": MODEL, "prompt": json.dumps(payload),
"stream": False, "format": "json",
"options": {"temperature": 0.05}}).encode()
req = urllib.request.Request(HOST + "/api/generate", data=body,
headers={"Content-Type": "application/json"})
with urllib.request.urlopen(req, timeout=300) as r:
return json.loads(r.read())["response"]
passed = 0
for name, payload, expect in CASES:
got = json.loads(ask(payload))
verdict = str(got.get("verdict", "")).upper()
flags = " ".join(str(x).upper() for x in got.get("failed_gates") or [])
ok = verdict in expect["verdict"] and (
not expect["must_flag"] or expect["must_flag"] in flags)
passed += ok
print("pass" if ok else "FAIL", name, "->", verdict, flags)
print(f"{passed} passed out of {len(CASES)}")
"format": "json" asks Ollama to force valid JSON out. That part worked fine, every answer parsed. The content was the problem.
What came back
Two passed. Four failed. And the failures was not cosmetic.
The worst one was the missing margin case. I left free_margin out of the input completely. The system prompt says in capitals to refuse when a number is missing. The model returned ALLOW, and one of its reasons was "margin above floor". It made up a passing check for data it never got, and it made it up in the direction of letting the trade through.
Then the clean trade. Every gate should pass, 174 minutes left before square-off. It came back REDUCE with this.
TIMING: 174 minutes to square-off is under the 45 minute minimum
174 is not under 45. That's a single integer comparison and it got it backwards.
The CNC short delivery case was sneaky. The model wrote the right reason in failed_gates, it even said "short delivery". But the verdict was REDUCE, not NO_TRADE. Any code that reads verdict would size the order down and still send a short sell. A wrong verdict with a correct explanation next to it is worse than a plain wrong answer, because it looks fine when you skim the log.
The margin plus timing case flagged only the margin and missed the timing gate.
Looking at all six together, five answers were REDUCE. It didn't matter much what I sent. My read is that it latched on to REDUCE from the few shot examples and just repeated it.
And here is the part that honestly annoyed me most. Four of my six test cases were the exact same inputs as the few shot examples in the Modelfile. The model had seen the right answer for those inputs, word for word, in its own prompt. It still got three of those four wrong.
What I did with it
I deleted the model from Ollama host right after the run, so nothing could point OLLAMA_MODEL at it by accident. The Modelfile stays in the repo with a big comment block on top saying it was built, tested and failed, along with the results file. A negative result is still a result, and I didn't want future me to try the same thing again in six months.
The jobs pick their model through one small interface in internal/llm, chosen by LLM_PROVIDER. The default stays Gemini. Switching to Ollama has to be done on purpose, and an unknown provider is an error, not a silent fallback. That design is why this experiment never got anywhere near a real order.
The gates in plain Go
These gates are integer and float comparisons. They don't need a model. Here is a small version of the checks from the eval, as a Go package.
package gate
import "fmt"
type Order struct {
Side string // BUY or SELL
Product string // MIS or CNC
Qty int
CurrentNet int
NetCap int
Holdings int // settled + T1 quantity, used for CNC sells
FreeMargin *float64 // nil means we don't know it
MarginFloor float64
MinsLeft int // minutes to the 15:20 MIS square-off
}
type Verdict struct {
Allow bool
Failed []string
}
func abs(n int) int {
if n < 0 {
return -n
}
return n
}
func Check(o Order) Verdict {
var failed []string
delta := o.Qty
if o.Side == "SELL" {
delta = -o.Qty
}
after := o.CurrentNet + delta
reducing := abs(after) < abs(o.CurrentNet)
if abs(after) > o.NetCap && !reducing {
failed = append(failed, fmt.Sprintf("NET_CAP: net after fill %d is over cap %d", after, o.NetCap))
}
if !reducing {
switch {
case o.FreeMargin == nil:
failed = append(failed, "MISSING: free_margin")
case *o.FreeMargin < o.MarginFloor:
failed = append(failed, fmt.Sprintf("MARGIN_FLOOR: %.0f is below %.0f", *o.FreeMargin, o.MarginFloor))
}
if o.Product == "MIS" && o.MinsLeft < 45 {
failed = append(failed, fmt.Sprintf("TIMING: %d minutes left, need 45", o.MinsLeft))
}
}
if o.Product == "CNC" && o.Side == "SELL" && o.Qty > o.Holdings {
failed = append(failed, fmt.Sprintf("PRODUCT: CNC sell of %d against holdings %d", o.Qty, o.Holdings))
}
return Verdict{Allow: len(failed) == 0, Failed: failed}
}
Margin is a pointer so "not given" and "zero" are different things. That's the exact case the model fumbled. Reducing a position skips the margin and timing checks, same as the rule in the Modelfile.
And the same cases as the eval, as a table test.
package gate
import "testing"
func margin(f float64) *float64 { return &f }
func TestCheck(t *testing.T) {
cases := []struct {
name string
order Order
allow bool
}{
{"clean CNC buy, plenty of time",
Order{Side: "BUY", Product: "CNC", Qty: 10, NetCap: 500,
FreeMargin: margin(90000), MarginFloor: 50000, MinsLeft: 174}, true},
{"CNC sell with nothing held",
Order{Side: "SELL", Product: "CNC", Qty: 40, NetCap: 500,
FreeMargin: margin(90000), MarginFloor: 50000, MinsLeft: 174}, false},
{"margin below floor and too late",
Order{Side: "BUY", Product: "MIS", Qty: 5, NetCap: 500,
FreeMargin: margin(40000), MarginFloor: 50000, MinsLeft: 30}, false},
{"margin not given",
Order{Side: "BUY", Product: "CNC", Qty: 10, NetCap: 500,
MarginFloor: 50000, MinsLeft: 200}, false},
{"already over cap, buying more",
Order{Side: "BUY", Product: "MIS", Qty: 100, CurrentNet: 800, NetCap: 500,
FreeMargin: margin(90000), MarginFloor: 50000, MinsLeft: 300}, false},
{"over cap, selling down is fine even on low margin",
Order{Side: "SELL", Product: "MIS", Qty: 300, CurrentNet: 800, NetCap: 500,
FreeMargin: margin(1000), MarginFloor: 50000, MinsLeft: 10}, true},
}
for _, c := range cases {
got := Check(c.order)
if got.Allow != c.allow {
t.Errorf("%s: allow=%v, want %v (failed: %v)", c.name, got.Allow, c.allow, got.Failed)
}
}
}
go mod init gate
go test -v .
All six pass, every time, in well under a second. No GPU and no host on the network to keep alive.
What I'd tell myself in September
If a rule can be written as if a < b, write it as if a < b. Putting it behind a model, any model, adds a way for it to fail and takes away the one thing you wanted from a gate, which is that it always says the same thing. A bigger model would probably have done better on these six cases. I still wouldn't let it decide whether 174 is less than 45.
The jobs still ask Gemini for advice. Anything that decides if an order can go out belongs in Go, with tests.