Subscribe

Switching between Gemini and Ollama behind one Go interface

My golang trading jobs call Gemini for advice, and I wanted to try a local Ollama model without editing every job. Here is the one method interface, the optional health check that fails fast, and how it compares with the Anthropic Go SDK.

Switching between Gemini and Ollama behind one Go interface

A few of my golang trading jobs send a big prompt to Gemini and parse a JSON answer out of the reply. Around September I wanted to see if a small model running on my own machine with Ollama could do some of that work. I didn't want to touch every job to try it. So I put both behind one tiny interface and made the choice an environment variable.

This post is about the code. It is not investment advice, and none of these models should be trusted with money just because they return valid JSON.

In this blog we will look at the interface, the optional health check that makes Ollama fail fast, how the backend gets picked, and at the end how my other repo talks to Claude through the Anthropic Go SDK, which looks quite different.

The interface is one method

The Gemini client I already had exposed one method: Generate(prompt string) (string, error). So I wrote the Ollama client to match it exactly, and the interface is just that.

// Generator is the one operation the trading jobs need from a model.
type Generator interface {
    Generate(prompt string) (string, error)
}

// Pinger is implemented by backends that can be health-checked cheaply.
// Use it to fail fast instead of waiting out an inference timeout.
type Pinger interface {
    Ping(ctx context.Context) error
}

Pinger is separate on purpose. Ollama can be checked for free: call /api/tags and see if the server answers and the model is pulled. Gemini can't. The only real way to check Gemini is to make a call, and that call is billed. So Gemini doesn't implement Ping at all, and a small helper checks only when it can.

// Check health-checks the backend when it supports it.
func Check(ctx context.Context, g Generator) error {
    if p, ok := g.(Pinger); ok {
        return p.Ping(ctx)
    }
    return nil
}

This is the same trick the standard library uses with io.WriterTo and friends. Keep the main interface tiny and check for an optional one with a type assertion.

Why fail fast matters here

My Ollama host is a CPU only machine. A 1B or 3B model on CPU makes single digit tokens per second, so a long structured answer can take minutes. Because of that the Ollama client has a 10 minute HTTP timeout by default, which you'd never set for a cloud API.

That long timeout has a nasty side. If host is off, or the model isn't pulled, a job could sit there for a long time before it finds out. Ping uses its own client with a 10 second timeout, so a dead host is known almost straight away.

// Ping checks the server answers and that the model is pulled.
func (c *Ollama) Ping(ctx context.Context) error {
    req, err := http.NewRequestWithContext(ctx, http.MethodGet, c.baseURL+"/api/tags", nil)
    if err != nil {
        return err
    }
    client := &http.Client{Timeout: 10 * time.Second}
    resp, err := client.Do(req)
    if err != nil {
        return fmt.Errorf("ollama %s unreachable: %w", c.baseURL, err)
    }
    defer resp.Body.Close()
    raw, err := io.ReadAll(resp.Body)
    if err != nil {
        return err
    }
    if resp.StatusCode != http.StatusOK {
        return fmt.Errorf("ollama /api/tags returned %d", resp.StatusCode)
    }
    var tags struct {
        Models []struct {
            Name string `json:"name"`
        } `json:"models"`
    }
    if err := json.Unmarshal(raw, &tags); err != nil {
        return fmt.Errorf("decode ollama tags: %w", err)
    }
    var available []string
    for _, m := range tags.Models {
        if m.Name == c.model {
            return nil
        }
        available = append(available, m.Name)
    }
    return fmt.Errorf("model %q not present on %s (available: %s), run: ollama pull %s",
        c.model, c.baseURL, strings.Join(available, ", "), c.model)
}

I like that the error tells you the fix. If the model is missing, it lists what is there and prints the ollama pull command.

Picking the backend

New reads the config and returns one or the other. Here it is trimmed down, with the config read straight from the environment.

type Config struct {
    LLMProvider  string
    GeminiAPIKey string
    OllamaHost   string
    OllamaModel  string
}

func getenvDefault(k, def string) string {
    if v := os.Getenv(k); v != "" {
        return v
    }
    return def
}

func LoadConfig() Config {
    return Config{
        LLMProvider:  strings.ToLower(getenvDefault("LLM_PROVIDER", "gemini")),
        GeminiAPIKey: os.Getenv("GEMINI_API_KEY"),
        OllamaHost:   getenvDefault("OLLAMA_HOST", "http://127.0.0.1:11434"),
        OllamaModel:  getenvDefault("OLLAMA_MODEL", "llama3.2:1b"),
    }
}

func New(cfg Config) (Generator, error) {
    switch strings.ToLower(strings.TrimSpace(cfg.LLMProvider)) {
    case "", "gemini":
        if strings.TrimSpace(cfg.GeminiAPIKey) == "" {
            return nil, fmt.Errorf("LLM_PROVIDER=gemini but GEMINI_API_KEY is not set")
        }
        return &Gemini{apiKey: cfg.GeminiAPIKey, http: &http.Client{}}, nil
    case "ollama":
        if strings.TrimSpace(cfg.OllamaModel) == "" {
            return nil, fmt.Errorf("LLM_PROVIDER=ollama but OLLAMA_MODEL is not set")
        }
        return &Ollama{
            baseURL: strings.TrimRight(cfg.OllamaHost, "/"),
            model:   cfg.OllamaModel,
            http:    &http.Client{Timeout: 10 * time.Minute},
        }, nil
    default:
        return nil, fmt.Errorf("unknown LLM_PROVIDER %q (want %q or %q)",
            cfg.LLMProvider, "gemini", "ollama")
    }
}

Two decisions in there that I'd make again.

The default is Gemini, and that is deliberate. The jobs that use this place real orders. Pointing them at a new model should be something I do on purpose, not a side effect of adding Ollama support to the repo. So adding the Ollama code changed nothing for any job until I set the variable.

An unknown provider is an error. No fallback. If I type LLM_PROVIDER=olama, I want the job to stop, not quietly carry on with Gemini while I think I'm testing the local model. There's also a Describe function that returns a string like gemini gemini-2.5-flash or ollama <model> @ <host> for the logs, so every run's output says which model produced the advice in it.

And the startup in a job looks like this.

func main() {
    g, err := New(LoadConfig())
    if err != nil {
        fmt.Println("config:", err)
        os.Exit(1)
    }
    ctx, cancel := context.WithTimeout(context.Background(), 15*time.Second)
    defer cancel()
    if err := Check(ctx, g); err != nil {
        fmt.Println("backend not ready:", err)
        os.Exit(1)
    }
    fmt.Println("backend ready")
}
LLM_PROVIDER=ollama OLLAMA_HOST=http://localhost:1 go run .
backend not ready: ollama http://localhost:1 unreachable: Get "http://localhost:1/api/tags": dial tcp [::1]:1: connect: connection refused

Instant. Not ten minutes.

Testing without any model

Since the interface is one method, a fake is two lines. And httptest covers the ping.

type fakeModel struct{ reply string }

func (f fakeModel) Generate(prompt string) (string, error) { return f.reply, nil }

func TestPingMissingModel(t *testing.T) {
    srv := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
        w.Write([]byte(`{"models":[{"name":"qwen2.5:3b"}]}`))
    }))
    defer srv.Close()
    g, err := New(Config{LLMProvider: "ollama", OllamaHost: srv.URL, OllamaModel: "llama3.2:1b"})
    if err != nil {
        t.Fatal(err)
    }
    err = Check(context.Background(), g)
    if err == nil || !strings.Contains(err.Error(), "ollama pull llama3.2:1b") {
        t.Fatalf("err = %v", err)
    }
}

Ollama specific traps

A few things in the Ollama client are not obvious, and each of them is there for a reason written in a comment.

num_ctx has to be set on every request. Ollama uses a context window of 4096 tokens when you don't say otherwise, and a longer prompt is cut off silently. No error. The model then answers confidently on half the data. My trading prompts run to several thousand tokens, so the client asks for 16384 by default. Honestly this is the one I'd tell anybody to check first.

format: "json" is on, since both callers parse JSON out of the reply and it stops the model wrapping the answer in markdown fences. Temperature is 0.1, because advice shouldn't wander between two runs on the same input.

As for how the 1B model actually did: on my small risk gate test it passed 2 cases out of 6. That's a separate post. The interface did its job though, swapping took one variable.

One honest note. The interface and New went in early October, but two of my older handlers still call gemini.New directly. Moving them over is a few lines each and it is still on my list.

How the Claude version looks

In a separate repo, claude-algo-trade, I used the official Anthropic Go SDK instead of raw HTTP. The shape is very different. It is typed, it takes a context, and that code streams the response with adaptive thinking on.

adaptive := anthropic.ThinkingConfigAdaptiveParam{}

stream := ca.client.Messages.NewStreaming(ctx, anthropic.MessageNewParams{
    Model:     anthropic.ModelClaudeOpus4_6,
    MaxTokens: 8000,
    Thinking:  anthropic.ThinkingConfigParamUnion{OfAdaptive: &adaptive},
    Messages: []anthropic.MessageParam{
        anthropic.NewUserMessage(anthropic.NewTextBlock(buildClaudePrompt(symbols))),
    },
})

msg := anthropic.Message{}
for stream.Next() {
    msg.Accumulate(stream.Current())
}
if err := stream.Err(); err != nil {
    return nil, fmt.Errorf("claude stream: %w", err)
}

The reply is a list of content blocks, and with thinking on not all of them is text. The code walks them and keeps the TextBlock. Then it strips any markdown fences and finds the outer { } before json.Unmarshal, since there's no JSON mode flag like Ollama's.

Nothing stops this from going behind the same Generator. A small adapter type with a Generate(prompt string) method that makes its own context with a timeout would do it. I haven't wired that, but if I did I'd change the interface to take a context.Context first. That's the one thing I'd do differently in hindsight. Generate(prompt string) was fine for two HTTP clients with their own timeouts, the SDK makes it obvious the caller should own cancellation.