Backend Setup: Go

This guide walks you through instrumenting a Go application with the SF Veritas SDK for Sailfish Enterprise. Go instrumentation captures logs, print statements, exceptions/panics, HTTP request/response telemetry, and function execution spans.

Auto-Installation

If you connected GitHub and received Auto-Installation PRs, the API key, service identifier, and graphql endpoint are already configured for you. Merge the PR and you're done.

Getting Your API Key

1. Open the Sailfish dashboard
2. Log in with your enterprise email
3. Navigate to Settings > Configuration
4. Copy your company's API key

Installation

Install the SF Veritas Go module:

go get github.com/SailfishAI/sf-veritas-go@latest

Basic Setup

Add the following to your application's main() function:

package main

import (
    "net/http"

    sfveritas "github.com/SailfishAI/sf-veritas-go"
)

func main() {
    sfveritas.SetupInterceptors(sfveritas.Options{
        APIKey:            "<see-api-key-from-your-account-settings-page>",
        ServiceIdentifier: "acme-corp/go-api/cmd/server/main.go", // Format: <org>/<repo>/<path-to-this-file>
        ServiceVersion:    "1.0.0",
    })
    defer sfveritas.Shutdown()

    mux := http.NewServeMux()
    mux.HandleFunc("/api/users", handleUsers)

    // Wrap your handler with SF Veritas middleware for HTTP tracing
    http.ListenAndServe(":8080", sfveritas.Middleware(mux))
}

Configuration Options

Option	Type	Required	Description
APIKey	string	Yes	Your Sailfish Enterprise API key
ServiceIdentifier	string	Yes	Unique identifier in <org>/<repo>/<path> format
ServiceVersion	string	No	Version of your service

What Gets Captured Automatically

Once SetupInterceptors is called, these are captured with zero additional code:

• Structured logs -- All slog.Info(), slog.Warn(), slog.Error() calls
• Print statements -- All fmt.Println(), fmt.Printf(), log.Println() output
• Inbound HTTP requests -- Timing, status codes, headers (via sfveritas.Middleware)
• Outbound HTTP requests -- All http.Get(), http.Post(), client.Do() calls via http.DefaultTransport
• Panics -- Recovered panics with full stack traces (via sfveritas.Middleware; for Gin see the Gin example below)
• Exceptions -- Manually reported errors with sfveritas.TransmitError()

Logs are linked to the request automatically

A plain slog.Info("...") or fmt.Println("...") made inside a request handler is correlated to that inbound request automatically -- you do not need to thread context.Context through your call stack or switch to slog.InfoContext. As long as the request is served through sfveritas.Middleware, the SDK associates telemetry emitted on the handler's goroutine with that request, so your logs appear nested under the right request in the dashboard.

Logs from goroutines you spawn

Go has no goroutine-local storage, so telemetry emitted from a new goroutine you start inside a handler (go func() { slog.Info(...) }()) can't be auto-linked to the request. For those, pass the request context explicitly:

go func() {
    slog.InfoContext(r.Context(), "background work done") // links to the request
}()

Automatic Function Instrumentation (toolexec)

For full function-level tracing in the Flamechart -- including automatic argument capture, return values, and local variable capture on panic -- use the sfveritas-instrument compile-time tool.

How It Works

Go's -toolexec flag lets you intercept the compiler. sfveritas-instrument parses your source code's AST (Abstract Syntax Tree) at compile time and injects lightweight instrumentation into every function:

1. Function entry -- Records the function name, file, line, and captures all argument values
2. Panic recovery -- A defer block captures all local variable values if a panic occurs
3. Function exit -- Records timing and ends the span

The transformation happens at compile time, not runtime. Your source files are never modified on disk.

Install the Tool

go install github.com/SailfishAI/sf-veritas-go/cmd/sfveritas-instrument@latest

Verify it's installed:

which sfveritas-instrument
# Should print: $GOPATH/bin/sfveritas-instrument

Usage with go build

go build -toolexec="sfveritas-instrument" -o myapp ./...

Usage with go run

go run -toolexec="sfveritas-instrument" .

Usage with go test

go test -toolexec="sfveritas-instrument" ./...

Integration Patterns

Makefile

TOOLEXEC := -toolexec="sfveritas-instrument"

.PHONY: dev prod test

# Build with instrumentation
dev:
	go run $(TOOLEXEC) .

# Production build (also instrumented for Enterprise telemetry)
prod:
	GIT_SHA=$$(git rev-parse HEAD) go build $(TOOLEXEC) -o myapp ./...

# Tests with instrumentation
test:
	go test $(TOOLEXEC) ./...

Air Hot-Reload

Air is a popular live-reload tool for Go. Add -toolexec to the build command in .air.toml:

# .air.toml
root = "."
tmp_dir = "tmp"

[build]
  cmd = "go build -toolexec='sfveritas-instrument' -o ./tmp/main ."
  bin = "tmp/main"
  delay = 1000
  exclude_dir = ["assets", "tmp", "vendor"]
  exclude_regex = ["_test\\.go"]
  include_ext = ["go", "tpl", "tmpl", "html"]
  kill_delay = "0s"

[misc]
  clean_on_exit = true

Docker Compose

# Dockerfile
FROM golang:1.22-alpine

WORKDIR /app

# Install the instrumentation tool
RUN go install github.com/SailfishAI/sf-veritas-go/cmd/sfveritas-instrument@latest

# Copy module files first for caching
COPY go.mod go.sum ./
RUN go mod download

COPY . .

# Build with instrumentation
ARG GIT_SHA
ENV GIT_SHA=$GIT_SHA
RUN go build -toolexec="sfveritas-instrument" -o /app/server .

CMD ["/app/server"]

# docker-compose.yml
services:
  api:
    build:
      context: .
      args:
        GIT_SHA: ${GIT_SHA:-$(git rev-parse HEAD)}
    ports:
      - "8080:8080"

IDE Run Buttons

VS Code

Add to .vscode/launch.json:

{
  "version": "0.2.0",
  "configurations": [
    {
      "name": "Run with SF Veritas",
      "type": "go",
      "request": "launch",
      "mode": "auto",
      "program": "${workspaceFolder}",
      "buildFlags": "-toolexec=sfveritas-instrument"
    }
  ]
}

GoLand / IntelliJ

1. Open Run/Debug Configurations
2. In the Go Build configuration, add to Go tool arguments:

   -toolexec=sfveritas-instrument

Is toolexec Production-Safe?

Yes — telemetry is sent on a non-blocking background transport (enqueue is ~5 ns), so it stays off the request path. The synchronous cost is per instrumented function call: roughly 8-10 microseconds per call (span create + close, dominated by Go stack introspection), measured on a benchmark VM — see BENCHMARKS.md. That is cheap for request handlers and service boundaries, but it is per call, so instrumenting every function on a very hot path adds up.

For hot paths, prefer manual tracing (StartSpan / TraceFunc) on the calls you care about, or enable sampling (SF_FUNCSPAN_ENABLE_SAMPLING=true, SF_FUNCSPAN_SAMPLE_RATE=0.1), rather than instrumenting everything via -toolexec.

Concern	Answer
Does toolexec modify my source files?	No. It operates on a copy during compilation. Your .go files are never changed.
Does the output binary differ?	Yes -- instrumented functions have extra span tracking code. This adds a small overhead per function call.
What's the overhead?	~8-10 µs per instrumented function call (span create/close); telemetry send is async (~5 ns to enqueue). Inbound middleware adds ~9 µs/request. See BENCHMARKS.md.
What if sfveritas-instrument isn't installed?	The build fails with exec: "sfveritas-instrument": executable file not found.

Framework Examples

net/http (stdlib)

package main

import (
    "fmt"
    "net/http"

    sfveritas "github.com/SailfishAI/sf-veritas-go"
)

func main() {
    sfveritas.SetupInterceptors(sfveritas.Options{
        APIKey:            "<see-api-key-from-your-account-settings-page>",
        ServiceIdentifier: "acme-corp/my-api/cmd/server/main.go", // Format: <org>/<repo>/<path-to-this-file>
    })
    defer sfveritas.Shutdown()

    mux := http.NewServeMux()
    mux.HandleFunc("/api/users", getUsers)
    mux.HandleFunc("/api/health", healthCheck)

    fmt.Println("Server starting on :8080")
    http.ListenAndServe(":8080", sfveritas.Middleware(mux))
}

func getUsers(w http.ResponseWriter, r *http.Request) {
    fmt.Println("Fetching users") // Appears in Sailfish
    w.Write([]byte(`{"users": []}`))
}

func healthCheck(w http.ResponseWriter, r *http.Request) {
    w.Write([]byte("ok"))
}

Run with instrumentation:

go run -toolexec="sfveritas-instrument" .

Gin

Gin needs two pieces of wiring:

1. Wrap the Gin engine with sfveritas.Middleware (the engine is an http.Handler) for inbound HTTP tracing and automatic log correlation.
2. Register veritasgin.Middleware() for exception capture. gin.Default() installs Gin's own Recovery(), which catches handler panics before the outer sfveritas.Middleware can see them -- so without this, panics and handled 5xx responses are never reported. The Gin middleware captures the panic (with its stack) and re-panics so Gin still returns the 500.

package main

import (
    "net/http"

    "github.com/gin-gonic/gin"
    sfveritas "github.com/SailfishAI/sf-veritas-go"
    veritasgin "github.com/SailfishAI/sf-veritas-go/gin"
)

func main() {
    sfveritas.SetupInterceptors(sfveritas.Options{
        APIKey:            "<see-api-key-from-your-account-settings-page>",
        ServiceIdentifier: "acme-corp/gin-api/cmd/server/main.go", // Format: <org>/<repo>/<path-to-this-file>
    })
    defer sfveritas.Shutdown()

    r := gin.Default()
    r.Use(veritasgin.Middleware()) // register BEFORE your routes -- captures panics + 5xx with stacks
    r.GET("/api/users", getUsers)

    // Wrap the engine for inbound tracing + automatic log correlation.
    http.ListenAndServe(":8080", sfveritas.Middleware(r))
}

func getUsers(c *gin.Context) {
    c.JSON(200, gin.H{"users": []string{}})
}

For the precise stack at a specific error site, report it directly: sfveritas.TransmitError(c.Request.Context(), err).

Echo

package main

import (
    "net/http"

    "github.com/labstack/echo/v4"
    sfveritas "github.com/SailfishAI/sf-veritas-go"
)

func main() {
    sfveritas.SetupInterceptors(sfveritas.Options{
        APIKey:            "<see-api-key-from-your-account-settings-page>",
        ServiceIdentifier: "acme-corp/echo-api/cmd/server/main.go", // Format: <org>/<repo>/<path-to-this-file>
    })
    defer sfveritas.Shutdown()

    e := echo.New()
    e.GET("/api/users", getUsers)

    // Start with SF Veritas middleware wrapping Echo
    http.ListenAndServe(":8080", sfveritas.Middleware(e))
}

func getUsers(c echo.Context) error {
    return c.JSON(200, map[string]interface{}{"users": []string{}})
}

Fiber

package main

import (
    "github.com/gofiber/fiber/v2"
    "github.com/gofiber/fiber/v2/middleware/adaptor"
    sfveritas "github.com/SailfishAI/sf-veritas-go"
)

func main() {
    sfveritas.SetupInterceptors(sfveritas.Options{
        APIKey:            "<see-api-key-from-your-account-settings-page>",
        ServiceIdentifier: "acme-corp/fiber-api/cmd/server/main.go", // Format: <org>/<repo>/<path-to-this-file>
    })
    defer sfveritas.Shutdown()

    app := fiber.New()

    // Use the adaptor to wrap SF Veritas middleware
    app.Use(adaptor.HTTPMiddleware(func(next http.Handler) http.Handler {
        return sfveritas.Middleware(next)
    }))

    app.Get("/api/users", func(c *fiber.Ctx) error {
        return c.JSON(fiber.Map{"users": []string{}})
    })

    app.Listen(":8080")
}

Manual Function Tracing

If you prefer not to use -toolexec, or need fine-grained control over which functions are traced, use the manual API:

Basic Span

func ProcessOrder(ctx context.Context, orderID string) error {
    span := sfveritas.StartSpan(ctx, "ProcessOrder")
    defer func() { span.End(nil) }()

    // Use span.Context() for child operations
    result, err := validateOrder(span.Context(), orderID)
    if err != nil {
        sfveritas.TransmitError(span.Context(), err)
        return err
    }
    span.End(result)
    return nil
}

Span with Arguments

func ProcessOrder(ctx context.Context, orderID string, amount float64) error {
    span := sfveritas.StartSpanWithArgs(ctx, "ProcessOrder", map[string]interface{}{
        "orderID": orderID,
        "amount":  amount,
    })
    defer func() { span.End(nil) }()

    // ... function body ...
    return nil
}

TraceFunc Helper

result, err := sfveritas.TraceFunc(ctx, "ProcessOrder", func(ctx context.Context) (string, error) {
    return doWork(ctx)
})

User Attribution

A backend request has no inherent end user, but you often know which users a request is acting on behalf of. The user-attribution API records which users are active over the lifetime of a request, so telemetry can be attributed to them in the Sailfish dashboard.

Attribution is explicit and request-scoped: each call binds to the current request's trace (served through sfveritas.Middleware). A call made outside a request is a no-op.

func GetArticles(c *gin.Context) {
    ctx := c.Request.Context()

    // Mark a user active for this request; pairs with a deferred close.
    defer sfveritas.ActiveUserScope(ctx, "customer-42")()

    // ... handle the request ...
}

Lower-level API:

sfveritas.AddActiveUser(ctx, "customer-42")      // open an interval (idempotent)
sfveritas.RemoveActiveUser(ctx, "customer-42")   // close it

sfveritas.SetActiveUsers(ctx, []string{"a", "b"}) // replace the active set:
                                                   // adds new users, ends removed ones
sfveritas.ClearActiveUsers(ctx)                    // end all open intervals

• Events are debounced (~50 ms) and sent as one batch, so rapid changes are cheap.
• AddActiveUser is idempotent; SetActiveUsers diffs against the current set.
• Call sfveritas.ClearActiveUsers(ctx) (or let the ActiveUserScope closer run) at the end of a unit of work so open intervals are closed. Before a short-lived process exits, call sfveritas.FlushUserAttribution() to flush any buffered events.

Requires the request trace

Attribution binds to the inbound request's trace, which is established by sfveritas.Middleware. Telemetry emitted from a goroutine you spawn won't carry the trace unless you pass the request context — see Logs are linked to the request automatically.

Debug Mode

Sailfish supports a server-triggered debug mode: from the dashboard, full function-span capture (arguments, return values, timing) can be turned on for your service for a time- and budget-bounded window — useful for live backend services with no HTTP surface (queue consumers, cron jobs, workers).

The SDK holds a lightweight WebSocket "uplink" to Sailfish. When debug mode is enabled for your service, capture rules are pushed to every running instance and applied to matching function spans; capture stops automatically when the TTL expires, a span/byte budget is exhausted, or it's cancelled. No code changes needed — the uplink starts with SetupInterceptors and stops on Shutdown.

Go specifics

Go has no runtime profiler, so debug mode can only elevate capture on functions that are already instrumented — manually (StartSpan / TraceFunc) or built with the -toolexec instrumenter. It cannot instrument arbitrary functions at runtime. For the most useful debug mode, build your service with -toolexec (spans exist but stay quiet) so a debug session can turn specific functions up on demand.

The uplink is on by default. Disable it with SF_UPLINK_ENABLE=false (e.g. FaaS/edge runtimes); point it at a non-default backend with SF_UPLINK_URL (otherwise it derives wss://…/ws/notify/ from your GraphQL endpoint).

Environment Variables

Variable	Default	Description
SF_API_KEY	--	API key (alternative to Options.APIKey)
SF_SERVICE_IDENTIFIER	--	Service name (alternative to Options.ServiceIdentifier)
SF_DEBUG	false	Enable debug logging to stderr
SF_UPLINK_ENABLE	true	WebSocket uplink for server-triggered Debug Mode; set false to disable
SF_UPLINK_URL	(derived)	Override the uplink URL (default: wss://…/ws/notify/ derived from the GraphQL endpoint)
SF_FUNCSPAN_SAMPLE_RATE	1.0	Function span sampling rate (0.0 to 1.0)
SF_FUNCSPAN_ENABLE_SAMPLING	false	Enable span sampling
SF_LOG_IGNORE_REGEX	--	Regex pattern to suppress logs from telemetry
SF_NETWORKHOP_CAPTURE_REQUEST_BODY	false	Capture HTTP request bodies
SF_NETWORKHOP_CAPTURE_RESPONSE_BODY	false	Capture HTTP response bodies
SF_NETWORKHOP_REQUEST_LIMIT_MB	1	Max request body capture size in MB
SF_NETWORKHOP_RESPONSE_LIMIT_MB	1	Max response body capture size in MB
SF_DISABLE_PRINT_CAPTURE	false	Disable stdout pipe capture
SF_FUNCSPAN_ARG_LIMIT_MB	1	Max argument capture size in MB
SF_FUNCSPAN_RETURN_LIMIT_MB	1	Max return value capture size in MB
SF_EXCLUDED_DOMAINS	--	Comma-separated domains to skip outbound tracing
SF_DISABLE_INBOUND_NETWORK_TRACING_ON_ROUTES	--	Comma-separated glob patterns for routes to skip
SF_INSTRUMENT_DEBUG	false	Enable debug output from the toolexec wrapper

Configuration File

Create a .sailfish file in your project root for per-file and per-function span configuration:

{
  "files": {
    "*.go": {
      "capture_arguments": true,
      "capture_return_value": true,
      "sample_rate": 1.0
    },
    "*_test.go": {
      "sample_rate": 0.0
    }
  },
  "functions": {
    "ProcessOrder": {
      "capture_arguments": true,
      "capture_return_value": true,
      "arg_limit_mb": 2,
      "return_limit_mb": 2
    },
    "healthCheck": {
      "sample_rate": 0.0
    }
  }
}

File Configuration

Option	Type	Description
capture_arguments	bool	Capture function arguments
capture_return_value	bool	Capture return values
arg_limit_mb	int	Max argument size in MB
return_limit_mb	int	Max return value size in MB
sample_rate	float	Sampling rate (0.0 to 1.0)

Function Configuration

Same options as file configuration but applied to specific function names. Function-level config takes priority over file-level config.

Verifying the Setup

1. Deploy your application with the Sailfish SDK configured
2. Trigger some HTTP requests
3. Open the Sailfish dashboard -- you should see telemetry appearing

Debug Mode

Enable debug output to verify instrumentation is working:

SF_DEBUG=true go run -toolexec="sfveritas-instrument" .

You'll see output like:

[sfveritas] Initializing Go Go BACKEND collector v0.1.5
[sfveritas] Endpoint: https://api-service.sailfish.ai/graphql/
[sfveritas] Setup complete. Interceptors active.

Troubleshooting

No logs appearing

1. Check the API key: Ensure APIKey is set to your Enterprise API key
2. Check the service identifier: Ensure ServiceIdentifier uses the <org>/<repo>/<path> format
3. Check terminal output: Look for [sfveritas] initialization messages

Logs appear but aren't nested under their request

1. Ensure requests are served through sfveritas.Middleware -- log correlation is established by the inbound middleware. With Gin, also wrap the engine (sfveritas.Middleware(r)), not just r.Run(). See the Gin example.
2. Logs from goroutines you spawn won't auto-correlate -- pass the request context with slog.InfoContext(r.Context(), ...). See Logs are linked to the request automatically.

sfveritas-instrument: command not found

1. Ensure $GOPATH/bin is in your $PATH:

   export PATH=$PATH:$(go env GOPATH)/bin

2. Re-install:

   go install github.com/SailfishAI/sf-veritas-go/cmd/sfveritas-instrument@latest

Connection errors

1. Ensure your deployment can reach https://api-service.sailfish.ai
2. Check that outbound HTTPS (port 443) is not blocked by a firewall or network policy

Build errors with toolexec

1. Verify Go version: Requires Go 1.22+
2. Check the tool is built: sfveritas-instrument --help should not error
3. Enable debug: Set SF_INSTRUMENT_DEBUG=true to see which files are being instrumented

Multi-Service Setup

When running multiple Go services, give each a unique ServiceIdentifier following the <org>/<repo>/<path> format:

// user-service/main.go
sfveritas.SetupInterceptors(sfveritas.Options{
    APIKey:            "<see-api-key-from-your-account-settings-page>",
    ServiceIdentifier: "acme-corp/user-service/cmd/server/main.go", // Format: <org>/<repo>/<path-to-this-file>
})

// order-service/main.go
sfveritas.SetupInterceptors(sfveritas.Options{
    APIKey:            "<ApiKey />",
    ServiceIdentifier: "acme-corp/order-service/cmd/server/main.go", // Format: <org>/<repo>/<path-to-this-file>
})

Use the service filter in the Sailfish dashboard to switch between services.

Next Steps

• Set up your frontend application (optional)
• Check the Sailfish dashboard to verify telemetry is flowing

---

Local Development

Looking to set up SF Veritas for local development with the Desktop App? See the Desktop App Go guide.