Skip to main content

Backend Setup: Go

This guide walks you through instrumenting a Go application with the SF Veritas SDK. Go instrumentation captures logs, print statements, exceptions/panics, HTTP request/response telemetry, and function execution spans.

Installation

Install the SF Veritas Go module:

go get github.com/SailfishAI/sf-veritas-go@latest

Basic Setup

Add the following to your application's main() function. Use an environment variable check so instrumentation only runs during local development:

package main

import (
"net/http"
"os"

sfveritas "github.com/SailfishAI/sf-veritas-go"
)

func main() {
// Initialize SF Veritas ONLY in development mode
if os.Getenv("SF_DEV") == "true" {
sfveritas.SetupInterceptors(sfveritas.Options{
APIKey: "sf-veritas-local",
GraphQLEndpoint: "http://localhost:6776/graphql/",
ServiceIdentifier: "my-go-service",
ServiceVersion: "1.0.0",
})
defer sfveritas.Shutdown()
}

mux := http.NewServeMux()
mux.HandleFunc("/api/users", handleUsers)

// Wrap your handler with SF Veritas middleware for HTTP tracing
http.ListenAndServe(":8080", sfveritas.Middleware(mux))
}

Configuration Options

OptionTypeRequiredDescription
APIKeystringYesUse "sf-veritas-local" for local development
GraphQLEndpointstringYesURL of the local collector (default port 6776)
ServiceIdentifierstringYesUnique name for your service
ServiceVersionstringNoVersion of your service

What Gets Captured Automatically

Once SetupInterceptors is called, these are captured with zero additional code:

  • Structured logs — All slog.Info(), slog.Warn(), slog.Error() calls
  • Print statements — All fmt.Println(), fmt.Printf(), log.Println() output
  • Inbound HTTP requests — Timing, status codes, headers (via sfveritas.Middleware)
  • Outbound HTTP requests — All http.Get(), http.Post(), client.Do() calls via http.DefaultTransport
  • Panics — Recovered panics with full stack traces (via sfveritas.Middleware; for Gin see the Gin example below)
  • Exceptions — Manually reported errors with sfveritas.TransmitError()

Logs are linked to the request automatically

A plain slog.Info("...") or fmt.Println("...") made inside a request handler is correlated to that inbound request automatically — you do not need to thread context.Context through your call stack or switch to slog.InfoContext. As long as the request is served through sfveritas.Middleware, the SDK associates telemetry emitted on the handler's goroutine with that request, so your logs appear nested under the right request in the Desktop App Console.

Logs from goroutines you spawn

Go has no goroutine-local storage, so telemetry emitted from a new goroutine you start inside a handler (go func() { slog.Info(...) }()) can't be auto-linked to the request. For those, pass the request context explicitly:

go func() {
slog.InfoContext(r.Context(), "background work done") // links to the request
}()

Automatic Function Instrumentation (toolexec)

For full function-level tracing in the Flamechart — including automatic argument capture, return values, and local variable capture on panic — use the sfveritas-instrument compile-time tool.

How It Works

Go's -toolexec flag lets you intercept the compiler. sfveritas-instrument parses your source code's AST (Abstract Syntax Tree) at compile time and injects lightweight instrumentation into every function:

  1. Function entry — Records the function name, file, line, and captures all argument values
  2. Panic recovery — A defer block captures all local variable values if a panic occurs
  3. Function exit — Records timing and ends the span

The transformation happens at compile time, not runtime. Your source files are never modified on disk.

Install the Tool

go install github.com/SailfishAI/sf-veritas-go/cmd/sfveritas-instrument@latest

Verify it's installed:

which sfveritas-instrument
# Should print: $GOPATH/bin/sfveritas-instrument

Usage with go build

go build -toolexec="sfveritas-instrument" -o myapp ./...

Usage with go run

go run -toolexec="sfveritas-instrument" .

Usage with go test

go test -toolexec="sfveritas-instrument" ./...

Integration Patterns

Makefile

TOOLEXEC := -toolexec="sfveritas-instrument"

.PHONY: dev prod test

# Development build with instrumentation
dev:
SF_DEV=true go run $(TOOLEXEC) .

# Production build without instrumentation (normal go build)
prod:
go build -o myapp ./...

# Tests with instrumentation
test:
go test $(TOOLEXEC) ./...

Run with:

make dev    # Development with full tracing
make prod # Production — no instrumentation, no overhead

Direct go run

# Development — with instrumentation
SF_DEV=true go run -toolexec="sfveritas-instrument" .

# Production — no toolexec flag, no overhead
go run .

Air Hot-Reload

Air is a popular live-reload tool for Go. Add -toolexec to the build command in .air.toml:

# .air.toml
root = "."
tmp_dir = "tmp"

[build]
cmd = "go build -toolexec='sfveritas-instrument' -o ./tmp/main ."
bin = "tmp/main"
delay = 1000
exclude_dir = ["assets", "tmp", "vendor"]
exclude_regex = ["_test\\.go"]
include_ext = ["go", "tpl", "tmpl", "html"]
kill_delay = "0s"

[misc]
clean_on_exit = true

Run with:

SF_DEV=true air
tip

Only add -toolexec in your development .air.toml. For CI/CD or production, use a separate config or remove the flag.

Docker Compose

For local development with Docker Compose, install the tool in the build stage and pass -toolexec to the build command:

# Dockerfile.dev
FROM golang:1.22-alpine

WORKDIR /app

# Install the instrumentation tool
RUN go install github.com/SailfishAI/sf-veritas-go/cmd/sfveritas-instrument@latest

# Copy module files first for caching
COPY go.mod go.sum ./
RUN go mod download

COPY . .

# Build with instrumentation
RUN go build -toolexec="sfveritas-instrument" -o /app/server .

# Set env vars for SF Veritas
ENV SF_DEV=true

CMD ["/app/server"]
# docker-compose.yml
services:
api:
build:
context: .
dockerfile: Dockerfile.dev
ports:
- "8080:8080"
environment:
- SF_DEV=true
# Point to host machine's SF Veritas Desktop App
- SF_GRAPHQL_ENDPOINT=http://host.docker.internal:6776/graphql/
Docker networking

Use host.docker.internal to reach the SF Veritas Desktop App running on your host machine. This works on Docker Desktop for Mac and Windows. On Linux, add extra_hosts: ["host.docker.internal:host-gateway"] to your service.

IDE Run Buttons

VS Code

Add to .vscode/launch.json:

{
"version": "0.2.0",
"configurations": [
{
"name": "Run with SF Veritas",
"type": "go",
"request": "launch",
"mode": "auto",
"program": "${workspaceFolder}",
"env": {
"SF_DEV": "true"
},
"buildFlags": "-toolexec=sfveritas-instrument"
}
]
}

GoLand / IntelliJ

  1. Open Run/Debug Configurations
  2. In the Go Build configuration, add to Go tool arguments:
    -toolexec=sfveritas-instrument
  3. In Environment, add:
    SF_DEV=true

Is toolexec Production-Safe?

Yes, but you don't need it in production. Here's the breakdown:

ConcernAnswer
Does toolexec modify my source files?No. It operates on a copy during compilation. Your .go files are never changed.
Does the output binary differ?Yes — instrumented functions have extra span tracking code. This adds a small overhead per function call.
Can I ship an instrumented binary?Technically yes, but don't. The instrumentation sends telemetry to your local desktop app. In production there's nothing listening.
What's the overhead?~8-10 µs per instrumented function call (span create/close); telemetry send is async (~5 ns to enqueue). Negligible for development, unnecessary for production. See BENCHMARKS.md.
What if sfveritas-instrument isn't installed?The build fails with exec: "sfveritas-instrument": executable file not found. This is a feature — it prevents accidental instrumented builds in CI.

Recommended approach: Use -toolexec only in development. Your production builds use normal go build with no flags — zero overhead, zero dependencies.

# This is the pattern we recommend
dev:
SF_DEV=true go run -toolexec="sfveritas-instrument" .

prod:
go build -o myapp ./...

Framework Examples

net/http (stdlib)

package main

import (
"fmt"
"net/http"
"os"

sfveritas "github.com/SailfishAI/sf-veritas-go"
)

func main() {
if os.Getenv("SF_DEV") == "true" {
sfveritas.SetupInterceptors(sfveritas.Options{
APIKey: "sf-veritas-local",
GraphQLEndpoint: "http://localhost:6776/graphql/",
ServiceIdentifier: "my-api",
})
defer sfveritas.Shutdown()
}

mux := http.NewServeMux()
mux.HandleFunc("/api/users", getUsers)
mux.HandleFunc("/api/health", healthCheck)

fmt.Println("Server starting on :8080")
http.ListenAndServe(":8080", sfveritas.Middleware(mux))
}

func getUsers(w http.ResponseWriter, r *http.Request) {
fmt.Println("Fetching users") // Appears in SF Veritas Console
w.Write([]byte(`{"users": []}`))
}

func healthCheck(w http.ResponseWriter, r *http.Request) {
w.Write([]byte("ok"))
}

Run with instrumentation:

SF_DEV=true go run -toolexec="sfveritas-instrument" .

Gin

Gin needs two pieces of wiring:

  1. Wrap the Gin engine with sfveritas.Middleware (the engine is an http.Handler) for inbound HTTP tracing and automatic log correlation.
  2. Register veritasgin.Middleware() for exception capture. gin.Default() installs Gin's own Recovery(), which catches handler panics before the outer sfveritas.Middleware can see them — so without this, panics and handled 5xx responses are never reported. The Gin middleware captures the panic (with its stack) and re-panics so Gin still returns the 500. It is a no-op until SetupInterceptors runs, so it is safe to register unconditionally.
package main

import (
"net/http"
"os"

"github.com/gin-gonic/gin"
sfveritas "github.com/SailfishAI/sf-veritas-go"
veritasgin "github.com/SailfishAI/sf-veritas-go/gin"
)

func main() {
dev := os.Getenv("SF_DEV") == "true"
if dev {
sfveritas.SetupInterceptors(sfveritas.Options{
APIKey: "sf-veritas-local",
GraphQLEndpoint: "http://localhost:6776/graphql/",
ServiceIdentifier: "gin-api",
})
defer sfveritas.Shutdown()
}

r := gin.Default()
r.Use(veritasgin.Middleware()) // register BEFORE your routes — captures panics + 5xx with stacks
r.GET("/api/users", getUsers)

if dev {
// Wrap the engine for inbound tracing + automatic log correlation.
http.ListenAndServe(":8080", sfveritas.Middleware(r))
} else {
r.Run(":8080")
}
}

func getUsers(c *gin.Context) {
c.JSON(200, gin.H{"users": []string{}})
}

For the precise stack at a specific error site, report it directly: sfveritas.TransmitError(c.Request.Context(), err).

Echo

package main

import (
"net/http"
"os"

"github.com/labstack/echo/v4"
sfveritas "github.com/SailfishAI/sf-veritas-go"
)

func main() {
if os.Getenv("SF_DEV") == "true" {
sfveritas.SetupInterceptors(sfveritas.Options{
APIKey: "sf-veritas-local",
GraphQLEndpoint: "http://localhost:6776/graphql/",
ServiceIdentifier: "echo-api",
})
defer sfveritas.Shutdown()
}

e := echo.New()
e.GET("/api/users", getUsers)

// Start with SF Veritas middleware wrapping Echo
http.ListenAndServe(":8080", sfveritas.Middleware(e))
}

func getUsers(c echo.Context) error {
return c.JSON(200, map[string]interface{}{"users": []string{}})
}

Fiber

package main

import (
"os"

"github.com/gofiber/fiber/v2"
"github.com/gofiber/fiber/v2/middleware/adaptor"
sfveritas "github.com/SailfishAI/sf-veritas-go"
)

func main() {
if os.Getenv("SF_DEV") == "true" {
sfveritas.SetupInterceptors(sfveritas.Options{
APIKey: "sf-veritas-local",
GraphQLEndpoint: "http://localhost:6776/graphql/",
ServiceIdentifier: "fiber-api",
})
defer sfveritas.Shutdown()
}

app := fiber.New()

// Use the adaptor to wrap SF Veritas middleware
app.Use(adaptor.HTTPMiddleware(func(next http.Handler) http.Handler {
return sfveritas.Middleware(next)
}))

app.Get("/api/users", func(c *fiber.Ctx) error {
return c.JSON(fiber.Map{"users": []string{}})
})

app.Listen(":8080")
}

Manual Function Tracing

If you prefer not to use -toolexec, or need fine-grained control over which functions are traced, use the manual API:

Basic Span

func ProcessOrder(ctx context.Context, orderID string) error {
span := sfveritas.StartSpan(ctx, "ProcessOrder")
defer func() { span.End(nil) }()

// Use span.Context() for child operations
result, err := validateOrder(span.Context(), orderID)
if err != nil {
sfveritas.TransmitError(span.Context(), err)
return err
}
span.End(result)
return nil
}

Span with Arguments

func ProcessOrder(ctx context.Context, orderID string, amount float64) error {
span := sfveritas.StartSpanWithArgs(ctx, "ProcessOrder", map[string]interface{}{
"orderID": orderID,
"amount": amount,
})
defer func() { span.End(nil) }()

// ... function body ...
return nil
}

TraceFunc Helper

result, err := sfveritas.TraceFunc(ctx, "ProcessOrder", func(ctx context.Context) (string, error) {
return doWork(ctx)
})

User Attribution

A backend request has no inherent end user, but you often know which users a request is acting on behalf of. The user-attribution API records which users are active over the lifetime of a request, so telemetry can be attributed to them in the SF Veritas Desktop App.

Attribution is explicit and request-scoped: each call binds to the current request's trace (served through sfveritas.Middleware). A call made outside a request is a no-op.

func GetArticles(c *gin.Context) {
ctx := c.Request.Context()

// Mark a user active for this request; pairs with a deferred close.
defer sfveritas.ActiveUserScope(ctx, "customer-42")()

// ... handle the request ...
}

Lower-level API:

sfveritas.AddActiveUser(ctx, "customer-42")      // open an interval (idempotent)
sfveritas.RemoveActiveUser(ctx, "customer-42") // close it

sfveritas.SetActiveUsers(ctx, []string{"a", "b"}) // replace the active set:
// adds new users, ends removed ones
sfveritas.ClearActiveUsers(ctx) // end all open intervals
  • Events are debounced (~50 ms) and sent as one batch, so rapid changes are cheap.
  • AddActiveUser is idempotent; SetActiveUsers diffs against the current set.
  • Call sfveritas.ClearActiveUsers(ctx) (or let the ActiveUserScope closer run) at the end of a unit of work so open intervals are closed. Before a short-lived process exits, call sfveritas.FlushUserAttribution() to flush any buffered events.
Requires the request trace

Attribution binds to the inbound request's trace, which is established by sfveritas.Middleware. Telemetry emitted from a goroutine you spawn won't carry the trace unless you pass the request context — see Logs are linked to the request automatically.

Debug Mode

Sailfish supports a server-triggered debug mode: full function-span capture (arguments, return values, timing) can be turned on for your service for a time- and budget-bounded window — useful for live backend services with no HTTP surface (queue consumers, cron jobs, workers).

The SDK holds a lightweight WebSocket "uplink" to Sailfish. When debug mode is enabled for your service, capture rules are pushed to every running instance and applied to matching function spans; capture stops automatically when the TTL expires, a span/byte budget is exhausted, or it's cancelled. No code changes needed — the uplink starts with SetupInterceptors and stops on Shutdown.

Go specifics

Go has no runtime profiler, so debug mode can only elevate capture on functions that are already instrumented — manually (StartSpan / TraceFunc) or built with the -toolexec instrumenter. It cannot instrument arbitrary functions at runtime. For the most useful debug mode, build your service with -toolexec so a debug session can turn specific functions up on demand.

The uplink is on by default. Disable it with SF_UPLINK_ENABLE=false; point it at a non-default backend with SF_UPLINK_URL (otherwise it derives ws(s)://…/ws/notify/ from GraphQLEndpoint).

Environment Variables

VariableDefaultDescription
SF_DEVSet to "true" to enable SF Veritas
SF_API_KEYAPI key (alternative to Options.APIKey)
SF_GRAPHQL_ENDPOINThttp://localhost:6776/graphql/Collector endpoint
SF_SERVICE_IDENTIFIERService name (alternative to Options.ServiceIdentifier)
SF_DEBUGfalseEnable debug logging to stderr
SF_UPLINK_ENABLEtrueWebSocket uplink for server-triggered Debug Mode; set false to disable
SF_UPLINK_URL(derived)Override the uplink URL (default derived from GraphQLEndpoint)
SF_FUNCSPAN_SAMPLE_RATE1.0Function span sampling rate (0.0 to 1.0)
SF_FUNCSPAN_ENABLE_SAMPLINGfalseEnable span sampling
SF_LOG_IGNORE_REGEXRegex pattern to suppress logs from telemetry
SF_NETWORKHOP_CAPTURE_REQUEST_BODYfalseCapture HTTP request bodies
SF_NETWORKHOP_CAPTURE_RESPONSE_BODYfalseCapture HTTP response bodies
SF_NETWORKHOP_REQUEST_LIMIT_MB1Max request body capture size in MB
SF_NETWORKHOP_RESPONSE_LIMIT_MB1Max response body capture size in MB
SF_DISABLE_PRINT_CAPTUREfalseDisable stdout pipe capture
SF_FUNCSPAN_ARG_LIMIT_MB1Max argument capture size in MB
SF_FUNCSPAN_RETURN_LIMIT_MB1Max return value capture size in MB
SF_EXCLUDED_DOMAINSComma-separated domains to skip outbound tracing
SF_DISABLE_INBOUND_NETWORK_TRACING_ON_ROUTESComma-separated glob patterns for routes to skip
SF_INSTRUMENT_DEBUGfalseEnable debug output from the toolexec wrapper

Configuration File

Create a .sailfish file in your project root for per-file and per-function span configuration:

{
"files": {
"*.go": {
"capture_arguments": true,
"capture_return_value": true,
"sample_rate": 1.0
},
"*_test.go": {
"sample_rate": 0.0
}
},
"functions": {
"ProcessOrder": {
"capture_arguments": true,
"capture_return_value": true,
"arg_limit_mb": 2,
"return_limit_mb": 2
},
"healthCheck": {
"sample_rate": 0.0
}
}
}

File Configuration

OptionTypeDescription
capture_argumentsboolCapture function arguments
capture_return_valueboolCapture return values
arg_limit_mbintMax argument size in MB
return_limit_mbintMax return value size in MB
sample_ratefloatSampling rate (0.0 to 1.0)

Function Configuration

Same options as file configuration but applied to specific function names. Function-level config takes priority over file-level config.

Verifying the Setup

  1. Start your application:
    SF_DEV=true go run -toolexec="sfveritas-instrument" .
  2. Open the SF Veritas Desktop App
  3. Open the Console panel — you should see logs and print statements
  4. Trigger some HTTP requests — you should see them in the Network panel
  5. Open the Flamechart — you should see function execution traces

Debug Mode

Enable debug output to verify instrumentation is working:

SF_DEV=true SF_DEBUG=true go run -toolexec="sfveritas-instrument" .

You'll see output like:

[sfveritas] Initializing Go Go BACKEND collector v0.1.5
[sfveritas] Endpoint: http://localhost:6776/graphql/
[sfveritas] Setup complete. Interceptors active.
[sfveritas] Function span: ProcessOrder (abc-123) duration=1234ns
[sfveritas] Outbound request: GET https://api.example.com/data → 200 (45ms)

Troubleshooting

No logs appearing

  1. Check the desktop app: Ensure the SF Veritas Desktop App is running
  2. Verify the endpoint: Ensure GraphQLEndpoint matches your server port
  3. Check SF_DEV: Must be set to "true"
  4. Check terminal output: Look for [sfveritas] initialization messages

Logs appear but aren't nested under their request

  1. Ensure requests are served through sfveritas.Middleware — log correlation is established by the inbound middleware. With Gin, also wrap the engine (sfveritas.Middleware(r)), not just r.Run(). See the Gin example.
  2. Logs from goroutines you spawn won't auto-correlate — pass the request context with slog.InfoContext(r.Context(), ...). See Logs are linked to the request automatically.

sfveritas-instrument: command not found

  1. Ensure $GOPATH/bin is in your $PATH:
    export PATH=$PATH:$(go env GOPATH)/bin
  2. Re-install:
    go install github.com/SailfishAI/sf-veritas-go/cmd/sfveritas-instrument@latest

Connection refused errors

  1. Verify the SF Veritas Desktop App is installed and running
  2. Check that the local server is running (look for server status in the Desktop App)
  3. Ensure port 6776 is not blocked by a firewall
  4. In Docker: use host.docker.internal instead of localhost

Build errors with toolexec

  1. Verify Go version: Requires Go 1.22+
  2. Check the tool is built: sfveritas-instrument --help should not error
  3. Enable debug: Set SF_INSTRUMENT_DEBUG=true to see which files are being instrumented

Multi-Service Setup

When running multiple Go services locally, give each a unique ServiceIdentifier:

// user-service/main.go
sfveritas.SetupInterceptors(sfveritas.Options{
APIKey: "sf-veritas-local",
GraphQLEndpoint: "http://localhost:6776/graphql/",
ServiceIdentifier: "user-service",
})

// order-service/main.go
sfveritas.SetupInterceptors(sfveritas.Options{
APIKey: "sf-veritas-local",
GraphQLEndpoint: "http://localhost:6776/graphql/",
ServiceIdentifier: "order-service",
})

Use the service filter in the Console to switch between services.

Enterprise Setup

Enterprise Only

This section applies to users with a Sailfish Enterprise account. If you're using the Desktop App for local development only, the basic setup above is all you need.

Enterprise users need two configurations — one for local development (Desktop App) and one for staging/production (Sailfish cloud). The SDK should detect which environment it's running in and configure itself accordingly.

Dual-Mode Configuration

package main

import (
"os"

sfveritas "github.com/SailfishAI/sf-veritas-go"
)

func main() {
if os.Getenv("SF_DEV") == "true" {
// Local development — send to Desktop App
sfveritas.SetupInterceptors(sfveritas.Options{
APIKey: "sf-veritas-local",
GraphQLEndpoint: "http://localhost:6776/graphql/",
ServiceIdentifier: "my-go-service",
ServiceVersion: "1.0.0",
})
defer sfveritas.Shutdown()
} else if apiKey := os.Getenv("SAILFISH_API_KEY"); apiKey != "" {
// Staging/Production — send to Sailfish cloud
// Do NOT set GraphQLEndpoint — the SDK defaults to the cloud endpoint
sfveritas.SetupInterceptors(sfveritas.Options{
APIKey: apiKey,
ServiceIdentifier: "my-go-service",
ServiceVersion: "1.0.0",
})
defer sfveritas.Shutdown()
}

// Start your application...
}

Environment Variables for Enterprise

Set these in your staging/production deployment:

# Required — your company's Enterprise API key
SAILFISH_API_KEY=your-enterprise-api-key

# Required — the git commit SHA for this build
GIT_SHA=$(git rev-parse HEAD)

# Optional — override the endpoint (only for local dev, do NOT set in production)
# SAILFISH_GRAPHQL_ENDPOINT=http://localhost:6776/graphql/

GIT_SHA

The GIT_SHA environment variable allows Sailfish to correlate telemetry with specific commits. The Go SDK reads it automatically from os.Getenv("GIT_SHA").

Set it in your build pipeline:

# Docker
docker build --build-arg GIT_SHA=$(git rev-parse HEAD) .

# In Dockerfile
ARG GIT_SHA
ENV GIT_SHA=$GIT_SHA

# Makefile
dev:
SF_DEV=true go run -toolexec="sfveritas-instrument" .

prod:
GIT_SHA=$$(git rev-parse HEAD) go build -o myapp ./...

URL Override

You can override the GraphQL endpoint in two ways:

  1. Environment variable: SAILFISH_GRAPHQL_ENDPOINT
  2. Constructor argument: GraphQLEndpoint in sfveritas.Options{}
EnvironmentEndpointHow
Local devhttp://localhost:6776/graphql/Set GraphQLEndpoint or SAILFISH_GRAPHQL_ENDPOINT
Staging/ProductionDefault (cloud)Do not set either — the SDK uses https://api-service.sailfishqa.com/graphql/ automatically

Next Steps