Backend Setup: Go
This guide walks you through instrumenting a Go application with the SF Veritas SDK. Go instrumentation captures logs, print statements, exceptions/panics, HTTP request/response telemetry, and function execution spans.
Installation
Install the SF Veritas Go module:
go get github.com/SailfishAI/sf-veritas-go@latest
Basic Setup
Add the following to your application's main() function. Use an environment variable check so instrumentation only runs during local development:
package main
import (
"net/http"
"os"
sfveritas "github.com/SailfishAI/sf-veritas-go"
)
func main() {
// Initialize SF Veritas ONLY in development mode
if os.Getenv("SF_DEV") == "true" {
sfveritas.SetupInterceptors(sfveritas.Options{
APIKey: "sf-veritas-local",
GraphQLEndpoint: "http://localhost:6776/graphql/",
ServiceIdentifier: "my-go-service",
ServiceVersion: "1.0.0",
})
defer sfveritas.Shutdown()
}
mux := http.NewServeMux()
mux.HandleFunc("/api/users", handleUsers)
// Wrap your handler with SF Veritas middleware for HTTP tracing
http.ListenAndServe(":8080", sfveritas.Middleware(mux))
}
Configuration Options
| Option | Type | Required | Description |
|---|---|---|---|
APIKey | string | Yes | Use "sf-veritas-local" for local development |
GraphQLEndpoint | string | Yes | URL of the local collector (default port 6776) |
ServiceIdentifier | string | Yes | Unique name for your service |
ServiceVersion | string | No | Version of your service |
What Gets Captured Automatically
Once SetupInterceptors is called, these are captured with zero additional code:
- Structured logs — All
slog.Info(),slog.Warn(),slog.Error()calls - Print statements — All
fmt.Println(),fmt.Printf(),log.Println()output - Inbound HTTP requests — Timing, status codes, headers (via
sfveritas.Middleware) - Outbound HTTP requests — All
http.Get(),http.Post(),client.Do()calls viahttp.DefaultTransport - Panics — Recovered panics with full stack traces (via
sfveritas.Middleware; for Gin see the Gin example below) - Exceptions — Manually reported errors with
sfveritas.TransmitError()
Logs are linked to the request automatically
A plain slog.Info("...") or fmt.Println("...") made inside a request handler
is correlated to that inbound request automatically — you do not need to thread
context.Context through your call stack or switch to slog.InfoContext. As long as
the request is served through sfveritas.Middleware, the SDK associates telemetry
emitted on the handler's goroutine with that request, so your logs appear nested under
the right request in the Desktop App Console.
Go has no goroutine-local storage, so telemetry emitted from a new goroutine you start
inside a handler (go func() { slog.Info(...) }()) can't be auto-linked to the request.
For those, pass the request context explicitly:
go func() {
slog.InfoContext(r.Context(), "background work done") // links to the request
}()
Automatic Function Instrumentation (toolexec)
For full function-level tracing in the Flamechart — including automatic argument capture, return values, and local variable capture on panic — use the sfveritas-instrument compile-time tool.
How It Works
Go's -toolexec flag lets you intercept the compiler. sfveritas-instrument parses your source code's AST (Abstract Syntax Tree) at compile time and injects lightweight instrumentation into every function:
- Function entry — Records the function name, file, line, and captures all argument values
- Panic recovery — A
deferblock captures all local variable values if a panic occurs - Function exit — Records timing and ends the span
The transformation happens at compile time, not runtime. Your source files are never modified on disk.
Install the Tool
go install github.com/SailfishAI/sf-veritas-go/cmd/sfveritas-instrument@latest
Verify it's installed:
which sfveritas-instrument
# Should print: $GOPATH/bin/sfveritas-instrument
Usage with go build
go build -toolexec="sfveritas-instrument" -o myapp ./...
Usage with go run
go run -toolexec="sfveritas-instrument" .
Usage with go test
go test -toolexec="sfveritas-instrument" ./...
Integration Patterns
Makefile
TOOLEXEC := -toolexec="sfveritas-instrument"
.PHONY: dev prod test
# Development build with instrumentation
dev:
SF_DEV=true go run $(TOOLEXEC) .
# Production build without instrumentation (normal go build)
prod:
go build -o myapp ./...
# Tests with instrumentation
test:
go test $(TOOLEXEC) ./...
Run with:
make dev # Development with full tracing
make prod # Production — no instrumentation, no overhead
Direct go run
# Development — with instrumentation
SF_DEV=true go run -toolexec="sfveritas-instrument" .
# Production — no toolexec flag, no overhead
go run .
Air Hot-Reload
Air is a popular live-reload tool for Go. Add -toolexec to the build command in .air.toml:
# .air.toml
root = "."
tmp_dir = "tmp"
[build]
cmd = "go build -toolexec='sfveritas-instrument' -o ./tmp/main ."
bin = "tmp/main"
delay = 1000
exclude_dir = ["assets", "tmp", "vendor"]
exclude_regex = ["_test\\.go"]
include_ext = ["go", "tpl", "tmpl", "html"]
kill_delay = "0s"
[misc]
clean_on_exit = true
Run with:
SF_DEV=true air
Only add -toolexec in your development .air.toml. For CI/CD or production, use a separate config or remove the flag.
Docker Compose
For local development with Docker Compose, install the tool in the build stage and pass -toolexec to the build command:
# Dockerfile.dev
FROM golang:1.22-alpine
WORKDIR /app
# Install the instrumentation tool
RUN go install github.com/SailfishAI/sf-veritas-go/cmd/sfveritas-instrument@latest
# Copy module files first for caching
COPY go.mod go.sum ./
RUN go mod download
COPY . .
# Build with instrumentation
RUN go build -toolexec="sfveritas-instrument" -o /app/server .
# Set env vars for SF Veritas
ENV SF_DEV=true
CMD ["/app/server"]
# docker-compose.yml
services:
api:
build:
context: .
dockerfile: Dockerfile.dev
ports:
- "8080:8080"
environment:
- SF_DEV=true
# Point to host machine's SF Veritas Desktop App
- SF_GRAPHQL_ENDPOINT=http://host.docker.internal:6776/graphql/
Use host.docker.internal to reach the SF Veritas Desktop App running on your host machine. This works on Docker Desktop for Mac and Windows. On Linux, add extra_hosts: ["host.docker.internal:host-gateway"] to your service.
IDE Run Buttons
VS Code
Add to .vscode/launch.json:
{
"version": "0.2.0",
"configurations": [
{
"name": "Run with SF Veritas",
"type": "go",
"request": "launch",
"mode": "auto",
"program": "${workspaceFolder}",
"env": {
"SF_DEV": "true"
},
"buildFlags": "-toolexec=sfveritas-instrument"
}
]
}
GoLand / IntelliJ
- Open Run/Debug Configurations
- In the Go Build configuration, add to Go tool arguments:
-toolexec=sfveritas-instrument - In Environment, add:
SF_DEV=true
Is toolexec Production-Safe?
Yes, but you don't need it in production. Here's the breakdown:
| Concern | Answer |
|---|---|
| Does toolexec modify my source files? | No. It operates on a copy during compilation. Your .go files are never changed. |
| Does the output binary differ? | Yes — instrumented functions have extra span tracking code. This adds a small overhead per function call. |
| Can I ship an instrumented binary? | Technically yes, but don't. The instrumentation sends telemetry to your local desktop app. In production there's nothing listening. |
| What's the overhead? | ~8-10 µs per instrumented function call (span create/close); telemetry send is async (~5 ns to enqueue). Negligible for development, unnecessary for production. See BENCHMARKS.md. |
What if sfveritas-instrument isn't installed? | The build fails with exec: "sfveritas-instrument": executable file not found. This is a feature — it prevents accidental instrumented builds in CI. |
Recommended approach: Use -toolexec only in development. Your production builds use normal go build with no flags — zero overhead, zero dependencies.
# This is the pattern we recommend
dev:
SF_DEV=true go run -toolexec="sfveritas-instrument" .
prod:
go build -o myapp ./...
Framework Examples
net/http (stdlib)
package main
import (
"fmt"
"net/http"
"os"
sfveritas "github.com/SailfishAI/sf-veritas-go"
)
func main() {
if os.Getenv("SF_DEV") == "true" {
sfveritas.SetupInterceptors(sfveritas.Options{
APIKey: "sf-veritas-local",
GraphQLEndpoint: "http://localhost:6776/graphql/",
ServiceIdentifier: "my-api",
})
defer sfveritas.Shutdown()
}
mux := http.NewServeMux()
mux.HandleFunc("/api/users", getUsers)
mux.HandleFunc("/api/health", healthCheck)
fmt.Println("Server starting on :8080")
http.ListenAndServe(":8080", sfveritas.Middleware(mux))
}
func getUsers(w http.ResponseWriter, r *http.Request) {
fmt.Println("Fetching users") // Appears in SF Veritas Console
w.Write([]byte(`{"users": []}`))
}
func healthCheck(w http.ResponseWriter, r *http.Request) {
w.Write([]byte("ok"))
}
Run with instrumentation:
SF_DEV=true go run -toolexec="sfveritas-instrument" .
Gin
Gin needs two pieces of wiring:
- Wrap the Gin engine with
sfveritas.Middleware(the engine is anhttp.Handler) for inbound HTTP tracing and automatic log correlation. - Register
veritasgin.Middleware()for exception capture.gin.Default()installs Gin's ownRecovery(), which catches handler panics before the outersfveritas.Middlewarecan see them — so without this, panics and handled5xxresponses are never reported. The Gin middleware captures the panic (with its stack) and re-panics so Gin still returns the 500. It is a no-op untilSetupInterceptorsruns, so it is safe to register unconditionally.
package main
import (
"net/http"
"os"
"github.com/gin-gonic/gin"
sfveritas "github.com/SailfishAI/sf-veritas-go"
veritasgin "github.com/SailfishAI/sf-veritas-go/gin"
)
func main() {
dev := os.Getenv("SF_DEV") == "true"
if dev {
sfveritas.SetupInterceptors(sfveritas.Options{
APIKey: "sf-veritas-local",
GraphQLEndpoint: "http://localhost:6776/graphql/",
ServiceIdentifier: "gin-api",
})
defer sfveritas.Shutdown()
}
r := gin.Default()
r.Use(veritasgin.Middleware()) // register BEFORE your routes — captures panics + 5xx with stacks
r.GET("/api/users", getUsers)
if dev {
// Wrap the engine for inbound tracing + automatic log correlation.
http.ListenAndServe(":8080", sfveritas.Middleware(r))
} else {
r.Run(":8080")
}
}
func getUsers(c *gin.Context) {
c.JSON(200, gin.H{"users": []string{}})
}
For the precise stack at a specific error site, report it directly:
sfveritas.TransmitError(c.Request.Context(), err).
Echo
package main
import (
"net/http"
"os"
"github.com/labstack/echo/v4"
sfveritas "github.com/SailfishAI/sf-veritas-go"
)
func main() {
if os.Getenv("SF_DEV") == "true" {
sfveritas.SetupInterceptors(sfveritas.Options{
APIKey: "sf-veritas-local",
GraphQLEndpoint: "http://localhost:6776/graphql/",
ServiceIdentifier: "echo-api",
})
defer sfveritas.Shutdown()
}
e := echo.New()
e.GET("/api/users", getUsers)
// Start with SF Veritas middleware wrapping Echo
http.ListenAndServe(":8080", sfveritas.Middleware(e))
}
func getUsers(c echo.Context) error {
return c.JSON(200, map[string]interface{}{"users": []string{}})
}
Fiber
package main
import (
"os"
"github.com/gofiber/fiber/v2"
"github.com/gofiber/fiber/v2/middleware/adaptor"
sfveritas "github.com/SailfishAI/sf-veritas-go"
)
func main() {
if os.Getenv("SF_DEV") == "true" {
sfveritas.SetupInterceptors(sfveritas.Options{
APIKey: "sf-veritas-local",
GraphQLEndpoint: "http://localhost:6776/graphql/",
ServiceIdentifier: "fiber-api",
})
defer sfveritas.Shutdown()
}
app := fiber.New()
// Use the adaptor to wrap SF Veritas middleware
app.Use(adaptor.HTTPMiddleware(func(next http.Handler) http.Handler {
return sfveritas.Middleware(next)
}))
app.Get("/api/users", func(c *fiber.Ctx) error {
return c.JSON(fiber.Map{"users": []string{}})
})
app.Listen(":8080")
}
Manual Function Tracing
If you prefer not to use -toolexec, or need fine-grained control over which functions are traced, use the manual API:
Basic Span
func ProcessOrder(ctx context.Context, orderID string) error {
span := sfveritas.StartSpan(ctx, "ProcessOrder")
defer func() { span.End(nil) }()
// Use span.Context() for child operations
result, err := validateOrder(span.Context(), orderID)
if err != nil {
sfveritas.TransmitError(span.Context(), err)
return err
}
span.End(result)
return nil
}
Span with Arguments
func ProcessOrder(ctx context.Context, orderID string, amount float64) error {
span := sfveritas.StartSpanWithArgs(ctx, "ProcessOrder", map[string]interface{}{
"orderID": orderID,
"amount": amount,
})
defer func() { span.End(nil) }()
// ... function body ...
return nil
}
TraceFunc Helper
result, err := sfveritas.TraceFunc(ctx, "ProcessOrder", func(ctx context.Context) (string, error) {
return doWork(ctx)
})
User Attribution
A backend request has no inherent end user, but you often know which users a request is acting on behalf of. The user-attribution API records which users are active over the lifetime of a request, so telemetry can be attributed to them in the SF Veritas Desktop App.
Attribution is explicit and request-scoped: each call binds to the current
request's trace (served through sfveritas.Middleware). A call made outside a
request is a no-op.
func GetArticles(c *gin.Context) {
ctx := c.Request.Context()
// Mark a user active for this request; pairs with a deferred close.
defer sfveritas.ActiveUserScope(ctx, "customer-42")()
// ... handle the request ...
}
Lower-level API:
sfveritas.AddActiveUser(ctx, "customer-42") // open an interval (idempotent)
sfveritas.RemoveActiveUser(ctx, "customer-42") // close it
sfveritas.SetActiveUsers(ctx, []string{"a", "b"}) // replace the active set:
// adds new users, ends removed ones
sfveritas.ClearActiveUsers(ctx) // end all open intervals
- Events are debounced (~50 ms) and sent as one batch, so rapid changes are cheap.
AddActiveUseris idempotent;SetActiveUsersdiffs against the current set.- Call
sfveritas.ClearActiveUsers(ctx)(or let theActiveUserScopecloser run) at the end of a unit of work so open intervals are closed. Before a short-lived process exits, callsfveritas.FlushUserAttribution()to flush any buffered events.
Attribution binds to the inbound request's trace, which is established by
sfveritas.Middleware. Telemetry emitted from a goroutine you spawn won't carry the
trace unless you pass the request context — see Logs are linked to the request automatically.
Debug Mode
Sailfish supports a server-triggered debug mode: full function-span capture (arguments, return values, timing) can be turned on for your service for a time- and budget-bounded window — useful for live backend services with no HTTP surface (queue consumers, cron jobs, workers).
The SDK holds a lightweight WebSocket "uplink" to Sailfish. When debug mode is
enabled for your service, capture rules are pushed to every running instance and
applied to matching function spans; capture stops automatically when the TTL
expires, a span/byte budget is exhausted, or it's cancelled. No code changes
needed — the uplink starts with SetupInterceptors and stops on Shutdown.
Go has no runtime profiler, so debug mode can only elevate capture on functions
that are already instrumented — manually (StartSpan / TraceFunc) or built
with the -toolexec instrumenter.
It cannot instrument arbitrary functions at runtime. For the most useful debug
mode, build your service with -toolexec so a debug session can turn specific
functions up on demand.
The uplink is on by default. Disable it with SF_UPLINK_ENABLE=false; point
it at a non-default backend with SF_UPLINK_URL (otherwise it derives
ws(s)://…/ws/notify/ from GraphQLEndpoint).
Environment Variables
| Variable | Default | Description |
|---|---|---|
SF_DEV | — | Set to "true" to enable SF Veritas |
SF_API_KEY | — | API key (alternative to Options.APIKey) |
SF_GRAPHQL_ENDPOINT | http://localhost:6776/graphql/ | Collector endpoint |
SF_SERVICE_IDENTIFIER | — | Service name (alternative to Options.ServiceIdentifier) |
SF_DEBUG | false | Enable debug logging to stderr |
SF_UPLINK_ENABLE | true | WebSocket uplink for server-triggered Debug Mode; set false to disable |
SF_UPLINK_URL | (derived) | Override the uplink URL (default derived from GraphQLEndpoint) |
SF_FUNCSPAN_SAMPLE_RATE | 1.0 | Function span sampling rate (0.0 to 1.0) |
SF_FUNCSPAN_ENABLE_SAMPLING | false | Enable span sampling |
SF_LOG_IGNORE_REGEX | — | Regex pattern to suppress logs from telemetry |
SF_NETWORKHOP_CAPTURE_REQUEST_BODY | false | Capture HTTP request bodies |
SF_NETWORKHOP_CAPTURE_RESPONSE_BODY | false | Capture HTTP response bodies |
SF_NETWORKHOP_REQUEST_LIMIT_MB | 1 | Max request body capture size in MB |
SF_NETWORKHOP_RESPONSE_LIMIT_MB | 1 | Max response body capture size in MB |
SF_DISABLE_PRINT_CAPTURE | false | Disable stdout pipe capture |
SF_FUNCSPAN_ARG_LIMIT_MB | 1 | Max argument capture size in MB |
SF_FUNCSPAN_RETURN_LIMIT_MB | 1 | Max return value capture size in MB |
SF_EXCLUDED_DOMAINS | — | Comma-separated domains to skip outbound tracing |
SF_DISABLE_INBOUND_NETWORK_TRACING_ON_ROUTES | — | Comma-separated glob patterns for routes to skip |
SF_INSTRUMENT_DEBUG | false | Enable debug output from the toolexec wrapper |
Configuration File
Create a .sailfish file in your project root for per-file and per-function span configuration:
{
"files": {
"*.go": {
"capture_arguments": true,
"capture_return_value": true,
"sample_rate": 1.0
},
"*_test.go": {
"sample_rate": 0.0
}
},
"functions": {
"ProcessOrder": {
"capture_arguments": true,
"capture_return_value": true,
"arg_limit_mb": 2,
"return_limit_mb": 2
},
"healthCheck": {
"sample_rate": 0.0
}
}
}
File Configuration
| Option | Type | Description |
|---|---|---|
capture_arguments | bool | Capture function arguments |
capture_return_value | bool | Capture return values |
arg_limit_mb | int | Max argument size in MB |
return_limit_mb | int | Max return value size in MB |
sample_rate | float | Sampling rate (0.0 to 1.0) |
Function Configuration
Same options as file configuration but applied to specific function names. Function-level config takes priority over file-level config.
Verifying the Setup
- Start your application:
SF_DEV=true go run -toolexec="sfveritas-instrument" . - Open the SF Veritas Desktop App
- Open the Console panel — you should see logs and print statements
- Trigger some HTTP requests — you should see them in the Network panel
- Open the Flamechart — you should see function execution traces
Debug Mode
Enable debug output to verify instrumentation is working:
SF_DEV=true SF_DEBUG=true go run -toolexec="sfveritas-instrument" .
You'll see output like:
[sfveritas] Initializing Go Go BACKEND collector v0.1.5
[sfveritas] Endpoint: http://localhost:6776/graphql/
[sfveritas] Setup complete. Interceptors active.
[sfveritas] Function span: ProcessOrder (abc-123) duration=1234ns
[sfveritas] Outbound request: GET https://api.example.com/data → 200 (45ms)
Troubleshooting
No logs appearing
- Check the desktop app: Ensure the SF Veritas Desktop App is running
- Verify the endpoint: Ensure
GraphQLEndpointmatches your server port - Check
SF_DEV: Must be set to"true" - Check terminal output: Look for
[sfveritas]initialization messages
Logs appear but aren't nested under their request
- Ensure requests are served through
sfveritas.Middleware— log correlation is established by the inbound middleware. With Gin, also wrap the engine (sfveritas.Middleware(r)), not justr.Run(). See the Gin example. - Logs from goroutines you spawn won't auto-correlate — pass the request context
with
slog.InfoContext(r.Context(), ...). See Logs are linked to the request automatically.
sfveritas-instrument: command not found
- Ensure
$GOPATH/binis in your$PATH:export PATH=$PATH:$(go env GOPATH)/bin - Re-install:
go install github.com/SailfishAI/sf-veritas-go/cmd/sfveritas-instrument@latest
Connection refused errors
- Verify the SF Veritas Desktop App is installed and running
- Check that the local server is running (look for server status in the Desktop App)
- Ensure port 6776 is not blocked by a firewall
- In Docker: use
host.docker.internalinstead oflocalhost
Build errors with toolexec
- Verify Go version: Requires Go 1.22+
- Check the tool is built:
sfveritas-instrument --helpshould not error - Enable debug: Set
SF_INSTRUMENT_DEBUG=trueto see which files are being instrumented
Multi-Service Setup
When running multiple Go services locally, give each a unique ServiceIdentifier:
// user-service/main.go
sfveritas.SetupInterceptors(sfveritas.Options{
APIKey: "sf-veritas-local",
GraphQLEndpoint: "http://localhost:6776/graphql/",
ServiceIdentifier: "user-service",
})
// order-service/main.go
sfveritas.SetupInterceptors(sfveritas.Options{
APIKey: "sf-veritas-local",
GraphQLEndpoint: "http://localhost:6776/graphql/",
ServiceIdentifier: "order-service",
})
Use the service filter in the Console to switch between services.
Enterprise Setup
This section applies to users with a Sailfish Enterprise account. If you're using the Desktop App for local development only, the basic setup above is all you need.
Enterprise users need two configurations — one for local development (Desktop App) and one for staging/production (Sailfish cloud). The SDK should detect which environment it's running in and configure itself accordingly.
Dual-Mode Configuration
package main
import (
"os"
sfveritas "github.com/SailfishAI/sf-veritas-go"
)
func main() {
if os.Getenv("SF_DEV") == "true" {
// Local development — send to Desktop App
sfveritas.SetupInterceptors(sfveritas.Options{
APIKey: "sf-veritas-local",
GraphQLEndpoint: "http://localhost:6776/graphql/",
ServiceIdentifier: "my-go-service",
ServiceVersion: "1.0.0",
})
defer sfveritas.Shutdown()
} else if apiKey := os.Getenv("SAILFISH_API_KEY"); apiKey != "" {
// Staging/Production — send to Sailfish cloud
// Do NOT set GraphQLEndpoint — the SDK defaults to the cloud endpoint
sfveritas.SetupInterceptors(sfveritas.Options{
APIKey: apiKey,
ServiceIdentifier: "my-go-service",
ServiceVersion: "1.0.0",
})
defer sfveritas.Shutdown()
}
// Start your application...
}
Environment Variables for Enterprise
Set these in your staging/production deployment:
# Required — your company's Enterprise API key
SAILFISH_API_KEY=your-enterprise-api-key
# Required — the git commit SHA for this build
GIT_SHA=$(git rev-parse HEAD)
# Optional — override the endpoint (only for local dev, do NOT set in production)
# SAILFISH_GRAPHQL_ENDPOINT=http://localhost:6776/graphql/
GIT_SHA
The GIT_SHA environment variable allows Sailfish to correlate telemetry with specific commits. The Go SDK reads it automatically from os.Getenv("GIT_SHA").
Set it in your build pipeline:
# Docker
docker build --build-arg GIT_SHA=$(git rev-parse HEAD) .
# In Dockerfile
ARG GIT_SHA
ENV GIT_SHA=$GIT_SHA
# Makefile
dev:
SF_DEV=true go run -toolexec="sfveritas-instrument" .
prod:
GIT_SHA=$$(git rev-parse HEAD) go build -o myapp ./...
URL Override
You can override the GraphQL endpoint in two ways:
- Environment variable:
SAILFISH_GRAPHQL_ENDPOINT - Constructor argument:
GraphQLEndpointinsfveritas.Options{}
| Environment | Endpoint | How |
|---|---|---|
| Local dev | http://localhost:6776/graphql/ | Set GraphQLEndpoint or SAILFISH_GRAPHQL_ENDPOINT |
| Staging/Production | Default (cloud) | Do not set either — the SDK uses https://api-service.sailfishqa.com/graphql/ automatically |