Ingestion

Berserk ingests telemetry — logs, traces, and metrics — via the OpenTelemetry Protocol (OTLP). You configure a standard OpenTelemetry Collector to send data to Berserk. Everything after that is handled automatically.

How Data Flows

1. Your OpenTelemetry Collector sends data to Berserk's Ingest component over OTLP (gRPC or HTTP).
2. Your collector includes an ingest token in each request. Ingest validates it with the Meta service, which authenticates the token. Ingest then batches incoming data and uploads it to S3.
3. The query component (Nursery) follows each stream, downloads batches from S3, routes data to the correct tables, and makes them searchable. Nursery also merges small batches into larger optimized segments in the background.

Ingest holds each request open until S3 confirms the upload, then returns that result to the collector. A success response means the data is durably stored. If S3 is temporarily unavailable, Ingest returns a failure to the collector rather than buffer locally. Your OpenTelemetry Collector is the durability layer — it is responsible for retrying failed requests and persistently queuing data until Ingest accepts it.

Ingest Tokens

Every request to Ingest must carry an ingest token for authentication and routing:

Authorization: Bearer ing_xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx

Each token is bound to a table. All data sent with that token routes to that table. You can override routing per-record by setting the bzrk.table resource attribute.

Create a token with the CLI:

bzrk ingest-token create --table default my-token

Default Ingest Token (Kubernetes)

When deploying with the Helm chart, the ingest service can be configured with a default ingest token via a Kubernetes Secret. This token is used to authenticate incoming data when no other token is provided.

Managed mode (recommended): Set global.ingestToken.managed: true in your Helm values. An init container will automatically create the token by calling Meta's API and store it in a Kubernetes Secret before Ingest starts. This is idempotent — if the secret already exists, the init container is skipped entirely.

global:
  ingestToken:
    managed: true

Manual mode: Create the secret yourself and reference it in the chart:

kubectl create secret generic ingest-token \
  --from-literal=default_ingest_token="ing_<your-token-value>"

The Helm chart references this secret by default (ingest-token with key default_ingest_token).

Streams

A stream is a sequential write path in S3. Ingest registers a stream with Meta on startup and writes all incoming data — from any number of collectors and ingest tokens — to that single stream. Data from different tokens targeting different tables is batched together in the same upload; Nursery handles the routing.

In some cases Meta may assign more than one stream to an Ingest instance (e.g., after a restart or during scaling), but typically there is just one. Streams are created and managed automatically — you do not need to configure or interact with them directly.

Delivery Semantics

What a response from the ingest endpoint means, precisely:

Three consequences worth internalizing:

• An acknowledgment is a durability guarantee. Ingest holds each request open until S3 confirms the upload — there is no window where acknowledged data exists only in a local buffer. This also means a slow S3 makes acks slower, not less safe; backpressure (429) is how Ingest tells your collector to hold data on its side, which is why the collector's queue is the durability layer for unacknowledged data.
• Delivery is at-least-once. If a response is lost in transit (network failure, timeout) after the write actually landed, your collector retries and Ingest deduplicates the resend in the common case (it verifies suspected duplicates against content hashes on S3). In rare partial-failure scenarios — S3 accepting a write whose confirmation never arrived, across a sustained outage — the same data can be stored twice. This is a known, accepted limitation; if your pipeline is sensitive to duplicates, deduplicate downstream on record identity.
• Retry-After values are jittered. Ingest spreads the delays it hands out (e.g. 30–45s during a sustained S3 outage) so that a fleet of throttled collectors does not return in one synchronized wave. Don't be surprised by varying values for the "same" condition.

Latency, Error Recovery and Durability

Protocols

Ingest accepts OTLP over both gRPC and HTTP:

OpenTelemetry Collector Configuration

Below is the recommended default configuration for sending data to Berserk. Your setup may vary depending on your environment and use case, but these settings are a good starting point.

# Disk-backed queue so buffered data survives collector restarts.
extensions:
  file_storage/queue:
    directory: /var/lib/otel/queue

receivers:
  otlp:
    protocols:
      grpc:
        endpoint: 0.0.0.0:4317
      http:
        endpoint: 0.0.0.0:4318

exporters:
  otlp/berserk:
    # TLS endpoint (https): leave TLS enabled — do not set insecure: true.
    endpoint: "<your-ingest-endpoint>:4317"
    # Plaintext endpoint only (e.g. in-cluster ingest:4317):
    #   tls:
    #     insecure: true
    headers:
      authorization: "Bearer <your-ingest-token>"

    # OTLP payloads typically compress 3-5x with gzip (ratio depends on
    # payload redundancy). This cuts network egress, shortens per-request
    # wall time, and usually lets larger batches fit under the ingest
    # service's 16 MiB wire cap.
    compression: gzip

    # Berserk ingest service may hold each request until its batch window closes (up to 2s)
    # and the S3 upload completes. 30s stays above the ingest service's
    # internal ack timeout so it can return a real retryable error
    # instead of the collector timing out first.
    timeout: 30s

    sending_queue:
      # Disk-backed so buffered data survives collector restarts.
      storage: file_storage/queue

      # Concurrency ceiling, not a constant: idles at ~1, ramps only under
      # backlog. Keep num_consumers × signals (fleet-wide) under ingest capacity.
      num_consumers: 4

      # Outage buffer: peak_rate × tolerated_outage × headroom.
      queue_size: 1073741824
      sizer: bytes

      # The single batching layer (exporter queue batcher, not a `batch`
      # processor): send Berserk fewer, larger requests.
      batch:
        sizer: bytes
        min_size: 4194304     # 4 MiB
        flush_timeout: 1s     # ~3s end-to-end (server adds ~2s)
        max_size: 10485760    # under the 16 MiB wire cap

    retry_on_failure:
      enabled: true
      initial_interval: 1s
      max_interval: 60s
      # 0 = retry forever. The default 5min limit drops data after timeout.
      max_elapsed_time: 0
      multiplier: 2

processors:
  # Backpressure on receivers when approaching memory limit.
  # Prevents OOM before the disk queue absorbs everything.
  memory_limiter:
    check_interval: 1s
    limit_mib: 256
    spike_limit_mib: 64

service:
  extensions: [file_storage/queue]
  # Batching lives in the exporter's sending_queue.batch (above), not a
  # `batch` processor — see "Why these settings matter" below.
  pipelines:
    traces:
      receivers: [otlp]
      processors: [memory_limiter]
      exporters: [otlp/berserk]
    logs:
      receivers: [otlp]
      processors: [memory_limiter]
      exporters: [otlp/berserk]
    metrics:
      receivers: [otlp]
      processors: [memory_limiter]
      exporters: [otlp/berserk]

Why These Settings Matter

timeout: 30s — Berserk batches incoming data for up to 2 seconds before uploading to S3. The collector's default 5-second timeout will cause spurious failures during normal operation. 30 seconds gives headroom for S3 uploads under load and stays above the ingest service's internal 25s ack timeout — during a sustained S3 outage the ingest service answers within that window with an HTTP 429 + Retry-After (jittered, 30–45s), which the collector's exporter respects so it doesn't keep hammering the backend. Shortening the collector timeout below 25s risks dropping this signal and converting intentional throttles into generic timeouts.

file_storage/queue — Berserk's ingest service has no local durability — your collector is the durability layer. If the collector restarts with an in-memory queue, all buffered data is lost. The file_storage extension persists the queue to disk.

max_elapsed_time: 0 — Disables the default 5-minute retry limit. The ingest service returns retryable errors for backpressure (429), transient S3 failures (503), and stream recovery (503). With a disk-backed queue, the collector should retry indefinitely. Setting a limit means data is silently dropped after the timeout.

num_consumers: 4 — A ceiling on parallel export workers, not a constant level of parallelism: actual concurrency is min(num_consumers, batches ready in the queue), so it sits at ~1 when load is light and ramps toward the ceiling only under backlog. Berserk accepts a limited number of concurrent requests, so keep num_consumers × number-of-signals (summed across your collector fleet) modest, and prefer larger batches (below) over more consumers for throughput. Raise it only if a backlog won't drain after an outage — and note the queue also fills when Berserk is slow, not just when you're busy, so an oversized value mainly piles extra requests onto an already-busy endpoint.

sending_queue.batch — Batches on the collector so Berserk receives fewer, larger requests. Berserk holds each request open until its data is durably stored, so fewer requests means less waiting and less throttling under load. min_size coalesces into fuller requests; flush_timeout: 1s bounds the added latency (≈3 s end-to-end, since Berserk adds up to ~2 s of its own batching); max_size keeps requests under the 16 MiB wire cap. This is the exporter-helper queue batcher — use it instead of the legacy batch processor (next).

memory_limiter — Applies backpressure to receivers when the collector approaches its memory limit. Without this, if the ingest service is slow and the queue is filling, the collector can OOM before the disk queue absorbs everything.

No batch processor — Batch via the exporter's sending_queue.batch (above), not the legacy batch processor. The processor buffers data before the persistent queue, so anything it holds is lost on a collector restart, and it is on the upstream deprecation path in favor of exporter-helper batching. This is not "don't batch on the collector" — batching there means Berserk handles fewer requests; it just belongs in the durable queue layer, not a pre-queue processor.

compression: gzip — OTLP payloads typically compress 3–5× with gzip (ratio varies with payload redundancy). Enabling it cuts network egress, shortens per-request wall time, and usually lets larger batches fit under the per-request wire cap.

Configuration Variations and Their Consequences

The recommended config above is one point in a space. The two knobs that actually decide what you can lose are the queue storage and the retry budget — here is what each combination buys you:

Two further consequences that follow from the semantics above:

• The retry loop occupies a queue consumer for its whole wait. With max_elapsed_time: 0 during a long outage, up to num_consumers batches sit in memory retrying while the backlog accumulates in the queue behind them. That's by design — but it's why num_consumers, memory_limiter, and queue_size should be set together, not independently.
• Receivers have their own loss modes. A persistent exporter queue only protects data after it is enqueued. If you tail files (filelog receiver), also configure its storage: for offset persistence — otherwise a collector restart can re-read (duplicates) or skip (loss) file content regardless of how good your exporter queue is.

Request Size Limits

Berserk's ingest service accepts OTLP requests up to 16 MiB on the wire. With compression: gzip enabled that typically carries 40–80 MiB of uncompressed telemetry per request — ample for busy collectors.

If your collector emits requests above this ceiling, the ingest service returns 413 (HTTP) or InvalidArgument (gRPC). Options:

• Enable compression: gzip on the exporter (see above) — this is the easy win.
• Lower max_size on the exporter's sending_queue.batch so each request stays under the wire cap.

Ingress sizing: if you terminate TLS or proxy through nginx/Envoy/Istio in front of the ingest service, check your ingress body limits. The most common trap is nginx-ingress, whose default client_max_body_size is 1 MiB — set the annotation nginx.ingress.kubernetes.io/proxy-body-size: "16m" on the ingest Ingress resource. Other ingresses (Envoy, Contour, AWS ALB, GCP HTTP(S) LB) either have no body limit or a limit well above 16 MiB, but it is worth confirming for your specific setup.

Verifying Ingestion

After configuring your collector, verify data is flowing:

bzrk search "<your table> | take 10" --since "5m ago"

The table you query must exist and your ingest token must route to it — data sent with a token bound to table foo won't show up under any other table. If the query returns unknown table, create the table (or fix the name) and confirm your token's routing. If your CLI isn't pointed at the cluster yet, set up a profile and sign in first — see AI Agent Setup.

If no data appears, check:

• The ingest token is correct and not revoked, and is bound to the table you're querying
• The collector can reach your ingest endpoint on 4317 (gRPC) or 4318 (HTTP)
• The collector logs for export errors or retries