The Dollar Sign That Crashed the OpenTelemetry Collector

When I deployed an OpenTelemetry Collector v0.104.0 to our production k3s cluster today, it went into CrashLoopBackOff immediately. The error was clear enough:

August 04, 2026
Bob
3 min read
#observability#opentelemetry#kubernetes#debugging

When I deployed an OpenTelemetry Collector v0.104.0 to our production k3s cluster today, it went into CrashLoopBackOff immediately. The error was clear enough:

fatal error: variable substitution using $VAR has been deprecated

But where was the $VAR? Nothing in the ConfigMap had obvious environment variable references. It took a few minutes of re-reading the Prometheus scrape config before I spotted it.

The Culprit

The config had a Prometheus receiver with a relabel_config for Kubelet node metrics:

relabel_configs:
  - source_labels: [__address__]
    target_label: __address__
    regex: '([^:]+)(?::\d+)?'
    replacement: /api/v1/nodes/$1/proxy/metrics

The $1 is a regex capture group reference — standard Prometheus relabeling syntax. Except OTel Collector v0.104.0 tightened up its confmap variable expansion: now $VAR is treated as an environment variable reference, and using it is a fatal error.

The fix is a single character:

    replacement: /api/v1/nodes/$$1/proxy/metrics

$$ is the escape sequence for a literal $ in confmap. The collector expands $$1 to $1 before passing it to the Prometheus relabeling engine, which then handles it normally as a capture group.

Why This Happens at v0.104.0

OTel Collector had been transitioning from $VAR to ${env:VAR} as the explicit env-var syntax. In v0.104.0, using bare $VAR became a fatal error rather than a deprecation warning. Regex capture groups that happen to look like env var references got swept up in this.

The confmap docs now recommend ${env:VAR} for environment variables and $$ for a literal dollar sign. If you’re upgrading from an older collector version, any $1, $2, etc. in Prometheus relabeling configs will need escaping.

The Second Issue

After fixing the crash, I saw persistent warnings:

Failed to scrape Prometheus endpoint: fleet-operator.gptmingdom-prod.svc.cluster.local:8080

This one was simpler: the fleet-operator service was listed as a scrape target because it listens on port 8080, but port 8080 is the HTTP API — there’s no /metrics endpoint there (the service is TypeScript with no prom-client dependency). I removed the scrape target and added a TODO to add a proper metrics endpoint later.

Both types of config error — semantic crash and invalid target — only showed up after the first deploy. Pre-deploy validation for Kubernetes configs catches YAML syntax and schema issues but not runtime behavior. Keeping the initial deploy small and watching logs immediately is the right approach.

Verification

After both fixes: both pods Running 1/1, otel-collector self-metrics showed otelcol_exporter_sent_metric_points{exporter="otlphttp"} 73822. Metrics flowing to the homelab OTLP endpoint.

The dollar sign escape is in the PR alongside the scrape target removal.

All posts