Skip to content
By Dharit Shah, Adam Scerra, Hector Martinez
Picture of Dharit Shah
Dharit Shah
Picture of Adam Scerra
Adam Scerra
Picture of Hector Martinez
Hector Martinez

Tracing Reference ​

Structured reference for fullsend's distributed tracing system: environment variables, span hierarchy, attributes, and operational behavior. For step-by-step setup, see How To Emit Traces. For implementation details, see the Tracing Development Guide.

Telemetry levels ​

LevelWhat it producesConfiguration required
1run-telemetry.jsonl file in the run output directoryNone
2OTLP/HTTP export to a remote backend (metadata only)OTEL_EXPORTER_OTLP_*ENDPOINT
3Conversation content (assistant text, reasoning, tool calls, and — on Claude runs — tool results) on agent spansOTEL_INSTRUMENTATION_GENAI_CAPTURE_MESSAGE_CONTENT=true

All levels produce metadata (timing, token counts, tool names, errors), including up to one execute_tool span per id-bearing tool call under each agent span (Claude Code today; pi and codex emit no call ids, #7414), capped at 1,024 per iteration, and — on mixed-model Pi iterations — one usage <model> child per per_model_usage entry. Level 3 adds the agent's conversation content to spans — enabled by one environment variable, exactly like Level 2's endpoint.

Environment variables ​

Endpoint configuration ​

VariablePurposeNotes
OTEL_EXPORTER_OTLP_TRACES_ENDPOINTSignal-specific endpoint URLTakes precedence; used as-is, no /v1/traces appended
OTEL_EXPORTER_OTLP_ENDPOINTBase endpoint URLSDK appends /v1/traces automatically

Authentication ​

VariablePurposeNotes
OTEL_EXPORTER_OTLP_TRACES_HEADERSSignal-specific headersTakes precedence; key=value pairs separated by commas; values are URL-decoded
OTEL_EXPORTER_OTLP_HEADERSBase headersSame format as above

Private CA ​

VariablePurposeNotes
OTEL_EXPORTER_OTLP_CERTIFICATEPEM file path for TLS root certificatesPoints at a CA bundle for verifying the OTLP backend's certificate; no skip-verify option exists. In managed workflows the PEM file must be committed into the repository checkout (e.g. .fullsend/otel-ca.pem) because the runner has no other persistent filesystem; bring-your-own-workflow runs can use any local path

Resource attributes ​

VariablePurposeNotes
OTEL_RESOURCE_ATTRIBUTESStatic k=v,k=v trace tagsMerged into the OTel resource; ${{ github.* }} expressions only evaluate in workflow YAML, not in Actions variables

Kill switches ​

VariableValueEffect
OTEL_SDK_DISABLEDtrue (case-insensitive)Disables all telemetry output: OTLP export and the local file

To disable only OTLP export without affecting the local file, unset the endpoint variables:

bash
unset OTEL_EXPORTER_OTLP_ENDPOINT
unset OTEL_EXPORTER_OTLP_TRACES_ENDPOINT

Trace propagation ​

VariablePurposeNotes
TRACEPARENTW3C Trace Context parentWhen present, the root span becomes SpanKindConsumer; when the sampled flag is unset (-00), OTLP export is suppressed but the local file is still written
TRACESTATEW3C Trace Context statePropagated alongside TRACEPARENT

Content capture (Level 3) ​

Fullsend assembles Level 3 content from the normalized event stream the console renders, redacts it through the security output pipeline, and attaches it to the per-iteration agent span. The agent runtime's own content-logging variables (OTEL_LOG_USER_PROMPTS, OTEL_LOG_ASSISTANT_RESPONSES, etc.) are never set.

VariableValues that enable captureValues that keep it off
OTEL_INSTRUMENTATION_GENAI_CAPTURE_MESSAGE_CONTENTtrue, span_only, span_and_event (case-insensitive)unset, false, NO_CONTENT, event_only, anything unrecognized

The variable name and accepted values follow the OpenTelemetry GenAI instrumentation convention. Fullsend records content on span attributes only, so event_only stays off. An unrecognized value disables capture; telemetry never fails a run.

Captured: assistant text, reasoning, tool calls (name plus short summary), and tool results — including any sub-agent activity, unattributed — as the gen_ai.output.messages span attribute: a JSON string following the GenAI output-messages schema with a finish_reason of stop or error. Tool results, and the id that correlates each with its call, are captured only when the runtime's stream provides them: Claude runs do; the pi and codex parsers emit neither yet (#7414).

Not captured: model input (gen_ai.input.messages) and pre/post-script content. First-iteration runs have no meaningful runner-side input; retry iterations carry the injected validation feedback, a natural input-capture follow-up.

Redaction and size: every part passes through security redaction (Unicode normalization, then secret masking) before reaching the span. Content is bounded at 256 KiB per iteration — each tool result at 8 KiB — kept as an ordered suffix; overflow drops the oldest content first. Those bounds count raw bytes; the exported JSON string is bounded as well, at 255,000 bytes, because encoding adds 9–11% at these bounds on real runs and up to six times on escape-dense content — a record over that is trimmed again, oldest first. Truncation is marked via fullsend.content.truncated on the span and fullsend.truncated on each cut part. A tool result whose stream line exceeds the parser's 1 MiB bound is kept as an empty, marked tool_call_response part — the call was answered, its content is lost and its is_error unknown — provided the line shows the call id within its first 1 MiB, where Claude Code normally writes it (ahead of the content; an id serialized after the content is not recovered, and that call closes unanswered). Two cases are absent rather than truncated: any other stream line beyond 1 MiB is skipped whole by the parser, and results whose content is entirely non-text (for example images) produce no part. A result that mixed text with non-text blocks keeps its text and is marked fullsend.truncated; a failed call with empty output survives as a tool_call_response part carrying is_error. The SDK's span attribute length cap is lifted while capture is on; an explicit OTEL_SPAN_ATTRIBUTE_VALUE_LENGTH_LIMIT still wins and will cut content mid-JSON — fullsend warns on stderr at startup.

Size limits: none of these bounds is a measured backend limit. The only acceptance proof is one 255,082-byte attribute, read back whole from the pilot MLflow backend on 2026-08-20; nothing larger was ever sent, so the backend's ceiling is unknown. The runner imposes nothing lower: the SDK cap is lifted, the exporter is OTLP over HTTP with no message limit of its own, and the file sink has none. The bounds stay because guessing wrong is costly — a backend that refuses an oversized request refuses the whole batch, up to 512 spans with their Level 1 metadata, and the exporter does not retry a refusal. On three captured review runs (117–255 tool results per iteration, sub-agents included) these bounds evict 28–56% of tool results. A 1 MiB total with the same per-result bound evicts none (records of 412–938 KB); raising only the per-result bound to 32 KiB evicts 47–91%; keeping every result whole takes 1.1–2.2 MB. So the total is the bound to raise first, once the size is proven on the target backend: send attributes of increasing size and read each back whole (#7415).

Sinks: content rides the span to both run-telemetry.jsonl and the OTLP endpoint (when configured). Spans may contain proprietary source code, PII, or credentials; the organization enabling capture is responsible for its backend's access controls. For how MLflow displays the content, see Tracing with MLflow.

Span hierarchy ​

A run produces this span tree. Span names match the name field in run-telemetry.jsonl; exported spans and the local file are two views of the same trace with identical span IDs.

run (root; Consumer when dispatched with TRACEPARENT, else Internal)
├── sandbox_create (gen_ai.operation.name=create_agent)
└── agent           (one per iteration; gen_ai.operation.name=invoke_agent)
    ├── execute_tool (one per id-bearing tool call; gen_ai.operation.name=execute_tool)
    └── usage <model> (mixed-model Pi only; one per PerModelUsage entry)

execute_tool spans are named execute_tool <tool name>. One starts when the runtime reports a tool call (its arguments complete) and ends when it reports the result — both are runner-side receipt times, which trail the sandbox by the pipe latency and the parser's decode of the line, stamped after the console renderer's output for the call (it prints nothing for a result) and before that event's Level 3 content processing. Events are handled one at a time, so with several calls open a time can still trail the processing of an earlier event, such as another call's large result. The span approximates execution rather than measuring it: both times trail the sandbox, and when the start trails by more than the end (an earlier event was still being processed when the call arrived) the span is shorter than the execution it covers. A call with no result by the end of the iteration is closed with error.type=unanswered; a result whose stream line exceeded the parser's 1 MiB bound ends its span on receipt, marked fullsend.tool.result_oversized with no status and no error.type — the tool answered, and whether it failed was never decoded; a result whose call was never reported (its stream line was skipped) is a near-zero-duration span marked fullsend.tool.unmatched. Runtimes whose parsers emit no call ids (pi, codex) produce no execute_tool spans, and neither do server-side tools, whose result never arrives as a tool_result. Tool names and call ids pass through the same sanitizer as span content (Unicode normalization, then secret redaction): a name is redacted in place and bounded to 256 bytes for the attribute and 128 for the span name; an id with any finding is dropped from the span. At most 1,024 execute_tool spans are recorded per iteration; calls past that are counted in fullsend.tool_spans.dropped on the agent span. Tool content never rides these spans — see Content capture.

SpanKind ​

SpanKindCondition
runConsumerValid inbound TRACEPARENT (dispatched by an instrumented system)
runInternalNo inbound TRACEPARENT (local/manual invocation)
sandbox_createInternalAlways
agentInternalAlways
execute_toolInternalAlways
usage <model>InternalMixed-model Pi iterations only

Span attributes ​

GenAI semantic convention attributes ​

These follow the OTel GenAI semantic conventions and are recognized by LLM-aware backends for GenAI dashboards.

AttributeExamplePresent on
gen_ai.operation.nameinvoke_agentrun, agent, mixed-model usage <model> (create_agent on sandbox_create; execute_tool on execute_tool)
gen_ai.agent.nametriagerun, agent
gen_ai.tool.nameBashexecute_tool (the runtime's tool name; absent when the call was never reported)
gen_ai.tool.call.idtoolu_01…execute_tool (absent when the id carried a security finding — a tainted id is dropped, never substituted, so it cannot collide with another call's)
gen_ai.system / gen_ai.provider.nameanthropic / anthropic-vertexagent, and each mixed-model usage <model> child (serving endpoint of the model used on this span — varies by runtime; system is the pre-v1.37 name — both keys are emitted with the same value so EM-001 and modern backends agree. Not the runtime name: fullsend.runtime is the harness.)
gen_ai.request.modelclaude-opus-4-6agent (parent resolved model); each mixed-model usage <model> child (that component's model id)
gen_ai.usage.input_tokens / output_tokens / cache_*_input_tokens109938agent on single-model runs; mixed-model Pi iterations emit these on usage <model> children instead of the agent span (see Per-model usage components)

Provider identity is the serving endpoint, not the model publisher and not the agent runtime. Claude Code reports anthropic. Pi reports the prefix of the resolved provider/id spec (anthropic-vertex, xai-vertex, google-vertex, openai, anthropic): a Claude model on Vertex is anthropic-vertex even though the publisher is Anthropic, because that is the catalog and credential path the run used. fullsend.runtime (claude, pi, …) stays a separate Fullsend attribute. Fullsend does not emit mlflow.* attributes; backends that derive native cost fields do so from these portable GenAI keys.

When a Pi run dispatches sub-agents on different vendors, the parent agent span keeps the parent serving-endpoint identity and the iteration's fullsend.cost_usd rollup, and one usage <model> child is emitted per per_model_usage entry. Each child carries that component's provider, model, tokens, request count, and cost. See Per-model usage components.

Breaking change — Pi runtime: agent spans from the Pi runtime used to report the literal string pi under gen_ai.system. They now report the resolved serving-endpoint provider under both gen_ai.system and the newly emitted gen_ai.provider.name (anthropic-vertex, xai-vertex, google-vertex, openai, anthropic). Downstream consumers that filtered or classified on gen_ai.system == "pi" will silently stop matching Pi-runtime spans and must switch to fullsend.runtime == "pi" to identify Pi-originated spans, or update their provider allowlist to include the resolved values above.

Breaking change — mixed-model Pi iterations: agent spans for mixed-model Pi iterations (more than one per_model_usage entry) no longer carry gen_ai.usage.input_tokens / output_tokens / cache_creation.input_tokens / cache_read.input_tokens. Those values move to the new usage <model> children, one per per_model_usage entry. Downstream consumers that summed gen_ai.usage.* directly off the agent span will silently see missing or zero token counts for mixed-model iterations going forward and must filter on fullsend.usage.component children instead. Single-model iterations are unaffected. See Per-model usage components.

Fullsend-specific attributes ​

AttributePresent onDescription
fullsend.runtimeagent, mixed-model usage <model>Harness identity (claude, pi, …), distinct from gen_ai.system (the serving endpoint)
fullsend.work_item_idrunWork item identity (e.g. owner/repo#123); primary cross-run correlation key
fullsend.agentrunAgent name
fullsend.cost_usdrun (aggregated), agent, mixed-model usage <model>Cost in USD, rounded to cents (see Cost data contract). On mixed-model Pi iterations the agent value is the iteration rollup and each usage <model> value is that component's share; do not sum both.
fullsend.usage.rollupagentPresent (true) on mixed-model Pi iterations: this span is the parent/root rollup, not a billable component. Absent on single-model runs.
fullsend.usage.componentusage <model>Present (true) on each per-model usage child. Query this flag (not span name) to sum billable tokens/cost without the parent rollup.
fullsend.usage.model_specusage <model>The per_model_usage key that produced this child (anthropic-vertex/claude-sonnet-5, xai-vertex/xai/grok-4.6, unknown).
fullsend.usage.requestsusage <model>Inference episodes attributed to this model spec (one for the parent iteration, one per sub-agent call).
fullsend.usage.droppedagentPresent when the iteration's per_model_usage breakdown had more distinct specs than the 1,024-spec usage-span cap allows: the number of specs refused a usage <model> component. Mirrors fullsend.tool_spans.dropped for the usage-component stream.
fullsend.tool_callsrun (aggregated), agentNumber of tool invocations
fullsend.num_turnsrunTotal conversation turns across all iterations
fullsend.iterationsrunNumber of agent iterations (validation loop included)
fullsend.security_trace_idrunSecurity scanner trace correlation ID
fullsend.harness.urlrunSource URL the harness was fetched from; omitted for local-path harnesses
fullsend.harness.pathrunLocal path of the resolved harness file; omitted when empty
fullsend.harness.content_sharunSHA-256 of the resolved harness file; absent when the file cannot be read
fullsend.prescript.skippedrunWhether the pre-script signaled a skip
fullsend.prescript.skip_reasonrunHuman-readable skip reason from the pre-script
fullsend.transcript_erroragentPresent (true) when the agent exited 0 but its transcript reported an error — the span's status is Error while exit_code keeps the raw process exit
gen_ai.output.messagesagentLevel 3 only: the iteration's conversation content as a JSON string (see Content capture)
fullsend.content.truncatedagentLevel 3 only: present (true) when the size budget cut or dropped content, or a kept tool result is a parser-side fragment or an oversized line's empty stand-in (fullsend.truncated on the part; no byte count)
fullsend.content.dropped_bytesagentLevel 3 only: exact part bytes removed by the size budget — content, ids, and the fixed footprint of an errored-empty or oversized stand-in part — in raw bytes whichever bound made the cut; the bytes of a skipped oversized line were never decoded and are not counted
fullsend.content.redactionsagentLevel 3 only: number of security findings raised while redacting content at assembly (including findings from parts the size budget later dropped)
fullsend.tool.unmatchedexecute_toolPresent (true) when a result arrived for a call the stream never reported; the span has near-zero duration
fullsend.tool.result_oversizedexecute_toolPresent (true) when the result's stream line exceeded the parser's 1 MiB bound: the call was answered but nothing of the result was decoded, so the span has no status and no error.type
fullsend.tool_spans.droppedagentPresent when the iteration hit the 1,024-span cap and at least one id-bearing call was refused a span: the number of tool_use events with a usable id that arrived past the cap, each counted once whatever its result later does; a result with no open span past the cap is not counted

Common attributes ​

AttributePresent onDescription
exit_coderun, agentProcess exit code
iterationagent1-based iteration index
error.typeexecute_tooltool_error when the runtime flagged the result is_error; unanswered when the call had no result by the end of the iteration (the runtime was stopped, or an over-long result line showed no call id within its first 1 MiB) or the runtime reported the same call id again (the earlier open call is superseded); absent on success

Resource attributes ​

Set on every span via the OTel resource:

AttributeValue
service.namefullsend
service.versionCLI version string

Additional resource attributes from OTEL_RESOURCE_ATTRIBUTES are merged in.

Per-model usage components ​

A Pi iteration that dispatched sub-agents on more than one model spec already records the breakdown in metrics.json (per_model_usage). The trace now exports the same components so model/provider dashboards do not attribute every child to the parent.

When they appear. One Internal child of that iteration's agent span per per_model_usage entry, named usage <model>, only when the iteration has more than one spec. A single-model Pi run (including an Agent-enabled iteration that dispatched nothing) and any runtime that leaves per_model_usage unset keep today's agent span and emit no children. Historical traces are not rewritten; metrics.json is unchanged. At most 1,024 children are emitted per iteration; a breakdown with more distinct specs than that reports the overflow as fullsend.usage.dropped on the agent span instead of emitting further children.

What each child carries. The serving-endpoint provider (gen_ai.system / gen_ai.provider.name), the effective model id (gen_ai.request.model), input/output/cache-creation/cache-read tokens (gen_ai.usage.*), fullsend.usage.requests, fullsend.usage.model_spec, fullsend.cost_usd (that component, rounded to cents), and fullsend.runtime. Status is Ok: the span reports consumed usage, not the agent's outcome. There is no prompt/output content, no credentials, and no mlflow.* attributes. A record with no model spec is keyed unknown (provider and model unknown) rather than dropped.

How to query without double-counting. The parent agent span is a rollup: it keeps the parent identity and the iteration's fullsend.cost_usd, and is marked fullsend.usage.rollup=true. It does not carry gen_ai.usage.input_tokens / output_tokens / cache_*_input_tokens on mixed-model iterations, because backends that auto-sum those keys (MLflow; ADR 0050, 2026-08-18) would otherwise add the rollup to the components. reasoning_tokens has no per-model counterpart and stays on the agent span.

Billable tokens and cost live on the children (fullsend.usage.component=true). Filter on that flag — not on span name — when summing. Do not add the agent span's fullsend.cost_usd to the children's — sum the children directly instead of using the agent span's rollup as a stand-in for that sum (see the rounding caveat below). The root run span's fullsend.cost_usd is the run total and is custom-namespaced, so it is not auto-summed with GenAI usage. Across retry iterations, each agent span (and its children, if mixed) covers that iteration only; the root remains the run rollup.

The sum of component input, output, cache-creation, and cache-read tokens equals that iteration's run totals — those are exact integers, never rounded. Component fullsend.cost_usd values are each rounded to cents independently, the same rounding the agent and root spans apply (see Rounding and precision by surface) — not by rounding once after summing. Because of that, the sum of the children's rounded costs can differ from the agent span's rounded rollup by up to a cent per component: three components each costing $0.006 round individually to $0.01 (summing to $0.03), while the raw iteration total of $0.018 rounds to $0.02. This is the same class of discrepancy as the agent-vs-root rounding caveat below, one level down.

This is usage attribution, not the recursive sub-agent span expansion ADR 0050 deferred: the children are near-zero-duration, carry no turns or content, and are keyed by model spec rather than by Agent-tool call.

Cost data contract ​

Fullsend does not calculate inference cost from token counts or maintain a model-price table. Each runtime reports a USD cost value and fullsend records it as-is. This section defines the source, aggregation, rounding, and display behavior of that value across every output surface.

Runtime cost extraction ​

Each runtime extracts cost differently from its agent process:

RuntimeSourceExtraction
claudeClaude Code stream JSONReads result.total_cost_usd from the final result event — a single value covering the entire iteration
piPi assistant message streamSums usage.cost.total across all assistant messages in the iteration
opencodeOpenCode step streamSums step_finish.part.cost across all step-finish events in the iteration

The runtime-reported value includes whatever the provider prices — input tokens, output tokens, cache-creation tokens, cache-read tokens, and reasoning tokens. Fullsend has no visibility into the provider's pricing breakdown; it accepts the reported total.

Token counts (including cache_creation_input_tokens and cache_read_input_tokens) are recorded as separate telemetry attributes. They are not inputs to any fullsend-side cost calculation.

Cross-iteration aggregation ​

When a run has multiple iterations (validation loop retries), the runner sums raw costs:

run_total_cost_usd = sum(iteration.total_cost_usd for each completed iteration)

This sum is the single aggregate cost for the run, used by every downstream surface.

Rounding and precision by surface ​

The raw aggregate is a floating-point sum. Different output surfaces apply different precision:

SurfaceValuePrecisionExample
metrics.json total_cost_usdRaw aggregateFull float640.8234567
fullsend.cost_usd on agent spansPer-iterationRounded to cents: round(value × 100) / 1000.41
fullsend.cost_usd on mixed-model usage <model> spansThat model spec's share of the iterationRounded to cents: round(value × 100) / 1000.20
fullsend.cost_usd on the root run spanAggregateRounded to cents: round(value × 100) / 1000.82
Console (per-iteration)Per-iterationFour decimal places ($%.4f)$0.4117
Status comment footerAggregateTwo decimal places ($%.2f)$0.82

The root span's rounded cost is computed by rounding the raw aggregate sum. Because rounding happens after summing, the root span value can differ from the sum of individually rounded agent-span values. For example, two iterations at $0.414 and $0.415 round individually to $0.41 and $0.42 (sum $0.83), but the aggregate $0.829 rounds to $0.83 — or, with different fractional values, the aggregate may round differently than the sum of parts.

No pricing-table fallback ​

If a runtime does not report cost (returns zero or the field is absent), fullsend records zero. There is no fallback cost calculation from token counts. A missing runtime cost propagates as $0.00 on all surfaces.

Distinction from backend-derived cost estimates ​

Tracing backends may display their own cost estimates alongside fullsend.cost_usd. These are independent calculations:

  • MLflow estimates cost from token counts against its internal model table. This estimate excludes cache-creation and cache-read token pricing, which can dominate agent-run cost. See Tracing with MLflow — Cost column caveat.
  • Other LLM-aware backends may apply similar token-based estimates.

The authoritative cost for a fullsend run is always fullsend.cost_usd (on spans) or total_cost_usd (in metrics.json). Backend-derived estimates are informational and may diverge.

Output file format ​

run-telemetry.jsonl contains one JSON object per line. Each object is a complete OTLP TracesData message with hex-encoded trace/span IDs (per the OTLP JSON spec, not base64).

Non-finite float values (NaN, Infinity) are encoded as proto3 JSON strings ("NaN", "Infinity", "-Infinity").

The file is written synchronously per span. Spans are flushed to disk as they complete; the file is the forensic record for crashed runs. Every execute_tool span is one such line, so a tool-heavy iteration (a hundred or more id-bearing calls — Claude Code today) adds up to that many, capped at 1,024.

Cross-run trace correlation ​

Multi-agent pipelines (triage, code, review) propagate trace context via the TRACEPARENT environment variable (W3C Trace Context).

When a workflow dispatches a child run:

yaml
env:
  TRACEPARENT: ${{ steps.parent.outputs.traceparent }}

The child run's root span becomes part of the parent trace.

For separate workflow runs on the same work item (e.g. triage, code, review as independent GHA workflows), TRACEPARENT must be propagated manually. GitHub webhooks do not support custom trace headers.

Within a single work item, fullsend.work_item_id on the root run span is the correlation key for filtering related traces in a backend.

Operational behavior ​

  • Export timing: spans are exported live via the batch processor. On shutdown, the provider flushes remaining spans within a 5-second budget. A dead endpoint does not block the run.
  • Retry: the exporter retries on transient failures (HTTP 503, etc.) with an initial interval of 250 ms and a max interval of 2 s. The 5-second context deadline passed to tp.Shutdown bounds both retries and in-flight requests, so a persistently failing or hanging endpoint does not extend shutdown.
  • Crashed runs: completed spans already flushed mid-run reach the backend; spans in the batch buffer are lost. The local file remains.
  • Sampling: when TRACEPARENT has the sampled flag unset (-00), OTLP export is suppressed. The local file is still written.
  • Endpoint validation: the CLI validates the endpoint before creating the OTLP exporter. An endpoint is invalid if it cannot be parsed as a URL, has no scheme, uses a scheme other than http or https, or has no host (e.g. localhost:4318 instead of http://localhost:4318). When invalid, the CLI prints a warning to stderr and skips OTLP export; the local file exporter is unaffected. A valid signal-specific endpoint is not blocked by an invalid generic endpoint.
  • Private CAs: OTEL_EXPORTER_OTLP_CERTIFICATE points at a PEM bundle. No skip-verify option exists.

GHA workflow configuration ​

Managed workflows ​

All agent stages (triage, code, review, fix, retro, prioritize, harness) forward OTEL configuration. To enable export, set on the org (or repo) that hosts the fullsend caller workflows:

NameTypeRequiredPurpose
OTEL_EXPORTER_OTLP_TRACES_ENDPOINTVariableYesBackend's full traces URL (e.g. https://mlflow.example.com/v1/traces). Alternatively, set OTEL_EXPORTER_OTLP_ENDPOINT (the base URL without a signal path); managed workflows forward both variants.
OTEL_EXPORTER_OTLP_TRACES_HEADERSSecretYesComplete header string, auth and routing included (e.g. Authorization=Bearer%20<token>,x-mlflow-experiment-id=42).
OTEL_EXPORTER_OTLP_HEADERSSecretNoGeneric (non-signal-specific) OTLP headers. Same format as the traces variant. Useful when a single header set covers all signals.
OTEL_EXPORTER_OTLP_CERTIFICATEVariableNoPath to a PEM CA bundle for backends behind a private CA. Commit the bundle into the config repo (e.g. .fullsend/otel-ca.pem) and set the variable to that checkout-relative path.
OTEL_RESOURCE_ATTRIBUTESVariableNoStatic k=v,k=v trace tags. The value is used verbatim; ${{ github.* }} expressions evaluate only in workflow YAML, not in variables.
OTEL_SDK_DISABLEDVariableNoSet to true to disable all telemetry, including the local file exporter.
OTEL_INSTRUMENTATION_GENAI_CAPTURE_MESSAGE_CONTENTVariableNoSet to true to attach conversation content to agent spans (Level 3; see Content capture).

Installations scaffolded before OTEL support was added must also forward the secrets (add OTEL_EXPORTER_OTLP_TRACES_HEADERS and OTEL_EXPORTER_OTLP_HEADERS under secrets:) until the scaffold is re-synced: in the .fullsend repo's stage workflows (per-org), or in the fullsend shim workflow's dispatch job (per-repo).

Bring your own workflow ​

Add the environment variables to any job that runs fullsend run:

yaml
env:
  OTEL_EXPORTER_OTLP_ENDPOINT: "${{ vars.OTEL_EXPORTER_OTLP_ENDPOINT }}"
  OTEL_EXPORTER_OTLP_TRACES_ENDPOINT: "${{ vars.OTEL_EXPORTER_OTLP_TRACES_ENDPOINT }}"
  OTEL_EXPORTER_OTLP_TRACES_HEADERS: "${{ secrets.OTEL_EXPORTER_OTLP_TRACES_HEADERS }}"
  OTEL_EXPORTER_OTLP_HEADERS: "${{ secrets.OTEL_EXPORTER_OTLP_HEADERS }}"
  OTEL_RESOURCE_ATTRIBUTES: "${{ vars.OTEL_RESOURCE_ATTRIBUTES }}"
  OTEL_SDK_DISABLED: "${{ vars.OTEL_SDK_DISABLED }}"
  OTEL_INSTRUMENTATION_GENAI_CAPTURE_MESSAGE_CONTENT: "${{ vars.OTEL_INSTRUMENTATION_GENAI_CAPTURE_MESSAGE_CONTENT }}"
  OTEL_EXPORTER_OTLP_CERTIFICATE: "${{ vars.OTEL_EXPORTER_OTLP_CERTIFICATE }}"

Any variable and secret names work here; the values reach the exporter as-is. Consult your backend's documentation for the endpoint URL and authentication mechanism.

Eval measurements ​

After each managed agent run, fullsend eval-measure scores run-telemetry.jsonl in the same job (fail-open). Scores land in eval-measurements.jsonl beside telemetry when at least one new score is produced (tool-agnostic artifact). When OTEL_EXPORTER_OTLP_* is set, those scores also export as gen_ai.evaluation.result span events on the same TraceID (fail-open; does not rewrite run-telemetry.jsonl).

Today's scorers (starting with EM-001) read the Level 1/2 metadata contract of run-telemetry.jsonl — span tree and attributes, not prompt or completion bodies. That foundation is intentional: fitness scores must trust the trace before quality scores can. Planned: content-aware scorers that consume Level 3 prompt/completion capture once Level 3 is implemented — that is where the real quality signal lives. See Eval Measurements and ADR 0087.

See also ​