Observability & Telemetry
MeetLoyd implements OpenTelemetry (OTel) for standardized observability of AI agent executions, providing distributed tracing and metrics following industry-standard semantic conventions.
Overview
The telemetry system provides:
- Distributed Tracing: Track requests across agent executions, LLM calls, and tool invocations
- Metrics Collection: Counters and histograms for executions, tokens, latency, and costs
- Gen AI Semantic Conventions: Industry-standard
gen_ai.*attributes for LLM observability - OTLP Export: Compatible with any OpenTelemetry-compatible backend
Architecture
The telemetry pipeline flows from the MeetLoyd platform (agent execution spans, LLM call spans, tool execution spans) through the OpenTelemetry SDK (BatchSpanProcessor, MetricReader) to the OTLP/HTTP Exporter, which delivers data to your observability backend (Grafana, Honeycomb, Datadog, Jaeger, etc.).
Traced Operations
Agent Execution Spans
Root spans created for each agent execution:
| Attribute | Description |
|---|---|
| gen_ai.agent.id | Unique agent identifier |
| gen_ai.agent.name | Human-readable agent name |
| gen_ai.request.model | LLM model used |
| gen_ai.provider.name | Provider (anthropic, openai, etc.) |
| meetloyd.tenant.id | Tenant identifier |
| meetloyd.user.id | User who triggered execution |
LLM Call Spans
Child spans for each LLM API call:
| Attribute | Description |
|---|---|
| gen_ai.operation.name | Operation type (chat, completion) |
| gen_ai.request.model | Model requested |
| gen_ai.request.max_tokens | Max tokens parameter |
| gen_ai.request.temperature | Temperature parameter |
| gen_ai.usage.input_tokens | Input tokens consumed |
| gen_ai.usage.output_tokens | Output tokens generated |
| gen_ai.response.id | Provider response ID |
| gen_ai.response.finish_reasons | Why generation stopped |
Tool Execution Spans
Child spans for each tool invocation:
| Attribute | Description |
|---|---|
| gen_ai.tool.name | Tool identifier |
| gen_ai.tool.call.id | Unique call identifier |
| gen_ai.tool.type | Tool type (builtin, custom) |
Metrics
Counters
| Metric | Description |
|---|---|
| gen_ai.agent.executions | Total agent executions |
| gen_ai.llm.calls | Total LLM API calls |
| gen_ai.tool.calls | Total tool invocations |
| gen_ai.tokens.input | Total input tokens |
| gen_ai.tokens.output | Total output tokens |
| gen_ai.errors | Total errors by type |
Histograms
| Metric | Description |
|---|---|
| gen_ai.agent.duration | Execution duration (ms) |
| gen_ai.llm.latency | LLM call latency (ms) |
| gen_ai.tool.latency | Tool execution latency (ms) |
| gen_ai.cost.usd | Execution cost (USD) |
| gen_ai.tokens.per_execution | Tokens per execution |
Supported Backends
MeetLoyd's OTLP export is compatible with:
| Backend | Notes |
|---|---|
| Grafana Cloud (Recommended) | Fully managed, generous free tier, cloud-agnostic |
| Honeycomb | Trace-first observability |
| Datadog | Full-stack monitoring |
| Axiom | Log analytics |
| AWS X-Ray | Via OpenTelemetry Collector |
| Azure Monitor | Via Azure Monitor Exporter |
| Google Cloud Trace | Via GCP exporter |
| Any OTLP-compatible collector | Universal |
Security Considerations
What's Captured
Telemetry data includes agent and user identifiers, model names and parameters, token counts and costs, and execution timing.
What's NOT Captured
Telemetry does not contain prompt content or responses, user messages, tool input/output data, or PII/sensitive business data.
Network Security
All OTLP exports use HTTPS. Authentication headers are supported. No data is exported when telemetry is disabled.
Compliance
OpenTelemetry supports SOC 2 (audit trail of AI operations), ISO 27001 (monitoring and incident detection), and GDPR (no PII in telemetry data by design).
Graceful Degradation
When telemetry is disabled or misconfigured, all tracing functions return no-ops with zero performance overhead. The application continues normally. A warning is logged if telemetry is enabled without an endpoint.
Additional Observability
Beyond OpenTelemetry, MeetLoyd provides:
- Structured Logging: Pino-based JSON logs with correlation IDs
- Chain of Thought Logging: Full reasoning capture in database
- Agent Run Tracking: Execution history in the agent runs table
- Audit Logs: Security-relevant events for compliance
See Audit Logs and SIEM Integration for more.