OpenTelemetry

1. Introduction

mirai provides comprehensive OpenTelemetry (otel) tracing support for observing asynchronous operations and distributed computation.

When the otel and otelsdk packages are installed and tracing is enabled, mirai automatically creates spans to track the lifecycle of daemon management, async operations, and task execution.

This enables detailed monitoring of:

• Task submission and completion times
• Daemon lifecycle and performance
• Error tracking and debugging
• Distributed tracing across network boundaries

2. Automatic Tracing Setup

Tracing is automatically enabled when:

1. The otel and otelsdk packages are installed and available
2. OpenTelemetry tracing is configured and enabled (see https://otelsdk.r-lib.org/reference/collecting.html)

No additional configuration is required - mirai will automatically detect the presence of OpenTelemetry and begin tracing.

3. Span Types and Hierarchy

mirai creates several types of spans to represent different operations:

3.1 Core Span Types

daemons / daemons->reset - Root span for a compute profile

• Kind: internal
• Attributes: url, n (number of daemons), dispatcher (true/false), compute_profile
• Lifecycle: Created when daemons are set, and when daemons are reset

daemon / daemon->end - Individual daemon process span

• Kind: internal
• Attributes: url
• Lifecycle: Created when a daemon process starts and when it ends

mirai_map - Parallel map operation span

• Kind: internal
• Lifecycle: Encompasses the entire map operation across multiple mirai tasks

mirai - Client-side async task span

• Kind: client
• Attributes: mirai.id (unique task identifier)
• Lifecycle: Created when mirai() is called, ended as soon as it returns

daemon->eval - Server-side task evaluation span

• Kind: server
• Lifecycle: Tracks for the duration of actual mirai evaluation on the daemon

3.2 Span Relationships and Context Propagation

The spans form a distributed structure that traces the complete lifecycle of async operations:

daemons (compute profile - top level)

daemon (daemon process 1 - top level)
...
daemon (daemon process N - top level)

mirai_map (top level) ──link→ daemons
├── mirai (task 1) ──link→ daemons
│   └── daemon->eval ──link→ daemon
├── mirai (task 2) ──link→ daemons
│   └── daemon->eval ──link→ daemon
└── mirai (task N) ──link→ daemons
    └── daemon->eval ──link→ daemon
    
mirai (top level) ──link→ daemons
└── daemon->eval ──link→ daemon

daemon->end (daemon process 1) ──link→ daemon
...
daemon->end (daemon process N) ──link→ daemon
daemons->reset ──link→ daemons

Context Propagation: The context is automatically packaged with each mirai() call and extracted on the daemon side, enabling proper parent-child relationships across process boundaries.

Span Links: Tasks are linked to their compute profile’s daemons span on the client side, and to each daemon span on the server side, showing exactly where each evaluation happened. When daemons are reset and the respective daemons terminated, these events are recorded in new spans which link back to the original spans.

4. Status and Error Tracking

daemon->eval spans automatically track the success or failure of operations:

Status Values:

• 'ok' or 'unset' - completed successfully
• 'error', with description 'miraiError' - failed with an error
• 'error', with description 'miraiInterrupt' - was interrupted

5. Monitoring and Observability

The OpenTelemetry spans provide rich observability into mirai operations:

Performance Monitoring:

• Track task execution times from submission to completion
• Monitor daemon utilization and load balancing
• Identify bottlenecks in distributed computation

Error Analysis:

• Correlate errors with specific tasks and daemons
• Track error rates across different types of operations
• Debug issues in distributed environments

Distributed Tracing:

• Follow task execution across network boundaries
• Understand the complete lifecycle of async operations
• Correlate client-side requests with server-side execution

6. Integration with Observability Platforms

mirai’s OpenTelemetry implementation works seamlessly with any OpenTelemetry-compatible observability platform, including:

• Grafana
• Pydantic Logfire
• Jaeger
• Any of those listed at https://otelsdk.r-lib.org/reference/collecting.html.

The tracer name used by mirai is "org.r-lib.mirai", making it easy to filter and identify mirai-related traces.