datarobot-external-agent-monitoring

DataRobot External Agent Monitoring Skill

This skill helps you instrument any AI agent — regardless of framework or deployment environment — to send OpenTelemetry telemetry (traces, logs, metrics) to DataRobot. It also creates a shell deployment in DataRobot as the telemetry routing target.

Quick Start

Most common use case: Instrument an existing agent project for DataRobot monitoring

1. Point the agent at your project directory
2. The skill detects your framework and existing OTel setup
3. It generates instrumentation code and creates a DataRobot shell deployment
4. Your agent sends traces, logs, and metrics to DataRobot

Example: "Instrument my agent in ./my_agent for DataRobot monitoring"

When to use this skill

Use this skill when you need to:

• Monitor an external AI agent in DataRobot
• Add OpenTelemetry tracing to an agent project
• Send agent traces, logs, and metrics to DataRobot
• Create a DataRobot deployment to receive external agent telemetry
• Instrument a Google ADK, LangChain, LangGraph, CrewAI, LlamaIndex, PydanticAI, or any Python agent

Supported Frameworks

| Framework | Detection | OTel Strategy | |-----------|-----------|---------------| | Google ADK | google-adk in deps or google.adk in imports | Lazy trace injection via callback (ADK overwrites TracerProvider) | | LangChain / LangGraph | langchain or langgraph in deps/imports | Auto-instrumentor + standard setup | | CrewAI | crewai in deps/imports | Auto-instrumentor + standard setup | | LlamaIndex | llama-index or llama_index in deps/imports | Auto-instrumentor + standard setup | | PydanticAI | pydantic-ai or pydantic_ai in deps/imports | Standard setup (respects global TracerProvider) | | Generic Python | None of the above detected | Manual span instrumentation |

Workflow

Follow these steps in order. Present the plan to the user and wait for approval before executing.

Step 1: Detect & Analyze

1. Read the project's dependency file (requirements.txt, pyproject.toml, setup.py, poetry.lock, or uv.lock)
2. Scan Python source files for framework imports
3. Check for existing OTel setup (look for opentelemetry imports, existing TracerProvider/LoggerProvider/MeterProvider configuration)
4. Identify the framework using the detection table above
5. Read the corresponding framework reference file from the frameworks/ directory next to this SKILL.md:
6. Google ADK → frameworks/google-adk.md
7. LangChain/LangGraph → frameworks/langchain-langgraph.md
8. CrewAI → frameworks/crewai.md
9. LlamaIndex → frameworks/llamaindex.md
10. PydanticAI → frameworks/pydantic-ai.md
11. Generic Python → frameworks/generic-python.md

Step 2: Check Prerequisites

1. Check if DATAROBOT_API_TOKEN env var is set. If not, ask the user to provide it.
2. Check if DATAROBOT_ENDPOINT env var is set. If not, ask the user (default: https://app.datarobot.com/api/v2).
3. Derive DATAROBOT_OTEL_ENDPOINT automatically: if DATAROBOT_ENDPOINT ends with /api/v2, strip it and append /otel (e.g., https://app.datarobot.com/api/v2 → https://app.datarobot.com/otel).
4. Check if the datarobot Python SDK is available. If not, install it: pip install datarobot.
5. Check if OTel packages are already in the project's dependencies.

Security note: Never echo API tokens or .env file contents into chat transcripts or logs. Use environment variables or CI secrets for credential management. If credentials are accidentally exposed, rotate them immediately.

Step 3: Present Plan

Tell the user what you detected and present the changes you will make:

• Framework detected (or generic Python)
• Existing OTel setup found (if any)
• New dependencies to add
• New files to create (dr_otel_config.py, and optionally dr_agent_metrics.py for frameworks with custom metrics)
• Existing files to modify (agent entrypoint, dependency file)
• Shell deployment to create in DataRobot

Wait for user approval before executing. If the user has already given explicit consent to implement or deploy, that counts as approval — no need to re-ask.

Step 4: Execute

1. Add dependencies to the project's dependency file:
2. opentelemetry-sdk
3. opentelemetry-api
4. opentelemetry-exporter-otlp-proto-http
5. Framework-specific packages (see framework reference file)

1. Generate dr_otel_config.py using the generic pattern below, adapted per the framework reference file.

1. Wire into agent entrypoint: Add import and call to configure_otel() at startup. Follow the framework reference file for specific wiring instructions (auto-instrumentors, callbacks, etc.).

1. Generate dr_agent_metrics.py if the framework reference file specifies custom metrics callbacks.

1. Create shell deployment: Run the helper script:

``bash python <skill_scripts_dir>/create_shell_deployment.py \ --name "<project_name> Monitoring" \ --description "OTel telemetry sink for <framework> agent" ``

The script automatically enables prediction row storage and automatic association ID generation on the deployment.

1. Report results: Show the deployment ID and a copy-paste env var block for the user's runtime:

``bash export DATAROBOT_API_TOKEN="<token>" export DATAROBOT_ENTITY_ID="deployment-<id>" export DATAROBOT_OTEL_ENDPOINT="<otel_endpoint>" ``

Step 5: Verify & Provide Runtime Instructions

1. Optionally run the verification script:

``bash DATAROBOT_API_TOKEN=<token> \ DATAROBOT_ENTITY_ID=deployment-<id> \ DATAROBOT_OTEL_ENDPOINT=<endpoint>/otel \ python <skill_scripts_dir>/verify_otel_connection.py ``

1. Provide the user with the env vars to set in their deployment environment:
2. DATAROBOT_API_TOKEN — DataRobot API key
3. DATAROBOT_ENTITY_ID — deployment-<id> (from shell deployment creation)
4. DATAROBOT_OTEL_ENDPOINT — {DATAROBOT_ENDPOINT}/otel

1. Explain what they'll see in DataRobot:
2. Data Exploration > Traces: Span hierarchy (agent orchestration, LLM calls, tool calls)
3. Data Exploration > Logs: Structured logs correlated with traces via traceId
4. Data Exploration > Metrics: Custom metrics (request count, latency, LLM calls, tool calls)

Generic OTel Configuration Pattern

This is the core configure_otel() function to generate for every project. Framework-specific files layer additional setup on top.

Critical rules:

1. Always pass endpoint= and headers= directly to exporters — NEVER use OTEL_EXPORTER_OTLP_* env vars (some frameworks detect these and create conflicting providers)
2. Be additive — add DataRobot as an additional span processor to any existing TracerProvider, don't replace it
3. Use SimpleSpanProcessor (not Batch) to avoid flush-before-shutdown issues
4. Use DELTA temporality for metrics (required by DataRobot)

Generated dr_otel_config.py template:

"""DataRobot OpenTelemetry configuration.

Configures traces, logs, and metrics export to DataRobot's OTel endpoint.
Call configure_otel() at application startup, before any agent code runs.

Required env vars at runtime:
    DATAROBOT_API_TOKEN      - DataRobot API key
    DATAROBOT_ENTITY_ID      - deployment-<deployment_id>
    DATAROBOT_OTEL_ENDPOINT  - https://<your-instance>.datarobot.com/otel
"""

import logging
import os

from opentelemetry import metrics, trace
from opentelemetry._logs import set_logger_provider
from opentelemetry.exporter.otlp.proto.http._log_exporter import OTLPLogExporter
from opentelemetry.exporter.otlp.proto.http.metric_exporter import OTLPMetricExporter
from opentelemetry.exporter.otlp.proto.http.trace_exporter import OTLPSpanExporter
from opentelemetry.sdk._logs import LoggerProvider, LoggingHandler
from opentelemetry.sdk._logs.export import SimpleLogRecordProcessor
from opentelemetry.sdk.metrics import Counter, Histogram, MeterProvider, ObservableCounter
from opentelemetry.sdk.metri