salesforce-agentforce-stdm-observer-skill

salesforce-agentforce-stdm-observer-skill

Production observability for Agentforce agents via STDM and Data Cloud. This skill is a live evidence reader, not a configuration reviewer. It queries session telemetry, quality scores, and action traces to answer: "Is my agent working correctly right now?" It does not modify agents, configurations, or any org data.

Adaptation note: Query mechanics in this skill are adapted from the observing-agentforce skill published by Salesforce in the forcedotcom/sf-skills repository (Apache-2.0). Vanguard-specific additions include the T1 least-privilege contract, structured audit envelope, explicit aggregate-only output policy, and the handoff routing model.

Verify-before-merge notice: All Agentforce, STDM, Data Cloud, and Einstein AI feature names evolve rapidly. Validate all product references, DMO field names, and API structures against current official Salesforce documentation before use in production.

When This Skill Owns the Task

Use salesforce-agentforce-stdm-observer-skill when the goal is live production observability for an Agentforce agent:

• "How is my Order Service agent performing over the last 7 days?"
• "Show me faithfulness and answer relevance scores for the Support agent"
• "Are there action invocation errors for the Case Routing agent?"
• "What is the session abandonment rate for the HR Self-Service agent?"
• "Show me STDM session telemetry for the last 48 hours"
• "Is the agent hallucinating? What do the quality scores say?"

Delegate elsewhere when:

| Situation | Skill to use | |---|---| | Static review of agent configuration (topics, actions, instructions) | salesforce-agentforce-risk-review-skill | | Agent is misconfigured and must be changed | T3 — requires human approval via salesforce-live-guard-agent | | Compliance/privacy review of session data handling | salesforce-compliance-privacy-agent | | General SOQL record queries unrelated to Agentforce | salesforce-soql-explorer-skill | | Metadata export and schema inspection | salesforce-metadata-fetcher-skill | | Authoring or editing .agent files | developing-agentforce (forcedotcom/sf-skills) | | Agent performance degrades and a fix must be deployed | Route through salesforce-live-guard-agent for human approval |

Required Context to Gather First

Before executing any STDM query, confirm all of the following. Ask if missing.

1. Target org alias — the --target-org value recognized by sf org list.

Never accept a raw instance URL or session token.

1. Org type — production or sandbox. Flag production orgs; apply stricter

output scrutiny and confirm the Connected App allowlist authorizes this alias.

1. Agent name(s) to observe — the agent's display label (MasterLabel) or

API name. Resolve against the org before querying (see Step 2).

1. Time window — ISO 8601 start and end timestamps (UTC). Default to last

7 days if not specified.

1. Specific metrics requested — sessions summary, faithfulness, answer

relevance, action telemetry, or all. Prefer getAggregatedMetrics as the first call to bound scope.

1. Sensitivity classification — does the org serve Health Cloud,

Financial Services Cloud, or other regulated verticals? If yes, extra redaction scrutiny applies; route any anomalies involving session content to salesforce-compliance-privacy-agent before sharing externally.

1. Data Cloud provisioned? — STDM requires Data Cloud and the Agentforce

Activity data stream enabled. Confirm before executing (see Step 1).

Recommended Workflow

Step 1 — Verify org connectivity and Data Cloud availability

Confirm the org alias is reachable and Data Cloud is provisioned:

sf org display --target-org <alias>

Then probe the Data Cloud data spaces endpoint to confirm STDM DMOs are available:

sf api request rest "/services/data/v63.0/ssot/data-spaces" \
  --target-org <alias>

Note: sf api request rest is a beta command — do not add --json (that flag is unsupported and causes an error in this command).

Decision logic:

• If the command fails (404 or permission error), assume DATA_SPACE=default

and log it as an assumption.

• Filter to only status: "Active" data spaces.
• If exactly one active data space exists, use it automatically.
• If multiple exist, show the list and ask the user which to use.
• Store the selected name as DATA_SPACE for all subsequent steps.

If Data Cloud is unavailable, stop and inform the user:

STDM requires Data Cloud with "Agentforce Activity" data stream active.

Navigate to Setup → Data Cloud → Data Streams to verify. This skill

cannot proceed without STDM. For local trace analysis without Data Cloud,

see the observing-agentforce skill from forcedotcom/sf-skills.

Step 2 — Resolve target agent and confirm AiAgentTag

Resolve the user-provided agent name to the exact MasterLabel used by STDM. Field names and exact object names are drift-prone — run this query and use the returned values, not the user-provided string:

sf data query \
  --query "SELECT Id, MasterLabel, DeveloperName FROM GenAiPlannerDefinition WHERE MasterLabel LIKE '%<user-provided-name>%' OR DeveloperName LIKE '%<user-provided-name>%'" \
  --target-org <alias> \
  --result-format json

Store:

• AGENT_MASTER_LABEL — for STDM findSessions agent filter
• PLANNER_ID — the Salesforce record ID for this agent (redact in output)

If the query returns no results: The agent does not exist in this org. Show the full list of agents and ask the user to identify the target.

Step 3 — Query AiAgentTagAssociation for tagging context

Retrieve tagging metadata to understand what quality evaluation definitions are configured. This confirms the org has quality scoring enabled before querying scores:

sf data query \
  --query "SELECT Id, AiAgentTagId, EntityId, EntityType FROM AiAgentTagAssociation LIMIT 10" \
  --target-org <alias> \
  --result-format json

If AiAgentTagAssociation returns no rows, quality scores may not be configured. Note this in the output and proceed with session-level metrics only.

Note: AiAgentTag, AiAgentTagDefinition, and AiAgentTagAssociation are Tooling API objects. Use --use-tooling-api if the standard SOQL path returns an "object not found" error:

sf data query \
  --query "SELECT Id, AiAgentTagId, EntityId, EntityType FROM AiAgentTagAssociation LIMIT 10" \
  --target-org <alias> \
  --use-tooling-api \
  --result-format json

Step 4 — Query STDM sessions via aggregated metrics

Start with getAggregatedMetrics via the AgentforceOptimizeService Apex helper class to get the health dashboard before drilling into individual sessions. This is the most efficient first call and avoids fetching session content.

Full Apex service deployment steps and invocation patterns are documented in references/stdm-queries.md (adapted from forcedotcom/sf-skills observing-agentforce).

String result = AgentforceOptimizeService.getAggregatedMetrics(
    '<DATA_SPACE>',
    '<START_ISO>',
    '<END_ISO>',
    50,
    '<AGENT_MASTER_LABEL>'
);
System.debug('STDM_RESULT:' + result);

sf apex run --json --file /tmp/stdm_metrics.apex --target-org <alias>

Parse the result using the DEBUG|STDM_RESULT: pattern (see references/stdm-queries.md). The aggregated metrics return:

• total_sessions, total_turns, avg_quality_score
• avg_faithfulness, avg_answer_relevance, avg_context_precision
• abandonment_rate, deflection_rate, escalation_rate
• end_type_counts, quality_distribution, top_intents
• unavailable_dmos — list of DMOs that could not be queried

If findSessions returns empty: No production sessions exist in this date window. Check that the date range is correct and that the agent is actively receiving traffic. Consider widening the window.

Step 5