observing-agentforce

Agentforce Observability

Improve Agentforce agents using session trace data and live preview testing.

Three-phase workflow:

• Observe -- Query STDM sessions from Data Cloud (if available), OR run test suites + preview with local traces as fallback
• Reproduce -- Use sf agent preview to simulate problematic conversations live
• Improve -- Edit the .agent file directly, validate, publish, verify

Platform Notes

• Shell examples below use bash syntax. On Windows, use PowerShell equivalents or Git Bash.
• Replace python3 with python on Windows.
• Replace /tmp/ with $env:TEMP\ (PowerShell) or %TEMP%\ (cmd).
• Replace jq with python -c "import json,sys; ..." if jq is not installed.

Routing

Gather these inputs before starting:

• Org alias (required)
• Agent API name (required for preview and deploy; ask if not provided)
• Agent file path (optional) -- path to the .agent file, typically force-app/main/default/aiAuthoringBundles/<AgentName>/<AgentName>.agent. Auto-detect if not provided.
• Session IDs (optional) -- analyze specific sessions; if absent, query last 7 days
• Days to look back (optional, default 7)

Determine intent from user input:

• No specific action -> run all three phases: Observe -> surface issues -> ask if user wants to Reproduce and/or Improve
• "analyze" / "sessions" / "what's wrong" -> Phase 1 only, then suggest next steps
• "reproduce" / "test" / "preview" -> Phase 2 (run Phase 1 first if no issues in hand)
• "fix" / "improve" / "update" -> Phase 3 (run Phase 1 first if no issues in hand)

Resolve agent name

Before any STDM query, resolve the user-provided agent name against the org to get the exact MasterLabel and DeveloperName:

sf data query --json \
  --query "SELECT Id, MasterLabel, DeveloperName FROM GenAiPlannerDefinition WHERE MasterLabel LIKE '%<user-provided-name>%' OR DeveloperName LIKE '%<user-provided-name>%'" \
  -o <org>

• MasterLabel = display name used by STDM findSessions and Agent Builder UI (e.g. "Order Service")
• DeveloperName = API name with version suffix used in metadata (e.g. "OrderService_v9")
• The --api-name flag for sf agent preview/activate/publish uses DeveloperName without the _vN suffix (e.g. "OrderService")

Store these values:

• AGENT_MASTER_LABEL -- for findSessions() agent filter
• AGENT_API_NAME -- DeveloperName without _vN suffix, for sf agent CLI commands
• PLANNER_ID -- the Salesforce record ID for this agent

Locate the .agent file

Step 1 -- Search locally:

find <project-root>/force-app/main/default/aiAuthoringBundles -name "*.agent" 2>/dev/null

If the user provided an agent file path, use that directly. Otherwise, search for files matching AGENT_API_NAME.

Step 2 -- If not found locally, retrieve from the org:

sf project retrieve start --json --metadata "AiAuthoringBundle:<AGENT_API_NAME>" -o <org>

Known bug: sf project retrieve start creates a double-nested path: force-app/main/default/main/default/aiAuthoringBundles/.... Fix it immediately after retrieve:

if [ -d "force-app/main/default/main/default/aiAuthoringBundles" ]; then
  mkdir -p force-app/main/default/aiAuthoringBundles
  cp -r force-app/main/default/main/default/aiAuthoringBundles/* \
    force-app/main/default/aiAuthoringBundles/
  rm -rf force-app/main/default/main
fi

Step 3 -- Validate the retrieved file:

Read the .agent file and verify it has proper Agent Script structure:

• system: block with instructions:
• config: block with developer_name:
• start_agent or subagent blocks with reasoning: instructions:
• Each subagent should have distinct instructions: content (not identical across subagents)

Store the resolved path as AGENT_FILE for Phase 3.

Phase 0: Discover Data Space

Before running any STDM query, determine the correct Data Cloud Data Space API name.

sf api request rest "/services/data/v63.0/ssot/data-spaces" -o <org>

Note: sf api request rest is a beta command -- do not add --json (that flag is unsupported and causes an error).

The response shape is:

{
  "dataSpaces": [
    {
      "id": "0vhKh000000g3DjIAI",
      "label": "default",
      "name": "default",
      "status": "Active",
      "description": "Your org's default data space."
    }
  ],
  "totalSize": 1
}

The name field is the API name to pass to AgentforceOptimizeService.

Decision logic:

• If the command fails (e.g. 404 or permission error), fall back to 'default' and note it as an assumption.
• Filter to only status: "Active" entries.
• If exactly one active Data Space exists, use it automatically and confirm to the user: "Using Data Space: <name>".
• If multiple active Data Spaces exist, show the list (label + name) and ask the user which to use.

Store the selected name value as DATA_SPACE for all subsequent steps.

Prerequisite check: STDM DMOs

After deploying the helper class (step 1.0), run a quick probe to verify the STDM Data Model Objects exist in Data Cloud:

sf apex run -o <org> -f /dev/stdin << 'APEX'
ConnectApi.CdpQueryInput qi = new ConnectApi.CdpQueryInput();
qi.sql = 'SELECT ssot__Id__c FROM "ssot__AiAgentSession__dlm" LIMIT 1';
try {
    ConnectApi.CdpQueryOutputV2 out = ConnectApi.CdpQuery.queryAnsiSqlV2(qi, '<DATA_SPACE>');
    System.debug('STDM_CHECK:OK rows=' + (out.data != null ? out.data.size() : 0));
} catch (Exception e) {
    System.debug('STDM_CHECK:FAIL ' + e.getMessage());
}
APEX

If STDM_CHECK:FAIL: STDM is not activated. Inform the user and switch to Phase 1-ALT:

STDM (Session Trace Data Model) is not available in this org. To enable: Setup -> Data Cloud -> Data Streams and verify "Agentforce Activity" is active. Proceeding with fallback: test suites + local traces.

If STDM_CHECK:OK, proceed to Phase 1 (STDM path).

Phase 1-ALT: Observe Without STDM (Fallback Path)

When STDM is not available, use test suites and sf agent preview --authoring-bundle with local trace analysis.

| Data source | When to use | Pros | Cons | |---|---|---|---| | STDM (Phase 1) | Historical production analysis | Real user data, volume | Requires Data Cloud, 15-min lag | | Test suites + local traces (Phase 1-ALT) | Dev iteration, orgs without STDM | Instant, full LLM prompt, variable state | Preview only, no real user data |

1-ALT.1 Run existing test suite (if available)

sf agent test list --json -o <org>
sf agent test run --json --api-name <TestSuiteName> --wait 10 --result-format json -o <org> | tee /tmp/test_run.json
JOB_ID=$(python3 -c "import json; print(json.load(open('/tmp/test_run.json'))['result']['runId'])")
sf agent test results --json --job-id "$JOB_ID" --result-format json -o <org>

1-ALT.2 Derive test utterances from .agent file (if no test suite)

If no test suite exists, derive utterances: one per non-entry subagent (from description: keywords), one per key action, one guardrail test, one multi-turn test.

1-ALT.3 Preview with --authoring-bundle (local traces)

Run each test utterance through preview to generate local trace files:

sf agent preview start --json --authoring-bundle <BundleName> -o <org> | tee /tmp/preview_start.json
SESSION_ID=$(python3 -c "import json; print(json.load(open('/tmp/preview_start.json'))['result']['sessionId'])")

sf agent preview send --json --session-id "$SESSION_ID" --authoring-bundle <BundleName> \
  --utterance "$UTT" -o <org> | tee /tmp/preview_response.json

sf agent preview end --json --session-id "$SESSION_ID" --authoring-bundle <BundleName> -o <org>

Trace file location: .sfdx/agents/{BundleName}/sessions/{sessionId}/traces/{planId}.json

1-ALT.4 Local trace diagnosis

| Issue type | Trace command | |---|---| | Subagent misroute | `jq -r '.plan[] \| select(.type=="NodeEntryStateStep") \| .dat