/agentforce-offline-experimentation-run

name: agentforce-offline-experimentation-run
description: "Use this skill when users want to execute this workflow when a user wants to test how a different LLM model would handle existing Agentforce agent conversations. Trigger when users mention agentforce, offline, experimentation."
metadata:
  relatedSkills:
  - dx-apexguru-scan
  version: '1.0'
  cliTools:
  - tool:
    - curl
    semver: '>=7.0'
  - tool:
    - jq
    semver: '>=1.6'
  - tool:
    - python3
    semver: '>=3.8'
  - tool:
    - sf
    semver: '>=2.0.0'

Managing Agentforce Offline Experimentation

When to Use This Skill

Execute this workflow when a user wants to test how a different LLM model would handle existing Agentforce agent conversations. Fetches session traces from Data Cloud STDM, replays prompts via the SFAP AI Gateway REST API with structured tool calling, scores across four metrics (topic agreement, function name accuracy, function argument accuracy via LLM-as-judge, text response quality via LLM-as-judge), and produces CSV + HTML reports with SAFE/REVIEW/UNSAFE verdict badges. Requires Data Cloud with Agentforce Activity data streaming enabled, an authenticated org, and a target model ID (llmgateway__ prefix for SFAP replay). Estimated runtime: 10-30 minutes for 10 sessions.

Workflow

discover-data-space → verify-stdm-availability → discover-agents → resolve-agent-label → list-available-models (if user needs help choosing) → find-sessions → get-conversation-details → get-llm-step-details → obtain-sfap-token → replay-turn-prompts → [score-exact-matches, judge-function-args, judge-text-responses] → generate-csv-report → generate-html-report. WORKFLOW GATE: Do not proceed past Phase 0 until org alias (auto-discover from authenticated orgs), agent API name (auto-discover from STDM), and target model ID (ask user) are all confirmed. SFAP JWT expires after ~2 hours. For 20+ session replays, refresh the token between batches. Each SFAP replay call takes ~1-3 seconds; keep under 20 sessions for reasonable wall-clock time (~10-30 minutes). The scoring and judging steps (Phase 3) can run in parallel once replay completes. Exact-match scoring is instant; LLM judge calls add ~1-2 seconds per turn.

Critical Constraints

**Preconditions:**

• Data Cloud is provisioned and enabled in the org (check: `DataCloudProvisioned`)
• Agentforce Activity data streaming is active in Data Cloud (check: `AgentforceActivityStream`)
• Einstein LLM / GenAI Gateway is configured in the org (check: `EinsteinLlmEnabled`)
• User has an authenticated session to the target org (check: `OrgAuthenticated`)

Operations Reference

Operations grouped by purpose. Use these as the building blocks for the workflows above.

Summary

| Operation | Purpose | Status | Call | Depends on | |-----------|---------|--------|------|------------| | `discover-data-space` | read | implemented | `getDataSpaces` | — | | `verify-stdm-availability` | validate | implemented | `queryCdpAnsiSqlV2` | `discover-data-space` | | `discover-agents` | read | implemented | `queryCdpAnsiSqlV2` | `verify-stdm-availability` | | `resolve-agent-label` | read | implemented | `queryGenAiPlannerDefinition` | `discover-agents` | | `list-available-models` | read | implemented | `queryAiModel` | — | | `find-sessions` | read | implemented | `queryCdpAnsiSqlV2` | `verify-stdm-availability`, `discover-agents` | | `get-conversation-details` | read | proposed | `queryCdpAnsiSqlV2` | `find-sessions` | | `get-llm-step-details` | read | proposed | `queryCdpAnsiSqlV2` | `get-conversation-details` | | `obtain-sfap-token` | validate | proposed | `obtainSfapJwt` | — | | `replay-turn-prompts` | read | proposed | `sfapChatGenerations` | `get-llm-step-details`, `obtain-sfap-token` | | `score-exact-matches` | read | proposed | `computeExactMatchScores` | `replay-turn-prompts` | | `judge-function-args` | read | proposed | `sfapChatGenerations` | `score-exact-matches`, `obtain-sfap-token` | | `judge-text-responses` | read | proposed | `sfapChatGenerations` | `replay-turn-prompts`, `obtain-sfap-token` | | `generate-csv-report` | write | proposed | `generateCsvReport` | `score-exact-matches`, `judge-function-args`, `judge-text-responses` | | `generate-html-report` | write | proposed | `generateHtmlReport` | `generate-csv-report` |

Dependency graph

graph TD
  discover_data_space["discover-data-space (read)"]
  verify_stdm_availability["verify-stdm-availability (validate)"]
  discover_agents["discover-agents (read)"]
  resolve_agent_label["resolve-agent-label (read)"]
  list_available_models["list-available-models (read)"]
  find_sessions["find-sessions (read)"]
  get_conversation_details["get-conversation-details (read)"]
  get_llm_step_details["get-llm-step-details (read)"]
  obtain_sfap_token["obtain-sfap-token (validate)"]
  replay_turn_prompts["replay-turn-prompts (read)"]
  score_exact_matches["score-exact-matches (read)"]
  judge_function_args["judge-function-args (read)"]
  judge_text_responses["judge-text-responses (read)"]
  generate_csv_report["generate-csv-report (write)"]
  generate_html_report["generate-html-report (write)"]
  discover_data_space --> verify_stdm_availability
  verify_stdm_availability --> discover_agents
  discover_agents --> resolve_agent_label
  verify_stdm_availability --> find_sessions
  discover_agents --> find_sessions
  find_sessions --> get_conversation_details
  get_conversation_details --> get_llm_step_details
  get_llm_step_details --> replay_turn_prompts
  obtain_sfap_token --> replay_turn_prompts
  replay_turn_prompts --> score_exact_matches
  score_exact_matches --> judge_function_args
  obtain_sfap_token --> judge_function_args
  replay_turn_prompts --> judge_text_responses
  obtain_sfap_token --> judge_text_responses
  score_exact_matches --> generate_csv_report
  judge_function_args --> generate_csv_report
  judge_text_responses --> generate_csv_report
  generate_csv_report --> generate_html_report

Read ope