AiEvaluationDefinition Metadata Type in SFDX

Architectural Overview#

Deterministic Apex code relies on @IsTest unit tests to guarantee 75%+ coverage, but autonomous AI agents require behavioral evaluation. The AiEvaluationDefinition metadata type powers the Agentforce Testing Center, allowing architects to define golden test datasets (customer utterances, expected GenAiPlugin topic selections, expected GenAiFunction invocations, and response quality metrics) as declarative metadata.

Storing AiEvaluationDefinition in SFDX lets your CI/CD pipeline deploy golden test suites alongside agent changes into every staging sandbox.

SFDX Directory & File Suffix#

Key Metadata Fields#

Declarative XML & package.xml Examples#

Sample .aiEvaluationDefinition-meta.xml#

<?xml version="1.0" encoding="UTF-8"?>
<AiEvaluationDefinition xmlns="http://soap.sforce.com/2006/04/metadata">
    <description>Golden regression evaluation suite verifying order lookup, RMA initiation, and off-topic guardrail refusal.</description>
    <masterLabel>Order Support Agent Regression Suite</masterLabel>
    <subjectName>Enterprise_Order_Support_Agent</subjectName>
    <subjectType>Agent</subjectType>
</AiEvaluationDefinition>

Sample package.xml Manifest#

<?xml version="1.0" encoding="UTF-8"?>
<Package xmlns="http://soap.sforce.com/2006/04/metadata">
    <types>
        <members>Order_Support_Agent_Regression_Suite</members>
        <name>AiEvaluationDefinition</name>
    </types>
    <version>68.0</version>
</Package>

SFDX CLI Retrieval & Deployment#

Retrieve and deploy AiEvaluationDefinition suites via the Salesforce CLI:

# Retrieve all AI Evaluation Definitions from the org
sf project retrieve start --metadata "AiEvaluationDefinition:*"

# Deploy the regression evaluation suite to a UAT sandbox
sf project deploy start --metadata "AiEvaluationDefinition:Order_Support_Agent_Regression_Suite"

Senior Architect Best Practices & Gotchas#

• Test Adversarial & Edge Utterances: Don’t only test happy-path utterances. Include prompt-injection attempts, ambiguous multi-intent questions, and out-of-scope requests in your AiEvaluationDefinition to verify that the planner routes to the off-topic guardrail rather than hallucinating actions.
• Seed Synthetic Test Data First: If the actions under evaluation query Order or Case records via Flow or Apex, run a pre-test data seeding script in your UAT sandbox so action execution outputs match your evaluation assertions.

Frequently asked questions

What is AiEvaluationDefinition used for in Salesforce?

It defines automated evaluation test suites in the Agentforce Testing Center to measure whether an AI agent selects the right topic, calls the expected actions, and produces accurate responses.

Where are AiEvaluationDefinition files stored in SFDX?

They are stored in the force-app/main/default/aiEvaluationDefinitions directory with the .aiEvaluationDefinition-meta.xml file extension.

Why is AiEvaluationDefinition critical for Agentforce CI/CD?

Because large language models are non-deterministic, modifying a single topic description or action instruction can alter routing behavior; automated evaluations catch regressions before production.

How do I retrieve AiEvaluationDefinition components with Salesforce CLI?

Run sf project retrieve start --metadata AiEvaluationDefinition to download all agent evaluation suites into your project.