DataStreamDefinition Metadata Type in SFDX

Architectural Overview#

In Salesforce Data Cloud, a Data Stream is the operational ingestion pipeline that pulls batch or streaming records from an external or internal source into a Data Lake Object (__dll). The DataStreamDefinition metadata type represents the declarative configuration of that ingestion pipeline—including the connector type, data source object reference, extraction parameters, and target lake storage binding.

For enterprise architects, DataStreamDefinition governs ingestion cadence, credit consumption, and data freshness SLAs. Whether ingesting real-time behavioral events via the Ingestion API or nightly batch files from Amazon S3, properly structuring DataStreamDefinition ensures predictable ingestion behavior across environments.

SFDX Directory and File Suffix#

Key Metadata Fields#

Declarative XML and Package.xml Examples#

DataStreamDefinition Metadata File#

<?xml version="1.0" encoding="UTF-8"?>
<DataStreamDefinition xmlns="http://soap.sforce.com/2006/04/metadata">
    <dataConnectorType>AmazonS3</dataConnectorType>
    <dataSource>AWS_S3_Enterprise_Lake</dataSource>
    <dataSourceObject>ERP_Order_Header_DSO</dataSourceObject>
    <isMissingFileFailure>false</isMissingFileFailure>
    <masterLabel>ERP Daily Orders Stream</masterLabel>
    <mktDataTranObject>ERP_Order_Header__dll</mktDataTranObject>
</DataStreamDefinition>

package.xml Manifest#

<?xml version="1.0" encoding="UTF-8"?>
<Package xmlns="http://soap.sforce.com/2006/04/metadata">
    <types>
        <members>ERP_Daily_Orders_Stream</members>
        <name>DataStreamDefinition</name>
    </types>
    <version>68.0</version>
</Package>

Monitoring Data Streams via SOQL and SFDX CLI#

While DataStreamDefinition is deployed via Metadata API, Data Cloud also exposes the runtime DataStream standard object in SOQL for operational health monitoring:

SELECT Id, Name, DataStreamStatus, ImportRunStatus,
       LastRefreshDate, LastNumberOfRowsAdded, TotalNumberOfRowsAdded
FROM DataStream
WHERE DataStreamStatus = 'ERROR' OR ImportRunStatus = 'FAILURE'
ORDER BY LastRefreshDate DESC

To retrieve or deploy the metadata definition using Salesforce CLI:

# Retrieve a DataStreamDefinition from your org
sf project retrieve start --metadata "DataStreamDefinition:ERP_Daily_Orders_Stream" --target-org data-cloud-prod

Senior Architect Best Practices and Gotchas#

• Prefer DataStreamTemplate for Packaging: In ISV or multi-org enterprise pipelines, deploying raw DataStreamDefinition files directly can fail if tenant-specific connector credentials are not yet initialized. Bundle streams as DataStreamTemplate items inside a Data Kit (DataPackageKitDefinition) and instantiate them after authenticating the connector.
• Credit Optimization on Refresh Mode: Use Upsert with an indexed timestamp or CDC watermark rather than Full Refresh on high-volume profile tables to minimize Data Services credit consumption.

Frequently asked questions

What is the difference between DataStreamDefinition and DataStreamTemplate?

DataStreamDefinition represents an instantiated ingestion pipeline in an org, whereas DataStreamTemplate is a reusable blueprint packaged inside a Data Kit to generate streams in subscriber orgs.

Where are DataStreamDefinition files stored in SFDX?

They are stored in the force-app/main/default/dataStreamDefinitions directory with the .dataStreamDefinition-meta.xml suffix.

Can I query DataStream runtime status via SOQL?

Yes. While DataStreamDefinition is the Metadata API definition, the runtime DataStream standard object can be queried via SOQL to inspect ImportRunStatus, LastRefreshDate, and TotalNumberOfRowsAdded.

What happens if I change a stream from Upsert to Full Refresh?

Full Refresh replaces the entire Data Lake Object dataset on each run, which can inadvertently delete historical engagement records if the upstream extract only contains delta rows.