DataConnectorS3 Metadata Type in SFDX

Architectural Overview#

The DataConnectorS3 metadata type defines the declarative integration configuration between Salesforce Data Cloud and Amazon Web Services (AWS) Simple Storage Service (S3). In modern enterprise lakehouse architectures, Amazon S3 commonly serves as the primary staging repository for enterprise data warehouses, customer transaction logs, and external ERP exports.

DataConnectorS3 establishes the connection metadata envelope, specifying target AWS S3 bucket identifiers, bucket directories, object prefix patterns, compression algorithms, and file formatting specifications. Once deployed, Data Cloud Data Streams bind to this connector to execute automated batch ingestion schedules into staging Data Lake Objects (DLOs).

Directory and File Suffix Conventions#

In SFDX source format, DataConnectorS3 metadata definitions reside in the dataConnectorS3s folder:

Key Metadata Fields#

Declarative XML and Manifest Examples#

Source Format XML (Enterprise_DataLake_S3.dataConnectorS3-meta.xml)#

<?xml version="1.0" encoding="UTF-8"?>
<DataConnectorS3 xmlns="http://soap.sforce.com/2006/04/metadata">
    <developerName>Enterprise_DataLake_S3</developerName>
    <masterLabel>Enterprise Data Lake S3 Connector</masterLabel>
    <bucketName>corp-lakehouse-data-cloud-sync</bucketName>
    <directoryPath>/inbound/customers/</directoryPath>
    <fileFormat>CSV</fileFormat>
    <compressionType>GZIP</compressionType>
    <status>Active</status>
</DataConnectorS3>

Manifest (package.xml)#

<?xml version="1.0" encoding="UTF-8"?>
<Package xmlns="http://soap.sforce.com/2006/04/metadata">
    <types>
        <members>Enterprise_DataLake_S3</members>
        <name>DataConnectorS3</name>
    </types>
    <version>68.0</version>
</Package>

Retrieval and Deployment Commands#

Use the Salesforce CLI to retrieve and deploy DataConnectorS3 metadata:

# Retrieve all S3 connector definitions
sf project retrieve start --metadata DataConnectorS3

# Retrieve a specific S3 connector by developer name
sf project retrieve start --metadata DataConnectorS3:Enterprise_DataLake_S3

# Deploy S3 connector metadata to target environment
sf project deploy start --metadata DataConnectorS3:Enterprise_DataLake_S3

Senior Architect Best Practices and Gotchas#

• Cross-Account AWS IAM Roles: Deploying the metadata definition creates the connection structure, but security handshake requires configuring AWS IAM AssumeRole policies with Salesforce’s 12-digit AWS account ID and External ID. Never attempt to hardcode static AWS credentials in CI/CD pipelines.
• Parquet Schema Compatibility: When ingesting Parquet format from S3, ensure upstream data engineering pipelines produce snappy or uncompressed Parquet files with fixed schemas matching your Data Stream field specifications.
• Data Kit Encapsulation: When deploying Data Cloud assets across staging and production orgs, package the DataConnectorS3 alongside linked DataSource and DataPackageKitDefinition components to preserve data lineage.

Frequently asked questions

What is the primary role of DataConnectorS3 in Data Cloud?

It defines the connection metadata and storage parameters for ingesting batch data files from Amazon S3 buckets into Data Cloud.

Where are DataConnectorS3 files stored in an SFDX project?

They are stored in force-app/main/default/dataConnectorS3s/ with the file suffix .dataConnectorS3-meta.xml.

Are AWS IAM secret access keys stored directly in metadata XML?

No. Sensitive AWS credentials and secret keys are never exported in metadata and must be configured in Data Cloud Setup or IAM trust roles.

Which file formats can DataConnectorS3 pipelines process?

Data Cloud S3 connectors natively process CSV, Parquet, and TSV files with gzip compression support.