Batch data sync

Not all integration is real-time. A great deal of enterprise data movement is bulk and periodic: nightly customer loads from an ERP, hourly product-catalogue refreshes, weekly exports to a data warehouse. The batch data synchronisation pattern addresses moving large volumes of records between Salesforce and other systems efficiently and on a schedule, and it is a distinct architectural choice from the real-time patterns.

The defining constraint is volume. Loading a million records through the normal REST API one at a time would be slow and would burn API limits, so this pattern uses the Bulk API (Bulk API 2.0), which is designed for high-throughput, asynchronous processing of large jobs in batches. On the tooling side, an ETL/middleware layer — MuleSoft, Informatica, a cloud data pipeline, or a custom job — typically orchestrates extraction from the source, transformation to the target's schema, and loading via the Bulk API.

Direction matters. Inbound sync brings external data into Salesforce (upserting on an External Id field so records match instead of duplicating). Outbound sync extracts Salesforce data to another system, often using a "last modified since" watermark to move only what changed — a delta or incremental sync — rather than reprocessing everything.

Batch sync design points:
  - Use Bulk API 2.0 for high volume (async, chunked jobs)
  - Upsert on an External Id to match records and avoid duplicates
  - Prefer delta (changed-since watermark) over full reload
  - Schedule off-peak; respect LDV selectivity on the Salesforce side
  - Plan error handling: capture per-record failures, retry, reconcile

Architectural considerations cluster around scale and reliability. Run large jobs off-peak to avoid contending with users. Ensure the Salesforce-side operations stay selective (the LDV lessons apply directly) so matching queries do not table-scan. And design robust error handling and reconciliation: bulk jobs partially fail, so capture failed records, retry transient errors, and reconcile record counts between systems.

As an exercise, design a nightly inbound customer sync: choose the External Id, decide full vs delta, pick the API, and outline how you would report and retry failures. The final lesson covers the pattern that increasingly ties everything together: event-driven architecture.