💃 Importing data with the Bulk API 🕺

Good morning, Salesforce Nerds! We all know Salesforce offers a lot in the way API’s. 💪 

I mean we’re talking REST, SOAP, Metadata, GraphQL, Pub/Sub, just to name a few.

Each serve their own purpose and offer us a ton of functionality.

One of the more interesting API’s we can play with as technical consultants is called the Bulk API.

If you’ve got a project where you need to migrate a bunch of data in our out of Salesforce then this API is for you! 🫵 

Table of Contents

Importing data with the Bulk API

• What is the Bulk API?

• How to use it

• Platform limits

What are we talking about?

What is the Bulk API?

Just a note - we’re going to be exploring v2.0. ✍️ 

This is an API based on based on REST principles that the mothership optimized for large-scale datasets.

You can perform any CRUD operation asynchronously. 🔥 

Meaning that you submit your request and come back later for the results after the process is done executing in the background.

Let’s see how ingesting data with this tool works … 👇️ 

What do I do with it?

How to use it

It’s actually pretty easy … it’s really just a few steps.

✅ Create a job - generates a job in your org that represents a bulk operation. Salesforce will give you back a jobId.

✅ Upload job data - use the jobId and send another request with the data in a CSV format.

✅ Close the job - use the jobId again to tell the platform that you’re done uploading data for this job.

🪄 At this point, Salesforce is doing it’s magic. The data you’ve sent is divided into multiple batches of 10k records and processed by the Bulk API engine.

✅ Poll the job - using the jobId to check the status of the job. How many records processed successfully or unsuccessfully? How many records are still waiting?

✅ Get the successful results - send in the jobId here to get a list of your successful records - including any new Id’s.

✅ Get the failed results - this one will give you back the records that failed, along with exception details for troubleshooting.

Halt! There be dragons here!

Platform limits

As with everything in the cloud we need to be careful. Especially when we’re dealing with LDV. ⚠️ 

The official documentation spells it out well.

Some of them include:

• Daily limit: You can process up to 100 million records per 24-hour period using Bulk API 2.0.

• Job limit: You can have up to 100,000 jobs in any state at any given time. Delete completed or aborted jobs to free up space for new jobs.

• Data limit: You can upload up to 150 MB of data per job.

• Batch limit: Salesforce creates a separate internal batch for every 10,000 records in your job data. Each batch can take up to 10 minutes to process. If a batch fails or times out, Salesforce will retry it up to 10 times. If a batch still cannot be processed after 10 retries, the entire job will fail and the remaining data will not be processed.

• Concurrency limit: You can have up to 25 concurrent jobs running at any given time.

• Query limit: You can query up to 15 GB of data per job. The query results must be retrieved within 7 days of job completion.

Soul Food

Today’s Principle

and now....Salesforce Memes