What are some considerations when loading data from multiple objects on a daily basis at scheduled times?
The master data for different objects is stored in a SharePoint folder in the form of CSV files. Each CSV file contains data for various objects (Accounts, Products, Invoices, etc.). There are around 7-8 objects in total. Each object is scheduled to load data at different intervals of time based on dependencies (based on the order in which data is upserted)
Mulesoft picks the file from SharePoint and upserts into Salesforce. When MuleSoft upserts data into Salesforce, there are Apex triggers and record-trigger flows that run on all the objects. The automations must run and cannot be bypassed because they maintain data integrity among different objects. What considerations should I keep in mind when loading data in bulk? Data volume is high for transactional objects (invoice - 10k, invoice items - 100,000). This data volume is initial data load though. Delta might be less than this.
These are some of my considerations:
- Record lock issues
- Automations need to be bulkified
- owner of the record should be at the top of role hierarchy
Anything else?
#Data Management #MuleSoft #Automation #Salesforce Developer #Integration #Sales CloudI assumed the data integration through Mulesoft and few terminology is in context to that, Let me clarify
reply has been framed for Mulesoft prospective It might not fully relevant if you are only seeing salesforce prospective.
1) What do you mean by transactional try block?
ANS: It need to be maintained transaction for dependent object. the transaction can be handle using try scope refer detail document here
2) I cannot disable the automations since they need to run when the records are created/updated.
ANS :- It's for initial data load only where the size of data may be very larger. It's for performance prospective.
3) I think the MuleSoft team would use the MuleSoft connector to load the data. I’m not sure if they would use Bulk API v2 specifically but I guess this is the API that is suitable for importing large records right?
Mukesh: yes it's for mulesoft developer
4) What is Vcore? I haven’t heard this before.
Mukesh: It's again for Mulesoft developer . it's computational resources allocated to mule application while deploying the application
5) Rollback/data loss prevention is carried out by Mulesoft so I’m not sure what needs to be done at Salesforce side.
Mukesh: You right it for Mulesoft team.
