My week started well before my normal business hours seeing messages from a team from a very important downstream system that listens for CDC events from Salesforce. No events had been coming through. For hours. Checking the Salesforce limits showed we weren't close to any eventing limits, yet messages just weren't coming through, even when we made test changes to records that should be delivering CDC events.
Ultimately a ticket with support returned the "Note that there is no defined SLA for CDC event delivery , and occasional delays are expected " which we'd seen before, but this time we also received:
"CDC uses shared resource on instance & when other Org publish tons of events, other customers' org may run into events delivery delay
We notice this phenomenon and continue to improve this performance issue."
There was no issue listed on
status.salesforce.comand I was surprised to see a delay of this many hours.
Questions I have:
Have others experienced this kind of delay in eventing throughput?
Do these delays only affect external subscribers or also Apex subscribers (which may make CDC as a method for asyncronous trigger logic problematic)?
What's the longest delay we can expect to see when leveraging CDC? (Enough of a delay and it doesn't seem as "near real time" as we're normally used to.)
CDC is built by Salesforce so is a shared service between customers. Would we have avoided this delay had we built on custom-defined triggered HVPE instead?
Thanks much to anyone who is able to provide any clarity or additional experience and info.
First, please accept my apologies for the delays you experienced. This is certainly not the standard we strive for, and my team has been working hard to resolve the underlying issues and prevent them from recurring. This specific technical issue last week impacted the processing of CDC events and there was a trust post but we resolved the delay over the weekend and got processing moving again and I suspect the trust post was resolved, however like a traffic jam there could still have been some orgs that were churning through their backlog.
To answer your questions when delays like this occur they do affect all subscribers. We've added a ton of alerting to notify us when the conditions for a delay are ripe so we can clear them out faster. Its hard to say what the longest delay would be as they are very situation specific.
This specific delay would have occurred on HVPE if you used publish after commit. If used publish immediate you would have not experienced this delay.
Please let me know if you have any other questions.
