Hi SFMC Community,
We're building an integration that syncs data from Data Extensions containing 87+ million records using the REST API (/data/v1/customobjectdata/key/{key}/rowset).
Current Approach
We're using offset-based pagination with $page and $pageSize parameters, along with a $filter on LastModifiedDate:
$filter=LastModifiedDate gte '2024-01-01' and LastModifiedDate lt '2024-12-01'$page=1&$pageSize=2500
We're observing response times increasing as page numbers grow:
Page 1: ~1s
Page 50: ~3s
Page 100: ~5s
Page 150+: ~6s+
We suspect this is due to offset-based pagination requiring SFMC to scan/skip all previous rows before returning the requested page.
- What are the recommended best practices for efficiently paginating through very large Data Extensions?
- Are there any internal optimizations (indexes on LastModifiedDate, etc.) we should be aware of when designing our sync strategy?
#Marketing Cloud
1 resposta
Hi, sorry for the inconvenience. If you don't receive a response here, we recommend reaching out to our support team for further assistance. You can do so by visiting https://help.salesforce.com/s/articleView?id=000393090&language=en_US&type=1
