Best practices for paginating large Data Extension datasets (87M+ records) via REST API

Hi SFMC Community, 

We're building an integration that syncs data from Data Extensions containing 87+ million records using the REST API (/data/v1/customobjectdata/key/{key}/rowset). 

Current Approach 

We're using offset-based pagination with $page and $pageSize parameters, along with a $filter on LastModifiedDate:

$filter=LastModifiedDate gte '2024-01-01' and LastModifiedDate lt '2024-12-01'$page=1&$pageSize=2500

We're observing response times increasing as page numbers grow: 

Page 1: ~1s 

Page 50: ~3s 

Page 100: ~5s 

Page 150+: ~6s+ 

We suspect this is due to offset-based pagination requiring SFMC to scan/skip all previous rows before returning the requested page. 

1. What are the recommended best practices for efficiently paginating through very large Data Extensions?
2. Are there any internal optimizations (indexes on LastModifiedDate, etc.) we should be aware of when designing our sync strategy?

@* Salesforce Developers *