@Daniel Ballinger, some questions regarding Database.Cursor:
Type safety
Why is Database.Cursor not templated? According to the material I've seen, I can declare and use a cursor like:
Database.Cursor contactCursor = Database.getCursor('SELECT Id FROM Contact');List<Contact> contacts = contactCursor.fetch(0, 200);I would have expected to have needed to declare the cursor like:
Database.Cursor<Contact> contactCursor = ...;
If we don't have templating, what should happen if I do:
Database.Cursor contactCursor = Database.getCursor('SELECT Id FROM Contact');List<Account> accounts = contactCursor.fetch(0, 200);Re-used cursors, user mode and sharing
Cursors can be passed around through Apex code, and are serializable to allow them to be retained between transactions.
- If I get a cursor for a user mode query, what happens if that cursor is re-used in a transaction for a different user with different permissions?
- If I get a cursor in a without sharing class then perform the fetch in a with sharing class, is sharing applied against the results?
Happy path cursor expiration
Cursors live for up to 48 hours from what I understand. If I'm caching the cursor, given the cursor's current beta API, I understand I have to catch exceptions against the cursor method(s) to detect that it has expired.
Handling exceptions in a "happy path" like this, where essentially I just want to know the cursor is expired, is something of an anti-pattern (I've mentioned this with other APIs added "recently" too). Can we please have a simple method that tells me that the cursor is valid for the duration of my transaction (and make sure that if the cursor could expire during the transaction it is either treated as already expired or that it is kept alive to the end of the transaction)?
Fetching the final partial scope/chunk
From the AMA session it seems that if you ask for too many records from a given offset, an exception is thrown.
Could we have a variant of fetch that doesn't throw an exception and instead simply returns as many records as are available?
Fetching when records have been deleted
According to the AMA session, if records have been deleted, the number of records returned for a given "chunk" can be fewer than requested (or even 0) simply because one or more of the records that would have been returned has been deleted.
Could we have a variant of fetch that tries to return the number of records requested, even if records had been deleted since the cursor was obtained? This actually aligns with the previous point where fetch will finally just return as many records as it can, up to the scope size, regardless of how many records have been deleted and whether the cursor's underlying results have been exhausted.
This suggests, too, that the cursor should be able to return the "next record offset" through another method that can be called after a fetch. A value of -1 can indicate that there are no further records. A non-negative value could still result in 0 records being returned if all the remaining records have been deleted, but that's OK.
@Daniel Ballinger thanks for the feedback.
On some of the comments and questions you posed:
>> Happy path cursor expiration
> Ideally I'd like to be able to guarantee that the cursor would be valid for an entire transaction and that we could do a single upfront check for that. In reality I couldn't do that.
I would approach this by determining the maximum duration for a transaction in the context of execution, double it for good measure, then if the cursor is within this period of expiration treating it as expired.
>> Could we have a variant of fetch that doesn't throw an exception and instead simply returns as many records as are available?
> Can you please give me an example of how you would want to use that?
>> Could we have a variant of fetch that tries to return the number of records requested, even if records had been deleted since the cursor was obtained?
> Trying to keep track of those gaps over time if we are constantly reindexing would be challenging.
These are both related to how I would want to process data without having potentially wasted iterations outside the API.
It seems to me (in an implementation ignorant way) that the record set is determined on cursor creation as an ordered set of record IDs. If that's the case, keeping track of pagination is as simple as knowing the last processed/returned ID.
Like any iterator, I want my iteration logic to simply ask for the next value; it happens that the "next value" is a chunk of records due to the way you have organized the API. I would expect that chunk of records to be the same size each time, except when I get the final chunk where I will get between 0 and chunk size records back.
The skipping of now deleted records is something the cursor infrastructure should deal with rather than expecting every implementation to try to deal with this outside the API.
I realize this isn't a good usage example, given the limits on cursor usage, but imagine using a cursor for UI pagination. If I cannot guarantee that the "next value" has the right number of (or even any) records in it, I have to jump serious hoops to try to fetch enough records to satisfy the next "page", while then holding the detail of the possible "overspill" for the subsequent pagination step. Clearly this issue is less visible when applying an "infinite scroll" approach to data loading.
