CAP Node.js pagination/$skiptoken/nextLink problem

[martinstenzig]

The details of this problem and reproduction is available in this GitHub repo: https://github.com/MartinStenzig/cds-bug-6

Problem description

• I run a first query with a $top=1100 and receive a nextLink of "@odata.nextLink": "sapWorkOrder?$top=1100&$skiptoken=1000"
• When I run that next link I get the next 1000 even though I wanted "only" the initial 1100

Alternative Example (outline in repo)

• As an alternative I have create a more complex local scenario
• SAP data is mocked
• I run the query /sapWorkOrder?$select=MaintenanceOrder,MaintenanceOrderDesc&$top=150 (see [sync.http](sync.http))
• "@odata.nextLink": "sapWorkOrder?$select=MaintenanceOrder,MaintenanceOrderDesc&$top=150&$skiptoken=100

Expectation

• I would assume that the next link has skiptoken set to 1000, but $top set to 100
• While on the subject. It would be great if the next link or at least next link information could be exposed in either
  • the .run result i.e const {runResult, {skip, top})= await SAP_CONN.run(req.query) or
  • an adjusted req.query object that contains the updated skip and top information.
• At the moment there is no easy way that I know of to loop through all (i.e. 10k records) in the SAP service as the SAP nextlink SAP provided is not accessible.

• Prerequisites
• - A SAP S/4 on-premise system (21XX) containing more than 1,000 work orders in the system.
• - A (proxy) [service](./srv/sync.cds) that all it does is expose the S/4 backend API (in my case Maintenance Order) as service (sapWorkOrder)

[Dinu]

Definitely it's a bug if following $nextlink results in more records than specified by $top in the first query. Northwind service does reduce $top value in $nextlink.  But, $skiptoken and $nextlink are supposed to be opaque, it is not supposed to be interpreted by clients. So, deconstruction to {skip, top} as suggested may not be correct from the perspective of the protocol.

In CAP.js, unlike in CAP Java, $nextlink was available by the same name in the query result. Have you tried result['$nextlink'] to access this value? The trouble I faced with using $nextlink with CAP.js, is that you have to use the REST api for following it. If you use REST api the results are not interpreted; for example, dates remain in /Date(nnnn) format with V2. There is need of a facility for following the $nextlink without falling back to REST api.  

[martinstenzig]

@Dinu,

Thanks for confirming that you think it is a bug.

I tried result['$nextlink'] and after debugging through the code realized that the OData middleware is generating it after the handlers are executed in (sap/cds/libx/odata/middleware/read.js). 

Hope the CAP team can help with either a fix, guidance or documentation adjustment  as to how to handle 1k+ recordsets in backend systems.

[vansyckel · Accepted solution]

Hi @martinstenzig 

Thanks for pointing this out. We will provide a fix for the incorrect nextLink behavior, and I took "convenient paging in service consumption" as a feature request.

Best,
Sebastian

[nicorunge · Accepted solution]

Hi,

good news! This will probably also solve my problem I had. Is there any way to follow the feature request?

Thanks!
Nico

[vansyckel · Accepted solution]

Hi Nico,

I'm afraid we don't have external BLI tracking. I'll try to remember to comment here. But we'll also announce via our Release Notes.

Best,
Sebastian