in reply to Paging with REST::Client?

When I was shelling out to curl, the way I was handing this was using _max_results and _paging with the REST calls (and checking if I needed to loop ove the _paging calls). So, I'm wondering if there's an easy way to pull this off with REST::Client? It doesn't look like it, however I can pass in my own LWP::UserAgent object -- is there a way to have LWP::UserAgent handle the paging under the hood? Guessing not based on my supersearch and duckduckgo searches, but thought I'd ask. Assuming not, is there another Perl module designed to handle this while providing a clean, simple interface?

Hi

So are _paging _max_results curl options? Http headers? Something else?

Replies are listed 'Best First'.
Re^2: Paging with REST::Client?
by Argel (Prior) on Jan 13, 2020 at 16:42 UTC
    I'm not an expert on RESTful APIs, so I'm not sure what specifically is supplying the functionality, but it's on the server side of things. It's something passed in via the URL. Here's an example URL from some debug output I have lying around: 
    https://nios.mycompany.com/wapi/v2.7/range?network=10.20.30.0/24\&star
+t_addr=10.20.30.40\&_return_fields%2B=disable,extattrs,network\&_pagi
+ng=1\&_max_results=500\&_return_as_object=1

    [download]

    Elda Taluta; Sarks Sark; Ark Arks
    My deviantART gallery
[reply]
[d/l]

      just a shot in the dark, the server may be sending data keyed on "address" (10.20.30.40). In order to get to the next 500 results you have to use the last "address" you got from the last results page and send this as "start_addr".

      sort of! In the link below, the server returns a reference to send it back to get the next page: perhaps you can figure it out by searching for paging in this page: https://ipam.illinois.edu/wapidoc/additional/sample.html?highlight=paging

[reply]
        Nice guess at the third party app!! Looking at my confusing old code, I don't think the query/search needs to change. You just have to pass in _next_page_id from the previous query in as _page_id in the new query. I'm not 100% on that as I haven't looked at this code in several months and there's a reason I wanted to get away from having to manually do the paging. It really could benefit from an OOP approach, and I was hoping there was a module that already handled it.

        Elda Taluta; Sarks Sark; Ark Arks
        My deviantART gallery
[reply]

      Hi try adding &page=2


      The way forward always starts with a minimal test.
[reply]
[d/l]
        Sorry, I should have been clear -- that's an example from my old Curl code. REST::Client isn't doing any of the paging.

        Elda Taluta; Sarks Sark; Ark Arks
        My deviantART gallery
[reply]

      I'm not an expert on RESTful APIs, so I'm not sure what specifically is supplying the functionality, but it's on the server side of things. It's something passed in via the URL. Here's an example URL from some debug output I have lying around:

      Hi

      ;) Have you read what you wrote?

      Surely you know exactly the actions you took, cause you say

      "passed in via URL"

      Well, there you go, the module does accepts URLs https://metacpan.org/pod/REST::Client#GET-(-$url,-%5B%25$headers%5D-)

      So what happened when you did that (pass via url)?

      "paging" ain't nothing but a loop (foreach) where you ask for some more

[reply]
        In theory I could pass in the paging parameters and see what REST::Client does with it, but I was hoping it supported it so it could be transparent to me -- i.e. just keep puling in thew data behind the scenes for me. Otherwise, it's back to some ugly code to deal with it. Maybe I can handle it slightly better than my Curl based scripts, but it adds a lot of complexity to the code just to handle something that feels like it should be part of the feature set of the module (e.g. a ->next function, or the abiltiy to tell it to slurp everything in).

        The "don't know" part -- I know the server side is handling the paging, but not sure what specifically is handling it (module in the web server, some process the WAPI calls are passed to, etc.).

        Elda Taluta; Sarks Sark; Ark Arks
        My deviantART gallery
[reply]

In Section Seekers of Perl Wisdom