Question

How to handle many new items between 2 Zaper requests

  • 11 January 2021
  • 3 replies
  • 32 views

Userlevel 2
Badge

Hi everyone.

Developing a zapier integraiton that consumes an GraphQl API, using cursor based pagination.

 

I have a hard time wrapping my head around the following scenario, and I hope you can help clarify.

Let’s say that the Zapier API sends a request every 15 minutes, and asks for a list of actions. The API responds with the latest 20 actions (i.e. page-size=20).

Simple scenario - one new action

In between 2 requests, a new action was returned by the API. Zapier correctly identifies this a new action, and the trigger fires away.

Advanced scenario - 25 new actions

In between 2 requests, 25 new actions are created. I somehow need to tell Zapier that we need to make another request. When I get the new response, I must inform Zapier that here we recognize an Id already processed.

 

How is the above accomplished in Zapier?


3 replies

Userlevel 4
Badge +6

Let’s call the things your trigger API is returning “objects”,  rather than “actions” as that has a specific meaning in Zapier integration world.  

The general approach is to return a big enough data set so that all the new objects created in the polling interval are in the result set returned by your endpoint, in a single page of data.  This is going to be the simplest implementation.  

Another straightforward approach is to use Rest Hooks.  Then de-duping and polling intervals are not a factor - you’ll send new data when you need to, as often as you need to, and we’ll consume it and process it.

Ok, so what if you’re stuck with polling and you’re stuck with sending a small page of data?  You don’t have an interface into the deduper to know what id’s have already been processed, so that’s not an avenue to figuring out whether or not you need a second request for more data.  What you might try is this: order the set of objects consistently; store the id of the most recent object returned in the request using z.cursor; when you get data back from your API check the result set to see if it contains the id stored in z.cursor - if it’s not there fetch another page until you reach that id; build up an array from all the requests and return it.  Be mindful you only have ~30 seconds in total to service all the requests.  This is more complicated than I like - you’ll need to consider all the edge cases, like, if the ‘last id’ actually gets deleted - you don’t want to page through your whole database.  (maybe using timestamp would be safer?).   I’d lean into one of the first two approaches first, before going this way. 

Note: even though “supports pagination” is only about triggers used for dynamic dropdowns, you’ll need to set it to true here, otherwise z.cursor will not be enabled in your trigger code.  

 

 

Userlevel 2
Badge

@Zane 

Thanks for taking the time to answer my question. In my humble opinion you should create an article for developers where you explain some of this information in a clear concise manner.

I develop the underlying GraphQl API - so I can implement Rest Hooks.

Based on your description of polling, I kind of conclude that pagination here is not recommended.

So in the next few days I will decide between:

  1. Increasing the page-size of each polling request substantially (to mitigate the problem).
  2. Implement Rest Hooks

Regarding Rest Hooks, do you have some blog/video/documentation walk-trough how that would look like in a Node.js Express application? When I walk through the online documentation, I’m still a bit confused 🙃

 

Thanks again!

Userlevel 4
Badge +6

You might have a look at http://resthooks.org/ for discussion of implementing the service side of Rest Hooks.  There are some examples linked.  I can’t recall if there’s one for Express specifically, and they aren’t GraphQL based, but will likely have some helpful info.  

 

Reply