Question

Is it possible to create a shared queue across multiple Zaps?

  • 16 November 2022
  • 10 replies
  • 425 views

Userlevel 1
Badge

We’re looking to move part of the API responsibility out of our back-end and into Zapier. To mitigate issues related to race conditions, we want to create a shared queue which ensures that Zaps that run concurrently all point to a single queue and release queued events once the Zap run is complete in the source Zap. 

Currently, we have a number of Zaps that point to various endpoints in our back-end. Concurrent events can result in issues related to race conditions. In an attempt to solve this, all those Zaps call the same Sub-Zap in which a Delay After Queue step has been setup, with a delay of 30 seconds. After the delay, the Sub-Zap returns to the source Zaps, and the source Zaps then finishes its sequence of steps. This allows us to neatly queue events from various Zaps in the same queue.

However, this does not entirely eliminate the possibility of running into race condition issues. 

So my question is this: Is there some way to ensure that a Zap run (run B) does not continue its sequence of steps until a previous Zap run (run A) has completed its sequence of steps? 

In this manner, a queue of events only continues to the next event in the queue once a prior Zap run has been concluded.   


This post has been closed for comments. Please create a new post if you need help or have a question about this topic.

10 replies

Userlevel 7
Badge +12

Hi @MartinKjeldP!

It is possible for Zaps to share queues created by the Delay by Zapier action. To do that, use the same queue name for both queues in the two separate Zaps and the items will share one queue. 

Although this answers part of your question, I’m not sure if it addresses everything because it’s not possible to specify that all items from one Zap should be added to the queue before all of the items for another Zap. Will creating one queue for both Zaps work for what you need?

If you need some more help trying to figure out this one, could you share some more information about how/when each Zap triggers? Thanks!

Userlevel 7
Badge +14

Hi @MartinKjeldP 

Good question.

Depending on how your Zaps are configured, you may want to chain Zaps together using Webhooks to ensure they process in the desired order.

Userlevel 1
Badge

@Danvers Thank you for the reply. I was unaware that it is possible to use the same queue just by using the same queue title across Zaps. I think perhaps this bit of information is lacking from the article about delays. This at least removes the Sub-Zap step that I had described in the above. I will have to test this and stress the Zap with many concurrent events. 

@Troy Tessalone Thank you for your reply, too. With regards to chaining Zaps together user webhooks: I am unsure how this would work. Our setup is something like this: 

  1. Changes for customers in our CRM trigger a number of webhooks, depending on which CRM sub-system is edited. For instance, one webhook is triggered on customer field changes, whereas another webhook is triggered if the customer’s account access/plans are created/edited.
  2. A number of Zaps catch the webhooks. Their immediate next step is to continue to the shared queue (right now, this is set up using a Delay After Queue step in Sub-Zap, which all of the Zaps call). 
  3. When each event passes through the queue, the Sub-Zap returns an event to the respective originating Zap, which then continues with its sequence of steps. 
  4. Depending on the state in both our back-end and our CRM, the Zaps branch out using Paths to create/POST or update/PUT users and accounts in our back-end. This is where issues with race conditions may arise, i.e., if several webhooks are triggered from our CRM at the same time and attempt to create/POST a new user/account in our back-end. 
    1. It should be noted that when the Zap POSTs to our back-end, it retrieves an account/user ID, which is then updated for the respective account/contact in our CRM. The Zap branches to POST and PUT paths depending on whether this ID is set in our CRM. 

In this event, although it may be difficult to assess given the details provided, how would we be able to ensure that they process in the desired order?

Userlevel 7
Badge +14

@MartinKjeldP 

The general concept is like this…

Zap 1

  1. Trigger: ???
  2. Action: POST Webhook
    1. Use the webhook URL generated from Zap 2, Step 1

 

Zap 2

  1. Trigger: Webhook - Catch Hook
    1. Use the webhook URL generated here in Zap 1, Step 2
  2. Action: ???
Userlevel 7
Badge +14

@MartinKjeldP 

Another concept might be to use an intermediary database such as Airtable.

  • Airtable has Views
  • Views have Filters (e.g. Field X = True AND Field Y = True)
  • Views can be used to trigger Zaps
  • Views can be used to trigger Airtable Automations
  • Airtable Automations have Run Script actions (on Pro+ plans) that can be used to trigger a Zap webhook

 

So it’s possible to only have Zaps enter a View when conditions are met giving you more control over the flow.

Userlevel 1
Badge

@Troy Tessalone 

I do not see how chaining webhooks together will solve the issue of race conditions. If Zap 1 is triggered multiple times within a few seconds, Zap 2 will also be triggered multiple times within a few seconds, which can lead to issues with race conditions.

Could you provide more details with regards to how chaining webhooks can ensure that multiple Zap runs are completed in a controlled sequence, thereby reducing or entirely eliminating race condition issues?

Userlevel 7
Badge +14

@MartinKjeldP

For each Zap Run of Zap 1, if you want Zap 2 to process after Zap 1, then chaining Zap 1 to Zap 2 will ensure that Zap 2 only processes after Zap 1, thus creating a controlled order of operations.

Userlevel 5
Badge +7

Hi there @MartinKjeldP - just checking in here, it looks like you’ve had some robust comments with Troy and Danvers, let us know if you were able to build what you were looking for or if you had any other questions or comments. 

Best,

Rachael

Userlevel 1
Badge

Hi there @MartinKjeldP - just checking in here, it looks like you’ve had some robust comments with Troy and Danvers, let us know if you were able to build what you were looking for or if you had any other questions or comments. 

Best,

Rachael

 

Hello. I had posted a comment as a reply to @Troy Tessalone’s latest comment, which looks to still be in review with the community moderators. 

The input provided did not solve the issue of race conditions, as described in the mentioned comment that is under review. 

Userlevel 5
Badge +7

Hi @MartinKjeldP - that’s odd, I am an admin in this Community and I can’t seem to find any post in review on our end. Sorry about that. @Troy Tessalone - maybe you can share a bit more about the process you described above about chaining Zaps to rely on the previous Zap completing in order to run? I’d also be curious to see what that might look like. 

 

Best,

Rachael