Monitoring Storage by Zapier to not run out of key space (What strategies work for you?)
I have one zap that submits a request to a backend system, and another zap that takes the asynchronous reply. And because the backend system doesn’t convey the parameters needed in the reply, I create an association in Storage by Zapier during the first call and do the lookup and during the reply.
So, create-key during call #1, and get and delete key during call #2.
Now, it’s possible that I don’t get a reply. In that case I need a way of finding the old keys and deleting them. Key Storage has a 500 key limit.
I need to expire them, so I’m looking for strategies. For instance, delete after 3 days because if the downstream system doesn’t get back to me by then we can assume it never will.
Suggestions?
Page 1 / 1
Has anyone else run into this issue before? @PaulKortman or @Saastronomical perhaps? Do you have any experience managing information in Storage?
Ooo this is a fun one.
There are a couple of ways to solve this but it’s mostly theoretical at this point because I’m not sure of your specific use case. For example I had a client once that all their storage usage was from 7am-7pm central, so daily at midnight (or 1am or whatever) I could clear all storage keys because I didn’t need them to last for longer than a workday.
On the other hand, you could look at using a list and pushing/popping values onto/off of that list and then deleting the list weekly or something like that.
And still, a third option is adding a webhook call to another zap that delays for a day or two and then checks for the storage key, use a filter to stop the zap if it doesn’t exist, and if it does exist delete the key.
And of course, you could combine these all together or make some combinations of the ideas. It really depends on how long you want to keep the key valid and when is a safe period to determine it’s gone.
@mrjcleaver let me know what of the above solutions sounds better for you and if you need a more spelled out recipe on making this work in a zap. I started this off by saying these are all theoretical because I wasn’t planning on doing a step by step for each idea… and some might work for you and others might not, so let us know which you’re interested in and if you need more help.
@Danvers After writing this and reviewing, I’m wondering if the “Remove All Values” and the “Remove Value” Actions in storage do or don’t actually delete the keys. I know that these functions remove the values/empty the keys, but I can only assume they also remove the keys as well because the above solutions have worked for me in the past. If it does remove the Keys as well then I think the documentation should explain that (or explain that it does not remove Keys as well.
Well, the easiest thing would be if Zapier implemented a key expire timestamp.
Otherwise we can hack it ourselves with another set of storage objects organized by date - I’d be interested seeing something like that.
I’ll come back to comment on your other suggestions this afternoon, thanks!
