About Zapier Community
New to our Community? Get to know more about our space and how to make the most of your time here.

Code of Conduct Community FAQ's Zapier Common Questions What to Expect

Get Started With Zapier
New to Zapier? We have got you covered with our Zapier 101 Resources.

Zapier Essentials

Get Support
Have a question? Let's get you an answer!

How Do I...? Troubleshooting Code & Webhooks Developer Zone

Other Help Resources
Wait, there's more! Check out these Community sourced spaces for additional resources.

Zapier Help Center Hire an Expert

Zapier Products
Learn more about Zapier's new products.

Zapier Tables Zapier Interfaces

Zapier Reading
All of our best learning resources can be found here.

Featured Articles Show & Tell AI Hub Zapier Tools Hub

Learning Groups
Excited to meet other builders? Check out our interest areas full of doers, dreamers, and in-betweeners.

User Groups

Other Learning Resources
Some of our favorite additional ways to upskill your Zapier knowledge.

Zapier Learn Zapier Blog Zapier Webinars

Product Updates
Check out what is new and upcoming with Zapier with our regular product updates.

Product Updates

Early Access Program
Want to join Early Access? Learn more here.

Best answer

My S3 file uploads are getting truncated!

A
Userlevel 1
Badge +1

Frankly,  I am having a host of problems with the S3 zap on the Upload file.   
1) There is a omission in the s3 connect dialog that essentially prevents you from creating a S3 connection to the US-East-2 region.  The drop-down does not have this option.  If you had buckets in that region,  it basically means you cannot upload anything using this Zap.   Zapier apparently hasn’t prioritized this fix very high - at least there is no ETA on a fix.    In any case,  I got around this by moving my bucket to us-west-1, a workaround that took me some 3-4 hours of work and is probably infeasible for anybody with a complicated AWS deployment on us-east-2.
2) A much more serious and showstopper issue is that the upload zap seems to be randomly chopping up and trunctating the file being uploaded.   This is the file loaded with the zap - Note the size: 89.2 kb
 

 

This is the same file that has been uploaded through S3 console.   Note the size is 181 kb.

Essentially, thanks to the truncation,  my .docx and .pdf files are essentially corrupted and unusable in S3.

I am a little baffled by how such a basic error is happening on a vanilla upload to S3.  I am also assuming this will be a widespread problem unless I am missing something obvious.
The original file is in a Google Drive if that is relevant (this also correctly shows 181Kb size)

Can folks comment on their usage and whether anybody else is having this problem and what you have done to fix it…

My zap is about as basic as it gets.
1. It is triggered by a new line being created in a Google sheet.  The Google sheet has the info for the upload step, in terms of s3 bucket, s3 key and the source file URL (Google Drive)
2. Second step is the S3 zap. which should be doing the upload.   The zap upload success message also shows the wrong size (about 91,0xx bytes) which is about 89.2

icon

Best answer by Troy Tessalone 11 March 2024, 05:23

View original

10 replies

Troy Tessalone
Userlevel 7
Badge +14

Hi @anatesanStream 

For us to have full context, post screenshots with how the steps are outlined and configured.

 

###

 

Help articles to reference for using AWS S3 in Zaps: https://zapier.com/apps/amazon-s3/help

 

I Can't Add to a Specific Bucket (400 error)

Is your bucket in a different region than the one you selected when connecting your S3 account to Zapier? If so, you will need to reconnect and select the matching region.

Note that often (depending on the security policy of the S3 account you have connected to Zapier), we are able to see buckets from regions other than the selected region, but we are not able to interact with them. For example, if you have a bucket in US East (Ohio) (which is us-east-2), you may be able to select that bucket in a drop-down, but if you selected us-east-1 when you connected your account, we won't be able to upload files to that bucket.

AWS has documentation on their region names here if you aren't sure which numerical region your bucket resides in: https://docs.aws.amazon.com/general/latest/gr/rande.html#s3_region

If you are still having trouble after this, please drop us an email so we can troubleshoot further with you!

⚡ Troy Tessalone - AutomationAce.com | Premier Certified Zapier Expert | #1 Zapier Community Contributor
A
Userlevel 1
Badge +1

Zapier’s S3 connection dialog shows a list of regions that you specify on the connect.   In this drop-down list,  us-east-2 is NOT present.   So,  you are out of luck if your buckets are currently located in us-east-2. 

If your buckets are in us-east-2, There is not much you can do to get the S3 upload zap working other than moving your buckets to a different region. In my case,  I moved them to us-west-1.   But other parts of my deployment and code on the AWS side were affected and all this drama was solely due to zapier simply not allowing a S3 connection to specify 1 of 4 US regions!   This seems like a bug/ommission on their part. But it has real ramifications to 25% of the AWS user community who have buckets in us-east-2.

picture below is of the drop-down on connection…   us-east-2 does not show up in this drop-down at all.  You can try this yourself.  Create a new connection in the S3 upload zap.  Try to set the region to us-east-2… it's missing.

 

 

This is however,  NOT the main problem I am reporting.  I worked around this by moving my buckets to a supported region.   I still need a fix to the problem that my uploaded files seem to be getting truncated by the time they get to my S3 buckets and the file is getting corrupted (unless I am missing something)

A
Userlevel 1
Badge +1
  1. OK - I created a simple Test zap to illustrate the problem. Here are all the details and screenshots…  Not sure how I simplify this test case any more, but happy to hear your thoughts.

    This is the zap…  (looks like file does not always get truncated… It can even expand!)
     

 

  1. This is the screenshot of the source file I tested with - this screenshot is details from step 1 of the zap above
Source file details on step 1 of the zap
​​​​​
  1. Source file details on Step 2 of the zap (before I hit Test)
Source file details as shown in the Step 2 before I hit Test button
  1. This is the output AFTER I hit the test button and the S3 upload had finished

     
S3 File Upload results - successfully uploaded but with the wrong file size

 

After I ran the zap, I could see the file on my S3 bucket.  I downloaded the file from S3 and compared it with the original source file.  You can see the difference in file sizes

 

The S3 file is corrupted… I cannot open it in my app.  I can however correctly open the source file as a png file.

If I am doing something obvious wrong… please LMK.   I am completely stuck with this problem on the zapier front and its an important part of my overall automation flow.

If Zapier support is looking at this,  you can try this out yourself and presumably you can reproduce the same problem.

A
Userlevel 1
Badge +1

Finally, are there other AWS S3 users who are using this upload Zap?   These problems seem so basic (both the connection dialog missing a major US AWS region + files getting mangled on an upload) that I am wondering how more users are not encountering the same behaviour.

Would also love to hear if I am missing something obvious which I can quickly fix at my end.

If you are actively using this zap,  please comment on your usage and if your file uploads to S3 are all working ok.   Thanks.

Troy Tessalone
Userlevel 7
Badge +14

@anatesanStream

Can you post screenshots with how your Zap steps are configured in EDIT mode?

 

⚡ Troy Tessalone - AutomationAce.com | Premier Certified Zapier Expert | #1 Zapier Community Contributor
Troy Tessalone
Userlevel 7
Badge +14

@anatesanStream 

Also, best to submit feedback and feature requests via a ticket to Zapier Support for it to be properly logged: https://zapier.com/app/get-help

⚡ Troy Tessalone - AutomationAce.com | Premier Certified Zapier Expert | #1 Zapier Community Contributor
Troy Tessalone
Userlevel 7
Badge +14

@anatesanStream 

Looks like you are using a GDrive file VIEW link, rather than a GDrive File direct download link.

Tool that will help you understand the proper link format to use: https://sites.google.com/site/gdocs2direct/

​NOTE: The GDrive File must be publicly accessible to be used so you may need to adjust your File Sharing Preferences for the Folder.

 

 

 

 

⚡ Troy Tessalone - AutomationAce.com | Premier Certified Zapier Expert | #1 Zapier Community Contributor
A
Userlevel 1
Badge +1

Thank you so much, @Troy Tessalone for the pointer.   Turns out that I did not have the right format URL.  There were 3 possible URLs:

  1. A URL which is in the format: https://drive.google.com/uc?export=download&id=<file_object_id>.   This is from your reference doc - https://sites.google.com/site/gdocs2direct/.  S3 UPLOAD ZAP WORKS WITH NO FILE CORRUPTION if this URL format is specified.   I had to compose this URL myself in another zap step (see Formatter step in my revised test zap)
  2. Google Drive triggers give you 2 other URLs for a file on the Google Drive:   the Attachment Link (in the format shown here + a few variants- https://drive.google.com/file/d/<file_object_id/view?usp=drivesdk.)and the Download Link URL (in the format - https://drive.google.com/uc?id=<file_object_id>&export=download).  Neither of these work correctly when specified in the S3 upload zap.  These result in file corruption on the upload.

I verified this by modifying my test-zap as shown below…  I added a new Formatter step to extract the object ID and recomposed the URL in format (1) above in the s3upload zap for the file location.

Revised test zap with a new Formatter step to extract the file_object_id from standard Gdrive URLs

Would be good if zapier included all this in the standard doc for this zap.   
 

Troy Tessalone
Userlevel 7
Badge +14

@anatesanStream 

fyi...help article about sending files in Zaps: https://help.zapier.com/hc/en-us/articles/8496288813453

 

To send a file in a Zap, you need to use a file object or certain types of URL links.

File objects are the bundles of data that represent the actual file. This is different from the name of the file or other file attributes. For example, the actual image of an apple is the file object, while the file name might be “apple.jpg”.

If you’re using a URL link to a file, it has to be:

  • Publicly-accessible: it can’t require a login to be accessed.
  • Downloadable: the app you’re sending it to must be able to upload it to their server.

Some apps will create a text (.txt) file if you enter text values into a field that accepts files.

Limitations

  • Some file fields may not accept text.
  • Files larger than 100 MB may result in a time out error.
  • You can only add 1 file per field. If you need to add additional files, you’ll need to add separate action steps.
ratingStar icon Tip

You can test your URL link by opening it in a private or incognito browser window. If it prompts you to download a file, the link can be used. If it prompts you to log in to an app, loads directly in the web browser instead of downloading, or results in an error code, the link can’t be used.

1. Check if your previous step provides a file

  • If a previous step has a File field with an (Exists but not shown) value, that is a file object.
  • If a previous step has an Attachments field, it may have a file object or a URL link to a file that you can use.

2. Add a file to your action step

  • If your field accepts files, you will see a file File: filled icon.
  • Insert your file into the field by selecting it from an earlier step.

If you need additional help using files in Zaps, contact Zapier support.

⚡ Troy Tessalone - AutomationAce.com | Premier Certified Zapier Expert | #1 Zapier Community Contributor
A
Userlevel 1
Badge +1

Thanks for this… 

The problem is that if you deal with Google Drive,  you will get many URLs - attachment links, download links, parent links and on and on.   If you deal with Google Mail to get attachments, its even worse - there are a huge number of URLs to puzzle over.

So, practically,  its pretty much a guessing game to figure out what will work for your use case and zap, even if you read generic articles like these.

This is not a Zapier-induced problem,  but it was a non-trivial problem to figure out what to use.  For my use case,  I had to actually reconstruct the right URL myself in Google Sheets/zap by breaking apart one of the standard Google Drive file URLs to get the object ID.

In any case,  I am glad that the pointer you gave me helped me get to a solution that works…. Thanks again.

Reply


Terms & Conditions