My S3 file uploads are getting truncated!

Hi @anatesanStream 

For us to have full context, post screenshots with how the steps are outlined and configured.

###

Help articles to reference for using AWS S3 in Zaps: https://zapier.com/apps/amazon-s3/help

I Can't Add to a Specific Bucket (400 error)

Is your bucket in a different region than the one you selected when connecting your S3 account to Zapier? If so, you will need to reconnect and select the matching region.

Note that often (depending on the security policy of the S3 account you have connected to Zapier), we are able to see buckets from regions other than the selected region, but we are not able to interact with them. For example, if you have a bucket in US East (Ohio) (which is us-east-2), you may be able to select that bucket in a drop-down, but if you selected us-east-1 when you connected your account, we won't be able to upload files to that bucket.

AWS has documentation on their region names here if you aren't sure which numerical region your bucket resides in: https://docs.aws.amazon.com/general/latest/gr/rande.html#s3_region

If you are still having trouble after this, please drop us an email so we can troubleshoot further with you!

Zapier’s S3 connection dialog shows a list of regions that you specify on the connect.   In this drop-down list,  us-east-2 is NOT present.   So,  you are out of luck if your buckets are currently located in us-east-2. 

If your buckets are in us-east-2, There is not much you can do to get the S3 upload zap working other than moving your buckets to a different region. In my case,  I moved them to us-west-1.   But other parts of my deployment and code on the AWS side were affected and all this drama was solely due to zapier simply not allowing a S3 connection to specify 1 of 4 US regions!   This seems like a bug/ommission on their part. But it has real ramifications to 25% of the AWS user community who have buckets in us-east-2.

picture below is of the drop-down on connection…   us-east-2 does not show up in this drop-down at all.  You can try this yourself.  Create a new connection in the S3 upload zap.  Try to set the region to us-east-2… it's missing.

 

This is however,  NOT the main problem I am reporting.  I worked around this by moving my buckets to a supported region.   I still need a fix to the problem that my uploaded files seem to be getting truncated by the time they get to my S3 buckets and the file is getting corrupted (unless I am missing something)

0. OK - I created a simple Test zap to illustrate the problem. Here are all the details and screenshots…  Not sure how I simplify this test case any more, but happy to hear your thoughts.
   
   This is the zap…  (looks like file does not always get truncated… It can even expand!)
    

1. This is the screenshot of the source file I tested with - this screenshot is details from step 1 of the zap above

2. Source file details on Step 2 of the zap (before I hit Test)

3. This is the output AFTER I hit the test button and the S3 upload had finished
   
    

After I ran the zap, I could see the file on my S3 bucket.  I downloaded the file from S3 and compared it with the original source file.  You can see the difference in file sizes

The S3 file is corrupted… I cannot open it in my app.  I can however correctly open the source file as a png file.

If I am doing something obvious wrong… please LMK.   I am completely stuck with this problem on the zapier front and its an important part of my overall automation flow.

If Zapier support is looking at this,  you can try this out yourself and presumably you can reproduce the same problem.

Finally, are there other AWS S3 users who are using this upload Zap?   These problems seem so basic (both the connection dialog missing a major US AWS region + files getting mangled on an upload) that I am wondering how more users are not encountering the same behaviour.

Would also love to hear if I am missing something obvious which I can quickly fix at my end.

If you are actively using this zap,  please comment on your usage and if your file uploads to S3 are all working ok.   Thanks.

@anatesanStream

Can you post screenshots with how your Zap steps are configured in EDIT mode?

@anatesanStream 

Also, best to submit feedback and feature requests via a ticket to Zapier Support for it to be properly logged: https://zapier.com/app/get-help

Thank you so much, @Troy Tessalone for the pointer.   Turns out that I did not have the right format URL.  There were 3 possible URLs:

1. A URL which is in the format: https://drive.google.com/uc?export=download&id=<file_object_id>.   This is from your reference doc - https://sites.google.com/site/gdocs2direct/.  S3 UPLOAD ZAP WORKS WITH NO FILE CORRUPTION if this URL format is specified.   I had to compose this URL myself in another zap step (see Formatter step in my revised test zap)
2. Google Drive triggers give you 2 other URLs for a file on the Google Drive:   the Attachment Link (in the format shown here + a few variants- https://drive.google.com/file/d/<file_object_id/view?usp=drivesdk.)and the Download Link URL (in the format - https://drive.google.com/uc?id=<file_object_id>&export=download).  Neither of these work correctly when specified in the S3 upload zap.  These result in file corruption on the upload.

I verified this by modifying my test-zap as shown below…  I added a new Formatter step to extract the object ID and recomposed the URL in format (1) above in the s3upload zap for the file location.

Would be good if zapier included all this in the standard doc for this zap.   
 

@anatesanStream 

fyi...help article about sending files in Zaps: https://help.zapier.com/hc/en-us/articles/8496288813453

To send a file in a Zap, you need to use a file object or certain types of URL links.

File objects are the bundles of data that represent the actual file. This is different from the name of the file or other file attributes. For example, the actual image of an apple is the file object, while the file name might be “apple.jpg”.

If you’re using a URL link to a file, it has to be:

• Publicly-accessible: it can’t require a login to be accessed.
• Downloadable: the app you’re sending it to must be able to upload it to their server.

Some apps will create a text (.txt) file if you enter text values into a field that accepts files.

Limitations

• Some file fields may not accept text.
• Files larger than 100 MB may result in a time out error.
• You can only add 1 file per field. If you need to add additional files, you’ll need to add separate action steps.

Tip

You can test your URL link by opening it in a private or incognito browser window. If it prompts you to download a file, the link can be used. If it prompts you to log in to an app, loads directly in the web browser instead of downloading, or results in an error code, the link can’t be used.

If you need additional help using files in Zaps, contact Zapier support.

Thanks for this… 

The problem is that if you deal with Google Drive,  you will get many URLs - attachment links, download links, parent links and on and on.   If you deal with Google Mail to get attachments, its even worse - there are a huge number of URLs to puzzle over.

So, practically,  its pretty much a guessing game to figure out what will work for your use case and zap, even if you read generic articles like these.

This is not a Zapier-induced problem,  but it was a non-trivial problem to figure out what to use.  For my use case,  I had to actually reconstruct the right URL myself in Google Sheets/zap by breaking apart one of the standard Google Drive file URLs to get the object ID.

In any case,  I am glad that the pointer you gave me helped me get to a solution that works…. Thanks again.