Hey @Tim Bezold @Danny Lai I have been testing the new AWS S3 Connector.

I was able to set up the sample_schema and copy a small dataset but when I tried to run an actual use case I started to face a bunch of errors.

The first big dataset I tried was a 30 columns and 1 million CSV file. I set up the DatSync job and ran it.. after 26 minutes I got this message:

Something went wrong while executing the remoteExtract_analytics node: Couldn't retrieve data from external source for object 'analytics': WCA Error. Received http '500'. {"errorMsg":"java.lang.OutOfMemoryError - GC overhead limit exceeded\nParser Configuration: CsvParserSettings:\n\tAuto configuration enabled=true\n\tAutodetect column delimiter=true\n\tAutodetect quotes=true\n\tColumn reordering enabled=true\n\tDelimiters for detection=[]\n\tEmpty value=null\n\tEscape unquoted values=false\n\tHeader extraction enabled=true\n\tHeaders=null\n\tIgnore leading whitespaces=true\n\tIgnore leading whitespaces in quotes=false\n\tIgnore trailing whitespaces=true\n\tIgnore trailing whitespaces in quotes=false\n\tInput buffer size=1048576\n\tInput reading on separate thread=true\n\tKeep escape sequences=false\n\tKeep quotes=false\n\tLength of content displayed on error=-1\n\tLine separator detection enabled=true\n\tMaximum number of characters per column=4096\n\tMaximum number of columns=512\n\tNormalize escaped line separators=true\n\tNull value=null\n\tNumber of records to read=all\n\tProcessor=com.univocity.parsers.common.processor.RowListProcessor\n\tRestricting data in exceptions=false\n\tRowProcessor error handler=null\n\t

Based on the "Out of memory" message I decided to split the 1M rows file into two 500k rows files. It worked perfectly. It took only 1 minute to copy the data.

Are there specific limits per file? I see some parameters in that error message like Maximum number of characters per column=4096 or input buffer size=1048576. But I couldn't find those in the documentation.

The only thing I found was this: Standard-volume connectors can sync up to 20 million rows or 10 GB per object, depending on which limit is reached first.

Again, is that limit per bucket? Per file?

We are planning on use this connector as our main data source so we would like to have a clear understanding on how it works. What are the limits and limitations, etc.