FTP Source - large files can make the connector crash
See original GitHub issueVersion: Confluent Platform 3.3
What is the expected behaviour?
When reading a big file from an FTP source, it is processed correctly
What was observed?
When reading a file from an FTP server with the FTP Source Connector, the connector crashes (out of memory).
From what I understand, the whole file is read in memory directly, meaning the big files will be an issue even if I increase the memory available to my connector.
Is this by design ? Is there a way around this ? Could we expect the source connector to become better with big files ?
java.lang.OutOfMemoryError: Java heap space
at java.util.Arrays.copyOf(Arrays.java:3236)
at java.io.ByteArrayOutputStream.grow(ByteArrayOutputStream.java:118)
at java.io.ByteArrayOutputStream.ensureCapacity(ByteArrayOutputStream.java:93)
at java.io.ByteArrayOutputStream.write(ByteArrayOutputStream.java:153)
at org.apache.commons.net.io.Util.copyStream(Util.java:124)
at org.apache.commons.net.ftp.FTPClient._retrieveFile(FTPClient.java:1924)
at org.apache.commons.net.ftp.FTPClient.retrieveFile(FTPClient.java:1889)
at com.datamountaineer.streamreactor.connect.ftp.source.FtpMonitor.fetch(FtpMonitor.scala:74)
at com.datamountaineer.streamreactor.connect.ftp.source.FtpMonitor$$anonfun$fetchFromMonitoredPlaces$3.apply(FtpMonitor.scala:142)
at com.datamountaineer.streamreactor.connect.ftp.source.FtpMonitor$$anonfun$fetchFromMonitoredPlaces$3.apply(FtpMonitor.scala:142)
at scala.collection.immutable.Stream.flatMap(Stream.scala:489)
at com.datamountaineer.streamreactor.connect.ftp.source.FtpMonitor.fetchFromMonitoredPlaces(FtpMonitor.scala:142)
at com.datamountaineer.streamreactor.connect.ftp.source.FtpMonitor$$anonfun$poll$1$$anonfun$apply$1.apply(FtpMonitor.scala:190)
at com.datamountaineer.streamreactor.connect.ftp.source.FtpMonitor$$anonfun$poll$1$$anonfun$apply$1.apply(FtpMonitor.scala:189)
at scala.collection.immutable.Stream.flatMap(Stream.scala:489)
at com.datamountaineer.streamreactor.connect.ftp.source.FtpMonitor$$anonfun$poll$1.apply(FtpMonitor.scala:189)
at com.datamountaineer.streamreactor.connect.ftp.source.FtpMonitor$$anonfun$poll$1.apply(FtpMonitor.scala:189)
at scala.util.Try$.apply(Try.scala:192)
at com.datamountaineer.streamreactor.connect.ftp.source.FtpMonitor.poll(FtpMonitor.scala:189)
at com.datamountaineer.streamreactor.connect.ftp.source.FtpSourcePoller.fetchRecords(FtpSourceTask.scala:78)
at com.datamountaineer.streamreactor.connect.ftp.source.FtpSourcePoller.poll(FtpSourceTask.scala:69)
at com.datamountaineer.streamreactor.connect.ftp.source.FtpSourceTask.poll(FtpSourceTask.scala:121)
at org.apache.kafka.connect.runtime.WorkerSourceTask.execute(WorkerSourceTask.java:163)
at org.apache.kafka.connect.runtime.WorkerTask.doRun(WorkerTask.java:146)
at org.apache.kafka.connect.runtime.WorkerTask.run(WorkerTask.java:190)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
``
Issue Analytics
- State:
- Created 5 years ago
- Comments:9
Top Results From Across the Web
FTP Source - large files can make the connector crash #462
When reading a file from an FTP server with the FTP Source Connector, the connector crashes (out of memory). From what I understand,...
Read more >Copying large number of files accross home network - crashing
The other option would be to zip up the files and transmit them as one large file. Save this answer. Show activity on...
Read more >Ftp freeze when I upload big files - Server Fault
The transfer is working, if the files are less then 800mb~1000mb, but if I try to upload bigger file it hangs up while...
Read more >FTP Connector - Covisint
Implementation – FTP Connector. Covisint has implemented FTP (File Transfer Protocol) access to the messaging hub using a custom FTP server.
Read more >Does list action load all files in the memory/disk?
I am using the FTP connector to list files from an FTP folder in Mule 4. I am using a matcher to filter...
Read more >
Top Related Medium Post
No results found
Top Related StackOverflow Question
No results found
Troubleshoot Live Code
Lightrun enables developers to add logs, metrics and snapshots to live code - no restarts or redeploys required.
Start Free
Top Related Reddit Thread
No results found
Top Related Hackernoon Post
No results found
Top Related Tweet
No results found
Top Related Dev.to Post
No results found
Top Related Hashnode Post
No results found
For your information, we are working on a fix. euranova/stream-reactor#1 .
We will submit a pull request to you when happy with the result. In the mean time, feel free to participate. Any input would be very appreciated.
@JoeInigo google might know 😃