Stuck on an issue?

Lightrun Answers was designed to reduce the constant googling that comes with debugging 3rd party libraries. It collects links to all the places you might be looking at while hunting down a tough bug.

And, if you’re still stuck at the end, we’re happy to hop on a call to see how we can help out.

FTP Source - large files can make the connector crash

See original GitHub issue

Version: Confluent Platform 3.3

What is the expected behaviour?

When reading a big file from an FTP source, it is processed correctly

What was observed?

When reading a file from an FTP server with the FTP Source Connector, the connector crashes (out of memory).

From what I understand, the whole file is read in memory directly, meaning the big files will be an issue even if I increase the memory available to my connector.

Is this by design ? Is there a way around this ? Could we expect the source connector to become better with big files ?

java.lang.OutOfMemoryError: Java heap space
    at java.util.Arrays.copyOf(Arrays.java:3236)
    at java.io.ByteArrayOutputStream.grow(ByteArrayOutputStream.java:118)
    at java.io.ByteArrayOutputStream.ensureCapacity(ByteArrayOutputStream.java:93)
    at java.io.ByteArrayOutputStream.write(ByteArrayOutputStream.java:153)
    at org.apache.commons.net.io.Util.copyStream(Util.java:124)
    at org.apache.commons.net.ftp.FTPClient._retrieveFile(FTPClient.java:1924)
    at org.apache.commons.net.ftp.FTPClient.retrieveFile(FTPClient.java:1889)
    at com.datamountaineer.streamreactor.connect.ftp.source.FtpMonitor.fetch(FtpMonitor.scala:74)
    at com.datamountaineer.streamreactor.connect.ftp.source.FtpMonitor$$anonfun$fetchFromMonitoredPlaces$3.apply(FtpMonitor.scala:142)
    at com.datamountaineer.streamreactor.connect.ftp.source.FtpMonitor$$anonfun$fetchFromMonitoredPlaces$3.apply(FtpMonitor.scala:142)
    at scala.collection.immutable.Stream.flatMap(Stream.scala:489)
    at com.datamountaineer.streamreactor.connect.ftp.source.FtpMonitor.fetchFromMonitoredPlaces(FtpMonitor.scala:142)
    at com.datamountaineer.streamreactor.connect.ftp.source.FtpMonitor$$anonfun$poll$1$$anonfun$apply$1.apply(FtpMonitor.scala:190)
    at com.datamountaineer.streamreactor.connect.ftp.source.FtpMonitor$$anonfun$poll$1$$anonfun$apply$1.apply(FtpMonitor.scala:189)
    at scala.collection.immutable.Stream.flatMap(Stream.scala:489)
    at com.datamountaineer.streamreactor.connect.ftp.source.FtpMonitor$$anonfun$poll$1.apply(FtpMonitor.scala:189)
    at com.datamountaineer.streamreactor.connect.ftp.source.FtpMonitor$$anonfun$poll$1.apply(FtpMonitor.scala:189)
    at scala.util.Try$.apply(Try.scala:192)
    at com.datamountaineer.streamreactor.connect.ftp.source.FtpMonitor.poll(FtpMonitor.scala:189)
    at com.datamountaineer.streamreactor.connect.ftp.source.FtpSourcePoller.fetchRecords(FtpSourceTask.scala:78)
    at com.datamountaineer.streamreactor.connect.ftp.source.FtpSourcePoller.poll(FtpSourceTask.scala:69)
    at com.datamountaineer.streamreactor.connect.ftp.source.FtpSourceTask.poll(FtpSourceTask.scala:121)
    at org.apache.kafka.connect.runtime.WorkerSourceTask.execute(WorkerSourceTask.java:163)
    at org.apache.kafka.connect.runtime.WorkerTask.doRun(WorkerTask.java:146)
    at org.apache.kafka.connect.runtime.WorkerTask.run(WorkerTask.java:190)
    at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
    at java.util.concurrent.FutureTask.run(FutureTask.java:266)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
    at java.lang.Thread.run(Thread.java:745)
``