[Upload Video 2.2GB] Upload large file

Describe the bug I run sample and upload big file, but not work

javax.net.ssl.SSLException: Write error: ssl=0x7a5418f8c8: I/O error during system call, Broken pipe
       at com.android.org.conscrypt.NativeCrypto.SSL_write(Native Method)
       at com.android.org.conscrypt.NativeSsl.write(NativeSsl.java:426)
       at com.android.org.conscrypt.ConscryptFileDescriptorSocket$SSLOutputStream.write(ConscryptFileDescriptorSocket.java:660)
       at okio.OutputStreamSink.write(JvmOkio.kt:53)
       at okio.AsyncTimeout$sink$1.write(AsyncTimeout.kt:103)
       at okio.RealBufferedSink.flush(RealBufferedSink.kt:247)
       at okhttp3.internal.http1.Http1ExchangeCodec$KnownLengthSink.flush(Http1ExchangeCodec.kt:276)
       at okio.ForwardingSink.flush(ForwardingSink.kt:32)
       at okhttp3.internal.connection.Exchange$RequestBodySink.flush(Exchange.kt:228)
       at okio.RealBufferedSink.flush(RealBufferedSink.kt:250)
       at net.gotev.uploadservice.okhttp.OkHttpBodyWriter.flush(OkHttpBodyWriter.kt:25)
       at net.gotev.uploadservice.network.BodyWriter.write(BodyWriter.kt:67)
       at net.gotev.uploadservice.network.BodyWriter.writeStream(BodyWriter.kt:42)
       at net.gotev.uploadservice.protocols.multipart.MultipartUploadTask.writeFiles(MultipartUploadTask.kt:52)
       at net.gotev.uploadservice.protocols.multipart.MultipartUploadTask.onWriteRequestBody(MultipartUploadTask.kt:81)
       at net.gotev.uploadservice.okhttp.OkHttpStackRequest$createBody$1.writeTo(OkHttpStackRequest.kt:76)
       at okhttp3.internal.http.CallServerInterceptor.intercept(CallServerInterceptor.kt:59)
       at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.kt:109)
       at net.gotev.uploadservicedemo.App.lambda$getOkHttpClient$1(App.java:87)
       at net.gotev.uploadservicedemo.-$$Lambda$App$i1SOtWVwIE4ECfByutde1MWQaGw.intercept(Unknown Source:0)
       at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.kt:109)
       at okhttp3.internal.connection.ConnectInterceptor.intercept(ConnectInterceptor.kt:34)
       at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.kt:109)
       at okhttp3.internal.cache.CacheInterceptor.intercept(CacheInterceptor.kt:95)
       at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.kt:109)
       at okhttp3.internal.http.BridgeInterceptor.intercept(BridgeInterceptor.kt:83)
       at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.kt:109)
       at okhttp3.internal.http.RetryAndFollowUpInterceptor.intercept(RetryAndFollowUpInterceptor.kt:76)
       at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.kt:109)
       at okhttp3.logging.HttpLoggingInterceptor.intercept(HttpLoggingInterceptor.java:212)
       at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.kt:109)
       at net.gotev.uploadservicedemo.App.lambda$getOkHttpClient$0(App.java:76)
       at net.gotev.uploadservicedemo.-$$Lambda$App$2XvpshJlC8JG_AkWVft4Xf_QR38.intercept(Unknown Source:0)
       at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.kt:109)
       at okhttp3.internal.connection.RealCall.getResponseWithInterceptorChain$okhttp(RealCall.kt:201)
       at okhttp3.internal.connection.RealCall.execute(RealCall.kt:154)
       at net.gotev.uploadservice.okhttp.OkHttpStackRequest.getResponse(OkHttpStackRequest.kt:95)
       at net.gotev.uploadservice.HttpUploadTask.upload(HttpUploadTask.kt:49)
       at net.gotev.uploadservice.UploadTask.run(UploadTask.kt:140)
       at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1167)
       at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:641)
       at java.lang.Thread.run(Thread.java:919)
   	Suppressed: java.net.ProtocolException: unexpected end of stream
       at okhttp3.internal.connection.Exchange$RequestBodySink.close(Exchange.kt:239)
       at okio.RealBufferedSink.close(RealBufferedSink.kt:268)
       at net.gotev.uploadservice.okhttp.OkHttpBodyWriter.close(OkHttpBodyWriter.kt:30)
       at kotlin.io.CloseableKt.closeFinally(Closeable.kt:60)
2020-08-25 13:57:43.367 6397-21586/net.gotev.uploadservicedemo E/UploadService:     at net.gotev.uploadservice.okhttp.OkHttpStackRequest$createBody$1.writeTo(OkHttpStackRequest.kt:75)

To Reproduce Steps to reproduce the behavior:

1. Open App
2. New Multipart Upload
3. Input link, header, add file
4. Click check icon on right top
5. Progress to 14% and broken

Device: Android 10 Samsung A30

[gotev]

@chihung93 @Borna-f using the testbed I tried a 2,5 GB multipart upload.

Created a test file generated like this:

mkfile -n 2500m large_test_file

and then pushed it to the emulator.

What happens systematically is that after 1 minute I get the following exception which makes the upload fail:

D/OkHttp: <-- HTTP FAILED: java.net.SocketException: Connection reset

This happens because the server is closing the connection prematurely and causes also the broken pipe exception, exactly like you observed @chihung93 (even when using the binary upload)

The demo server of the testbed is using node.js + express which has 1 minute socket timeout by default.

Tried uploading the same file using curl with the following commands:

curl -vvv -X POST -w %{time_total} --limit-rate 1M -F mykey=myvalue -F upload=@large_test_file http://192.168.1.132:3000/upload/multipart

Added --limit-rate 1M on purpose to simulate some latency, otherwise, being on localhost, the transfer is immediate. This is what I get:

*   Trying 192.168.1.132...
* TCP_NODELAY set
* Connected to 192.168.1.132 (192.168.1.132) port 3000 (#0)
> POST /upload/multipart HTTP/1.1
> Host: 192.168.1.132:3000
> User-Agent: curl/7.64.1
> Accept: */*
> Content-Length: 2621440310
> Content-Type: multipart/form-data; boundary=------------------------81736c9f77eb9483
> Expect: 100-continue
>
< HTTP/1.1 100 Continue
* Send failure: Broken pipe
* Closing connection 0
curl: (55) Send failure: Broken pipe
60,153441

The last number 60,153441 represents the total seconds of the request.

Long story short:

It’s not a library bug.

Avoid uploading very big chunks of data at once. Prefer many smaller pieces. If you need it at all costs, inform your business people, backend and DevOps engineers that you need very large socket timeouts. This would make your server more exposed to hacker attacks though and it’s not recommended. The final choice is yours.

[gotev]

@chihung93 thanks for the video, but it doesn’t help addressing the error you’re encountering. Seeing what happened on the UI doesn’t give any detail.

To understand what’s really happening, a tcpdump or Charles log is needed. Do an upload using Android Upload Service and save also LogCat log. Then, upload the same file with iOS and compare the two, looking at the initial request to spot the differences. First of all check the Content-Length count and the other headers. They should be the same. If not, that could be the reason.