Unable to download text file through Chunked transfer encoding

Issue

1. What problem do you get? I’m not able to download one text file (VTT file). Presumably, it’s because the server is using Chunked transfer encoding.

2. Which version of FileDownloader are you using when you produce such problem? Version 1.7.4

3. How to reproduce such problem? Just try downloading the following VTT file. Please note that I added the VTT url to Bitly, because the url expires and I can keep updating it for testing purposes: http://bit.ly/2HtV0gP You might need to get the redirect url first. Please let me know if the url expires.

4. Do you set FileDownloadLog.NEED_LOG=true? Yes

5. Could you please reproduce this problem and provide all main process and :filedownloader process logcat See the logcat below.

6. Can you fix it by yourself and request PR, if not, what’s problem do you get when you try to fix it Not at this moment

///////////////////////////////////////////////////////////////////////////////////////////////////// ////////////////////////////////////MAIN PROCESS LOG//////////////////////////////////// ///////////////////////////////////////////////////////////////////////////////////////////////////// 06-07 17:10:45.678 15132-15132/com.example.bugapp I/zygote64: Late-enabling -Xcheck:jni 06-07 17:10:45.786 15132-15150/com.example.bugapp D/libEGL: loaded /vendor/lib64/egl/libEGL_adreno.so 06-07 17:10:45.792 15132-15150/com.example.bugapp D/libEGL: loaded /vendor/lib64/egl/libGLESv1_CM_adreno.so 06-07 17:10:45.805 15132-15150/com.example.bugapp D/libEGL: loaded /vendor/lib64/egl/libGLESv2_adreno.so 06-07 17:10:45.864 15132-15171/com.example.bugapp D/OpenGLRenderer: HWUI GL Pipeline 06-07 17:10:45.900 15132-15171/com.example.bugapp I/Adreno: QUALCOMM build : 594927b, I916dfac403 Build Date : 10/11/17 OpenGL ES Shader Compiler Version: EV031.21.02.00 Local Branch : mybranch28618966 Remote Branch : quic/gfx-adreno.lnx.6.4.9-rel Remote Branch : NONE Reconstruct Branch : NOTHING 06-07 17:10:45.905 15132-15171/com.example.bugapp I/Adreno: PFP: 0x005ff087, ME: 0x005ff063 06-07 17:10:45.910 15132-15171/com.example.bugapp I/zygote64: android::hardware::configstore::V1_0::ISurfaceFlingerConfigs::hasWideColorDisplay retrieved: 1 06-07 17:10:45.910 15132-15171/com.example.bugapp I/OpenGLRenderer: Initialized EGL, version 1.4 06-07 17:10:45.910 15132-15171/com.example.bugapp D/OpenGLRenderer: Swap behavior 2 06-07 17:10:55.359 15132-15301/com.example.bugapp I/ODRM: Starting downloadOneByOne 06-07 17:10:55.373 15132-15301/com.example.bugapp D/FileDownloader.DownloadTask: setPath /storage/emulated/0/Android/data/com.example.bugapp/files/Download 06-07 17:10:55.374 15132-15301/com.example.bugapp D/FileDownloader.DownloadTask: setListener com.example.bugapp.util.DownloadIntentService$MyFileDownloadListener@f4140ac 06-07 17:10:55.374 15132-15301/com.example.bugapp V/FileDownloader.DownloadTaskHunter: call start Url[http://bcboltproduse1-a.akamaihd.net/media/v1/text/vtt/clear/5783296997001/9f8be68c-fd34-4b94-922e-ce97768dd48d/8d06ebef-49db-43fe-921a-c6a38b777a91/text.vtt?akamai_token=exp=1528428442~acl=/media/v1/text/vtt/clear/5783296997001/9f8be68c-fd34-4b94-922e-ce97768dd48d/8d06ebef-49db-43fe-921a-c6a38b777a91/text.vtt*~hmac=77a5a4c6cb010b8594b9e55a7f35f3a6e0fda9f7713b752dd81815860882b896], Path[/storage/emulated/0/Android/data/com.example.bugapp/files/Download] Listener[com.example.bugapp.util.DownloadIntentService$MyFileDownloadListener@f4140ac], Tag[null] 06-07 17:10:55.377 15132-15303/com.example.bugapp V/FileDownloader.FileDownloadEventPool: setListener event.service.connect.changed 06-07 17:10:55.378 15132-15301/com.example.bugapp V/FileDownloader.DownloadTaskHunter: the task[-818857745] has been into the launch pool. 06-07 17:10:55.378 15132-15301/com.example.bugapp I/ODRM: Checking that atomic integer is ZERO 06-07 17:10:55.380 15132-15303/com.example.bugapp I/FileDownloader.FileDownloadProperties: init properties 0 load properties: http.lenient=FALSE; process.non-separate=FALSE; download.min-progress-step=65536; download.min-progress-time=2000; download.max-network-thread-count=3; file.non-pre-allocation=FALSE; broadcast.completed=FALSE; download.trial-connection-head-method=FALSE 06-07 17:10:55.381 15132-15303/com.example.bugapp D/FileDownloader.LostServiceConnectedHandler: Waiting for connecting with the downloader service… -818857745 06-07 17:10:55.383 15132-15303/com.example.bugapp D/FileDownloader.FileDownloadServiceUIGuard: bindStartByContext PlaybackTestApplication 06-07 17:10:55.398 15132-15303/com.example.bugapp D/FileDownloader.DownloadTaskHunter: free the task -818857745, when the status is 10 06-07 17:10:55.493 15132-15132/com.example.bugapp D/FileDownloader.FileDownloadServiceUIGuard: onServiceConnected ComponentInfo{com.example.bugapp/com.liulishuo.filedownloader.services.FileDownloadService$SeparateProcessService} com.liulishuo.filedownloader.i.IFileDownloadIPCService$Stub$Proxy@a935ff1 06-07 17:10:55.494 15132-15132/com.example.bugapp V/FileDownloader.FileDownloadEventPool: asyncPublishInNewThread event.service.connect.changed 06-07 17:10:55.495 15132-15321/com.example.bugapp V/FileDownloader.FileDownloadEventPool: publish event.service.connect.changed 06-07 17:10:55.496 15132-15321/com.example.bugapp D/FileDownloader.LostServiceConnectedHandler: The downloader service is connected. 06-07 17:10:55.496 15132-15321/com.example.bugapp V/FileDownloader.DownloadTaskHunter: call start Url[http://bcboltproduse1-a.akamaihd.net/media/v1/text/vtt/clear/5783296997001/9f8be68c-fd34-4b94-922e-ce97768dd48d/8d06ebef-49db-43fe-921a-c6a38b777a91/text.vtt?akamai_token=exp=1528428442~acl=/media/v1/text/vtt/clear/5783296997001/9f8be68c-fd34-4b94-922e-ce97768dd48d/8d06ebef-49db-43fe-921a-c6a38b777a91/text.vtt*~hmac=77a5a4c6cb010b8594b9e55a7f35f3a6e0fda9f7713b752dd81815860882b896], Path[/storage/emulated/0/Android/data/com.example.bugapp/files/Download] Listener[com.example.bugapp.util.DownloadIntentService$MyFileDownloadListener@f4140ac], Tag[null] 06-07 17:10:55.497 15132-15321/com.example.bugapp V/FileDownloader.DownloadTaskHunter: the task[-818857745] has been into the launch pool. 06-07 17:10:55.498 15132-15322/com.example.bugapp D/FileDownloader.FileDownloadMessenger: notify begin [e-mail]@458f944 06-07 17:10:55.498 15132-15322/com.example.bugapp V/FileDownloader.DownloadTaskHunter: filedownloader:lifecycle:start com.liulishuo.filedownloader.DownloadTaskHunter@46d0a2d by 11 06-07 17:10:55.499 15132-15322/com.example.bugapp V/FileDownloader.FileDownloadList: add list in all [e-mail]@458f944 11 1 06-07 17:10:55.521 15132-15325/com.example.bugapp D/FileDownloader.MessageSnapshotGate: ~~~callback -818857745 old[11] new[1] 1 06-07 17:10:55.522 15132-15325/com.example.bugapp D/FileDownloader.FileDownloadMessenger: notify pending [e-mail]@458f944 06-07 17:10:55.523 15132-15325/com.example.bugapp D/FileDownloader.MessageSnapshotGate: updateKeepFlow 06-07 17:10:55.524 15132-15325/com.example.bugapp D/FileDownloader.MessageSnapshotGate: ~~~callback -818857745 old[1] new[6] 1 06-07 17:10:55.524 15132-15325/com.example.bugapp D/FileDownloader.FileDownloadMessenger: notify started [e-mail]@458f944 06-07 17:10:55.524 15132-15325/com.example.bugapp D/FileDownloader.MessageSnapshotGate: updateKeepFlow 06-07 17:10:55.729 15132-15325/com.example.bugapp D/FileDownloader.MessageSnapshotGate: ~~~callback -818857745 old[6] new[2] 1 06-07 17:10:55.730 15132-15325/com.example.bugapp D/FileDownloader.FileDownloadMessenger: notify connected [e-mail]@458f944 06-07 17:10:55.731 15132-15325/com.example.bugapp D/FileDownloader.MessageSnapshotGate: updateKeepFlow 06-07 17:10:55.831 15132-15325/com.example.bugapp D/FileDownloader.MessageSnapshotGate: ~~~callback -818857745 old[2] new[-1] 1 06-07 17:10:55.832 15132-15325/com.example.bugapp V/FileDownloader.FileDownloadList: remove [e-mail]@458f944 left -1 0 06-07 17:10:55.833 15132-15325/com.example.bugapp D/FileDownloader.FileDownloadMessenger: notify error [e-mail]@458f944 com.liulishuo.filedownloader.exception.FileDownloadGiveUpRetryException: can’t know the size of the download file, and its Transfer-Encoding is not Chunked either. you can ignore such exception by add http.lenient=true to the filedownloader.properties 06-07 17:10:55.833 15132-15325/com.example.bugapp V/FileDownloader.DownloadTaskHunter: filedownloader:lifecycle:over com.liulishuo.filedownloader.DownloadTaskHunter@46d0a2d by -1 06-07 17:10:55.834 15132-15325/com.example.bugapp D/FileDownloader.MessageSnapshotGate: updateKeepFlow

///////////////////////////////////////////////////////////////////////////////////////////////////////////////////////// ///////////////////////////////////////FILEDOWNLOADER service LOG////////////////////////////////////// ///////////////////////////////////////////////////////////////////////////////////////////////////////////////////////// 06-07 17:10:55.401 15304-15304/? I/zygote64: Late-enabling -Xcheck:jni 06-07 17:10:55.523 15304-15324/com.example.bugapp:filedownloader D/NetworkSecurityConfig: No Network Security Config specified, using platform default

[sergiomartinez4]

FYI: I found a workaround for this issue. I added the “Accept-Encoding” header with the “identity” value when downloading VTT files.

BaseDownloadTask task = FileDownloader.getImpl().create(...);
task.addHeader("Accept-Encoding", "identity");
...

Hope this help anyone else having this problem.

[bcdarius]

@rantianhua I think one detail is being omitted here. FileDownloader sets Accept-encoding: gzip in the request

To which server responds with Content-Length of gzipped data which is correct behaviour based on RFC 2616.

The file size you’re seeing after downloading via Chrome and saving to file is most likely uncompressed. That’s probably why you’re seeing discrepancy there. That said, if FileDownloader sends Accept-encoding: gzip in the request and expects Content-Length value of uncompressed file, that is, I believe, incorrect behaviour.