Stuck on an issue?

Lightrun Answers was designed to reduce the constant googling that comes with debugging 3rd party libraries. It collects links to all the places you might be looking at while hunting down a tough bug.

And, if you’re still stuck at the end, we’re happy to hop on a call to see how we can help out.

[BUG] - Dump is completing without backing up all data

See original GitHub issue

Hello,

Below is the index that I’m trying to migrate using backup and restore process which is huge in size 38GB, yellow open ces_articles_1.17 5 1 263551842 2854819 38.2gb 38.2gb

Below is the used command using multielasticdump,

nohup  multielasticdump --direction=dump --match='ces_articles_1.17' --input=<SOURCE_URL> --output=/content/bce/elastic-dump/backup_9_30 --limit=3000 --noRefresh --fsCompress --overwrite=true --includeType='data'> ces_indexes_9_30.log 2>&1 &

Below is the log output for starting and end entries,

Starting entries,

Fri, 30 Sep 2022 03:45:32 GMT | We are performing : dump
Fri, 30 Sep 2022 03:45:32 GMT | options: {"debug":true,"parallel":8,"match":"ces_articles_1.17","order":"asc","input":"<SOURCE_URL>","output":"/content/bce/elastic-dump/backup_9_30","scrollId":null,"scrollTime":"10m","scroll-with-post":false,"timeout":null,"limit":3000,"offset":0,"size":-1,"direction":"dump","support-big-int":false,"big-int-fields":"","ignoreAnalyzer":true,"ignoreChildError":true,"ignoreData":false,"ignoreMapping":true,"ignoreSettings":true,"ignoreTemplate":true,"ignoreAlias":true,"ignoreType":true,"includeType":"data","interval":1000,"delete":false,"prefix":"","suffix":"","transform":null,"headers":null,"searchBody":null,"searchWithTemplate":null,"cert":null,"key":null,"pass":null,"ca":null,"tlsAuth":false,"input-cert":null,"input-key":null,"input-pass":null,"input-ca":null,"output-cert":null,"output-key":null,"output-pass":null,"output-ca":null,"httpAuthFile":null,"concurrency":1,"carryoverConcurrencyCount":true,"intervalCap":5,"concurrencyInterval":5000,"overwrite":true,"fsCompress":true,"awsChain":false,"awsAccessKeyId":null,"awsSecretAccessKey":null,"awsIniFileProfile":null,"awsService":null,"awsRegion":null,"awsUrlRegex":null,"s3AccessKeyId":null,"s3SecretAccessKey":null,"s3Region":null,"s3Endpoint":null,"s3SSLEnabled":true,"s3ForcePathStyle":false,"s3Compress":false,"s3ServerSideEncryption":null,"s3SSEKMSKeyId":null,"s3ACL":null,"quiet":false}
Fri, 30 Sep 2022 03:45:32 GMT [debug] | GET /_aliases
Fri, 30 Sep 2022 03:45:32 GMT [debug] | GET /_aliases -> 200 OK
Fri, 30 Sep 2022 03:45:32 GMT | dumping <SOURCE_URL>/ces_articles_1.17 to /content/bce/elastic-dump/backup_9_30/ces_articles_1.17.json.gz
Fri, 30 Sep 2022 03:45:32 GMT [debug] | fork: /usr/lib/node_modules/elasticdump/bin/elasticdump --type=data,--input=<SOURCE_URL>/ces_articles_1.17,--output=/content/bce/elastic-dump/backup_9_30/ces_articles_1.17.json.gz,--scrollId=null,--scrollTime=10m,--limit=3000,--offset=0,--size=-1,--searchBody=null,--searchWithTemplate=null,--prefix=,--suffix=,--support-big-int=false,--big-int-fields=,--headers=null,--cert=null,--key=null,--pass=null,--ca=null,--tlsAuth=false,--input-cert=null,--input-key=null,--input-pass=null,--input-ca=null,--output-cert=null,--output-key=null,--output-pass=null,--output-ca=null,--httpAuthFile=null,--concurrency=1,--carryoverConcurrencyCount=true,--intervalCap=5,--concurrencyInterval=5000,--overwrite=true,--fsCompress=true,--awsChain=false,--awsAccessKeyId=null,--awsSecretAccessKey=null,--awsIniFileProfile=null,--awsService=null,--awsRegion=null,--awsUrlRegex=null,--s3AccessKeyId=null,--s3SecretAccessKey=null,--s3Region=null,--s3Endpoint=null,--s3SSLEnabled=true,--s3ForcePathStyle=false,--s3Compress=false,--s3ServerSideEncryption=null,--s3SSEKMSKeyId=null,--s3ACL=null,--quiet=false,--scroll-with-post=false

Ending Entries,

Fri, 30 Sep 2022 04:12:31 GMT | sent 3000 objects to destination file, wrote 3000
Fri, 30 Sep 2022 04:12:34 GMT | got 3000 objects from source elasticsearch (offset: 1686000)
Fri, 30 Sep 2022 04:12:35 GMT | sent 3000 objects to destination file, wrote 3000
Fri, 30 Sep 2022 04:12:38 GMT | got 3000 objects from source elasticsearch (offset: 1689000)
Fri, 30 Sep 2022 04:12:39 GMT | sent 3000 objects to destination file, wrote 3000
Fri, 30 Sep 2022 04:12:42 GMT |  dumping all done
Fri, 30 Sep 2022 04:12:42 GMT |  bye

Here the issue is, elastic dump is not backup all the data instead small portion of it and end the process without any issue. Could you plz assist me on this? Or else plz suggest me an alternative solution for this migration, if dump is not supporting that much of size index migration.

Note: I managed to migrate index which is in size 9GB without any issue with dump tool, If you need I can provide that index migration statistics.

And one more question, is this the correct usage for elasticdump with --fsCompress

nohup  elasticdump --input=<SOURCE_URL>/ces_articles_1.17 --output=/content/bce/elastic-dump/backup_9_30/ --limit=10000 --noRefresh --fsCompress --overwrite=true --type='data' > ces_indexes_9_30.log 2>&1 &`