Stuck on an issue?

Lightrun Answers was designed to reduce the constant googling that comes with debugging 3rd party libraries. It collects links to all the places you might be looking at while hunting down a tough bug.

And, if you’re still stuck at the end, we’re happy to hop on a call to see how we can help out.

EmrEtlRunner: fix srcPattern for copying stream enriched data to HDFS

See original GitHub issue

When we’re resuming from shred in stream enrich mode, S3DistCp tries to copy data from enriched/good, not from enriched/good/run=2018... and fails because of that.

Due this bug, we can recover R102 stream enrich mode only by re-staging enriched data back to enriched.stream.

Issue Analytics

State:
Created 5 years ago
Comments:15 (15 by maintainers)

Top GitHub Comments

2reactions

chuwycommented, Apr 16, 2018

Sorry, I was wrong this regex handles $folder$ files and tries to move them from enriched/good to HDFS for shredding. I cannot find any evidences that S3DistCp ignores empty files, so I think it still would be better to stick with .gz.

1reaction

chuwycommented, Apr 13, 2018

Turns out fix should (can) be different from one described in title. When we’re resuming from shred in batch-enrich mode, S3DistCp knows nothing about run-folder and also uses enriched/good, but with --srcPattern .*part-.*, which seems to handle files from subfolders. With stream-enrich mode we use --srcPattern .+, which somehow doesn’t handle subfolders in the same way.