String size exceeds the maximum allowed size of 131072 in U-SQL script

Hello, I am hitting the error of string data type size limitation of 128KB during extract phase.

I tried the option of Extractors.Tsv(silent:true) to omit the corresponding rows which does not fit the 128KB but that doesn’t seem to solve the problem.

I do not want to parse using custom extractors and want to consume the entire line and then process if necessary.

Code snippet: @eventLogs = EXTRACT Event string FROM @path USING Extractors.Tsv(silent:true);

Error: MESSAGE: String size 215710 exceeds the maximum allowed size of 131072

Questions: • Is there a way to store larger literals than 128KB during extraction? • If not is there a way to omit those lines instead of run-time failure?

[narensoft]

Can you please share the sample with us @MikeRys and @ullasm

[anbudhanush]

Hi Mike, Big thank you for this piece. It is a life saver. Is it mandatory to have the start the document with “[” and end with “]”. Sometimes I have to churn bigdata sets of json through this extractor and it is bit of work to add these brackets in those massive files. I do it with emeditor.