spark input_file_name() not working in cobrix

Hi,

Thank you for creating and maintaining Cobrix. It’s a tool we discovered recently, and plan to implement it in our cloud data platform for our Mainframe project.

Just a small question to ask. We notice the input_file_name() command in spark always returns blanks when using cobrix. This in combination with the option(“is_record_sequence”, “true”) option.

spark.read.format("cobol").option("copybook", "/mnt/inputMDP/BIWA_GUTEX/Copybooks/"+dbutils.widgets.get("version")+"/GAGUSECO_20070115.txt").option("is_record_sequence", "true").load("/mnt/inputMDP/BIWA_GUTEX/Datafiles/"+dbutils.widgets.get("version")+"/GA-GA324001*").withColumn("ISN_Source", input_file_name).createOrReplaceTempView("vw_gutex_GA")

Do you notice the same behaviour? Is there any chance to get this working?

Keep up the good work!

Regards,

Kris

[kriswijnants]

Hi Ruslan,

I just tried, and it works perfect! It’s now showing the filename of ebcdic files using the option is_record_sequence = true.

Thanks a lot for your efforts!

Regards,

Kris

Kris Wijnants Innovation Wizard m [phone]

From: Ruslan Yushchenko [e-mail] Sent: vrijdag 20 december 2019 13:53 To: AbsaOSS/cobrix [e-mail] Cc: Wijnants Kris [e-mail]; Author [e-mail] Subject: Re: [AbsaOSS/cobrix] spark input_file_name() not working in cobrix (#221)

Hi Kris,

Snapshot version linking requires additional configuration in .m2/settings.xml. It might be even harder for managed clusters.

Try setting the version to 2.0.1 which was released today.

And please let me know if it worked for you.

Thank you, Ruslan

— You are receiving this because you authored the thread. Reply to this email directly, view it on GitHubhttps://eur02.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2FAbsaOSS%2Fcobrix%2Fissues%2F221%3Femail_source%3Dnotifications%26email_token%3DANWTSU5KQGAQARCJRBOVQ5TQZS53PA5CNFSM4JWVEGA2YY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOEHM3DJA%23issuecomment-567914916&data=02|01|kris.wijnants%40kohera.be|70dd2f07eec548269d8e08d7854b92b6|49c3d703357947bfa8887c913fbdced9|0|0|637124431937930582&sdata=au4S1vmXJI2QBWqOMbfmBhfV2WWfv5aLPA6ZOdZJYxg%3D&reserved=0, or unsubscribehttps://eur02.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Fnotifications%2Funsubscribe-auth%2FANWTSU6Z6E2GAZPXY47OKPTQZS53PANCNFSM4JWVEGAQ&data=02|01|kris.wijnants%40kohera.be|70dd2f07eec548269d8e08d7854b92b6|49c3d703357947bfa8887c913fbdced9|0|0|637124431937940579&sdata=yDl50HT2c2RxFEJLJbYJDt7jZo%2FxNl%2F3zuMiJ7WGK9g%3D&reserved=0.

This email has been scanned by BullGuard antivirus protection. For more info visit www.bullguard.comhttp://www.bullguard.com/tracking.aspx?affiliate=bullguard&buyaffiliate=smtp&url=/

[kriswijnants]

Thanks!

Kris Wijnants Innovation Wizard m [phone]

From: Ruslan Yushchenko [e-mail] Sent: vrijdag 21 februari 2020 16:12 To: AbsaOSS/cobrix [e-mail] Cc: Wijnants Kris [e-mail]; Author [e-mail] Subject: Re: [AbsaOSS/cobrix] spark input_file_name() not working in cobrix (#221)

Interesting. I will take a look. I think this can be easily fixed so that with_input_file_name_col would work in your case.

— You are receiving this because you authored the thread. Reply to this email directly, view it on GitHubhttps://eur02.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2FAbsaOSS%2Fcobrix%2Fissues%2F221%3Femail_source%3Dnotifications%26email_token%3DANWTSU2SY7IF7TKCEUFO6FDRD7VMTA5CNFSM4JWVEGA2YY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOEMTAOJI%23issuecomment-589694757&data=02|01|kris.wijnants%40kohera.be|ffffb189c9a44811c11308d7b6e06c31|49c3d703357947bfa8887c913fbdced9|0|0|637178947318866001&sdata=afZ%2FCBk4Dk7cazHhvPHbmzZt7Zx%2FKHATWoTO%2FLv%2B52o%3D&reserved=0, or unsubscribehttps://eur02.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Fnotifications%2Funsubscribe-auth%2FANWTSU5WHWXW4RVZO7NOOWLRD7VMTANCNFSM4JWVEGAQ&data=02|01|kris.wijnants%40kohera.be|ffffb189c9a44811c11308d7b6e06c31|49c3d703357947bfa8887c913fbdced9|0|0|637178947318875997&sdata=vYkRTLaKg%2BDx4tvJXMak3ap8E%2Fmuyot75UtRJBIy4f4%3D&reserved=0.

This email has been scanned by BullGuard antivirus protection. For more info visit www.bullguard.comhttp://www.bullguard.com/tracking.aspx?affiliate=bullguard&buyaffiliate=smtp&url=/