question-mark
Stuck on an issue?

Lightrun Answers was designed to reduce the constant googling that comes with debugging 3rd party libraries. It collects links to all the places you might be looking at while hunting down a tough bug.

And, if you’re still stuck at the end, we’re happy to hop on a call to see how we can help out.

Issue with LONG and INT columns when extracting from Oracle

See original GitHub issue

Hello,

I am extracting data from oracle and generating avro schema based on column types from cx_Oracle. I have the following:

` if defaultType == cx_Oracle.NUMBER:

type = {
    "name": name,
    "type": ["null", {"name": name, "type": "long"}]
}
avro_schema.append(type)

` The main issue is that output_type_handler in cx_oracle puts LONG and INT under NUMBER datatype (or at least that’s how dba had the tables setup). When trying to generate the avro file with the schema and I encounter an INT column, I get the following error:

ValueError: 3453453453532423323L (type <type 'long'>) do not match ['null', {'type': 'long'}]

So i tried to convert all int and long to decimal:

` if defaultType == cx_Oracle.NUMBER:

type = {
    "name": name,
    "type": ["null", {"name": name, "type": "bytes", "logicalType": "decimal", "precision": 38, "scale": 0}]
}
avro_schema.append(type)

`

and i get error (i also tried replacing bytes with long) and same issue:

with long:

ValueError: 3453453453532423323L (type <type 'long'>) do not match [{'logicalType': 'decimal', 'scale': 0, 'type': 'long', 'precision': 38}, 'null']

with bytes:

ValueError: 3453232446456443 (type <type 'int'>) do not match [{'logicalType': 'decimal', 'scale': 0, 'type': 'bytes', 'precision': 38}, 'null']

Please let me know if there is any way around this.

I am using python==2.7.12 fastavro==0.21.5 cx-Oracle==6.2

Thank you.

Issue Analytics

  • State:closed
  • Created 5 years ago
  • Comments:13 (2 by maintainers)

github_iconTop GitHub Comments

1reaction
scottbeldencommented, Jul 9, 2019

I’m going to close this. I don’t have a way to reproduce it and my suspicion is that they are using the C-API to create their NUMBER such that it claims to be type <int> but doesn’t pass the isinstance test.

0reactions
scottbeldencommented, Oct 4, 2018

@oyamin Somewhere in your code where you have one of these cx_Oracle.NUMBER objects could you add the following prints and let me know what they return?

# some_value is the cx_Oracle.NUMBER
print("isinstance({}, int) = {}".format(some_value, isinstance(some_value, int)))
print("isinstance({}, long) = {}".format(some_value, isinstance(some_value, long)))
import numbers
print("isinstance({}, numbers.Number) = {}".format(some_value, isinstance(some_value, numbers.Number)))
print("type({}) = {}".format(some_value, type(some_value)))


Read more comments on GitHub >

github_iconTop Results From Across the Web

Conversion/Extraction of Longs - Ask TOM
I'm pulling data into Oracle from access and carrying out some validation on the data retrieved ( checking the length of remote columns...
Read more >
how to select and insert a long type column - Ask TOM
The SQLPLUS copy command works as well. In Oracle8i, release 8.1.6 which you have, the correct datatype to use is a CLOB. Longs...
Read more >
ORA-01401: Inserted value too large for column - Ask TOM
It doesn't tell you which column is too large, it leaves it to us to figure out which column is causing problem.
Read more >
Data Type Support and Conversion in Oracle Database ...
7 Data Type Support and Conversion in Oracle Database Provider for DRDA. This chapter discusses data type support in Oracle, and conversion between...
Read more >
Long to Varchar2 conversion.... - Ask TOM
in the particular remarks (long datatype) column. suppose if the length of data increase means, it returns just null... how can i solve...
Read more >

github_iconTop Related Medium Post

No results found

github_iconTop Related StackOverflow Question

No results found

github_iconTroubleshoot Live Code

Lightrun enables developers to add logs, metrics and snapshots to live code - no restarts or redeploys required.
Start Free

github_iconTop Related Reddit Thread

No results found

github_iconTop Related Hackernoon Post

No results found

github_iconTop Related Tweet

No results found

github_iconTop Related Dev.to Post

No results found

github_iconTop Related Hashnode Post

No results found