ValueError: invalid literal for int() with base 36

Hi Team,

I am trying to run diff between postgres and bq. My primary key column is string(uuid). Example - ‘zZsNL-xxt2o-REN’

When doing the diff, I am getting the following exception:

[2022-09-05, 11:40:45 AEST] {taskinstance.py:1902} ERROR - Task failed with exception
Traceback (most recent call last):
  File "/home/airflow/.local/lib/python3.8/site-packages/data_diff/diff_tables.py", line 421, in _parse_key_range_result
    return cls(mn), cls(mx) + 1
  File "/home/airflow/.local/lib/python3.8/site-packages/data_diff/utils.py", line 94, in __add__
    res = self.new(int=self.int + other)
  File "/home/airflow/.local/lib/python3.8/site-packages/data_diff/utils.py", line 75, in int
    return int(self._str, len(alphanums))
ValueError: invalid literal for int() with base 36: 'zZsNL-xxt2o-REN'

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/home/airflow/.local/lib/python3.8/site-packages/airflow/operators/python.py", line 171, in execute
    return_value = self.execute_callable()
  File "/home/airflow/.local/lib/python3.8/site-packages/airflow/operators/python.py", line 189, in execute_callable
    return self.python_callable(*self.op_args, **self.op_kwargs)
  File "/opt/airflow/dags/dataquality_metrics/utils/quality_metrics.py", line 54, in find_missing_rows
    raise ev
  File "/opt/airflow/dags/dataquality_metrics/utils/quality_metrics.py", line 48, in find_missing_rows
    diff = list(dif_iter)
  File "/home/airflow/.local/lib/python3.8/site-packages/data_diff/diff_tables.py", line 414, in diff_tables
    raise error
  File "/home/airflow/.local/lib/python3.8/site-packages/data_diff/diff_tables.py", line 367, in diff_tables
    min_key1, max_key1 = self._parse_key_range_result(key_type, next(key_ranges))
  File "/home/airflow/.local/lib/python3.8/site-packages/data_diff/diff_tables.py", line 423, in _parse_key_range_result
    raise type(e)(f"Cannot apply {key_type} to {mn}, {mx}.") from e
ValueError: Cannot apply String_Alphanum(length=15) to 016AL-NjNQP-REN, zZsNL-xxt2o-REN.

I have tried recreating this error by taking the following samples from utils.py code

import string 
alphanums = string.digits + string.ascii_lowercase
def numberToBase(num, base):
    digits = []
    while num > 0:
        num, remainder = divmod(num, base)
        digits.append(remainder)
    return "".join(alphanums[i] for i in digits[::-1])

xx = 'zZsNL-xxt2o-REN'
yy = int(xx, len(alphanums))
str = numberToBase(yy, len(alphanums))  

[erezsh]

Yes, I understand your use-case, it makes sense to me.

Keys of varying size aren’t supported right now, but they are on our TODO. I believe they shouldn’t be too hard to add.

As for (2), I wouldn’t consider - to be alphanumeric. Do you have any other non-alphanumeric characters? Or is it only “a” to “z”, “A” to “Z”, and “-” ?

[erezsh]

Fixed and merged to master.