Filters feature on Postgres/Redshift sources not working correctly

When trying to use the filters feature, whenever a Redshift or Postgres is a source or destination, the execution of column validation fails. With an exception from ibis/backends/base_sqlalchemy/compiler.py: Value error: ibis.common.exceptions.OperationNotDefinedError: No translation rule for <class 'third_party.ibis.ibis_addon.operations.RawSQL'>

In this config yaml source is Postgres and target is Redshift.

result_handler: {}
source: Postgres
target: Redshift
validations:
- aggregates:
  - field_alias: count
    source_column: null
    target_column: null
    type: count
  filters: 
  - source: id=1
    target: id=1
    type: custom
  format: table
  grouped_columns:
  - cast: null
    field_alias: created_at
    source_column: created_at
    target_column: created_at
  labels: []
  schema_name: public
  table_name: accounts
  target_schema_name: public
  target_table_name: accounts
  threshold: 0.0
  type: Column

When both source or targets are Redshift or Postgres it also breaks. Here is the exception:

-- ** Source Query ** --
Traceback (most recent call last):
  File "/Users/y.ralko/.local/share/virtualenvs/analytics-dbt-7zrpQIZD/bin/data-validation", line 8, in <module>
    sys.exit(main())
  File "/Users/y.ralko/.local/share/virtualenvs/analytics-dbt-7zrpQIZD/lib/python3.7/site-packages/data_validation/__main__.py", line 373, in main
    run_validations(args, config_managers)
  File "/Users/y.ralko/.local/share/virtualenvs/analytics-dbt-7zrpQIZD/lib/python3.7/site-packages/data_validation/__main__.py", line 315, in run_validations
    run_validation(config_manager, verbose=args.verbose)
  File "/Users/y.ralko/.local/share/virtualenvs/analytics-dbt-7zrpQIZD/lib/python3.7/site-packages/data_validation/__main__.py", line 304, in run_validation
    validator.execute()
  File "/Users/y.ralko/.local/share/virtualenvs/analytics-dbt-7zrpQIZD/lib/python3.7/site-packages/data_validation/data_validation.py", line 94, in execute
    self.validation_builder, process_in_memory=True
  File "/Users/y.ralko/.local/share/virtualenvs/analytics-dbt-7zrpQIZD/lib/python3.7/site-packages/data_validation/data_validation.py", line 246, in _execute_validation
    source_query = validation_builder.get_source_query()
  File "/Users/y.ralko/.local/share/virtualenvs/analytics-dbt-7zrpQIZD/lib/python3.7/site-packages/data_validation/validation_builder.py", line 260, in get_source_query
    print(query.compile())
  File "/Users/y.ralko/.local/share/virtualenvs/analytics-dbt-7zrpQIZD/lib/python3.7/site-packages/ibis/expr/types.py", line 244, in compile
    self, limit=limit, timecontext=timecontext, params=params
  File "/Users/y.ralko/.local/share/virtualenvs/analytics-dbt-7zrpQIZD/lib/python3.7/site-packages/ibis/client.py", line 410, in compile
    expr, limit=limit, params=params, timecontext=timecontext, **kwargs
  File "/Users/y.ralko/.local/share/virtualenvs/analytics-dbt-7zrpQIZD/lib/python3.7/site-packages/ibis/client.py", line 256, in compile
    return query_ast.compile()
  File "/Users/y.ralko/.local/share/virtualenvs/analytics-dbt-7zrpQIZD/lib/python3.7/site-packages/ibis/backends/base_sqlalchemy/compiler.py", line 55, in compile
    compiled_queries = [q.compile() for q in self.queries]
  File "/Users/y.ralko/.local/share/virtualenvs/analytics-dbt-7zrpQIZD/lib/python3.7/site-packages/ibis/backends/base_sqlalchemy/compiler.py", line 55, in <listcomp>
    compiled_queries = [q.compile() for q in self.queries]
  File "/Users/y.ralko/.local/share/virtualenvs/analytics-dbt-7zrpQIZD/lib/python3.7/site-packages/ibis/backends/base_sqlalchemy/alchemy.py", line 1333, in compile
    frag = step(frag)
  File "/Users/y.ralko/.local/share/virtualenvs/analytics-dbt-7zrpQIZD/lib/python3.7/site-packages/ibis/backends/base_sqlalchemy/alchemy.py", line 1421, in _add_where
    self._translate(pred, permit_subquery=True) for pred in self.where
  File "/Users/y.ralko/.local/share/virtualenvs/analytics-dbt-7zrpQIZD/lib/python3.7/site-packages/ibis/backends/base_sqlalchemy/alchemy.py", line 1421, in <listcomp>
    self._translate(pred, permit_subquery=True) for pred in self.where
  File "/Users/y.ralko/.local/share/virtualenvs/analytics-dbt-7zrpQIZD/lib/python3.7/site-packages/ibis/backends/base_sqlalchemy/compiler.py", line 1598, in _translate
    return translator.get_result()
  File "/Users/y.ralko/.local/share/virtualenvs/analytics-dbt-7zrpQIZD/lib/python3.7/site-packages/ibis/backends/base_sqlalchemy/compiler.py", line 1362, in get_result
    translated = self.translate(self.expr)
  File "/Users/y.ralko/.local/share/virtualenvs/analytics-dbt-7zrpQIZD/lib/python3.7/site-packages/ibis/backends/base_sqlalchemy/compiler.py", line 1404, in translate
    'No translation rule for {}'.format(type(op))
ibis.common.exceptions.OperationNotDefinedError: No translation rule for <class 'third_party.ibis.ibis_addon.operations.RawSQL'>

Please add support for Postgres/Redshift RawSQL.

[letmardi]

@helensilva14 @nehanene15 thanks! now everything works correctly

[helensilva14]

Hello everyone! Since today is a holiday in US (Labor Day), we’re still pending review at the fix for this issue, but if you’re willing to test it already, you can see the changes at #576, check out to the specified branch and install this version locally using pip install . at the root repo folder. Any feedbacks are appreciated, thank you!