Parameterized unload throws an error

Driver version

2.0.877

Redshift version

PostgreSQL 8.0.2 on i686-pc-linux-gnu, compiled by GCC gcc (GCC) 3.4.2 20041017 (Red Hat 3.4.2-6.fc3), Redshift 1.0.25109

Client Operating System

Ubuntu 20.04

Python version

3.8.5

Table schema

N/A

Problem description

1. Expected behaviour: An unload is executed successfully when a parameterized expression is used.
2. Actual behaviour: An error is thrown.
3. Error message/stack trace:

> python3 test_unload.py
Traceback (most recent call last):
  File "test_unload.py", line 16, in <module>
    cursor.execute(f"""
  File ".../lib/python3.8/site-packages/redshift_connector/cursor.py", line 162, in execute
    self._c.execute(self, operation, args)
  File ".../lib/python3.8/site-packages/redshift_connector/core.py", line 1016, in execute
    self.handle_messages(cursor)
  File ".../lib/python3.8/site-packages/redshift_connector/core.py", line 1172, in handle_messages
    raise self.error
redshift_connector.error.ProgrammingError: {'S': 'ERROR', 'C': '42601', 'M': 'syntax error at or near "$1"', 'P': '14', 'F': '/home/ec2-user/padb/src/pg/src/backend/parser/parser_scan.l', 'L': '718', 'R': 'yyerror'}

1. Any other details that can be helpful: the test code works when all parameters are hardcoded.

Python Driver trace logs

N/A

Reproduction code

test_unload.py:

import os

import redshift_connector

conn = redshift_connector.connect(
    host=os.getenv("REDSHIFT_HOST"),
    database=os.getenv("REDSHIFT_DATABASE"),
    user=os.getenv("REDSHIFT_USER"),
    password=os.getenv("REDSHIFT_PASSWORD"),
)

destination = os.getenv("REDSHIFT_UNLOAD_DESTINATION")
role = os.getenv("REDSHIFT_UNLOAD_ROLE")

cursor = conn.cursor()
cursor.execute(f"""
    UNLOAD (%s)
    TO '{destination}'
    IAM_ROLE '{role}'
    PARALLEL OFF
""", ['SELECT id FROM mydb.accounts WHERE id = 0'])
print(cursor.fetchall())

[Brooke-white]

Hey @djciregethigher ,

Thank you for sharing your findings and congrats on your first post 😃. When comparing the output from the two code blocks, it seems to be identical. Could you please help me understand?

startDate = '2022-01-01'
endDate = '2022-02-01'

string1 = f""" select * from table where time between {startDate} and {endDate}"""
print(string1)

startDate = '2022-01-01'
endDate = '2022-02-01'

string2 = """ select * from table where time between {0} and {1}""".format(startDate, endDate)
print(string2)

print(string1 == string2)

>>> select * from table where time between 2022-01-01 and 2022-02-01
>>> select * from table where time between 2022-01-01 and 2022-02-01
>>>True

Not sure if this applies to your use case, but redshift_connector supports parameterization without needing to use format strings which can be a bit cleaner and provide protection against sql injection. I’ve put an example of how to do this below using Redshift MONTHS_BETWEEN function.

with redshift_connector.connect(...) as conn:
    with conn.cursor() as cursor:
        cursor.execute("select months_between(%s,%s);", (endDate, startDate))

>>> ([1.0],)

If you continue to experience issue with this or any other redshift_connector functionality feel free to open a new issue and we can work to debug/fix any issue at hand 😃.

[djciregethigher]

Hi @Brooke-white, thank you for the solution for parametrizing dates!

In reference to my findings, I’m using slightly different techniques to create/modify/parametrize query-strings of SQL that I then use as an input for the .execute() method.

I found that using conventional f-strings to create formatted strings were throwing me an error due to all the reasons discussed in this thread.

I decided to use the Python’s native .format() method for strings to build the string first, and then feed the string to the .execute() method of Redshift Connector.

To rephrase,

instead of using this:

https://docs.python.org/3/tutorial/inputoutput.html#tut-f-strings

I use this to create your strings before feeding them to .execute():

https://docs.python.org/3/library/stdtypes.html#str.format

I hope this better helps you understand my findings!

Cheers,

DJ CIRE

P.S. if not helpful, then please disregard! LOL