% string formatting with multiple inputs via of tuples doesn't match python3.5 behavior

I’m new to cython but this seems like incorrect behavior.

Python code put in cytest/test.py:

def test_func_no_tuples(a, b):
  print((a.__class__, b.__class__))
  print((a, b))
  s = 'string being formatted=%d' % a
  print(s)


def test_func_with_tuple(a,b):
  print((a.__class__, b.__class__))
  print((a, b))
  s = 'string being formatted=%d-%d' % (a,b)
  print(s)

test_func_no_tuples(50.,50.)

test_func_with_tuple(50.,50.)

Before compilation the output of python3 -c "from cytest import test" is:

python3 -c "from cytest import test"
(<class 'float'>, <class 'float'>)
(50.0, 50.0)
string being formatted=50
(<class 'float'>, <class 'float'>)
(50.0, 50.0)
string being formatted=50-50

After a pip install of cython==0.29.13 I compile with:

PYTHONLIB=/usr/include/python3.5
CFLAGS="-shared -pthread -fPIC -fwrapv -O2 -Wall -fno-strict-aliasing -I${PYTHONLIB}"
cython --no-docstrings -3 cytest/test.py -o cytest/test.c
gcc $CFLAGS -o cytest/test.so cytest/test.c

Now the output is:

(<class 'float'>, <class 'float'>)
(50.0, 50.0)
string being formatted=50
(<class 'float'>, <class 'float'>)
(50.0, 50.0)
Traceback (most recent call last):
  File "<string>", line 1, in <module>
  File "cytest/test.py", line 16, in init cytest.test
    test_func_with_tuple(50.,50.)
  File "cytest/test.py", line 11, in cytest.test.test_func_with_tuple
    s = 'string being formatted=%d-%d' % (a,b)
ValueError: Unknown format code 'd' for object of type 'float'

This is the error I would get if did 1.2.__format__('d'), but it appears that in the other case cython isn’t using that function.

The relevant c code output by cython is:

  /* "cytest/test.py":4
 *   print((a.__class__, b.__class__))
 *   print((a, b))
 *   s = 'string being formatted=%d' % a             # <<<<<<<<<<<<<<
 *   print(s)
 * 
 */
  __pyx_t_3 = __Pyx_PyUnicode_FormatSafe(__pyx_kp_u_string_being_formatted_d, __pyx_v_a); if (unlikely(!__pyx_t_3)) __PYX_ERR(0, 4, __pyx_L1_error)
  __Pyx_GOTREF(__pyx_t_3);
  __pyx_v_s = ((PyObject*)__pyx_t_3);
  __pyx_t_3 = 0;

  /* "cytest/test.py":11
 *   print((a.__class__, b.__class__))
 *   print((a, b))
 *   s = 'string being formatted=%d-%d' % (a,b)             # <<<<<<<<<<<<<<
 *   print(s)
 * 
 */
  __pyx_t_3 = PyTuple_New(4); if (unlikely(!__pyx_t_3)) __PYX_ERR(0, 11, __pyx_L1_error)
  __Pyx_GOTREF(__pyx_t_3);
  __pyx_t_4 = 0;
  __pyx_t_5 = 127;
  __Pyx_INCREF(__pyx_kp_u_string_being_formatted);
  __pyx_t_4 += 23;
  __Pyx_GIVEREF(__pyx_kp_u_string_being_formatted);
  PyTuple_SET_ITEM(__pyx_t_3, 0, __pyx_kp_u_string_being_formatted);
  __pyx_t_2 = __Pyx_PyObject_Format(__pyx_v_a, __pyx_n_u_d); if (unlikely(!__pyx_t_2)) __PYX_ERR(0, 11, __pyx_L1_error)
  __Pyx_GOTREF(__pyx_t_2);
  __pyx_t_5 = (__Pyx_PyUnicode_MAX_CHAR_VALUE(__pyx_t_2) > __pyx_t_5) ? __Pyx_PyUnicode_MAX_CHAR_VALUE(__pyx_t_2) : __pyx_t_5;
  __pyx_t_4 += __Pyx_PyUnicode_GET_LENGTH(__pyx_t_2);
  __Pyx_GIVEREF(__pyx_t_2);
  PyTuple_SET_ITEM(__pyx_t_3, 1, __pyx_t_2);
  __pyx_t_2 = 0;
  __Pyx_INCREF(__pyx_kp_u_);
  __pyx_t_4 += 1;
  __Pyx_GIVEREF(__pyx_kp_u_);
  PyTuple_SET_ITEM(__pyx_t_3, 2, __pyx_kp_u_);
  __pyx_t_2 = __Pyx_PyObject_Format(__pyx_v_b, __pyx_n_u_d); if (unlikely(!__pyx_t_2)) __PYX_ERR(0, 11, __pyx_L1_error)
  __Pyx_GOTREF(__pyx_t_2);
  __pyx_t_5 = (__Pyx_PyUnicode_MAX_CHAR_VALUE(__pyx_t_2) > __pyx_t_5) ? __Pyx_PyUnicode_MAX_CHAR_VALUE(__pyx_t_2) : __pyx_t_5;
  __pyx_t_4 += __Pyx_PyUnicode_GET_LENGTH(__pyx_t_2);
  __Pyx_GIVEREF(__pyx_t_2);
  PyTuple_SET_ITEM(__pyx_t_3, 3, __pyx_t_2);
  __pyx_t_2 = 0;
  __pyx_t_2 = __Pyx_PyUnicode_Join(__pyx_t_3, 4, __pyx_t_4, __pyx_t_5); if (unlikely(!__pyx_t_2)) __PYX_ERR(0, 11, __pyx_L1_error)
  __Pyx_GOTREF(__pyx_t_2);
  __Pyx_DECREF(__pyx_t_3); __pyx_t_3 = 0;
  __pyx_v_s = ((PyObject*)__pyx_t_2);
  __pyx_t_2 = 0;

It looks like the __Pyx_PyUnicode_FormatSafe function is being used only in the first case and not in the second, which may or may not be relevant. I am using Python 3.5.2.

If there is any other information needed let me know

[scoder]

Looks like this is a difference between %-formatting and format()/f-strings. Cython optimises some %-formatting cases into f-strings, that’s why this occurs.

I think the right fix would be to generate a bit of additional code that checks if the format value is an integer, and if not, passes it through PyNumber_Int(). Tests will show if the behaviour is the same in Py2 and Py3, and if both need this adaptation.

[scoder]

I think I found a nice way of doing this in #3589.