Custom boolean values create a field_error when trying to validate

Overview

We have a data package that defines custom values to represent “true” or “false” on boolean fields, as defined in the specs. When trying to validate this, frictionless-py raises a field_error.

How to reproduce

1. Create a file named bool.csv like this:

id,question
0,confirm
1,deny
2,deny
3,confirm

1. Create a Python environment and pip install frictionless==4.1.2.

2. Code to test:

from frictionless import describe_resource, validate_schema, validate_resource

r = describe_resource('bool.csv')
f = r.schema.get_field('question')
f.type = 'boolean'
f.format = {
        'trueValues' : ['confirm'],
        'falseValues' : ['deny']
}

1. Now the problem:

[6]: validate_schema(r.schema)

[6]: {'version': '4.2.1',
 'time': 0.001,
 'errors': [{'code': 'field-error',
   'name': 'Field Error',
   'tags': [],
   'note': '"{\'name\': \'question\', \'type\': \'boolean\', \'format\': {\'trueValues\': [\'confirm\'], \'falseValues\': [\'deny\']}} is not valid under any of the given schemas" at "" in metadata and at "anyOf" in profile',
   'message': 'Field is not valid: "{\'name\': \'question\', \'type\': \'boolean\', \'format\': {\'trueValues\': [\'confirm\'], \'falseValues\': [\'deny\']}} is not valid under any of the given schemas" at "" in metadata and at "anyOf" in profile',
   'description': 'Provided field is not valid.'}],
 'tasks': [],
 'stats': {'errors': 1, 'tasks': 0},
 'valid': False}

1. Also:

[7]: validate_resource(r)

[7]: {'version': '4.2.1',
 'time': 0.004,
 'errors': [{'code': 'field-error',
   'name': 'Field Error',
   'tags': [],
   'note': '"{\'name\': \'question\', \'type\': \'boolean\', \'format\': {\'trueValues\': [\'confirm\'], \'falseValues\': [\'deny\']}} is not valid under any of the given schemas" at "" in metadata and at "anyOf" in profile',
   'message': 'Field is not valid: "{\'name\': \'question\', \'type\': \'boolean\', \'format\': {\'trueValues\': [\'confirm\'], \'falseValues\': [\'deny\']}} is not valid under any of the given schemas" at "" in metadata and at "anyOf" in profile',
   'description': 'Provided field is not valid.'}],
 'tasks': [],
 'stats': {'errors': 1, 'tasks': 0},
 'valid': False}

---

Please preserve this line to notify @roll (lead of this repository)

[augusto-herrmann]

Oh, I see.

Thanks for looking into it, @roll . 😄

Sorry for misunderstanding the specs. 😟

[roll]

Hi @augusto-herrmann

I guess, the specs’ text is confusing. It’s not the field.format it’s the field.[property]:

from pprint import pprint
from frictionless import describe_resource, validate_schema, validate_resource

r = describe_resource("tmp/issue773.csv")
f = r.schema.get_field("question")
f.type = "boolean"
f.true_values = ["confirm"]
f.false_values = ["deny"]
pprint(r.schema.metadata_valid)
pprint(r.schema)

True
{'fields': [{'name': 'id', 'type': 'integer'},
            {'falseValues': ['deny'],
             'name': 'question',
             'trueValues': ['confirm'],
             'type': 'boolean'}]}