Type and Field extension / change behavior

Overview

I’m using Frictionless pretty much as Schema Detector. I have two issues here:

1. I would like to introduce a custom type called BigInt to be able to differentiate normal Integer from a BigInt. Issue is that for how it works the detector, as soon it find a correct type (https://github.com/frictionlessdata/frictionless-py/blob/a0cd70d03df4ca6571b1e8b26445e6274ba46c3d/frictionless/detector.py#L270), it will break from the loop of the runners, invalidate in case a custom BigInteger type cause the Integer type would always succeed before.

2. I have a issue also with datetime format, still in a scenario of schema detection. Since automatically the detector will apply a default as format, instead of a any for a timestamp, it will always arise a issue on the assert (https://github.com/frictionlessdata/frictionless-py/blob/a0cd70d03df4ca6571b1e8b26445e6274ba46c3d/frictionless/types/datetime.py#L32) avoiding in case a correct parser.parse(cell) for different format that doesn’t follow isoformat.

I’m not sure how I could patch in case these two cases, and if it’s even possible.

---

Please preserve this line to notify @roll (lead of this repository)

[antoniocali]

Thank you. It was very on point!

[roll]

@antoniocali Hi, from frictionless@4.16 there is a plugin.create_candidates hook. Here is a full example with a custom type:

    # Type
    class CustomType(Type):
        def read_cell(self, cell):
            return [cell] # make everything an array just for example

    # Plugin
    class CustomPlugin(Plugin):
        def create_candidates(self, candidates):
            candidates.insert(0, {"type": "custom"})

        def create_type(self, field):
            if field.type == "custom":
                return CustomType(field)

    # System
    plugin = CustomPlugin()
    system.register("custom", plugin)

That’s how system.create_candidates returns without custom plugins (a detector will try these types on an detected cell from top to bottom):

[{'type': 'yearmonth'},
 {'type': 'geopoint'},
 {'type': 'duration'},
 {'type': 'geojson'},
 {'type': 'object'},
 {'type': 'array'},
 {'type': 'datetime'},
 {'type': 'time'},
 {'type': 'date'},
 {'type': 'integer'},
 {'type': 'number'},
 {'type': 'boolean'},
 {'type': 'year'},
 {'type': 'string'}]

Using this hook you can also alter other detections details, for example, insert {'type': 'datetime', 'format': 'any'} before the default datetime candidate.