train_ner.py train data format to spaCy's json

Hi, I’m trying to use the CLI train command to train a NER model. I was able to train it following the example from train_ner.py on which the data needed to be formatted like this:

TRAIN_DATA = [
    ("Who is Shaka Khan?", {"entities": [(7, 17, "PERSON")]}),
    ("I like London and Berlin.", {"entities": [(7, 13, "LOC"), (18, 24, "LOC")]}),
]

I now want to use the more powerful CLI.train command, but I have all my data in the format above, is there an existing script for this conversion? As far as I can see this isn’t supported by CLI.convert

Thanks.

Your Environment

• spaCy version: 2.2.4
• Platform: Linux-4.19.104±x86_64-with-Ubuntu-18.04-bionic
• Python version: 3.6.9
• Models: en, es

[svlandeg]

Wouldn’t it make sense to add it into the CLI convert script as another supported format?

You’re right that this has been lacking. For spaCy v.3, we’re working on an overhaul of the convert function and the data formats in general, which should hopefully make all of this more intuitive!

[adrianeboyd]

Here’s my stackoverflow answer on how to do this: https://stackoverflow.com/a/59209377/461847

It would probably make sense to add an example script to do this, since this is the main missing step for people who want to move from the super simple example training scripts to real training with the train CLI.