Trouble running "Quick Start"-scripts

Hi! Firstly, thanks for publishing your research and models! 😃

I have trouble evaluating the NER model with the given command CUDA_VISIBLE_DEVICES=0 python3 run_acener.py --model_type bertspanmarker ... The output is a json-file with only one line: {"dev_best_f1": 0}

The last 3 lines of the log-output are:

02/04/2022 17:37:16 - INFO - __main__ -   Training/evaluation parameters Namespace(adam_epsilon=1e-08, alpha=1, cache_dir='', config_name='', data_dir='../scierc/raw_data', dev_file='dev.json', device=device(type='cuda'), do_eval=True, do_lower_case=True, do_test=False, do_train=False, eval_all_checkpoints=True, evaluate_during_training=True, fp16=True, fp16_opt_level='O1', gradient_accumulation_steps=1, group_axis=-1, group_edge=False, group_sort=False, learning_rate=2e-05, lminit=True, local_rank=-1, logging_steps=5, max_grad_norm=1.0, max_mention_ori_length=8, max_pair_length=256, max_seq_length=512, max_steps=-1, model_name_or_path='../bert_models/scibert_scivocab_uncased', model_type='bertspanmarker', n_gpu=1, no_cuda=False, no_test=False, norm_emb=False, num_train_epochs=50.0, onedropout=True, output_dir='../sciner_models/sciner-scibert', output_results=True, overwrite_cache=False, overwrite_output_dir=True, per_gpu_eval_batch_size=16, per_gpu_train_batch_size=8, save_steps=2000, save_total_limit=1, seed=42, server_ip='', server_port='', shuffle=False, test_file='test.json', tokenizer_name='', train_file='train.json', use_full_layer=-1, warmup_steps=-1, weight_decay=0.0)
02/04/2022 17:37:16 - INFO - __main__ -   Evaluate on test set
02/04/2022 17:37:16 - INFO - __main__ -   Evaluate the following checkpoints: []

As you can see in the first line, I changed the original command in the following way:

1. --model_name_or_path ../bert_models/scibert_scivocab_uncased I couldn’t find a folder “scibert-uncased”, so I downloaded the 4th model from huggingface as described in the “Training Script”-section (AllenAI) - is this maybe the wrong model?
2. --data_dir ../scierc/raw_data I downloaded the SciERC raw_data from their website to execute the evaluation on - is this the wrong dataset?

[Clemens123]

Thanks for the help, I got it running now 😃

For future reference (or you might add it to the repo-description): The problem was the wrong data path: you need SCIERC_processed and then refer to the json-folder as data_dir. Note that the data-directory-path needs to contain “scierc” in lowercase. Also, the model-path and the output-path have to be the same.

CUDA_VISIBLE_DEVICES=0  python3  run_acener.py  --model_type bertspanmarker  \
    --model_name_or_path  ../sciner-scibert  --do_lower_case  \
    --data_dir ../scierc/processed_data/json  \
    --learning_rate 2e-5  --num_train_epochs 50  --per_gpu_train_batch_size  8  --per_gpu_eval_batch_size 16  --gradient_accumulation_steps 1  \
    --max_seq_length 512  --save_steps 2000  --max_pair_length 256  --max_mention_ori_length 8    \
    --do_eval  --evaluate_during_training   --eval_all_checkpoints  \
    --fp16  --seed 42  --onedropout  --lminit  \
    --train_file train.json --dev_file dev.json --test_file test.json  \
    --output_dir ../sciner-scibert  --overwrite_output_dir  --output_results

Same folder issues for evaluating the relation-extraction model:

1. copy a file from the ner-model to the data_dir: cp ../sciner-scibert/ent_pred_test.json ../scierc/processed_data/json
2. then run the following script:

CUDA_VISIBLE_DEVICES=0  python3  run_re.py  --model_type bertsub  \
    --model_name_or_path  ../scire-scibert  --do_lower_case  \
    --data_dir ../scierc/processed_data/json  \
    --learning_rate 2e-5  --num_train_epochs 10  --per_gpu_train_batch_size  8  --per_gpu_eval_batch_size 16  --gradient_accumulation_steps 1  \
    --max_seq_length 256  --max_pair_length 16  --save_steps 2500  \
    --do_eval  --evaluate_during_training   --eval_all_checkpoints  --eval_logsoftmax  \
    --fp16  --lminit  \
    --test_file ent_pred_test.json  \
    --use_ner_results \
    --output_dir ../scire-scibert

Also, this was the setup it required:

git clone https://github.com/thunlp/PL-Marker.git
sed -i 's/=/==/g' PL-Marker/requirement.txt  #replace wrong syntax in requirement.txt
pip install -r PL-Marker/requirement.txt
pip uninstall -y urllib3 chardet
pip install requests
cd PL-Marker/transformers
python3 setup.py install

• Download of the trained models from Google Cloud and the SCIERC-processed dataset

[YeDeming]

You can try:

CUDA_VISIBLE_DEVICES=0  python3  run_acener.py  --model_type bertspanmarker  \
    --model_name_or_path  downloaded_model_path  --do_lower_case  \
    --data_dir scierc  \
    --learning_rate 2e-5  --num_train_epochs 50  --per_gpu_train_batch_size  8  --per_gpu_eval_batch_size 16  --gradient_accumulation_steps 1  \
    --max_seq_length 512  --save_steps 2000  --max_pair_length 256  --max_mention_ori_length 8    \
    --do_eval  --evaluate_during_training   --eval_all_checkpoints  \
    --fp16  --seed 42  --onedropout  --lminit  \
    --train_file train.json --dev_file dev.json --test_file test.json  \
    --output_dir downloaded_model_path  --overwrite_output_dir  --output_results

And then replace downloaded_model_path to the downloaded trained model path.