Unable to complete Onboarding, the submit button does not work?

Hi,

I’m trying the single model evaluation example, but after being presented with an onboarding task, I’m not able to move forward to the main task. When I remove the onboarding requirement in example.yaml file, I’m able to complete the task. I also tried model_chat example and got the same result.

I’d really appreciate your help. Thanks!

[JackUrb]

Fixed in https://github.com/facebookresearch/ParlAI/pull/4426/commits/385e97b8fce5a1170eed5e77fdb9c8a7d5d25b30. Closing from here as this is ultimately a ParlAI issue (and will be resolved once mephisto-1.0 merges over there.

[evelynkyl]

Hi @JackUrb - thank you for the prompt response!

Yes, I was using ParlAI with Mephisto version 1.0.0. I checked out the mephisto-1.0 branch in the ParlAI repo but the issue remains.

I tested both model_chat and single_model_eval. Below is the output from single_model_eval

Hydra config:
mephisto:
  blueprint:
    _blueprint_type: model_chat_blueprint
    block_qualification: single_model_eval__${current_time}__block
    onboarding_qualification: single_model_eval__onboarding
    _group: ModelChatBlueprint
    world_file: /home/evelyn/parley/parlai/crowdsourcing/tasks/model_chat/worlds.py
    preview_source: ???
    task_description_file: ${task_dir}/task_config/task_description.html
    custom_source_bundle: ???
    custom_source_dir: /home/evelyn/parley/parlai/crowdsourcing/tasks/model_chat/frontend
    extra_source_dir: ???
    context_csv: ???
    context_jsonl: ???
    num_conversations: ???
    num_turns: 6
    random_seed: 42
    annotation_question: 'Please answer the following:'
    model_opt_path: ${task_dir}/task_config/model_opts.yaml
    task_model_parallel: true
    max_resp_time: 300
    chat_data_folder: ${task_dir}/model_chat/
    check_acceptability: true
    context_seed: ???
    task_config_path: /home/evelyn/parley/parlai/crowdsourcing/tasks/model_chat/task_config
    left_pane_text_path: ${task_dir}/task_config/left_pane_text.html
    annotations_config_path: ${task_dir}/task_config/annotations_config.json
    final_rating_question: 'Please rate how much you''d prefer to talk to your partner
      for a long conversation. (1: Would not at all prefer, 5: Would very much prefer)|Please
      rate how human your partner sounds. (1: Very inhuman, 5: Very human)|Please
      rate how interesting your partner is. (1: Very boring, 5: Very interesting)'
    max_concurrent_responses: 1
    override_opt: {}
    conversation_start_mode: hi
    include_persona: true
    conversations_needed_string: blender_90M:10
    max_onboard_time: 300
    onboard_task_data_path: ${task_dir}/task_config/onboard_task_data.json
  provider:
    _provider_type: mock
    requester_name: MOCK_REQUESTER
  architect:
    _architect_type: local
    server_type: node
    server_source_path: ???
    hostname: localhost
    port: '3000'
  task:
    task_name: single_model_eval__example
    task_title: Chat with a fellow conversationalist!
    task_description: dummy_text
    task_reward: 3.0
    task_tags: chat,conversation,dialog,partner
    assignment_duration_in_seconds: 900
    allowed_concurrent: 1
    maximum_units_per_worker: 1
    max_num_concurrent_units: 5
    submission_timeout: 600
  database:
    _database_type: singleton
  log_level: info
task_dir: /home/evelyn/parley/parlai/crowdsourcing/projects/humaneval/single_model_eval
num_tasks: 5
current_time: 1648569259
mturk:
  worker_blocklist_paths: null
monitoring_log_rate: 30

[2022-03-29 17:54:19,711][mephisto.utils.metrics][WARNING] - Prometheus PID existed, but server doesn't appear to be up.
[2022-03-29 17:54:19,711][mephisto.utils.metrics][WARNING] - Clearing prometheus pid as the server isn't running. Use `shutdown_prometheus_server` in the future for proper cleanup.
[2022-03-29 17:54:19,719][mephisto.utils.metrics][INFO] - Prometheus server launched at process 984766
[2022-03-29 17:54:19,727][root][WARNING] - `validate_and_run_config` functions are deprecated in 1.0, and will be removed in Mephisto 1.1. Use `launch_task_run` versions instead.
[2022-03-29 17:54:19,731][mephisto.operations.operator][INFO] - Creating a task run under task name: single_model_eval__example_local
17:54:19 | WARNING: MTurk task has no_cuda FALSE. Models will run on GPU. Will not work if loading many models at once.
17:54:19 | Got 1 models: dict_keys(['blender_90M']).
17:54:19 | 

--------------------------------
17:54:19 | model_name: blender_90M, opt_dict: {'init_opt': None, 'allow_missing_init_opts': False, 'task': None, 'download_path': None, 'loglevel': 'info', 'datatype': 'train', 'image_mode': 'raw', 'hide_labels': False, 'multitask_weights': [1], 'batchsize': 1, 'dynamic_batching': None, 'verbose': False, 'is_debug': False, 'datapath': '/home/evelyn/parley/data', 'model': None, 'model_file': '/home/evelyn/parley/data/models/blender/blender_90M/model', 'init_model': None, 'dict_class': 'parlai.core.dict:DictionaryAgent', 'image_size': 256, 'image_cropsize': 224, 'embedding_size': 300, 'n_layers': 2, 'ffn_size': 300, 'dropout': 0.0, 'attention_dropout': 0.0, 'relu_dropout': 0.0, 'n_heads': 2, 'learn_positional_embeddings': False, 'embeddings_scale': True, 'n_positions': None, 'n_segments': 0, 'variant': 'aiayn', 'activation': 'relu', 'output_scaling': 1.0, 'share_word_embeddings': True, 'n_encoder_layers': -1, 'n_decoder_layers': -1, 'model_parallel': True, 'checkpoint_activations': False, 'beam_size': 10, 'beam_min_length': 20, 'beam_context_block_ngram': 3, 'beam_block_ngram': 3, 'beam_block_full_context': False, 'beam_length_penalty': 0.65, 'skip_generation': False, 'inference': 'beam', 'topk': 10, 'topp': 0.9, 'beam_delay': 30, 'beam_block_list_filename': None, 'temperature': 1.0, 'compute_tokenized_bleu': False, 'interactive_mode': False, 'embedding_type': 'random', 'embedding_projection': 'random', 'fp16': False, 'fp16_impl': 'safe', 'force_fp16_tokens': False, 'optimizer': 'sgd', 'learningrate': 1, 'gradient_clip': 0.1, 'adam_eps': 1e-08, 'adafactor_eps': (1e-30, 0.001), 'momentum': 0, 'nesterov': True, 'nus': (0.7,), 'betas': (0.9, 0.999), 'weight_decay': None, 'rank_candidates': False, 'truncate': -1, 'text_truncate': None, 'label_truncate': None, 'history_reversed': False, 'history_size': -1, 'person_tokens': False, 'split_lines': False, 'use_reply': 'label', 'add_p1_after_newln': False, 'delimiter': '\n', 'history_add_global_end_token': None, 'special_tok_lst': None, 'gpu': -1, 'no_cuda': False, 'dict_file': None, 'dict_initpath': None, 'dict_language': 'english', 'dict_max_ngram_size': -1, 'dict_minfreq': 0, 'dict_maxtokens': -1, 'dict_nulltoken': '__null__', 'dict_starttoken': '__start__', 'dict_endtoken': '__end__', 'dict_unktoken': '__unk__', 'dict_tokenizer': 're', 'dict_lower': False, 'bpe_debug': False, 'dict_textfields': 'text,labels', 'bpe_vocab': None, 'bpe_merge': None, 'bpe_add_prefix_space': None, 'bpe_dropout': None, 'lr_scheduler': 'reduceonplateau', 'lr_scheduler_patience': 3, 'lr_scheduler_decay': 0.5, 'invsqrt_lr_decay_gamma': -1, 'warmup_updates': -1, 'warmup_rate': 0.0001, 'update_freq': 1, 'parlai_home': '/home/evelyn/parley', 'override': {'model_parallel': True, 'model_file': '/home/evelyn/parley/data/models/blender/blender_90M/model', 'beam_block_full_context': False, 'beam_block_ngram': 3, 'beam_context_block_ngram': 3, 'beam_min_length': 20, 'beam_size': 10, 'inference': 'beam'}, 'starttime': 'Mar29_17-54'}
17:54:19 | Overriding opt["model_file"] to /home/evelyn/parley/data/models/blender/blender_90M/model (previously: /checkpoint/edinan/20200210/baseline_BST_retnref/lr=7.5e-06_attention-dropout=0.0_relu-dropout=0.0/model)
17:54:19 | Using CUDA
17:54:19 | loading dictionary from /home/evelyn/parley/data/models/blender/blender_90M/model.dict
17:54:19 | num words = 54944
17:54:20 | DEPRECATED: XLM should only be used for backwards compatibility, as it involves a less-stable layernorm operation.
17:54:22 | Total parameters: 87,508,992 (87,508,992 trainable)
17:54:22 | Loading existing model params from /home/evelyn/parley/data/models/blender/blender_90M/model
17:54:22 | WARNING: Test set not included. Setting datatype to valid.
17:54:22 | loading fbdialog data: /home/evelyn/parley/data/ConvAI2/valid_both_original.txt
[EmpatheticDialoguesTeacher] Only use experiencer side? True, datatype: test
loading: /home/evelyn/parley/data/wizard_of_wikipedia/test_random_split.json
17:54:23 | Some data not being used. If you are not trying to reproduce the previous results, it is recommended that you run with the flag --add-missing-turns train or --add-missing-turns all.
Starting to map topics to episodes.
Finished mapping topics to episodes.
Starting to map personas to topics.
Finished mapping personas to topics.
npm WARN server@1.0.0 No description
npm WARN server@1.0.0 No repository field.
npm WARN server@1.0.0 No license field.

audited 74 packages in 0.561s
found 1 high severity vulnerability
  run `npm audit fix` to fix them, or `npm audit` for details
[2022-03-29 17:54:39,567][sh.command][INFO] - <Command '/usr/bin/rm -rf /home/evelyn/Mephisto/data/data/runs/NO_PROJECT/35/build/router'>: starting process
[2022-03-29 17:54:39,622][sh.command][INFO] - <Command '/usr/bin/rm -rf /home/evelyn/Mephisto/data/data/runs/NO_PROJECT/35/build/router', pid 984959>: process started
[2022-03-29 17:54:39,700][sh.command][INFO] - <Command '/usr/bin/rm -rf /home/evelyn/Mephisto/data/data/runs/NO_PROJECT/35/build/router', pid 984959>: process completed
Listening on 3000
Server running locally with pid 984964.
[2022-03-29 17:54:40,893][sh.command][INFO] - <Command '/usr/bin/rm -rf /home/evelyn/Mephisto/data/data/runs/NO_PROJECT/35/build/router'>: starting process
[2022-03-29 17:54:40,948][sh.command][INFO] - <Command '/usr/bin/rm -rf /home/evelyn/Mephisto/data/data/runs/NO_PROJECT/35/build/router', pid 984974>: process started
[2022-03-29 17:54:41,030][sh.command][INFO] - <Command '/usr/bin/rm -rf /home/evelyn/Mephisto/data/data/runs/NO_PROJECT/35/build/router', pid 984974>: process completed
[2022-03-29 17:54:41,031][mephisto.operations.client_io_handler][INFO] - Sending alive
[2022-03-29 17:54:41,031][mephisto.operations.client_io_handler][INFO] - Sending alive
[2022-03-29 17:54:41,040][mephisto.operations.client_io_handler][INFO] - Sending alive
[2022-03-29 17:54:41,040][mephisto.abstractions.architects.channels.websocket_channel][INFO] - channel open
Client connected
System socket attached:
::1
[2022-03-29 17:54:41,391][mephisto.operations.operator][INFO] - Operator running task ID = 35
localhost:3000
Mock task launched: localhost:3000 for preview, localhost:3000/?worker_id=x&assignment_id=665
[2022-03-29 17:54:41,393][mephisto.abstractions.providers.mock.mock_unit][INFO] - Mock task launched: localhost:3000 for preview, localhost:3000/?worker_id=x&assignment_id=665 for assignment 664
localhost:3000
Mock task launched: localhost:3000 for preview, localhost:3000/?worker_id=x&assignment_id=666
[2022-03-29 17:54:41,395][mephisto.abstractions.providers.mock.mock_unit][INFO] - Mock task launched: localhost:3000 for preview, localhost:3000/?worker_id=x&assignment_id=666 for assignment 665
localhost:3000
Mock task launched: localhost:3000 for preview, localhost:3000/?worker_id=x&assignment_id=667
[2022-03-29 17:54:41,397][mephisto.abstractions.providers.mock.mock_unit][INFO] - Mock task launched: localhost:3000 for preview, localhost:3000/?worker_id=x&assignment_id=667 for assignment 666
localhost:3000
Mock task launched: localhost:3000 for preview, localhost:3000/?worker_id=x&assignment_id=668
[2022-03-29 17:54:41,399][mephisto.abstractions.providers.mock.mock_unit][INFO] - Mock task launched: localhost:3000 for preview, localhost:3000/?worker_id=x&assignment_id=668 for assignment 667
localhost:3000
Mock task launched: localhost:3000 for preview, localhost:3000/?worker_id=x&assignment_id=669
[2022-03-29 17:54:41,401][mephisto.abstractions.providers.mock.mock_unit][INFO] - Mock task launched: localhost:3000 for preview, localhost:3000/?worker_id=x&assignment_id=669 for assignment 668
[2022-03-29 17:54:48,465][mephisto.operations.worker_pool][INFO] - MockWorker(1) is starting onboarding thread with onboarding OnboardingAgent(20).
ModelChatOnboardWorld: starting parley for worker_id: --NOT-MTURK-AGENT-x_sandbox
Client connected
[2022-03-29 17:55:11,398][mephisto.operations.operator][INFO] - Operator running task ID = 35

Thanks so much!