BB3 Response Time

Hi, I’m trying to host a chat service with blenderbot3 (3B) across 5 NVIDIA GeForce RTX 2080 Ti gpus. Here is my config file

tasks:
  default:
    onboard_world: MessengerBotChatOnboardWorld
    task_world: MessengerBotChatTaskWorld
    timeout: 1800
    agents_required: 1
task_name: chatbot
world_module: parlai.chat_service.tasks.chatbot.worlds
overworld: MessengerOverworld
max_workers: 1000
opt:
  debug: True
  models:
    blenderbot3_3B:
      model: projects.seeker.agents.seeker:ComboFidGoldDocumentAgent
      init_opt: gen/r2c2_bb3
      model_file: zoo:bb3/bb3_3B/model
      interactive_mode: True
      no_cuda: False
      override:
        init_opt: gen/r2c2_bb3
        search_server: https://www.google.com
        model_parallel: True
additional_args:
  page_id: 1 # Configure Your Own Page

I was wondering if there are any methods I could use to increase the response time for the bot. Currently it takes the bot around 10-15 seconds to respond with my current setup which seems a bit slow, especially compared to the bb3 demo which is faster and uses a larger model. Let me know if there are any other details I can provide

[mojtaba-komeili]

3-4 seconds seems reasonable. And you are right that decoding is taking most of the time during inference. There are some ongoing projects for improving decoding. So stay tuned on that. For serving multiple users, you may try a batching mechanism that keeps requests in a queue and batches them between each inference. Have a look at batching in this code for reference.

[ryanshea10]

Great! Thanks for the help with this. I think I’ve managed to mostly resolve the issue so I will close it for now.