__new__() got an unexpected keyword argument 'delay_ms' when testing inference request

The Environment

I tried using tensorboard, tensorboard plugin and tensorflow installations outside of the virtual env suggested in the README, they satisfy the versions specified but produce an error. tensorboard: ‘2.4.0’ tensorflow: ‘2.3.1’ tensorboard_plugin_profile: 2.4

The Error

I’m running the tutorial line by line here, but I cannot capture any profile data. When I try to, I get a notice in tensorboard that says

Failed to capture profile: [object Object]

And in my terminal window where I’m running tensorboard, this shows up when I request:

tensorboard --logdir ~/logs/inference_demo/ --port 6006
2021-01-12 15:13:15.239907: I tensorflow/stream_executor/platform/default/dso_loader.cc:48] Successfully opened dynamic library libcudart.so.10.1
Serving TensorBoard on localhost; to expose to the network, use a proxy or pass --bind_all
TensorBoard 2.4.0 at http://localhost:6006/ (Press CTRL+C to quit)
W0112 15:13:47.703725 140358682797824 security_validator.py:51] In 3.0, this warning will become an error:
Illegal Content-Security-Policy for script-src: 'unsafe-inline'
E0112 15:14:02.757672 140358682797824 _internal.py:113] Error on request:
Traceback (most recent call last):
  File "/home/rave/miniconda3/envs/data-science-stack-2.5.0/lib/python3.7/site-packages/werkzeug/serving.py", line 323, in run_wsgi
    execute(self.server.app)
  File "/home/rave/miniconda3/envs/data-science-stack-2.5.0/lib/python3.7/site-packages/werkzeug/serving.py", line 312, in execute
    application_iter = app(environ, start_response)
  File "/home/rave/miniconda3/envs/data-science-stack-2.5.0/lib/python3.7/site-packages/tensorboard/backend/application.py", line 526, in __call__
    return self._app(environ, start_response)
  File "/home/rave/miniconda3/envs/data-science-stack-2.5.0/lib/python3.7/site-packages/tensorboard/backend/application.py", line 567, in wrapper
    return wsgi_app(environ, start_response)
  File "/home/rave/miniconda3/envs/data-science-stack-2.5.0/lib/python3.7/site-packages/tensorboard/backend/security_validator.py", line 82, in __call__
    return self._application(environ, start_response_proxy)
  File "/home/rave/miniconda3/envs/data-science-stack-2.5.0/lib/python3.7/site-packages/tensorboard/backend/path_prefix.py", line 71, in __call__
    return self._application(environ, start_response)
  File "/home/rave/miniconda3/envs/data-science-stack-2.5.0/lib/python3.7/site-packages/tensorboard/backend/experiment_id.py", line 76, in __call__
    return self._application(environ, start_response)
  File "/home/rave/miniconda3/envs/data-science-stack-2.5.0/lib/python3.7/site-packages/tensorboard/backend/empty_path_redirect.py", line 47, in __call__
    return self._application(environ, start_response)
  File "/home/rave/miniconda3/envs/data-science-stack-2.5.0/lib/python3.7/site-packages/tensorboard/backend/application.py", line 590, in wrapper
    return wsgi_app(environ, start_response)
  File "/home/rave/miniconda3/envs/data-science-stack-2.5.0/lib/python3.7/site-packages/tensorboard/backend/application.py", line 549, in _route_request
    return self.exact_routes[clean_path](environ, start_response)
  File "/home/rave/miniconda3/envs/data-science-stack-2.5.0/lib/python3.7/site-packages/werkzeug/wrappers/base_request.py", line 238, in application
    resp = f(*args[:-2] + (request,))
  File "/home/rave/miniconda3/envs/data-science-stack-2.5.0/lib/python3.7/site-packages/tensorboard_plugin_profile/profile_plugin.py", line 526, in capture_route
    delay_ms=int(request.args.get('delay', '0'))
TypeError: __new__() got an unexpected keyword argument 'delay_ms'
E0112 15:14:55.211221 140358682797824 _internal.py:113] Error on request:
Traceback (most recent call last):
  File "/home/rave/miniconda3/envs/data-science-stack-2.5.0/lib/python3.7/site-packages/werkzeug/serving.py", line 323, in run_wsgi
    execute(self.server.app)
  File "/home/rave/miniconda3/envs/data-science-stack-2.5.0/lib/python3.7/site-packages/werkzeug/serving.py", line 312, in execute
    application_iter = app(environ, start_response)
  File "/home/rave/miniconda3/envs/data-science-stack-2.5.0/lib/python3.7/site-packages/tensorboard/backend/application.py", line 526, in __call__
    return self._app(environ, start_response)
  File "/home/rave/miniconda3/envs/data-science-stack-2.5.0/lib/python3.7/site-packages/tensorboard/backend/application.py", line 567, in wrapper
    return wsgi_app(environ, start_response)
  File "/home/rave/miniconda3/envs/data-science-stack-2.5.0/lib/python3.7/site-packages/tensorboard/backend/security_validator.py", line 82, in __call__
    return self._application(environ, start_response_proxy)
  File "/home/rave/miniconda3/envs/data-science-stack-2.5.0/lib/python3.7/site-packages/tensorboard/backend/path_prefix.py", line 71, in __call__
    return self._application(environ, start_response)
  File "/home/rave/miniconda3/envs/data-science-stack-2.5.0/lib/python3.7/site-packages/tensorboard/backend/experiment_id.py", line 76, in __call__
    return self._application(environ, start_response)
  File "/home/rave/miniconda3/envs/data-science-stack-2.5.0/lib/python3.7/site-packages/tensorboard/backend/empty_path_redirect.py", line 47, in __call__
    return self._application(environ, start_response)
  File "/home/rave/miniconda3/envs/data-science-stack-2.5.0/lib/python3.7/site-packages/tensorboard/backend/application.py", line 590, in wrapper
    return wsgi_app(environ, start_response)
  File "/home/rave/miniconda3/envs/data-science-stack-2.5.0/lib/python3.7/site-packages/tensorboard/backend/application.py", line 549, in _route_request
    return self.exact_routes[clean_path](environ, start_response)
  File "/home/rave/miniconda3/envs/data-science-stack-2.5.0/lib/python3.7/site-packages/werkzeug/wrappers/base_request.py", line 238, in application
    resp = f(*args[:-2] + (request,))
  File "/home/rave/miniconda3/envs/data-science-stack-2.5.0/lib/python3.7/site-packages/tensorboard_plugin_profile/profile_plugin.py", line 526, in capture_route
    delay_ms=int(request.args.get('delay', '0'))
TypeError: __new__() got an unexpected keyword argument 'delay_ms'

What I’ve tried

From the instructions:

Press Capture on the TensorBoard UI and send an inference request to TF Serving quickly thereafter.

I’ve tried to send the post request to the example tfserving container “quickly” following the instructions, but no matter how quickly I do so, it seems like the unexpected keyword is the problem.

It seems like there is a version conflict between tensorflow 's profiler and the plugin, so perhaps the suggested prerequisites can be updated.

[yisitu]

This is because ‘delay_ms’ is a new parameter that we added in TensorFlow 2.4 and set in the profiler plugin. A fix is on the way. Once it lands you can use a nightly release of TensorBoard profiler plugin[1]. Otherwise, you can also upgrade to TensorFlow 2.4.

1. https://pypi.org/project/tbp-nightly/#history

[yisitu]

That output looks good. That script defaults to nightly, essentially the same as my previous comment. Since the issue does not show up any more, I’ll close this bug.