Stuck on an issue?

Lightrun Answers was designed to reduce the constant googling that comes with debugging 3rd party libraries. It collects links to all the places you might be looking at while hunting down a tough bug.

And, if you’re still stuck at the end, we’re happy to hop on a call to see how we can help out.

imdb plugin failing with a 405 error code

See original GitHub issue

Expected behaviour:

Flexget is parsing the imdb.com and doesn’t throw an error while doing it.

Actual behaviour:

Steps to reproduce:

Run the attached sample of config
See the error from flexget saying the HTTP method not allowed

Config:

# Import sensitive data

variables: secrets.yml

####################
# Define templates #
####################

templates:

  slack-notification-movies:
    notify:
      entries:
        via:
          - slack:
              web_hook_url: '{? notify.slack ?}'
              message: "{{ movie_name }}"
              attachments:
                - title: "{{ movie_name }} ({{ movie_year }})"
                  image_url: "{{ imdb_photo }}" # better quality
                  title_link: "{{ imdb_url }}"
                  fallback: "{{ imdb_name }} ({{ imdb_year }})"
                  text: "{{ imdb_plot_outline }}"
                  color: "#e67e22"
                  footer: "{{ task }}"
                  footer_icon: "https://avatars2.githubusercontent.com/u/17483320?s=400&v=4"
                  fields:
                    - title: Score
                      value: "{{ imdb_score }}"
                      short: true
                    - title: Genres
                      value: "{{ imdb_genres | join(', ') | title }}"
                      short: true
                    - title: Resolution
                      value: "{{ quality.resolution }}"
                      short: true
                    - title: Filesize
                      value: "{{ content_size }} MB"
                      short: true


####################################
# Organizing the movies and series #
####################################

tasks:

  copy-movies:
    priority: 2
    template: [ slack-notification-movies ]
    metainfo_movie: yes
    imdb_lookup: yes
    #disable: seen
    seen: local
    accept_all: yes
    content_size:
      min: 1300
    filesystem:
      path: /transmission/downloads/complete/movies/
      regexp: '.*.(avi|mkv|mp4|m4v)$'
      recursive: yes
    copy:
      to: "/MoviesFromIMDBList/{{ movie_name|replace(' ','.')|replace(':','.')|replace('?','.') }}.{{ movie_year }}.{{ quality.resolution }}/"
      rename: "{{ movie_name|replace(' ','.')|replace(':','.')|replace('?','.') }}.{{ movie_year }}.{{ quality.resolution }}{{ location|pathext }}"

Log:

(click to expand)

2022-11-21 11:46:50 DEBUG    backlog       copy-movies     0 entries purged from backlog
2022-11-21 11:46:50 VERBOSE  details       copy-movies     Produced 1 entries.
2022-11-21 11:46:50 DEBUG    content_size  copy-movies     Long.Lost.Christmas.2022.1080p.WEBRip.x264-RARBG size 1643 MB
2022-11-21 11:46:50 VERBOSE  task          copy-movies     ACCEPTED: `Long.Lost.Christmas.2022.1080p.WEBRip.x264-RARBG` by accept_all plugin
2022-11-21 11:46:50 DEBUG    urlrewriter   copy-movies     Checking 1 entries
2022-11-21 11:46:50 VERBOSE  details       copy-movies     Summary - Accepted: 1 (Rejected: 0 Undecided: 0 Failed: 0)
2022-11-21 11:46:50 INFO     content_size  copy-movies     Plugin is partially disabled with --test and --learn because size information may not be available
2022-11-21 11:46:50 DEBUG    imdb_lookup   copy-movies     lookup for Long.Lost.Christmas.2022.1080p.WEBRip.x264-RARBG
2022-11-21 11:46:51 VERBOSE  imdb_lookup   copy-movies     Parsing imdb for `Long.Lost.Christmas.2022.1080p.WEBRip.x264-RARBG`
2022-11-21 11:46:51 DEBUG    utils.requests copy-movies     GETing URL https://www.imdb.com/title/tt22307560/ with args () and kwargs {'allow_redirects': True, 'timeout': 30}
2022-11-21 11:46:51 DEBUG    plugin        copy-movies     decorator caught RequestException. handled traceback:
Traceback (most recent call last):

  File "/usr/lib/python3.8/threading.py", line 890, in _bootstrap
    self._bootstrap_inner()
    │    └ <function Thread._bootstrap_inner at 0x7f4c5cf27ee0>
    └ <Thread(task_queue, started daemon 139965851674368)>
  File "/usr/lib/python3.8/threading.py", line 932, in _bootstrap_inner
    self.run()
    │    └ <function Thread.run at 0x7f4c5cf27c10>
    └ <Thread(task_queue, started daemon 139965851674368)>
  File "/usr/lib/python3.8/threading.py", line 870, in run
    self._target(*self._args, **self._kwargs)
    │    │        │    │        │    └ {}
    │    │        │    │        └ <Thread(task_queue, started daemon 139965851674368)>
    │    │        │    └ ()
    │    │        └ <Thread(task_queue, started daemon 139965851674368)>
    │    └ <bound method TaskQueue.run of <flexget.task_queue.TaskQueue object at 0x7f4c5716d160>>
    └ <Thread(task_queue, started daemon 139965851674368)>

  File "/usr/local/lib/python3.8/dist-packages/flexget/task_queue.py", line 47, in run
    self.current_task.execute()
    │    │            └ <function Task.execute at 0x7f4c59f66d30>
    │    └ <flexget.task.Task object at 0x7f4c56e02910>
    └ <flexget.task_queue.TaskQueue object at 0x7f4c5716d160>

  File "/usr/local/lib/python3.8/dist-packages/flexget/task.py", line 87, in wrapper
    return func(self, *args, **kw)
           │    │      │       └ {}
           │    │      └ ()
           │    └ <flexget.task.Task object at 0x7f4c56e02910>
           └ <function Task.execute at 0x7f4c59f66ca0>

  File "/usr/local/lib/python3.8/dist-packages/flexget/task.py", line 727, in execute
    self._execute()
    │    └ <function Task._execute at 0x7f4c59f66c10>
    └ <flexget.task.Task object at 0x7f4c56e02910>

  File "/usr/local/lib/python3.8/dist-packages/flexget/task.py", line 696, in _execute
    self.__run_task_phase(phase)
    │                     └ 'output'
    └ <flexget.task.Task object at 0x7f4c56e02910>

  File "/usr/local/lib/python3.8/dist-packages/flexget/task.py", line 514, in __run_task_phase
    response = self.__run_plugin(plugin, phase, args)
               │                 │       │      └ (<flexget.task.Task object at 0x7f4c56e02910>, {'to': "/home/vilius/Downloads/synonology.nas/Media/Filmai/MoviesFromIMDBList/...
               │                 │       └ 'output'
               │                 └ <PluginInfo(name=copy)>
               └ <flexget.task.Task object at 0x7f4c56e02910>

  File "/usr/local/lib/python3.8/dist-packages/flexget/task.py", line 547, in __run_plugin
    result = method(*args, **kwargs)
             │       │       └ {}
             │       └ (<flexget.task.Task object at 0x7f4c56e02910>, {'to': "/home/vilius/Downloads/synonology.nas/Media/Filmai/MoviesFromIMDBList/...
             └ <Event(name=plugin.copy.output,func=on_task_output,priority=128)>

  File "/usr/local/lib/python3.8/dist-packages/flexget/event.py", line 20, in __call__
    return self.func(*args, **kwargs)
           │    │     │       └ {}
           │    │     └ (<flexget.task.Task object at 0x7f4c56e02910>, {'to': "/home/vilius/Downloads/synonology.nas/Media/Filmai/MoviesFromIMDBList/...
           │    └ <bound method BaseFileOps.on_task_output of <flexget.plugins.output.file_operations.CopyFiles object at 0x7f4c57d56b20>>
           └ <Event(name=plugin.copy.output,func=on_task_output,priority=128)>

  File "/usr/local/lib/python3.8/dist-packages/flexget/plugins/output/file_operations.py", line 128, in on_task_output
    self.handle_entry(task, config, entry, siblings)
    │    │            │     │       │      └ {}
    │    │            │     │       └ <Entry(title=Long.Lost.Christmas.2022.1080p.WEBRip.x264-RARBG,state=accepted)>
    │    │            │     └ {'to': "/home/vilius/Downloads/synonology.nas/Media/Filmai/MoviesFromIMDBList/{{ movie_name|replace(' ','.')|replace(':','.')...
    │    │            └ <flexget.task.Task object at 0x7f4c56e02910>
    │    └ <function TransformingOps.handle_entry at 0x7f4c585f3ee0>
    └ <flexget.plugins.output.file_operations.CopyFiles object at 0x7f4c57d56b20>

  File "/usr/local/lib/python3.8/dist-packages/flexget/plugins/output/file_operations.py", line 239, in handle_entry
    dst_path = entry.render(dst_path)
               │     │      └ "/home/vilius/Downloads/synonology.nas/Media/Filmai/MoviesFromIMDBList/{{ movie_name|replace(' ','.')|replace(':','.')|replac...
               │     └ <function Entry.render at 0x7f4c5a338040>
               └ <Entry(title=Long.Lost.Christmas.2022.1080p.WEBRip.x264-RARBG,state=accepted)>

  File "/usr/local/lib/python3.8/dist-packages/flexget/entry.py", line 317, in render
    return render_from_entry(template, self, native=native)
           │                 │         │            └ False
           │                 │         └ <Entry(title=Long.Lost.Christmas.2022.1080p.WEBRip.x264-RARBG,state=accepted)>
           │                 └ "/home/vilius/Downloads/synonology.nas/Media/Filmai/MoviesFromIMDBList/{{ movie_name|replace(' ','.')|replace(':','.')|replac...
           └ <function render_from_entry at 0x7f4c5a387310>

  File "/usr/local/lib/python3.8/dist-packages/flexget/utils/template.py", line 330, in render_from_entry
    return render(template, variables, native=native)
           │      │         │                 └ False
           │      │         └ {'location': '/home/vilius/docker-compose/transmission/downloads/complete/movies/Long.Lost.Christmas.2022.1080p.WEBRip.x264-R...
           │      └ "/home/vilius/Downloads/synonology.nas/Media/Filmai/MoviesFromIMDBList/{{ movie_name|replace(' ','.')|replace(':','.')|replac...
           └ <function render at 0x7f4c5a387280>

  File "/usr/local/lib/python3.8/dist-packages/flexget/utils/template.py", line 306, in render
    result = template.render(context)
             │        │      └ {'location': '/home/vilius/docker-compose/transmission/downloads/complete/movies/Long.Lost.Christmas.2022.1080p.WEBRip.x264-R...
             │        └ <function Template.render at 0x7f4c5a395ca0>
             └ <FlexGetTemplate memory:7f4c55d40610>

  File "/usr/local/lib/python3.8/dist-packages/jinja2/environment.py", line 1299, in render
    return self.environment.concat(self.root_render_func(ctx))  # type: ignore
           │    │           │      │    │                └ <Context <flexget.utils.lazy_dict.LazyDict object at 0x7f4c55d5f430> of None>
           │    │           │      │    └ <function root at 0x7f4c55d3ee50>
           │    │           │      └ <FlexGetTemplate memory:7f4c55d40610>
           │    │           └ <built-in method join of str object at 0x7f4c5d22b2f0>
           │    └ <flexget.utils.template.FlexGetEnvironment object at 0x7f4c57080cd0>
           └ <FlexGetTemplate memory:7f4c55d40610>

  File "<template>", line 10, in root

  File "/usr/local/lib/python3.8/dist-packages/jinja2/runtime.py", line 240, in resolve_or_missing
    if key in self.parent:
       │      │    └ <flexget.utils.lazy_dict.LazyDict object at 0x7f4c55d5f430>
       │      └ <Context <flexget.utils.lazy_dict.LazyDict object at 0x7f4c55d5f430> of None>
       └ 'movie_name'

  File "/usr/lib/python3.8/_collections_abc.py", line 666, in __contains__
    self[key]
    │    └ 'movie_name'
    └ <flexget.utils.lazy_dict.LazyDict object at 0x7f4c55d5f430>

  File "/usr/local/lib/python3.8/dist-packages/flexget/utils/lazy_dict.py", line 77, in __getitem__
    return item[key]
           │    └ 'movie_name'
           └ <LazyLookup([LazyCallee(func=<bound method MetainfoQuality.get_quality of <flexget.plugins.metainfo.quality.MetainfoQuality o...

  File "/usr/local/lib/python3.8/dist-packages/flexget/utils/lazy_dict.py", line 41, in __getitem__
    callee.func(self.store, *(callee.args or []), **(callee.kwargs or {}))
    │      │    │    │        │      │               │      └ <_collections._tuplegetter object at 0x7f4c5a372f10>
    │      │    │    │        │      │               └ LazyCallee(func=<bound method ImdbLookup.lazy_loader of <flexget.components.imdb.imdb_lookup.ImdbLookup object at 0x7f4c571b9...
    │      │    │    │        │      └ <_collections._tuplegetter object at 0x7f4c5a372f40>
    │      │    │    │        └ LazyCallee(func=<bound method ImdbLookup.lazy_loader of <flexget.components.imdb.imdb_lookup.ImdbLookup object at 0x7f4c571b9...
    │      │    │    └ <Entry(title=Long.Lost.Christmas.2022.1080p.WEBRip.x264-RARBG,state=accepted)>
    │      │    └ <LazyLookup([LazyCallee(func=<bound method MetainfoQuality.get_quality of <flexget.plugins.metainfo.quality.MetainfoQuality o...
    │      └ <_collections._tuplegetter object at 0x7f4c5a372e80>
    └ LazyCallee(func=<bound method ImdbLookup.lazy_loader of <flexget.components.imdb.imdb_lookup.ImdbLookup object at 0x7f4c571b9...

  File "/usr/local/lib/python3.8/dist-packages/flexget/components/imdb/imdb_lookup.py", line 69, in lazy_loader
    self.lookup(entry)
    │    │      └ <Entry(title=Long.Lost.Christmas.2022.1080p.WEBRip.x264-RARBG,state=accepted)>
    │    └ <function internet.__call__.<locals>.wrapped_func at 0x7f4c57d6f790>
    └ <flexget.components.imdb.imdb_lookup.ImdbLookup object at 0x7f4c571b99d0>

> File "/usr/local/lib/python3.8/dist-packages/flexget/plugin.py", line 126, in wrapped_func
    return func(*args, **kwargs)
           │     │       └ {}
           │     └ (<flexget.components.imdb.imdb_lookup.ImdbLookup object at 0x7f4c571b99d0>, <Entry(title=Long.Lost.Christmas.2022.1080p.WEBRi...
           └ <function with_session.<locals>.decorator.<locals>.wrapper at 0x7f4c57d6f820>

  File "/usr/local/lib/python3.8/dist-packages/flexget/utils/database.py", line 30, in wrapper
    return func(*args, **kwargs)
           │     │       └ {'session': <sqlalchemy.orm.session.ContextSession object at 0x7f4c55d5f7f0>}
           │     └ (<flexget.components.imdb.imdb_lookup.ImdbLookup object at 0x7f4c571b99d0>, <Entry(title=Long.Lost.Christmas.2022.1080p.WEBRi...
           └ <function ImdbLookup.lookup at 0x7f4c57d6f670>

  File "/usr/local/lib/python3.8/dist-packages/flexget/components/imdb/imdb_lookup.py", line 222, in lookup
    movie = self._parse_new_movie(entry['imdb_url'], session)
            │    │                │                  └ <sqlalchemy.orm.session.ContextSession object at 0x7f4c55d5f7f0>
            │    │                └ <Entry(title=Long.Lost.Christmas.2022.1080p.WEBRip.x264-RARBG,state=accepted)>
            │    └ <function ImdbLookup._parse_new_movie at 0x7f4c57d6f8b0>
            └ <flexget.components.imdb.imdb_lookup.ImdbLookup object at 0x7f4c571b99d0>

  File "/usr/local/lib/python3.8/dist-packages/flexget/components/imdb/imdb_lookup.py", line 267, in _parse_new_movie
    parser.parse(imdb_url)
    │      │     └ 'https://www.imdb.com/title/tt22307560/'
    │      └ <function ImdbParser.parse at 0x7f4c586ec670>
    └ <flexget.components.imdb.utils.ImdbParser object at 0x7f4c55cc8760>

  File "/usr/local/lib/python3.8/dist-packages/flexget/components/imdb/utils.py", line 250, in parse
    page = requests.get(url)
           │        │   └ 'https://www.imdb.com/title/tt22307560/'
           │        └ <function Session.get at 0x7f4c5bc2e8b0>
           └ <flexget.utils.requests.Session object at 0x7f4c586e36d0>

  File "/usr/local/lib/python3.8/dist-packages/requests/sessions.py", line 600, in get
    return self.request("GET", url, **kwargs)
           │    │              │      └ {'allow_redirects': True}
           │    │              └ 'https://www.imdb.com/title/tt22307560/'
           │    └ <function Session.request at 0x7f4c59fbf820>
           └ <flexget.utils.requests.Session object at 0x7f4c586e36d0>

  File "/usr/local/lib/python3.8/dist-packages/flexget/utils/requests.py", line 271, in request
    result.raise_for_status()
    │      └ <function Response.raise_for_status at 0x7f4c5bc2c5e0>
    └ <Response [405]>

  File "/usr/local/lib/python3.8/dist-packages/requests/models.py", line 1021, in raise_for_status
    raise HTTPError(http_error_msg, response=self)
          │         │                        └ <Response [405]>
          │         └ '405 Client Error: Not Allowed for url: https://www.imdb.com/title/tt22307560/'
          └ <class 'requests.exceptions.HTTPError'>

requests.exceptions.HTTPError: 405 Client Error: Not Allowed for url: https://www.imdb.com/title/tt22307560/
2022-11-21 11:46:51 VERBOSE  imdb_lookup   copy-movies     Requestexception: 405 client error: not allowed for url: https://www.imdb.com/title/tt22307560/

Additional information:

FlexGet version: 3.5.5
Python version: 3.8.10
Installation method: pip
Using daemon (yes/no): no
OS and version: 20.04.2-Ubuntu

Issue Analytics

State:
Created 10 months ago
Reactions:2
Comments:12 (3 by maintainers)

Top GitHub Comments

5reactions

ProteinPigcommented, Nov 23, 2022

I got a 403 client error when use imdb_lookup here.

FlexGet version: 3.5.5 Installation method: docker/wiserain

  imdb_test:
    manual: yes
    mock:
      - {title: 'A.Beautiful.Day.in.the.Neighborhood.2019.BluRay.1080p.DTS-HD.MA.5.1.x265.10bit-BeiTai'}
    imdb_lookup: yes
    accept_all: yes
    exec: echo "{{imdb_name}} ({{imdb_year}})"

2022-11-23 18:38:59 INFO     manager                       Config change detected. Reloading.
2022-11-23 18:38:59 INFO     manager                       Config successfully reloaded!
2022-11-23 18:38:59 VERBOSE  details       imdb_test       Produced 1 entries.
2022-11-23 18:39:00 VERBOSE  task          imdb_test       ACCEPTED: `A.Beautiful.Day.in.the.Neighborhood.2019.BluRay.1080p.DTS-HD.MA.5.1.x265.10bit-BeiTai` by accept_all plugin
2022-11-23 18:39:00 VERBOSE  details       imdb_test       Summary - Accepted: 1 (Rejected: 0 Undecided: 0 Failed: 0)
2022-11-23 18:39:00 VERBOSE  imdb_lookup   imdb_test       Movie `A Beautiful Day in the Neighborhood` details expired, refreshing ...
2022-11-23 18:39:00 VERBOSE  imdb_lookup   imdb_test       Parsing imdb for `A.Beautiful.Day.in.the.Neighborhood.2019.BluRay.1080p.DTS-HD.MA.5.1.x265.10bit-BeiTai`
2022-11-23 18:39:00 INFO     imdb_lookup   imdb_test       Requestexception: 403 client error: forbidden for url: https://www.imdb.com/title/tt3224458/
2022-11-23 18:39:00 VERBOSE  exec          imdb_test       Executing: echo "None (None)"
2022-11-23 18:39:00 INFO     exec          imdb_test       Stdout: None (None)

2reactions

Larrikinaucommented, Nov 30, 2022

I’m still getting these errors. The source in flexget/components/imdb/utils.py is intentionally identifying itself as a bot:
# IMDb delivers a version of the page which is unparsable to unknown (and some known) user agents, such as requests'
# Spoof the old urllib user agent to keep results consistent
requests = Session()
requests.headers.update({'User-Agent': 'Python-urllib/2.6'})
Changing the user-agent in that last line to a standard browser agent fixes the problem for me.

@grr Would you mind posting the exact line you’ve entered to replace the User-Agent so I can see what to edit mine to whilst waiting for a permanent fix? Thanks!

FYI, I tried this and it didn’t work (note - do I have to recompile anything - I haven’t as I am assuming it runs this python script):

requests.headers.update({'User-Agent': 'Mozilla/5.0'})