Lightrun
question-mark
Stuck on an issue?

Lightrun Answers was designed to reduce the constant googling that comes with debugging 3rd party libraries. It collects links to all the places you might be looking at while hunting down a tough bug.

And, if you’re still stuck at the end, we’re happy to hop on a call to see how we can help out.

iptorrents searches return '429 Client Error: Too Many Requests for url' with long lists

See original GitHub issue

Description

When searching with the iptorrents plugin, long lists of movies/series result in 429 Client Error: Too Many Requests for url error responses from IPTorrents. GET requests should be rate-limited for this plugin.

Expected Behavior

Searching with long lists of movies/series will return any discovered entries (or possibly zero entries) without any failed HTTP requests.

Actual Behavior

Searching with long lists of movies/series can result in 429 Client Error: Too Many Requests for url responses after iterating over a portion of the list.

Example with series:

2022-12-02 22:09:54 VERBOSE  discover      SEARCH_TV       Discovering 24 titles ...
2022-12-02 22:09:54 INFO     discover      SEARCH_TV       Ignoring interval because of --discover-now
2022-12-02 22:09:54 VERBOSE  discover      SEARCH_TV       Searching for `Yellowstone (2018) S04E11` with plugin `iptorrents` (1 of 24)
2022-12-02 22:09:56 VERBOSE  discover      SEARCH_TV       No search results for `Yellowstone (2018) S04E11`
2022-12-02 22:09:56 INFO     task          SEARCH_TV       Plugin next_series_episodes has requested task to be ran again after execution has completed. Reason: Look for next season
2022-12-02 22:09:56 VERBOSE  discover      SEARCH_TV       Searching for `What We Do in the Shadows S04E11` with plugin `iptorrents` (2 of 24)
2022-12-02 22:09:57 VERBOSE  discover      SEARCH_TV       No search results for `What We Do in the Shadows S04E11`
2022-12-02 22:09:57 VERBOSE  discover      SEARCH_TV       Searching for `The Righteous Gemstones S02E10` with plugin `iptorrents` (3 of 24)
2022-12-02 22:09:59 VERBOSE  discover      SEARCH_TV       No search results for `The Righteous Gemstones S02E10`
2022-12-02 22:09:59 VERBOSE  discover      SEARCH_TV       Searching for `The Morning Show S02E11` with plugin `iptorrents` (4 of 24)
2022-12-02 22:10:00 VERBOSE  discover      SEARCH_TV       No search results for `The Morning Show S02E11`
2022-12-02 22:10:00 VERBOSE  discover      SEARCH_TV       Searching for `The Boulet Brothers DRAGULA S04E11` with plugin `iptorrents` (5 of 24)
2022-12-02 22:10:01 VERBOSE  discover      SEARCH_TV       No search results for `The Boulet Brothers DRAGULA S04E11`
2022-12-02 22:10:01 VERBOSE  discover      SEARCH_TV       Searching for `The Bear S01E09` with plugin `iptorrents` (6 of 24)
2022-12-02 22:10:03 VERBOSE  discover      SEARCH_TV       Searching for `Ted Lasso S02E13` with plugin `iptorrents` (7 of 24)
2022-12-02 22:10:04 VERBOSE  discover      SEARCH_TV       No search results for `Ted Lasso S02E13`
2022-12-02 22:10:04 VERBOSE  discover      SEARCH_TV       Searching for `Severance S01E10` with plugin `iptorrents` (8 of 24)
2022-12-02 22:10:05 VERBOSE  discover      SEARCH_TV       No search results for `Severance S01E10`
2022-12-02 22:10:05 VERBOSE  discover      SEARCH_TV       Searching for `RuPauls Secret Celebrity Drag Race S02E09` with plugin `iptorrents` (9 of 24)
2022-12-02 22:10:07 VERBOSE  discover      SEARCH_TV       No search results for `RuPauls Secret Celebrity Drag Race S02E09`
2022-12-02 22:10:07 VERBOSE  discover      SEARCH_TV       Searching for `RuPauls Drag Race: Vegas Revue S01E05` with plugin `iptorrents` (10 of 24)
2022-12-02 22:10:08 VERBOSE  discover      SEARCH_TV       No search results for `RuPauls Drag Race: Vegas Revue S01E05`
2022-12-02 22:10:08 VERBOSE  discover      SEARCH_TV       Searching for `RuPauls Drag Race: Untucked! S14E15` with plugin `iptorrents` (11 of 24)
2022-12-02 22:10:09 VERBOSE  discover      SEARCH_TV       No search results for `RuPauls Drag Race: Untucked! S14E15`
2022-12-02 22:10:09 VERBOSE  discover      SEARCH_TV       Searching for `RuPauls Drag Race UK vs the World S01E07` with plugin `iptorrents` (12 of 24)
2022-12-02 22:10:10 ERROR    discover      SEARCH_TV       Error searching with iptorrents: RequestException: 429 Client Error: Too Many Requests for url: https://iptorrents.com/t?q=tt16968320+S01E07&qf=
2022-12-02 22:10:10 VERBOSE  discover      SEARCH_TV       No search results for `RuPauls Drag Race UK vs the World S01E07`
2022-12-02 22:10:10 VERBOSE  discover      SEARCH_TV       Searching for `RuPauls Drag Race UK S03E11` with plugin `iptorrents` (13 of 24)
2022-12-02 22:10:10 ERROR    discover      SEARCH_TV       Error searching with iptorrents: RequestException: 429 Client Error: Too Many Requests for url: https://iptorrents.com/t?q=RuPauls+Drag+Race+UK+S03E11&qf=
2022-12-02 22:10:10 VERBOSE  discover      SEARCH_TV       No search results for `RuPauls Drag Race UK S03E11`
2022-12-02 22:10:10 VERBOSE  discover      SEARCH_TV       Searching for `RuPauls Drag Race Down Under S01E09` with plugin `iptorrents` (14 of 24)
2022-12-02 22:10:10 ERROR    discover      SEARCH_TV       Error searching with iptorrents: RequestException: 429 Client Error: Too Many Requests for url: https://iptorrents.com/t?q=RuPauls+Drag+Race+Down+Under+S01E09&qf=
2022-12-02 22:10:10 VERBOSE  discover      SEARCH_TV       No search results for `RuPauls Drag Race Down Under S01E09`
2022-12-02 22:10:10 VERBOSE  discover      SEARCH_TV       Searching for `RuPauls Drag Race All Stars Untucked S07E13` with plugin `iptorrents` (15 of 24)
2022-12-02 22:10:10 ERROR    discover      SEARCH_TV       Error searching with iptorrents: RequestException: 429 Client Error: Too Many Requests for url: https://iptorrents.com/t?q=RuPauls+Drag+Race+All+Stars+Untucked+S07E13&qf=
2022-12-02 22:10:10 VERBOSE  discover      SEARCH_TV       No search results for `RuPauls Drag Race All Stars Untucked S07E13`
2022-12-02 22:10:10 VERBOSE  discover      SEARCH_TV       Searching for `RuPauls Drag Race All Stars S07E13` with plugin `iptorrents` (16 of 24)
2022-12-02 22:10:10 ERROR    discover      SEARCH_TV       Error searching with iptorrents: RequestException: 429 Client Error: Too Many Requests for url: https://iptorrents.com/t?q=RuPauls+Drag+Race+All+Stars+S07E13&qf=
2022-12-02 22:10:10 VERBOSE  discover      SEARCH_TV       No search results for `RuPauls Drag Race All Stars S07E13`
2022-12-02 22:10:10 VERBOSE  discover      SEARCH_TV       Searching for `RuPauls Drag Race S14E17` with plugin `iptorrents` (17 of 24)
2022-12-02 22:10:10 ERROR    discover      SEARCH_TV       Error searching with iptorrents: RequestException: 429 Client Error: Too Many Requests for url: https://iptorrents.com/t?q=RuPauls+Drag+Race+S14E17&qf=
2022-12-02 22:10:10 VERBOSE  discover      SEARCH_TV       No search results for `RuPauls Drag Race S14E17`
2022-12-02 22:10:10 VERBOSE  discover      SEARCH_TV       Searching for `Rick and Morty S06E05` with plugin `iptorrents` (18 of 24)
2022-12-02 22:10:10 ERROR    discover      SEARCH_TV       Error searching with iptorrents: RequestException: 429 Client Error: Too Many Requests for url: https://iptorrents.com/t?q=Rick+and+Morty+S06E05&qf=
2022-12-02 22:10:10 VERBOSE  discover      SEARCH_TV       No search results for `Rick and Morty S06E05`
2022-12-02 22:10:10 VERBOSE  discover      SEARCH_TV       Searching for `Peacemaker S01E09` with plugin `iptorrents` (19 of 24)
2022-12-02 22:10:11 ERROR    discover      SEARCH_TV       Error searching with iptorrents: RequestException: 429 Client Error: Too Many Requests for url: https://iptorrents.com/t?q=Peacemaker+S01E09&qf=
2022-12-02 22:10:11 VERBOSE  discover      SEARCH_TV       No search results for `Peacemaker S01E09`
2022-12-02 22:10:11 VERBOSE  discover      SEARCH_TV       Searching for `Drag Race Holland S02E09` with plugin `iptorrents` (20 of 24)
2022-12-02 22:10:11 ERROR    discover      SEARCH_TV       Error searching with iptorrents: RequestException: 429 Client Error: Too Many Requests for url: https://iptorrents.com/t?q=Drag+Race+Holland+S02E09&qf=
2022-12-02 22:10:11 VERBOSE  discover      SEARCH_TV       No search results for `Drag Race Holland S02E09`
2022-12-02 22:10:11 VERBOSE  discover      SEARCH_TV       Searching for `Clarice S01E14` with plugin `iptorrents` (21 of 24)
2022-12-02 22:10:11 ERROR    discover      SEARCH_TV       Error searching with iptorrents: RequestException: 429 Client Error: Too Many Requests for url: https://iptorrents.com/t?q=Clarice+S01E14&qf=
2022-12-02 22:10:11 VERBOSE  discover      SEARCH_TV       No search results for `Clarice S01E14`
2022-12-02 22:10:11 VERBOSE  discover      SEARCH_TV       Searching for `Canadas Drag Race S02E11` with plugin `iptorrents` (22 of 24)
2022-12-02 22:10:11 ERROR    discover      SEARCH_TV       Error searching with iptorrents: RequestException: 429 Client Error: Too Many Requests for url: https://iptorrents.com/t?q=Canadas+Drag+Race+S02E11&qf=
2022-12-02 22:10:11 VERBOSE  discover      SEARCH_TV       No search results for `Canadas Drag Race S02E11`
2022-12-02 22:10:11 VERBOSE  discover      SEARCH_TV       Searching for `Better Call Saul S06E14` with plugin `iptorrents` (23 of 24)
2022-12-02 22:10:11 ERROR    discover      SEARCH_TV       Error searching with iptorrents: RequestException: 429 Client Error: Too Many Requests for url: https://iptorrents.com/t?q=Better+Call+Saul+S06E14&qf=
2022-12-02 22:10:11 VERBOSE  discover      SEARCH_TV       No search results for `Better Call Saul S06E14`
2022-12-02 22:10:11 VERBOSE  discover      SEARCH_TV       Searching for `Alone S09E12` with plugin `iptorrents` (24 of 24)
2022-12-02 22:10:11 ERROR    discover      SEARCH_TV       Error searching with iptorrents: RequestException: 429 Client Error: Too Many Requests for url: https://iptorrents.com/t?q=Alone+S09E12&qf=
2022-12-02 22:10:11 VERBOSE  discover      SEARCH_TV       No search results for `Alone S09E12`
2022-12-02 22:10:11 VERBOSE  details       SEARCH_TV       Produced 2 entries.
*****
2022-12-02 22:10:11 VERBOSE  details       SEARCH_TV       Summary - Accepted: 0 (Rejected: 2 Undecided: 0 Failed: 0)
2022-12-02 22:10:11 INFO     task          SEARCH_TV       Rerunning the task in case better resolution can be achieved.
2022-12-02 22:10:11 VERBOSE  discover      SEARCH_TV       Discovering 24 titles ...
2022-12-02 22:10:11 INFO     discover      SEARCH_TV       Ignoring interval because of --discover-now
2022-12-02 22:10:11 VERBOSE  details       SEARCH_TV       Task didn't produce any entries.
2022-12-02 22:10:11 VERBOSE  details       SEARCH_TV       Summary - Accepted: 0 (Rejected: 0 Undecided: 0 Failed: 0)

FlexGet Version

$ flexget -V
3.5.6
You are on the latest release.

Relevant Config

tasks:
  SEARCH_TV:
    template: tv
    priority: 2
    free_space:
      path: /var/lib/deluge/Downloads
      space: 20000
    limit_new: 5
    discover:
      what:
        - next_series_episodes: yes
      from:
        - iptorrents:
            rss_key: *****
            uid: *****
            password: *****
      release_estimations: ignore

templates:
  tv:
    deluge:
      host: *****
      port: *****
      username: *****
      password: *****
      path: /var/lib/deluge/Downloads/TV
      config_path: /var/lib/deluge/.config/deluge
      action: add
      label: "tv"
    series:
      - "Better Call Saul":
          alternate_name:
            - tt3032476
          exact: yes
          quality: 1080p webdl+
      - "Clarice":
          alternate_name:
            - tt2177268
          exact: yes
          quality: 1080p webdl+
      - "Yellowstone (2018)":
          alternate_name:
            - tt4236770
          exact: yes
          quality: webrip+
      - "Ted Lasso":
          alternate_name:
            - tt10986410
          exact: yes
          quality: 1080p webdl+
      - "What We Do in the Shadows":
          alternate_name:
            - tt7908628
          exact: yes
          quality: 1080p webdl+
      - "Peacemaker":
          alternate_name:
            - tt13146488
          exact: yes
          quality: 1080p webdl+
      - "The Righteous Gemstones":
          alternate_name:
            - tt8634332
          exact: yes
          quality: 1080p webdl+
      - "The Bear":
          alternate_name:
            - tt14452776
          exact: yes
          quality: 1080p webdl+
      - "Alone":
          alternate_name:
            - tt4803766
          exact: yes
          quality: 720p webdl+
      - "The Morning Show":
          alternate_name:
            - tt7203552
          exact: yes
          quality: 1080p webdl+
      - "Severance":
          alternate_name:
            - tt11280740
          exact: yes
          quality: 1080p webdl+
      - "Rick and Morty":
          alternate_name:
            - tt2861424
          exact: yes
          quality: 1080p webdl+
      - "RuPauls Drag Race":
          alternate_name:
            - tt1353056
          exact: yes
          quality: webrip+
      - "RuPauls Drag Race: Untucked!":
          alternate_name:
            - tt1588754
          exact: yes
          quality: webrip+
      - "RuPauls Drag Race All Stars":
          alternate_name:
            - tt2301351
          exact: yes
          quality: webrip+
      - "RuPauls Drag Race All Stars Untucked":
          exact: yes
          quality: webrip+
      - "Canadas Drag Race":
          alternate_name:
            - tt11382554
          exact: yes
          quality: webrip+
      - "RuPauls Drag Race: Vegas Revue":
          alternate_name:
            - tt12937714
          exact: yes
          quality: webrip+
      - "RuPauls Secret Celebrity Drag Race":
          alternate_name:
            - tt11187480
          exact: yes
          quality: webrip+
      - "RuPauls Drag Race UK":
          alternate_name:
            - tt9780442
          exact: yes
          quality: webrip+
      - "Drag Race Holland":
          alternate_name:
            - tt13033136
          exact: yes
          quality: webrip+
      - "RuPauls Drag Race Down Under":
          alternate_name:
            - tt14192040
          exact: yes
          quality: webrip+
      - "The Boulet Brothers DRAGULA":
          alternate_name:
            - tt6289132
          exact: yes
          quality: webrip+
      - "RuPauls Drag Race UK vs the World":
          alternate_name:
            - tt16968320
          exact: yes
          quality: webrip+
    thetvdb_lookup: yes
    regexp:
      from: title
      reject:
        - msd
        - afg
        - line
    content_size:
      min: 0
      max: 10000
      strict: yes
    magnets: no
    torrent_alive: 5

Notes

I think this is triggered in this situation due to the alternate_name setting on many series in this list. Trying one name after another in quick succession could be the issue.

I believe this could be solved with a TimedLimiter.

We would want to use a flexget.utils.requests.Session that is updated via add_domain_limiter with a TimedLimiter instead of the default flexget.utils.requests.get method (also used here).

I think the interval for the TimedLimiter could be 1 second based on the logging that I’ve shared.

We could also use a session without a TimedLimiter to start, and if a 429 response is received, modify the session to begin limiting the requests from that point forward.

Issue Analytics

  • State:closed
  • Created 10 months ago
  • Comments:7 (7 by maintainers)

github_iconTop GitHub Comments

1reaction
bobcatthebakercommented, Dec 3, 2022

In the meantime, the domain_delay plugin might be an acceptable workaround

I just tried this workaround, and unfortunately the iptorrents plugin doesn’t use the session that’s built into the task passed to search. It references the default flexget.utils.requests.get method. For clarity, I can see in the logs that the delay is set:

2022-12-02 22:49:39 DEBUG    domain_delay  SEARCH_TV       Adding minimum interval of 5 seconds between requests to iptorrents.com

However, it isn’t respected due to the described reason:

...
2022-12-02 22:58:17 VERBOSE  discover      SEARCH_TV       Searching for `RuPauls Drag Race: Untucked! S14E15` with plugin `iptorrents` (11 of 24)
2022-12-02 22:58:18 ERROR    discover      SEARCH_TV       Error searching with iptorrents: RequestException: 429 Client Error: Too Many Requests for url: https://iptorrents.com/t?q=tt1588754+14x15&qf=
2022-12-02 22:58:18 VERBOSE  discover      SEARCH_TV       No search results for `RuPauls Drag Race: Untucked! S14E15`
2022-12-02 22:58:18 VERBOSE  discover      SEARCH_TV       Searching for `RuPauls Drag Race UK vs the World S01E07` with plugin `iptorrents` (12 of 24)
2022-12-02 22:58:18 ERROR    discover      SEARCH_TV       Error searching with iptorrents: RequestException: 429 Client Error: Too Many Requests for url: https://iptorrents.com/t?q=RuPauls+Drag+Race+UK+vs+the+World+S01E07&qf=
2022-12-02 22:58:18 VERBOSE  discover      SEARCH_TV       No search results for `RuPauls Drag Race UK vs the World S01E07`
2022-12-02 22:58:18 VERBOSE  discover      SEARCH_TV       Searching for `RuPauls Drag Race UK S03E11` with plugin `iptorrents` (13 of 24)
2022-12-02 22:58:19 ERROR    discover      SEARCH_TV       Error searching with iptorrents: RequestException: 429 Client Error: Too Many Requests for url: https://iptorrents.com/t?q=RuPauls+Drag+Race+UK+S03E11&qf=
2022-12-02 22:58:19 VERBOSE  discover      SEARCH_TV       No search results for `RuPauls Drag Race UK S03E11`
2022-12-02 22:58:19 VERBOSE  discover      SEARCH_TV       Searching for `RuPauls Drag Race Down Under S01E09` with plugin `iptorrents` (14 of 24)
2022-12-02 22:58:19 ERROR    discover      SEARCH_TV       Error searching with iptorrents: RequestException: 429 Client Error: Too Many Requests for url: https://iptorrents.com/t?q=RuPauls+Drag+Race+Down+Under+S01E09&qf=
2022-12-02 22:58:19 VERBOSE  discover      SEARCH_TV       No search results for `RuPauls Drag Race Down Under S01E09`
...
1reaction
gazpachokingcommented, Dec 3, 2022

A PR would be great to build it straight into the plugin. We already have the utility classes to add rate limits.

Read more comments on GitHub >

github_iconTop Results From Across the Web

iptorrents searches return '429 Client Error: Too Many ... - GitHub
Searching with long lists of movies/series can result in 429 Client Error: Too Many Requests for url responses after iterating over a portion...
Read more >
How to Fix 429 Too Many Requests Error - Kinsta
The HTTP 429 error is returned when too many requests are made to a page within a short period of time. Find out...
Read more >
429 Client Error: Too Many Requests for url - Stack Overflow
The HTTP 429 Too Many Requests response status code indicates the user has sent too many requests in a given amount of time...
Read more >
Nyaa error 429 too many requests : r/sonarr - Reddit
I cant search torrents through nyaa anymore saying i made too many requests, how long does it take to come back and is...
Read more >
Untitled
Long island picture frames, Hook pull off stage, Modulos de atencion sat, ... Pontop pike transmitter maintenance, Blue link customer service, ...
Read more >

github_iconTop Related Medium Post

No results found

github_iconTop Related StackOverflow Question

No results found

github_iconTroubleshoot Live Code

Lightrun enables developers to add logs, metrics and snapshots to live code - no restarts or redeploys required.
Start Free

github_iconTop Related Reddit Thread

No results found

github_iconTop Related Hackernoon Post

No results found

github_iconTop Related Tweet

No results found

github_iconTop Related Dev.to Post

No results found

github_iconTop Related Hashnode Post

No results found
Previous page

Trakt throwing errors

Next page

imdb plugin failing with a 405 error code