Cookie, mux.com and datadome issues
See original GitHub issueHey, getting an Error recently any idea how to fix it?
2022-05-14 13:52:50.0560 DEBUG [PatreonDownloader.Implementation.PatreonPageCrawler] Page #4: https://www.patreon.com/api/posts?include=user%2Cattachments%2Ccampaign%2Cpoll.choices%2Cpoll.current_user_responses.user%2Cpoll.current_user_responses.choice%2Cpoll.current_user_responses.poll%2Caccess_rules.tier.null%2Cimages.null%2Caudio.null&fields%5Bpost%5D=change_visibility_at%2Ccomment_count%2Ccontent%2Ccurrent_user_can_delete%2Ccurrent_user_can_view%2Ccurrent_user_has_liked%2Cembed%2Cimage%2Cis_paid%2Clike_count%2Cmin_cents_pledged_to_view%2Cpost_file%2Cpost_metadata%2Cpublished_at%2Cpatron_count%2Cpatreon_url%2Cpost_type%2Cpledge_url%2Cthumbnail_url%2Cteaser_text%2Ctitle%2Cupgrade_url%2Curl%2Cwas_posted_by_campaign_owner&fields%5Buser%5D=image_url%2Cfull_name%2Curl&fields%5Bcampaign%5D=show_audio_post_download_links%2Cavatar_photo_url%2Cearnings_visibility%2Cis_nsfw%2Cis_monthly%2Cname%2Curl&fields%5Baccess_rule%5D=access_rule_type%2Camount_cents&fields%5Bmedia%5D=id%2Cimage_urls%2Cdownload_url%2Cmetadata%2Cfile_name&sort=-published_at&filter%5Bis_draft%5D=false&filter%5Bcontains_exclusive_posts%5D=true&json-api-use-default-includes=false&json-api-version=1.0&filter%5Bcampaign_id%5D=3133042&page%5Bcursor%5D=01SUSjbQm6uGXMGHMnHbaLxrQ_ 2022-05-14 13:52:50.3300 FATAL [PatreonDownloader.App.Program] Fatal error, application will be closed: UniversalDownloaderPlatform.Common.Exceptions.DownloadException: Error status code returned: BadRequest at UniversalDownloaderPlatform.DefaultImplementations.WebDownloader.DownloadStringInternal(String url, Int32 retry, Int32 retryTooManyRequests) in F:\Sources\BigProjects\PatreonDownloader\submodules\UniversalDownloaderPlatform\UniversalDownloaderPlatform.DefaultImplementations\WebDownloader.cs:line 323 at UniversalDownloaderPlatform.DefaultImplementations.WebDownloader.DownloadString(String url) in F:\Sources\BigProjects\PatreonDownloader\submodules\UniversalDownloaderPlatform\UniversalDownloaderPlatform.DefaultImplementations\WebDownloader.cs:line 288 at PatreonDownloader.Implementation.PatreonWebDownloader.DownloadString(String url) in F:\Sources\BigProjects\PatreonDownloader\PatreonDownloader.Implementation\PatreonWebDownloader.cs:line 55 at PatreonDownloader.Implementation.PatreonWebDownloader.DownloadString(String url) in F:\Sources\BigProjects\PatreonDownloader\PatreonDownloader.Implementation\PatreonWebDownloader.cs:line 73 at PatreonDownloader.Implementation.PatreonPageCrawler.Crawl(ICrawlTargetInfo crawlTargetInfo, String downloadDirectory) in F:\Sources\BigProjects\PatreonDownloader\PatreonDownloader.Implementation\PatreonPageCrawler.cs:line 84 at UniversalDownloaderPlatform.Engine.UniversalDownloader.Download(String url, String downloadDirectory, IUniversalDownloaderPlatformSettings settings) in F:\Sources\BigProjects\PatreonDownloader\submodules\UniversalDownloaderPlatform\UniversalDownloaderPlatform.Engine\UniversalDownloader.cs:line 198 at PatreonDownloader.App.Program.RunPatreonDownloader(CommandLineOptions commandLineOptions) in F:\Sources\BigProjects\PatreonDownloader\PatreonDownloader.App\Program.cs:line 143 at PatreonDownloader.App.Program.Main(String[] args) in F:\Sources\BigProjects\PatreonDownloader\PatreonDownloader.App\Program.cs:line 69
Issue Analytics
- State:
- Created a year ago
- Reactions:5
- Comments:81 (29 by maintainers)
Ah ha! I was able to figure this out and luckily there’s a quick workaround for your https://github.com/AlexCSDev/PatreonDownloader/tree/125_fix build.
The default
CookieContainer
has aPerDomainCapacity
of 20 cookies and it seems like it quickly hits this after the first page or two. Not sure exactly why this would be the case, but my guess is there’s something going on where you are setting the cookies on line 73 ofHttpCookieClient
. It keeps getting back adatadome
cookie but setting it with a different request url each time.In any case, changing line 34 of
HttpCookieClient
to specify a largerPerDomainCapacity
of 300 got me back up and running:_cookieContainer = new CookieContainer(CookieContainer.DefaultCookieLimit,300,CookieContainer.DefaultCookieLengthLimit);
Yep, if you count all tracking garbage there are 24 cookies set by patreon. Judging by the logic in microsoft’s code it doesn’t care if you update an existing cookie, it will keep cleaning up the collection until it reaches target size.
Great work, I will prepare a new test build tomorrow or the day after.