Cookie, mux.com and datadome issues

Hey, getting an Error recently any idea how to fix it?

2022-05-14 13:52:50.0560 DEBUG [PatreonDownloader.Implementation.PatreonPageCrawler] Page #4: https://www.patreon.com/api/posts?include=user%2Cattachments%2Ccampaign%2Cpoll.choices%2Cpoll.current_user_responses.user%2Cpoll.current_user_responses.choice%2Cpoll.current_user_responses.poll%2Caccess_rules.tier.null%2Cimages.null%2Caudio.null&fields%5Bpost%5D=change_visibility_at%2Ccomment_count%2Ccontent%2Ccurrent_user_can_delete%2Ccurrent_user_can_view%2Ccurrent_user_has_liked%2Cembed%2Cimage%2Cis_paid%2Clike_count%2Cmin_cents_pledged_to_view%2Cpost_file%2Cpost_metadata%2Cpublished_at%2Cpatron_count%2Cpatreon_url%2Cpost_type%2Cpledge_url%2Cthumbnail_url%2Cteaser_text%2Ctitle%2Cupgrade_url%2Curl%2Cwas_posted_by_campaign_owner&fields%5Buser%5D=image_url%2Cfull_name%2Curl&fields%5Bcampaign%5D=show_audio_post_download_links%2Cavatar_photo_url%2Cearnings_visibility%2Cis_nsfw%2Cis_monthly%2Cname%2Curl&fields%5Baccess_rule%5D=access_rule_type%2Camount_cents&fields%5Bmedia%5D=id%2Cimage_urls%2Cdownload_url%2Cmetadata%2Cfile_name&sort=-published_at&filter%5Bis_draft%5D=false&filter%5Bcontains_exclusive_posts%5D=true&json-api-use-default-includes=false&json-api-version=1.0&filter%5Bcampaign_id%5D=3133042&page%5Bcursor%5D=01SUSjbQm6uGXMGHMnHbaLxrQ_ 2022-05-14 13:52:50.3300 FATAL [PatreonDownloader.App.Program] Fatal error, application will be closed: UniversalDownloaderPlatform.Common.Exceptions.DownloadException: Error status code returned: BadRequest at UniversalDownloaderPlatform.DefaultImplementations.WebDownloader.DownloadStringInternal(String url, Int32 retry, Int32 retryTooManyRequests) in F:\Sources\BigProjects\PatreonDownloader\submodules\UniversalDownloaderPlatform\UniversalDownloaderPlatform.DefaultImplementations\WebDownloader.cs:line 323 at UniversalDownloaderPlatform.DefaultImplementations.WebDownloader.DownloadString(String url) in F:\Sources\BigProjects\PatreonDownloader\submodules\UniversalDownloaderPlatform\UniversalDownloaderPlatform.DefaultImplementations\WebDownloader.cs:line 288 at PatreonDownloader.Implementation.PatreonWebDownloader.DownloadString(String url) in F:\Sources\BigProjects\PatreonDownloader\PatreonDownloader.Implementation\PatreonWebDownloader.cs:line 55 at PatreonDownloader.Implementation.PatreonWebDownloader.DownloadString(String url) in F:\Sources\BigProjects\PatreonDownloader\PatreonDownloader.Implementation\PatreonWebDownloader.cs:line 73 at PatreonDownloader.Implementation.PatreonPageCrawler.Crawl(ICrawlTargetInfo crawlTargetInfo, String downloadDirectory) in F:\Sources\BigProjects\PatreonDownloader\PatreonDownloader.Implementation\PatreonPageCrawler.cs:line 84 at UniversalDownloaderPlatform.Engine.UniversalDownloader.Download(String url, String downloadDirectory, IUniversalDownloaderPlatformSettings settings) in F:\Sources\BigProjects\PatreonDownloader\submodules\UniversalDownloaderPlatform\UniversalDownloaderPlatform.Engine\UniversalDownloader.cs:line 198 at PatreonDownloader.App.Program.RunPatreonDownloader(CommandLineOptions commandLineOptions) in F:\Sources\BigProjects\PatreonDownloader\PatreonDownloader.App\Program.cs:line 143 at PatreonDownloader.App.Program.Main(String[] args) in F:\Sources\BigProjects\PatreonDownloader\PatreonDownloader.App\Program.cs:line 69

[urbanyeti]

Ah ha! I was able to figure this out and luckily there’s a quick workaround for your https://github.com/AlexCSDev/PatreonDownloader/tree/125_fix build.

The default CookieContainer has a PerDomainCapacity of 20 cookies and it seems like it quickly hits this after the first page or two. Not sure exactly why this would be the case, but my guess is there’s something going on where you are setting the cookies on line 73 of HttpCookieClient. It keeps getting back a datadome cookie but setting it with a different request url each time.

In any case, changing line 34 of HttpCookieClient to specify a larger PerDomainCapacity of 300 got me back up and running: _cookieContainer = new CookieContainer(CookieContainer.DefaultCookieLimit,300,CookieContainer.DefaultCookieLengthLimit);

[AlexCSDev]

Yep, if you count all tracking garbage there are 24 cookies set by patreon. Judging by the logic in microsoft’s code it doesn’t care if you update an existing cookie, it will keep cleaning up the collection until it reaches target size.

Great work, I will prepare a new test build tomorrow or the day after.