question-mark
Stuck on an issue?

Lightrun Answers was designed to reduce the constant googling that comes with debugging 3rd party libraries. It collects links to all the places you might be looking at while hunting down a tough bug.

And, if you’re still stuck at the end, we’re happy to hop on a call to see how we can help out.

Session state is not changed after failed request

See original GitHub issue

Which package is this bug report for? If unsure which one to select, leave blank

No response

Issue description

If a crawler request handler fails, the session state is not changed at all. That means it will never be replaced and might lead to infinite blocking.

I think in the initial SessionPool specs, we wanted to call session.markBad() when the request handler threw an error. At a minimum, it should increase session.usageCount. Right now, it only marks the session good if it succeeds but does not touch it on error, .e.g in this catch block

Example run https://console.apify.com/view/runs/PuagSQ0VRYFbRnDO1

Code sample

await Actor.init();

const proxyConfiguration = await Actor.createProxyConfiguration();

const crawler = new PuppeteerCrawler({
    proxyConfiguration,
    sessionPoolOptions: {
        maxPoolSize: 1,
    },
    requestHandler: async ({ page, request, session }) => {
        // The log prints incremented retryCount of request but session usageCount and errorScore stay at 0.
        log.info(`Request retryCount: ${request.retryCount}, sessionId: ${session.id}, `
            + `session usageCount: ${session.usageCount}, session errorScore: ${session.errorScore}`);
        throw 'Retry';
    },
});

await crawler.run([{"url":"https://example.com"}]);

// Exit successfully
await Actor.exit();

Package version

3.0.4

Node.js version

16.17

Operating system

No response

Apify platform

  • Tick me if you encountered this issue on the Apify platform

Priority this issue should have

Medium (should be fixed soon)

I have tested this on the next release

No response

Other context

No response

Issue Analytics

  • State:closed
  • Created a year ago
  • Comments:10 (10 by maintainers)

github_iconTop GitHub Comments

1reaction
metalwarrior665commented, Nov 1, 2022
  1. I would mark it bad on any error as it should by default do what it takes to get into working sessions. When it goes through without error, it marks itself as good which removes some of the error scores. I think this was the original idea when we discussed the specs with @petrpatek . At worst when you throw a lot but it is not tied to the session necessarily, you will rotate the sessions faster. No huge deal if we don’t do it but as I described as well in https://github.com/apify/crawlee/issues/1573, SessionPool is kinda our enfant terrible as the only way to really understand what it is doing is to dig into the source code so we need to document it really well.
  2. Not marking it as used on error is a pure bug I think
0reactions
B4nancommented, Dec 8, 2022

Yes please, preferably E2E test.

Read more comments on GitHub >

github_iconTop Results From Across the Web

Unable to make the session state request to ... - Stack Overflow
Start–> Administrative Tools –> Services; Right-click on the ASP.NET State Service and click “start”. Additionally you could set the service ...
Read more >
Unable to use session state server - ASP.NET - Microsoft Learn
This exception occurs the first time the session cache is used after the application is published to the server, or after IIS is...
Read more >
How to fix Session state is not available in this context IIS Error ...
Once we changed the setting to turn sessionstate off, the product page will load. Usually you would need to remove the configuration section...
Read more >
Ensuring Application Continuity - Oracle Help Center
After a successful replay, the application can continue where that database session left off, instead of having users left in doubt, not knowing...
Read more >
Why isn't my session state working in ASP.NET Core? Session ...
In this post I describe why session state appears to not work in ASP. ... NET Core application; Set a string in session...
Read more >

github_iconTop Related Medium Post

No results found

github_iconTop Related StackOverflow Question

No results found

github_iconTroubleshoot Live Code

Lightrun enables developers to add logs, metrics and snapshots to live code - no restarts or redeploys required.
Start Free

github_iconTop Related Reddit Thread

No results found

github_iconTop Related Hackernoon Post

No results found

github_iconTop Related Tweet

No results found

github_iconTop Related Dev.to Post

No results found

github_iconTop Related Hashnode Post

No results found