Spider: Illinois Department of Corrections Advisory Board

URL: https://www2.illinois.gov/idoc/aboutus/advisoryboard/Pages/default.aspx Spider Name: il_corrections Agency Name: Illinois Department of Corrections Advisory Board

See the contribution guide for information on how to get started

Some information will need to be parsed from PDFs, so it could be useful to look at chi_human_relations for an example of how to handle this

[pjsier]

@cherdeman sure! Scrapy requests are made asynchronously, so if you want to guarantee you have data available for another method you’ll need to make sure you’re chaining callbacks so that they’re called in the right order.

You can see an example of this in chi_human_relations where the first request (in start_urls) goes to a department detail page. The parse method is the default callback, so the response from the detail page is checked there, and from there look for a PDF link to parse and yield a request to that link that will be handled by _parse_schedule. In _parse_schedule we handle the response body and then chain another request to _parse_documents now that we know the schedule details will be available and can be used to yield meetings.

From taking an initial look at your scraper, it looks like you could follow a similar pattern of first parsing all of the links, then yield a request to each meeting, pull the PDF, and then potentially yield back to the meeting (which is where this one might be a bit circular). Let me know if that’s helpful! The Scrapy docs are also generally good, so if there’s anything I can help with from those let me know

[pjsier]

@cherdeman great! I’ll assign you