Order of calling close_spider in pipelines

process_item is called based on the order of the pipeline classes mentioned in ITEM_PIPELINES setting. But, close_spider follows the exact opposite order with the close_spider of the last pipeline getting called first [link].

I can’t think of a good reason for this. Should I submit a PR?

[debosmit]

I made a bad choice in wording. For my use-case, *pipeline A* post items to a ES in batches, hence it needs to hold the items till it has enough of them to make a batch request. Once, all the items have been batch-posted by *pipeline A*, *pipeline B* does some stuff based on the ES indices affected by *pipeline A*'s actions.

I totally agree with you about the backward compatibility point. I was thinking of some kind of setting that would allow the user to choose either having close_spider called in reverse order (as it is now), or else, have it called in the same order as that of the pipelines – a simple boolean setting would suffice for this. I would love to hear your thoughts and ideas about this.

[debosmit]

@kmike Sure, I totally see the benefits in the open_spider and process_item cases. I am just not sure what benefit we get from calling B.close_spider before A.close_spider. Isn’t it more intuitive for all the functions to be serial? For example, a call order like,

1. A.open_spider
2. B.open_spider
3. A.process_item
4. B.process_item
5. A.close_spider
6. B.close_spider

I stumbled upon this in a usage where, my *pipeline A* posts to ES and *pipeline B* then works on the indices touched by *pipeline A*. Say, *pipeline A* stores a buffer of items, that is completely flushed on close spider, then if *pipeline B*'s close_spider is called before *pipeline A*'s close_spider, the datastore has not yet been updated with all the data and it appears like the overall spider pipeline is behaving like a bi-directional valve. Given that an ordering for pipelines exists, I think there should be a way to support both orderings at the least.

Can you mention a use-case where you would have to have pipelines structured such that B would require being closed before A?