Allow multiple items through pipelines?

The documentation for pipeline specifies that process_item must either return a dict with data, Item object or raise a DropItem exception. Is there a reason why we aren’t allowed to return an iterable of dictionaries with data (or Item objects)? It seems impossible to write a pipeline that modifies the input item and returns multiple items under the current framework.

Thank you!

[kmike]

1. A pipeline has to be able to return either a single object, or an iterable collection of it. It also has to accept both, otherwise it wouldn’t make sense.

For me it makes more sense for pipeline to accept a single item, but return either a single item or a list/iterable. Before:

item1 --> [pipeline A] --> item1 --> [pipeline B] --> ...
item2 --> [pipeline A] --x raise DropItem()

New feature:

item3 --> [pipeline A],--> item4 --> [pipeline B] --> ...
                      '--> item5 --> [pipeline B] --> ...

[dxue2012]

Hi all,

I ended up defining a custom post-processing step after ItemPipelines in the following manner:

1. Store all the items in MongoDB at the end of Pipelines
2. Read all items from MongoDB, and do more stuff on the data with “Processors”

Each processor (analogous to pipeline) defines a function called process_iter_items, which takes in an iterable of dicts, and must return an iterable of dicts. The set of processors is managed by BatchProcessorManager, a MiddlewareManager class similar to ItemPipelineManager, which supports the chaining of process_iter_items functions instead of process_item functions.

The chain of process_iter_items is connected to the signal emitted by the last ItemPipeline that stores the items in MongoDB.