Export `iterator` optimizations bypass `prefetch_related` optimizations

When using .iterator() prefetch_related is not taken in consideration (this is a django side effect of iterator()).

This is a sample of export method of Resource class:

        if isinstance(queryset, QuerySet):
            # Iterate without the queryset cache, to avoid wasting memory when
            # exporting large datasets.
            iterable = queryset.iterator()
        else:
            iterable = queryset
        for obj in iterable:
            data.append(self.export_resource(obj))

When queryset is an instance of QuerySet prefetch_related defined in get_queryset() override will be ignored.

Example (source of the code example):

class Book(models.Model):
    tags = TaggableManager()

class BookResource(resources.ModelResource):
    tags = fields.Field()

    class Meta:
        model = Book

    def dehydrate_tags(self, book):
        return ','.join([tag.name for tag in book.tags.all()])

    def get_queryset(self, queryset=None):
        if queryset is None:
            queryset = Book.objects.all()
        return queryset.prefetch_related('tags')

Since it is a QuerySet and the code of export will use iterator() it will result with 1 SQL query to fetch the books and 1 SQL query per row to fetch the tags.

The workaround I found was to list the queryset which will bypass the export iterator() logic:

    def export(self, queryset=None):
        queryset = self.get_queryset(queryset)
        fetched_queryset = list(queryset)
        return super().export(fetched_queryset)

With this fix I have now 2 SQL queries only

It works but it is not a clean solution.

Suggested fix:

• In export validate if the queryset instance has prefetch_related and not use iterator in this situation
• Or add some sort of configuration and documentation to avoid this kind of pointless and complicated override logic.

[kingbuzzman]

@jrobichaud I have a working solution:

def page_queryset(queryset, per_page=2000):
    """
    While in a perfect world you would use queryset.iterator(), but there is an issue that it fails to
    load any prefetch_related() fields specified. Using Paginator() we can mimic the same functionality

    > Note that if you use iterator() to run the query, prefetch_related() calls will be ignored since these two
    > optimizations do not make sense together.

    https://docs.djangoproject.com/en/2.0/ref/models/querysets/
    """
    if queryset._prefetch_related_lookups:
        if not queryset.query.order_by:
            # Paginator() throws a warning if there is no sorting attached to the queryset
            queryset = queryset.order_by('pk')
        paginator = Paginator(queryset, per_page)
        for index in range(paginator.num_pages):
            yield from paginator.get_page(index + 1)
    else:
        yield from queryset.iterator(chunk_size=per_page)

PS. I should state, I do not use django-import-export – I was just googling “django prefetch_related iterator” and this issue came up.

[Blejwi]

Same issue here, it would be nice to have this option configurable, because in some cases exporting breaks sensible timeouts for HTTP requests.