Performance: slow export with ForeignKeys id

I tried to investigate slow export on my model. I found out, that if I write dehydrate function on ModelResource like this:

    def dehydrate_user_profile(self, search_report):
        return search_report.user_profile_id

The export rapidly speeds up. I think, that django-import-export has bad method of getting the id values for ForeignKeys.

The bad performance is similar to this (notice the . instead of _)

    def dehydrate_user_profile(self, search_report):
        return search_report.user_profile.id

[PetrDlouhy]

@andrewgy8 In my case the increase was extreme - when I try with just 1300 records it is almost instant vs. 20 seconds.

I investigated a little bit further. The problem is, that on line 90 in fields.py in get_value() function the value of foreign key is fetched by:

value = getattr(value, attr, None)

If the value is not already cached or not fetch by the queryset with select_related, it will trigger new DB query for every row of the export. That value is further processed by Widget.render() where the value of certain key is fetched from the related object. This is unnecessary, if the key we want is primary key of the ForeignKey relation.

I am not sure, how to fix this properly. One method might be to detect, that we want to fetch pk/id key and bypass the get_value()->Widget.render() mechanism. Downside is that it might change behaviour if one of these function was overridden by user.

Second method is to automatically include select_related to the queryset. Upside is that it can greatly improve performance not only for primary keys, but also for related fields. Downside is that it might not work for some user modified querysets properly. Also in some cases it can lead to slowdown, because the select_related would increase the query complexity and/or fetch unwanted data (which might be solved by using only).

[matthewhegarty]

@mikaraunio Thanks for this update. I feel that updating the ‘Getting started’ docs would be a great way to move this forward. If you are able to submit a PR for this documentation update that would be appreciated.