select_sublcasses does not work with prefetch_related

Consider having the following example models:

from django.db import models
from model_utils.managers import InheritanceManager

class Post(models.Model):
    creation_date = models.DateTimeField(auto_now_add=True)
    objects = InheritanceManager()

class TextPost(Post):
    body = models.TextField()

class Photo(models.Model):
    image = models.ImageField(upload_to='photos')

class PhotoPost(Post):
    photos = models.ManyToManyField(Photo)

When I attempt to select all posts and prefetch PhotoPost’s photos:

Post.objects.select_subclasses().prefetch_related('photos').all()

The following error is raised:

    (0.001) SELECT `photos_post`.`id`, `photos_post`.`creation_date`, `photos_textpost`.`post_ptr_id`, `photos_textpost`.`body`, `photos_photopost`.`post_ptr_id` FROM `photos_post` LEFT OUTER JOIN `photos_textpost` ON (`photos_post`.`id` = `photos_textpost`.`post_ptr_id`) LEFT OUTER JOIN `photos_photopost` ON (`photos_post`.`id` = `photos_photopost`.`post_ptr_id`) LIMIT 21; args=()
Traceback (most recent call last):
  File "<console>", line 1, in <module>
  File "/Users/anislav/.virtualenvs/mysite/lib/python2.7/site-packages/django/db/models/query.py", line 77, in __repr__
    data = list(self[:REPR_OUTPUT_SIZE + 1])
  File "/Users/anislav/.virtualenvs/mysite/lib/python2.7/site-packages/django/db/models/query.py", line 102, in __iter__
    len(self)
  File "/Users/anislav/.virtualenvs/mysite/lib/python2.7/site-packages/django/db/models/query.py", line 94, in __len__
    self._prefetch_related_objects()
  File "/Users/anislav/.virtualenvs/mysite/lib/python2.7/site-packages/django/db/models/query.py", line 613, in _prefetch_related_objects
    prefetch_related_objects(self._result_cache, self._prefetch_related_lookups)
  File "/Users/anislav/.virtualenvs/mysite/lib/python2.7/site-packages/django/db/models/query.py", line 1742, in prefetch_related_objects
    (attr, first_obj.__class__.__name__, lookup))
AttributeError: Cannot find 'photos' on TextPost object, 'photos' is an invalid parameter to prefetch_related()

Is prefetch_related supported in combination with select_sublcasses? Note that I have tested this on Django 1.5 and 1.6 with the latest release of django-model-utils.

Disclaimer: the given example is reworked version of the following one: http://stackoverflow.com/questions/19511368/prefetch-related-with-django-1-5-django-model-utils

[BrendaH]

I also bumped into this problem with prefetch_related as mentioned in https://github.com/jazzband/django-model-utils/issues/110#issuecomment-34044670

Django version: 1.11.16 django-model-utils version: 3.1.2 Python version: 3.5 Windows machine

Looks like the prefetch cache is not where one would expect it. With Post.objects.select_subclasses().prefetch_related('photopost__photos').all() we would expect all Post objects to have prefetched their PhotoPosts and their Photos. So looping over all PhotoPosts (as returned by select_subclasses) we would expect their Photos to be prefetched and photopost_instance.photos.all() not to result in extra queries. However, the actual prefetch result seems to be stored in photopost_instance.photoposts.photos.all() (so with the extra photoposts in there). Using this last way of accessing the photos does not result in extra queries, but it is a bit weird to have to do so.

I suspect that this happens because prefetch_related is handled after the iterator has converted the found instances to the appropriate child classes.

Unfortunately I do not have time (or the know-how) to make a PR to fix this. (I was hoping this package would be plug-and-play to save me time. 😉 ) So for now I’m going with the weird but working fix of using the extra classname when looking up the prefetched objects. I just wanted to make it known that the bug still exists and what might be causing it.

[carljm]

Hi @Anislav, thanks for the report. I think @kezabelle is likely correct that #78 would fix the reverse-related cache, which seems like part of the fix needed here.

Regarding the three queries, I’m not sure what’s going on there, would need to dig into it. Probably won’t have time to do that soon, but if you figure out the cause I’d be happy to take a pull request!