Import file flow breaks when horizontally scaled app

I’m getting this error on file upload of my prod server, only recently when I bumped up the number of dynos:

  File "/app/.heroku/python/lib/python2.7/site-packages/import_export/admin.py", line 122, in process_import
    import_file = open(import_file_name, input_format.get_read_mode())

IOError: [Errno 2] No such file or directory: u'/tmp/tmpPWSbPm'

The only thing that makes sense to me is the upload and process is happening in separate web requests? Am I missing a config value?

[Bubbassauro]

Leaving this here since I had the same issue and it wasn’t clear to me from the documentation.

What may happen if you’re running on an EC2 server behind a load balancer is:

1. You upload the file
2. You see a preview. By default the temporary file is stored in the server’s file system
3. You click upload and there’s a chance you’ll hit the other server behind the LB, which doesn’t have the file.

So you can use the CacheStorage instead in your admin.py, for example:

from import_export.tmp_storages import CacheStorage

class CustomBookAdmin(ImportMixin, admin.ModelAdmin)
    tmp_storage_class = CacheStorage
    resource_class = BookResource

In our case, we’re using the ImportExportMixin in multiple classes, so I added this:

from import_export.tmp_storages import CacheStorage

class ImportExport(ImportExportMixin):
    """Override temporary storage to use cache instead of file system because
    filesystem has issues when the application is running behind a load balancer"""
    tmp_storage_class = CacheStorage

And replaced on the admin classes that were using it:

class CustomBookAdmin(ImportExport, admin.ModelAdmin):
    # other admin options...

[zehawki]

@Bubbassauro you are a lifesaver. I couldnt figure out why local was fine and production crashed until I found this https://stackoverflow.com/a/42452966 and this thread here.

In my case, I just temporarily removed all but one instances from the LB and ran the import. I didnt want to take a risk with CacheStorage which I understand needs memcache. Each of my EC2 instances has its own memcache, and all are synced, but I wasnt sure whether that would still work. I also didnt know how the the 1MB per key limit would affect when importing 1000s of records.

PS: This is such an excellent library, but the documentation is really lacking.