Allow for zero-copy file uploads.

Right now, Bottle directly calls cgi.FieldStorage to parse incoming data.

cgi.FieldStorage has a ‘make_file’ method defined with the following comment:

def make_file(self, binary=None):
    """Overridable: return a readable & writable file.

    The file will be used as follows:
    - data is written to it
    - seek(0)
    - data is read from it

    The 'binary' argument is unused -- the file is always opened
    in binary mode.

    This version opens a temporary file for reading and writing,
    and immediately deletes (unlinks) it.  The trick (on Unix!) is
    that the file can still be used, but it can't be opened by
    another process, and it will automatically be deleted when it
    is closed or when the current process terminates.

    If you want a more permanent file, you derive a class which
    overrides this method.  If you want a visible temporary file
    that is nevertheless automatically deleted when the script
    terminates, try defining a __del__ method in a derived class
    which unlinks the temporary files you have created.

    """

It would be nice if Bottle allowed more easy access to providing your own FieldStorage class implementation, or perhaps specifically allowed for optionally providing a make_file implementation.

My use case is that I would like to have Bottle upload file data directly to its final storage location, instead of being uploaded to a tempfile and then being copied again to its final destination.

[Nbelles]

@defnull Thank you for explaining in such great detail.

I think if we are unable to decrease the time taken to parse the multipart requests, then it would be nice to decrease the time required to move the files to their final resting place.

To make that happen, it sounds like the ideal scenario would be using the TMPDIR method to set the directory where the file will be written, combined with a tempfile.NamedTemporaryFile such that the file can easily be renamed to its final name without needing to be moved again. This would ensure that you are writing to the destination file system when parsing the file and then instead of using file.save(...) the designer would just have to rename the file instead of moving the file anywhere. Does this make sense?

Is there a place where I can find more info about using the TMPDIR environment variable method to change the location where the file is place? I hadn’t heard of doing something like this before you mentioned it. Thanks!

[Nbelles]

I would love to see this implemented if possible. There is still interest in this use case. I tried implementing the pull request on my local machine but there have been so many changes since this was suggested that it was returning some errors. If it is possible to have this option implemented, that would be awesome. I am happy to help with any testing!