Cant upload files with lambda to s3

Hi,

I’m trying to use django with zappa and AWS. I’ve created a basic project and some buckets to run with it.

Tried so many permissions and policies on the bucket and IAM role, with no success.

Any idea?

Context

Python 3.6 running with virtualenvironment

Expected Behavior

The file is uploaded to S3 and im transfered back to django admin

Actual Behavior

The action fiails, no file is created and I recieve {“message”: “Endpoint request timed out”}

Your Environment

‘storages’ is in my INSTALLED_APPS

This is in my settings file

AWS_ACCESS_KEY_ID = os.environ.get('AWS_ACCESS_KEY_ID')
AWS_SECRET_ACCESS_KEY = os.environ.get('AWS_SECRET_ACCESS_KEY')
AWS_STORAGE_BUCKET_NAME = os.environ.get('S3_BUCKET_NAME')
AWS_STORAGE_REGION = os.environ.get('S3_REGION')
AWS_HEADERS = {  # see http://developer.yahoo.com/performance/rules.html#expires
    'Expires': 'Thu, 31 Dec 2099 20:00:00 GMT',
    'Cache-Control': 'max-age=94608000',
}

MEDIA_URL = 'http://%s.s3.amazonaws.com/' % AWS_STORAGE_BUCKET_NAME
DEFAULT_FILE_STORAGE = "storages.backends.s3boto.S3BotoStorage"

in my zappa settings I have

 "aws_region": "eu-west-1",
        "vpc_config" : {
            "SubnetIds": [ "*****", "*******"], 
            "SecurityGroupIds": [ "******" ]
        },
        "profile_name": "default",
        "project_name": "****",
        "runtime": "python3.6",
        "s3_bucket": "MU BUCKET",
         "environment_variables": {
            "S3_REGION": "eu-west-1",
            "S3_BUCKET_NAME": "***********",
            "AWS_ACCESS_KEY_ID":"***********",
            "AWS_SECRET_ACCESS_KEY": "*************",
            "AWS_ACCESS_KEY":"***********",
            "AWS_SECRET_KEY": "**********************"
         },
      	"certificate_arn": "arn:aws:acm:***********,

This is my model

class Map(models.Model):
    name = models.CharField(max_length=128, default='')
    description = models.TextField(max_length=1023, default='', blank=True)

    pgm_file = models.FileField(upload_to=path_and_rename,
                                blank=False,
                                default='',
                                max_length=200)

from utils.utils import PathAndRename
path_and_rename = PathAndRename('./test')

class PathAndRename(object):

    def __init__(self, sub_path):
        self.path = sub_path

    def __call__(self, instance, filename):
        ext = filename.split('.')[-1]
        day = time.strftime('%y%m%d')
        # get filename
        if instance.pk:
            filename = '{}.{}'.format(instance.pk, ext)
        else:
            # set filename as random string
            timestamp = int(time.time())
            name = str(uuid4().hex[:3] + str(timestamp)[-5:])
            filename = '{}.{}'.format(name, ext)

        try:
            instance_type = type(instance).__name__
            username = instance.created_by.username
        except Exception:
            username = ''
            instance_type = ''

        return os.path.join(self.path, instance_type, username, day, filename)

Bucket policy

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Sid": "PublicReadGetObject",
            "Effect": "Allow",
            "Principal": "*",
            "Action": "s3:GetObject",
            "Resource": "arn:aws:s3::MY_BUCKET/*"
        },
        {
            "Sid": "AllowS3Access",
            "Effect": "Allow",
            "Principal": {
                "AWS": [
                    "arn:aws:iam::***********:role/ZAPPA-ROLE"
                ]
            },
            "Action": "s3:*",
            "Resource": [
                "arn:aws:s3:::MY_BUCKET",
                "arn:aws:s3:::MY_BUCKET/*"
            ]
        }
    ]
}

CORS

<CORSRule>
    <AllowedOrigin>*</AllowedOrigin>
    <AllowedMethod>HEAD</AllowedMethod>
    <AllowedMethod>GET</AllowedMethod>
    <AllowedMethod>PUT</AllowedMethod>
    <AllowedMethod>POST</AllowedMethod>
    <AllowedMethod>DELETE</AllowedMethod>
    <AllowedHeader>*</AllowedHeader>
</CORSRule>

[edgarroman]

If your VPC subnets are private you will either:

1. have to create a NAT gateway to get to external S3
2. Create an S3 endpoint in your VPC subnets to allow your lambda to get access.

[talbenbasat]

@edgarroman Thank you very very much!! The endpoint solved the problem