[python] 3rd party modules

currently, for 3rd party module, we have to do pip install -t . on the same directory as template.yaml and the main python application file. This pollutes the working directory when you have a lot of modules. Is there a better way to handle this? maybe on the file zipping of a subdirectory?

[randyjohnston]

Could you please clarify how to set up template.yml for Python module with dependencies? The example template.yml does not make sense and references a non-existent zip file.

[heitorlessa]

I’ll make a proper PR as soon as I get time to clean comments up and test in other OSes, but if that helps here’s what I came up with (tested in OSX only) to have dependencies installed + hot-reloading experience similar to NodeJS.

Essentially it takes advantage of hard links and clones the entire source code/folders (lib, etc.) onto a folder called dev. All you have to do is to change your SAM template to point to <source_code>/dev.

Makefile - Hardcoded for Python 3.6 and pip :

# VirtualEnv vars
SERVICE ?= users
VENV := ${SERVICE}/.venv
VENV_LIBS := $(VENV)/lib/python3.6/site-packages

# help credit (https://gist.github.com/prwhite/8168133)
help: ## Show this help

	@fgrep -h "##" $(MAKEFILE_LIST) | fgrep -v fgrep | sed -e 's/\\$$//' | sed -e 's/##//'

target: help

	exit 0

clean: ## -> Deletes current virtual env environment

	$(info "[-] Who needs all that anyway? Destroying environment....")
	rm -rf ./$(VENV)
	rm -rf ./$(SERVICE)/dev
	rm -rf ./*.zip

build-dev: _check_service_definition ## -> creates dev folder and install dependencies (requirements.txt) into dev/ folder

	echo "[+] Cloning ${SERVICE} directory structure to ${SERVICE}/dev"
	rsync -a -f "+ */" -f "- *" -f "- dev/" ${SERVICE}/ ${SERVICE}/dev/
	echo "[+] Cloning source files from ${SERVICE} to ${SERVICE}/dev"
	find ${SERVICE} -type f \
			-not -name "*.pyc" \
			-not -name "*__pycache__" \
			-not -name "requirements.txt" \
			-not -name "event.json" \
			-not -name "dev" | cut -d '/' -f2- > .results.txt
	@while read line; do \
		ln -f ${SERVICE}/$$line ${SERVICE}/dev/$$line; \
	done < .results.txt
	echo "[+] Installing dependencies into dev/"
	pip3.6 install \
		--isolated \
		--disable-pip-version-check \
		-Ur $(SERVICE)/requirements.txt -t ${SERVICE}/dev/

_check_service_definition:

	echo "[*] Checking whether service $(SERVICE) exists..."

# SERVICE="<name_of_service>" must be passed as ARG for target or else fail
ifndef SERVICE
	echo "[!] SERVICE argument not defined...FAIL"
	echo "[*] Try running 'make <target> SERVICE=<service_name>'"
	exit 1
endif

ifeq ($(wildcard $(SERVICE)/.),)
	echo "[!] $(SERVICE) folder doesn't exist"
	exit 1
endif

ifeq ($(wildcard $(SERVICE)/requirements.txt),)
	echo "[!] Pip requirements file missing from $(SERVICE) folder..."
	exit 1
endif

run-dev: ## -> Run SAM Local API GW
	sam local start-api

Folder structure - Sample with multiple private packages (vendor/db, vendor/dispatcher) and external dependencies (boto3, requirements, etc.)

.
├── Makefile
├── bootstrap_dynamodb.py
├── setup.cfg
├── template.yaml
└── users
    ├── requirements.txt
    ├── users.py
    ├── something_else.py
    └── vendor
        ├── db
        │   └── __init__.py
        ├── dispatcher
        │   ├── __init__.py
        ├── helper
        │   └── __init__.py
        └── users
            ├── __init__.py

SAM

    UsersFunction:
      Type: AWS::Serverless::Function
      Properties:
        CodeUri: ./users/dev/
        Handler: users.lambda_handler
        Runtime: python3.6
        Environment:
          Variables:
            TABLE_NAME: "users"
        Events:
          CatchAll:
            Type: Api
            Properties:
              Path: /{proxy+}
              Method: any

Usage

make build-dev SERVICE="users"

Where users is where my source code is as it’s a users service sample.