Starting scrapyd docker container with eggs included

Hi I’ve been experimenting a little with scrapyd on docker, and done the following:

• in the config file, i specified different directory for eggs eggs_dir = /src/eggs
• in dockerfile, i added prebuilt projects to this directory ADD eggs /src/eggs

At first glance, it looked like it’s working

but, when I wanted to make a scheduje.json post, it returned me an error

{“node_name”: “295e305bea8e”, “status”: “error”, “message”: “Scrapy 1.4.0 - no active project\n\nUnknown command: list\n\nUse "scrapy" to see available commands\n”}

I could type anything into project and spider fields and the result was the same. How can I fix this issue?

[jacob1237]

@VanDavv @iamprageeth @radyz

I managed to solve the problem without using the API. Unfortunately, there is no way to deploy Scrapy projects without the egg files completely (the only way is to override some scrapyd components), so you’ll need a simple deployment script:

build.sh:

#!/bin/sh

set -e

# The alternative way to build eggs is to use setup.py
# if you already have it in the Scrapy project's root
scrapy-deploy --build-egg=myproject.egg

# your docker container build commands
# ...

Dockerfile:

RUN mkdir -p eggs/myproject
COPY myproject.egg eggs/myproject/1_0.egg

CMD ["scrapyd"]

That’s all! So instead of deploying myproject.egg into the eggs folder directly, you have to create the following structure: eggs/myproject/1_0.egg where myproject is your project name, and 1_0 is a version of your project in scrapyd

[radyz]

I managed to get through this by running a background deploy after my scrapyd instance has started. Not sure it’s the best way but it works for me now

Dockerfile

FROM python:3.6

COPY requirements.txt /requirements.txt
RUN pip install -r requirements.txt
COPY docker-entrypoint /usr/local/bin/
RUN chmod 0755 /usr/local/bin/docker-entrypoint
COPY . /scrapyd
WORKDIR /scrapyd

ENTRYPOINT ["/usr/local/bin/docker-entrypoint"]

Entrypoint script

#!/bin/bash
bash -c 'sleep 15; scrapyd-deploy' &
scrapyd

scrapy.cfg

[settings]
default = scraper.settings

[deploy]
url = http://localhost:6800
project = projectname

This assumes you are copying your scrapy project folder into /scrapyd and have the requirements.tx with all your dependencies (including scrapyd server)