Stuck on an issue?

Lightrun Answers was designed to reduce the constant googling that comes with debugging 3rd party libraries. It collects links to all the places you might be looking at while hunting down a tough bug.

And, if you’re still stuck at the end, we’re happy to hop on a call to see how we can help out.

S3 compatible API services

See original GitHub issue

Can we use Feed exporter for S3 to store the items in an s3 compatible service, like Minio or Riak CS?

I tried setting the FEED_URI to my local instance of minio http://192.168.1.111:9000/minio/mybucket/%(name)s.json

but scrapy complains that “Unknown feed storage shceme: http”.

If I try changing it to s3://192.168.1.111:9000/minio/mybucket/%(name)s.json it tries to pre-pend it to amazon:

CertificateError: hostname 192.168.1.111.s3.amazonaws.com doesn't match either of '*.s3.amazonaws.com', 's3.amazon.com'

Any thoughts? Am I doing something wrong?

Issue Analytics

State:
Created 7 years ago
Comments:6 (1 by maintainers)

Top GitHub Comments

1reaction

LaundroMatcommented, Jul 19, 2017

Waw, I must have been dead tired for not having seen the link 🙄- sorry for that! (and thanks for replying)

1reaction

drozzycommented, Feb 1, 2017

Solved, by writing my own item exporter, and item pipeline as per the docs: https://doc.scrapy.org/en/latest/topics/exporters.html

Used the minio client to push events to s3 service directly. Works great. If anyone is interested, the source is here: https://bitbucket.org/drozdyuk/scrapy_quotes