question-mark
Stuck on an issue?

Lightrun Answers was designed to reduce the constant googling that comes with debugging 3rd party libraries. It collects links to all the places you might be looking at while hunting down a tough bug.

And, if you’re still stuck at the end, we’re happy to hop on a call to see how we can help out.

Can I pass more than one setting to `schedule.json` ?

See original GitHub issue

Hello! Thanks for seeing this issue.

The doc of schedule.jsonsays:

setting (string, optional) - a Scrapy setting to use when running the spider

So can I pass two or more Scrapy settings to the spider?

For example, I can run the following command in command line:

scrapy crawl somespider -s JOBDIR='./jobs' -s CLOSESPIDER_TIMEOUT=100

Then it can pass both two settings to the job.

What should I do if I want to pass those two settings via API schedule.json ?

Thank you!

Issue Analytics

  • State:closed
  • Created 5 years ago
  • Reactions:2
  • Comments:6

github_iconTop GitHub Comments

6reactions
wlh320commented, Jul 19, 2018

@nadzimo Hi! You can pass settings with any HTTP client like requests or urllib. I use requests in my project. The following code is for your reference.

settings = [
    ('project', 'default'),
    ('spider', 'xxx'),
    ('setting', 'JOBDIR=%s' % jobdir),
    ('setting', 'CLOSESPIDER_TIMEOUT=%d' % duration)
]
res = requests.post('http://localhost:6800/schedule.json', data=settings)
res = res.json()
if res.get('status') == 'ok':
    pass

In the above code, I pass a list of tuples as POST data instead of dict to pass more than one setting to scrapyd API, or values with same key will be overridden.

2reactions
Apuyusengcommented, Aug 6, 2019

you can

import requests

data = {
      ...
      setting=["JOBDIR=%s"% jobdir, "CLOSESPIDER_TIMEOUT=%s"%timeout]
}
resp = requests.post(url,data=data)

by soure code add print ,this is able

Read more comments on GitHub >

github_iconTop Results From Across the Web

Pass multiple parameters to rest API - Spring - Stack Overflow
Yes. I'd suggest you do a web search. I'd also suggest using a library, such as jQuery , to make it easier.
Read more >
Combine multiple requests in one HTTP call using JSON ...
Use JSON batching to optimize your application by combining multiple requests into a single JSON object, saving the application significant network latency.
Read more >
Scheduling queries | BigQuery - Google Cloud
Shows how to set up, update, and delete scheduled queries in BigQuery, and describes configuration options, quotas, pricing, and supported regions for ...
Read more >
JSON Event - Qualtrics
The JSON Event can be used to integrate with Zendesk as an HTTP target in a trigger. That means events in Zendesk can...
Read more >
Schedules - Zendesk Developer Docs
You can use this API to create multiple schedules with different business hours and holidays. To learn more about schedules, see Setting your...
Read more >

github_iconTop Related Medium Post

No results found

github_iconTop Related StackOverflow Question

No results found

github_iconTroubleshoot Live Code

Lightrun enables developers to add logs, metrics and snapshots to live code - no restarts or redeploys required.
Start Free

github_iconTop Related Reddit Thread

No results found

github_iconTop Related Hackernoon Post

No results found

github_iconTop Related Tweet

No results found

github_iconTop Related Dev.to Post

No results found

github_iconTop Related Hashnode Post

No results found