question-mark
Stuck on an issue?

Lightrun Answers was designed to reduce the constant googling that comes with debugging 3rd party libraries. It collects links to all the places you might be looking at while hunting down a tough bug.

And, if you’re still stuck at the end, we’re happy to hop on a call to see how we can help out.

Create scrapy command to get the equivalent Request from a cURL command

See original GitHub issue

Hi!

Since https://github.com/scrapy/scrapy/pull/3862 was accepted I was suggested to create a scrapy command that receive a cURL command and return the equivalent Request __repr__.

Something like:

scrapy requestfromcurl "curl 'http://example.org/post' -X POST -H 'Cookie: _gauges_unique_year=1; _gauges_unique=1'"

would print:

Request(method='POST', url='http://example.org/post', cookies={'_gauges_unique_year': '1', '_gauges_unique': '1'})

I just did a PR (https://github.com/scrapy/scrapy/pull/3990) to show you the idea, but I would like to ask you:

1st. Do you like the idea? Is it useful? (I’m not sure about it)

In the affirmative case:

2nd. how would you name that command? (ideas: requestfromcurl, curltorequest…) 3rd. where would you mention that command in the docs?

Thanks in advance

Issue Analytics

  • State:open
  • Created 4 years ago
  • Comments:12 (5 by maintainers)

github_iconTop GitHub Comments

5reactions
michael-shubcommented, Dec 3, 2019

Hi! I’ve recently built small visual tool which help to translate curl requests to scrapy code. Here’s a link https://michael-shub.github.io/curl2scrapy/ and here’s github page: https://github.com/michael-shub/curl2scrapy

1reaction
michael-shubcommented, Dec 5, 2019

Updated curl2scrapy a bit. Now seems to be working with simpler curl requests. Also added onPaste handler to avoid pressing extra buttons and save your time.

Read more comments on GitHub >

github_iconTop Results From Across the Web

Requests and Responses — Scrapy 2.7.1 documentation
Create a Request object from a string containing a cURL command. It populates the HTTP method, the URL, the headers, the cookies and...
Read more >
curl2scrapy - cURL command -> Scrapy request translator.
Curl source: Scrapy code: Press Ctrl + ENTER to translate..
Read more >
Using curl information with scrapy shell - python - Stack Overflow
I'm using Scrapy to get some information of some websites. To be more precise, I'm using it from the scrapy shell, with this...
Read more >
Convert curl commands to code
curl from Google Chrome. Open the Network tab in the DevTools; Right click (or Ctrl-click) a request; Click "Copy" → "Copy as cURL""Copy...
Read more >
Using your browser's Developer Tools for scraping
Note that to translate a cURL command into a Scrapy request, you may use curl2scrapy. As you can see, with a few inspections...
Read more >

github_iconTop Related Medium Post

No results found

github_iconTop Related StackOverflow Question

No results found

github_iconTroubleshoot Live Code

Lightrun enables developers to add logs, metrics and snapshots to live code - no restarts or redeploys required.
Start Free

github_iconTop Related Reddit Thread

No results found

github_iconTop Related Hackernoon Post

No results found

github_iconTop Related Tweet

No results found

github_iconTop Related Dev.to Post

No results found

github_iconTop Related Hashnode Post

No results found