question-mark
Stuck on an issue?

Lightrun Answers was designed to reduce the constant googling that comes with debugging 3rd party libraries. It collects links to all the places you might be looking at while hunting down a tough bug.

And, if you’re still stuck at the end, we’re happy to hop on a call to see how we can help out.

scrapy FormRequest ignore Content-Type other values

See original GitHub issue

The following code

super(FormRequest, self).__init__(*args, **kwargs)
if formdata:
    items = formdata.items() if isinstance(formdata, dict) else formdata
    querystr = _urlencode(items, self.encoding)
    if self.method == 'POST':
        self.headers.setdefault('Content-Type', 'application/x-www-form-urlencoded')
        self._set_body(querystr)
    else:
        self._set_url(self.url + ('&' if '?' in self.url else '?') + querystr)

Although the FormRequest base class Request sets the request header based on the value of the headers band, such as setting Content-Type = “application / json; charset = utf-8”, the code

 self.headers.setdefault('Content-Type', 'application/x-www-form-urlencoded')

sets the Content- Type value back to "application / x -www-form-urlencoded ". He did not consider headers own Content-Type situation.

Issue Analytics

  • State:closed
  • Created 6 years ago
  • Comments:5 (4 by maintainers)

github_iconTop GitHub Comments

1reaction
cathalgarveycommented, Feb 15, 2018

Hi @andywubit - I’m inclined to close this unless we can get a reproducible example case to demonstrate the issue. It appears, from my reading and others, to be perhaps a case of misunderstanding. If you think we could improve our documentation to make this clearer, that’s a potential way we could take this?

Looking forward to your response.

0reactions
dangracommented, Dec 12, 2017

@andywubit can you provide a code snippet that reproduces your issue? thanks

Read more comments on GitHub >

github_iconTop Results From Across the Web

Requests and Responses — Scrapy 2.7.1 documentation
It accepts the same arguments as the Request class, taking preference and overriding the values of the same arguments contained in the cURL ......
Read more >
Python Scrapy override content type to be multipart/form-data ...
Just use scrapy.http.FormRequest instead of scrapy.Request, passing the parameters in the formdata argument. Sample code:
Read more >
Requests and Responses - Scrapy documentation
Scrapy uses Request and Response objects for crawling web sites. ... Returns a new FormRequest object with its form field values pre-populated with...
Read more >
Scrapy Documentation - Read the Docs
Scrapy (/skrepa/) is an application framework for crawling web sites and extracting structured data which can be used.
Read more >
Easy Way to Handle Form POST with Python Scrapy - YouTube
Handling POST requests can be a little tricky with scrapy. In this video, we will see a specific scenario, where the POST request...
Read more >

github_iconTop Related Medium Post

No results found

github_iconTop Related StackOverflow Question

No results found

github_iconTroubleshoot Live Code

Lightrun enables developers to add logs, metrics and snapshots to live code - no restarts or redeploys required.
Start Free

github_iconTop Related Reddit Thread

No results found

github_iconTop Related Hackernoon Post

No results found

github_iconTop Related Tweet

No results found

github_iconTop Related Dev.to Post

No results found

github_iconTop Related Hashnode Post

No results found