question-mark
Stuck on an issue?

Lightrun Answers was designed to reduce the constant googling that comes with debugging 3rd party libraries. It collects links to all the places you might be looking at while hunting down a tough bug.

And, if you’re still stuck at the end, we’re happy to hop on a call to see how we can help out.

The encoding problem in outputing data to file

See original GitHub issue

Here I got an upset problem. When I use -o file option, I find the text in file just display as below: {"title": "Docker \u5b9e\u6218\u7cfb\u5217 \u2014 \u8fd0\u884c\u7b2c\u4e00\u4e2a\u5bb9\u5668", "short_desc": "\u5f53\u4f60\u5bf9\u4e00\u4e9b\u6982\u5ff5\u7406\u89e3\u4e0d\u6df1\u7684\u65f6\u5019\uff0c\u5b9e\u8df5\u662f\u6700\u597d\u7684\u89e3\u51b3\u529e\u6cd5\n\n", "time": "07 Apr 2015"},

while I run file outputfile command in terminal, it shows blog.json: ASCII text, with very long lines. I can’t understand, really confused. I just want the file display with my local language. And I know write utf-8 string to file will work correctly. So I added encode('utf-8') before assigning to Item[‘key’], but still no any change. Please help me, thx

Issue Analytics

  • State:closed
  • Created 7 years ago
  • Comments:8 (5 by maintainers)

github_iconTop GitHub Comments

13reactions
redapplecommented, Apr 29, 2016

If you ever want to write JSON items as UTF-8, you can do it like this:

  1. define a custom item exporter, e.g. in an exporters.py file in your project
$ cat myproject/exporters.py 
from scrapy.exporters import JsonItemExporter


class Utf8JsonItemExporter(JsonItemExporter):

    def __init__(self, file, **kwargs):
        super(Utf8JsonItemExporter, self).__init__(
            file, ensure_ascii=False, **kwargs)
  1. replace the default JSON item exporter in your settings.py
FEED_EXPORTERS = {
    'json': 'myproject.exporters.Utf8JsonItemExporter',
}
1reaction
wi-cuckoocommented, Apr 30, 2016

I don’t think overwriting the JsonItemExporter class is a simple solution. If we can add an option in -o file , like --encoding=‘utf-8’, this would be more elegant. Well, only personal idea.

Read more comments on GitHub >

github_iconTop Results From Across the Web

Encoding problem when writing to a file in x86 - Stack Overflow
I'm a new to x86 and currently writing a program where I need to write a file after doing some operations with data...
Read more >
Choose text encoding when you open and save files
To avoid problems with encoding and decoding text files, you can save files with Unicode encoding. Unicode accommodates most characters sets across all...
Read more >
Error reading/writing file: (The number of encoding ... - IBM
So the issue is not with the length of the data itself but with the file content. Something with the file is either...
Read more >
Text File Encoding Problem - FME Community - Safe Software
Hi,. I have a problem encoding a text file to the correct format. The below images show: 1) the *.prj document properties of...
Read more >
Solved: Help reading in File, might be encoding problem?
Solved: Hello everyone. My company has a tool that exporst data to csv all in one column, but comma seperated. The output file...
Read more >

github_iconTop Related Medium Post

No results found

github_iconTop Related StackOverflow Question

No results found

github_iconTroubleshoot Live Code

Lightrun enables developers to add logs, metrics and snapshots to live code - no restarts or redeploys required.
Start Free

github_iconTop Related Reddit Thread

No results found

github_iconTop Related Hackernoon Post

No results found

github_iconTop Related Tweet

No results found

github_iconTop Related Dev.to Post

No results found

github_iconTop Related Hashnode Post

No results found