Stuck on an issue?

Lightrun Answers was designed to reduce the constant googling that comes with debugging 3rd party libraries. It collects links to all the places you might be looking at while hunting down a tough bug.

And, if you’re still stuck at the end, we’re happy to hop on a call to see how we can help out.

After adding request flags subclasses of logformatter that rely on 'flags' format string are broken

See original GitHub issue

#2082 added flags to request but it also renamed formatting string key from flags to response_flags/request_flags

CRAWLEDMSG = u"Crawled (%(status)s) %(request)s (referer: %(referer)s)%(flags)s"
 +CRAWLEDMSG = u"Crawled (%(status)s) %(request)s%(request_flags)s (referer: %(referer)s)%(response_flags)s"

Scrapy allows you to override logformatter and this is what I have in my project. I have logformatter looking rouhgly like this

# dirbot/logf.py
from scrapy.logformatter import LogFormatter


class CustomLogFormatter(LogFormatter):
    def crawled(self, request, response, spider):
        kwargs = super(CustomLogFormatter, self).crawled(
            request, response, spider)
        kwargs['msg'] = (
            u"Crawled (%(status)s) %(request)s "
            u"(referer: %(referer)s, latency: %(latency).2f s)%(flags)s"
        )
        kwargs['args']['latency'] = response.meta.get('download_latency', 0)
        return kwargs

now if you enable it in settings LOG_FORMATTER = 'dirbot.logf.CustomLogFormatter' and try to run it with recent master you’ll get KeyError

2017-03-13 14:15:26 [scrapy.extensions.telnet] DEBUG: Telnet console listening on 127.0.0.1:6023
Traceback (most recent call last):
  File "/usr/lib/python2.7/logging/__init__.py", line 851, in emit
    msg = self.format(record)
  File "/usr/lib/python2.7/logging/__init__.py", line 724, in format
    return fmt.format(record)
  File "/usr/lib/python2.7/logging/__init__.py", line 464, in format
    record.message = record.getMessage()
  File "/usr/lib/python2.7/logging/__init__.py", line 328, in getMessage
    msg = msg % self.args
KeyError: u'flags'
Logged from file engine.py, line 238
Traceback (most recent call last):
  File "/usr/lib/python2.7/logging/__init__.py", line 851, in emit
    msg = self.format(record)
  File "/usr/lib/python2.7/logging/__init__.py", line 724, in format
    return fmt.format(record)
  File "/usr/lib/python2.7/logging/__init__.py", line 464, in format
    record.message = record.getMessage()
  File "/usr/lib/python2.7/logging/__init__.py", line 328, in getMessage
    msg = msg % self.args
KeyError: u'flags'
Logged from file engine.py, line 238
2017-03-13 14:15:27 [scrapy.core.scraper] DEBUG: Scraped from <200 http://www.dmoz.org/Computers/Programming/Languages/Python/Resources/>

So this change that renamed flags to response_flags/request_flags seems backward incompatible.

Issue Analytics

State:
Created 7 years ago
Comments:5 (5 by maintainers)

Top GitHub Comments

1reaction

kmikecommented, Mar 13, 2017

Please go on with a pull request! 👍

0reactions

pawelmhmcommented, Mar 13, 2017

ah yeah that fixes it and looks super simple, should I create patch with unit test or you’d like to fix it

Top Results From Across the Web

Access logging — envoy 1.25.0-dev-367763 documentation

Format strings are plain strings, specified using the format key. They may contain either command operators or other characters interpreted as a plain...

Logging Cookbook — Python 3.11.1 documentation

INFO) # set a format which is simpler for console use formatter = logging. ... be output using a suitable format string, or...

java.util Class Formatter

The optional flags is a set of characters that modify the output format. The set of valid flags depends on the conversion.

Manual Page - GoAccess

Additional static files can be added to the configuration file. ... This panel is displayed if %v is used within the log-format string....

printf format identifiers. - LIX

printf formatting is controlled by 'format identifiers' which, are shown below in their ... %s String. ... Flags, Minimum field width, Period, Precision....