BUG: Logstash in Python 3, byte to string in handler_tcp.py

The following error is thrown when configuring logstash for Gunicorn.

Traceback (most recent call last):                                                                                          
  File "/usr/lib/python3.5/logging/handlers.py", line 621, in emit                                                          
    s = self.makePickle(record)                                                                                             
  File "~/Projects/env/lib/python3.5/site-packages/logstash/handler_tcp.py", line 25, in makePickle    
    return self.formatter.format(record) + b'\n'                                                                            
TypeError: Can't convert 'bytes' object to str implicitly                                                                   

Changing handler_tcp.py to:

from logging.handlers import DatagramHandler, SocketHandler
from logstash import formatter


# Derive from object to force a new-style class and thus allow super() to work
# on Python 2.6
class TCPLogstashHandler(SocketHandler, object):
    """Python logging handler for Logstash. Sends events over TCP.
    :param host: The host of the logstash server.
    :param port: The port of the logstash server (default 5959).
    :param message_type: The type of the message (default logstash).
    :param fqdn; Indicates whether to show fully qualified domain name or not (default False).
    :param version: version of logstash event schema (default is 0).
    :param tags: list of tags for a logger (default is None).
    """

    def __init__(self, host, port=5959, message_type='logstash', tags=None, fqdn=False, version=0):
        super(TCPLogstashHandler, self).__init__(host, port)
        if version == 1:
            self.formatter = formatter.LogstashFormatterVersion1(message_type, tags, fqdn)
        else:
            self.formatter = formatter.LogstashFormatterVersion0(message_type, tags, fqdn)

    def makePickle(self, record):
        return str.encode(self.formatter.format(record)) + b'\n'

Specifically the last line:

return str.encode(self.formatter.format(record)) + b'\n'

Encoding the string to bytes allows the library to run.

Didn’t want to submit a pull request in case you or someone else had a better solution.

[israel-fl]

@SalahAdDin I’ve modified the handler_tcp.py file to encode the string. You can actually copy and paste this block into your file

from logging.handlers import DatagramHandler, SocketHandler
from logstash import formatter


# Derive from object to force a new-style class and thus allow super() to work
# on Python 2.6
class TCPLogstashHandler(SocketHandler, object):
    """Python logging handler for Logstash. Sends events over TCP.
    :param host: The host of the logstash server.
    :param port: The port of the logstash server (default 5959).
    :param message_type: The type of the message (default logstash).
    :param fqdn; Indicates whether to show fully qualified domain name or not (default False).
    :param version: version of logstash event schema (default is 0).
    :param tags: list of tags for a logger (default is None).
    """

    def __init__(self, host, port=5959, message_type='logstash', tags=None, fqdn=False, version=0):
        super(TCPLogstashHandler, self).__init__(host, port)
        if version == 1:
            self.formatter = formatter.LogstashFormatterVersion1(message_type, tags, fqdn)
        else:
            self.formatter = formatter.LogstashFormatterVersion0(message_type, tags, fqdn)

    def makePickle(self, record):
        return str.encode(self.formatter.format(record)) + b'\n'

Alternatively, I’ve made a fork of this repo and updated it to work with Python3. You can run

pip3 install python3-logstash

to install it.

[sibiryoff]

TypeError: Can’t convert ‘bytes’ object to str implicitly

I believe this is an issue with Python 3.5/Python 3.6. Can you confirm that this problem still exists within the 3.7.x version? Or probably, there are some suggestions about upgrading ulule/python-logstash-formatter#41 to 0.5.16 should resolve the issue.

Problem still exists in 3.7. I’m using JSON-log-formatter.