Node.js Winston log formatter pollutes all user-defined transports with NodeJS Agent-specific fields

Description

When using the Winston logging library, the New Relic Node.js agent replaces the value for any field on the logging object named timestamp with the equivalent Unix timestamp. In addition, the NodeJS agent is adding additional items to the log data for all transports:

• span.id
• trace.id
• entity.guid
• entity.name
• entity.type

Expected Behavior

While the documentation does state that any timestamp field on the logging data object would be copied to a new field named original_timestamp and the timestamp field replaced with the UNIX timestamp equivalent format, and that the other fields listed above would also be added to the logging data object, I expected that it would only be for the data being sent to New Relic, and not polluting all of my other configured transport/sink outputs.

Any data manipulation of the logging data should only be apparent in the log data directly sent to New Relic, and not in any other log outputs that may be configured by the application.

Steps to Reproduce

Here’s how to reproduce today’s behavior.

First, here’s my winston logger initializer, in logging.ts, which is imported in my application’s composition root:

import jsonStringify from 'safe-stable-stringify';
import { SPLAT } from 'triple-beam'; // This is a winston "component"...
import {
  Logform,
  Logger,
  LoggerOptions,
  config as Config,
  createLogger,
  format,
  transports as Transports
} from 'winston';
import TransportStream from 'winston-transport';

export const initializeLogger = () : Logger => {
  let logSinks: TransportStream[] = [];

  const parseLogLevel = (s: string): LogLevel => {
    // Code in here returns an expected log level string for configuring the logger...
    // no need to bother with the actual implementation...
    return s as LogLevel;
  }

  // ***********************************************************************************
  // *    ***** ****     I    M    P    O    R    T    A    N    T    ****    *****    *
  // ***********************************************************************************
  // ! format.combine creates a "pipeline" of formatters.
  // ! I.e., when adding a formatter to the pipeline, each one is applied, in order,
  // ! to the `info` object. So ordering is IMPORTANT when creating the base logger
  // ! formatter and subsequent transport formatters.
  // ! Also be aware that some formatters MUTATE the `info` object, causing subsequent
  // ! transports to not get the same data as was present on the original, unmutated
  // ! `info` object.
  // ! All formatters created for this app ensure that they operate on a clone of the
  // ! `info` object so that each transport gets the ORIGINAL log data as it came into
  // ! the logging pipeline so that, for each transport, the data can be formatted as
  // ! desired.

  // A custom formatter that prints plain-text log files where each entry has the format:
  //   [2021-08-07T18:35:01.159-0400] [INFO   ] [ApiLocation    ] The loge message { extra: "Other data" }
  const StandardUnstructuredFormat = format.printf(((info: Logform.TransformableInfo) => {
    const { timestamp, level, message, location, ...rest } = info;
    return `[${timestamp}] [${level}] ${location ? `[${(location as string).padEnd('LongestLengthLocation'.length)}] ` : ''}${message}${!(!!rest && Object.keys(rest).length === 0 && rest.constructor === Object) ? ' ' + jsonStringify(rest) : ''}`
  }));

  const loggerOptions: LoggerOptions = {
    exitOnError:  false,

    // This is a "base" log formatter. It gets
    // `format.combine()`ed with specified transport formatters,
    // if any, and runs BEFORE any other formatters defined on
    // transports.
    format: format.combine(
      // THIS GETS REPLACED BY THE NEW RELIC NODEJS AGENT :(
      format.timestamp({ format: 'YYYY-MM-DD HH:mm:ss.SSSZZ' } as Logform.TimestampOptions)
    ),
    levels: Config.syslog.levels
  };

  // Here, I setup my log transports
  if (logToFile) {
    logSinks = [
      ...logSinks,
      new Transports.File({
        dirname: `${process.env.MY_APP_LOG_FILE_DIR ?? '.'}`,
        filename: `${process.env.MY_APP_LOG_FILE_BASE_FILENAME ?? 'my-app'}.log.json`,
        // REMEMBER: formatters listed here get COMBINED with the "base" formatters set
        //           on the logging instance itself!
        format: format.json(),
        level: fileLogLevel
      }),
      new Transports.File({
        dirname: `${process.env.MY_APP_LOG_FILE_DIR ?? '.'}`,
        filename: `${process.env.MY_APP_LOG_FILE_BASE_FILENAME ?? 'my-app'}.log`,
        // REMEMBER: formatters listed here get COMBINED with the "base" formatters set
        //           on the logging instance itself!
        format: format.combine(StandardUnstructuredFormat),
        level: fileLogLevel
      })
    ]
  }

  if (logToConsole) {
    logSinks = [
      ...logSinks,
      new Transports.Console({
        // REMEMBER: formatters listed here get COMBINED with the "base" formatters set
        //           on the logging instance itself!
        format: format.combine( StandardUnstructuredFormat),
        level: consoleLogLevel
      })
    ]
  }

  // Now add the log sinks/transports to the logger options
  loggerOptions.transports = logSinks;

  // and create the logger
  return createLogger(loggerOPtions);
}

export default initializeLogger;

Next, in my application’s composition root, index.ts:

import dotEnvExpand from 'dotenv-expand';
import dotEnvFlow from 'dotenv-flow';
import initializeLogger from './logging';
import setupApp from './app';

// Setup the environment by reading the environment variables and any specified .env.* files.
dotEnvExpand(dotEnvFlow.config());

// Some other basic app initialization happens here, such as the expected URL and port
// that Express will listen on...
const baseAppUrl = `${process.env.MY_APP_SCHEME}://${process.env.MY_APP_HOSTNAME}`;
const port = process.env.MY_APP_API_PORT || process.env.PORT;
const generateAppUrl = (port?: string) => `${baseAppUrl}:${tryConvertEnvVarToNumber(port, 5000)}`.replace(/.*:$/, '');

// This initializes the logger using the initializer above.
// New Relic stomps all over my logger configuration! :(
// If I ONLY wanted to send logs to New Relic, that'd be one thing.
// But in this early stage of "getting to know you", I want to continue
// to log to file and/or console.
//
// Also, for development purposes, I don't want to log to New Relic
// (easily accomplished by configuring the environment variables), but
// do want to log to console (and/or file).
const logger = initilaizeLogger().child({location: "MyApp"});

cost app = setupApp(/* Application configuration goes here */);
app.listen(port, () => {
  console.log('Starting MyApp...\n')

  try {
    logger.info('Application starting...');
    logger.info(`Running in '${process.env.NODE_ENV?.toUpperCase()}' mode.`);
    logger.info('Application started.');
    logger.info(`Listening on ${generateAppUrl(port)}.`);
  } catch (error) {
    logger.error('An error occurred while starting MyApp.', { error });
    process.abort();
  }
});

function tryConvertEnvVarToNumber(s? : string | number, def?: number): number {
  try {
    return s !== undefined ? (((s as any) instanceof Number) ? +s : parseInt(s as string)) : (def !== undefined ? def : 0)
  } catch (error) {
    logger.error('Unable to convert the environment variable value to a number: %s', s);
    throw new Error(`Unable to convert the environment variable value to a number: '${s}'`)
  }
}

Now, in the code above, particularly within logging.ts inside of initializeLogger, the console transport and the second file transport that is setup is formatted using the StandardUnstructuredFormat formatter, which is a semi-structured plain-text format, I still want to have the timestamp displayed as YYYY-MM-DD HH:mm:ss.SSSZZ. But New Relic’s NodeJS agent completely stomps on my logger’s timestamp formatting by overriding my logger format.

Log Examples

Below are examples of my log output before introducing the NodeJS agent and after.

Plain-text Logs Before Configuring the Node JS Agent:

[2022-05-10 16:14:45.176-0400] [INFO   ] [MyApp                    ] Received a request to enable the deployment target Tentacle agent on Machines-1134 {"meta0":"203e4f37-33d0-4fb2-ad99-18154f79047b"}
[2022-05-10 16:16:02.172-0400] [INFO   ] [DeploymentTargetProcessor] [203e4f37-33d0-4fb2-ad99-18154f79047b] Successfully enabled the Octopus Deploy Tentacle agent on Machines-1134 (ASERVER.STAGE.LOCAL). {"correlationId":"203e4f37-33d0-4fb2-ad99-18154f79047b"}

Plain-text Logs After Configuring the Node JS Agent:

[1653430613504] [INFO   ] [MyApp                    ] Received a request to enable the deployment target Tentacle agent on Machines-395 {"entity.guid":"MjA0NzY0N3xBUE18QVBQTElDQVRJT058MTMxNDU0OTMzMQ","entity.name":"\"MyApp (Test)\"","entity.type":"SERVICE","hostname":"OCTOPUS1","meta0":"7caa8c46-4e61-492f-b48f-f0f9566547f9","original_timestamp":"2022-05-24 18:16:53.504-0400","span.id":"49fdb788465b0000","trace.id":"8ef1c04f9f7d388059f4c9e24208dd43"}
[1653430613986] [INFO   ] [DeploymentTargetProcessor] [7caa8c46-4e61-492f-b48f-f0f9566547f9] Successfully enabled the Octopus Deploy Tentacle agent on Machines-395 (ASERVER.STAGE.LOCAL). {"correlationId":"7caa8c46-4e61-492f-b48f-f0f9566547f9","entity.guid":"MjA0NzY0N3xBUE18QVBQTElDQVRJT058MTMxNDU0OTMzMQ","entity.name":"\"MyApp (Test)\"","entity.type":"SERVICE","hostname":"OCTOPUS1","original_timestamp":"2022-05-24 18:16:53.986-0400","span.id":"49fdb788465b0000","trace.id":"8ef1c04f9f7d388059f4c9e24208dd43"}

Your Environment

Browser name and version: N/A Node version: 14.17.0 and 16.15.0 OS: Windows Server 2019 (17763.2931)

[bizob2828]

@fourpastmidnight this was released yesterday in 8.13.0.

[bizob2828]

@fourpastmidnight just run npm install instead of npm ci. If that doesn’t work you’re going to have to wait for us to release this