Technology Sharing

한어Русский языкEnglishFrançaisIndonesianSanskrit日本語DeutschPortuguêsΕλληνικάespañolItalianoSuomalainenLatina

This is a follow-up to the previous article "Elasticsearch: Node.js ECS Logging - Pino" This is a sequel to the previous article. We will continue to talk about using the Winston package to target Node.js The application generates ECS to match the day. This Node.js package iswinston loggerProvides a formatter that is compatible with Elastic Common Schema (ECS) loggingCompatible. Combined Filebeat With this shipper, you can monitor all logs in Elastic Stack in one place. Supports winston 3.x versions >= 3.3.3.

set up

Install

npm install @elastic/ecs-winston-format
npm install winston

Configuration

winston-logging.js

const winston = require('winston');
const { ecsFormat } = require('@elastic/ecs-winston-format');
 
const logger = winston.createLogger({
  format: ecsFormat(/* options */),  // 1
  transports: [
    new winston.transports.Console()
  ]
});
 
logger.info('hi');
logger.error('oops there is a problem', { err: new Error('boom') });

• Pass the ECS formatter to winston.

The result of running the above code is:

Configure Filebeat

Filebeat 7.16+

filebeat.yml

filebeat.inputs:
- type: filestream     # 1
  paths: /path/to/logs.json
  parsers:
    - ndjson:
      overwrite_keys: true # 2
      add_error_key: true  # 3
      expand_keys: true    # 4
 
processors: // 5
  - add_host_metadata: ~
  - add_cloud_metadata: ~
  - add_docker_metadata: ~
  - add_kubernetes_metadata: ~

1. Use filestream input to read lines from the active log file.
2. If a conflict occurs, the values ​​of the decoded JSON object will overwrite the fields that Filebeat would normally add (type, source, offset, etc.).
3. If a JSON unmarshaling error occurs, Filebeat adds the "error.message" and "error.type: json" keys.
4. Filebeat will recursively strip dot keys from the decoded JSON and expand it into a hierarchical object structure.
5. Processors enhance your data. See processors for more information.

Filebeat < 7.16

filebeat.yml

filebeat.inputs:
- type: log
  paths: /path/to/logs.json
  json.keys_under_root: true
  json.overwrite_keys: true
  json.add_error_key: true
  json.expand_keys: true
 
processors:
- add_host_metadata: ~
- add_cloud_metadata: ~
- add_docker_metadata: ~
- add_kubernetes_metadata: ~

For more information, see Filebeat reference。

how to use

winston-logging.js

const winston = require('winston');
const { ecsFormat } = require('@elastic/ecs-winston-format');
 
const logger = winston.createLogger({
  level: 'info',
  format: ecsFormat(/* options */), // 1
  transports: [
    new winston.transports.Console()
  ]
});
 
logger.info('hi');
logger.error('oops there is a problem', { foo: 'bar' });

1. See below for available options.

node winston-logging.js | jq .

Run this script (available inHerewill produce log output similar to the above.

The formatter takes care of serializing the data into JSON, so you don't need to add json In addition, the formatter automatically generates the timestamp, so you don't need to addtimestamp Formatter.

Error logging

By default, the formatter converts the err metafield of Error instances to ECS Error Field.For exampleexample：

winston-logging.js

const winston = require('winston');
const { ecsFormat } = require('@elastic/ecs-winston-format');
const logger = winston.createLogger({
  format: ecsFormat(), 
  transports: [
    new winston.transports.Console()
  ]
});
 
const myErr = new Error('boom');
logger.info('oops', { err: myErr }); 

Special handling of the err meta field can be disabled with the convertErr: false option:

winston-logging.js

const winston = require('winston');
const { ecsFormat } = require('@elastic/ecs-winston-format');
const logger = winston.createLogger({
  format: ecsFormat({convertErr: false} ), 
  transports: [
    new winston.transports.Console()
  ]
});
 
const myErr = new Error('boom');
logger.info('oops', { err: myErr }); 

HTTP request and response logging

With the convertReqRes: true option, the formatter will automatically convert Node.js core request andresponse object.

winston-logging.js

const http = require('http');
const winston = require('winston');
const { ecsFormat } = require('@elastic/ecs-winston-format');
 
const logger = winston.createLogger({
  level: 'info',
  format: ecsFormat({ convertReqRes: true }), // 1
  transports: [
    new winston.transports.Console()
  ]
});
 
const server = http.createServer(handler);
server.listen(3000, () => {
  logger.info('listening at http://localhost:3000')
});
 
function handler (req, res) {
  res.setHeader('Foo', 'Bar');
  res.end('ok');
  logger.info('handled request', { req, res }); // 2
}

1. Using the convertReqRes option
2. Record the req and/or res meta fields

This will use ECS HTTP fieldsGenerates a log containing request and response information. For exampleexample：

On top, we need to access http://localhost:3000 To see the day information as shown above.

Log correlation using APM

this ECS Log formatter withElastic APM Integration. If your Node application is usingNode.js Elastic APM Agent, multiple fields are added to the log record to correlate APM service or trace and log data:

• Logging statements (such as logger.info(...) ) called while the current tracing span is in progress will includeTracking Fields — trace.id、transaction.id、span.id。
• Several service identifier fields determined by or configured on the APM agent allow for cross-linking between services and logs in Kibana — service.name, service.version, service.environment, service.node.name.
• Enable event.dataset in your Elastic Observability applicationLog rate anomaly detection。

For example, running examples/http-with-elastic-apm.js andcurl -i localhost:3000/ Will produce log records containing the following:

% node examples/http-with-elastic-apm.js | jq .
...
  "service.name": "http-with-elastic-apm",
  "service.version": "1.4.0",
  "service.environment": "development",
  "event.dataset": "http-with-elastic-apm"
  "trace.id": "7fd75f0f33ff49aba85d060b46dcad7e",
  "transaction.id": "6c97c7c1b468fa05"
}

These IDs are matched with the trace data reported by the APM agent.

Integration with Elastic APM can be explicitly disabled via the apmIntegration: false option, for example:

const logger = winston.createLogger({
  format: ecsFormat({ apmIntegration: false }),
  // ...
})

Limitations and considerations

ecs-logging SpecificationIt is recommended that the first three fields in a log record should be @timestamp, log.level, and message. As of version 1.5.0, this formatter does not follow this recommendation. This is possible but requires creating a new object in ecsFields for each log record. Given that the ordering of ecs-logging fields is for readability and does not affect interoperability, the decision was made to prioritize performance.

refer to

ecsFormat([options])

• options {type-object} supports the following options:
  • convertErr {type-boolean} Whether to convert the err field of the record to the ECS error field.default value：true。
  • convertReqRes {type-boolean} Whether to log the req and res HTTP request and response fields to the ECS HTTP, user-agent, and URL fields.default value：false。
  • apmIntegration {type-boolean} Whether to enable APM agent integration.default value：true。
  • serviceName {type-string} “service.name” value. If specified, overrides any value from the active APM agent.
  • serviceVersion {type-string} "service.version" value. If specified, overrides any value from the active APM agent.
  • serviceEnvironment {type-string} "service.environment" value. If specified, overrides any value from the active APM agent.
  • serviceNodeName {type-string} “service.node.name” value. If specified, will override any value from the active APM agent.
  • eventDataset {type-string} "event.dataset" value. If specified, will override the default of using ${serviceVersion}.

Creates a formatter for winston that emits in the ECS logging format. This is a process ecsFields([options]) andecsStringify([options]) The following two are equivalent:

const { ecsFormat, ecsFields, ecsStringify } = require('@elastic/ecs-winston-format');
const winston = require('winston');
 
const logger = winston.createLogger({
  format: ecsFormat(/* options */),
  // ...
});
 
const logger = winston.createLogger({
  format: winston.format.combine(
    ecsFields(/* options */),
    ecsStringify()
  ),
  // ...
});

ecsFields([options])

• options {type-object} supports the following options:
  • convertErr {type-boolean} Whether to convert the err field of the record to the ECS error field.default value：true。
  • convertReqRes {type-boolean} Whether to log the req and res HTTP request and response fields to the ECS HTTP, user-agent, and URL fields.default value：false。
  • apmIntegration {type-boolean} Whether to enable APM agent integration.default value：true。
  • serviceName {type-string} “service.name” value. If specified, overrides any value from the active APM agent.
  • serviceVersion {type-string} "service.version" value. If specified, overrides any value from the active APM agent.
  • serviceEnvironment {type-string} "service.environment" value. If specified, overrides any value from the active APM agent.
  • serviceNodeName {type-string} “service.node.name” value. If specified, will override any value from the active APM agent.
  • eventDataset {type-string} "event.dataset" value. If specified, will override the default of using ${serviceVersion}.

Create a formatter for winston that converts the fields on the logging info object to the ECS logging format.

ecsStringify([options])

Create a formatter for winston that stringifies/serializes log records into JSON.

This is similar to logform.json(). They both use the safe-stable-stringify package to generate JSON. Some differences:

• This stringifier skips serializing the level field because it is not an ECS field.
• Winston provides a replacer that converts a bigint to a string. The rationale for this is that the JavaScript JSON parser loses precision when parsing a bigint. The deprecation is that BigInt changes the type to a string instead of a number. Currently, this stringifier does not convert a BitInt to a string.