Configuring Logstash

You need to configure Logstash to ingest data from Transaction Analysis Workbench.

The starter dashboards require a Logstash pipeline config with the following characteristics:

• Maps the value of the time field in the incoming JSON Lines to the @timestamp field.

  The starter dashboards use @timestamp as the event time stamp.

• Matches Kibana index pattern fuw-%{type}-*

  The starter dashboards use index patterns such as fuw-atf-aes-*.

• Creates data streams (create action) rather than time-based indices.

Here is a starter Logstash pipeline config:

input {
  tcp {
    port => 5046
    codec => json_lines
  }
}
filter {
  date {
    match => ["time", "ISO8601"]
  }
}
output {
  elasticsearch {
    hosts => ["localhost"]

    index => "fuw-%{type}-ds"
	action => "create"
  }
}

This starter config assumes that you have configured Transaction Analysis Workbench to forward data over TCP in JSON Lines format.

This starter config assumes unsecure TCP: no Transport Layer Security (SSL/TLS).

In input.tcp.port, specify the port on which to listen for data from Transaction Analysis Workbench.

TIP

If you deploy the Elastic Stack in Docker containers, then you need to understand the difference between port numbers exposed by the Docker host and port numbers used inside the Docker containers.

In output.elasticsearch.hosts, specify the host name of the computer that is running Elasticsearch.

One data stream per type

The combination of the create action in this starter Logstash config and the data_stream object in the corresponding sample Elasticsearch index template cause Elasticsearch to store Transaction Analysis Workbench data in data streamsopen in new window.

Each record type, as identified by the type field in the incoming JSON Lines, has its own data stream.

Single or multiple Logstash pipelines?

You need to know whether your instance of Logstash is for use only with these starter dashboards or is also used for other purposes, other inputs. Specifically, you need to know whether your use of Logstash involves a single pipeline or multiple pipelines.

If you have installed a new instance of Elastic Stack as a sandbox environment for testing these starter dashboards, then you can use a single Logstash pipeline.

However, if you are using these starter dashboards in an existing instance of Elastic Stack that already has other inputs, then it is more likely that you will need to use multiple pipelines.

Single pipeline

If your instance of Logstash is for use only with these starter dashboards, copy the starter config provided here to the file fuw-tcp-to-local-elasticsearch.conf and save it in the logstash bin folder and run the following command.

logstash -f fuw-tcp-to-local-elasticsearch.conf

Multiple pipelines

For information about configuring multiple pipelines, see the Logstash documentationopen in new window.