Logstash Configuration File Structure

A Logstash configuration file contains three sections for specifying each type of plugin we want to use. The three sections are input, filter, and output. Consider the following example:

input { 
 
      } 
filter { 
 
      } 
output { 
 
      } 

Each section contains the configuration for single or multiple plugins. To configure a plugin, provide the plugin name inside the section, followed by the settings or parameters of that plugin. It is specified in terms of key => value pair. For each section, if you define multiple plugins, then the order of execution is the order in which they appear in the configuration file. Each plugin has its own set of settings, which needs to be used with the plugin name.

Note

The => sign is an assignment operator which assigns the value to the key in the configuration file.

Before going into depth about the various plugins available in each of the sections, let's understand the various value types, which can be defined as the value of the settings in the configuration file, and how to use conditions for conditional statements in the configuration file.

Value types

Every Logstash plugin contains a set of settings to be used. Some settings that are mandatory to specify are marked as required fields. For each setting, we define a value, which is as per the different value types supported by Logstash. The various available value types are discussed in the following sections.

Array

This is a collection of one or more values.

For a single value, the syntax is as follows:

Key => "value" 

For multiple values, the syntax is as follows:

Key => ["value1","value2","value3"] 

If you specify the same settings multiple times in an array, the values get appended to the array, as shown here:

Key => "value" 
Key => "value1" 
Key => ["value2","value3","value4"] 

The key will contain all the five values, that is, value, value1, value2, value3, and value4.

Boolean

This is used to specify the value as either true or false.

For example:

Key => true 
Key1 => false 

Note

The value of a Boolean type, that is, true or false, must not be disclosed in quotes.

Bytes

A byte is a string type field (enclosed in double quotes), which is used to represent a unit of bytes. It uses both the International System of Units (SI Units) (kB, MB, or GB) and binary units (KiB, MiB, or GiB) to calculate the bytes. It is used to define the value followed by the unit. It is case insensitive and also accepts a space between the value of the key and the unit. Also, SI units are based on base-1000, whereas binary units are based on base-1024, that is to say, 1 kB = 1000 bytes, whereas 1 KiB = 1024 bytes.

For example:

size => "2467KiB" 
size => "9872miB" 
Key => "452 GB" 

Note

If no unit is specified, then the value represents the number of bytes.

Key => "1234"

Codec

A codec is not a value type but is used to represent the data. It is used to decode the data coming from the input and encode the data before going to the output. It eliminates the need to have an additional filter to specify how the data is:

codec => "plain" 

Comments

This is not a value type but is used to define a comment in the configuration file. Its syntax is the same as that used in Perl, Python, or Ruby. It is specified by #. It can appear anywhere in the line:

Key => "value" #It is string value type 
#Hope you are learning 

Hash

A hash is a collection of key-value pairs, wherein both the key and the value are specified within double quotes. Multiple entries of key-value pairs are not separated by commas, but are instead separated by spaces:

match => { 
  "field1" => "value1" 
  "field2" => "value2" 
  ... 
} 

Number

A number must contain valid numeric values of either an integer or float type:

number => 44 
amount => 1.28 

String

A string contains a value which can be enclosed either in single or double quotes. If the string value contains the same quote as the string specifier, then it needs to be escaped with a backslash:

name => "yuvraj" 
escape => "value"ue" 
single => 'Hello It's nice you are reading it' 

Use of Conditionals

Conditionals are used to check for conditions based on which action can be taken. In the case of configuration files, we can check for conditions in the plugins based on which settings or configuration will be used. It is handled in a similar way to other programming languages and it supports if, else, and else if statements.

The structure of the configuration file is as follows:

if EXPRESSION { 
  ... 
} else if EXPRESSION { 
  ... 
} else { 
  ... 
} 

Expressions contain operators such as comparison operators, Boolean operators, and unary operators. The comparison operators are subpided into equality operators, regex operators, and inclusion operators:

• Equality operators: Equality operators contain the following list of operators: ==, !=, <, >, <=, and >=
• Regex operators: Regex operators contain the following list of operators: =~ and !~
• Inclusion operators: Inclusion operators contain the following list of operators: in and not in
• Boolean operators: Boolean operators contain the following list of operators: and, or, nand, and xor
• Unary operators: Unary operators contain the following list of operators: !

We will have a look at how to use these operators in the examples set out throughout this chapter.

As described previously, Logstash configuration is pided into multiple sections. Let's have a look at the various plugins available for use within these sections.