ELK is the group of three open source projects in Linux.  Elastic Search, Logstash and Kibana respectively. Before going into depth let’s have a short definition about these:

Elasticsearch:

This is an open source distribution, reliable, scalable, easy to use and flexible Lucene library based search engine. It provides multitenant-capable text with an HTTP web interface.

Logstash:

It is an open source tool used to store data, collect information, and store it for further use. Kibana is used to retrieve the logs stored by Logstash.

Kibana:

Let’s you transform your data into your own format or specific shape like charts and graphs in Elasticsearch. 

So, in this article we will cover the following :

• How to Install Java on Centos 8

• How to add ELK repository to Centos 8

• How to install and Configure Elasticsearch 

• How to instaall and configure Kibana on Centos 8

• How to install and configure Logstash on Centos 8

• How to install other ELK tools(Optional)

Step 1: Install Java on Centos 8

Before installing Elasticsearch we must have java installed on our system as Elasticsearch depends on java.  So install it before further proceeding.

How to install Java 11 (OpenJdk 11 on RHEL / Centos 8

Step 2: Add ELK repository to Centos 8

After installing java, add ELK repository  to Centos 8 and run the following command as Sudo.

For Elasticsearch 7.x 

cat <<EOF | sudo tee /etc/yum.repos.d/elasticsearch.repo

[elasticsearch -7.x]

name=Elasticsearch repository for 7.x packages

baseurl=https://artifacts.elasticsearch.co/packages/7.x/yum

gpgcheck=1

gpgkey=https://artifacts.elastic.co/GPG-KEY-elasticse

enabled=1

autorefresh=1

type=rpm-md

EOF

For Elasticsearch 6.x

cat <<EOF | sudo tee /etc/yum.repos.d/elasticsearch.repo

[elasticsearch -6.x]

name=Elasticsearch repository for 6.x packages

baseurl=https://artifacts.elasticsearch.co/packages/6.x/yum

gpgcheck=1

gpgkey=https://artifacts.elastic.co/GPG-KEY-elasticse

enabled=1

autorefresh=1

type=rpm-md

EOF

For Elasticsearch 5.x

cat <<EOF | sudo tee /etc/yum.repos.d/elasticsearch.repo

[elasticsearch -5.x]

name=Elasticsearch repository for 5.x packages

baseurl=https://artifacts.elasticsearch.co/packages/5.x/yum

gpgcheck=1

gpgkey=https://artifacts.elastic.co/GPG-KEY-elasticse

enabled=1

autorefresh=1

type=rpm-md

EOF

After doing so, import GPG key

sudo rpm –import https://artifacts.elastic.co/GPG-KEY-elasticsearch

Now,  clear and update your YUM package index. 

sudo yum clean all

sudo yum makecache

Step 3: Install and Configure Elasticsearch

As we have done with Elasticsearch repository and now it’s ready to use. Now make sure to run this command to install Elasticsearch.

sudo yum -y install elasticsearch

Double-check that installation completed successfully. 

rpm -qi elasticsearch

Set up the JVM options like memory limits and others according to your own needs. For this edit the following file:

Here we have set up maximum size of total heap space.

/etc/elasticsearch/jvm.options

You can adjust according to your system requirements. 

Now,  start and enable the Elasticsearch services.

Make sure these are properly working. 

Let’s create a test index.

curl -X PUT “http://127.0.0.1:9200/mytest_index”

Step 4: Install / Configure Kibana on Centos 8

From added Elasticsearch repository download and install kibana. 

sudo yum -y install kibana

Configure it after installation completed. 

sudo vim /etc/kibana/kibana.yml
server.host: “0.0.0.0“
server.name: “kibana.example.com”
elasticsearch.url: “http://localhost:9200“

Set up other settings to your own requirements and start kibana services.

sudo systemctl enable –now kibana

Visit http://ip-address:5601 to open kibana dashboard 

If you have firewall service active make sure to allow TCP port 5601.

sudo firewall-cmd –add-port=5601/tcp –permanent
sudo firewall-cmd –reload

Step 4: Install / Configure Logstash on Centos 8

The last step is to install and configure Logstash which will act like a centralized logs server for your client systems and runs an agent like filebeat.

sudo yum -y install logstash

Customize settings under the following directory: /etc/logstash/conf.d/ For further information you can check out Logstash configuration manual. 

Step 5: Install other ELK tools – (optional) 

Some of these tools help you to work smoothly.

Filebeat:

It makes things simple by following lightweight way to forward and centralized logs and files. 

Metricbeat:

Helps you to send and collect metrics from your systems and services, from CPU to memory,  Redis to NGINX,  and many more.  It’s also a lightweight way to access system and services statistics.

Packetbeat:

Packetbeat provides a lightweight way for Network Data to increase performance.

Heartbeat:

Monitors the up time of Services. Helps you to know Availability of services. 

Auditbeat:

Useful for auditing the activities and processes on your system by users. The tools we have discussed so far can be installed with the give command one time or you can install individually by this command.

sudo yum install filebeat auditbeat metricbeat packetbeat heartbeat-elastic

These add-on tools help you better experience.To configure any tool you can check  official ELK stack documentation.

Hope you are all done!  If have any queries regarding this tutorial leave a comment!

The post How to Install ELK Stack on Centos 8? appeared first on Linux Windows and android Tutorials.

• How to setup Elasaticsearch Node.
• What options do we have to ingest data into Elasticsearch database.
• Elasticsearch uses an entity called “INDEX” to store data.
• Every message that’s gets stored is a “Document” at Elasticsearch.
• And even we know that using Kibana it is possible to view those data back for analysis.

In this post, I am going to cover the native Query language that Elasticsearch use to search data. It is DSL (Domain Specific Language).

REST API:

9200/tcp is one of the network communication socket that Elasticsearch use. And when running, Elasticsearch expose its REST API on this port for external communication and that is what we can connect with to perform these DSL queries. This API is build on top of HTTP protocol, so its aware any http calls, for example, GET, POST, PUT..

If you are wondering on which client that will support executing these queries, well, there are 3rd party applications, such as “Postman”. However, since we already have Kibana Installed, let’s check that on how to access the REST API.

• Since Kibana is aware the Elasticsearch node url, it is not required to mention in the query itself, like if you are working on 3rd party tool.

 match_all Query:

If we don’t know what to search, but we need to see every document inside of a INDEX, this query comes handy.

GET 192.168.0.1:9200/index_name/_search

{
  "size": 1,
  "query": {
    "match_all": {}
  }
}

• index_name => on which index that perform the query
• _search          => one of the API endpoint that elasticsearch exposes. _search of course is what provide the search facility.
• {}                      => everything inside of the curly braces will be HTTP body
• size                 => number of matched document that need to filter out.
• match_all     => this is a catch all statement where anything inside the “index_name” INDEX we would be able to see.

exists Query:

We know that Elasticsearch is a Json store, so if we know the exact FieldName that the document we are searching upon, this query comes handy.

GET 192.168.0.1:9200/index_name/_search

{
  "query": {
    "exists": {
      "field": "firstName"
    }
  }
}

• field                      => We are searching any document inside the “index_name” INDEX, but each document should have a filed named called “firstName”

term Query:

This query is best when If we know the exact field name and we also want to limit the search result only on documents that have filed value of exactly what we are searching.

GET 192.168.0.1:9200/index_name/_search

{
  "query": {
    "term": {
      "userName": "john"
    }
  }
}

Note that this query should be perform on Fields within a document that has single term, not on Fields that has many terms inside of it, for example;
“userName”: “don john mcalister”

match Query:

If “term” query is not best for search multiple terms, then which query is best of doing the task. This is where match query comes into play which does full-text search at it’s best.

GET 192.168.0.1:9200/index_name/_search

{
  "query": {
    "match": {
      "userCommnet": "Sri Lanka beautiful country"
    }
  }
}

Note that even in the actual document’s “userComment“:  contains “Sri Lanka is one of the best beautiful countries in the world” the above query get you that document out of the result search result. 

query_string Query:

If you are looking at AND/OR operators to match document, this would do it for you.

GET 192.168.0.1:9200/index_name/_search

{
  "query": {
    "query_string": {
      "query": "city:(new york) OR (new-york)"
    }
  }
}

Note that “city” is the field name & rest will be the search text that we are looking.. 

For basic operations, these type of queries are best for getting task done, but like any other database query, DSL also has many powerful options that leverage to narrow down the search result as to the best of requirement. Once you get these basics covered, you can comment on more for such advanced queries. 

“I hope this has been informative for you.”

The post Elasticsearch DSL Query Examples appeared first on Linux Windows and android Tutorials.