- SAP Managed Tags:
In this article, I will be demonstrating the installation of opensource Elastic search, Kibana, Logstash plugin and how to stream the log data of a build of a Job in a Jenkins pipeline.
Logstash plugin is used to stream the log data from Jenkins instance to the any indexer. In the current scenario, i would be using Elastic search. Hence, the build logs for a job can be streamed to the elastic search and eventually, can be visualized in the Kibana dashbaord.
Prerequisities:
Install an opensource helm chart for elastic search from the following url Elastic search.
Similarly, install the helm chart for Kibana from here. Ensure, to configure the 'elasticsearchHosts' and 'elasticsearchURL' while installing the Kibana. Those two parameters refers to the elastic search instance url and it's hostname. These two parameters can be set during the command input for instance as shown below:
Lastly, it is required to install the logstash plugin in our Jenkins instance. To do so, login into the Jenkins instance -> Manage Jenkins -> Manage plugins as shown below
Manage plugins-> Available plugins -> Logstash plugin
After installing the plugin, you should be able to see it in the installed plugins under manage plugins.
Configuration:
After successfully installing the plugin, in the global configuration of the Jenkins instance, we need to configure the logstash plugin with the indexer (in the current scenario elastic search endpoint) and the credentials to access it if required. Please follow the below steps to do the configuration:
In the above image, configure the uri, where the elastic search is running.Moreover, define the indexer name followed by the the instance type for example(logstashtest/jenkins). In case, if it is a protected url, enter the credentials for username and password and save the configuration.
Logstash working:
After configuring the elastic search endpoint in the logstash configuration in the jenkins, create a test pipeline with the logstash step to see how the logs inside the logstash step would be send to the elastic search indexer. Sample pipeline looks like the following:
from the above pipeline, all the logs produced inside the logstash step, will be streamed to the elastic search endpoint.
The logs streamed to the elastic search can be viewed by accessing the elastic search endpoint. You can see if the elastic search is up by accessing
The list of indexers can be viewed with the following command:
In my case, I have configured the indexer with the name "logstashtest" , Hence I could see all the logs of the build under this indexer by accessing the following command :
And the event payload data will look like the following:
Finally, the required parameters from the event payload can be extracted using cat API of the elastic search.
Similarly, the required logs in the Kibana can be visualized after accessing to the following url:
and the logs would look in the following manner:
The required parameters and required logs can be extracted using the filters in Kibana.
Ensure to configure the indexer pattern in the Kibana, before analyzing the logs. It is recommended to use the type "logstash*" for the pattern. As we have already configured our indexer name as "logstashtest", Hence, now we are able to stream the log data from Jenkins instance to elastic search and finally to the Kibana.
Logstash plugin is used to stream the log data from Jenkins instance to the any indexer. In the current scenario, i would be using Elastic search. Hence, the build logs for a job can be streamed to the elastic search and eventually, can be visualized in the Kibana dashbaord.
Prerequisities:
Install an opensource helm chart for elastic search from the following url Elastic search.
Similarly, install the helm chart for Kibana from here. Ensure, to configure the 'elasticsearchHosts' and 'elasticsearchURL' while installing the Kibana. Those two parameters refers to the elastic search instance url and it's hostname. These two parameters can be set during the command input for instance as shown below:
helm install --name Kibana elastic/kibana --set elasticsearchHosts="http://elasticsearch-master:9200" --set elasticsearchURL="http://elasticsearch-master:9200"Lastly, it is required to install the logstash plugin in our Jenkins instance. To do so, login into the Jenkins instance -> Manage Jenkins -> Manage plugins as shown below
Manage plugins-> Available plugins -> Logstash plugin
After installing the plugin, you should be able to see it in the installed plugins under manage plugins.
Configuration:
After successfully installing the plugin, in the global configuration of the Jenkins instance, we need to configure the logstash plugin with the indexer (in the current scenario elastic search endpoint) and the credentials to access it if required. Please follow the below steps to do the configuration:
In the above image, configure the uri, where the elastic search is running.Moreover, define the indexer name followed by the the instance type for example(logstashtest/jenkins). In case, if it is a protected url, enter the credentials for username and password and save the configuration.
Logstash working:
After configuring the elastic search endpoint in the logstash configuration in the jenkins, create a test pipeline with the logstash step to see how the logs inside the logstash step would be send to the elastic search indexer. Sample pipeline looks like the following:
pipeline{
agent none
stages {
stage("first"){
steps {
timestamps {
logstash{
echo "hello world 1"
}
}
}
}
stage("second"){
steps{
timestamps {
logstash {
echo "hello world 2"
}
}
}
}
}
}from the above pipeline, all the logs produced inside the logstash step, will be streamed to the elastic search endpoint.
The logs streamed to the elastic search can be viewed by accessing the elastic search endpoint. You can see if the elastic search is up by accessing
http://localhost:9200The list of indexers can be viewed with the following command:
http://localhost:9200/_cat/indicesIn my case, I have configured the indexer with the name "logstashtest" , Hence I could see all the logs of the build under this indexer by accessing the following command :
http://localhost:9200/logstashtest/_search?prettyAnd the event payload data will look like the following:
{
"took" : 2,
"timed_out" : false,
"_shards" : {
"total" : 5,
"successful" : 5,
"skipped" : 0,
"failed" : 0
},
"hits" : {
"total" : 1,
"max_score" : 1.0,
"hits" : [
{
"_index" : "logstashtest",
"_type" : "jenkins",
"_id" : "uoOWWWoB3m4VW96z1LI3",
"_score" : 1.0,
"_source" : {
"data" : {
"id" : "4",
"projectName" : "test",
"fullProjectName" : "test",
"displayName" : "#4",
"fullDisplayName" : "test #4",
"url" : "job/test/4/",
"buildHost" : "Jenkins",
"buildLabel" : "master",
"buildNum" : 4,
"buildDuration" : 478,
"rootProjectName" : "test",
"rootFullProjectName" : "test",
"rootProjectDisplayName" : "#4",
"rootBuildNum" : 4,
"buildVariables" : {
"BUILD_DISPLAY_NAME" : "#4",
"BUILD_ID" : "4",
"BUILD_NUMBER" : "4",
"BUILD_TAG" : "jenkins-test-4",
"BUILD_URL" : "https://test.sap.com/job/test/4/",
"CLASSPATH" : "",
"HUDSON_HOME" : "/var/jenkins_home",
"HUDSON_SERVER_COOKIE" : "240b1176c4e3f0fd",
"HUDSON_URL" : "https://test.sap.com/",
"JENKINS_HOME" : "/var/jenkins_home",
"JENKINS_SERVER_COOKIE" : "240b1176c4e3f0fd",
"JENKINS_URL" : "https://test.sap.com/",
"JOB_BASE_NAME" : "test",
"JOB_DISPLAY_URL" : "https://test.sap.com/job/test/display/redirect",
"JOB_NAME" : "test",
"JOB_URL" : "https://test.sap.com/job/test/",
"RUN_CHANGES_DISPLAY_URL" : "https://test.sap.com/job/test/4/display/redirect?page=changes",
"RUN_DISPLAY_URL" : "https://test.sap.com/job/test/4/display/redirect"
}
},
"message" : [
"hello world 2"
],
"source" : "jenkins",
"source_host" : "https://test.sap.com/",
"@buildTimestamp" : "2019-04-26T12:20:17.393+0000",
"@timestamp" : "2019-04-26T12:20:17.877+0000",
"@version" : 1
}
}
]
}
}Finally, the required parameters from the event payload can be extracted using cat API of the elastic search.
Similarly, the required logs in the Kibana can be visualized after accessing to the following url:
http://localhost:5601and the logs would look in the following manner:
The required parameters and required logs can be extracted using the filters in Kibana.
Ensure to configure the indexer pattern in the Kibana, before analyzing the logs. It is recommended to use the type "logstash*" for the pattern. As we have already configured our indexer name as "logstashtest", Hence, now we are able to stream the log data from Jenkins instance to elastic search and finally to the Kibana.
5 Comments
