ELK (Elastic search ,LogStash ,Kibana)Stack installation script.

Hi Guyes,

I am providing you a script to install single node ELK stack.

Hope you will find it useful.

NOTE- Script will run on debian/ubuntu.

Please find the script below.

 

sudo apt-get update
sudo add-apt-repository -y ppa:webupd8team/java
echo debconf shared/accepted-oracle-license-v1-1 select true | sudo debconf-set-selections
echo debconf shared/accepted-oracle-license-v1-1 seen true | sudo debconf-set-selections
sudo apt-get update
sudo apt-get -y install oracle-java8-installer

############ INSTALL ELASTICSEARCH ##############
wget -qO – https://packages.elastic.co/GPG-KEY-elasticsearch | sudo apt-key add –
echo “deb http://packages.elastic.co/elasticsearch/2.x/debian stable main” | sudo tee -a /etc/apt/sources.list.d/elasticsearch-2.x.list
sudo apt-get update
sudo apt-get -y install elasticsearch
sudo sed -i ‘s/# network.host: 192.168.0.1/network.host: localhost/g’ /etc/elasticsearch/elasticsearch.yml
sudo service elasticsearch restart
sudo update-rc.d elasticsearch defaults 95 10

############### INSTALL KIBANA ####################
echo “deb http://packages.elastic.co/kibana/4.4/debian stable main” | sudo tee -a /etc/apt/sources.list.d/kibana-4.4.x.list
sudo apt-get update
sudo apt-get -y install kibana
sudo sed -i ‘s/# server.host: “0.0.0.0”/server.host: “localhost”/g’ /opt/kibana/config/kibana.yml
sudo update-rc.d kibana defaults 96 9
sudo service kibana start

################## INSTALL NGINX #######################
sudo apt-get -y install nginx apache2-utils
sudo mv /etc/nginx/sites-available/default /etc/nginx/sites-available/default.bk
cat >>/etc/nginx/sites-available/default <<EOF
server {
listen 80;
server_name example.com;
location / {
proxy_pass http://localhost:5601;
proxy_http_version 1.1;
proxy_set_header Upgrade \$http_upgrade;
proxy_set_header Connection ‘upgrade’;
proxy_set_header Host \$host;
proxy_cache_bypass \$http_upgrade;
}
}
EOF

service nginx restart
sudo apt-get update
################################## INSTALL LOGSTASH ####################
echo ‘deb http://packages.elastic.co/logstash/2.2/debian stable main’ | sudo tee /etc/apt/sources.list.d/logstash-2.2.x.list
sudo apt-get update
sudo apt-get -y install logstash

# ##################create certificate to be used by filebeat for forwarding logs to logstash######################
sudo mkdir -p /etc/pki/tls/certs
sudo mkdir /etc/pki/tls/private
ELK_server_private_IP=$(ifconfig eth0 | grep “inet addr:” | cut -d ‘:’ -f2 | cut -d ‘ ‘ -f1)
sed -i “s/v3_ca ]/v3_ca ]\nsubjectAltName = IP: $ELK_server_private_IP/” /etc/ssl/openssl.cnf
cd /etc/pki/tls
sudo openssl req -config /etc/ssl/openssl.cnf -x509 -days 3650 -batch -nodes -newkey rsa:2048 -keyout private/logstash-forwarder.key -out certs/logstash-forwarder.crt

################## create logstash input######################
cat >>/etc/logstash/conf.d/02-beats-input.conf<<EOF
input {
beats {
port => 5044
ssl => true
ssl_certificate => “/etc/pki/tls/certs/logstash-forwarder.crt”
ssl_key => “/etc/pki/tls/private/logstash-forwarder.key”
}
}
EOF

####################create logstash filter#######################
cat >> /etc/logstash/conf.d/10-syslog-filter.conf<<EOF
filter {
if [type] == “syslog” {
grok {
match => { “message” => “%{SYSLOGTIMESTAMP:syslog_timestamp} %{SYSLOGHOST:syslog_hostname} %{DATA:syslog_program}(?:\[%{POSINT:syslog_pid}\])?: %{GREEDYDATA:syslog_message}” }
add_field => [ “received_at”, “%{@timestamp}” ]
add_field => [ “received_from”, “%{host}” ]
}
syslog_pri { }
date {
match => [ “syslog_timestamp”, “MMM d HH:mm:ss”, “MMM dd HH:mm:ss” ]
}
}
}
EOF

################ create logstash output###############################
cat>> /etc/logstash/conf.d/30-elasticsearch-output.conf<<EOF
output {
elasticsearch {
hosts => [“localhost:9200”]
sniffing => true
manage_template => false
index => “%{[@metadata][beat]}-%{+YYYY.MM.dd}”
document_type => “%{[@metadata][type]}”
}
}
EOF

sudo service logstash configtest
sudo service logstash restart
sudo update-rc.d logstash defaults 96 9

###################### install filebeat dashboard########################
cd /tmp
curl -L -O https://download.elastic.co/beats/dashboards/beats-dashboards-1.1.0.zip
sudo apt-get -y install unzip
unzip beats-dashboards-*.zip
cd beats-dashboards-*
./load.sh

############################# install filebeat template###################
cd /tmp/
curl -O https://gist.githubusercontent.com/thisismitch/3429023e8438cc25b86c/raw/d8c479e2a1adcea8b1fe86570e42abab0f10f364/filebeat-index-template.json
curl -XPUT ‘http://localhost:9200/_template/filebeat?pretty&#8217; -d@filebeat-index-template.json

########TODO remove filebeat installation and configuration as this will be done on the server from which we want to forward the logs to this server######
######## install filebeat on the same server for testing##############
echo “deb https://packages.elastic.co/beats/apt stable main” | sudo tee -a /etc/apt/sources.list.d/beats.list
wget -qO – https://packages.elastic.co/GPG-KEY-elasticsearch | sudo apt-key add –
sudo apt-get update
sudo apt-get -y install filebeat
sed -i ‘s/#document_type: log/document_type: syslog/g’ /etc/filebeat/filebeat.yml
sed -i ‘s/#logstash/logstash/g’ /etc/filebeat/filebeat.yml
sed -i “s/#hosts: \[\”localhost:5044\”\]/hosts: [\”$ELK_server_private_IP:5044\”]\n bulk_max_size: 1024/g” /etc/filebeat/filebeat.yml
sed -i ‘s/#tls:/tls:/g’ /etc/filebeat/filebeat.yml
sed -i ‘s!#certificate_authorities: \[“/etc/pki/root/ca.pem”\]!certificate_authorities: [“/etc/pki/tls/certs/logstash-forwarder.crt”]!g’ /etc/filebeat/filebeat.yml
sudo service filebeat restart
sudo update-rc.d filebeat defaults 95 10

How check the ELK stack is installed properly or not.-

In a web browser, go to the FQDN or public IP address of your ELK Server. After entering  you should see a page prompting you to configure a default index pattern.

 

 

 

In VI/VIM editor,Make changes in file without root user even if file has root permissions.

Hi Guyes,

This is very common problem that every software engineer has may suffered since from student level.

Problem Statement-

In vi/vim editor,Suppose you have created a file and saved it using root user.
And now there is a situation that you have opened this file using without root user and you are trying to write in file and save it.

Then we get an error like file is readonly or you can not save it.

what normal people do –

Normally people quit the file without saving(as they not able to save the file).
(:q!) and they change the user to root user / change the permission of file,open the file and write in it and save it.

But as we are software engineer, we have habit to forgot, and in a scenario mentioned in above problem statement we dont want to do the stuff as normal people does or want to save the time and efforts (haha).

What this blog reader do in the above senario(solution)-

(:w !sudo tee %)

esacpe : !sudo tee % and enter ….by doing this you will able to save the changes in the file. And you have to do :q! to exit from the files