ELK (Elastic search ,LogStash ,Kibana)Stack installation script.

Hi Guyes,

I am providing you a script to install single node ELK stack.

Hope you will find it useful.

NOTE- Script will run on debian/ubuntu.

Please find the script below.

 

sudo apt-get update
sudo add-apt-repository -y ppa:webupd8team/java
echo debconf shared/accepted-oracle-license-v1-1 select true | sudo debconf-set-selections
echo debconf shared/accepted-oracle-license-v1-1 seen true | sudo debconf-set-selections
sudo apt-get update
sudo apt-get -y install oracle-java8-installer

############ INSTALL ELASTICSEARCH ##############
wget -qO – https://packages.elastic.co/GPG-KEY-elasticsearch | sudo apt-key add –
echo “deb http://packages.elastic.co/elasticsearch/2.x/debian stable main” | sudo tee -a /etc/apt/sources.list.d/elasticsearch-2.x.list
sudo apt-get update
sudo apt-get -y install elasticsearch
sudo sed -i ‘s/# network.host: 192.168.0.1/network.host: localhost/g’ /etc/elasticsearch/elasticsearch.yml
sudo service elasticsearch restart
sudo update-rc.d elasticsearch defaults 95 10

############### INSTALL KIBANA ####################
echo “deb http://packages.elastic.co/kibana/4.4/debian stable main” | sudo tee -a /etc/apt/sources.list.d/kibana-4.4.x.list
sudo apt-get update
sudo apt-get -y install kibana
sudo sed -i ‘s/# server.host: “0.0.0.0”/server.host: “localhost”/g’ /opt/kibana/config/kibana.yml
sudo update-rc.d kibana defaults 96 9
sudo service kibana start

################## INSTALL NGINX #######################
sudo apt-get -y install nginx apache2-utils
sudo mv /etc/nginx/sites-available/default /etc/nginx/sites-available/default.bk
cat >>/etc/nginx/sites-available/default <<EOF
server {
listen 80;
server_name example.com;
location / {
proxy_pass http://localhost:5601;
proxy_http_version 1.1;
proxy_set_header Upgrade \$http_upgrade;
proxy_set_header Connection ‘upgrade’;
proxy_set_header Host \$host;
proxy_cache_bypass \$http_upgrade;
}
}
EOF

service nginx restart
sudo apt-get update
################################## INSTALL LOGSTASH ####################
echo ‘deb http://packages.elastic.co/logstash/2.2/debian stable main’ | sudo tee /etc/apt/sources.list.d/logstash-2.2.x.list
sudo apt-get update
sudo apt-get -y install logstash

# ##################create certificate to be used by filebeat for forwarding logs to logstash######################
sudo mkdir -p /etc/pki/tls/certs
sudo mkdir /etc/pki/tls/private
ELK_server_private_IP=$(ifconfig eth0 | grep “inet addr:” | cut -d ‘:’ -f2 | cut -d ‘ ‘ -f1)
sed -i “s/v3_ca ]/v3_ca ]\nsubjectAltName = IP: $ELK_server_private_IP/” /etc/ssl/openssl.cnf
cd /etc/pki/tls
sudo openssl req -config /etc/ssl/openssl.cnf -x509 -days 3650 -batch -nodes -newkey rsa:2048 -keyout private/logstash-forwarder.key -out certs/logstash-forwarder.crt

################## create logstash input######################
cat >>/etc/logstash/conf.d/02-beats-input.conf<<EOF
input {
beats {
port => 5044
ssl => true
ssl_certificate => “/etc/pki/tls/certs/logstash-forwarder.crt”
ssl_key => “/etc/pki/tls/private/logstash-forwarder.key”
}
}
EOF

####################create logstash filter#######################
cat >> /etc/logstash/conf.d/10-syslog-filter.conf<<EOF
filter {
if [type] == “syslog” {
grok {
match => { “message” => “%{SYSLOGTIMESTAMP:syslog_timestamp} %{SYSLOGHOST:syslog_hostname} %{DATA:syslog_program}(?:\[%{POSINT:syslog_pid}\])?: %{GREEDYDATA:syslog_message}” }
add_field => [ “received_at”, “%{@timestamp}” ]
add_field => [ “received_from”, “%{host}” ]
}
syslog_pri { }
date {
match => [ “syslog_timestamp”, “MMM d HH:mm:ss”, “MMM dd HH:mm:ss” ]
}
}
}
EOF

################ create logstash output###############################
cat>> /etc/logstash/conf.d/30-elasticsearch-output.conf<<EOF
output {
elasticsearch {
hosts => [“localhost:9200”]
sniffing => true
manage_template => false
index => “%{[@metadata][beat]}-%{+YYYY.MM.dd}”
document_type => “%{[@metadata][type]}”
}
}
EOF

sudo service logstash configtest
sudo service logstash restart
sudo update-rc.d logstash defaults 96 9

###################### install filebeat dashboard########################
cd /tmp
curl -L -O https://download.elastic.co/beats/dashboards/beats-dashboards-1.1.0.zip
sudo apt-get -y install unzip
unzip beats-dashboards-*.zip
cd beats-dashboards-*
./load.sh

############################# install filebeat template###################
cd /tmp/
curl -O https://gist.githubusercontent.com/thisismitch/3429023e8438cc25b86c/raw/d8c479e2a1adcea8b1fe86570e42abab0f10f364/filebeat-index-template.json
curl -XPUT ‘http://localhost:9200/_template/filebeat?pretty&#8217; -d@filebeat-index-template.json

########TODO remove filebeat installation and configuration as this will be done on the server from which we want to forward the logs to this server######
######## install filebeat on the same server for testing##############
echo “deb https://packages.elastic.co/beats/apt stable main” | sudo tee -a /etc/apt/sources.list.d/beats.list
wget -qO – https://packages.elastic.co/GPG-KEY-elasticsearch | sudo apt-key add –
sudo apt-get update
sudo apt-get -y install filebeat
sed -i ‘s/#document_type: log/document_type: syslog/g’ /etc/filebeat/filebeat.yml
sed -i ‘s/#logstash/logstash/g’ /etc/filebeat/filebeat.yml
sed -i “s/#hosts: \[\”localhost:5044\”\]/hosts: [\”$ELK_server_private_IP:5044\”]\n bulk_max_size: 1024/g” /etc/filebeat/filebeat.yml
sed -i ‘s/#tls:/tls:/g’ /etc/filebeat/filebeat.yml
sed -i ‘s!#certificate_authorities: \[“/etc/pki/root/ca.pem”\]!certificate_authorities: [“/etc/pki/tls/certs/logstash-forwarder.crt”]!g’ /etc/filebeat/filebeat.yml
sudo service filebeat restart
sudo update-rc.d filebeat defaults 95 10

How check the ELK stack is installed properly or not.-

In a web browser, go to the FQDN or public IP address of your ELK Server. After entering  you should see a page prompting you to configure a default index pattern.

 

 

 

Advertisements

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s