转载

使用Elasticsearch + Logstash + Kibana搭建日志集中分析平台实践

在上周的上海Gopher Meetup的聚会上，听了ASTA谢的演讲。然后公司最近也需要实现一个日志集中分析平台。ASTA谢恰好也讲了他使用了Elasticsearch + Logstash + Kibana这个组合进行日志分析。回来之后就买了一本书然后各种google把它配置好了，当然只是把框架搭好了。这三个组建还有很多功能并没有熟悉。本文只是简单的介绍在Centos如果配置ELK(因为公司的服务器是Centos的,个人比较喜欢Ubuntu 哈哈)

什么是ELK：

Elasticsearch + Logstash + Kibana（ELK）是一套开源的日志管理方案，分析网站的访问情况时我们一般会借助Google/百度/CNZZ等方式嵌入JS做数据统计，但是当网站访问异常或者被攻击时我们需要在后台分析如Nginx的具体日志，而Nginx日志分割/GoAccess/Awstats都是相对简单的单节点解决方案，针对分布式集群或者数据量级较大时会显得心有余而力不足，而ELK的出现可以使我们从容面对新的挑战。

Logstash：负责日志的收集，处理和储存
Elasticsearch：负责日志检索和分析
Kibana：负责日志的可视化

官方网站：

JDK – http://www.oracle.com/technetwork/java/javase/downloads/index.html

Elasticsearch – https://www.elastic.co/downloads/elasticsearch

Logstash – https://www.elastic.co/downloads/logstash

Kibana – https://www.elastic.co/downloads/kibana

Nginx- https://www.nginx.com/

服务端配置：

安装Java JDK：

cat /etc/redhat-release //这是我linux的版本 CentOS Linux release 7.1.1503 (Core)  //我们通过yum 方式安装Java Jdk yum install java-1.7.0-openjdk

cat/etc/redhat-release //这是我linux的版本 CentOS Linuxrelease7.1.1503(Core) //我们通过yum 方式安装Java Jdk yum installjava-1.7.0-openjdk

Elasticsearch安装：

#下载安装 wget https://download.elastic.co/elasticsearch/elasticsearch/elasticsearch-1.7.1.noarch.rpm yum localinstall elasticsearch-1.7.1.noarch.rpm   #启动相关服务 service elasticsearch start service elasticsearch status  #查看Elasticsearch的配置文件 rpm -qc elasticsearch  /etc/elasticsearch/elasticsearch.yml /etc/elasticsearch/logging.yml /etc/init.d/elasticsearch /etc/sysconfig/elasticsearch /usr/lib/sysctl.d/elasticsearch.conf /usr/lib/systemd/system/elasticsearch.service /usr/lib/tmpfiles.d/elasticsearch.conf  #查看端口使用情况 netstat -nltp  Active Internet connections (only servers) Proto Recv-Q Send-Q Local Address           Foreign Address         State       PID/Program name     tcp        0      0 127.0.0.1:25            0.0.0.0:*               LISTEN      1817/master          tcp        0      0 0.0.0.0:5601            0.0.0.0:*               LISTEN      27369/node           tcp        0      0 0.0.0.0:80              0.0.0.0:*               LISTEN      31848/nginx: master  tcp        0      0 0.0.0.0:22              0.0.0.0:*               LISTEN      16567/sshd           tcp6       0      0 127.0.0.1:8005          :::*                    LISTEN      8263/java            tcp6       0      0 :::5000                 :::*                    LISTEN      2771/java            tcp6       0      0 :::8009                 :::*                    LISTEN      8263/java            tcp6       0      0 :::3306                 :::*                    LISTEN      28839/mysqld         tcp6       0      0 :::80                   :::*                    LISTEN      31848/nginx: master  tcp6       0      0 :::8080                 :::*                    LISTEN      8263/java            tcp6       0      0 :::9200                 :::*                    LISTEN      25808/java           tcp6       0      0 :::9300                 :::*                    LISTEN      25808/java           tcp6       0      0 :::9301                 :::*                    LISTEN      2771/java            tcp6       0      0 :::22                   :::*                    LISTEN      16567/sshd

#下载安装 wget https://download.elastic.co/elasticsearch/elasticsearch/elasticsearch-1.7.1.noarch.rpm yum localinstallelasticsearch-1.7.1.noarch.rpm   #启动相关服务 service elasticsearchstart service elasticsearchstatus   #查看Elasticsearch的配置文件 rpm-qcelasticsearch   /etc/elasticsearch/elasticsearch.yml /etc/elasticsearch/logging.yml /etc/init.d/elasticsearch /etc/sysconfig/elasticsearch /usr/lib/sysctl.d/elasticsearch.conf /usr/lib/systemd/system/elasticsearch.service /usr/lib/tmpfiles.d/elasticsearch.conf   #查看端口使用情况 netstat-nltp   Active Internetconnections(onlyservers) Proto Recv-QSend-QLocalAddress          ForeignAddress        State      PID/Programname     tcp        0      0127.0.0.1:25            0.0.0.0:*              LISTEN      1817/master         tcp        0      00.0.0.0:5601            0.0.0.0:*              LISTEN      27369/node           tcp        0      00.0.0.0:80              0.0.0.0:*              LISTEN      31848/nginx:master tcp        0      00.0.0.0:22              0.0.0.0:*              LISTEN      16567/sshd           tcp6      0      0127.0.0.1:8005          :::*                    LISTEN      8263/java           tcp6      0      0:::5000                :::*                    LISTEN      2771/java           tcp6      0      0:::8009                :::*                    LISTEN      8263/java           tcp6      0      0:::3306                :::*                    LISTEN      28839/mysqld         tcp6      0      0:::80                  :::*                    LISTEN      31848/nginx:master tcp6      0      0:::8080                :::*                    LISTEN      8263/java           tcp6      0      0:::9200                :::*                    LISTEN      25808/java           tcp6      0      0:::9300                :::*                    LISTEN      25808/java           tcp6      0      0:::9301                :::*                    LISTEN      2771/java           tcp6      0      0:::22                  :::*                    LISTEN      16567/sshd

我们看到9200端口了说明我们安装成功了，我们可以在终端输入

#测试访问 curl -X GET http://localhost:9200/

#测试访问 curl-XGEThttp://localhost:9200/

或者直接浏览器打开我们可以看到

{ status: 200, name: "Pip the Troll", cluster_name: "elasticsearch", version: { number: "1.7.2", build_hash: "e43676b1385b8125d647f593f7202acbd816e8ec", build_timestamp: "2015-09-14T09:49:53Z", build_snapshot: false, lucene_version: "4.10.4" }, tagline: "You Know, for Search" }

{ status:200, name:"Pip the Troll", cluster_name:"elasticsearch", version:{ number:"1.7.2", build_hash:"e43676b1385b8125d647f593f7202acbd816e8ec", build_timestamp:"2015-09-14T09:49:53Z", build_snapshot:false, lucene_version:"4.10.4" }, tagline:"You Know, for Search" }

说明我们的程序是运行正常的。

Kibana的安装：

#下载tar包 wget https://download.elastic.co/kibana/kibana/kibana-4.1.1-linux-x64.tar.gz #解压 tar zxf kibana-4.1.1-linux-x64.tar.gz -C /usr/local/ cd /usr/local/ mv kibana-4.1.1-linux-x64 kibana  #创建kibana服务 vim /etc/rc.d/init.d/kibana  #!/bin/bash ### BEGIN INIT INFO # Provides:          kibana # Default-Start:     2 3 4 5 # Default-Stop:      0 1 6 # Short-Description: Runs kibana daemon # Description: Runs the kibana daemon as a non-root user ### END INIT INFO  # Process name NAME=kibana DESC="Kibana4" PROG="/etc/init.d/kibana"  # Configure location of Kibana bin KIBANA_BIN=/usr/local/kibana/bin  # PID Info PID_FOLDER=/var/run/kibana/ PID_FILE=/var/run/kibana/$NAME.pid LOCK_FILE=/var/lock/subsys/$NAME PATH=/bin:/usr/bin:/sbin:/usr/sbin:$KIBANA_BIN DAEMON=$KIBANA_BIN/$NAME  # Configure User to run daemon process DAEMON_USER=root # Configure logging location KIBANA_LOG=/var/log/kibana.log  # Begin Script RETVAL=0  if [ `id -u` -ne 0 ]; then         echo "You need root privileges to run this script"         exit 1 fi  # Function library . /etc/init.d/functions   start() {         echo -n "Starting $DESC : "  pid=`pidofproc -p $PID_FILE kibana`         if [ -n "$pid" ] ; then                 echo "Already running."                 exit 0         else         # Start Daemon if [ ! -d "$PID_FOLDER" ] ; then                         mkdir $PID_FOLDER                 fi daemon --user=$DAEMON_USER --pidfile=$PID_FILE $DAEMON 1>"$KIBANA_LOG" 2>&1 &                 sleep 2                 pidofproc node > $PID_FILE                 RETVAL=$?                 [[ $? -eq 0 ]] && success || failure echo                 [ $RETVAL = 0 ] && touch $LOCK_FILE                 return $RETVAL         fi }  reload() {     echo "Reload command is not implemented for this service."     return $RETVAL }  stop() {         echo -n "Stopping $DESC : "         killproc -p $PID_FILE $DAEMON         RETVAL=$? echo         [ $RETVAL = 0 ] && rm -f $PID_FILE $LOCK_FILE }   case "$1" in   start)         start ;;   stop)         stop         ;;   status)         status -p $PID_FILE $DAEMON         RETVAL=$?         ;;   restart)         stop         start         ;;   reload) reload ;;   *) # Invalid Arguments, print the following message.         echo "Usage: $0 {start|stop|status|restart}" >&2 exit 2         ;; esac  #修改启动权限 chmod +x /etc/rc.d/init.d/kibana  #启动kibana服务 service kibana start service kibana status  #查看端口 netstat -nltp

#下载tar包 wget https://download.elastic.co/kibana/kibana/kibana-4.1.1-linux-x64.tar.gz #解压 tar zxfkibana-4.1.1-linux-x64.tar.gz-C/usr/local/ cd/usr/local/ mv kibana-4.1.1-linux-x64kibana #创建kibana服务 vim/etc/rc.d/init.d/kibana #!/bin/bash ### BEGIN INIT INFO # Provides:   kibana # Default-Start:     2 3 4 5 # Default-Stop:      0 1 6 # Short-Description: Runs kibana daemon # Description: Runs the kibana daemon as a non-root user ### END INIT INFO # Process name NAME=kibana DESC="Kibana4" PROG="/etc/init.d/kibana" # Configure location of Kibana bin KIBANA_BIN=/usr/local/kibana/bin # PID Info PID_FOLDER=/var/run/kibana/ PID_FILE=/var/run/kibana/$NAME.pid LOCK_FILE=/var/lock/subsys/$NAME PATH=/bin:/usr/bin:/sbin:/usr/sbin:$KIBANA_BIN DAEMON=$KIBANA_BIN/$NAME # Configure User to run daemon process DAEMON_USER=root # Configure logging location KIBANA_LOG=/var/log/kibana.log # Begin Script RETVAL=0 if[`id-u`-ne0];then  echo"You need root privileges to run this script"  exit1 fi # Function library ./etc/init.d/functions start(){  echo-n"Starting $DESC : " pid=`pidofproc-p$PID_FILEkibana`  if[-n"$pid"];then   echo"Already running."   exit0  else  # Start Daemon if[!-d"$PID_FOLDER"];then    mkdir$PID_FOLDER   fi daemon--user=$DAEMON_USER--pidfile=$PID_FILE$DAEMON1>"$KIBANA_LOG"2>&1&   sleep2   pidofprocnode>$PID_FILE   RETVAL=$?   [[$?-eq0]]&&success||failure echo   [$RETVAL=0]&&touch$LOCK_FILE   return$RETVAL  fi } reload() {     echo"Reload command is not implemented for this service."     return$RETVAL } stop(){  echo-n"Stopping $DESC : "  killproc-p$PID_FILE$DAEMON  RETVAL=$? echo  [$RETVAL=0]&&rm-f$PID_FILE$LOCK_FILE } case"$1"in   start)  start ;;   stop)  stop  ;;   status)  status-p$PID_FILE$DAEMON  RETVAL=$?  ;;   restart)  stop  start  ;;   reload) reload ;;   *) # Invalid Arguments, print the following message.  echo"Usage: $0 {start|stop|status|restart}">&2 exit2  ;; esac #修改启动权限 chmod+x/etc/rc.d/init.d/kibana #启动kibana服务 service kibanastart service kibanastatus #查看端口 netstat-nltp

因为刚刚已经执行过

netstat -nltp

netstat-nltp

所以显示的效果我这里就不贴了，如果我们能看到5601端口就说明我们安装成功了。

Option 1：Generate SSL Certificates：

生成SSL证书是为了服务端和客户端进行验证：

sudo vi /etc/pki/tls/openssl.cnf

sudo vi/etc/pki/tls/openssl.cnf

Find the [ v3_ca ] section in the file, and add this line under it (substituting in the Logstash Server’s private IP address):

subjectAltName = IP: logstash_server_private_ip

subjectAltName=IP:logstash_server_private_ip

cd /etc/pki/tls sudo openssl req -config /etc/pki/tls/openssl.cnf -x509 -days 3650 -batch -nodes -newkey rsa:2048 -keyout private/logstash-forwarder.key -out certs/logstash-forwarder.crt

cd/etc/pki/tls sudo opensslreq-config/etc/pki/tls/openssl.cnf-x509-days3650-batch-nodes-newkeyrsa:2048-keyoutprivate/logstash-forwarder.key-outcerts/logstash-forwarder.crt

Option 2: FQDN (DNS)：

cd /etc/pki/tls sudo openssl req -subj '/CN=<^>logstash_server_fqdn/' -x509 -days 3650 -batch -nodes -newkey rsa:2048 -keyout private/logstash-forwarder.key -out certs/logstash-forwarder.crt

cd/etc/pki/tls sudo opensslreq-subj'/CN=<^>logstash_server_fqdn/'-x509-days3650-batch-nodes-newkeyrsa:2048-keyoutprivate/logstash-forwarder.key-outcerts/logstash-forwarder.crt

Logstash安装：

Logstash Forwarder（客户端）：

安装Logstash Forwarder wget https://download.elastic.co/logstash-forwarder/binaries/logstash-forwarder-0.4.0-1.x86_64.rpm yum localinstall logstash-forwarder-0.4.0-1.x86_64.rpm  #查看logstash-forwarder的配置文件位置 rpm -qc logstash-forwarder /etc/logstash-forwarder.conf  #备份配置文件 cp /etc/logstash-forwarder.conf /etc/logstash-forwarder.conf.save  #编辑 /etc/logstash-forwarder.conf，需要根据实际情况进行修改  vim /etc/logstash-forwarder.conf  {   "network": {     "servers": [ "这里写服务器的ip:5000" ],      "ssl ca": "/etc/pki/tls/certs/logstash-forwarder.crt",      "timeout": 15   },    "files": [     {       "paths": [         "/var/log/messages",         "/var/log/secure"       ],        "fields": { "type": "syslog" }     }   ] }

安装LogstashForwarder wget https://download.elastic.co/logstash-forwarder/binaries/logstash-forwarder-0.4.0-1.x86_64.rpm yum localinstalllogstash-forwarder-0.4.0-1.x86_64.rpm   #查看logstash-forwarder的配置文件位置 rpm-qclogstash-forwarder /etc/logstash-forwarder.conf   #备份配置文件 cp/etc/logstash-forwarder.conf/etc/logstash-forwarder.conf.save   #编辑 /etc/logstash-forwarder.conf，需要根据实际情况进行修改   vim/etc/logstash-forwarder.conf {   "network":{     "servers":["这里写服务器的ip:5000"],       "ssl ca":"/etc/pki/tls/certs/logstash-forwarder.crt",       "timeout":15   },     "files":[     {       "paths":[         "/var/log/messages",         "/var/log/secure"       ],         "fields":{"type":"syslog"}     }   ] }

Logstash Server（服务端）：

#下载rpm包 wget https://download.elastic.co/logstash/logstash/packages/centos/logstash-1.5.4-1.noarch.rpm #安装 yum localinstall logstash-1.5.4-1.noarch.rpm  #创建一个01-logstash-initial.conf文件 vim /etc/logstash/conf.d/01-logstash-initial.conf  input {   lumberjack {     port => 5000     type => "logs"     ssl_certificate => "/etc/pki/tls/certs/logstash-forwarder.crt"     ssl_key => "/etc/pki/tls/private/logstash-forwarder.key"   } }   filter {   if [type] == "syslog" {     grok {       match => { "message" => "%{SYSLOGTIMESTAMP:syslog_timestamp} %{SYSLOGHOST:syslog_hostname} %{DATA:syslog_program}(?:[%{POSINT:syslog_pid}])?: %{GREEDYDATA:syslog_message}" }       add_field => [ "received_at", "%{@timestamp}" ]       add_field => [ "received_from", "%{host}" ]     }     syslog_pri { }     date {       match => [ "syslog_timestamp", "MMM  d HH:mm:ss", "MMM dd HH:mm:ss" ]     }   } }  output {   elasticsearch { host => localhost }   stdout { codec => rubydebug } }  #启动logstash服务 service logstash start service logstash status  #访问Kibana，Time-field name 选择 @timestamp 要在下一步操作 Nginx 日志配置之后访问 不然会没有数据不能创建 http://localhost:5601/  #增加节点和客户端配置一样，注意同步证书(可以通过SSH的方式同步) /etc/pki/tls/certs/logstash-forwarder.crt

#下载rpm包 wget https://download.elastic.co/logstash/logstash/packages/centos/logstash-1.5.4-1.noarch.rpm #安装 yum localinstalllogstash-1.5.4-1.noarch.rpm #创建一个01-logstash-initial.conf文件 vim/etc/logstash/conf.d/01-logstash-initial.conf input{   lumberjack{     port=>5000     type=>"logs"     ssl_certificate=>"/etc/pki/tls/certs/logstash-forwarder.crt"     ssl_key=>"/etc/pki/tls/private/logstash-forwarder.key"   } }     filter{   if[type]=="syslog"{     grok{       match=>{"message"=>"%{SYSLOGTIMESTAMP:syslog_timestamp} %{SYSLOGHOST:syslog_hostname} %{DATA:syslog_program}(?:[%{POSINT:syslog_pid}])?: %{GREEDYDATA:syslog_message}"}       add_field=>["received_at","%{@timestamp}"]       add_field=>["received_from","%{host}"]     }     syslog_pri{}     date{       match=>["syslog_timestamp","MMM  d HH:mm:ss","MMM dd HH:mm:ss"]     }   } }   output{   elasticsearch{host=>localhost}   stdout{codec=>rubydebug} }   #启动logstash服务 service logstashstart service logstashstatus   #访问Kibana，Time-field name 选择 @timestamp 要在下一步操作 Nginx 日志配置之后访问 不然会没有数据不能创建 http://localhost:5601/   #增加节点和客户端配置一样，注意同步证书(可以通过SSH的方式同步) /etc/pki/tls/certs/logstash-forwarder.crt

配置Nginx日志：

#修改客户端配置 vim /etc/logstash-forwarder.conf  {   "network": {     "servers": [ "自己服务器的ip地址:5000" ],      "ssl ca": "/etc/pki/tls/certs/logstash-forwarder.crt",      "timeout": 15   },    "files": [     {       "paths": [         "/var/log/messages",         "/var/log/secure"       ],       "fields": { "type": "syslog" }     }, {       "paths": [         "/app/local/nginx/logs/access.log"       ],       "fields": { "type": "nginx" }     }   ] }  #服务端增加patterns mkdir /opt/logstash/patterns vim /opt/logstash/patterns/nginx  NGUSERNAME [a-zA-Z.@-+_%]+ NGUSER %{NGUSERNAME} NGINXACCESS %{IPORHOST:remote_addr} - - [%{HTTPDATE:time_local}] "%{WORD:method} %{URIPATH:path}(?:%{URIPARAM:param})? HTTP/%{NUMBER:httpversion}" %{INT:status} %{INT:body_bytes_sent} %{QS:http_referer} %{QS:http_user_agent}   #修改logstash权限 chown -R logstash:logstash /opt/logstash/patterns  #修改服务端配置 vim /etc/logstash/conf.d/01-logstash-initial.conf   input {   lumberjack {     port => 5000     type => "logs"     ssl_certificate => "/etc/pki/tls/certs/logstash-forwarder.crt"     ssl_key => "/etc/pki/tls/private/logstash-forwarder.key"   } }   filter {   if [type] == "syslog" {     grok {       match => { "message" => "%{SYSLOGTIMESTAMP:syslog_timestamp} %{SYSLOGHOST:syslog_hostname} %{DATA:syslog_program}(?:[%{POSINT:syslog_pid}])?: %{GREEDYDATA:syslog_message}" }       add_field => [ "received_at", "%{@timestamp}" ]       add_field => [ "received_from", "%{host}" ]     }     syslog_pri { }     date {       match => [ "syslog_timestamp", "MMM  d HH:mm:ss", "MMM dd HH:mm:ss" ]     }   }   if [type] == "nginx" {     grok {        match => { "message" => "%{NGINXACCESS}" }     }   } }  output {   elasticsearch { host => localhost }   stdout { codec => rubydebug } }

#修改客户端配置 vim/etc/logstash-forwarder.conf   {   "network":{     "servers":["自己服务器的ip地址:5000"],       "ssl ca":"/etc/pki/tls/certs/logstash-forwarder.crt",       "timeout":15   },     "files":[     {       "paths":[         "/var/log/messages",         "/var/log/secure"       ],       "fields":{"type":"syslog"}     },{       "paths":[         "/app/local/nginx/logs/access.log"       ],       "fields":{"type":"nginx"}     }   ] }   #服务端增加patterns mkdir/opt/logstash/patterns vim/opt/logstash/patterns/nginx   NGUSERNAME[a-zA-Z.@-+_%]+ NGUSER%{NGUSERNAME} NGINXACCESS%{IPORHOST:remote_addr}--[%{HTTPDATE:time_local}]"%{WORD:method} %{URIPATH:path}(?:%{URIPARAM:param})? HTTP/%{NUMBER:httpversion}"%{INT:status}%{INT:body_bytes_sent}%{QS:http_referer}%{QS:http_user_agent}     #修改logstash权限 chown-Rlogstash:logstash/opt/logstash/patterns   #修改服务端配置 vim/etc/logstash/conf.d/01-logstash-initial.conf   input{   lumberjack{     port=>5000     type=>"logs"     ssl_certificate=>"/etc/pki/tls/certs/logstash-forwarder.crt"     ssl_key=>"/etc/pki/tls/private/logstash-forwarder.key"   } }     filter{   if[type]=="syslog"{     grok{       match=>{"message"=>"%{SYSLOGTIMESTAMP:syslog_timestamp} %{SYSLOGHOST:syslog_hostname} %{DATA:syslog_program}(?:[%{POSINT:syslog_pid}])?: %{GREEDYDATA:syslog_message}"}       add_field=>["received_at","%{@timestamp}"]       add_field=>["received_from","%{host}"]     }     syslog_pri{}     date{       match=>["syslog_timestamp","MMM  d HH:mm:ss","MMM dd HH:mm:ss"]     }   }   if[type]=="nginx"{     grok{        match=>{"message"=>"%{NGINXACCESS}"}     }   } }   output{   elasticsearch{host=>localhost}   stdout{codec=>rubydebug} }

我们看一下完成配置之后的效果：

使用Elasticsearch + Logstash + Kibana搭建日志集中分析平台实践