ELK-logstash在搬运日志的时候会出现多行日志，普通的搬运会造成保存到ES中单条单条，很丑，而且不方便读取，logstash-filter-multiline可以解决该问题

github地址：https://github.com/logstash-plugins/logstash-filter-multiline

其他插件的地址：https://github.com/logstash-plugins

官网地址：https://www.elastic.co/cn/products/logstash

普通日志如下：

记录到es会的记录则是：

我们希望的结果肯定是这样的

 

 

  1. # /usr/share/logstash/bin/logstash-plugin install logstash-filter-multiline
  2. Validating logstash-filter-multiline
  3. Installing logstash-filter-multiline
  4. WARNING: can not set Session#timeout=(0) no session context

 

给 Ruby 加上国内的镜像站：https://gems.ruby-china.com/，替代https://rubygems.org。

注意原https://gems.ruby-china.org/目前已经可用，需要使用https://gems.ruby-china.com/

  1. # yum install -y gem
  2. # gem -v
  3. 2.0.14.1
  4. # gem update --system
  5. # gem -v
  6. 2.6.13
  1. # gem sources -l
  2. *** CURRENT SOURCES ***
  3.  
  4. https://rubygems.org/
  1. # gem sources --add https://gems.ruby-china.com/ --remove https://rubygems.org/
  2. https://gems.ruby-china.com/ added to sources
  3. https://rubygems.org/ removed from sources
  1. # cat ~/.gemrc 
  2. ---
  3. :backtrace: false
  4. :bulk_threshold: 1000
  5. :sources:
  6. - https://gems.ruby-china.com/
  7. :update_sources: true
  8. :verbose: true
  1. # gem install bundler
  2. # bundle config mirror.https://rubygems.org https://gems.ruby-china.com
  1. # vim /usr/share/logstash/Gemfile
  2.  
  3. # This is a Logstash generated Gemfile.
  4. # If you modify this file manually all comments and formatting will be lost.
  5.  
  6. source "https://rubygems.org"
  7. gem "logstash-core", :path => "./logstash-core"
  8. ......
  1. # 更改默认的 https://rubygems.org 为https://gems.ruby-china.com

Gemfile.jruby-1.9.lock文件不用改，会自动更新。

  1. # /usr/share/logstash/bin/logstash-plugin install logstash-filter-multiline
  2. Validating logstash-filter-multiline
  3. Installing logstash-filter-multiline
  4. Installation successful

我们看看logstash都安装了哪些插件

  1. # /usr/share/logstash/bin/logstash-plugin list
  2. logstash-codec-cef
  3. logstash-codec-collectd
  4. logstash-codec-dots
  5. logstash-codec-edn
  6. logstash-codec-edn_lines
  7. logstash-codec-es_bulk
  8. logstash-codec-fluent
  9. logstash-codec-graphite
  10. logstash-codec-json
  11. logstash-codec-json_lines
  12. logstash-codec-line
  13. logstash-codec-msgpack
  14. logstash-codec-multiline
    logstash-filter-multiline
  15. logstash-codec-netflow
  16. logstash-codec-plain
  17. logstash-codec-rubydebug
  18. logstash-filter-aggregate
  19. logstash-filter-anonymize
  20. logstash-filter-cidr
  21. logstash-filter-clone
  22. logstash-filter-csv
  23. logstash-filter-date
  24. logstash-filter-de_dot
  25. logstash-filter-dissect
  26. logstash-filter-dns
  27. logstash-filter-drop
  28. logstash-filter-elasticsearch
  29. logstash-filter-fingerprint
  30. logstash-filter-geoip
  31. logstash-filter-grok
  32. logstash-filter-jdbc_static
  33. logstash-filter-jdbc_streaming
  34. logstash-filter-json
  35. logstash-filter-kv
  36. logstash-filter-metrics
  37. logstash-filter-mutate
  38. logstash-filter-ruby
  39. logstash-filter-sleep
  40. logstash-filter-split
  41. logstash-filter-syslog_pri
  42. logstash-filter-throttle
  43. logstash-filter-translate
  44. logstash-filter-truncate
  45. logstash-filter-urldecode
  46. logstash-filter-useragent
  47. logstash-filter-xml
  48. logstash-input-beats
  49. logstash-input-dead_letter_queue
  50. logstash-input-elasticsearch
  51. logstash-input-exec
  52. logstash-input-file
  53. logstash-input-ganglia
  54. logstash-input-gelf
  55. logstash-input-generator
  56. logstash-input-graphite
  57. logstash-input-heartbeat
  58. logstash-input-http
  59. logstash-input-http_poller
  60. logstash-input-imap
  61. logstash-input-jdbc
  62. logstash-input-kafka
  63. logstash-input-pipe
  64. logstash-input-rabbitmq
  65. logstash-input-redis
  66. logstash-input-s3
  67. logstash-input-snmptrap
  68. logstash-input-sqs
  69. logstash-input-stdin
  70. logstash-input-syslog
  71. logstash-input-tcp
  72. logstash-input-twitter
  73. logstash-input-udp
  74. logstash-input-unix
  75. logstash-output-cloudwatch
  76. logstash-output-csv
  77. logstash-output-elasticsearch
  78. logstash-output-email
  79. logstash-output-file
  80. logstash-output-graphite
  81. logstash-output-http
  82. logstash-output-kafka
  83. logstash-output-lumberjack
  84. logstash-output-nagios
  85. logstash-output-null
  86. logstash-output-pagerduty
  87. logstash-output-pipe
  88. logstash-output-rabbitmq
  89. logstash-output-redis
  90. logstash-output-s3
  91. logstash-output-sns
  92. logstash-output-sqs
  93. logstash-output-stdout
  94. logstash-output-tcp
  95. logstash-output-udp
  96. logstash-output-webhdfs
  97. logstash-patterns-core

这里安装好了，那么就可以打个离线的 zip 包，方便生产环境直接安装。离线包将包含所有依赖的包。

  1. # cd /usr/share/logstash/bin
  2. # ./logstash-plugin prepare-offline-pack --overwrite --output logstash-filter-multiline.zip logstash-filter-multiline
  3. Offline package created at: logstash-filter-multiline.zip
  4.  
  5. You can install it with this command `bin/logstash-plugin install file:///usr/share/logstash/bin/logstash-filter-multiline.zip`

 

  1. codec =>multiline {
  2.      charset=>...          #可选                    字符编码        
  3.      max_bytes=>...        #可选     bytes类型       设置最大的字节数
  4.      max_lines=>...        #可选     number类型      设置最大的行数,默认是500行
  5.      multiline_tag...      #可选     string类型      设置一个事件标签,默认是multiline
  6.      pattern=>...          #必选     string类型      设置匹配的正则表达式
  7.      patterns_dir=>...     #可选     array类型       可以设置多个正则表达式
  8.      negate=>...           #可选     boolean类型     默认false不显示，可设置ture
  9.      what=>...             #必选                    向前previous ， 向后 next
  10.  }
  1. ## negate  只支持布尔值，true 或者false，默认为false。
  2. 如果设置为true，表示与正则表达式（pattern）不匹配的内容都需要整合，
  3. 具体整合在前还是在后，看what参数。如果设置为false，即与pattern匹配的内容
  4.  
  5. ## what  前一行 或者后一行，指出上面对应的规则与前一行内容收集为一行，还是与后一行整合在一起
  6.  
  7. negate默认是 false，不显示与patten匹配的行
  8. 由what决定 向前或向后 匹配
  9.  
  10. negate 设置为true
  11. 则与patten 不匹配的行
  12. 由what决定 向前或向后 匹配

 

版权声明：本文为JetpropelledSnake原创文章，遵循 CC 4.0 BY-SA 版权协议，转载请附上原文出处链接和本声明。
本文链接：https://www.cnblogs.com/JetpropelledSnake/p/10725815.html