Chemmy's Blog

Nlog、elasticsearch、Kibana以及logstash在项目中的应用（二） - 风~继续吹 - 博客园

Excerpt

上一篇说如何搭建elk的环境（不清楚的可以看我的上一篇博客http://www.cnblogs.com/never-give-up-1015/p/5715904.html），现在来说一下如何用Nlog将日志通过logstash写入elasticsearch。 新建一个项目，用nuget引入Nlog，

---

上一篇说如何搭建elk的环境（不清楚的可以看我的上一篇博客http://www.cnblogs.com/never-give-up-1015/p/5715904.html），现在来说一下如何用Nlog将日志通过logstash写入elasticsearch。

 新建一个项目，用nuget引入Nlog，随便写几行打日志的代码（写成循环是为了方便测试）

<span> 1</span>             <span>var</span> log =<span> LogManager.GetCurrentClassLogger();
</span><span> 2</span>             <span>while</span> (<span>true</span><span>)
</span><span> 3</span> <span>            {
</span><span> 4</span>                 log.Info(<span>"</span><span>high Hkaos one</span><span>"</span><span>);
</span><span> 5</span>                 log.Debug(<span>"</span><span>high Hkaos two</span><span>"</span><span>);
</span><span> 6</span>                 log.Warn(<span>"</span><span>high Hkaos three</span><span>"</span><span>);
</span><span> 7</span>                 log.Error(<span>"</span><span>high Hkaos four</span><span>"</span><span>);
</span><span> 8</span>                 Console.WriteLine(<span>"</span><span>success</span><span>"</span><span>);
</span><span> 9</span> <span>                Console.ReadKey();
</span><span>10</span>             }

我是采用udp的方式传输，所以在配置Nlog的时候需要设置type为Network，至于message的格式可以在layout中配置（网上一大堆，一搜就出来了），下面是我的配置文件

&lt;?xml version=<span>"</span><span>1.0</span><span>"</span> encoding=<span>"</span><span>utf-8</span><span>"</span> ?&gt;
&lt;configuration&gt;
  &lt;configSections&gt;
    &lt;section name=<span>"</span><span>nlog</span><span>"</span> type=<span>"</span><span>NLog.Config.ConfigSectionHandler, NLog</span><span>"</span>/&gt;
  &lt;/configSections&gt;
  &lt;startup&gt;
    &lt;supportedRuntime version=<span>"</span><span>v4.0</span><span>"</span> sku=<span>"</span><span>.NETFramework,Version=v4.5</span><span>"</span> /&gt;
  &lt;/startup&gt;
  &lt;nlog xmlns=<span>"</span><span>http://www.nlog-project.org/schemas/NLog.xsd</span><span>"</span><span>
        xmlns:xsi</span>=<span>"</span><span>http://www.w3.org/2001/XMLSchema-instance</span><span>"</span>&gt;
    &lt;targets&gt;
      &lt;target name=<span>"</span><span>network</span><span>"</span> xsi:type=<span>"</span><span>Network</span><span>"</span> address=<span>"</span><span>udp://192.168.4.12:4561</span><span>"</span> layout=<span>"</span><span>${message}</span><span>"</span>/&gt;
    &lt;/targets&gt;
    &lt;rules&gt;
          &lt;logger name=<span>"</span><span>*</span><span>"</span> minlevel=<span>"</span><span>Debug</span><span>"</span> writeTo=<span>"</span><span>network</span><span>"</span> /&gt;
    &lt;/rules&gt;
  &lt;/nlog&gt;
&lt;/configuration&gt;

一开始没有写configSections和rules，所以一直没有打日志，晕死。

配置完毕之后，在配置logstash。由于Nlog是通过udp一直向配置的地址端口发送日志，所以在配置文件（这个配置就是自己手动建的文件，不懂的看上一篇博客）中的input要监听Nlog中配置的地址。logstash的input中可以写多种搜集日志的方式，有兴趣的可以去官网地址看看（https://www.elastic.co/guide/en/logstash/current/input-plugins.html）。

<span>input { 
    udp{
        host</span>=&gt;<span>"</span><span>192.168.4.12</span><span>"</span><span>
        port</span>=&gt;<span>4561</span><span>
    }    
}<br></span>

input下面可以写多个搜集方式，比如

<span>input { 
    stdin{
    
    }
    udp{
        host</span>=&gt;<span>"</span><span>192.168.4.12</span><span>"</span><span>
        port</span>=&gt;<span>4561</span><span>
    }    
}</span>

配置完成之后就可以测试一下。运行程序之前记得启动logstash，elasticsearch和kibana暂时不用可以不用启动。

同时观察启动logstash的窗口

出现了刚刚打印的日志，yes!　成功了。

那我们先看看如何把日志写到elasticsearch里面并且通过kibana查看。很简单，我们在output中把原先输入到屏幕上的信息同时也输入到elasticsearch中就Ok了。同时呢，output的输出方式也有很多，可以去看看（https://www.elastic.co/guide/en/logstash/current/output-plugins.html），elasticsearch的选项有很多，这里有详细的介绍（https://www.elastic.co/guide/en/logstash/current/plugins-outputs-elasticsearch.html），下面是我的配置

<span>output {
        stdout{
        }
        elasticsearch {
        hosts </span>=&gt; [<span>"</span><span>127.0.0.1:9200</span><span>"</span><span>]
        index </span>=&gt; <span>"</span><span>logstash-%{+YYYY.MM.dd}</span><span>"</span><span>
        document_type </span>=&gt; <span>"</span><span>logs</span><span>"</span><span>
        }
}</span>

hosts就是elasticsearch的地址，可以配置多个，是个数组，index是索引的名称，相当于数据库的库名字，这里就是命名就是logstash-当天日期，每天都会新建一个库，document_type是类型的名称，相当于数据库中的表，老版本是index_type，新版本不让用了，如果用的话就会报错，告诉你这个方法被弃用了，让你用document_type。配置好之后需要重新启动logstash，首先要先启动elasticsearch和kibana。（Tip：重启logstash的话可以用Ctrl+C重启，连续按，能输入为止）。走起!

界面是没啥变化，这时候要登录kibana看看，地址在启动kibana的时候被被打印出来

界面如下：

刚刚的日志被打印了出来，bingo！右边的字段可以自己选择添加是否显示在右边，右上角有个last 15 minutes，表示查看最新15分钟的，点击之后你可以任意选择你想查看的时间段。

同时呢，elasticsearch也自带了查询的插件，不过需要安装，方法也简单，cmd进入elasticsearch的目录 运行plugin -install mobz/elasticsearch-head，就会自动安装了，成功之后地址为http://localhost:9200/\_plugin/head/，可以直接访问，下面是界面

在基本查询里面你可以查询打印的日志，还有执行的查询语句和原始的json。

但是这些只能打印出message，比如我要打印其他的东西怎么办，比如方法名，hash值，是debug还是info还是error以及其他自定义的字段，这时候会有人说那就全部打印在message里面，那为啥还要这么麻烦用这个工具呢，直接打印到文件里面直接Ctrl+F就好了。所以我们要通过Nlog打印自定义的字段。这就要自己在Nlog里面定义了。

这也是在网上查找了资料才找到的方法，需要继承Nlog的LayoutRenderer，重写Append方法

[LayoutRenderer(<span>"</span><span>json</span><span>"</span><span>)]
</span><span>public</span> <span>class</span><span> JsonLayoutRenderer : LayoutRenderer
{
    </span><span>protected</span> <span>override</span> <span>void</span><span> Append(StringBuilder builder, LogEventInfo loggingEvent)
    {
        </span><span>var</span> msg = loggingEvent.FormattedMessage.AsJson&lt;JsonLogMessage&gt;<span>();

        msg.priority </span>=<span> loggingEvent.Level.ToString();
        msg.logger_name </span>=<span> loggingEvent.LoggerName;
        msg.thread </span>=<span> loggingEvent.SequenceID.ToString();
        msg.application </span>=<span> "TestApplication";
        </span><span>var</span> json =<span> msg.AsJsonString();
        builder.Append(json);
    }
}</span>

本来是通过赋值的方法在LogEventInfo中取我想要的值比如Method,file,class等等，Log4net中是直接可以取得到的，但是Nlog不行，唉，蛋疼，所以只能吧数据写到json序列化之后写到message里面，然后在这个方法里面反序列化回来。大神们有什么好方法希望给解惑啊！

这是对象信息

<span>public</span> <span>class</span><span> JsonLogMessage
  {
      </span><span>public</span> <span>string</span> @class { <span>get</span>; <span>set</span><span>; }
      </span><span>public</span> <span>string</span> file { <span>get</span>; <span>set</span><span>; }
      </span><span>public</span> <span>string</span> host { <span>get</span>; <span>set</span><span>; }
      </span><span>public</span> <span>string</span> logger_name { <span>get</span>; <span>set</span><span>; }
      </span><span>public</span> <span>string</span> path { <span>get</span>; <span>set</span><span>; }
      </span><span>public</span> <span>string</span> priority { <span>get</span>; <span>set</span><span>; }
      </span><span>public</span> <span>string</span> thread { <span>get</span>; <span>set</span><span>; }
      </span><span>public</span> <span>string</span> application { <span>get</span>; <span>set</span><span>; }
      </span><span>public</span> <span>string</span> message { <span>get</span>; <span>set</span><span>; }
      </span><span>public</span> <span>string</span> addUser { <span>get</span>; <span>set</span><span>; }
      </span><span>public</span> <span>string</span> addUserName { <span>get</span>; <span>set</span><span>; }
      </span><span>public</span> <span>string</span> objName { <span>get</span>; <span>set</span><span>; }
      </span><span>public</span> <span>string</span> objId { <span>get</span>; <span>set</span><span>; }
      </span><span>public</span> <span>int</span> hash { <span>get</span>; <span>set</span><span>; }
      </span><span>public</span> <span>string</span> method { <span>get</span>; <span>set</span><span>; }
      </span><span>public</span> DateTime timestamp { <span>get</span>; <span>set</span><span>; }
  }</span>

本来timestamp字段不想写上去的，但是后来通过代码取不到时间，也是一个蛋疼的问题。

那message的格式修改了，所以打日志的方法也就要变了，由于Append的方式改变了，所以布局方式需要重新修改，只要在程序运行前加一段代码就好,代码里面用的AsJosn和AsJosnString是封装好的json序列化和反序列化的方法，所以大家要用的话就直接用json原生的方法就好了。另外在配置文件中需要改变layout的布局方式，将layout=”${message}修改成layout=”${json}

<span>private</span> <span>static</span> <span>void</span> Main(<span>string</span><span>[] args)
        {
            ConfigurationItemFactory.Default.LayoutRenderers.RegisterDefinition(</span><span>"</span><span>json</span><span>"</span>, <span>typeof</span><span>(JsonLayoutRenderer));

            </span><span>var</span> log =<span> LogManager.GetCurrentClassLogger();
            </span><span>while</span> (<span>true</span><span>)
            {
                </span><span>var</span> josn1 = <span>new</span> JsonLogMessage(<span>"</span><span>Main</span><span>"</span>, -<span>1</span>, <span>"</span><span>high Hkaos one</span><span>"</span>, <span>"</span><span>tester</span><span>"</span>, <span>"</span><span>testName</span><span>"</span>, <span>"</span><span>objName</span><span>"</span>, <span>"</span><span>objId</span><span>"</span><span>);
                </span><span>var</span> josn2 = <span>new</span> JsonLogMessage(<span>"</span><span>Main</span><span>"</span>, -<span>1</span>, <span>"</span><span>high Hkaos two</span><span>"</span>, <span>"</span><span>tester</span><span>"</span>, <span>"</span><span>testName</span><span>"</span>, <span>"</span><span>objName</span><span>"</span>, <span>"</span><span>objId</span><span>"</span><span>);
                </span><span>var</span> josn3 = <span>new</span> JsonLogMessage(<span>"</span><span>Main</span><span>"</span>, -<span>1</span>, <span>"</span><span>high Hkaos three</span><span>"</span>, <span>"</span><span>tester</span><span>"</span>, <span>"</span><span>testName</span><span>"</span>, <span>"</span><span>objName</span><span>"</span>, <span>"</span><span>objId</span><span>"</span><span>);
                </span><span>var</span> josn4 = <span>new</span> JsonLogMessage(<span>"</span><span>Main</span><span>"</span>, -<span>1</span>, <span>"</span><span>high Hkaos four</span><span>"</span>, <span>"</span><span>tester</span><span>"</span>, <span>"</span><span>testName</span><span>"</span>, <span>"</span><span>objName</span><span>"</span>, <span>"</span><span>objId</span><span>"</span><span>);

                log.Info(josn1.AsJsonString());
                log.Debug(josn2.AsJsonString());
                log.Warn(josn3.AsJsonString());
                log.Error(josn4.AsJsonString());
                Console.WriteLine(</span><span>"</span><span>success</span><span>"</span><span>);
                Console.ReadKey();
            }
        }</span>

运行！

完蛋了，看到这个我就知道有错了，虽然日志出来了，但是我们在head插件或者kibana中看看（我用的是head插件）

日志的信息全部打印到message里面，操蛋了。我要的不是这样子的。去Google了一下，发现要是用json的话，需要在input里面加codec=>”json”，好了，那就修改一下input

<span>input { 
    stdin{

    }
    udp{
        host</span>=&gt;<span>"</span><span>192.168.4.12</span><span>"</span><span>
        port</span>=&gt;<span>4561</span><span>
        codec</span>=&gt;<span>"</span><span>json</span><span>"</span><span>
    }    
}</span>

重启一下logstash，再来一遍

这次明显打印的不一样了，再看看head里面数据

yes!要的就是这个效果。

之后我吧Nlog的配置改成了代码写的，使用配置文件总有一些不方便的地方，比如说地址。下面是代码

<span>private</span> <span>static</span> <span>void</span> Main(<span>string</span><span>[] args)
       {
           ConfigurationItemFactory.Default.LayoutRenderers.RegisterDefinition(</span><span>"</span><span>json</span><span>"</span>, <span>typeof</span><span>(JsonLayoutRenderer));

           </span><span>var</span> log =<span> LogManager.GetCurrentClassLogger();
           </span><span>var</span> udpTarget = <span>new</span><span> NetworkTarget()
           {
               Address </span>= <span>"</span><span>udp://192.168.4.12:4561</span><span>"</span><span>,
               Encoding </span>=<span> Encoding.UTF8,
               Layout </span>= <span>"</span><span>${json}</span><span>"</span><span>,
               NewLine </span>= <span>true</span><span>,
           };

           </span><span>var</span> config = <span>new</span><span> LoggingConfiguration();
           config.LoggingRules.Add(</span><span>new</span> LoggingRule(<span>"</span><span>*</span><span>"</span><span>, LogLevel.Debug, udpTarget));
           LogManager.Configuration </span>=<span> config;

           </span><span>while</span> (<span>true</span><span>)
           {
               </span><span>var</span> josn1 = <span>new</span> JsonLogMessage(<span>"</span><span>Main</span><span>"</span>, -<span>1</span>, <span>"</span><span>high Hkaos one</span><span>"</span>, <span>"</span><span>tester</span><span>"</span>, <span>"</span><span>testName</span><span>"</span>, <span>"</span><span>objName</span><span>"</span>, <span>"</span><span>objId</span><span>"</span><span>);
               </span><span>var</span> josn2 = <span>new</span> JsonLogMessage(<span>"</span><span>Main</span><span>"</span>, -<span>1</span>, <span>"</span><span>high Hkaos two</span><span>"</span>, <span>"</span><span>tester</span><span>"</span>, <span>"</span><span>testName</span><span>"</span>, <span>"</span><span>objName</span><span>"</span>, <span>"</span><span>objId</span><span>"</span><span>);
               </span><span>var</span> josn3 = <span>new</span> JsonLogMessage(<span>"</span><span>Main</span><span>"</span>, -<span>1</span>, <span>"</span><span>high Hkaos three</span><span>"</span>, <span>"</span><span>tester</span><span>"</span>, <span>"</span><span>testName</span><span>"</span>, <span>"</span><span>objName</span><span>"</span>, <span>"</span><span>objId</span><span>"</span><span>);
               </span><span>var</span> josn4 = <span>new</span> JsonLogMessage(<span>"</span><span>Main</span><span>"</span>, -<span>1</span>, <span>"</span><span>high Hkaos four</span><span>"</span>, <span>"</span><span>tester</span><span>"</span>, <span>"</span><span>testName</span><span>"</span>, <span>"</span><span>objName</span><span>"</span>, <span>"</span><span>objId</span><span>"</span><span>);

               log.Info(josn1.AsJsonString());
               log.Debug(josn2.AsJsonString());
               log.Warn(josn3.AsJsonString());
               log.Error(josn4.AsJsonString());
               Console.WriteLine(</span><span>"</span><span>success</span><span>"</span><span>);
               Console.ReadKey();
           }
       }</span>

配置文件里面直接删除原来添加的东西就好了。

就写到这里吧，下一篇准备些一下如何用C#链接日志服务，根据条件查询日志，如何配置模板和映射等。