p f S e n s e r s y s l o g f l u e L n o t k w d i r i t e d a t a s o u r c e G r a f a n a

Introduction

This post is about pfSense syslog(especially to filterlog) collection and visualization by using Fluentd, Loki and Grafana. We assume you already have running fluentd, loki and grafana instances. The following instruction won’t cover any installation.

Run fluentd and check the syslog of pfSense

  1. First of all, we prepare a very basic fluentd config file (~/fluentd/fluent.conf) for receiving pfsense syslog and printing them out. 
    <source>
  ### https://docs.fluentd.org/input/syslog
  @type syslog
  port 5141
  bind 0.0.0.0
  <transport udp>
  </transport>
  <parse>
    message_format rfc5424
  </parse>
  tag pfsense
</source>

<match pfsense.**>
  @type stdout
</match>
  2. Run fluentd with the config file. 
    ### https://docs.fluentd.org/installation
/opt/fluent/bin/fluentd -c ~/fluentd/fluent.conf
  3. Enable and configure the remote logging on pfsense host.
    • Status » System Logs » Settings » General Logging Options
      • Log Message Format: syslog (RFC 5424, with RFC 3339 ...)
    • Status » System Logs » Settings » Remote Logging Options
      • Remote Log Servers: fluentd_listening_address:port (Ex: 192.168.1.99:5141)
      • Remote Syslog Contents: Firewall Events
  4. Now you should see the log from pfsense keep comming in. 
    2024-04-25 06:09:34.632899000 +0000 pfsense.local0.info: {"host":"pfsense.willyhu.local","ident":"filterlog","pid":"44226","msgid":"-","extradata":"-","message":"4,,,1000000103,pppoe0,match,block,in,4,0x0,,242,26160,0,none,6,tcp,44,89.248.165.17,125.229.96.130,44961,30129,0,S,3258086147,,1025,,mss"}
2024-04-25 06:09:41.934155000 +0000 pfsense.local0.info: {"host":"pfsense.willyhu.local","ident":"filterlog","pid":"44226","msgid":"-","extradata":"-","message":"4,,,1000000103,pppoe0,match,block,in,4,0x0,,233,61911,0,none,6,tcp,40,45.141.87.109,125.229.96.130,44210,5040,0,S,246121189,,1024,,"}
2024-04-25 06:09:42.959281000 +0000 pfsense.local0.info: {"host":"pfsense.willyhu.local","ident":"filterlog","pid":"44226","msgid":"-","extradata":"-","message":"4,,,1000000103,pppoe0,match,block,in,4,0x0,,107,19362,0,DF,6,tcp,52,177.229.216.18,125.229.96.130,51305,445,0,S,1581211380,,8192,,mss;nop;wscale;nop;nop;sackOK"}
  5. As we’ve configured the message format(rfc5424) in fluentd config, it has already extracted several fields and values for us. However, as you can see. We still have to parse the message field before insert them into Loki server.
    • host: pfsense.willyhu.local
    • ident: filterlog
    • pid: 44226
    • msgid: -
    • extradata: -
    • message: 4,,,1000000103,pppoe0,match,block,in,4,0x0,,111,23330,0,DF,6,tcp,52,115.73.209.120,125.229.96.130,56735,445,0,S,2152994813,,8192,,mss;nop;wscale;nop;nop;sackOK

Parsing the raw message of filterlog

Every kind of log has its own format. In this case, we can find the very detailed explanation of filterlog format at pfsense official document. That <CSV data> in the document is actually the value of the message field. So, let’s break it down!

  1. Install extra fluentd plugins: rewrite_tag_filter and filter_record_modifier. 
    sudo /opt/fluent/bin/fluent-gem install fluent-plugin-rewrite-tag-filter
sudo /opt/fluent/bin/fluent-gem install fluent-plugin-record-modifier
  2. Add following filters to ~/fluentd/fluent.conf after <source> section. 
    ### rewrite tag: pf.${ident}
<match pf.**>
  @type rewrite_tag_filter
  <rule>
    key ident
    pattern ^(.+)$
    tag pf.$1
  </rule>
</match>

### parsing filterlog: basic - before ip-verison
<filter pf.filterlog>
  @type parser
  key_name message
  reserve_data true
  reserve_time true
  <parse>
    @type csv
    keys rule-number,sub-rule-number,anchor,tracker,real-interface,reason,action,direction,ip-version
    parser_type fast
  </parse>
</filter>

### rewrite tag: pf.${ident}.${ip-version}
<match pf.filterlog>
  @type rewrite_tag_filter
  <rule>
    key ip-version
    pattern ^(.+)$
    tag ${tag}.$1
  </rule>
</match>

### parsing filterlog: ipv4 - before destination-address
<filter pf.filterlog.4>
  @type parser
  key_name message
  reserve_data true
  reserve_time true
  <parse>
    @type csv
    keys rule-number,sub-rule-number,anchor,tracker,real-interface,reason,action,direction,ip-version,tos,ecn,ttl,id,offset,flags,protocol-id,protocol-text,length,source-address,destination-address
    parser_type fast
  </parse>
</filter>

### parsing filterlog: ipv6 - before destination-address
<filter pf.filterlog.6>
  @type parser
  key_name message
  reserve_data true
  reserve_time true
  <parse>
    @type csv
    keys rule-number,sub-rule-number,anchor,tracker,real-interface,reason,action,direction,ip-version,class,flow-label,hop-limit,protocol-text,protocol-id,length,source-address,destination-address
    parser_type fast
  </parse>
</filter>

### rewrite tag: pf.${ident}.${ip-version}.${protocol-text}
<match pf.filterlog.*>
  @type rewrite_tag_filter
  <rule>
    key protocol-text
    pattern ^(.+)$
    tag ${tag}.$1
  </rule>
</match>

### parsing filterlog: ipv4/tcp
<filter pf.filterlog.4.tcp>
  @type parser
  key_name message
  reserve_data true
  reserve_time true
  <parse>
    @type csv
    keys rule-number,sub-rule-number,anchor,tracker,real-interface,reason,action,direction,ip-version,tos,ecn,ttl,id,offset,flags,protocol-id,protocol-text,length,source-address,destination-address,source-port,destination-port,data-length,tcp-flags,sequence-number,ack-number,tcp-window,urg,tcp-options
    parser_type fast
  </parse>
</filter>

### parsing filterlog: ipv4/udp
<filter pf.filterlog.4.udp>
  @type parser
  key_name message
  reserve_data true
  reserve_time true
  <parse>
    @type csv
    keys rule-number,sub-rule-number,anchor,tracker,real-interface,reason,action,direction,ip-version,tos,ecn,ttl,id,offset,flags,protocol-id,protocol-text,length,source-address,destination-address,source-port,destination-port,data-length
    parser_type fast
  </parse>
</filter>

### parsing filterlog: ipv6/tcp
<filter pf.filterlog.6.tcp>
  @type parser
  key_name message
  reserve_data true
  reserve_time true
  <parse>
    @type csv
    keys rule-number,sub-rule-number,anchor,tracker,real-interface,reason,action,direction,ip-version,class,flow-label,hop-limit,protocol-text,protocol-id,length,source-address,destination-address,source-port,destination-port,data-length,tcp-flags,sequence-number,ack-number,tcp-window,urg,tcp-options
    parser_type fast
  </parse>
</filter>

### parsing filterlog: ipv6/udp
<filter pf.filterlog.6.udp>
  @type parser
  key_name message
  reserve_data true
  reserve_time true
  <parse>
    @type csv
    keys rule-number,sub-rule-number,anchor,tracker,real-interface,reason,action,direction,ip-version,class,flow-label,hop-limit,protocol-text,protocol-id,length,source-address,destination-address,source-port,destination-port,data-length
    parser_type fast
  </parse>
</filter>

### remove unused fields in this case
<filter pf.**>
  @type record_modifier
  <record>
    log-type ${record['ident']}
  </record>
  remove_keys host,pri,ident,pid,msgid,extradata,rule-number,sub-rule-number,anchor,tracker,reason,ip-version,tos,ecn,ttl,id,offset,flags,protocol-id,length,source-port,data-length,sequence-number,ack-number,tcp-window,urg,tcp-options,class,flow-label,hop-limit
</filter>

### print out results
<match pf.**>
  @type stdout
</match>
  3. Re-run fluentd again you should see the message has been extracted (exclude the fields we deleted) to key/value pair.
    • real-interface: pppoe0
    • action: block
    • direction: in
    • protocol-text: tcp
    • source-address: 157.245.252.5
    • destination-address: 125.229.96.130
    • destination-port: 22
    • tcp-flags: S
    • log-type: filterlog

IP address lookup - The Geolocation

We can find more information from ip address. It’s helpful for further aggregation and visualization.

  1. Install extra fluentd geoip plugin. 
    ### https://github.com/y-ken/fluent-plugin-geoip?tab=readme-ov-file#dependency
sudo /opt/fluent/bin/fluent-gem install fluent-plugin-geoip
  2. Download the geoip database. 
    sudo curl -sSfL -o /opt/GeoLite2-City.mmdb https://github.com/P3TERX/GeoLite.mmdb/raw/download/GeoLite2-City.mmdb
  3. Add geoip filter below to ~/fluentd/fluent.conf before the record_modifier section. 
    <filter pf.filterlog.**>
  @type geoip
  geoip_lookup_keys source-address,destination-address
  geoip2_database /opt/GeoLite2-City.mmdb
  <record>
    source-country ${country.names.en["source-address"]}
    source-country-code ${country.iso_code["source-address"]}
    source-city ${city.names.en["source-address"]}
    destination-country ${country.names.en["destination-address"]}
    destination-country-code ${country.iso_code["destination-address"]}
    destination-city ${city.names.en["destination-address"]}
  </record>
</filter>
  4. Re-run fluentd you should see extra 6 fields that we’ve extracted through geoip filter:
    • source-country: The Netherlands
    • source-country-code: NL
    • source-city: Amsterdam
    • destination-country: Taiwan
    • destination-country-code: TW
    • destination-city: Zhongli District

Writing logs to Loki

We’ve done a lot so far. Time to add an output and send logs to our lightweight log storage: Loki.

  1. Install extra fluentd fluent-plugin-grafana-loki plugin. 
    ### https://grafana.com/docs/loki/latest/send-data/fluentd/
sudo /opt/fluent/bin/fluent-gem install fluent-plugin-grafana-loki
  2. Add output config below to the bottom of ~/fluentd/fluent.conf and don’t forget to update the address of the loki server. The whole fluentd config for this case can be downloaded HERE. 
    <match pf.**>
  @type loki
  url "http://loki:3100"
  include_thread_label false
  drop_single_key true
  <label>
    log_type $.log-type
    real_interface $.real-interface
    direction $.direction
    protocol_text $.protocol-text
    action $.action
    source_address $.source-address
    source_country $.source-country
    source_country_code $.source-country-code
    destination_address $.destination-address
    destination_country $.destination-country
    destination_country_code $.destination-country-code
    destination_port $.destination-port
  </label>
  <buffer>
    flush_interval 10s
    flush_at_shutdown true
  </buffer>
</match>

Visualization - Grafana

Once logs have been written into Loki with proper labels. We can easily create a dashboard in Grafana with Loki datasource. Here is an example dashboard of pfSense filterlog in this case. As usual, try download and import into your own Grafana server.

The Grafana dashboard of pfSense filterlog.