RHEL 7 UDP metrics into splunk metrics index

We were discussing this on splunk-usergroups slack, and I said I should post it here and vraptor and dawnrise urged me to do so quickly — so here I am.  (Thanks vraptor and dawnrise!)

First up, a script to use the nstat tool to grab some kernel UDP metrics and write them out in a format compatible with Splunk’s metrics store:

typeset -A MAPPER
populate_metrics() {
  NOW=`date +%s`
  printf $FORMAT "metric_timestamp" "metric_name" "_value"
  while read METRIC VALUE JUNK; do
        printf $FORMAT "$NOW" "${MAPPER[$METRIC]}" "$VALUE"
  done <  <( nstat -z ${!MAPPER[@]} | egrep -v "^#" )

The relevant inputs.conf:

index = my_metrics
sourcetype = metrics_csv
interval = 60

A search that uses it:

| mstats span=5m sum(_value) as value where index=my_metrics metric_name=udp.packets_received by host 
| xyseries _time host value

Obligatory picture:

4 thoughts on “RHEL 7 UDP metrics into splunk metrics index

  1. This is awesome Duane! Such a clever way to collect kernel UDP metrics, and now I can see almost unlimited awesomeness with bash scripts and the use of the metrics_csv sourcetype!

    • Thanks Luke! Truth told, this was our quick hack to be able to keep our syslog servers measured as healthy until we could get statsd rolled out. But, it’s still something that I was really proud of for a few minutes of work. One neat thing is that the nstat command does a good job of giving you deltas from the most previous run, so you don’t have to do any streamstats magic.

  2. Hi Duane, this is awesome. Thanks for sharing.

    One question though… the script is generating a splunk error on line 15; and specifically one of the two “<“ after done. Any thoughts on how I can fix it?

Leave a Reply

Your email address will not be published. Required fields are marked *