M0AGX / LB9MG

Amateur radio and embedded systems

Simple hard drive SMART monitoring with collectd

Even though I don't consider myself a datahoarder and follow the 3-2-1 backup strategy I like to know that I can rely on my hard drives and replace them at the first hint of problems. Of course I learned this the hard way. 😁

All modern drives provide basic health data through SMART but you need to put some effort to read the attributes and figure out yourself if the drive is healthy or not. You can view SMART data with smartctl manually, you can set up e-mail notifications for self-tests with smartd, but what really matters to me are the trends. Are there more bad sectors? Some bad sectors can be expected even on brand new drives. When did they start appearing? Are there any more reallocations? Is the drive running hotter? Is there more read/write load on the drive? To gather this information in a longer period (months or years) an automated approach is needed.

I briefly considered running smartctl -a, parsing the output, and storing it into a database but... it is a solved problem! The tool is called collectd and it does what says on the label. It collects all kinds of data from various sources using plugins and puts them into a database. There is a SMART plugin to gather data from the disks. Several database backends are supported. My choice fell on RRDtool because I think it is the easiest one to work with for very simple use cases. RRDtool?! In 2025?! Not Grafana? Not InfluxDB? Not Graphite? Well, if all you need is a bunch of static charts (like this one below) that are refreshed maybe once per week then RRDtool is perfectly adequate. 🙂

Temperature chart

collectd configuration

The collectd package is available in every major distro. collectd has a hundred different plugins that can gather data from almost any source. The default /etc/collectd.conf config file can be quite intimidating but the only relevant plugins are rrdtool and smart. My config is very short:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
BaseDir "/var/lib/collectd"
PIDFile "/run/collectd.pid"

LoadPlugin logfile
<Plugin logfile>
    LogLevel info
    File "/var/log/collectd.log"
    Timestamp true
    PrintSeverity false
</Plugin>

LoadPlugin rrdtool
<LoadPlugin smart>
Interval 3600
</LoadPlugin>

<Plugin rrdtool>
    DataDir "/var/lib/collectd/rrd"
</Plugin>

<Plugin smart>
    Disk "/^[hs]d[a-f][0-9]?$/"
    IgnoreSleepMode true
</Plugin>

The interesting entries in the config are: the Interval in the rrdtool plugin, and Disk in the smart plugin. The interval can be quite long. One hour granularity is more than enough when it comes to SMART data.

The disks can also be specified explicitly like Disk "nvme0n1". The Disk entry can appear multiple times.

Database files

A while after starting collectd (exactly: after Interval seconds) a bunch of files should appear. The exact names will depend on what the SMART plugin can get from the drives in the system.

My files look like this:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
smart-sda
├── smart_attribute-attribute-165.rrd
├── smart_attribute-attribute-166.rrd
├── smart_attribute-attribute-167.rrd
├── smart_attribute-attribute-168.rrd
├── smart_attribute-attribute-169.rrd
├── smart_attribute-attribute-173.rrd
├── smart_attribute-attribute-174.rrd
├── smart_attribute-available-reserved-space.rrd
├── smart_attribute-command-timeout.rrd
├── smart_attribute-end-to-end-error.rrd
├── smart_attribute-endurance-remaining.rrd
├── smart_attribute-erase-fail-count.rrd
├── smart_attribute-head-amplitude.rrd
├── smart_attribute-power-cycle-count.rrd
├── smart_attribute-power-on-hours.rrd
├── smart_attribute-power-on-seconds-2.rrd
├── smart_attribute-program-fail-count.rrd
├── smart_attribute-reallocated-sector-count.rrd
├── smart_attribute-reported-uncorrect.rrd
├── smart_attribute-temperature-celsius-2.rrd
├── smart_attribute-total-lbas-read.rrd
├── smart_attribute-total-lbas-written.rrd
├── smart_attribute-udma-crc-error-count.rrd
├── smart_attribute-uncorrectable-ecc-count.rrd
├── smart_badsectors.rrd
├── smart_powercycles.rrd
├── smart_poweron.rrd
└── smart_temperature.rrd
smart-sdb
├── smart_attribute-attribute-165.rrd
├── smart_attribute-attribute-166.rrd
├── smart_attribute-attribute-167.rrd
├── smart_attribute-attribute-168.rrd
├── smart_attribute-attribute-169.rrd
├── smart_attribute-attribute-173.rrd
├── smart_attribute-attribute-174.rrd
├── smart_attribute-available-reserved-space.rrd
├── smart_attribute-command-timeout.rrd
├── smart_attribute-end-to-end-error.rrd
├── smart_attribute-endurance-remaining.rrd
├── smart_attribute-erase-fail-count.rrd
├── smart_attribute-head-amplitude.rrd
├── smart_attribute-power-cycle-count.rrd
├── smart_attribute-power-on-hours.rrd
├── smart_attribute-power-on-seconds-2.rrd
├── smart_attribute-program-fail-count.rrd
├── smart_attribute-reallocated-sector-count.rrd
├── smart_attribute-reported-uncorrect.rrd
├── smart_attribute-temperature-celsius-2.rrd
├── smart_attribute-total-lbas-read.rrd
├── smart_attribute-total-lbas-written.rrd
├── smart_attribute-udma-crc-error-count.rrd
├── smart_attribute-uncorrectable-ecc-count.rrd
├── smart_badsectors.rrd
├── smart_powercycles.rrd
├── smart_poweron.rrd
└── smart_temperature.rrd
smart-sdc
├── smart_attribute-attribute-165.rrd
├── smart_attribute-attribute-166.rrd
├── smart_attribute-attribute-167.rrd
├── smart_attribute-attribute-168.rrd
├── smart_attribute-attribute-169.rrd
├── smart_attribute-attribute-173.rrd
├── smart_attribute-attribute-174.rrd
├── smart_attribute-available-reserved-space.rrd
├── smart_attribute-command-timeout.rrd
├── smart_attribute-end-to-end-error.rrd
├── smart_attribute-endurance-remaining.rrd
├── smart_attribute-erase-fail-count.rrd
├── smart_attribute-head-amplitude.rrd
├── smart_attribute-power-cycle-count.rrd
├── smart_attribute-power-on-hours.rrd
├── smart_attribute-power-on-seconds-2.rrd
├── smart_attribute-program-fail-count.rrd
├── smart_attribute-reallocated-sector-count.rrd
├── smart_attribute-reported-uncorrect.rrd
├── smart_attribute-temperature-celsius-2.rrd
├── smart_attribute-total-lbas-read.rrd
├── smart_attribute-total-lbas-written.rrd
├── smart_attribute-udma-crc-error-count.rrd
├── smart_attribute-uncorrectable-ecc-count.rrd
├── smart_badsectors.rrd
├── smart_powercycles.rrd
├── smart_poweron.rrd
└── smart_temperature.rrd

You can see that they are identical for every drive. This is only because all my SSDs are from WD. Some files have a self-explanatory name. Some files don't have a human-readable name and only contain the attribute number. I was able to find out the meaning of the attributes by running smartctl -a on the drives. It seems that the drive database of smartmontools and the smart plugin is slightly different.

Altogether the files consume around 43 MB on my machine.

Plotting the data

One of the weaker spots of the collectd and RRDtool combination is that you have to write your own plotting scripts. The tools I have found (as of 2025) were less usable than doing raw shell scripting. Fortunately, plotting with RRDtool is not too hard so I made this shell script to generate the images:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
#!/bin/bash
set -e

DISK_COLORS=(0000FFB4 00FF00B4 FF0000B4)
DISKS="smart-sda smart-sdb smart-sdc"

DIMENSIONS="-w 650 -h 200"
OUT_DIR=/tmp/rrd_plots
SRC_DIR=/var/lib/collectd/rrd/hostname

mkdir -p $OUT_DIR

make_smart_graph(){
        TITLE=$1   # function
        SERIES=$2  # arguments
        OUT_FILE=$3
        BASE_DIR=$4
        SRC_FILE=$5
        TS_START=$6
        shift 6 #now $1 is the first drive to plot

        cmd="rrdtool graph $OUT_FILE -t \"${TITLE}\" ${DIMENSIONS} --start=$TS_START --legend-position=south --full-size-mode"
        index=0
        for device in "$@"; do # iterate over function arguments that are left
                cmd="${cmd} DEF:ds${index}=$BASE_DIR/$device/$SRC_FILE:$SERIES:AVERAGE "
                cmd="${cmd} LINE2:ds${index}#${DISK_COLORS[$index]}:\"${device}\""
                ((++index)) #pre-increment to avoid returning 1 and ending the script by set -e
        done
        eval $cmd # to see the full command you can add: echo $cmd
}

TS_START=-30d # plot last 30 days

make_smart_graph "SMART temperature" value $OUT_DIR/smart-temperature${TS_START}.png \
        $SRC_DIR smart_temperature.rrd $TS_START $DISKS
make_smart_graph "SMART endurance remaining" current $OUT_DIR/endurance-remaining${TS_START}.png \
        $SRC_DIR smart_attribute-endurance-remaining.rrd $TS_START $DISKS
make_smart_graph "SMART bad sectors" value $OUT_DIR/bad-sectors${TS_START}.png \
        $SRC_DIR smart_badsectors.rrd $TS_START $DISKS
make_smart_graph "SMART total LBAs written" pretty $OUT_DIR/total-lbas-written${TS_START}.png \
        $SRC_DIR smart_attribute-total-lbas-written.rrd $TS_START $DISKS
make_smart_graph "SMART total LBAs read" pretty $OUT_DIR/total-lbas-read${TS_START}.png \
        $SRC_DIR smart_attribute-total-lbas-read.rrd $TS_START $DISKS
make_smart_graph "SMART total bad blocks" pretty $OUT_DIR/total-bad-blocks${TS_START}.png \
        $SRC_DIR smart_attribute-attribute-169.rrd $TS_START $DISKS
make_smart_graph "SMART program fail count" pretty $OUT_DIR/program-fail-count${TS_START}.png \
        $SRC_DIR smart_attribute-program-fail-count.rrd $TS_START $DISKS

As usual, shell syntax is maybe not the prettiest but gets the job done. The single function can generate an arbitrary chart for multiple drives by concatenating the command that goes into RRDtool. All that has to be changed is in the DISK_COLORS and DISKS variables. Rest of the script should be self-explanatory. I run the script once per day from cron.

Final result

Thanks to the plots I can see that the temperature of the drives is not constant. It may be varying due to the load on the whole machine, not necessarily the drives themselves. The correlation seems to be 100% (no wonder when the drives are sitting close together).

Temperature chart

The most interesting plot is the one that shows amount of written data (blocks). You can see the variable load on sdc. This plot tells me basically the "SSD consumption rate".

Total blocks written chart

Read side is less interesting. sda and sdb correlation is visible because they are part of the same RAID.

Total blocks read chart

All drives seem healthy with 100% endurance remaining and zero problems. 🙂

Endurance remaining chart

Program fail count chart

Bad sectors chart

Total bad blocks chart