Real-Time Data Logging

Analogy categoryAnalogies

The Anatomy of a Data Logger

A data logger records measurements over time so they can be analyzed later, and a good one has four concerns: acquiring samples at a consistent rate, timestamping them accurately, storing them durably, and doing all of that without missing readings. On Arduino the storage target is usually an SD card via SPI for large capacity, the internal EEPROM for a handful of values that must survive power loss, or a network upstream to the cloud. The classic beginner mistake is using delay() to space samples, which blocks the processor and makes the true interval drift; robust loggers use millis() timing so the CPU stays free to service the SD card and sensors between samples.

🏏

Cricket analogy: Like a scorer's book that must record every ball with its over number and outcome in real time — miss one delivery and the whole scorecard's timeline is corrupted for later analysis.

Timestamps and the Real-Time Clock

Arduino has no concept of wall-clock time on its own; millis() only counts milliseconds since the last reset and stops when power is lost. For meaningful timestamps you add a real-time clock module such as the DS3231, a temperature-compensated chip accurate to a couple of minutes per year, backed by a coin cell so it keeps time even when the Arduino is off. You talk to it over I2C, read the current date and time, and prepend it to each logged row. Without an RTC your CSV rows are relative and become useless the moment the board resets; with one, every measurement is anchored to a real calendar timestamp you can correlate with external events.

🏏

Cricket analogy: Like a match clock that keeps running across a rain delay so play resumes at the correct real time — the DS3231's coin cell preserves the true clock even when the main system is off.

cpp

#include <SPI.h>
#include <SD.h>
#include <Wire.h>
#include <RTClib.h>       // Adafruit RTClib for DS3231

RTC_DS3231 rtc;
const int CS = 10;         // SD card chip-select
const unsigned long INTERVAL = 1000;   // 1 Hz logging
unsigned long lastSample = 0;

void setup() {
  Serial.begin(9600);
  Wire.begin();
  rtc.begin();
  if (!SD.begin(CS)) { Serial.println(F("SD fail")); while (1); }
  // write a header if the file is new
  if (!SD.exists("log.csv")) {
    File f = SD.open("log.csv", FILE_WRITE);
    f.println(F("timestamp,temperature,light"));
    f.close();
  }
}

void loop() {
  unsigned long now = millis();
  if (now - lastSample < INTERVAL) return;   // non-blocking timing
  lastSample = now;

  DateTime t = rtc.now();
  int temp  = analogRead(A0);
  int light = analogRead(A1);

  File f = SD.open("log.csv", FILE_WRITE);
  if (f) {
    char stamp[20];
    snprintf(stamp, sizeof(stamp), "%04d-%02d-%02d %02d:%02d:%02d",
             t.year(), t.month(), t.day(),
             t.hour(), t.minute(), t.second());
    f.print(stamp);  f.print(',');
    f.print(temp);   f.print(',');
    f.println(light);
    f.close();        // flush to card so data survives power loss
  }
}

Writing to an SD card is relatively slow and involves buffered blocks. Calling f.close() after each row forces a flush so a sudden power loss cannot lose the most recent data, at the cost of throughput. For high sample rates, keep the file open and flush() periodically instead, accepting a small window of potential data loss.

Choosing a Storage Medium

The right storage depends on volume and lifetime. SD cards give gigabytes of space and human-readable CSV files you can pop into a spreadsheet, but need clean power and a proper file close to avoid corruption. The internal EEPROM holds only about 1 KB on an Uno and, critically, is rated for roughly 100,000 write cycles per cell, so it suits configuration and occasional milestones — not a value written every second, which would wear a cell out in about a day. For always-connected loggers, streaming rows to a cloud time-series database or an MQTT topic removes local storage limits entirely but makes you dependent on the network. Many robust designs combine approaches: buffer to SD when offline and sync to the cloud when a connection returns.

🏏

Cricket analogy: Like team selection for conditions — a spinner (EEPROM) suits a few key overs, a pace battery (SD) handles the bulk workload, and a cloud feed is the broadcast that reaches everyone; you pick per the situation.

The internal EEPROM is rated for only about 100,000 write cycles per byte. Writing a logged value to the same EEPROM address every second would exhaust that cell in roughly a day and permanently damage it. Reserve EEPROM for configuration and infrequent milestones, and use wear-leveling (rotating addresses) if you must write often.

A good logger handles consistent sampling, accurate timestamps, durable storage, and never blocking long enough to miss data.
Use millis()-based non-blocking timing, not delay(), so the CPU can service sensors and storage between samples.
Arduino has no wall clock; add a DS3231 RTC with a coin cell for accurate, power-loss-proof timestamps over I2C.
SD cards over SPI give gigabytes and CSV convenience but need a proper file close to survive power loss.
Internal EEPROM is tiny (~1 KB) and wears out after ~100,000 writes per cell, so use it only for config/milestones.
Cloud/MQTT streaming removes local limits but depends on the network; hybrid designs buffer to SD and sync when online.
Write a CSV header row once so the log opens cleanly in a spreadsheet for later analysis.