Synchronized Devices

Instrumentation for psychological experiments can be simple or complex. It can be as simple as a pencil-and-paper survey, or as complex as an fMRI outfitted with custom electronics. The researchers I work with generally want something on the complex side. The most common challenge I've seen is the problem of time-aligning heterogeneous data.

Researchers I work with usually want to measure some combination of:

  • intentional behaviors (e.g. pressing buttons, answering questions)
  • physiological changes in response to stimuli (e.g. heart rate, posture, fMRI data, facial expression, gaze)

My job is to help researchers build new instrumentation or get different kinds of instrumentation to work together.

Most devices I see in this line of work connect over USB. FireWire, serial, and custom interfaces are becoming increasingly rare. Wireless devices are a growing minority. Networked (Ethernet, Wi-Fi) devices are very rare. The kind of device varies greatly, from humble laptop keyboards to multi-channel specialized sensors like ECG machines.

Example

Most researchers are pragmatic. They like to start simple and add complexity as needed. Usually this approach wins because the system they end up with does exactly what they need and nothing else. But sometimes, especially when precise timing is desired, a more top-down design is better.

Let's follow an example scenario to illustrate.

The initial experiment design revolves around a subject solving a crossword puzzle in a room. While they solve the puzzle, someone knocks on the door. The experiment looks at how they respond to the knock. The hypothesis is that people are less likely to answer the door if they are told they must complete the crossword in 10 minutes.

Additionally, the reasercher wants to record the subject's heart rate during the experiment. A particular brand of heart rate monitor has been used successfully in other experiments and is generally regarded as accurate. The monitor connects over USB and interfaces with a closed-source program. From within this program, we can view waveforms and graphs of the heart rate, and generate text-format recordings of the monitor.

Task: Measure the time between the knock on the door and an increase in heart rate.

This task, like so many in the real world, requires aggregating heterogeneous data from heterogeneous sources. One data source is our heart rate monitor. Another is the PC's system clock.

Approaches

There are several approaches to choose from depending on the problem at hand. One is to embed an out-of-band signal into multiple data streams. For example, to synchronize mulitple cameras one could put an infrared blinker within view of all the cameras. This infrared marker would be invisible to humans participating in a scene, but visible to the cameras. After capturing a scene, we can (manually or programmatically) time-align the cameras on the blink rate.

In our heart rate monitor + system clock example, we need a different approach. The software interface is output-only and we can't modify it. The USB interface is handled by the host operating system. Obtaining precise (1ms resolution) timing from a user-space application is not practical, because USB interrupts will be serviced on non-deterministic intervals (at least from the perspective of a user-space app).

Sending a "start" signal to the USB device also suffers from user-space non-determinicity, because getting a timestamp from the system is one system call, and sending it to a device is another, which requires jumping into and out of the kernel at least twice, incurring arbitrary delays each time.

The approach I've used to date has been to encode timestamps into the device's data output stream. Most microcontrollers have a real-time clock which can be read from and written to output data (such as a serial stream or raw HID byte array). This provides a precise clock that solves half of the problem. The other half is matching this clock up with a non-real-time clock such as the PC system time. For 1ms precision it's not enough to emit `milliseconds since program start` because there is latency between when the user tells the program to start, and when the microcontroller receives the message. To synchronize the two clocks, we have to compute the latency on each side.

The procedure would look something like this:

PC: hi microcontroller, begin an experiment; The time is HHMMSS.SSS (system timestamp).

uC: hi PC, I'm beginning an experiment at HHMMSS.SSS and resetting my clock to 0.

uC: here's the first data point at `0 + t1`.

PC: I got your first data point at `HHMMSS.SSS + t2`

uC: here's another data point at `0 + t3`, with a round-trip latency of `t2 - t1`

And so on. This means that the experimental data contains both the wall clock accuracy and the real time clock precision.

Future Work

The approach I'd like to implement soon is an "in-line real-time wrapper". In the heart rate example, we don't have access to the internals of either the hardware or software, so the "wrapper" is the USB channel itself. By running the USB device through a deterministic, real-time kernel such as an embedded Linux variant, we can capture its data, timestamp it (within the context of the real-time OS), and export this timestamped data separately from the USB data.