HEAR: Learning to Assess Heartbeat Observability

HEAR teaser — **(a)** Simulated observations with similar motion projections can exhibit markedly different heartbeat observability — geometry alone cannot determine whether the heartbeat component is readable. **(b)** Trained solely on simulated observations, HEAR is applied to real recordings without adaptation and jointly predicts an observability score and heart rate from the phase spectrum of each 25 s window.

Chest-Wall Motion Across Postures

Simulated chest-wall motion on SMPL body meshes posed with CMU MoCap motion-capture data via AMASS. Heartbeat displacement is magnified for visibility; respiration and heartbeat are driven by waveforms extracted from real radar recordings.

Standing

Left: respiration + heartbeat synthesis on a standing posture. Right: close-up of the chest-wall scatterers under the same motion.

Sitting

Left: respiration + heartbeat synthesis on a sitting posture. Right: close-up of the chest-wall scatterers under the same motion.

Lying

Left: respiration + heartbeat synthesis on a lying posture. Right: close-up of the chest-wall scatterers under the same motion.

Four body types, same drive

Male/female × normal/fat body models in the same standing posture, animated by the same respiratory and heartbeat waveforms and shown at a common scale. Each is paired with its chest close-up.

Male, normal

Male, fat

Female, normal

Female, fat

HEAR in Action

Trained only in simulation, HEAR runs zero-shot on real 60 GHz radar recordings. For each 25 s window it predicts an observability score and a heart rate; windows with g ≥ 0.7 report their estimate (green), and low-score windows abstain (red).

Zero-shot selective estimation on Parralejo recordings

A resting record where every window is reported, followed by a post-exercise record where HEAR abstains on windows without a readable heartbeat peak.

Abstract

Contactless heart-rate sensing with millimeter-wave (mmWave) radar requires assessing whether individual measurements support reliable estimation. We study learning to assess heartbeat observability, defined as the readability of the heartbeat component in an acquired phase spectrum, for selective heart-rate estimation. Coherent superposition of scatterer returns can suppress this component even under similar macroscopic observation geometry, motivating assessment directly from acquired measurements. To obtain training supervision across different observability conditions, we develop a controllable multi-scatterer frequency-modulated continuous-wave (FMCW) simulator. Agreement between the dominant heartbeat-band peak and the known heart rate provides an automatic observability label for each simulated measurement. We propose HEAR (Heartbeat Estimation with Assessed Reliability), a compact dual-task Transformer that jointly predicts an observability score and heart rate. Trained solely on simulated observations, HEAR transfers zero-shot to two public real-world datasets collected at 60 and 120 GHz from 134 subjects. On the 120 GHz dataset, score-based selection reduces the HR head's mean absolute error from 17.9 BPM at full coverage to 1.6 BPM at 50% coverage. The complete pipeline achieves an end-to-end processing latency of 50.8 ms on an edge device.

Why Geometry Alone Is Not Enough

Echoes from many chest-wall scatterers superpose coherently within one range bin. Sub-wavelength differences in scatterer ranges set their relative phases, which can preserve or suppress the heartbeat component even when the macroscopic motion projection is identical. Controlled simulation isolates this effect — something physical experiments cannot do, because the same physiological motion cannot be replayed across hundreds of precisely controlled positions.

ROC: geometric prior vs learned gate — **Geometry is a weak prior.** The motion-projection score Q_k separates readable from unreadable observations with AUC 0.635, while the learned gate reaches AUC 0.868 on the same data.

Phase-only perturbation experiment — **Phases decide readability.** With geometry and heart rate fixed, resampling only the sub-wavelength scatterer phases flips HR correctness: 85% in the above-median gate group vs. 23% below the median.

Body-surface map of simulated band quality — **Quality is spatially irregular.** Truth-guided band quality over 492 candidate observation points on the chest surface: high- and low-quality points interleave at neighboring positions rather than forming smooth geometric regions.

HPQ vs simulated quality scatter — **No geometric shortcut.** Across 492 body-surface candidates, the geometric prior is nearly uncorrelated with truth-guided band quality (r ≈ −0.03).

Multi-Scatterer FMCW Simulator

Simulation signal generation pipeline — Paired observations under identical respiratory and heartbeat motion: SMPL body meshes, posed with CMU MoCap data via AMASS, provide scatterer positions and normals; real-derived drive waveforms animate the chest wall; the full FMCW chain (per-scatterer mixing, coherent summation, noise, range FFT, phase processing) produces the phase spectra used for analysis and training.

body models
12 postures × 4 body types

2,240

drive waveforms from self-collected recordings, frequency-shifted over 0.8–3.0 Hz

~600 → 10%

candidate radar positions per body, screened by projection quality

107,520

simulated observation instances with automatic observability labels

Simulated observations and automatic labels

One body and one physiological drive, observed from 630 candidate points on the chest. Each observation is labeled automatically by comparing its dominant heartbeat-band peak with the known heart rate.

Simulated observation samples with automatic labels — Left: all 630 observation points, colored by the automatic label y_gate; readable (green) and unreadable (red) points interleave across the chest. Right: three adjacent pairs (less than 4 cm apart) with nearly identical geometric quality Q_k. For each point we show the phase signal (blue) with its 0.8–2 Hz heartbeat-band component below it, and the corresponding spectrum. In each pair, one spectrum keeps its dominant heartbeat-band peak (▼) at the true heart rate (dashed line), while the other's dominant peak moves away from it.

Scanning the observation point across the chest

Same motion, different observation points

The observation point moves along three rows at heart level, about 2 cm per step, under the same body and physiological drive. Each scanned point is colored by its automatic label y_gate. The strip below the map compares the geometric quality Q_k (line) with y_gate (bars) along the path: Q_k changes gradually, while the heartbeat peak switches between readable and lost from one point to the next.

HEAR: A 5,295-Parameter Dual-Task Transformer

HEAR network architecture — Each candidate heartbeat bin becomes a 2-D token [X(b), f(b)/f_b] — normalized spectral magnitude plus the frequency ratio to the respiration fundamental, which encodes proximity to respiratory harmonics. A 2-layer Transformer (4 heads, d = 16) is mean-pooled into a shared representation; two heads jointly predict the observability score g and the HR distribution. The observability label y_gate = exp(−|b_peak−b_true|/τ) follows the task-performance view of sample quality: it measures whether a peak-based estimator can read the heartbeat frequency from the spectrum.

Zero-Shot Results on Real Radar

HEAR is trained purely in simulation and evaluated zero-shot on two public datasets: Parralejo (60 GHz FMCW, 110 subjects, lying/sitting × rest/post-exercise, 1,408 windows) Dataset Paper and VitalSense (120 GHz, 24 subjects, rest/breath-hold, 480 windows) Dataset Paper. No window from either dataset is used for training, validation, model selection, or threshold tuning.

HR distributions of the two datasets — Ground-truth HR distributions of the evaluation windows: Parralejo includes a post-exercise tail up to 144 BPM, whereas VitalSense stays at or below 83 BPM.

Observability discrimination (gate AUC)

	Simulation	Parralejo 60 GHz	VitalSense 120 GHz
Geometric prior Q_k	0.635	—	—
HEAR gate (zero-shot)	0.868	0.856	0.941

Selective heart-rate estimation (MAE, HR head)

	Full coverage	At 50% coverage
Parralejo 60 GHz	31.0 BPM	19.8 BPM
VitalSense 120 GHz	17.9 BPM	1.6 BPM

One score, many estimators

The same gate (threshold g ≥ 0.7) improves the accuracy of five downstream estimators it was never trained with. For the plain FFT peak, Acc@5 BPM rises from 29.8% to 86.7% in simulation, from 32.0% to 65.8% on Parralejo, and from 81.7% to 94.1% on VitalSense; weighted FFT, autocorrelation, harmonic product spectrum, and VMD estimators improve consistently as well.

Risk–coverage analysis

Risk-coverage curves — Lowering coverage by raising the score threshold monotonically reduces the error of accepted windows on both real datasets, approaching the oracle ordering.

Sim-to-real alignment

t-SNE of learned representations — t-SNE of HEAR's pooled representations: simulated and real observations overlap for (a) Parralejo postures and (b) VitalSense scenarios, supporting zero-shot transfer.

Learning to Assess Heartbeat Observability
for mmWave Heart-Rate Sensing