Pulse2Symphony

Pulse2Symphony is a mobile pipeline that turns a smartphone camera into a biosignal sensor for personalised music generation. It extracts heart-rate-variability (HRV) features from a short photoplethysmography (PPG) recording, classifies the user's affective state in a two-dimensional valence-arousal space, and generates MIDI music conditioned on that state through a Transformer decoder trained on REMI tokens.

HRV is the variation in time between consecutive heartbeats. Short-term HRV statistics like SDNN, RMSSD, and the LF/HF ratio correlate reliably with autonomic nervous-system balance and have been used as proxies for arousal and, less directly, for valence. A smartphone camera with the flash on can recover the PPG waveform from fingertip perfusion, making HRV extraction possible without specialised hardware.

Russell's circumplex model places affective states on two axes (valence and arousal), a compact representation that maps naturally onto music-generation conditioning. REMI tokenisation represents music as structured events (position, bar, note, duration) that a Transformer can learn from efficiently.

The pipeline has three stages. The first extracts SDNN, RMSSD, and LF/HF from the PPG signal using standard time- and frequency-domain methods. The second is a CNN-LSTM that maps the HRV feature window onto a point in Russell's valence-arousal plane. The third is an emotion-conditioned Transformer decoder that generates REMI token sequences from the predicted affect coordinates, which are decoded back into MIDI.