[" "] // Resonance Loop

Resonance Loop explores what happens when external observation of the self becomes internal dialogue. Standing silently before a camera, the artist is subject to a custom system that reads microexpressions in real time, translating them into inner monologue fragments synthesized in the artist's own voice. Through headphones, the artist hears these system generated "thoughts" and reacts. These reactions become new facial data, generating new monologue. The loop continues.

This creates an impossible circuit: a living subject reduced to 468 facial landmark coordinates, transformed into language, then as synthesized voice. A fundamental gap exists between what the algorithm captures as expression and what it presumes as interiority. Yet the system translates this into definitive narrative.

The work doesn't try to step outside the problem of using technology to critique technology. It makes that contradiction the structure itself. The system operates through processes familiar from surveillance facial recognition, emotion detection interfaces, and affective computing applications. But here, the artist is both the subject and the observer. Audiences watch the artist being watched, witnessing how algorithms translate lived experience into data, then back into representation. In this act of watching, they become implicated in the same mechanisms of observation the work examines. When are we observers and when are we participants in systems of capture?

System Architecture

Recursive Loop Structure

The loop is intentionally closed. The system takes the artist's reactions to its own outputs as new inputs.

System Architecture

Resonance Loop operates through the following integrated pipeline:

1. Real-time facial tracking: MediaPipe extracts 468 facial landmarks at 30fps.
2. XYZ meta-coordinate transformation: Landmark movement is interpreted as directionality, intensity, and temporal cohesion.
3. Narrative phase classification: Temporal changes in XYZ values are mapped to six phases that describe different positions within unfolding affect.
4. Custom ML model application: A Gaussian Mixture Model (GMM) and Hidden Markov Model (HMM) trained on paired expression–word data provide a small dictionary of words associated with stable microexpression states and their transitions.
5. Word selection: Narrative phase output is combined with neighbouring cluster words to form a short candidate set for inner monologue.
6. LLM Based Inner monologue generation: Prompt engineering produces brief, fragmentary internal speech shaped by the current phase.
7. Voice synthesis: Text is rendered through a TTS model trained on the artist's voice.
8. Recursive feedback: The artist's facial reactions to the generated voice are tracked again as new input.

Pre-Verbal Affective Dynamics (PVAD)

PVAD models how affective expression unfolds over time by linking microexpressions, linguistic tendencies, and temporal behaviour into a continuous loop. Facial microexpression data was captured simultaneously with natural speech and later processed to align specific moments of movement with the words spoken at those moments, producing a paired dataset.

A GMM identifies similarity structures within this data, while an HMM describes how these structures transition over time. Together, they produce a limited dictionary of words statistically associated with stable expressive states. At runtime, this dictionary is combined with real-time XYZ analysis and narrative phase classification. The resulting internal speech is played back, prompting new microexpressions that continue the loop.

468 Landmarks and the XYZ Meta-Coordinate System

The 468 landmarks extracted through MediaPipe are transformed into a three-dimensional representation of temporal dynamics:

• X-axis (Directionality/Flow): Expansion versus contraction of expression, computed from frame-to-frame changes in features such as mouth width and brow height.
• Y-axis (Intensity/Amplitude): Speed and strength of facial change. Aggregates movement velocity, asymmetry, and z-depth tension.
• Z-axis (Temporal Cohesion): Duration and layering of states, increasing with sustained expression, directional reversals, and hesitation patterns.

Six Narrative Phases

PVAD identifies six phases that describe temporal positions within affective experience:

1. Neutral: Minimal facial activity. XYZ characteristics: X≈0, Y<0.5, Z<0.3. Baseline between affective events.
2. Sensory Reaction: Sudden intensity spike. XYZ: Y>15 OR ΔY>8. Immediate, involuntary response to stimulus, occurring before conscious emotional regulation.
3. Pre-Verbal Hesitation: Microexpressions before speech commitment. XYZ: |X|>0.5, Y<2.0, Z<1.0. The "thinking face": contemplation, uncertainty, processing. The gap between internal experience and external expression.
4. Post-Verbal Residue: Expressions lingering after verbal utterance. XYZ: Y>2.0, Z>2.0. Emotional "hangover", feelings persisting beyond the immediate moment. What remains unsaid but still felt.
5. Sharp Transition: Brief, intense changes. XYZ: Y>1.5, Z<1.0. Punctuated shifts: decisions, rejections, conclusions.
6. Systemic Shift: Broader baseline changes. Default classification for patterns not fitting other categories.

Custom Machine Learning Model

Facial microexpression data was captured alongside natural speech and later aligned at the level of specific moments. These expression–word pairs trained a GMM to identify similarity structures and an HMM to describe their transitions over time. Clusters emerge from statistical proximity across both expression features and associated word tendencies.

Dual-Signal Semantic Analysis

Brief user utterances are supplemented through two signals:

• Temporal sequences: Probabilistic chains based on observed pairing patterns.
• Cluster similarity: Neighbouring words in feature space, maintaining thematic coherence.

These signals produce a short candidate set for internal speech.

semantic_coherence = (cluster_similarity × 0.5) + (temporal_probability × 0.5)

Inner Monologue Generation

XYZ values and narrative phases condition a language model to produce fragmentary internal speech. Prompt engineering shapes:

• Incomplete phrasing
• Dropped subjects
• Interjections
• Private verb endings
• Short length

This style reflects immediate, non-communicative inner speech.

Linguistic Style Mapping

Changes in XYZ influence linguistic qualities:

1. High Y: Single-word utterances.
2. Low Y: Soft, hesitant phrasing.
3. High Z: Topic drift and lingering reflection.
4. Phase-specific: Direct perception, hesitation, trailing memory, etc.

Philosophical Implications: The Affective Mirror

Ambiguity as Resistance in Affective Computation

The PVAD framework builds on the ongoing debates in affective computing, where researchers increasingly recognise that emotion cannot be fully formalised or captured through discrete categories. Rather than viewing this limitation as a technical flaw, PVAD makes it the central condition of the system. Emotion is treated not as information to be resolved but as a dynamic field of unfolding uncertainty.

In most research environments, affective systems are optimised to minimise ambiguity and maximise prediction accuracy. In this work, ambiguity itself becomes method. The continuous loop between face, language, and voice does not aim for stable classification but sustains the tension between sensation and interpretation. This refusal of closure functions as a quiet form of resistance to the instrumental logic of affective AI, which depends on certainty to operate.

Within the art context, PVAD serves as a mirror that reflects the epistemological limits of machine perception. It exposes how data-driven systems convert lived immediacy into representation and how that conversion reshapes the sense of self. The artist's participation is not an act of self-portraiture but of self-translation under algorithmic observation. Emotion, here, is not private interiority but a relational negotiation between the biological, the linguistic, and the computational.

By bringing this process into public view, Resonance Loop transforms what would normally occur within a closed research environment into a shared inquiry into perception and ethics. As viewers watch the system unfold, they become implicated in the same question: when affect is made visible, what remains unseen? In Resonance Loop, that question extends into the very technologies of observation themselves.

Systems of Capture: Surveillance and Self-Observation

Resonance Loop operates in a public setting: a book café in Yeouido, Seoul's financial district. Surrounded by offices, screens and surveillance cameras, the work sits within daily life where reading, rest and observation coexist. Surveillance here is not exceptional but atmospheric, quietly woven into how people appear to one another.

The system mirrors facial recognition and affect detection technologies used in digital surveillance and consumer analytics. Here, those mechanisms are exposed as aesthetic experience. The algorithm's gaze becomes visible, its interpretive logic audible through the artist's continuous observation.

This blurs voluntary and involuntary visibility. The artist chooses to be seen, yet that choice reveals how limited agency becomes under constant observation. The algorithm reads the face, translates it into words, returns those words as voice. Each cycle erodes the boundary between self-expression and system interpretation.

By placing this process in public space, the work reveals surveillance as an ordinary affective condition. Observation is quiet, continuous, relational. Viewers share the same informational field as the artist, aware that presence itself becomes data.

The work reframes surveillance beyond control mechanisms. It becomes a way of sensing modern subjectivity, where to exist is already to be translated, measured, inferred upon. What remains is not spectacle but mutual exposure, where watching becomes participation and recognition.