Technology
From raw audio capture to the visual reading of the 12 Zones — what is standard signal processing, what is proprietary framework, and where each one ends.
Step 1
Every session begins with a 60-second calibration. During this window, the system establishes the mean and variability of the patient's fundamental frequency (F0) at rest, maps the noise profile of the room and microphone, and locks (baseline_locked) the reference values that will be used for every comparison in the session.
This exists because absolute acoustic analysis makes no clinical sense: a naturally low voice cannot be mistaken for psychomotor retardation, and a high-pitched voice cannot be read as hyperactivation. Each patient is compared only against themselves.
Baseline F0 (mean and standard deviation), power spectral density per band (via FFT in the browser — Web Audio API, no DSP library on the backend), background noise, and the initial state of the 468 tracked facial landmarks.
Session state implemented as a state machine CALIBRATING → LIVE → ENDED, with a BASELINE_LOCK action triggering the lock of reference values (froid-dashboard/src/pages/LiveSession.tsx).
Step 2
At every analysis window, FROID extracts a set of vocal features and compares each one against the locked baseline.
Cepstral coefficients on the Mel scale and their derivatives (Δ, ΔΔ). In the anchor study, MFCC7 under negative content is predictive of the depression scale (β=0.90) and MFCC9 correlates inversely with Hamilton somatic anxiety (β=−0.45) — which is why FROID reads ΔΔMFCC9 as a marker of accelerating laryngeal tension.
Fundamental frequency and zero-crossing rate, extracted continuously and compared to baseline variability.
Normalized internal indices (not classic clinical jitter% / shimmer dB) derived from scaled ZCR and the coefficient of variation of the RMS envelope — they signal relative vocal instability.
In the production code (LiveSession.tsx), these metrics are explicitly labeled as proxies, with a documented unit:
In other words: they do not directly equate to jitter in % or shimmer in dB as defined by Boersma & Weenink (Praat). They are a derived proxy, used as a relative signal of instability — not as a normative clinical value. Validated physical extraction (Praat-equivalent) is on the roadmap.
Sustained spikes in the Delta-Delta derivative acceleration (ΔΔMFCC9 > 1.8) signal reflexive vocal cord micro-contractions and serve as a biophysical trigger for somatoaffective friction. The 1.8 is a calibratable engineering threshold, not a published constant.
Step 3
468 facial landmarks are tracked by local computer vision, classified into Action Units (AUs) according to the Facial Action Coding System, with temporal modeling of onset, apex, and offset.
When a facial configuration (e.g., AU12 without AU6, or sustained AU23/AU24) coincides with a relevant acoustic deviation in a zone, the system applies a dissonance multiplier to that zone's reading.
Binary form applied in production when the facial dissonance flag triggers. The value 2.5 is a FROID product parameter, not derived from a published study — treat it as calibratable engineering heuristic, not a scientific constant. The full IDM assembly (average of the 12 zones × M_fac) and the continuous form of the multiplier are in Step 5.
The classic hypothesis that absent AU6 indicates a forced smile has partial support in the literature, but there is also recent research that directly contests it, finding that AU6 relates more to smile intensity than to emotional authenticity. FROID uses this configuration as one factor among several for dissonance, not as a standalone lie detector.
Step 4 · Proprietary framework
The visual, interpretive layer that organizes all the preceding signals into a circular map of 12 zones, each associated with a clinical theme dichotomy.
A proprietary visualization interface that distributes vocal spectral energy across 12 bands, each tuned to a note of the chromatic scale and associated by FROID with an interpretive clinical theme (e.g., Zone 3 — "Sadness vs. Inner Peace"). The physics behind it (power spectral density via FFT) is real and measurable.
There is no peer-reviewed literature validating the specific mapping between a narrow Hz band and a psychological theme. The structure is an adapted transposition of commercial voice biofeedback technology. We treat this explicitly as a FROID product heuristic.
Step 5 · Proprietary framework
The human brain does not run Fourier Transforms or track 468 facial landmarks at 30 FPS while sustaining the therapeutic alliance. FROID solves this bottleneck through dimensionality reduction: it fuses voice, face, and semantics into two vectors that run continuously on the care panel.
The "speedometer" of the session's biological energy — quantifies, on a 0–100 scale, the patient's expressive intensity and psychomotor vitality every second.
Combines the amplitude and variability of three channels: acoustic energy and F0 variability (voice), amplitude and contraction speed of Action Units (face), and density of high emotional-impact terms detected by NLP (semantics). A dynamic per-channel reliability factor discards spurious variations, such as a cough or an abrupt change in room lighting.
Clinical reading: a sustained drop points to lethargy or depressive motor retardation; frequent spikes indicate hyperactivation, anxious agitation, or racing thoughts.
The "compass" that points to the direction and coherence of that energy — assesses whether the patient's reactions are congruent with the content being discussed, or whether there is a symmetry break revealing resistance, repression, or somatization.
Compares instantaneous readings against the individual baseline locked in the first 60 seconds. When the facial and vocal channels contradict each other — sad content with vocal infra-tremor (5–12 Hz) but only a voluntary smile (AU12 without AU6) — it applies the Facial Dissonance Multiplier, shifting the index toward alert colors.
Clinical reading: locates emotional "hot spots" — reveals Somatoaffective Dissonance (False Calm), when stress has been repressed from the face but keeps tensioning the larynx.
σ is the sigmoid that compresses the result into 0–100. |zᵢ(t)| is the magnitude of the standardized deviation (z-score) of the channel against the 60-second baseline; rᵢ(t) ∈ [0,1] is the channel's dynamic reliability; wᵢ are weights calibrated through phonation engineering to balance each sensor.
At rest or in congruence, M_fac = 1.0. At maximum dissonance threshold (pure social smile, no orbital contraction, Conf_apex ≥ 0.75) the multiplier reaches the ceiling of 2.5. Current production applies the binary form of this multiplier (2.5 when the dissonance flag triggers, 1.0 otherwise — see Step 3); the continuous expression above is the reference model. Implemented in froid-engine.ts → calculateIDM().
Engineering honesty: the weights, the fusion coefficients, and the 2.5 itself are product parameters refined in the lab — qualified clinical hypotheses, not published scientific constants. Definitive population validation depends on the anonymous Data-Froid (k-anonymity + one-way hashes). See framing in Science.
Step 6 · Proprietary framework
In testing with professionals, a screen that changes every few seconds proved too fast to assess all the metrics with clinical calm. FROID's response: raw processing continues at high resolution (1-second cadence), but the visual presentation is stabilized within a configurable clinical window — no freezing, and no loss of accuracy.
Every 1 minute, FROID closes a technical micro-window. The screen shows a 5-minute moving clinical window, made up of the last 5 micro-windows — consolidated by temporal weighted average, never a simple average: the current minute weighs more than the previous ones.
The professional sees stability, but FROID keeps reacting gradually to what is happening right now.
Real time — no visual freezing
1 minute — light clinical window
3 minutes — dynamic supervision
5 minutes — recommended default
7 minutes — slower/more reflective session
"Real time" means charts and indicators follow the available technical cadence (≈1s/10s depending on the metric's origin) — it is not absolute laboratory real time, but rather the absence of clinical screen freezing.
The panel displays Clinical window: 5min and the counter Next update in 03:42, with the Update now button always available.
Safety exception: a critical alert breaks the stabilization and updates the screen immediately, ahead of schedule.
m_k is the consolidated value of technical micro-window k (1 minute). Raw processing continues at 1-second cadence; only the presentation layer is stabilized.
Proprietary reference: FROID_Estabilizacao_Clinica_Da_Tela.md — documents the metric, the math, and the clinical rationale, and is part of the knowledge base consulted by FROID Explains. The session's semantic segments are not affected by this module.