Political Analysis 3
a word error rate (WER)1 of 0.05. A
The authors caution these validations are performed in laboratory
conditions and do not account for levels of noise or crosstalk often present in political speech. Using the European Union corpus for English, they find WER of 0.03 with the YouTube API and 0.21 with the Google API, with generally worse results for other languages, ranging from WER of 0.10 to 0.26 for French and German. They demonstrate that "automatic transcription tools have reached accuracy levels that make them useful for study of parliamentary debates, campaign speeches, and intergovernmental negotiations" (Proch, Windisch-Wiedemann, and Wiedemann 2018). They report that APIs are "different depending on length and learn about legal issues related to data protection, copies of the audio files are sent to YouTube or Google servers." Recent sophisticated models, like OpenAI's Whisper, have been analyzed for accent and speaker differences, Graham and Roll (2021) found Whisper performed best for U.S. and Canadian accents but poorly for Vietnamese and Thai accents in English, and for native but regional accents, such as the British Leeds accent, with a match error rate (MER) of almost 100%.
Our work is motivated by addressing alignment, feature extraction, and classification
of speech and emotion using modern audio embeddings. As a case study, we analyze televised U.S. presidential debates since first airing in 1960, until 2020, including vice-presidential debates. We obtained 47,150 sentences from the debate transcripts (38,649 from candidates and 8,501 from moderators, panelists, or audience members) initially unlabeled to their respective videos. We obtained 243,023 or 4,050 min or 67.5 h across 45 videos. In these videos, 34 participants (16 Democrats, 15 Republicans, and 3 Independents) spoke in 1,000 sentences. We used the 2020 debate as a case study, but we also analyzed the 1960, 1964, 1968, 1972, 1976, 1980, 1984, 1988, 1992, 1996, 2000, 2004, 2008, 2012, and 2016 debates.
We begin by explaining core audio data concepts relevant to political scientists (Section 3). We present four progressive applications to demonstrate ways of leveraging audio data for political science research, namely audio-text forced alignment (Section 4.1), analysis of individual speaking styles using low-level features (Section 4.2), construction of custom-made machine learning models for classification tasks (Section 4.3), and off-the-shelf machine learning models for emotion recognition (Section 4.4). We provide all replication codes in Python, transcripts, alignment data, and results in the Diverse record Metadata Repository (https://doi.org/10.5281/zenodo.4800000).
To fully understand the affordances of audio as data, it is necessary to appreciate the complex and multidimensional nature of audio signals. In speech processing and recognition, several features distinguish various aspects of audio signals. We explore two classical audio feature sets, low-level descriptors (LLDs) and Mel-frequency cepstral coefficients (MFCCs), as well as novel audio embeddings, which attempt to capture multidimensional representations of audio signals, discerning patterns, and relationships in a way akin to word embeddings. Table 1 introduces features' key attributes, strengths, and limitations.
1WER is a standard metric used in speech recognition and natural language processing that calculates the percentage of words in a transcribed text that were incorrectly recognized, compared with a reference transcription, taking into account substitutions, insertions, and deletions.4 Rafael Mestre and Matt Ryan

Table 1. Summary of audio feature extraction techniques

| Features | Key attributes | Strengths | Limitations |
|----------|----------------|-----------|-------------|
| LLDs     | Basic acoustic features: pitch, energy, spectral contrast, zero-crossing rate | Simple, computationally inexpensive, interpretable | Capture only surface-level patterns; sensitive to noise |
| MFCCs    | Derived from Mel-scaled spectrogram; models timbre and spectral envelope | Captures perceptual characteristics of human hearing | Fixed frame size; require preprocessing; less effective for context |
| Audio embeddings | Learned representations from models like Wav2Vec, HubBERT, and Whisper | Captures complex, holistic attributes; adaptable to various tasks | Computationally expensive; requires large datasets |

Abbreviations: LLD, low-level descriptor; MFCC, Mel-frequency cepstral coefficient.

Figure 1. Representation of an utterance in different modalities. (a) As a discrete mathematical function with an amplitude changing over time; (b) As a spectrum of energy in decibels with respect to time.

3. Understanding Audio Features

An audio signal is a series of air vibrations measured over time. Its typical representation is a waveform like Figure 1a, which shows the amplitude (or strength) of vibrations at each point in time. This time-based view reveals little about the nature of the sound. Speech, music, or noise may look similar in waveform form but differ significantly in content. We often transform the signal into the frequency domain, producing a spectrogram (Figure 1b). The x-axis (time) and the z-axis (color bar) show how much energy is present at each frequency over time. This spectral view decomposes the signal into a combination of simple waves at varying frequencies and amplitudes, making it more analytically useful.

3.1. Low-Level Descriptors

Derived from spectral representations, LLDs describe basic properties of an audio signal and serve as inputs for more complex feature extraction. Typical LLDs include root-mean-squared energy (also known as power or intensity), pitch, spectral contrast, spectral flux, zero-crossing rate, and various spectral shape descriptors, such as spectral centroid, spread, skewness, and kurtosis. We explore two of the most important LLDs for speech analysis: energy and pitch.

Energy refers to the loudness or intensity of an audio signal. Mathematically, for a discrete-time signal x[n] of duration T, discretized into N points, and sampled at a rate/frequency f_s = N/T, the energy is calculated as E = ∑_{n=0}^{N-1} x^2[n]. This measure quantifies the 'power' or 'volume' of the signal,
