[1]

Human speech: an exciting field of research

Hardly any human performance is as complicated as that of speaking. Initiated by neuronal signals and processes in the brain, more than 100 muscles and several organs throughout the body are involved - for example, in addition to the larynx, lips and nose, the diaphragm comes into play as soon as we raise our voice. Body tension also plays a decisive role. In view of this complex interplay of motor and cognitive processes, it is obvious that a person's voice reveals a lot about his or her physical and mental condition. For Prof. Dr. Heiko Gewald (opens in a new window), Fabian Thaler (opens in a new window), and Prof. Dr. Stefan Fausser (opens in a new window)at the Institute for Service Management at HNU (CROSS), language and the human voice are therefore one thing above all: a highly interesting field of research.

[2]

Aim of the project: filtering out the "inner conviction" at the auditory level. 

The scientists are primarily concerned with the great potential offered by voice analysis through AI: Whether speech-based coronation tests or personality tests in recruiting processes, AI-supported speech and voice analyses are being used more and more. Gewald ,Thaler, and Fausser are dedicated to the technically supported analysis of beliefs in spoken language. With the VACE project, the scientists are pursuing an overarching goal in two different sub-projects and data sets: they want to extract the "inner conviction" of speakers at the auditory level and thereby operationalise actual intentions. The markers identified in this way will be used in the VACE project to develop optimised marketing strategies. The more precisely the needs, desires and intentions of customers are sounded out, the more individually individualised the response to individual needs can be, and ultimately not only can customer satisfaction be increased, but the marketing budget can also be used in a more targeted manner. 
To this end, the researchers are teaching a prototypically developed AI to recognise the most subtle modifications in the voice - in principle a procedure similar to the lie detector. However, the latter includes other parameters in its analysis, such as skin conductivity, blood pressure and pulse, whereas the VACE studies focus purely on the acoustics of the spoken word.

Curious? If you would like to be a test person and oscillate between fiction and truth in the service of science, or if you would like to work on the project as a student assistant, you can contact Fabian Thaler directly by email.

The Technology Transfer Centre Big Data based Marketing (TTZ Günzburg) (opens in a new window), supported by the Free State of Bavaria, was founded in 2020 as an institute of the University of Applied Sciences Neu-Ulm (HNU) in the Swabian town of Günzburg and conducts applied research and development in the field of data-driven marketing.
The interdisciplinary team advises companies strategically and operationally on the development and implementation of prototypes and concepts for the use of artificial intelligence in marketing.

[3]

Hula-hoop on Mount Everest - or: it's all about setting the right tone 

The researchers are not interested in direct communication between man and machine or in extracting meaning from machine-recorded speech; that falls into the field of application of so-called Natural Language Processing (NLP). In their experiments, it is much more imperative to let the semantic level take a back seat to the acoustic level - what is important is not what is spoken, but how. "If I tell you, for example, that I danced the hula hoop naked on Mount Everest last week, you know immediately that I am just pulling your leg," says Fabian Thaler. "The algorithm doesn't understand that. But it can, and that's the interesting thing for us, recognise purely on an acoustic level that something must be wrong with the story - for example, because my pitch changes or my voice trembles." The AI uses voice samples for comparison: The algorithm learns to detect changes in pitch, speech tempo or intonation in front of a comparison foil that is collected in advance.

[4]

"It's on the tip of my tongue": Early Alzheimer's detection through voice analysis

In the long term, this research should serve another purpose besides optimising marketing strategies: early Alzheimer's detection. Even in the early stages, the most subtle nuances in speech production can indicate the presence of this degenerative disease. Even before clearly recognisable and audible speech and communication disorders or aphasia, the complete loss of speech, word-finding problems and syntactic or semantic deficits, for example, are symptoms whose early recognition enables intervention. The analysis of language can therefore help to detect this disorder earlier - for example, within the framework of apps that provide for regular language tests and thus allow an individual measurement of language.
The research at CROSS closes a research gap. While corresponding data sets have already been generated in the international context, no equivalent studies are yet available for the German-speaking world, as Fabian Thaler explains. However, because possible markers can differ greatly from language to language and from culture to culture, and the international results are therefore not transferable without compromises, a German corpus is absolutely necessary.  

[5]

Data, data, data: voice sample fodder for the algorithm

"The first and most important task is, of course, to create a suitable database," says Fabian Thaler. The more voice samples there are, the better the prototype can be trained to identify any deviations in the audio streams. The CROSS team is collecting the corresponding samples in two different studies.
The researchers had already conducted the study of the first sub-project, the debating club. The aim was to find out whether verbal statements that do not correspond to the actual opinion of the speakers show discrepancies in the acoustic and linguistic features. The algorithm worked successfully: it was able to identify the "inner conviction" of the speakers with an accuracy of around 98 percent. However, because the study did not have enough participants, the project is now going into a third round. The test persons receive a randomly selected, polarising topic by email, 30 minutes of preparation time and a specific role assignment; they are therefore only told shortly beforehand whether they are to take a pro or con stance on the said question and argue accordingly.
In the second sub-project, the participants are allowed to be even more creative: They present the researchers with five previously prepared short stories, at least one of which must be invented. "There were some extremely imaginative narratives that we in the evaluation team were able to identify relatively unanimously as lies," Fabian Thaler reports from the samples so far. "The exciting thing now is to find out whether the AI can also identify this as a lie, regardless of the abstruse content."

[6]

From spectrum to cepstrum

To do this, however, the audio data must first be fed in. Fabian Thaler shows us how this works directly in the system. The first step in automatic speech recognition is to extract certain features from the digital audio signal. "The classic spectrum, as everyone knows it from audio recordings, is too inaccurate for our analysis," he explains. "We have to turn the whole thing around again, so to speak, and approximate the human voice." This is called cepstrum - an anagram of spectrum. So-called MFCCs (Mel Frequency Cepstral Coefficients), which are filtered out in the process. These allow conclusions to be drawn about a variety of acoustic speech features.

[7]

Quo vadis, voice analysis

Automated natural language processing is on the rise and opens up manifold possibilities in almost all areas of our lives. Speech recognition systems are getting better and better at understanding human speech - but an AI is only as good as the data it is provided with. In the coming weeks and months, Fabian Thaler will therefore have many more fairy tale lessons on his calendar: to ensure a sufficient database, many more voice samples will be collected from volunteers, fed in and measured. We will continue to accompany him and the project and will publish regular updates here.