Semantic content effects on the perception of movieclips

Abstract

Our brain is skilled with the ability to perceive and process multimodal stimuli. This process known as crossmodal perceptual integration, has been in the research spotlight for a long time, providing evidence for the integration of information coming from different modalities. Prior experiments on the field mostly utilized pictures and were limited in the semantic content of a single sound or word. The present study aims to investigate crossmodal perceptual integration in realistic conditions using short movieclips (1500ms) and auditory meaningful three-word sentences in cases of target detection judgments. This study (N=36) is the first to introduce trials without a target that always include target-related information, which was present, either only through vision or audition (incongruent movieclips) or through both (congruent movieclips). For each target condition (present or absent) the movieclips were made up of a combination of 12 videos and 12 sentences, which were repeated in a pseudorandomized order four times for each participant (total trials= 288). The results from the two-way repeated measures ANOVA indicate a similar pattern between the two modalities for semantically incongruent movieclips, with statistically lower accuracy scores in trials where the target was present only in one modality (Maudio=0.647, SDaudio=0.305; Mvisual=0.841, SDvisual=0.235), whereas in target absent trials both showed superior performance (Maudio=0.931, SDaudio=0.038; Mvisual= 0.986, SDvisual=0.018). On the other hand, we observed the opposite pattern for semantically congruent movieclips (Target present trials: Maudiovisual=0.981, SDaudiovisual= 0.036 vs. Target absent trials: Maudiovisual=0.898, SDaudiovisual=0.111). Reaction times were the same for the two modalities (F(2,70)=0.384, p=0.683). In accordance with previous research using images and single words, our results show that when auditory and visual information is congruent, performance is superior and when the target is only present through audio but visual information is incongruent, performance is evidently compromised, and vice versa. Regarding the role of semantics, when the audio sentence included a target-related noun accompanied by a semantically incongruent video, accuracy in judgements was statistically better compared to when it was a verb (tincVerb vs. incNoun=-8.428, p< .001; tconVeb vs. incNoun=-4.256, p< .001). The present results could provide more evidence regarding the role of complexity of semantics, and especially the different role verbs and nouns could play in crossmodal perceptual integration in more realistic situations. Our findings can enrich the content of learning techniques, as well as the design of AI models, by taking advantage of the supporting role of semantic audiovisual information, while taking into consideration the confusion that the complexity in semantic information could cause to perception experience.