Kypseli Logo
    • Ελληνικά
    • English
  •  Home
  •  Browse 
    • Communities & Collections
    • By Issue Date
    • Authors
    • Titles
    • Subjects
    • By Issue number
  • Language elLanguage en
  •  Login 
    • Sign in
    View Item 
    • Home
    • Αποθετήριο Ανοικτού Πανεπιστημίου Κύπρου (Repository of the Open University of Cyprus)
    • Μεταπτυχιακές διατριβές / Master Τhesis
    • Cognitive Systems (in English)
    • View Item
    •   Home
    • Αποθετήριο Ανοικτού Πανεπιστημίου Κύπρου (Repository of the Open University of Cyprus)
    • Μεταπτυχιακές διατριβές / Master Τhesis
    • Cognitive Systems (in English)
    • View Item
    JavaScript is disabled for your browser. Some features of this site may not work without it.

    Semantic content effects on the perception of movieclips

    Thumbnail
    View/Open
    COS-2023-00020.pdf (4.602Mb)
    Date
    2023-05
    Author
    Kesoglou, Anastasia Maria
    Metadata
    Show full item record
    Abstract
    Our brain is skilled with the ability to perceive and process multimodal stimuli. This process known as crossmodal perceptual integration, has been in the research spotlight for a long time, providing evidence for the integration of information coming from different modalities. Prior experiments on the field mostly utilized pictures and were limited in the semantic content of a single sound or word. The present study aims to investigate crossmodal perceptual integration in realistic conditions using short movieclips (1500ms) and auditory meaningful three-word sentences in cases of target detection judgments. This study (N=36) is the first to introduce trials without a target that always include target-related information, which was present, either only through vision or audition (incongruent movieclips) or through both (congruent movieclips). For each target condition (present or absent) the movieclips were made up of a combination of 12 videos and 12 sentences, which were repeated in a pseudorandomized order four times for each participant (total trials= 288). The results from the two-way repeated measures ANOVA indicate a similar pattern between the two modalities for semantically incongruent movieclips, with statistically lower accuracy scores in trials where the target was present only in one modality (Maudio=0.647, SDaudio=0.305; Mvisual=0.841, SDvisual=0.235), whereas in target absent trials both showed superior performance (Maudio=0.931, SDaudio=0.038; Mvisual= 0.986, SDvisual=0.018). On the other hand, we observed the opposite pattern for semantically congruent movieclips (Target present trials: Maudiovisual=0.981, SDaudiovisual= 0.036 vs. Target absent trials: Maudiovisual=0.898, SDaudiovisual=0.111). Reaction times were the same for the two modalities (F(2,70)=0.384, p=0.683). In accordance with previous research using images and single words, our results show that when auditory and visual information is congruent, performance is superior and when the target is only present through audio but visual information is incongruent, performance is evidently compromised, and vice versa. Regarding the role of semantics, when the audio sentence included a target-related noun accompanied by a semantically incongruent video, accuracy in judgements was statistically better compared to when it was a verb (tincVerb vs. incNoun=-8.428, p< .001; tconVeb vs. incNoun=-4.256, p< .001). The present results could provide more evidence regarding the role of complexity of semantics, and especially the different role verbs and nouns could play in crossmodal perceptual integration in more realistic situations. Our findings can enrich the content of learning techniques, as well as the design of AI models, by taking advantage of the supporting role of semantic audiovisual information, while taking into consideration the confusion that the complexity in semantic information could cause to perception experience.
    URI
    http://hdl.handle.net/11128/5772
    Collections
    • Cognitive Systems (in English)

    Open University of Cyprus

    PO Box 12794,

    2252, Latsia

    Cyprus

    Tel.: +357 22 411600

    Fax.: +357 22 411601

    • Help
    • Contact Us
    • Open University of Cyprus
    • OUC Library
    • Policies
    • Accessibility and Data Protection

    Find us on:

    • FacebookFacebook
    • EU Flag
    • Republic of Cyprus
    • Structural Funds
    • e University
    • Open University of Cyprus

    The eUniversity Project is co-founded by the European Regional Development Fund and National Funds in the Programmatic Period 2007-2013

     

    Browse

    All of DSpaceCommunities & CollectionsBy Issue DateAuthorsTitlesSubjectsBy Issue numberThis CollectionBy Issue DateAuthorsTitlesSubjectsBy Issue number

    My Account

    Sign inRegister

    Open University of Cyprus

    PO Box 12794,

    2252, Latsia

    Cyprus

    Tel.: +357 22 411600

    Fax.: +357 22 411601

    • Help
    • Contact Us
    • Open University of Cyprus
    • OUC Library
    • Policies
    • Accessibility and Data Protection

    Find us on:

    • FacebookFacebook
    • EU Flag
    • Republic of Cyprus
    • Structural Funds
    • e University
    • Open University of Cyprus

    The eUniversity Project is co-founded by the European Regional Development Fund and National Funds in the Programmatic Period 2007-2013