+
Skip to main content
Springer Nature Link
Log in

Exploring Physiology-Based Classification of Flow During Musical Improvisation in Mixed Reality

  • Conference paper
  • First Online:
Deep Learning Theory and Applications (DeLTA 2024)

Part of the book series: Communications in Computer and Information Science ((CCIS,volume 2171))

Included in the following conference series:

  • 403 Accesses

Abstract

The flow state is desirable in many activities, e.g., while making music or being active in Virtual, Augmented, and Mixed Realities. With the long-term goal of creating affective systems that can consider the user’s flow state in real-time, we evaluated an approach for real-time flow classification during networked music performance using a deep neural network. We trained our classifier based on physiological signals (PPG and GSR) that we recorded and annotated in a laboratory study, including jamming musicians. The results that we present in this paper confirm the technical validity of this approach while also facing challenges that stem from inter-rater reliability and a heavily unbalanced dataset.

This is a preview of subscription content, log in via an institution to check access.

Access this chapter

Log in via an institution

Subscribe and save

Springer+
from $39.99 /Month
  • Starting from 10 chapters or articles per month
  • Access and download chapters and articles from more than 300k books and 2,500 journals
  • Cancel anytime
View plans

Buy Now

Chapter
USD 29.95
Price excludes VAT (USA)
  • Available as PDF
  • Read on any device
  • Instant download
  • Own it forever
eBook
USD 139.99
Price excludes VAT (USA)
  • Available as EPUB and PDF
  • Read on any device
  • Instant download
  • Own it forever
Softcover Book
USD 84.99
Price excludes VAT (USA)
  • Compact, lightweight edition
  • Dispatched in 3 to 5 business days
  • Free shipping worldwide - see info

Tax calculation will be finalised at checkout

Purchases are for personal use only

Institutional subscriptions

Similar content being viewed by others

Explore related subjects

Discover the latest articles, books and news in related subjects, suggested using machine learning.

References

  1. Barrass, S., Barrass, T.: Musical creativity in collaborative virtual environments. Virtual Reality 10(2), 149–157 (2006)

    Article  Google Scholar 

  2. Bian, Y., et al.: A framework for physiological indicators of flow in VR games: construction and preliminary evaluation. Pers. Ubiquit. Comput. 20(5), 821–832 (2016)

    Article  Google Scholar 

  3. Csikszentmihalyi, M.: Beyond Boredom and Anxiety. The Jossey-Bass Behavioral Science Series (1975)

    Google Scholar 

  4. Csikszentmihalyi, M.: Beyond Boredom and Anxiety. Jossey-Bass (2000)

    Google Scholar 

  5. Davis, C.A., et al.: Dopamine for “wanting’’ and opioids for “liking’’: a comparison of obese adults with and without binge eating. Obesity 17(6), 1220–1225 (2009)

    Article  Google Scholar 

  6. De Manzano, Ö., Theorell, T., Harmat, L., Ullén, F.: The psychophysiology of flow during piano playing. Emotion 10(3), 301 (2010)

    Article  Google Scholar 

  7. Diener, E., Emmons, R.A.: The independence of positive and negative affect. J. Pers. Soc. Psychol. 47(5) (1984)

    Google Scholar 

  8. Dietrich, A.: Neurocognitive mechanisms underlying the experience of flow. Conscious. Cogn. 13(4), 746–761 (2004)

    Article  Google Scholar 

  9. Dunbar, R.I., Kaskatis, K., MacDonald, I., Barra, V.: Performance of music elevates pain threshold and positive affect: implications for the evolutionary function of music. Evol. Psychol. 10(4), 147470491201000420 (2012)

    Article  Google Scholar 

  10. Elvers, P., Fischinger, T., Steffens, J.: Music listening as self-enhancement: effects of empowering music on momentary explicit and implicit self-esteem. Psychol. Music 46(3), 307–325 (2018)

    Article  Google Scholar 

  11. Forbes, M.: Giving voice to jazz singers’ experiences of flow in improvisation. Psychol. Music 49(4), 789–803 (2021)

    Article  Google Scholar 

  12. Ford, J.L., Vosloo, J., Arvinen-Barrow, M.: ‘pouring everything that you are’: musicians’ experiences of optimal performances. Br. J. Music Educ. 37(2), 141–153 (2020)

    Article  Google Scholar 

  13. Fredenburg, H.A., Silverman, M.J.: Effects of music therapy on positive and negative affect and pain with hospitalized patients recovering from a blood and marrow transplant: a randomized effectiveness study. Arts Psychother. 41(2), 174–180 (2014)

    Article  Google Scholar 

  14. Fritz, B.S., Avsec, A.: The experience of flow and subjective well-being of music students. Horiz. Psychol. 16(2), 5–17 (2007)

    Google Scholar 

  15. Groarke, J.M., Hogan, M.J.: Listening to self-chosen music regulates induced negative affect for both younger and older adults. PLoS ONE 14(6), e0218017 (2019)

    Article  Google Scholar 

  16. Hamilton, R., Platz, C.: Gesture-based collaborative virtual reality performance in carillon. In: Proceedings of the 2016 International Computer Music Conference, pp. 337–340 (2016)

    Google Scholar 

  17. Harmat, L., de Manzano, Ö., Theorell, T., Högman, L., Fischer, H., Ullén, F.: Physiological correlates of the flow experience during computer game playing. Int. J. Psychophysiol. 97(1), 1–7 (2015)

    Article  Google Scholar 

  18. Jackson, S.A., Csikszentmihalyi, M.: Flow in sports. The keys to optimal experiences and performances: human kinetics (1999)

    Google Scholar 

  19. Keeler, J.R., et al.: The neurochemistry and social flow of singing: bonding and oxytocin. Front. Hum. Neurosci. 518 (2015)

    Google Scholar 

  20. Kivikangas, J.M., et al.: Psychophysiology of flow experience: an explorative study (2006)

    Google Scholar 

  21. Kusunoki, F., Sugimoto, M., Hashizume, H.: Symphony-q: a support system for learning music in collaborative learning. In: IEEE International Conference on Systems, Man and Cybernetics, vol. 4, pp. 6–pp. IEEE (2002)

    Google Scholar 

  22. Lindlbauer, D., Wilson, A.D.: Remixed reality: manipulating space and time in augmented reality. In: Proceedings of the 2018 CHI Conference on Human Factors in Computing Systems, pp. 1–13 (2018)

    Google Scholar 

  23. Loepthien, T., Leipold, B.: Flow in music performance and music-listening: differences in intensity, predictors, and the relationship between flow and subjective well-being. Psychol. Music 50(1), 111–126 (2022)

    Article  Google Scholar 

  24. Makowski, D., et al.: Neurokit2: a Python toolbox for neurophysiological signal processing. Behav. Res. Methods 1–8 (2021)

    Google Scholar 

  25. Men, L., Bryan-Kinns, N.: Lemo: supporting collaborative music making in virtual reality. In: 2018 IEEE 4th VR Workshop on Sonic Interactions for Virtual Environments (SIVE), pp. 1–6. IEEE (2018)

    Google Scholar 

  26. Merry, M., Silverman, M.J.: Effects of patient-preferred live music on positive and negative affect and pain with adults on a post-surgical oncology unit: a randomized study. Arts Psychother. 72, 101739 (2021)

    Article  Google Scholar 

  27. Morgan, E., Gunes, H., Bryan-Kinns, N.: Instrumenting the interaction: affective and psychophysiological features of live collaborative musical improvisation (2014)

    Google Scholar 

  28. Morgan, E., Gunes, H., Bryan-Kinns, N.: Using affective and behavioural sensors to explore aspects of collaborative music making. Int. J. Hum. Comput. Stud. 82, 31–47 (2015)

    Article  Google Scholar 

  29. Park, S.: ARLooper: a mobile AR application for collaborative sound recording and performance. In: Kronland-Martinet, R., Ystad, S., Aramaki, M. (eds.) CMMR 2019. LNCS, vol. 12631, pp. 554–562. Springer, Cham (2021). https://doi.org/10.1007/978-3-030-70210-6_35

    Chapter  Google Scholar 

  30. Peifer, C., Schulz, A., Schächinger, H., Baumann, N., Antoni, C.H.: The relation of flow-experience and physiological arousal under stress-can u shape it? J. Exp. Soc. Psychol. 53, 62–69 (2014)

    Article  Google Scholar 

  31. Pejsa, T., Kantor, J., Benko, H., Ofek, E., Wilson, A.: Room2room: enabling life-size telepresence in a projected augmented reality environment. In: Proceedings of the 19th ACM Conference on Computer-Supported Cooperative Work & Social Computing, pp. 1716–1725 (2016)

    Google Scholar 

  32. Pfaff, S., Lervik, O.: Games in concert. In: 2015 IEEE 2nd VR Workshop on Sonic Interactions for Virtual Environments (SIVE), pp. 1–4 (2015)

    Google Scholar 

  33. Poupyrev, I., et al.: Augmented groove: collaborative jamming in augmented reality. In: ACM SIGGRAPH 2000 Conference Abstracts and Applications, vol. 17, p. 77 (2000)

    Google Scholar 

  34. Regenbrecht, H., Ott, C., Park, N., Duncan, S., Collins, J.: Voxelvideos for entertainment, education, and training. IEEE Access 9, 68185–68196 (2021)

    Article  Google Scholar 

  35. Regenbrecht, H., Park, J.W., Ott, C., Mills, S., Cook, M., Langlotz, T.: Preaching voxels: an alternative approach to mixed reality. Front. ICT 6, 7 (2019)

    Article  Google Scholar 

  36. Schäfer, K., Saarikallio, S., Eerola, T.: Music may reduce loneliness and act as social surrogate for a friend: evidence from an experimental listening study. Music Sci. 3, 2059204320935709 (2020)

    Article  Google Scholar 

  37. Schellenberg, E.G.: Music and cognitive abilities. Curr. Dir. Psychol. Sci. 14(6), 317–320 (2005)

    Article  Google Scholar 

  38. Schlagowski, R., Gupta, K., Mertes, S., Billinghurst, M., Metzner, S., André, E.: Jamming in MR: towards real-time music collaboration in mixed reality. In: 2022 IEEE Conference on Virtual Reality and 3D User Interfaces Abstracts and Workshops (VRW), pp. 854–855. IEEE (2022)

    Google Scholar 

  39. Schlagowski, R., et al.: Wish you were here: mental and physiological effects of remote music collaboration in mixed reality. In: Proceedings of the 2023 CHI Conference on Human Factors in Computing Systems, pp. 1–16 (2023)

    Google Scholar 

  40. Schlagowski, R., Wildgrube, F., Mertes, S., George, C., André, E.: Flow with the beat! human-centered design of virtual environments for musical creativity support in VR. In: Proceedings of the 14th Conference on Creativity and Cognition, pp. 428–442 (2022)

    Google Scholar 

  41. Tan, L., Sin, H.X.: Flow research in music contexts: a systematic literature review. Music Sci. 25(4), 399–428 (2021)

    Article  Google Scholar 

  42. Tozman, T., Magdas, E.S., MacDougall, H.G., Vollmeyer, R.: Understanding the psychophysiology of flow: a driving simulator experiment to investigate the relationship between flow and heart rate variability. Comput. Hum. Behav. 52, 408–418 (2015)

    Article  Google Scholar 

  43. Turchet, L., Hamilton, R., Çamci, A.: Music in extended realities. IEEE Access 9, 15810–15832 (2021)

    Article  Google Scholar 

  44. Welch, G.F., Biasutti, M., MacRitchie, J., McPherson, G.E., Himonides, E.: The impact of music on human development and well-being. Front. Psychol. 11, 1246 (2020)

    Article  Google Scholar 

  45. Wrigley, W.J., Emmerson, S.B.: The experience of the flow state in live music performance. Psychol. Music 41(3), 292–305 (2013)

    Article  Google Scholar 

  46. Yoon, B., Kim, H., Lee, G.A., Billinghurst, M., Woo, W.: The effect of avatar appearance on social presence in an augmented reality remote collaboration. In: 2019 IEEE Conference on Virtual Reality and 3D User Interfaces (VR), pp. 547–556. IEEE (2019)

    Google Scholar 

Download references

Acknowledgments

This paper was partially funded by the DFG through the Leibniz award of Elisabeth André (AN 559/10-1).

Author information

Authors and Affiliations

  1. Chair for Human-Centered Artificial Intelligence, University of Augsburg, Augsburg, Germany

    Ruben Schlagowski, Silvan Mertes, Dominik Schiller, Yekta Said Can & Elisabeth André

Authors
  1. Ruben Schlagowski
    View author publications

    Search author on:PubMed Google Scholar

  2. Silvan Mertes
    View author publications

    Search author on:PubMed Google Scholar

  3. Dominik Schiller
    View author publications

    Search author on:PubMed Google Scholar

  4. Yekta Said Can
    View author publications

    Search author on:PubMed Google Scholar

  5. Elisabeth André
    View author publications

    Search author on:PubMed Google Scholar

Corresponding author

Correspondence to Ruben Schlagowski .

Editor information

Editors and Affiliations

  1. Instituto de Telecomunicações and University of Lisbon, Lisbon, Portugal

    Ana Fred

  2. LIAS, LIAS/ENSMA, Poitiers, France

    Allel Hadjali

  3. Ford Motor Company, Dearborn, MI, USA

    Oleg Gusikhin

  4. University of Naples Federico II, Naples, Italy

    Carlo Sansone

Ethics declarations

Disclosure of Interests

The authors have no competing interests to declare that are relevant to the content of this article.

Rights and permissions

Reprints and permissions

Copyright information

© 2024 The Author(s), under exclusive license to Springer Nature Switzerland AG

About this paper

Check for updates. Verify currency and authenticity via CrossMark

Cite this paper

Schlagowski, R., Mertes, S., Schiller, D., Can, Y.S., André, E. (2024). Exploring Physiology-Based Classification of Flow During Musical Improvisation in Mixed Reality. In: Fred, A., Hadjali, A., Gusikhin, O., Sansone, C. (eds) Deep Learning Theory and Applications. DeLTA 2024. Communications in Computer and Information Science, vol 2171. Springer, Cham. https://doi.org/10.1007/978-3-031-66694-0_18

Download citation

  • DOI: https://doi.org/10.1007/978-3-031-66694-0_18

  • Published:

  • Publisher Name: Springer, Cham

  • Print ISBN: 978-3-031-66693-3

  • Online ISBN: 978-3-031-66694-0

  • eBook Packages: Computer ScienceComputer Science (R0)

Keywords

Publish with us

Policies and ethics

点击 这是indexloc提供的php浏览器服务，不要输入任何密码和下载