WO2025079272A1 - Method for simultaneous reproduction of multiple performance sound sources, simultaneous reproduction system, and program for simultaneous reproduction of multiple performance sound sources - Google Patents

Abstract

[Problem] To provide a method for simultaneous reproduction of multiple performance sound sources, a simultaneous reproduction system, and a program for simultaneous reproduction of multiple performance sound sources, with which an excellent performance sound source for appreciation can be provided by simulating a sense of presence or immersion corresponding to a performance listening position when reproducing multiple performance sound sources, and which can be utilized as a more effective accompaniment performance sound source for practice. [Solution] A method for simultaneous reproduction of multiple (N) performance sound sources, said method characterized by comprising: a step of individually and independently recording multiple performance sound sources by multi-tracking; a step of setting, on an operation screen, a space size to be used during reproduction of the multiple performance sound sources; a step of specifying, on the operation screen, a listening position to be used during reproduction of the performance sound sources; a step of calculating three-dimensional coordinates of the listening position on the basis of the specified listening position; a step of calculating a distance and an orientation between the listening position and a position of each of N-1 performance sound sources; a step of selecting, on the operation screen, one of 1, ...N-1, which are the N-1 performance sound sources; a step of adding three-dimensional position information to performance sound source information by using object-based stereophonic sound processing, and performing reproduction processing of a sound generated by each performance sound source in accordance with a positional relationship between the position of each performance sound source and the listening position according to the selected space size and the listening position, thereby constructing a simulation of a sound field within a specific space size; and a step of simultaneously reproducing one or multiple selected performance recording performance sound sources that have been subjected to the stereophonic sound processing.　

Description

Method for simultaneously playing back a plurality of performance sound sources, simultaneous playback system, and program for simultaneously playing back a plurality of performance sound sources

The present invention relates to a method, a system, and a program for simultaneously playing multiple performance sound sources, and more specifically, to a method, a system, and a program for simultaneously playing multiple performance sound sources, which provide an excellent performance sound source for appreciation by simulating a sense of realism or immersion according to the performance listening position when playing multiple performance sound sources, and which can also be used as a more effective accompaniment performance sound source for practice.

In many music genres, including jazz, rock, and fusion, so-called N-1 accompaniment apps are used by professionals and amateurs alike.
For example, when a user practices a saxophone solo for a song, the N-1 accompaniment app plays back rhythmic accompaniment from piano, bass, and drums, and plays back accompaniment sound sources for parts other than the user's own instrument. One such N-1 accompaniment app is known as the software "IREAL PRO."
This app plays back a large number of songs from a wide variety of music genres, including pop, jazz, and rock, with piano, bass, and drum accompaniment, while displaying the progression of the so-called chord chart on the screen, allowing, for example, a saxophonist or other instrumentalist, or a vocalist, to practice along with the songs. It is also possible to select the rhythm accompaniment, choose the rhythm from swing, funk, Latin, etc., adjust the tempo, transpose, and select the number of repetitions for a song or just a part of a song.
However, this app is solely focused on providing audio accompaniment, and is not intended to provide audio for listening to music. As an accompaniment, the audio can be listened to in stereo, but it does not provide the same immersive and realistic feel as a live jazz session, where musicians are gathered on the spot to perform live.

In this regard, recently, accompaniment apps for multiple performance styles, especially big bands, have been developed. For example, the software name "WDR BIG BAND"
The "BOB Minzer BIG BAND" app is well known.
More specifically, a big band is different from the so-called small combo style, and is composed of a rhythm section consisting of piano, drums, bass, guitar and vocals, a trumpet section consisting of multiple trumpets, a trombone section consisting of multiple trombones, and a saxophone section consisting of multiple saxophones, and mainly plays jazz.
This accompaniment app contains recordings of trumpet, trombone and saxophone front playing along with rhythmic accompaniment from piano, bass and drums for a large number of songs, and by selecting a song it is possible to use it not only as a performance sound source for appreciation, but also to some extent as an N-1 accompaniment app and as an accompaniment performance sound source for practice.
However, this is only possible for a limited number of songs, and it cannot be used as a so-called virtual session.
More specifically, in the case of a small combo style session, for example, the rhythm section, front section instruments, and vocals gather on the spot and perform a selected song live at the same time, but a so-called virtual session is not a live performance by just gathering on the spot, or even a simultaneous performance, but rather involves adding one's own performance to a recorded performance sound source, ultimately completing the song, creating a virtual simulation of a session.
With the recent development of communication networks, such virtual sessions have become possible to a certain extent, but in the big band application mentioned above, such virtual sessions have not been realized.

Meanwhile, advances have been made in acoustics and sound field software processing technology in recent years, and as disclosed in, for example, Patent Documents 1, 2, and 3, by subjecting a performance sound source to so-called reverb software processing, it is now possible to simulate a realistic performance through stereophonic reproduction.
Patent Document 1 discloses a stereophonic effect adding device, a program executed by a computer, and a performance sound source device. The stereophonic effect adding device comprises storage means for storing music data, and stereophonic effect adding means for extracting information that changes periodically with the progress of a music piece from the music data stored in the storage means, generating stereophonic effect information that changes the localization of the stereophonic sound in synchronism with the periodically changing information when stereophonic sound is generated by playing the music data, and outputting music data with the stereophonic effect information added to the music data. The program executed by a computer comprises an analysis step for analyzing music data and extracting from the music data information that changes periodically with the progress of the music, and a stereophonic effect adding step for generating stereophonic effect information that changes the localization of the stereophonic sound in synchronism with the periodically changing information extracted in the analysis step when stereophonic sound is generated by playing the music data, and outputting music data with the stereophonic effect information added to the music data. The performance sound source device is configured to cause a computer to execute the above, and the performance sound source device reproduces stereophonic sound according to music data, and is configured to include: analysis means for analyzing music data and extracting from the music data information that changes periodically with the progress of the music; and control means for generating stereophonic effect information that changes the position of the stereophonic sound in synchronization with the periodically changing information extracted by the analysis means when stereophonic sound is generated by reproducing the music data, and for controlling the reproduction of the stereophonic sound based on the music data based on the stereophonic effect information.
Such a stereophonic effect adding device, a program that is executed by a computer, and a performance sound source device make it possible to perform stereophonic playback using music data even if the music data was not created with stereophonic playback in mind or if the music data is for stereophonic playback with little stereophonic effect.

Patent Document 2 discloses a three-dimensional sound reproducing device and a program for causing a computer to function as the three-dimensional sound reproducing device. This three-dimensional sound reproducing device includes an analysis unit that separates a multi-channel sound signal into a sound image component and an encapsulated component, a sound image component processing unit that binauralizes the sound image component, an encapsulated component emphasis processing unit that emphasizes the encapsulated component to binauralize it, and an adder that adds the binauralized sound image component and the encapsulated component, wherein the analysis unit performs principal component analysis on the multi-channel sound signal using a variance-covariance matrix and separates the multi-channel sound signal into the sound image component and the encapsulated component based on the magnitude of the obtained eigenvalue, and the encapsulated component emphasis processing unit changes the number of channels of the encapsulated component so that the interaural correlation coefficient is smaller with the number of channels of the multi-channel sound signal than with the signal obtained by binauralizing the encapsulated component, and emphasizes the encapsulated component by binauralizing the encapsulated component with the changed number of channels.
According to such a three-dimensional sound reproducing device and program, when a multi-channel sound signal is binaurally reproduced, it is possible to reproduce the enveloping components of the multi-channel sound signal with emphasis.

Patent Document 3 discloses a stereophonic reproduction device that includes a first speaker whose main axis of directionality is directed toward one ear of a listener, a second speaker whose main axis of directionality is directed toward the other ear of the listener, and a signal processing unit that performs crosstalk cancellation processing to cancel out sound reaching the one ear from the second speaker and sound reaching the other ear from the first speaker, and generates a processed signal, wherein the first speaker and the second speaker emit sounds based on the processed signal, and the first speaker and the second speaker are arranged on the median plane or the same sagittal plane of the listener.
According to such a stereophonic sound reproduction device, by arranging two speakers on the median plane or on the same sagittal plane of the listener, it becomes possible to reproduce stereophonic sound with higher robustness against the movement of the listener than in the past.

However, in none of Patent Documents 1, 2, and 3 does this type of acoustic or sound field processing technology apply to the reproduction of multiple performance sound sources, such as a big band performance, and the provision of a performance sound source with a sense of realism or immersion according to the performance position is achieved.
More specifically, it has not been realized to select any one of a plurality of recorded performance sound sources and select a listening position, for example, to use as an accompaniment performance sound source for N-1, or to simulate the performance space size of multiple recorded performances when used as an appreciation performance sound source for a musical piece, or to simulate listening to any one of the multiple recorded performance sound sources at a desired position.
Furthermore, for a selected song, musicians from the rhythm section, front section, etc., as well as vocalists, are recruited not only from within Japan but also from all over the world, and instead of performing simultaneously, performances are added gradually based on recorded accompaniment performance audio, and once the song is completed, a virtual session has not been realized in which a realistic performance is simulated through surround sound reproduction.

In this regard, for example, Patent Document 4 discloses an online session server device. This online session server device is connected to terminal devices used by multiple users in a communicable manner, and includes a setting unit that sets the performance part to be assigned to each terminal device, a generating unit that processes karaoke performance data of a certain song used in a karaoke performance to generate part-specific audio data corresponding to each of the set parts, a transmitting unit that transmits minus-one audio data, which is part-specific audio data corresponding to another part different from the first part, to the terminal device in which the first part is set, an acquiring unit that acquires live performance data based on a performance sound obtained by actually playing the first part in accordance with the performance based on the minus-one audio data from the terminal device in which the first part is set, and a storage processing unit that stores the live performance data of the certain song acquired from each of the multiple terminal devices.
According to such an online session server device, an online session performance can be carried out by utilizing karaoke performance data used in a karaoke performance.

Also, for example, Patent Document 5 discloses a session adjustment device. In this session adjustment device, a calculation device acquires music composition information for each of the performance sounds of a plurality of pieces of music, and also acquires music composition information for a model sound. The calculation device calculates a difference value between the music composition information for the performance sounds and the music composition information for the model sound based on the music composition information. Then, an overall optimization processing unit performs an averaging process based on the difference value. A synthesis unit synthesizes each of the plurality of performance sounds based on an average value obtained by the averaging process. The synthesis unit functions as an adjustment unit by adjusting the volume, rhythm, etc. of the plurality of performance sounds to match the average value.
Such a session adjustment device makes it possible to some extent to adjust the session while eliminating delays in the network.
However, in both Patent Document 4 and Patent Document 5, even if online session performance, that is, simultaneous performance, is possible, a virtual session that simulates a realistic performance by using stereophonic reproduction is not realized.

Patent No. 4983012 Patent No. 6463955 Patent Publication 2023-121744 Patent Publication No. 2022-114309 Patent Publication No. 2021-196556

Based on the above, an object of the present invention is to provide a performance sound source for excellent appreciation by simulating a sense of realism or immersion according to the listening position when playing back multiple performance sound sources, and to provide a method, a system, and a program for simultaneously playing back multiple performance sound sources that can be used as a more effective accompaniment performance sound source for practice.
Based on the above, an object of the present invention is to provide a method, a system, and a program for simultaneously playing multiple performance sound sources, which enable a so-called virtual session when playing a song using multiple performance sound sources, and simulate a sense of realism or immersion, thereby providing an excellent performance sound source for listening, and which can also be used as a more effective accompaniment performance sound source for practice.

In order to achieve the above object, the present invention provides a method for simultaneously playing back a plurality of performance sound sources, comprising:
A method for simultaneously playing multiple (N) performance sound sources, comprising:
A stage in which each of the multiple performance sound sources is recorded individually and independently using multi-track recording;
setting a space size for playing a plurality of sound sources on an operation screen;
specifying a listening position when playing back the performance sound source on an operation screen;
calculating three-dimensional coordinates of the listening position based on the identified listening position;
Calculating the distance and direction between the position of each of the N-1 performance sound sources and the listening position;
a step of selecting one of N-1 performance sound sources 1, ...N-1 on an operation screen, and using object-based stereophonic processing to add three-dimensional position information to the performance sound source information, while performing a reproduction process of sounds from each performance sound source according to the positional relationship between the position of each performance sound source and the listening position based on the selected space size and listening position, thereby simulating and constructing a sound field within a specific space size;
and simultaneously playing back the selected one or more recorded performance sound sources that have been subjected to stereophonic processing.

According to the method for simultaneously playing a plurality of performance sound sources having the above-mentioned configuration, each of the plurality of performance sound sources is recorded individually and independently using multi-tracks, the space size of the plurality of performance sound sources is set on the operation screen, the listening position of the performance sound source is specified on the operation screen, the three-dimensional coordinates of the listening position are calculated based on the specified listening position, the distance and direction between the position of each of the N-1 performance recording sound sources and the listening position are calculated, and then three-dimensional position information is added to the performance sound source using object-based stereophonic processing, while the sound is reproduced by each performance sound source according to the selected space size and listening position, thereby simulating a sound field within a specific space size. Therefore, by selecting one of the N-1 performance sound sources 1, . . . N-1 on the operation screen and simultaneously playing back the selected single or multiple performance recording sound sources that have been subjected to stereophonic processing, when playing back the plurality of performance sound sources, a sense of realism or a sense of immersion according to the performance listening position is simulated, thereby providing an excellent performance sound source for appreciation and being usable as a more effective accompaniment performance sound source for practice.
In summary, when playing back music from multiple performance sound sources simultaneously, any of the multiple performance sound sources is selected, and one of the multiple performance sound sources that is not selected is specified as the listening position of the music, a space size for the music performance from the multiple performance sound sources is set, and position information is associated with each performance sound source information selected from the multiple performance sound sources according to the position vector relationship between each performance sound source position and the listening position.By doing this, without any restrictions on the recording conditions such as microphone position and number of microphones, or the playback conditions such as speaker position and number of speakers, the sound image is localized by using the head-related transfer function (HRTF) to perform object-based stereophonic processing, and a sound field is simulated within the selected space size, thereby making it possible to use a music piece with a rich sense of immersion or realism for appreciation or as a practice accompaniment.

In addition, it is preferable to perform stereophonic processing based on the selected space size and listening position, including the direction of the sound produced by the multiple sound sources, to reflect volume differences, time differences, changes in frequency characteristics, phase changes, changes in reverberation, and the Doppler effect.
Furthermore, when the plurality of performance sound sources are reproduced, it is preferable to perform a synchronization setting between the selected performance sound sources before performing a stereophonic sound processing.
Furthermore, the multiple performance sound sources may be composed of a rhythm section including a piano, drums, bass, and guitar, a trumpet section consisting of multiple trumpets, a trombone section consisting of multiple trumpets, and a saxophone section consisting of multiple saxophones, and the simultaneous playback stage of the performance recording sound sources may be a performance playback of a musical piece by a big band.
In addition, a listening position may be selected from a plurality (N) of performance sound sources, and any one of the remaining performance sound sources 1, ..., N-1 may be selected as the performance sound source, and the multiple performance sound sources may be simultaneously played back to provide a practice accompaniment performance sound source as a performance at the listening position.

In addition, an audience position may be set separately from the multiple (N) performance sound sources, and by selecting the audience position, the multiple performance sound sources may be played simultaneously to create a musical piece for appreciation using multiple full performance sound sources.
Furthermore, any one of the multiple (N) performance sound sources for the rhythm section may be selected and played, and another one of the multiple (N) performance sound sources for the rhythm section may be selected and played, thereby enabling the user to acquire a sense of rhythm by comparing the individual performance sound sources of each part in the rhythm section.
Furthermore, in cases where one of the multiple performance sound sources and/or the listening position moves and the one selected as the performance sound source among the multiple performance sound sources moves during performance, a BPM value may be selected when playing back the multiple performance sound sources.
In addition, in the case of multiple recorded performance sound sources, after simultaneous playback of the multiple performance sound sources is started based on a selected recorded performance sound source and a selected listening position, stereophonic processing may be performed on the selected recorded performance sound source.

In addition, by setting the playback start position and playback end position within the selected song and/or the number of repetitions, when multiple performance sound sources are played simultaneously, the performance up to the set playback start position and playback end position can be repeated according to the set number of repetitions.
Furthermore, when recording a plurality of performance sound sources individually and independently using a multi-track system, the recording may be performed using binaural processing.

Furthermore, in the method for simultaneously reproducing a plurality of performance sound sources according to claim 1, when reproducing the master performance sound source, a step of selecting a performance sound source to be muted from the plurality of performance sound sources is performed.
recording the performance sound source while listening to the master performance sound source as an accompaniment through stereo headphones;
The method may further include a step of playing the music piece in a virtual session with additional participation using the master performance sound source and the recorded performance sound source in accordance with the method for simultaneously playing a plurality of performance sound sources as described in claim 1.
In addition, when participating as an additional participation virtual session, the system may determine whether to respond to invitations from existing participants and participation requests from those wishing to participate, and whether to accept or reject the participation requests.
In addition, in the additional participation virtual session, each existing participant may be determined to be either a Master who has the authority to set up the session and reject participants, or a member who has the authority only to reject the sharing of their own recorded performances.
Furthermore, when participating in the additional participation type virtual session, the recorded performance sound source may be transferred to an external storage unit and shared.
In addition, the Master may decide which of the multiple performance sound sources will be combined with the master performance sound source to form a collaboration musical piece.

Also, the Master may decide whether sharing of the collaboration music piece is to be limited to the participants or to be made public to the users of the method for simultaneously reproducing a plurality of performance sound sources as described in claim 1.
Furthermore, if a participant is listening to a master performance sound source as accompaniment through stereo headphones and there is a recorded performance sound source of the performance sound source, the next participant may use the master performance sound source and the recorded performance sound source as an accompaniment performance sound source by subjecting them to stereophonic processing using the method for simultaneously playing multiple performance sound sources described in claim 1.
Furthermore, a performance sound source of a rhythm section including piano, drums, and bass may be used as a master performance sound source, and participation from the front section of guitar, saxophone, trumpet, and trombone may be included.
Additionally, the evaluation results may be presented based on the number of plays and/or "likes" given to the published collaboration song.
In addition, in the additional participation type virtual session, the tempo, key, and surround sound processing conditions of the music may be set commonly between each participant for the accompaniment sound source that is the basis for joining the virtual session.

In order to achieve the above object, the present invention provides a simultaneous playback program for multiple performance sound sources,
A music playback program for a music playback device having storage means for storing music data consisting of a plurality of performance sound sources, playback means for playing back the music data, sound image localization processing means for processing a musical sound signal output from the playback means based on a control signal to localize a sound image as stereophonic sound at an arbitrary position within a three-dimensional sound space, and a localization control table defining the contents of the music and a localization method for localizing the corresponding sound image as stereophonic sound within the three-dimensional sound space, the contents of which can be arbitrarily changed by a user,
The computer system is configured to function as: analysis means for acquiring information about the content of the music from the music data when playback of the music data is instructed; means for judging whether the information acquired by the analysis means is information about the content of the music defined in the localization control table; means for causing the playback means to start playback of the music data when the judging means has judged that the information acquired by the analysis means is information about the content of the music defined in the localization control table, and means for sending a control signal to the sound image localization processing means for executing a method of localizing a sound image in a three-dimensional acoustic space corresponding to the localization control table defined in the localization control table; and means for causing the playback means to start playback of the music data without performing sound image localization processing in the three-dimensional acoustic space by the sound image localization processing means when the judging means has judged that the information acquired by the analysis means is not information about the content of the music defined in the localization control table.

In order to achieve the above object, the present invention provides a system for simultaneously reproducing a plurality of musical performance sound sources, comprising:
A system for simultaneously playing multiple performance sound sources, comprising: a control unit for controlling the operation of an entire music playback device when simultaneously playing multiple performance sound sources; a memory for storing various control programs and data; a music information storage unit for storing music data; a timer for outputting various timing signals; a music analysis unit for acquiring information related to the content of the music data stored in the music information storage unit; a display control unit for displaying various information on a display unit; a performance control unit for playing back the performance information included in the music data stored in the music information storage unit and outputting it to a speaker; an operation unit for inputting operation signals; a communication unit for transmitting and receiving signals to and from an external device; a display unit for displaying information related to the performance information; at least two speakers for stereoscopically playing back the performance information from the performance control unit; an operation unit for inputting various operation information; and a bus for transferring various data,
The system has a localization control table that defines the content of the music piece and the localization method for localizing the sound image as the corresponding stereophonic sound in a three-dimensional sound space for each of a plurality of performance sound sources.
In addition, multiple simultaneous playback systems for multiple performance sound sources, each having a simultaneous playback program for multiple performance sound sources, are provided, and the multiple systems are connected via network communication, and the multiple performance sound source data being simultaneously played back in each of the multiple systems can be accessed by each other, so that each simultaneous playback system can be used in an additional participation virtual session by recording its own performance while using the performance sound sources of the other simultaneous playback systems.

Below, with reference to the attached drawings, an embodiment of the method, system, and program for simultaneously playing multiple performance sound sources of the present invention will be described, taking as an example a big band having a rhythm section with multiple types of instruments and a front section with multiple types of instruments or vocals as multiple performance sound sources.

The simultaneous playback program for multiple performance sound sources according to the present invention is an application program used in electronic devices such as smartphones and playback devices specialized for music playback (hereinafter, these are collectively referred to as the simultaneous playback system for multiple performance sound sources 10), and the hardware configuration must include a CPU 12, memory 14, an operation unit 28 such as a touch panel or key switches, a display device such as an LCD, storage 32 for storing performance sound source data, and an audio output unit such as a speaker or earphone output. In other words, it is intended to be used in the simultaneous playback system for multiple performance sound sources 10 that has the same computational processing capabilities as a computer.

For example, the simultaneous reproduction system 10 of multiple performance sound sources according to an embodiment of the present disclosure may function as a computer that performs processing of the session adjustment method of the present disclosure. Fig. 1 is a diagram showing an example of a hardware configuration of the simultaneous reproduction system 10 of multiple performance sound sources according to an embodiment of the present disclosure. The simultaneous reproduction system 10 of multiple performance sound sources may be physically configured as a computer device including a CPU 12, a memory 14, a storage 32, a communication unit 30, an operation unit 28, an output device, a bus, etc.
(Basic configuration)

More specifically, as shown in FIG. 1, the simultaneous playback system 10 for multiple performance sound sources can be applied to various devices such as mobile terminals such as mobile phones, portable music players, and personal computers with AV functions, but FIG. 1 shows the case of a mobile terminal such as a mobile phone. In FIG. 1, 12 is a CPU that controls the operation of the entire music playback device 10, 14 is a memory consisting of ROM and RAM that stores various control programs and data, 18 is a music information storage unit that stores music data, 16 is a timer that outputs various timing signals, 20 is a display control unit 12 for displaying various information on a display unit 22, 24 is a performance control unit that plays the performance information included in the music data stored in the music information storage unit 18 and outputs it to a speaker 26, 28 is an operation unit that inputs an operation signal from an operation device, 30 is a communication unit, 22 is a display unit such as a liquid crystal display unit, 26 is at least two speakers (or headphones) that can play back the performance information from the performance control unit 24 in three dimensions, and 24 is a bus for transferring various data.

The performance control unit 24 includes a first interface circuit connected to the bus, a first FIFO (First in first out) buffer in which performance information (sequence data) input via the first interface circuit is stored, a sequencer that receives performance information via the first buffer and outputs event data to a performance sound source at a timing specified by duration data in the performance information, a performance sound source that generates corresponding musical tones based on the performance information (event data) supplied from the sequencer, and a musical tone signal input via the first interface circuit in response to the musical tone signal input from the performance sound source or the musical tone signal from the MP3 decoder. A sound image localization processing circuit (3D circuit) that performs sound image localization processing based on an input sound image localization control signal (3D control signal), a second interface circuit connected to the bus, a second FIFO buffer that stores audio compression format data (in this example, MP3 data) supplied via the second interface circuit, an MP3 decoder that decodes the MP3 data supplied from the second buffer and outputs it to the sound image localization processing circuit, and an A/D converter (DAC) that converts the digital audio signal output from the sound image localization processing circuit into an analog signal, and the output signal of the DAC is amplified and output from the stereo speaker 26.

The system 10 for simultaneously playing back multiple performance sound sources comprises a CPU 12 equivalent to the CPU of a computer, a memory 14 for temporary storage such as a playback buffer, a storage 32 for storing performance sound source data, an operation unit 28 used by the user for operation, a display for displaying the user interface of the program for simultaneously playing back multiple performance sound sources, a playback unit that contributes to actual music playback, and a network connection unit having a function for realizing a network connection. In addition, a network attached storage (NAS) must comprise the CPU 12, memory 14, storage 32, and network connection unit of the system 10 for simultaneously playing back multiple performance sound sources. Similarly, an external playback device must comprise the CPU 12, memory 14, operation unit 28, playback unit, and network connection unit of the system 10 for simultaneously playing back multiple performance sound sources. The external playback device may have storage 32 and function like a NAS.

Memory 14 is a computer-readable recording medium and may be composed of, for example, at least one of ROM (Read Only Memory), EPROM (Erasable Programmable ROM), EEPROM (Electrically Erasable Programmable ROM), RAM (Random Access Memory), etc. Memory 14 may also be called a register, cache, main memory 14 (primary storage device), etc. Memory 14 can store executable programs (program codes), software modules, etc. for implementing a method for simultaneously playing multiple performance sound sources according to one embodiment of the present disclosure.

Storage 32 is a computer-readable recording medium and may be, for example, at least one of an optical disk such as a CD-ROM (Compact Disc ROM), a hard disk drive, a flexible disk, a magneto-optical disk (e.g., a compact disk, a digital versatile disk, a Blu-ray® disk), a smart card, a flash memory 14 (e.g., a card, a stick, a key drive), a floppy disk, a magnetic strip, etc. Storage 32 may also be referred to as an auxiliary memory device. The above-mentioned storage medium may be, for example, a database, a server or other suitable medium including at least one of memory 14 and storage 32.

The sequencer is a process that, when a playback instruction is given from the communication control process, reads out the music data for playback designated in advance by the user from among the music data in the music storage area, and controls the performance sound source section according to the music data. The communication control process is a process that controls the communication section to perform the process for establishing the above-mentioned communication link, passes information received by the communication section to the audio processing section, and controls the communication section to send terminal information supplied from the audio processing section.
In this process, when reading out music data, certain event information is read out and sent to the performance sound source section, and if duration information is read out, the process waits for the time specified by the duration information to elapse before reading out the next event information. This process is repeated to control the formation of musical tone signals by the performance sound source section.

(Network Configuration)

Figure 2 shows the overall configuration of a system to which the simultaneous playback system for multiple performance sound sources, which is the mobile terminal shown in Figure 1, is connected. In this figure, it is a mobile terminal 40, which can be connected to a large-scale network 34 such as the Internet via a base station 38. A distribution server 36 is connected to the large-scale network 34, and the mobile terminal 40 is able to download music data from the distribution server 36, store it in the music information storage unit 18 (Figure 1), and play it back using the performance control unit 24.

In a situation where an external playback device such as a Network Attached Storage (NAS) or a music playback component is connected to a network using mutually compatible standards, the simultaneous playback program for multiple performance sound sources according to the present invention can be used like a remote control device to easily play back performance sound source data stored in the NAS on the external playback device 12, or play back performance sound source data stored in the simultaneous playback system for multiple performance sound sources 10 on the external playback device 12. Note that in this embodiment, when the term NAS is used, it includes not only literal network attached storage, but also all devices that have storage 32 capable of storing performance sound sources and can be connected to a network.

The communication unit 30 is hardware (transmission/reception device) for performing communication between computers via at least one of a wired network and a wireless network, and is also called, for example, a network device, a network controller, a network card, a communication module, etc. The communication unit 30 may be configured to include a high-frequency switch, a duplexer, a filter, a frequency synthesizer, etc. (e.g., a touch panel) in order to realize at least one of Frequency Division Duplex (FDD) and Time Division Duplex (TDD).

(Data format)

The music data stored in the music information storage unit is basically data including performance information (sequence data) such as SMF (Standard MIDI File) and SMAF (Synthetic music Mobile Application Format), but may also be audio data (audio compression format data such as MP3 (MPEG Audio Layer‑3) and AAC (MPEG‑2 Audio AAC (Advanced Audio Coding))). This music data may originally be data in a stereo or 3D audio format, or may be completely monaural data. The music analysis unit may be realized by software processing by a CPU. Furthermore, the performance control unit 24 is a performance sound source unit capable of reproducing music data, and includes a decoder such as an FM performance sound source, a WT (Wave Table) performance sound source, and an MP3 decoder, or a composite performance sound source of these.

(Operation screen)

The simultaneous playback system 10 for multiple performance sound sources has a user interface in the form of a touch panel layered on a display, and corresponds to, for example, a personal computer, a smartphone, a digital still camera, a digital video camera, a navigation device, etc. Of course, the present invention can also be applied to other simultaneous playback systems for multiple performance sound sources.

The touch panel is transparent to the display screen and can detect not only contact with the user's finger, but also approach (meaning approaching within a certain distance without touching). Here, the user's finger refers to a body part other than the user's finger or a stylus (a pen-shaped pointer) that is compatible with the touch panel. In the following, it will be simply referred to as the user's finger, but this includes a body part other than a finger and a stylus.

The touch panel may be of a capacitance type, for example. However, other types may be used as long as they are capable of detecting not only the touch of the user's finger but also the approach of the user's finger. The touch panel notifies the CPU 12 via the bus of detection information indicating the detection results of the touch or approach of the user's finger.

The display may be, for example, an LCD or an organic EL display, and displays a screen image supplied from the CPU 12 via the bus. The memory 14 stores control programs executed by the CPU 12. The memory 14 may also store various types of content (image data, audio data, video data, application programs, etc.) to be processed by the system 10 for simultaneously playing multiple performance sound sources.

The CPU 12 executes a control program recorded in the memory 14 to control the entire system 10 for simultaneously playing multiple musical sound sources. In addition to the above-mentioned configuration, a push button or other hardware that serves as a user interface may be provided.

Next, an operation screen when using the simultaneous playback program of multiple performance sound sources of the present invention will be described. With regard to the graphical user interface, which is the operation screen of the simultaneous playback program of multiple performance sound sources, what is displayed on the display of the simultaneous playback system 10 of multiple performance sound sources is an example of a graphical user interface (hereinafter, GUI) by the simultaneous playback program of multiple performance sound sources. In the simultaneous playback program of multiple performance sound sources of the present invention, a playback device selection area and a performance sound source data selection area may be fixedly arranged on the screen of the GUI so that the selection of the playback device and the selection of the playback performance sound source data can be performed on the same screen. Furthermore, when making a selection using the playback device selection area and the performance sound source data selection area, a hierarchical display area may be provided to display the selection items hierarchically.

(Surface Processing)

In the stereophonic effect addition process, the music data specified by the operation of the operation unit is read from among the music data in the music storage area, converted into music data with appropriate stereophonic effect information added, and the converted music data is stored in the music storage area. In one preferred embodiment, the program for the stereophonic effect addition process is written in advance into the ROM of memory 14. In another embodiment, the program for the stereophonic effect addition process is downloaded into memory 14 by the user from a specified site on the Internet.

There are three types of processes for adding stereophonic effect information that can be executed in the stereophonic effect addition process. When the CPU 10 executes the stereophonic effect addition process, the user can specify the execution of one or more of these processes by operating the operation unit 28.

Outputs the target position for sound image localization. This process calculates the position of the sound image in a three-dimensional orthogonal coordinate system and outputs the position to the 3D sound system.

For example, if the positions of multiple performance sound sources and the focus position, which is the listening position, are calculated as distance R, azimuth angle θ, and elevation angle φ, the target localization position (X, Y, Z) in a three-dimensional orthogonal coordinate system is calculated using the following equations 1 to 3.

X=R・cosφ・cosθ ⋯Formula 1
Y=R・cosφ・sinθ ⋯Formula 2
Z=R・sinφ ⋯Formula 3
The stereophonic system instructs the audio system to set the transfer functions to be used in the filters L and R. For example, the audio system stores in advance a plurality of transfer functions corresponding to the instruction from the stereophonic system, and selectively sets the transfer functions. As a result, the sound image formed by the speakers L and R is localized at a localization position corresponding to the position of interest.

a. Perspective effect addition process: In this perspective effect addition process, sound intensity information (e.g. velocity, volume) contained in the original music data is converted into distance to the virtual performance sound source position, and three-dimensional sound effect information that sets the virtual performance sound source position at that distance away from the user is generated and added to the original music data.

b. Movement effect addition process: In this movement effect addition process, periodic changes such as note information contained in the original music data are extracted, and stereophonic effect information that periodically moves the virtual performance sound source position in synchronization with the period of the change is generated and added to the original music data. The trajectory for periodically moving the virtual performance sound source position includes a straight trajectory extending in the left and right directions in front of the user, a circular trajectory that surrounds the user, and an elliptical trajectory that surrounds the user, and trajectory definition information for these trajectories (such as functions for calculating the coordinates of each point on the trajectory) is stored in the storage unit. The user can select a desired trajectory from among these in advance by operating the operation unit. In this movement effect addition process, the virtual performance sound source position to be assigned to each periodically changing note is calculated based on the trajectory definition information of the trajectory specified by the user, and stereophonic effect information that localizes the sound image of the speaker playback sound at the virtual performance sound source position is generated.

c. Doppler effect addition processing: In this Doppler effect addition processing, pitch control information such as pitch bend events contained in the original music data is converted into three-dimensional sound effect information that moves the virtual performance sound source position closer to the user from a distant location, or moves the virtual performance sound source position away from the user, and is added to the music data.

The stereophonic effect addition process is started when the user operates the operation unit to specify a song and instructs conversion for adding the stereophonic effect. First, the conversion conditions are set. Specifically, a screen inquiring which of the above-mentioned perspective effect addition process, movement effect addition process, and Doppler effect addition process the user wishes to execute is displayed on the display unit 22, and the user's instruction is obtained via the operation unit 28. At this time, the user may instruct one, two, or all of the perspective effect addition process, movement effect addition process, and Doppler effect addition process. Also, when the execution of the movement effect addition process is instructed by the user, a screen inquiring about the trajectory for moving the virtual performance sound source position is displayed on the display unit 22, and the user's instruction is obtained via the operation unit 28.

Next, a perspective effect addition process, a movement effect addition process or a Doppler effect addition process is executed to generate music data in which stereophonic effect information has been added to the original music data, and store the data in the music storage area of memory 14. The generated music data may overwrite the original music data in the music storage area, or may be stored in the music storage area with a file name different from that of the original music data. The user instructs which storage method is to be used by operating the operation unit 28. This concludes the processing content of the stereophonic effect addition process.

The subsystem includes a stereophonic system and an audio system. In order to localize a sound image, the stereophonic system instructs the audio system to set a transfer function. Specifically, as described below, the audio system instructs the audio system to set a transfer function based on a target localization position output from the CPU 12. The transfer function is used in well-known sound image localization technology, and more specifically, it is a head-related transfer function (HRTF) that represents the transfer characteristics of sound from a virtual performance sound source to the listener's eardrum.

The audio system has a performance sound source, filters L and R, and speakers L and R.

One of the speakers L and R is the left speaker L for outputting sound from the left side of the user, and the other is the right speaker R for outputting sound from the right side of the user.

The left filter L corresponds to the left speaker L, and the right filter R corresponds to the right speaker R. When localizing a sound image as described above, the left filter L and the right filter R perform filtering based on a transfer function. The signals from the performance sound source that have been filtered in this way are output to the left speaker L and the right speaker R, respectively.

Note that the left speaker L and the right speaker R indicate that there are two channels, and the left speaker L and the right speaker R may each be composed of multiple speakers. Furthermore, during normal times when gaze guidance is not being performed, signals are output from the performance sound source based on the audio environment set by the user. At this time, the audio system outputs signals without passing through filters L and R. Note that the left speaker L and the right speaker R may be composed of stereo headphones.

The audio environment may be an equalizer setting that changes the frequency characteristics of the output signal. It may also be a left/right balance setting. If the speakers are arranged in the front/rear direction, it also includes a front/rear balance setting. The audio environment is set via a group of operation switches and stored in the external storage unit 32. Note that if no setting has been made by the user, a default audio environment is stored in the external storage unit 32.

FIG. 3 shows the touch panel operation screen 42 .
More specifically, the touch panel operation screen 42 is an operation screen for playing back a digital recording of a big band performance of a selected song under various conditions when playing back a song performed by a big band consisting of a rhythm section 74 consisting of piano, drums, bass, guitar and vocals, a trumpet section 76 consisting of multiple trumpets, a trombone section 78 consisting of multiple trombones, and a saxophone section 80 consisting of multiple saxophones.
When playing back the digital recording of the selected music piece, a section layout diagram 44 of the musical instruments played by the rhythm section 74, trumpet section 76, trombone section 78 and saxophone section 80 is displayed at the top of the screen, and one or more of the instruments in the rhythm section 74, trumpet section 76, trombone section 78 and saxophone section 80 are selected.
More specifically, the rhythm section 74 displays a drum icon 46, a bass icon 48, a piano icon 50, a guitar icon 52 and a vocal icon 54, the trumpet section 76 displays a number of trumpet icons 56, the trombone section 78 displays a number of trombone icons 58, and the saxophone section 80 displays a number of saxophone icons, each of which is clickable.

In the center of the screen, an image is displayed that allows the user to select sound, volume, and metronome by adjusting the length of the horizontal bar, and each can be operated by adjusting the length of the horizontal bar.
More specifically, from the top to the bottom of the screen, a reset icon 64, a listening position setting icon 66, a space size selection bar 68, a volume selection bar 70, and a metronome volume adjustment bar 72 are displayed.
Clicking a reset icon 64 resets the state of the operation screen, clicking a listening position setting icon 66 sets the listening position, and clicking a space size selection bar 68 sets the space size for simultaneously playing back a plurality of performance sound sources.
In the left vertical section of the screen, the length of the song is indicated by a left vertical bar, and an image for selecting a performance part of the song by the length and position of the right vertical bar is displayed by a right vertical bar, and the operation is performed by the length and position of the right vertical bar. More specifically, the left vertical bar is a progress display bar 60, the right vertical bar is a performance part display bar 62, and in the lower part of the left vertical section of the screen, a repeat display icon 82 and a repeat time display icon 84 are displayed, and an image of a repeat operation for repeatedly playing the performance part of the selected song is displayed.
In addition to the section layout diagram 44, the lower part of the screen displays operation images that can be selected or deselected for each of the rhythm section 74, trumpet section 76, trombone section 78 and saxophone section 80.
Instructions can be given either through the operation image or through the section layout diagram 44, but on the operation screen 42, instructions from one are reflected in instructions from the other.
The display of the operation screen 42 and instructions given by clicking on the touch panel screen are performed by the CPU 12 in the same manner as in the conventional case.

4 shows a modified example of the operation screen 42. In more detail, in addition to the center position 90, a left position 92 and a right position 94 are added as audience positions, and when a space size is selected and an instrument to be performed is selected from a plurality of performance sound sources, the size of the section layout diagram 44 is enlarged or reduced in accordance with the selected space size in the space size selection bar 68 (FIG. 4(A) shows a case where the space size is large, and FIG. 4(B) shows a case where the space size is small), allowing the user to visually grasp the image of the selected space size, and the direction of the selected instrument to be performed can be selected by an arrow display 96, reflecting the direction of the sound produced by each instrument, and adding a function for stereophonic processing.
As described above, when creating a session, the selection of music, playback conditions, and conditions for stereophonic sound processing can all be set visually.

As shown in FIG. 6, in step 1, a system having a program for simultaneously playing multiple performance sound sources is started by an icon. Next, in step 2, if the performance sound source of a new song is saved in storage 32, which is an external memory unit, the new song information is imported into memory 14, which is an internal memory unit. Next, in step 3, if an existing session already exists, a selection is made as to whether or not to access the existing session; if yes, proceed to step 5; if no, proceed to step 4. Next, in step 4, songs are selected, including any new songs that have been added. Next, in step 5, a selection is made as to whether or not to play back and listen to the selected song; if yes, proceed to step 6; if no, return to step 4.

Next, in step 7, the user selects whether or not to create a session for the song that has been previewed. If a session is to be created, the user proceeds to step 8. If a session is not to be created, the application is terminated. Next, in step 8, the user sets the tempo, volume, and space size for the song for which a session is to be created on the operation screen. Note that the defaults are　　　　　　Next, in step 9, the performing instruments and listening positions are selected for the song for which a session is to be created. More specifically, in a big band consisting of a rhythm section consisting of piano, drums, bass, guitar, and vocals, a trumpet section consisting of multiple trumpets, a trombone section consisting of multiple trombones, and a saxophone section consisting of multiple saxophones, for example, only the drums are selected in the rhythm section, the first trumpet in the trumpet section, the first trombone in the trombone section, and the tenor, the first tenor, the alto, and the baritone in the saxophone section are all selected, and the listening position is set to the piano position. Note that the performing instruments and listening positions may be selected by directly clicking on the instrument layout diagram on the operation screen 42 as described above, or may be selected in each section on the operation screen 42.

Next, in step 10, stereophonic processing is performed based on the selected space size, musical instruments, and listening position. Next, in step 11, the start position, end position, and number of repetitions of the selected music piece are selected. Next, in step 12, multiple musical performance sound sources are simultaneously played back, and can be used as N-1 accompaniment sound sources for the user's musical instrument performance practice. In this case, for example, in the case of piano practice, the accompaniment sound source to be listened to is listened to as an immersive and realistic accompaniment sound source as if it were being listened to from the piano position by stereophonic processing, so that even if it is a solo practice, it is possible to practice effectively with the feeling of playing live on the spot. Note that, when all musical instruments are selected and the audience position is selected as the listening position, it is possible to listen to a music piece with a sense of immersion and realism, for example, as if appreciating a completed music piece in a large performance hall, according to the selected space size.
Next, in step 14, the user selects whether to save the simultaneously played songs as a session. If the user selects to save the songs, the user proceeds to step 14 and ends the process. If the user selects not to save the songs, the user returns to step 4 and repeats the same process for another song.

According to the method for simultaneously playing back a plurality of performance sound sources having the above configuration, each of the plurality of performance sound sources is recorded individually and independently by multi-track, the space size of the plurality of performance sound sources is set on the operation screen, the listening position of the performance sound source is specified on the operation screen, the three-dimensional coordinates of the listening position are calculated based on the specified listening position, the distance and direction between the position of each of the N-1 performance recording sound sources and the listening position are calculated, and then, using object-based stereophonic processing, three-dimensional position information is added to the performance sound source, while the positional relationship between the position of each performance sound source and the listening position is calculated according to the selected space size and listening position. In accordance with the above, a sound field within a specific space size is simulated by performing sound playback processing for each performance sound source, reflecting volume differences, time differences, changes in frequency characteristics, phase changes, changes in reverberation, and the Doppler effect. Therefore, by selecting one of the N-1 performance sound sources 1, ... N-1 on the operation screen and simultaneously playing back one or more selected performance recording sound sources that have been processed for stereophonic sound, the sense of presence or immersion according to the performance listening position when playing back multiple performance sound sources can be simulated, providing an excellent performance sound source for appreciation and also being usable as a more effective accompaniment performance sound source for practice.

A second embodiment of the present invention will be described below. In the following description, components similar to those in the first embodiment are given the same reference numerals and their description will be omitted. Below, the characteristic parts of this embodiment will be described in detail with reference to Figs. 6 to 28.
The second embodiment of the present invention is characterized by the use of the method of simultaneously playing multiple performance sound sources, and while the first embodiment is for N-1 practice accompaniment for big bands or for listening to music, this embodiment is for a so-called additional participation type virtual session using multiple performance sound sources. Here, the "virtual" in virtual session means that, while a normal session is performed live by a combo arrangement of a rhythm section such as piano, drums, and bass, and a front section such as saxophone, trumpet, and trombone, each section of the combo does not perform live, but rather creates recorded performance sound sources and participates in sequence, so it is not a simultaneous performance, but rather the completed music is processed into stereophonic sound and played simultaneously, virtually simulating a session.

To realize an additive virtual session, similar to the first embodiment, it is necessary to add functions for adding session members, recording performances, sharing performances, setting up performance collaborations, and participating in public sessions, all of which are prerequisites for starting and listening to sessions.
The main functions are explained below using a flowchart.

Figure 6 shows a flowchart for starting a session after launching the app. As shown in Figure 6, when the app is launched in step 1, data on the song, artist, and session (information only) is read in step 2. Next, in step 3, a choice is made as to whether to select a song, and if so, proceed to step 4, or if not, proceed to step 6. Next, in step 4, session information regarding the title, description, and public setting (public or private) is entered. Meanwhile, in step 6, an existing session is selected, and in step 7, the session is started. Next, in step 5, a new session is created, and in step 7, the session is started.

FIG. 7 shows a flow chart from the start of a session to the saving of the session. As shown in FIG. 7, in step 1, a session is started, and in step 2, data is read in for music audio data, spatial arrangement, tempo, session tonality, and reflection of user settings. Next, in step 3, it is selected whether to set a session, and if so, proceed to step 8, and if not, proceed to step 4. Next, in step 4, it is determined whether the session is stopped, and if so, proceed to step 5, and if not, proceed to step 6. Next, in step 5, the session is played back. Next, in step 6, the stream data is subjected to stereophonic processing, and multiple performance sound sources are simultaneously played back in a synchronized state, and the session is put into a playback state. Next, in step 7, it is selected whether to set a session, and if so, proceed to step 8, and if not, proceed to step 13. Next, in step 8, a session is set, and in step 9, the performance parts are set by muting the specified part, muting parts other than the specified part, adjusting the volume of the specified part, turning the metronome on and off, and adjusting the volume. Next, in step 10, the playback position is set by setting the playback start position, the playback range, and turning repeat playback on and off. Next, in step 11, the conditions for stereophonic processing are set by setting the listening position and the space size of the listening space. Next, in step 12, the music is set by setting the tempo and adjustment. Next, in step 13, the session is stopped. Next, in step 14, it is selected whether to save the settings. If the settings are to be saved, the process proceeds to step 15 and the settings are saved, and if the settings are not to be saved, the process proceeds to step 16 and ends.

Fig. 11 shows a flowchart for adding a session member. As shown in Fig. 11, when a session member is added in step 1, in step 2, the invitee is notified using a link URL, email or LINE, a communication app, or a two-dimensional barcode as a means of invitation. Next, in step 3, it is selected whether to cancel the invitation, and if it is canceled, the process returns to step 4, and if it is not canceled, the process returns to step 1. Next, in step 4, it is determined whether to resend the invitation notification, and if it is resent, the process returns to step 5, and if it is not resent, the process returns to step 2. Next, in step 5, the person who received the invitation accepts the invitation.
On the other hand, if the join request is accepted in step 6, the session is made public in step 7, and then the request to join the session is received in step 8. Next, in step 9, it is determined whether to approve the request, and if approved, the process proceeds to step 10, and if not, to step 11. Next, in step 10, if the person who received the invite approves, or if the person approves the join request, participation in the session is completed and the process ends in step 16. If the join request is not approved, a notice of removal from the session is sent in step 11, and the process ends in step 16.
Furthermore, if the participating members are to be managed in step 12, a decision is made in step 13 as to whether to change the authority, and if so, proceed to step 14, or if not, proceed to step 15. Note that changing the authority means a change between Member and Master, with respect to Master being able to decide on changing the settings related to the session and adding members to participate in the session, and Member being able to decide only on sharing his/her own performance. Next, in step 14, the authority is changed and the process returns to step 12. On the other hand, in step 15, the participating member is cancelled, and the process proceeds to step 11.
8 to 10 show related operation screens. In the drawings, 90 denotes a session invitation screen, 92 denotes an approval icon, 94 denotes a resend icon, 96 denotes a master icon, 98 denotes a member icon, 102 denotes a role change screen, and 104 denotes an execution icon.

FIG. 14 shows a flowchart for recording a performance. As shown in FIG. 14, in step 1, when recording a performance, in step 2, the recording environment is set, such as muting the part to be played. Next, in step 3, recording is started by pressing the record button, etc. Next, in step 4, the performance recording is converted into digital data by the microphone and saved in a temporary folder. Next, in step 5, it is determined whether to pause the recording, and if it is paused, it proceeds to step 6, and if it is not paused, it proceeds to step 7. Next, in step 6, it is determined whether to resume recording, and it returns to step 3 to resume, and if it is not resumed, it proceeds to step 7. Next, in step 7, recording is terminated. Next, in step 8, the performance part is designated, a file name, and comment information are assigned to the recording data and saved, and in step 9, it is terminated.
12 and 13 show related operation screens. In the figures, 106 indicates a volume reset icon, 108 indicates a mute icon, and 110 indicates a recording entry screen.

Next, we will discuss how to share a session song among members when the session songs and session members have been decided and the performance by the decided session members has been recorded.
Fig. 18 shows a flowchart of sharing a performance. As shown in Fig. 18, when a performance is shared in step 1, the performance is shared in step 2, and a file is uploaded to a specified server via a network in step 3. Then, in step 4, the performance is shared with session members, and the process ends in step 5.
On the other hand, if the sharing of the performance is to be deleted in step 1, it is determined in step 2 whether or not the performance data exists on the server, and if it exists, the process proceeds to step 3, and if it does not exist, to step 5. Next, in step 3, the sharing of the performance is released. Next, in step 4, the file on the server is deleted. Next, in step 5, the session members are no longer able to listen to the performance, and the process ends in step 6.
Furthermore, if all performance-related data is to be deleted in step 1, all performance-related data is deleted in step 2. Next, in step 3, it is determined whether performance data exists on the server, and if it exists, the process proceeds to step 4, and if it does not exist, the process proceeds to step 5. Next, in step 4, the performance data is deleted from the server. Next, in step 5, the data on the terminal is deleted. Next, in step 6, the data added to the recording data is deleted, and the process ends in step 7.
15 to 17 show related operation screens. In the drawings, reference numeral 112 denotes a session progress screen, 114 denotes a session information screen, 116 denotes a cloud share icon, and 118 denotes a download icon.

Fig. 23 shows a flowchart for setting up collaboration in performance. Session collaboration is data in which performance data is adjusted so that other users can listen to it as a virtual session, and when it is made public, it is adjusted so that not only session members but also users of the simultaneous playback program application of multiple performance sound sources can check it.
As shown in Fig. 23, when setting up a performance collaboration in step 1, in step 2, the performance tracks of the members to be used for playback are selected. Next, in step 3, the scope of disclosure is set. Next, in step 4, it is determined whether to disclose to the general public, and if so, proceed to step 5, and if not, proceed to step 6. Next, in step 5, the application user becomes able to listen to the session collaboration, and in step 7, the process ends. Meanwhile, in step 6, only the session members can listen to the session collaboration, and in step 7, the process ends.
19 to 22 show related operation screens. In the figures, reference numeral 120 denotes a collaboration screen, 122 denotes a solo icon, 124 denotes a rhythm section icon, 126 denotes a trumpet icon, 128 denotes a pin icon, 130 denotes a song title screen, 132 denotes a session member screen, and 134 denotes a public selection screen.

FIG. 28 shows a flowchart of participation in a public session. As shown in FIG. 28, in the case of participation in a public session in step 1, in step 2, a public session is searched for by oneself, by a system recommendation, or by sharing from another user. Next, in step 3, the session is confirmed by listening to the session information, session participating members, or session collaboration. Next, in step 4, a participation request is sent to the confirmed session. Next, in step 5, among the members of the session that sent the request, a person with Master authority determines whether the request is approved, and if approved, proceeds to step 7, and if not, proceeds to step 6. Next, in step 7, participation in the session is started, and ends in step 8. On the other hand, in step 6, a refusal of the request is notified, and the process returns to step 2.
24 to 26 show related operation screens. In the drawings, reference numeral 136 denotes a session detail information screen, 138 denotes a listening icon, 140 denotes a participation request icon, 142 denotes a participation request icon, 144 denotes a recommendation screen, 146 denotes a popular session screen, and 148 denotes a popular artist screen.

By using the above-mentioned method for simultaneously playing multiple performance sound sources, the system for simultaneously playing multiple performance sound sources, and the program for simultaneously playing multiple performance sound sources in a virtual session, even if a combo does not gather at the same place and perform live at the same time, it is possible to play a performance with a sense of immersion and realism like a real live performance by performing stereophonic processing and simultaneously playing multiple performance sound sources. In addition, as an additional participation type virtual session, when each participant records their own performance while listening to the accompaniment performance sound source, they can perform their own performance while listening to the stereophonic processed accompaniment performance sound source, making it possible to record with a greater sense of unity. As a modified example, when each participant records their own performance while listening to the accompaniment performance sound source, they can record using a microphone equipped, for example, as with an iPhone. In that case, when recording while listening to the accompaniment performance sound source without stereophonic processing, and when playing back the recorded performance sound source completed as a musical piece, the listener can select the space size, listening position, and performance sound source of their own choice, and then play and appreciate the stereophonic processing.

Although the present disclosure has been described in detail above, it is clear to those skilled in the art that the present disclosure is not limited to the embodiments described in the present disclosure. The present disclosure can be implemented as modified and altered forms without departing from the spirit and scope of the present disclosure as defined by the claims. Therefore, the description of the present disclosure is intended to be illustrative and does not have any limiting meaning on the present disclosure.
For example, in this embodiment, it has been described as an N-1 practice accompaniment for a big band having a rhythm section of piano, bass, and drums, and a front section of saxophone, trumpet, and trombone, but it is not limited to this. For example, by selecting the listening position as the audience position, it may be used as a performance sound source for appreciation that provides a sense of realism and immersion.
For example, in this embodiment, it has been described as an N-1 practice accompaniment for a big band having a rhythm section of piano, bass, and drums, and a front section of saxophone, trumpet, and trombone, but it is not limited to this. For example, any one of the rhythm section of piano, bass, and drums, or any one of the front section of saxophone, trumpet, and trombone, such as a saxophone, could be selected and used as an N-1 practice accompaniment for a so-called combo.

For example, in this embodiment, the playback of multiple performance sound sources including the rhythm section and the front section can be any of the multiple performance sound sources, for example, a vocalist singing while moving, but this is not limited to this, and the listening position can also move during performance, or any of the multiple performance sound sources can move while the listening position moves during performance.
For example, in this embodiment, it has been described that acoustic processing is performed when the performance of the selected song is completed as a so-called virtual session and then played back, but this is not limited to this. For example, when recording one's own performance based on the N-1 recorded performance sound source before the virtual session is completed, the N-1 recorded performance sound source may be acoustically processed individually.
For example, in the second embodiment, in an additional participation type virtual session, the tempo, tonality, and surround sound processing conditions of the music may be set commonly between each participant for the accompaniment sound source that serves as the basis for joining the virtual session.

The various modes/implementations described in this disclosure include LTE (Long Term Evolution), LTE-A (LTE-Advanced), SUPER 3G, IMT-Advanced, 4G (4th generation mobile communication system), 5G (5th generation mobile communication system), and FRA (Future Radio Access). , NR (new Radio), W-CDMA (registered trademark), GSM (registered trademark), CDMA 2000, UMB (Ultra Mobile Broadband), IEEE 802.11 (Wi-Fi (registered trademark)), IEEE 802.16 (WiMAX (registered trademark)), IEEE 802.20, UWB (Ultra Wide Band), Bluetooth (registered trademark), and other suitable systems, and next generation systems that are based on and extend these. Also, multiple systems may be combined (for example, a combination of at least one of LTE and LTE-A with 5G, etc.).

The processing steps, sequences, flow charts, etc. of each aspect/embodiment described in this disclosure may be reordered unless inconsistent. For example, the methods described in this disclosure present elements of various steps using an example order and are not limited to the particular order presented.

The input and output information, etc. may be stored in a specific location (e.g., memory 14) or may be managed using a management table. The input and output information, etc. may be overwritten, updated, or added to. The output information, etc. may be deleted. The input information, etc. may be sent to another device.

Each aspect/embodiment described in this disclosure may be used alone, in combination, or switched depending on the execution. In addition, notification of specific information (e.g., notification that "X is the case") is not limited to being done explicitly, but may be done implicitly (e.g., not notifying the specific information).

Software shall be interpreted broadly to mean instructions, instruction sets, code, code segments, program code, programs, subprograms, software modules, applications, software applications, software packages, routines, subroutines, objects, executable files, threads of execution, procedures, functions, etc., whether referred to as software, firmware, middleware, microcode, hardware description language, or otherwise.

In addition, software, instructions, information, etc. may be transmitted and received via a transmission medium. For example, if the software is transmitted from a website, server, or other remote source using at least one of wired technologies (such as coaxial cable, fiber optic cable, twisted pair, Digital Subscriber Line (DSL)), and/or wireless technologies (such as infrared, microwave, etc.), then at least one of these wired and/or wireless technologies is included within the definition of a transmission medium.

The block diagrams used to explain the embodiments show functional blocks. These functional blocks (components) are realized by any combination of at least one of hardware and software. Furthermore, the method of realizing each functional block is not particularly limited. That is, each functional block may be realized using one device that is physically or logically coupled, or may be realized using two or more devices that are physically or logically separated and connected directly or indirectly (e.g., using wires, wirelessly, etc.). The functional blocks may be realized by combining software with one device or multiple devices.

Functions include, but are not limited to, judgement, determination, judgment, calculation, processing, search, confirmation, receiving, transmitting, output, access, resolution, selection, comparison, broadcasting, notifying, communicating, forwarding, configuring, reconfiguring, allocating, mapping, and assigning. For example, a functional block (component) that enables transmission is called a transmitting unit or transmitter. As mentioned above, there are no particular limitations on the method of realization for either of these.

The term "apparatus" may be interpreted as a circuit, device, unit, etc. The hardware configuration of the system 10 for simultaneous playback of multiple performance sound sources may be configured to include one or more of the devices shown in the figure, or may be configured to exclude some of the devices.

Furthermore, each device such as the CPU 12 and memory 14 is connected by a bus for communicating information. The bus may be configured using a single bus, or may be configured using different buses between each device.

Furthermore, the system 10 for simultaneous playback of multiple performance sound sources may be configured to include hardware such as a micro CPU 12, a digital signal CPU 12 (DSP: Digital Signal Processor), an ASIC (Application Specific Integrated Circuit), a PLD (Programmable Logic Device), and an FPGA (Field Programmable Gate Array), and some or all of the functional blocks may be realized by the hardware. For example, the CPU 121001 may be implemented using at least one of these pieces of hardware.

The notification of information is not limited to the aspects/embodiments described in the present disclosure and may be performed using other methods. For example, the notification of information may be performed by physical layer signaling (e.g., Downlink Control Information (DCI), Uplink Control Information (UCI)), higher layer signaling (e.g., Radio Resource Control (RRC) signaling, Medium Access Control (MAC) signaling, broadcast information (Master Information Block (MIB), System Information Block (SIB)), other signals, or a combination thereof. In addition, the RRC signaling may be referred to as an RRC message, such as an RRC Connection Setup message, an RRC Connection Reconfiguration message, etc.

The system 10 for simultaneously playing multiple musical sound sources, the NAS, and the external playback device are connected to a network to recognize and link each other. One possible method of network connection in this case is mutual recognition using a communication protocol called UPnP (Universal Plug and Play). For example, by adopting an industry standard (guideline) such as DLNA (an abbreviation for the industry group Digital Living Network Alliance, "DLNA" is a registered trademark of said group) that makes products compatible with each other and enables network connections between electronic devices within the home, a system can be created in which any electronic device that adopts this industry standard can easily be connected to the network. DLNA is a standard for LAN connections, but other communication standards are not limited to LAN connections and may be adopted as long as they meet the requirements for music playback, such as transfer speed.

1 is a diagram showing the overall configuration of a system for simultaneously reproducing a plurality of performance sound sources according to a first embodiment of the present invention; 1 is a diagram showing a network communication configuration in a system for simultaneously reproducing a plurality of performance sound sources according to a first embodiment of the present invention; 1 is a diagram showing an operation screen in a simultaneous reproduction system for multiple performance sound sources according to a first embodiment of the present invention; FIG. 13 is a diagram showing a modified example of the operation screen in the system for simultaneously reproducing a plurality of performance sound sources according to the first embodiment of the present invention. 1 is a flowchart showing simultaneous reproduction of a plurality of performance sound sources in a simultaneous reproduction system for a plurality of performance sound sources according to a first embodiment of the present invention. 13 is a flowchart showing a process from launching an application to starting a session in a simultaneous reproduction system for a plurality of performance sound sources according to a second embodiment of the present invention. 13 is a flowchart showing a process from the start of a session to the saving of the session in a simultaneous reproduction system for a plurality of performance sound sources according to a second embodiment of the present invention. FIG. 13 is a diagram showing a screen for inviting a user to participate in a session in the simultaneous reproduction system for multiple performance sound sources according to the second embodiment of the present invention. FIG. 13 is a diagram showing a screen for inviting a user to participate in a session in the simultaneous reproduction system for multiple performance sound sources according to the second embodiment of the present invention. FIG. 13 is a diagram showing a screen for inviting a user to participate in a session in the simultaneous reproduction system for multiple performance sound sources according to the second embodiment of the present invention. 13 is a flowchart showing a process for adding a session member in a system for simultaneously reproducing a plurality of musical performance sound sources according to a second embodiment of the present invention. FIG. 13 is a diagram showing a screen for recording a performance in a simultaneous reproduction system for multiple performance sound sources according to a second embodiment of the present invention. FIG. 13 is a diagram showing a screen for recording a performance in a simultaneous reproduction system for multiple performance sound sources according to a second embodiment of the present invention. 11 is a flowchart of a performance recording process in a simultaneous reproduction system for multiple performance sound sources according to a second embodiment of the present invention. FIG. 11 is a diagram of a screen showing management and sharing of recorded data in a simultaneous playback system for multiple performance sound sources according to a second embodiment of the present invention. FIG. 11 is a diagram of a screen showing management and sharing of recorded data in a simultaneous playback system for multiple performance sound sources according to a second embodiment of the present invention. FIG. 11 is a diagram of a screen showing management and sharing of recorded data in a simultaneous playback system for multiple performance sound sources according to a second embodiment of the present invention. 13 is a flowchart showing a process of sharing a performance in a simultaneous reproduction system for a plurality of performance sound sources according to a second embodiment of the present invention. FIG. 13 is a diagram showing a performance collaboration setting screen in a simultaneous playback system for multiple performance sound sources according to a second embodiment of the present invention. FIG. 13 is a diagram showing a performance collaboration setting screen in a simultaneous playback system for multiple performance sound sources according to a second embodiment of the present invention. FIG. 13 is a diagram showing a performance collaboration setting screen in a simultaneous playback system for multiple performance sound sources according to a second embodiment of the present invention. FIG. 13 is a diagram showing a performance collaboration setting screen in a simultaneous playback system for multiple performance sound sources according to a second embodiment of the present invention. 13 is a flowchart showing a process for setting collaboration of musical performances in a simultaneous reproduction system for a plurality of musical performance sound sources according to a second embodiment of the present invention. FIG. 13 is a diagram showing a screen for detailed session information in a simultaneous playback system for multiple performance sound sources according to a second embodiment of the present invention. FIG. 13 is a diagram showing a screen for detailed session information in a simultaneous playback system for multiple performance sound sources according to a second embodiment of the present invention. FIG. 13 is a diagram showing a screen for detailed session information in a simultaneous playback system for multiple performance sound sources according to a second embodiment of the present invention. FIG. 13 is a diagram showing a screen for evaluating a published session in a simultaneous reproduction system for multiple performance sound sources according to a second embodiment of the present invention. 13 is a flowchart showing a process of participating in a public session in a simultaneous reproduction system for a plurality of musical performance sound sources according to a second embodiment of the present invention.

10: System for simultaneously reproducing multiple musical sound sources 12: CPU
14: Memory 16: Timer 18: Music information storage unit 20: Display control unit 22: Display unit 24: Performance control unit 26: Speaker 28: Operation unit 30: Communication unit 34: Network communication unit 38: Base station 40: Terminal 36: Distribution server 42: Operation screen 44: Section layout diagram 46: Drum icon 48: Bass icon 50: Piano icon 52: Guitar icon 54: Vocal icon 56: Trumpet icon 58: Trombone icon 60: Progression display bar 62: Performance part display bar 64: Reset icon 66: Listening position setting icon 68: Space size selection bar 70: Volume selection bar 72: Metronome volume adjustment bar 74: Rhythm section 76: Trumpet section 78: Trombone section 80: Saxophone section 82: Repeat display icon 84: Repeat time display icon 90: Session invitation screen 92: Approval icon 94: Resend icon 96: Master icon 98: Member icon 102: Role change screen 104: Execution icon 106 Reset volume icon 108 Mute icon 110 Recording entry screen 112 Session progress screen 114 Session information screen 116 Cloud share icon 118 Download icon 120 Collaboration screen 122 Solo icon 124 Rhythm section icon 126 Trumpet icon 128 Pin icon 130 Song name screen 132 Session members screen 134 Release selection screen 136 Session details screen 138 Listen icon 140 Participation request icon 142 Participation request icon 144 Recommendation screen 146 Popular sessions screen 148 Popular artists screen

Claims (24)

A method for simultaneously playing multiple (N) performance sound sources, comprising:
A stage in which each of the multiple performance sound sources is recorded individually and independently using multi-track recording;
setting a space size for playing a plurality of sound sources on an operation screen;
specifying a listening position when playing back the performance sound source on an operation screen;
calculating three-dimensional coordinates of the listening position based on the identified listening position;
Calculating the distance and direction between the position of each of the N-1 performance sound sources and the listening position;
a step of selecting one of N-1 performance sound sources 1, ...N-1 on an operation screen, and using object-based stereophonic processing to add three-dimensional position information to the performance sound source information, while performing a reproduction process of sounds from each performance sound source according to the positional relationship between the position of each performance sound source and the listening position based on the selected space size and listening position, thereby simulating and constructing a sound field within a specific space size;
and a step of simultaneously playing back the selected one or more recorded performance sound sources. The method for simultaneously playing multiple performance sound sources according to claim 1, which performs stereophonic processing based on a selected space size and listening position, including the sound direction of the performance of the multiple performance sound sources, and reflects volume differences, time differences, changes in frequency characteristics, phase changes, changes in reverberation, and the Doppler effect. The method for simultaneously playing multiple performance sound sources according to claim 1, wherein, when playing the multiple performance sound sources, synchronization is set between the selected performance sound sources before performing stereophonic processing. The method for simultaneously playing multiple performance sound sources according to claim 1, wherein the multiple performance sound sources are composed of a rhythm section including a piano, drums, bass, and guitar, a trumpet section consisting of multiple trumpets, a trombone section consisting of multiple trombones, and a saxophone section consisting of multiple saxophones, and the simultaneous playback step of the performance recording performance sound sources is a performance playback of a musical piece by a big band. The method for simultaneously playing multiple performance sound sources according to claim 1, which selects a listening position from multiple (N) performance sound sources, and then selects any one of the remaining performance sound sources 1, ..., N-1 as the performance sound source, thereby simultaneously playing back the multiple performance sound sources and using them as a practice accompaniment performance sound source as a performance at the listening position. The method for simultaneously playing multiple performance sound sources according to claim 1, in which an audience position is set separately from the multiple (N) performance sound sources, and an audience position is selected to simultaneously play the multiple performance sound sources to create a musical piece for appreciation that is played using multiple full performance sound sources. The method for simultaneously playing multiple performance sound sources according to claim 1, which is used to acquire a sense of rhythm by comparing the individual performance sound sources of each part in a rhythm section by selecting and playing any performance sound source of the rhythm section from among the multiple (N) performance sound sources, and selecting and playing another performance sound source of the rhythm section from among the multiple (N) performance sound sources. A method for simultaneously playing multiple performance sound sources as claimed in claim 1, in which a BPM value is selected when playing multiple performance sound sources in the case where one of the multiple performance sound sources and/or the listening position moves and the one selected as the performance sound source among the multiple performance sound sources moves during the performance. The method for simultaneously playing multiple performance sound sources according to claim 1, in which, after simultaneous playback of multiple performance sound sources is started based on a selected recorded performance sound source and a selected listening position, stereophonic processing is performed on the selected recorded performance sound source. The method for simultaneously playing multiple performance sound sources according to claim 1, which sets a playback start position and a playback end position in a selected piece of music and/or sets a number of repetitions, and when simultaneously playing multiple performance sound sources, repeats the performance up to the set playback start position and playback end position according to the set number of repetitions. A method for simultaneously playing multiple performance sound sources according to claim 1, in which the recording is performed using binaural processing when each of the multiple performance sound sources is recorded individually and independently using multi-track. 2. The method for simultaneously playing a plurality of performance sound sources according to claim 1, comprising the steps of: selecting a performance sound source to be muted from the plurality of performance sound sources when playing a master performance sound source;
recording the performance sound source while listening to the master performance sound source as an accompaniment through stereo headphones;
A method for simultaneously playing multiple performance sound sources, comprising the steps of: playing a piece of music through an additional participation type virtual session using a master performance sound source and a recorded performance sound source according to the method for simultaneously playing multiple performance sound sources described in claim 1. The method for simultaneously playing multiple performance sound sources according to claim 12, in which, when participating as an additional participation type virtual session, a response to an invitation from an existing participant and a participation request from a person wishing to participate is determined, and a decision is made as to whether or not to accept the participation request. The method for simultaneously playing multiple performance sound sources according to claim 13, in which, for each existing participant in the additional participation type virtual session, it is determined whether the participant is a Master who has the authority to set up the session and to reject participants, or a Member who has the authority only to reject the sharing of the participant's own recorded performance. The method for simultaneously playing multiple performance sound sources according to claim 13, in which the recorded performance sound source is shared by transferring it to an external storage unit when participating in the additional participation type virtual session. The method for simultaneously playing multiple performance sound sources according to claim 14, in which the Master decides which of the multiple performance sound sources will be combined with the Master performance sound source to create a collaboration song. The method for simultaneously playing multiple performance sound sources described in claim 15, in which the Master decides whether sharing of the collaboration song is limited to participants or made public to users of the method for simultaneously playing multiple performance sound sources described in claim 1. When a participant listens to a master performance sound source as accompaniment through stereo headphones and there is a recorded performance sound source, the next participant uses the master performance sound source and the recorded performance sound source as an accompaniment sound source by processing them into stereophonic sound using the method for simultaneously playing multiple performance sound sources described in claim 12. The method for simultaneously playing multiple performance sound sources according to claim 12, in which the performance sound source of the rhythm section including piano, drums, and bass is the master performance sound source, and the front section including guitar, saxophone, trumpet, and trombone participates. The method for simultaneously playing multiple performance sound sources according to claim 16, which presents evaluation results based on the number of plays and/or "like" responses to a published collaboration song. A method for simultaneously playing multiple performance sound sources according to claim 12, in which the tempo, key, and spatial sound processing conditions of the music are set in common between each participant for the accompaniment performance sound source that is the basis for participating in the virtual session in an additional participation type virtual session. A simultaneous playback program for a plurality of performance sound sources in a music playback device having storage means for storing music data consisting of a plurality of performance sound sources, playback means for playing back the music data, sound image localization processing means for processing a musical sound signal output from the playback means based on a control signal to localize a sound image as stereophonic sound at an arbitrary position in a three-dimensional sound space, and a localization control table defining the contents of the music and a localization method for localizing the corresponding sound image as stereophonic sound in the three-dimensional sound space, the contents of which can be arbitrarily changed by a user,
a control signal for transmitting to the sound image localization processing means a control signal for executing a sound image localization method in a three-dimensional acoustic space corresponding to the sound image localization processing means when the information acquired by the analysis means is determined to be information related to the content of the music piece defined in the localization control table; and a control signal for transmitting to the sound image localization processing means a control signal for transmitting to the sound image localization processing means a control signal for executing a sound image localization method in a three-dimensional acoustic space corresponding to the sound image localization processing means when the information acquired by the analysis means is determined to be information related to the content of the music piece defined in the localization control table. A system for simultaneously playing multiple performance sound sources, comprising: a control unit for controlling the operation of an entire music playback device when simultaneously playing multiple performance sound sources; a memory for storing various control programs and data; a music information storage unit for storing music data; a timer for outputting various timing signals; a music analysis unit for acquiring information related to the content of the music data stored in the music information storage unit; a display control unit for displaying various information on a display unit; a performance control unit for playing back the performance information included in the music data stored in the music information storage unit and outputting it to a speaker; an operation unit for inputting operation signals; a communication unit for transmitting and receiving signals to and from an external device; a display unit for displaying information related to the performance information; at least two speakers for stereoscopically playing back the performance information from the performance control unit; an operation unit for inputting various operation information; and a bus for transferring various data,
This system for simultaneously playing a plurality of performance sound sources is characterized in that it has a localization control table in which the content of a musical piece and a localization method for localizing a sound image as a corresponding stereophonic sound in a three-dimensional acoustic space are defined for each of the plurality of performance sound sources. The system for simultaneous playback of multiple performance sound sources according to claim 23, in which multiple simultaneous playback systems for multiple performance sound sources are provided, each of which has a simultaneous playback program for multiple performance sound sources, and the multiple simultaneous playback systems are connected by network communication, and the multiple performance sound source data simultaneously played in each of the multiple simultaneous playback systems can be accessed by each other, thereby allowing each simultaneous playback system to use the performance sound sources of the other simultaneous playback systems while recording its own performance in an additional participation type virtual session.

Images