WO2018148803A1 - A speech signal recognition sports match scoring system - Google Patents
A speech signal recognition sports match scoring system Download PDFInfo
- Publication number
- WO2018148803A1 WO2018148803A1 PCT/AU2018/050127 AU2018050127W WO2018148803A1 WO 2018148803 A1 WO2018148803 A1 WO 2018148803A1 AU 2018050127 W AU2018050127 W AU 2018050127W WO 2018148803 A1 WO2018148803 A1 WO 2018148803A1
- Authority
- WO
- WIPO (PCT)
- Prior art keywords
- field device
- data
- signal
- signals
- spoken
- Prior art date
Links
Classifications
-
- G—PHYSICS
- G08—SIGNALLING
- G08C—TRANSMISSION SYSTEMS FOR MEASURED VALUES, CONTROL OR SIMILAR SIGNALS
- G08C17/00—Arrangements for transmitting signals characterised by the use of a wireless electrical link
- G08C17/02—Arrangements for transmitting signals characterised by the use of a wireless electrical link using a radio link
-
- G—PHYSICS
- G10—MUSICAL INSTRUMENTS; ACOUSTICS
- G10L—SPEECH ANALYSIS TECHNIQUES OR SPEECH SYNTHESIS; SPEECH RECOGNITION; SPEECH OR VOICE PROCESSING TECHNIQUES; SPEECH OR AUDIO CODING OR DECODING
- G10L15/00—Speech recognition
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F2218/00—Aspects of pattern recognition specially adapted for signal processing
-
- G—PHYSICS
- G08—SIGNALLING
- G08C—TRANSMISSION SYSTEMS FOR MEASURED VALUES, CONTROL OR SIMILAR SIGNALS
- G08C2201/00—Transmission systems of control signals via wireless link
- G08C2201/30—User interface
- G08C2201/31—Voice input
-
- G—PHYSICS
- G08—SIGNALLING
- G08C—TRANSMISSION SYSTEMS FOR MEASURED VALUES, CONTROL OR SIMILAR SIGNALS
- G08C2201/00—Transmission systems of control signals via wireless link
- G08C2201/90—Additional features
- G08C2201/93—Remote control using other portable devices, e.g. mobile phone, PDA, laptop
Definitions
- This invention relates generally to autonomous computer scoring match tracking systems. More particularly, this invention relates to an autonomous speech signal recognition match scoring system.
- Scoring of sporting matches typically entails manual score keeping by an off-field scorer. However, scorers are not always available for all sporting events, especially nonprofessional sporting events.
- the umpire/referee of the sporting match may keep score, such is undesirous in that the umpire/referee generally cannot be distracted such as by take eyes off gameplay for input of data into a portable computing device, for example.
- the umpire/referee may issue hand score signals but which again require interpretation by an off-field scorer.
- the present invention seeks to provide a scoring system, which will overcome or substantially ameliorate at least some of the deficiencies of the prior art, or to at least provide an alternative.
- a sports match scoring system for semiautonomous scoring of matches that uses speech signal recognition in a ways that allows the umpire/referee to score the match without being unduly distracted such as by having to take eyes off gameplay.
- the system comprises an in-field device comprising an audio input which may receive and digitise umpire/referee speech signals and wirelessly transmit the speech signals using a wireless transmitter.
- the system may further comprise an off-field device comprising a wireless receiver for receiving the speech signals from the wireless transmitter of the in-field device.
- the off-field device comprises a signal acquisition controller for receiving and digitally sampling the speech signals and signal pre-processor and feature extraction controllers for preprocessing the signals and extracting features therefrom.
- the off-field device further comprises an automatic speech recognition (AS ) decoder for classifying patterns of the features as one or more spoken signals.
- AS automatic speech recognition
- the off-field device further comprises a scoring controller configured for calculating and updating an in-memory match score according to the one or more spoken signals classified by the ASR decoder.
- the system is configured for enhancing the classification accuracy of the ASR decoder.
- the off-field device may comprise a data dictionary of a finite of applicable spoken commands which, in embodiments, may be game type specific.
- the ASR decoder may classify the signals according to the finite set of available spoken commands from the dictionary, thereby enhancing the classification accuracy thereof.
- the off-field device may comprise a decision tree comprising a plurality of interconnected gameplay progress data nodes which may be used by the system to track gameplay progress in accordance with gameplay rules and events.
- the classification accuracy of the ASR decoder may be further enhanced with reference to the gameplay rules decision tree to filter out signal classifications which would be inapplicable according to the current gameplay progress.
- the classification accuracy of the ASR decoder and/or the scoring of the scoring controller may be further enhanced with auxiliary sensor data.
- the system may comprise an audio sensor and an associated audio analysis controller configured for detecting gameplay events according to audio signal patterns and updating the gameplay progress within the decision tree.
- the classification accuracy of the ASR decoder and/or the scoring of the scoring controller may be further enhanced with auxiliary vision data (which may include charged couples device (CCD) image sensors, Light Detection and Ranging (LIDAR) sensing sensors and the like) wherein the system comprises a video analysis controller for processing vision data received therefrom for detecting certain match events and updating the gameplay progress within the decision tree accordingly.
- auxiliary vision data which may include charged couples device (CCD) image sensors, Light Detection and Ranging (LIDAR) sensing sensors and the like
- the wireless communication between the in-field device and the off-field device may be bidirectional wherein the off-field device is able to transmit data indicative of the correct recognition or the inability to correctly recognise spoken signals which may be conveyed to the umpire/referee by way of a feedback device such as a haptic feedback device.
- the in-field device may comprise a user interface, such as a pushbutton interface so as to allow the umpire/referee to use pushbutton depresses for common scoring signals, such as "dot ball" signals in cricket.
- the in-field device may comprise an associated wirelessly connected handheld device having the pushbutton user interface thereon for the more convenient provision of these pushbutton inputs.
- the handheld device may comprise a pushbutton input for push-to-talk functionality.
- the in-field device may comprise an accelerometer and an associated motion controller configured classifying hand signal gestures of the umpire/referee.
- the accelerometer may be either inbuilt within the in-field device or take the form of an electronic wristband in wireless communication with the in-field device so as to be able to detect hand motions of the umpire/referee, such as the "wide" and "out” hand signals of the umpire/referee in cricket.
- the in-field device is further configured for transmitting acceleration data to the off-field device which uses the acceleration data for classifying the acceleration data according to a plurality of acceleration classification patents to identify hand gestures.
- the hand gestures may be used to further enhance the classification accuracy of the AS decoder.
- a speech signal recognition match scoring system comprising: an in-field device comprising: an audio input; a wireless transmitter for transmitting the speech signals received from the audio input; an off-field device comprising: a processor for processing digital data; a wireless receiver for receiving the speech signals from the wireless transmitter of the in-field device wireless receiver in operable communication with the processor across a system bus; a memory device for storing digital data, including computer program code instruction controllers and associated data, the memory device being in operable communication with the processor across the system bus wherein the controllers comprise: a signal acquisition controller for receiving digitised speech signals via the wireless receiver; signal preprocessor and feature extraction controllers for pre-processing the speech signals and extracting futures therefrom; an automated speech recognition (ASR) decoder for classifying patterns of the features as one or more spoken signals; and the scoring controller for calculating and updating an in- memory score according to the one or more spoken signals.
- ASR automated speech recognition
- the data may comprise a signal command dictionary comprising a finite set of signal commands and the ASR decoder may be configured for classifying the spoken signals according to the finite set of signal commands.
- the data may comprise a gameplay progress decision tree and the scoring controller may be configured for updating gameplay progress according to gameplay progress data nodes of the decision tree.
- the off-field device AS decoder may be further configured for classifying the one or more spoken signals according to a current node position of decision tree.
- the off-field device may be configured for calculating applicable spoken signals according to a current node position of the gameplay decision tree.
- the in-field device may further comprise a wireless receiver and the off-field device may further comprise a wireless transmitter configured for transmitting data from the off-field device to the in-field device and the off-field device may be configured for transmitting data indicative of a spoken signal being inapplicable to the current node position to the in-field device.
- the in-field device may further comprise a wireless receiver and the off-field device may further comprise a wireless transmitter configured for transmitting data from the off-field device to the in-field device.
- the off-field device may be configured for transmitting feedback data indicative of at least one of successful classification of a speech signal and the unsuccessful classification of a speech signal.
- the in-field device may comprise a feedback device and the feedback device may be configured for providing feedback when receiving the feedback data.
- the in-field device may further comprise a user interface and the in-field device transmitter may be configured for transmitting user interface data to the off-field device receiver and the scoring controller may be configured for calculating and updating the in-memory score further in accordance with the user interface data.
- the system may further comprise an in-field accelerometer and the in-field device transmitter may be configured for transmitting acceleration data to the off-field device receiver.
- the off-field device may comprise a motion analyser configured for template-based classifying the acceleration data according to at least one gesture pattern.
- the ASR decoder may be configured for further classifying the one or more spoken signals in accordance with the at least one gesture pattern.
- the system may further comprise a wrist wearable comprising the accelerometer, the wrist wearable being in wireless short range communication with the in-field device.
- the system may further comprise a handheld device in operable communication with the infield device and the handheld device may comprise a user interface and the handheld device may be configured for transmitting user interface data to the in-field device.
- the handheld device may be in wireless communication with the in-field device.
- Figure 1 shows a speech signal recognition match scoring system in accordance with an embodiment.
- FIG. 1 shows a speech recognition scoring system 100 in accordance with an embodiment.
- the system 100 comprises an in-field device 102 comprising an audio input 110.
- the in-field device 102 may comprise an analogue-to-digital (A/D) converter 111 for digitising speech signals received from the audio input 110.
- A/D analogue-to-digital
- the in-field device is used by the match umpire/referee within the sporting field/ring.
- the in-field device 102 further comprises a wireless transmitter 108 for transmitting the speech signals.
- the system 100 further comprises an off-field device comprising a processor 112 for processing digital data.
- the off-field device 100 may further comprises a wireless receiver 109 for receiving speech signals from the wireless transmitter 108 of the in-field device 102.
- the wireless receiver 109 is in operable communication with the processor 112 across a system bus 107.
- the off-field device 101 further comprises a memory device 106 for storing digital data, including computer program code instruction controllers and associated data.
- the memory device 106 is similarly in operable communication with the processor 112 across a system bus 107.
- the controllers comprise a signal acquisition controller 115 for receiving speech signals from the wireless receiver 109.
- the A/D converter 112 may sample speech signals from the audio input 110 at 8 kHz with 12 bits resolution.
- An analogue filter may be utilised between the audio input 110 and the A/D converter 111.
- the transmitter 108 and the receiver 109 digitally transmit the digitised speech signals from the field.
- the in-field device 112 may transmit analogue signals to the off-field device 101 wherein sampling is performed by the off-field device 101.
- the transmitter 108 and the receiver 109 may be relatively short range of Bluetooth transmitters/receivers, in other embodiments, the transmitter 108 and receiver 109 are preferably at least intermediate range transmitters and receivers so as, for example, to be able to transmit the speech signals over more than 100 m in certain applications such as for cricket matches.
- the off-field device 101 controllers may further comprise a signal pre-processor controller 116 and a feature extraction controller 117 for pre-processing the speech signals and extracting features therefrom.
- the pre-processor controller 116 may detect the beginning of a speech signal and remove DC harmonics, such as filtering 50 Hz soundcard interference ripples. The pre-processing controller 116 may further shift and normalise the speech signals.
- the feature extraction controller 117 may calculate a Fast Fourier Transform (FFT) of the signal an extract FFT coefficients between 100 Hz and 500 Hz for example.
- FFT Fast Fourier Transform
- the controllers may further comprise an automated speech recognition (AS ) decoder 118 for classifying the features as one or more spoken signals.
- AS automated speech recognition
- the ASR decoder 118 may implement a multilayer perceptron for pattern classification.
- the functionality of at least one of the signal acquisition, preprocessor feature extraction and ASR decoding controllers 115 - 118 may be implemented by a commercially available speech recognition API such as the GoogleTM Speech API, SpeechmaticsTM ASR and the like.
- the controllers may further comprise a scoring controller 119 configured for calculating and updating an in-memory score 120 according to the one or more spoken words recognised by the ASR decoder 118.
- the off-field device 101 comprises a video output 131 for outputting a graphical representation of the calculated score on a digital display 128. Furthermore, the off-field device 101 may comprise a network interface card 130 for transmitting the score data across a data network 127, such as the Internet.
- the in-field device 102 is similarly a digital device comprising a processor 112 and a memory device 106 operably coupled thereto across a system bus 107, the memory device 106 similarly having computer program code instruction controllers therein for configuring the functionality of the digital in-field device 102.
- the audio device 110 may be a suitably located microphone either inbuilt or connected via an electrical lead to the lapel or the like of the referee so as to be suitably located for receiving spoken commands.
- the in-field device 102 may be a small form factor portable battery-powered electronic device.
- the ASR decoder 118 classifies spoken signals according to a finite set of signal commands from a data dictionary 124 in memory 106.
- the dictionary 124 may be further configured with a finite set of signal commands particular to a sports type.
- the dictionary 124 may be configured with a finite set of in play signal commands such as “dot ball”, “out”, “no-ball", “free hit”, “wide”, “dead ball” and the like and scoring signals such as “1 Run”, “2 Runs”, “3 Runs”, “Four”, “5 Runs”, “Six”, “Bye”, “Leg bye, “Short run”, “Television replay”, “Penalty runs”, “Last hour”, “Revoke last signal", “New Ball”, “Challenge System”, “Power Play” and the like.
- the memory device 106 of the off-field device 101 comprises a gameplay progress decision tree 123.
- off-field device 101 may track gameplay progress by branching to various gameplay progress data nodes of the decision tree 123 according to a set of gameplay rules 122 and gameplay events, including detected gameplay events.
- the decision tree 123 and gameplay rules 122 for a game of cricket may currently represent the gameplay progress of the current ball.
- a spoken signal is received for the current ball (i.e. "one run”, “no-ball” and the like)
- the current position within the decision tree 123 will move to the next ball position.
- the scoring controller 119 won't allow gameplay progress to progress up the decision tree 123 a signal is received for the current ball.
- the decision tree 123 may be utilised for enhancing the classification accuracy of the ASR decoder 118.
- the ASR decoder 118 may reference the decision tree 123 to confine the applicable set of possible spoken signals applicable for the particular gameplay position and filter out inapplicable spoken signals.
- the ASR decoder 118 may infer from the decision tree 123 that a subsequent classification of spoken signals of, for example "fifteen-thirty", “love-forty” and the like would be incorrect classifications of the signals.
- the scoring controller 180 would also may also make reference to the gameplay progress of the decision tree 123 when calculating and updating the match score 120.
- wireless data transmission between the in-field device 102 and the off field device 101 may be bidirectional in that the off-field device 101 comprises a transmitter 108 for wirelessly transmitting data to a receiver 109 of the in-field device 102.
- off-field device 101 may be configured for transmitting data indicative of the correct recognition or the inability to recognise a spoken signal.
- the in-field device 100 to may comprise a feedback device 113, such as a vibrational haptic device.
- a short pulse of the haptic device may be indicative to the umpire/referee of the correct recognition of the spoken signal whereas two short pulses may be indicative of the inability to recognise the spoken signal, prompting the umpire/referee to reissue the signal until such time that a short pulse is received.
- off-field device 101 may wirelessly transmit an indication of such to the in-field device 102.
- the umpire may miscalculate the number of balls bowled in an over and allow seventh ball to be bowled.
- the feedback device 103 may provide a long pulse indicative of the potential gameplay error prompting rectification by the umpire/referee oi possible.
- the in-field device 100 to may comprise a user interface 114, such as a pushbutton interface which may, for example, be used to input more commonly used signals, such as a "dot ball” signals in cricket. As such, as opposed to having to verbalise this signal for each ball bowled, the umpire may rather depress the button user interface 114.
- a user interface 114 such as a pushbutton interface which may, for example, be used to input more commonly used signals, such as a "dot ball” signals in cricket.
- the umpire may rather depress the button user interface 114.
- the in-field device 102 is further in operable communication with a handheld device 103.
- the handheld device 103 may be in wireless communication with the infield device 102 by way of a wireless data connection, such as a Bluetooth data connection.
- the handheld device 103 may comprise the user interface 114 in this embodiment and may take the form of a smaller more conveniently held handheld device more ergonomically used for depressing the pushbutton user interface 114.
- the handheld device 103 may comprise a push-to-talk button for selectively capturing speech signals.
- auxiliary data may be received and analysed by the off-field device 101 to detect certain signals and gameplay events.
- the in-field device 102 comprises a Micro-electromechanical Systems (MEMS) accelerometer 112, such as one which is worn on the wrist of the umpire/referee, and an associated motion analyser controller 135.
- MEMS Micro-electromechanical Systems
- the accelerometer 112 monitors acceleration such that the motion analyser controller 135 is able to infer hand gestures from the motion of the wrist(s) of the umpire/referee.
- the motion analysis controller 135 employs template based matching wherein a plurality of acceleration classification patterns 133 store reference patterns for each class/gesture for matching against the acceleration data using similarity measurements, such as Euclidean distance matching.
- the motion analysis controller 135 may employ model- based methods based on the probabilistic interpretation of acceleration data such as by using Hidden Markov Models (HMM)
- the acceleration classification patterns 133 may comprise a "wide" signal acceleration pattern indicative of the umpires wrist moving from side to side. Furthermore, the acceleration classification patterns 133 may comprise an "out" acceleration pattern indicative of the umpire's wrist moving upwardly.
- the acceleration pattern classification may be utilised to enhance the detection accuracy of the AS decoder 118 wherein, for example, should the spoken command be inaudible and may potentially be classified as both "out” or "wide", the accelerometer data may be determinative for the correct classification.
- the utilisation of the accelerometer data may allow the umpire to issue spoken or hand gesture commands alternatively.
- audio data may be analysed/classified wherein the system 100 comprises an in-field audio sensor 105 and audio analyser 125 for detecting/classifying audio signals received from the audio sensor 105 according to various well-known audio signal classification (ASC) techniques.
- ASC audio signal classification
- the audio sensor 105 may be located near the batsmen so as to be able to classify audio signals indicative of a ball snick event.
- the audio analyser 125 is able to classify the "ball snick" signature which, if followed by a wicket-keeper "glove strike” or “crowd cheering" audio signature event, is determinative of an "out” signal.
- the off-field device may provide the long pulse signal via the feedback device 113 indicative of the potential inaccuracy of the signal.
- the audio signature classification may be utilised to enhance the classification accuracy of the ASR decoder 118.
- the gameplay progress of the decision tree 123 may be updated to indicate the successful hitting of the ball wherein, for a subsequent spoken signal from the umpire which could either be classified as "wide” or "1 run", the latter may be favoured as the correct classification by the ASR decoder 118.
- vision data may be analysed wherein the system 100 comprises an image sensor 104 and/or LIDAR sensor 134 and an associated vision analyser 126.
- the vision analyser 126 may implement motion detection of the vision signals received via the image sensor 104 to detect a "batsmen crossing the pitch" event and update the gameplay position within the decision tree 123 accordingly.
- sensors may be utilised such as boundary laser sensors, cricket wicked/tennis net accelerometers and the like.
- the scoring controller 119 may utilise match specific data 121 within memory 106.
- player numbers may rather be provided on field.
- the umpire may issue concatenated signals such as "player 2 - bye - 1 run”.
- the signal dictionary 124 may comprise multilingual signals so as to allow the recognition of spoken signals in differing languages.
- the ASR decoder 118 and the signal dictionary 124 may be configured for allowing bilingual or multilingual capabilities such that a single installation may recognise spoken command signals in differing languages alternately, such as for karate matches for receiving signal commands in both English and Japanese.
Landscapes
- Engineering & Computer Science (AREA)
- Physics & Mathematics (AREA)
- Computer Networks & Wireless Communication (AREA)
- General Physics & Mathematics (AREA)
- Computational Linguistics (AREA)
- Health & Medical Sciences (AREA)
- Audiology, Speech & Language Pathology (AREA)
- Human Computer Interaction (AREA)
- Acoustics & Sound (AREA)
- Multimedia (AREA)
- Arrangements For Transmission Of Measured Signals (AREA)
Abstract
A speech signal recognition sports scoring system comprises an in-field device comprising an audio input which receives and wirelessly transmits speech signals using a wireless transmitter to an off-field device comprising. The off-field device comprises an automatic speech recognition (ASR) decoder for classifying patterns of the features as one or more spoken signals from a signal command dictionary and a scoring controller configured for calculating and updating an in-memory match score according to the one or more spoken signals classified by the ASR decoder and a gameplay progress decision tree. The current node position within the gameplay progress decision tree may be used to enhance the classification accuracy of the ASR decoder.
Description
A speech signal recognition sports match scoring system
Field of the Invention
[1] This invention relates generally to autonomous computer scoring match tracking systems. More particularly, this invention relates to an autonomous speech signal recognition match scoring system.
Background of the Invention
[2] Scoring of sporting matches typically entails manual score keeping by an off-field scorer. However, scorers are not always available for all sporting events, especially nonprofessional sporting events.
[3] Whereas the umpire/referee of the sporting match may keep score, such is undesirous in that the umpire/referee generally cannot be distracted such as by take eyes off gameplay for input of data into a portable computing device, for example.
[4] In certain sporting events, such as Australian football league (AFL) and cricket, the umpire/referee may issue hand score signals but which again require interpretation by an off-field scorer.
[5] The present invention seeks to provide a scoring system, which will overcome or substantially ameliorate at least some of the deficiencies of the prior art, or to at least provide an alternative.
[6] It is to be understood that, if any prior art information is referred to herein, such reference does not constitute an admission that the information forms part of the common general knowledge in the art, in Australia or any other country.
Summary of the Disclosure
[7] There is provided herein a sports match scoring system for semiautonomous scoring of matches that uses speech signal recognition in a ways that allows the umpire/referee to score the match without being unduly distracted such as by having to take eyes off gameplay.
[8] In an embodiment, the system comprises an in-field device comprising an audio input which may receive and digitise umpire/referee speech signals and wirelessly transmit the speech signals using a wireless transmitter. The system may further comprise an off-field device comprising a wireless receiver for receiving the speech signals from the wireless transmitter of the in-field device.
[9] The off-field device comprises a signal acquisition controller for receiving and digitally sampling the speech signals and signal pre-processor and feature extraction controllers for preprocessing the signals and extracting features therefrom.
[10] The off-field device further comprises an automatic speech recognition (AS ) decoder for classifying patterns of the features as one or more spoken signals.
[11] As such, the off-field device further comprises a scoring controller configured for calculating and updating an in-memory match score according to the one or more spoken signals classified by the ASR decoder.
[12] In embodiments, the system is configured for enhancing the classification accuracy of the ASR decoder.
[13] For example, in one embodiment, the off-field device may comprise a data dictionary of a finite of applicable spoken commands which, in embodiments, may be game type specific. As such, when classifying the spoken signals, the ASR decoder may classify the signals according to the finite set of available spoken commands from the dictionary, thereby enhancing the classification accuracy thereof.
[14] In further in embodiments, the off-field device may comprise a decision tree comprising a plurality of interconnected gameplay progress data nodes which may be used by the system to track gameplay progress in accordance with gameplay rules and events.
[15] The classification accuracy of the ASR decoder may be further enhanced with reference to the gameplay rules decision tree to filter out signal classifications which would be inapplicable according to the current gameplay progress.
[16] In further embodiments, the classification accuracy of the ASR decoder and/or the scoring of the scoring controller may be further enhanced with auxiliary sensor data. For example, in one embodiment, the system may comprise an audio sensor and an associated audio analysis controller configured for detecting gameplay events according to audio signal patterns and updating the gameplay progress within the decision tree.
[17] In further embodiment, the classification accuracy of the ASR decoder and/or the scoring of the scoring controller may be further enhanced with auxiliary vision data (which may include charged couples device (CCD) image sensors, Light Detection and Ranging (LIDAR) sensing sensors and the like) wherein the system comprises a video analysis controller for processing vision data received therefrom for detecting certain match events and updating the gameplay progress within the decision tree accordingly.
[18] In embodiments, the wireless communication between the in-field device and the off-field device may be bidirectional wherein the off-field device is able to transmit data indicative of the correct recognition or the inability to correctly recognise spoken signals which may be conveyed to the umpire/referee by way of a feedback device such as a haptic feedback device.
[19] In further embodiments, the in-field device may comprise a user interface, such as a pushbutton interface so as to allow the umpire/referee to use pushbutton depresses for common scoring signals, such as "dot ball" signals in cricket. In embodiments, the in-field device may comprise
an associated wirelessly connected handheld device having the pushbutton user interface thereon for the more convenient provision of these pushbutton inputs. In embodiments, the handheld device may comprise a pushbutton input for push-to-talk functionality.
[20] In further embodiments, the in-field device may comprise an accelerometer and an associated motion controller configured classifying hand signal gestures of the umpire/referee. The accelerometer may be either inbuilt within the in-field device or take the form of an electronic wristband in wireless communication with the in-field device so as to be able to detect hand motions of the umpire/referee, such as the "wide" and "out" hand signals of the umpire/referee in cricket.
[21] In accordance with this embodiment, the in-field device is further configured for transmitting acceleration data to the off-field device which uses the acceleration data for classifying the acceleration data according to a plurality of acceleration classification patents to identify hand gestures.
[22] In embodiments, the hand gestures may be used to further enhance the classification accuracy of the AS decoder.
[23] In accordance with one aspect, there is provided a speech signal recognition match scoring system comprising: an in-field device comprising: an audio input; a wireless transmitter for transmitting the speech signals received from the audio input; an off-field device comprising: a processor for processing digital data; a wireless receiver for receiving the speech signals from the wireless transmitter of the in-field device wireless receiver in operable communication with the processor across a system bus; a memory device for storing digital data, including computer program code instruction controllers and associated data, the memory device being in operable communication with the processor across the system bus wherein the controllers comprise: a signal acquisition controller for receiving digitised speech signals via the wireless receiver; signal preprocessor and feature extraction controllers for pre-processing the speech signals and extracting futures therefrom; an automated speech recognition (ASR) decoder for classifying patterns of the features as one or more spoken signals; and the scoring controller for calculating and updating an in- memory score according to the one or more spoken signals.
[24] The data may comprise a signal command dictionary comprising a finite set of signal commands and the ASR decoder may be configured for classifying the spoken signals according to the finite set of signal commands.
[25] The data may comprise a gameplay progress decision tree and the scoring controller may be configured for updating gameplay progress according to gameplay progress data nodes of the decision tree.
[26] The off-field device AS decoder may be further configured for classifying the one or more spoken signals according to a current node position of decision tree.
[27] In use, the off-field device may be configured for calculating applicable spoken signals according to a current node position of the gameplay decision tree.
[28] The in-field device may further comprise a wireless receiver and the off-field device may further comprise a wireless transmitter configured for transmitting data from the off-field device to the in-field device and the off-field device may be configured for transmitting data indicative of a spoken signal being inapplicable to the current node position to the in-field device.
[29] The in-field device may further comprise a wireless receiver and the off-field device may further comprise a wireless transmitter configured for transmitting data from the off-field device to the in-field device.
[30] In use, the off-field device may be configured for transmitting feedback data indicative of at least one of successful classification of a speech signal and the unsuccessful classification of a speech signal.
[31] The in-field device may comprise a feedback device and the feedback device may be configured for providing feedback when receiving the feedback data.
[32] The in-field device may further comprise a user interface and the in-field device transmitter may be configured for transmitting user interface data to the off-field device receiver and the scoring controller may be configured for calculating and updating the in-memory score further in accordance with the user interface data.
[33] The system may further comprise an in-field accelerometer and the in-field device transmitter may be configured for transmitting acceleration data to the off-field device receiver.
[34] The off-field device may comprise a motion analyser configured for template-based classifying the acceleration data according to at least one gesture pattern.
[35] The ASR decoder may be configured for further classifying the one or more spoken signals in accordance with the at least one gesture pattern.
[36] The system may further comprise a wrist wearable comprising the accelerometer, the wrist wearable being in wireless short range communication with the in-field device.
[37] The system may further comprise a handheld device in operable communication with the infield device and the handheld device may comprise a user interface and the handheld device may be configured for transmitting user interface data to the in-field device.
[38] The handheld device may be in wireless communication with the in-field device.
[39] Other aspects of the invention are also disclosed.
Brief Description of the Drawings
[40] Notwithstanding any other forms which may fall within the scope of the present invention, preferred embodiments of the disclosure will now be described, by way of example only, with reference to the accompanying drawings in which:
[41] Figure 1 shows a speech signal recognition match scoring system in accordance with an embodiment.
Description of Embodiments
[42] Figure 1 shows a speech recognition scoring system 100 in accordance with an embodiment. The system 100 comprises an in-field device 102 comprising an audio input 110. In embodiments, the in-field device 102 may comprise an analogue-to-digital (A/D) converter 111 for digitising speech signals received from the audio input 110. The in-field device is used by the match umpire/referee within the sporting field/ring.
[43] The in-field device 102 further comprises a wireless transmitter 108 for transmitting the speech signals.
[44] The system 100 further comprises an off-field device comprising a processor 112 for processing digital data. The off-field device 100 may further comprises a wireless receiver 109 for receiving speech signals from the wireless transmitter 108 of the in-field device 102. The wireless receiver 109 is in operable communication with the processor 112 across a system bus 107.
[45] The off-field device 101 further comprises a memory device 106 for storing digital data, including computer program code instruction controllers and associated data. The memory device 106 is similarly in operable communication with the processor 112 across a system bus 107.
[46] The controllers comprise a signal acquisition controller 115 for receiving speech signals from the wireless receiver 109. For example, the A/D converter 112 may sample speech signals from the audio input 110 at 8 kHz with 12 bits resolution. An analogue filter may be utilised between the audio input 110 and the A/D converter 111.
[47] In embodiments, the transmitter 108 and the receiver 109 digitally transmit the digitised speech signals from the field. Alternatively, the in-field device 112 may transmit analogue signals to the off-field device 101 wherein sampling is performed by the off-field device 101.
[48] Whereas in embodiments, the transmitter 108 and the receiver 109 may be relatively short range of Bluetooth transmitters/receivers, in other embodiments, the transmitter 108 and receiver 109 are preferably at least intermediate range transmitters and receivers so as, for example, to be able to transmit the speech signals over more than 100 m in certain applications such as for cricket matches.
[49] The off-field device 101 controllers may further comprise a signal pre-processor controller 116 and a feature extraction controller 117 for pre-processing the speech signals and extracting features therefrom.
[50] The pre-processor controller 116 may detect the beginning of a speech signal and remove DC harmonics, such as filtering 50 Hz soundcard interference ripples. The pre-processing controller 116 may further shift and normalise the speech signals.
[51] The feature extraction controller 117 may calculate a Fast Fourier Transform (FFT) of the signal an extract FFT coefficients between 100 Hz and 500 Hz for example.
[52] The controllers may further comprise an automated speech recognition (AS ) decoder 118 for classifying the features as one or more spoken signals. The ASR decoder 118 may implement a multilayer perceptron for pattern classification.
[53] The output of the ASR decoder 118 are recognised spoken signals.
[54] In embodiments, the functionality of at least one of the signal acquisition, preprocessor feature extraction and ASR decoding controllers 115 - 118 may be implemented by a commercially available speech recognition API such as the Google™ Speech API, Speechmatics™ ASR and the like.
[55] The controllers may further comprise a scoring controller 119 configured for calculating and updating an in-memory score 120 according to the one or more spoken words recognised by the ASR decoder 118.
[56] In embodiments, the off-field device 101 comprises a video output 131 for outputting a graphical representation of the calculated score on a digital display 128. Furthermore, the off-field device 101 may comprise a network interface card 130 for transmitting the score data across a data network 127, such as the Internet.
[57] In embodiments, the in-field device 102 is similarly a digital device comprising a processor 112 and a memory device 106 operably coupled thereto across a system bus 107, the memory device 106 similarly having computer program code instruction controllers therein for configuring the functionality of the digital in-field device 102. The audio device 110 may be a suitably located microphone either inbuilt or connected via an electrical lead to the lapel or the like of the referee so as to be suitably located for receiving spoken commands.
[58] Typically, the in-field device 102 may be a small form factor portable battery-powered electronic device.
[59] In embodiments, the ASR decoder 118 classifies spoken signals according to a finite set of signal commands from a data dictionary 124 in memory 106.
[60] Furthermore, the dictionary 124 may be further configured with a finite set of signal commands particular to a sports type. For example, for cricket, the dictionary 124 may be configured
with a finite set of in play signal commands such as "dot ball", "out", "no-ball", "free hit", "wide", "dead ball" and the like and scoring signals such as "1 Run", "2 Runs", "3 Runs", "Four", "5 Runs", "Six", "Bye", "Leg bye, "Short run", "Television replay", "Penalty runs", "Last hour", "Revoke last signal", "New Ball", "Challenge System", "Power Play" and the like.
[61] By limiting the set of signal commands to a finite set of signal commands within the dictionary 124 including for a particular type of sport, the recognition accuracy of the ACR decoder 118 is increased.
[62] In further embodiments, the memory device 106 of the off-field device 101 comprises a gameplay progress decision tree 123. During gameplay, off-field device 101 may track gameplay progress by branching to various gameplay progress data nodes of the decision tree 123 according to a set of gameplay rules 122 and gameplay events, including detected gameplay events.
[63] For example, the decision tree 123 and gameplay rules 122 for a game of cricket may currently represent the gameplay progress of the current ball. When a spoken signal is received for the current ball (i.e. "one run", "no-ball" and the like), the current position within the decision tree 123 will move to the next ball position. In embodiments, the scoring controller 119 won't allow gameplay progress to progress up the decision tree 123 a signal is received for the current ball.
[64] In embodiments, the decision tree 123 may be utilised for enhancing the classification accuracy of the ASR decoder 118.
[65] For example, when classifying the features into one or more spoken signals, the ASR decoder 118 may reference the decision tree 123 to confine the applicable set of possible spoken signals applicable for the particular gameplay position and filter out inapplicable spoken signals.
[66] For example, should the gameplay progress of the decision tree 123 for a tennis match indicate that the current score is love-fifteen, the ASR decoder 118 may infer from the decision tree 123 that a subsequent classification of spoken signals of, for example "fifteen-thirty", "love-forty" and the like would be incorrect classifications of the signals.
[67] As such, by tracking gameplay progress with reference to the decision tree 123 and rules 122, the signal classification accuracy of the ASR decoder 118 is enhanced.
[68] The scoring controller 180 would also may also make reference to the gameplay progress of the decision tree 123 when calculating and updating the match score 120.
[69] For example, for cricket, when receiving the "bye" signal, the scoring controller 119 would update the gameplay progress of the decision tree 123 such that when receiving a subsequent "1 run" scoring signal, the scoring controller 118 would, by referencing the gameplay position of the decision tree, increment the team's total by one run but leave the current batsman's score unchanged.
[70] In embodiments, wireless data transmission between the in-field device 102 and the off field device 101 may be bidirectional in that the off-field device 101 comprises a transmitter 108 for wirelessly transmitting data to a receiver 109 of the in-field device 102.
[71] In embodiments, off-field device 101 may be configured for transmitting data indicative of the correct recognition or the inability to recognise a spoken signal.
[72] Furthermore, the in-field device 100 to may comprise a feedback device 113, such as a vibrational haptic device. As such, a short pulse of the haptic device may be indicative to the umpire/referee of the correct recognition of the spoken signal whereas two short pulses may be indicative of the inability to recognise the spoken signal, prompting the umpire/referee to reissue the signal until such time that a short pulse is received.
[73] In embodiments, should the off-field device 101 determine that a signal is illogical in accordance with a current gameplay position of the decision tree 123, off-field device 101 may wirelessly transmit an indication of such to the in-field device 102. For example, the umpire may miscalculate the number of balls bowled in an over and allow seventh ball to be bowled. As such, the feedback device 103 may provide a long pulse indicative of the potential gameplay error prompting rectification by the umpire/referee oi possible.
[74] In embodiments, the in-field device 100 to may comprise a user interface 114, such as a pushbutton interface which may, for example, be used to input more commonly used signals, such as a "dot ball" signals in cricket. As such, as opposed to having to verbalise this signal for each ball bowled, the umpire may rather depress the button user interface 114.
[75] In embodiments, the in-field device 102 is further in operable communication with a handheld device 103. In embodiments, the handheld device 103 may be in wireless communication with the infield device 102 by way of a wireless data connection, such as a Bluetooth data connection.
[76] The handheld device 103 may comprise the user interface 114 in this embodiment and may take the form of a smaller more conveniently held handheld device more ergonomically used for depressing the pushbutton user interface 114.
[77] In embodiments, the handheld device 103 may comprise a push-to-talk button for selectively capturing speech signals.
[78] In embodiments, so as to further enhance the classification accuracy of the AS decoder 118 and/or the accuracy of the scoring controller 118, auxiliary data may be received and analysed by the off-field device 101 to detect certain signals and gameplay events.
[79] In one embodiment, the in-field device 102 comprises a Micro-electromechanical Systems (MEMS) accelerometer 112, such as one which is worn on the wrist of the umpire/referee, and an associated motion analyser controller 135. As such, in accordance with this embodiment, the
accelerometer 112 monitors acceleration such that the motion analyser controller 135 is able to infer hand gestures from the motion of the wrist(s) of the umpire/referee.
[80] In embodiments, the motion analysis controller 135 employs template based matching wherein a plurality of acceleration classification patterns 133 store reference patterns for each class/gesture for matching against the acceleration data using similarity measurements, such as Euclidean distance matching. Alternatively, the motion analysis controller 135 may employ model- based methods based on the probabilistic interpretation of acceleration data such as by using Hidden Markov Models (HMM)
[81] For example, the acceleration classification patterns 133 may comprise a "wide" signal acceleration pattern indicative of the umpires wrist moving from side to side. Furthermore, the acceleration classification patterns 133 may comprise an "out" acceleration pattern indicative of the umpire's wrist moving upwardly.
[82] In one embodiment, the acceleration pattern classification may be utilised to enhance the detection accuracy of the AS decoder 118 wherein, for example, should the spoken command be inaudible and may potentially be classified as both "out" or "wide", the accelerometer data may be determinative for the correct classification.
[83] Furthermore, in embodiments, the utilisation of the accelerometer data may allow the umpire to issue spoken or hand gesture commands alternatively.
[84] In further embodiments, audio data may be analysed/classified wherein the system 100 comprises an in-field audio sensor 105 and audio analyser 125 for detecting/classifying audio signals received from the audio sensor 105 according to various well-known audio signal classification (ASC) techniques.
[85] For example, for cricket, the audio sensor 105 may be located near the batsmen so as to be able to classify audio signals indicative of a ball snick event. The audio analyser 125 is able to classify the "ball snick" signature which, if followed by a wicket-keeper "glove strike" or "crowd cheering" audio signature event, is determinative of an "out" signal. By way of further example, should the audio analyser 125 classify the "ball snick" signature and the umpire/referee/issue the "wide" verbal signal, the off-field device may provide the long pulse signal via the feedback device 113 indicative of the potential inaccuracy of the signal.
[86] In embodiments, the audio signature classification may be utilised to enhance the classification accuracy of the ASR decoder 118. For example, when classifying a "bat strike" audio signature, the gameplay progress of the decision tree 123 may be updated to indicate the successful hitting of the ball wherein, for a subsequent spoken signal from the umpire which could either be
classified as "wide" or "1 run", the latter may be favoured as the correct classification by the ASR decoder 118.
[87] In further embodiments, vision data may be analysed wherein the system 100 comprises an image sensor 104 and/or LIDAR sensor 134 and an associated vision analyser 126.
[88] For example, the vision analyser 126 may implement motion detection of the vision signals received via the image sensor 104 to detect a "batsmen crossing the pitch" event and update the gameplay position within the decision tree 123 accordingly.
[89] Similarly, such an updated gameplay position may be used to subsequently enhance the detection accuracy of the ASR decoder 118.
[90] In embodiments, other sensors may be utilised such as boundary laser sensors, cricket wicked/tennis net accelerometers and the like.
[91] In embodiments, the scoring controller 119 may utilise match specific data 121 within memory 106. For example, as opposed to the umpire/referee having to memorise player names, player numbers may rather be provided on field. As such, the umpire may issue concatenated signals such as "player 2 - bye - 1 run".
[92] In embodiments, the signal dictionary 124 may comprise multilingual signals so as to allow the recognition of spoken signals in differing languages. In embodiments the ASR decoder 118 and the signal dictionary 124 may be configured for allowing bilingual or multilingual capabilities such that a single installation may recognise spoken command signals in differing languages alternately, such as for karate matches for receiving signal commands in both English and Japanese.
[93] Whereas the embodiment shown in figure 1 shows two devices 102, 101 in wireless communication with each other, it should be appreciated that, in other embodiments, a differing system configuration may be employed within the purposive scope of the embodiments described herein.
[94] The foregoing description, for purposes of explanation, used specific nomenclature to provide a thorough understanding of the invention. However, it will be apparent to one skilled in the art that specific details are not required in order to practice the invention. Thus, the foregoing descriptions of specific embodiments of the invention are presented for purposes of illustration and description. They are not intended to be exhaustive or to limit the invention to the precise forms disclosed; obviously, many modifications and variations are possible in view of the above teachings. The embodiments were chosen and described in order to best explain the principles of the invention and its practical applications, they thereby enable others skilled in the art to best utilize the invention and various embodiments with various modifications as are suited to the particular use contemplated. It is intended that the following claims and their equivalents define the scope of the invention.
Claims
1. A speech signal recognition match scoring system comprising:
an in-field device comprising:
an audio input;
a wireless transmitter for transmitting the speech signals received from the audio input;
an off-field device comprising:
a processor for processing digital data;
a wireless receiver for receiving the speech signals from the wireless transmitter of the in-field device wireless receiver in operable communication with the processor across a system bus;
a memory device for storing digital data, including computer program code instruction controllers and associated data, the memory device being in operable communication with the processor across the system bus wherein the controllers comprise:
a signal acquisition controller for receiving digitised speech signals via the wireless receiver;
signal pre-processor and feature extraction controllers for pre-processing the speech signals and extracting futures therefrom;
an automated speech recognition (ASR) decoder for classifying patterns of the features as one or more spoken signals; and
the scoring controller for calculating and updating an in-memory score according to the one or more spoken signals.
2. A system as claimed in claim 1, wherein the data comprises a signal command dictionary comprising a finite set of signal commands and wherein the ASR decoder is configured for classifying the spoken signals according to the finite set of signal commands.
3. A system as claimed in claim 1, wherein the data comprises a gameplay progress decision tree and wherein the scoring controller is configured for updating gameplay progress according to gameplay progress data nodes of the decision tree.
4. A system as claimed in claim 3, wherein the off-field device ASR decoder is further configured for classifying the one or more spoken signals according to a current node position of decision tree.
5. A system as claimed in claim 3, wherein, in use, the off-field device is configured for calculating applicable spoken signals according to a current node position of the gameplay decision tree.
6. A system as claimed in claim 5, wherein the in-field device further comprises a wireless receiver and wherein the off-field device further comprises a wireless transmitter configured for transmitting data from the off-field device to the in-field device and wherein the off-field device is configured for transmitting data indicative of a spoken signal being inapplicable to the current node position to the in-field device.
7. A system as claimed in claim 1, wherein the in-field device further comprises a wireless receiver and wherein the off-field device further comprises a wireless transmitter configured for transmitting data from the off-field device to the in-field device.
8. A system as claimed in claim 7, in use, the off-field device is configured for transmitting feedback data indicative of at least one of successful classification of a speech signal and the unsuccessful classification of a speech signal.
9. A system as claimed in claim 8, wherein the in-field device comprises a feedback device and wherein the feedback device is configured for providing feedback when receiving the feedback data.
10. A system as claimed in claim 1, wherein the in-field device further comprises a user interface and wherein the in-field device transmitter is configured for transmitting user interface data to the off-field device receiver and wherein the scoring controller is configured for calculating and updating the in-memory score further in accordance with the user interface data.
11. A system as claimed in claim 1, further comprising an in-field accelerometer and wherein the in-field device transmitter is configured for transmitting acceleration data to the off-field device receiver.
12. A system as claimed in claim 11, wherein the off-field device comprises a motion analyser configured for template-based classifying the acceleration data according to at least one gesture pattern.
13. A system as claimed in claim 12, wherein the AS decoder is configured for further classifying the one or more spoken signals in accordance with the at least one gesture pattern.
14. A system as claimed in claim 11, further comprising a wrist wearable comprising the accelerometer, the wrist wearable being in wireless short range communication with the in-field device.
15. A system as claimed in claim 10, further comprising a handheld device in operable communication with the in-field device and wherein the handheld device comprises a user interface and wherein the handheld device is configured for transmitting user interface data to the in-field device.
16. A system as claimed in claim 15, wherein the handheld device is in wireless communication with the in-field device.
Applications Claiming Priority (2)
Application Number | Priority Date | Filing Date | Title |
---|---|---|---|
AU2017900488 | 2017-02-15 | ||
AU2017900488A AU2017900488A0 (en) | 2017-02-15 | A command recognition automated sporting scoring system |
Publications (1)
Publication Number | Publication Date |
---|---|
WO2018148803A1 true WO2018148803A1 (en) | 2018-08-23 |
Family
ID=63169120
Family Applications (1)
Application Number | Title | Priority Date | Filing Date |
---|---|---|---|
PCT/AU2018/050127 WO2018148803A1 (en) | 2017-02-15 | 2018-02-15 | A speech signal recognition sports match scoring system |
Country Status (1)
Country | Link |
---|---|
WO (1) | WO2018148803A1 (en) |
Citations (5)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
GB2355203A (en) * | 1999-09-11 | 2001-04-18 | Inv S Direct Ltd | Score-keeping apparatus for racket games |
US20030054905A1 (en) * | 2001-09-14 | 2003-03-20 | King Willie A. | Monitoring computer system for court and field ball games |
US7616098B2 (en) * | 2005-07-05 | 2009-11-10 | Sabah Naser Al-Sabah | Electronic personal assistant device for soccer game official |
CN102512814A (en) * | 2011-12-12 | 2012-06-27 | 延安大学 | Intelligent electronic scoreboard based on voice recognition |
US20130120123A1 (en) * | 2010-07-08 | 2013-05-16 | James Andrew Aman | Sports scorekeeping system with integrated scoreboard and automatic entertainment system |
-
2018
- 2018-02-15 WO PCT/AU2018/050127 patent/WO2018148803A1/en active Application Filing
Patent Citations (5)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
GB2355203A (en) * | 1999-09-11 | 2001-04-18 | Inv S Direct Ltd | Score-keeping apparatus for racket games |
US20030054905A1 (en) * | 2001-09-14 | 2003-03-20 | King Willie A. | Monitoring computer system for court and field ball games |
US7616098B2 (en) * | 2005-07-05 | 2009-11-10 | Sabah Naser Al-Sabah | Electronic personal assistant device for soccer game official |
US20130120123A1 (en) * | 2010-07-08 | 2013-05-16 | James Andrew Aman | Sports scorekeeping system with integrated scoreboard and automatic entertainment system |
CN102512814A (en) * | 2011-12-12 | 2012-06-27 | 延安大学 | Intelligent electronic scoreboard based on voice recognition |
Similar Documents
Publication | Publication Date | Title |
---|---|---|
WO2021051579A1 (en) | Body pose recognition method, system, and apparatus, and storage medium | |
US11322044B2 (en) | Information processing device, sensor device, information processing system, and storage medium | |
Rana et al. | Wearable sensors for real-time kinematics analysis in sports: A review | |
US10446059B2 (en) | Hand motion interpretation and communication apparatus | |
US7565295B1 (en) | Method and apparatus for translating hand gestures | |
US20180249908A1 (en) | Multi-state performance monitoring system | |
Büthe et al. | A wearable sensing system for timing analysis in tennis | |
Kos et al. | Tennis stroke detection and classification using miniature wearable IMU device | |
CN103857446B (en) | Equipment and its data processing method for catching tennis match data | |
US10114462B2 (en) | Device and method for entering information in sports applications | |
US20080036737A1 (en) | Arm Skeleton for Capturing Arm Position and Movement | |
KR102388939B1 (en) | Method for providing feedback service for user's motion correction | |
KR102089002B1 (en) | Method and wearable device for providing feedback on action | |
US10558854B2 (en) | Analysis device, recording medium, and analysis method | |
US20160228744A1 (en) | Device and method for the classification and the reclassification of a user activity | |
CN113384861A (en) | Table tennis training device, table tennis training method, and computer-readable storage medium | |
CN109758154B (en) | Motion state determination method, device, equipment and storage medium | |
US10751601B2 (en) | Automatic rally detection and scoring | |
KR102089934B1 (en) | Method for extracting impact time in golf swing | |
Taghavi et al. | Tennis stroke detection using inertial data of a smartwatch | |
US10729960B2 (en) | System and method for improving bowling shot performance | |
JP6912363B2 (en) | Information processing system, information processing method and information processing program | |
WO2018148803A1 (en) | A speech signal recognition sports match scoring system | |
CN114170868A (en) | Intelligent piano training method and system | |
CN111353345A (en) | Method, device and system for providing training feedback, electronic equipment and storage medium |
Legal Events
Date | Code | Title | Description |
---|---|---|---|
121 | Ep: the epo has been informed by wipo that ep was designated in this application |
Ref document number: 18754416 Country of ref document: EP Kind code of ref document: A1 |
|
NENP | Non-entry into the national phase |
Ref country code: DE |
|
122 | Ep: pct application non-entry in european phase |
Ref document number: 18754416 Country of ref document: EP Kind code of ref document: A1 |