feat(vertexai): Add flutter_soloud for sound output in Live API audio streaming example. #17305
Conversation
|
Hi there, thanks for the PR! I understand that this is a draft but can you please give me some more detail about this? I recently added a sound example to vertexai https://github.com/firebase/flutterfire/blob/main/packages/firebase_vertexai/firebase_vertexai/example/lib/pages/audio_page.dart unless yours is different? |
|
Hey @MichaelVerdon! From my understanding, the audio page that you linked to is for recording a it of audio and sending it to the Gemini API as opposed to streaming live audio, right? I'm working with @cynthiajoan on refining the Live API example page, so in this case it's bidirectional audio streaming here => https://github.com/firebase/flutterfire/blob/main/packages/firebase_vertexai/firebase_vertexai/example/lib/pages/bidi_page.dart. Docs: https://firebase.google.com/docs/vertex-ai/live-api?platform=flutter Edit: I realize my PR title was unclear, so I added more detail! |
|
@khanhnwin - is this still being worked on? |
|
@russellwheatley Yes! @cynthiajoan is going to clean it up and integrate it into the example app is my understanding. |
Yeah I'm working on this and not planning to have this PR included in the I/O release. |
|
The current implementation of Live AI API stream gets the AI to enter an endless loop of self replying, the audio of the speaker feeds back into the user audio stream due to the lack of echo cancelling. Tested against both flutter_soloud and just_audio where both packages does not have AEC implementation so far. What is the recommended flutter audio package for this use case as currently this approach is not usable for production |
In this PR we updated the audio_input.dart to add some attempts for AEC in the recorder config. Would you like to try to see if that improves the AI self hearing in your target platform? |
packages/firebase_vertexai/firebase_vertexai/example/lib/pages/bidi_page.dart
Outdated
Show resolved
Hide resolved
packages/firebase_ai/firebase_ai/example/lib/pages/bidi_page.dart
Outdated
Show resolved
Hide resolved
packages/firebase_ai/firebase_ai/example/lib/pages/bidi_page.dart
Outdated
Show resolved
Hide resolved
packages/firebase_ai/firebase_ai/example/lib/pages/bidi_page.dart
Outdated
Show resolved
Hide resolved
packages/firebase_ai/firebase_ai/example/lib/pages/function_calling_page.dart
Outdated
Show resolved
Hide resolved
packages/firebase_ai/firebase_ai/example/lib/utils/audio_input.dart
Outdated
Show resolved
Hide resolved
packages/firebase_ai/firebase_ai/example/lib/utils/audio_output.dart
Outdated
Show resolved
Hide resolved
packages/firebase_ai/firebase_ai/example/lib/utils/audio_output.dart
Outdated
Show resolved
Hide resolved
Co-authored-by: Nate Bosch <nbosch1@gmail.com>
Description
Replace this paragraph with a description of what this PR is doing. If you're modifying existing behavior, describe the existing behavior, how this PR is changing it, and what motivated the change.
Related Issues
Replace this paragraph with a list of issues related to this PR from the issue database. Indicate, which of these issues are resolved or fixed by this PR. Note that you'll have to prefix the issue numbers with flutter/flutter#.
Checklist
Before you create this PR confirm that it meets all requirements listed below by checking the relevant checkboxes (
[x]).This will ensure a smooth and quick review process. Updating the
pubspec.yamland changelogs is not required.///).melos run analyze) does not report any problems on my PR.Breaking Change
Does your PR require plugin users to manually update their apps to accommodate your change?