+
Skip to content

Conversation

rudrankriyam
Copy link
Contributor

@rudrankriyam rudrankriyam commented Aug 31, 2025

Add Sesame TTS integration to mlx-swift-audio package.

Components

  • SesameTTS: Main API for text-to-speech generation
  • SesameModel: Decoder models for text understanding and audio generation
  • SesameModelWrapper: Generation pipeline with tokenization and voice prompting
  • Mimi Codec: Audio tokenizer/decoder
  • SesameAttention: Custom attention mechanism for TTS

Changes

  • Swift implementation following mlx-audio Python reference
  • iOS app now supports both Kokoro and Sesame TTS providers
  • Fixed compilation errors in ContentView.swift
  • Added conversational voice options (A & B)

Test Plan

  • SesameTest executable validates functionality
  • Swift package builds successfully
  • iOS app supports provider switching

Video: https://x.com/rudrankriyam/status/1963537827544580607

@rudrankriyam rudrankriyam marked this pull request as ready for review September 4, 2025 10:02
@rudrankriyam
Copy link
Contributor Author

PR run is failing due to metal toolchain issue

@lucasnewman
Copy link
Collaborator

Generally looks good! I noticed your xcuserdata/ directories made it in -- it's best to remove those (you can add a .gitignore rule for it if you want).

- Removed problematic mlx-swift-audio package dependency from Xcode project
- Added proper MLX package dependencies (MLX, MLXNN, MLXRandom, MLXLMCommon, MLXLLM, Transformers)
- Updated workspace configuration to use parent directory
- Modified SesameSession.swift to support optional audio playback

Note: Swift Package Manager builds successfully, but Xcode has issues resolving MLX dependencies
- Removed explicit mlx-swift and swift-transformers dependencies from Xcode
- Keep only mlx-swift-examples which brings in both as transitive dependencies
- This matches the Package.swift structure and avoids duplication

Note: Xcode still has MLX dependency resolution issues, but Swift Package Manager works correctly
- Removed duplicate mlx-swift and swift-transformers dependencies
- Keep only mlx-swift-examples which provides both as transitive dependencies
- Updated package version to upToNextMajorVersion with minimumVersion 2.25.7
- Xcode project now builds successfully ✅

Before: mlx-swift + mlx-swift-examples + swift-transformers (redundant)
After: mlx-swift-examples (includes mlx-swift + swift-transformers)
@lucasnewman lucasnewman merged commit 93e042d into Blaizzy:main Sep 15, 2025
1 of 2 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants

点击 这是indexloc提供的php浏览器服务,不要输入任何密码和下载