这是indexloc提供的服务,不要输入任何密码
Skip to content

feat: Implement hair swapping and enhance realism #1298

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
wants to merge 24 commits into
base: main
Choose a base branch
from

Conversation

rehanbgmi
Copy link

@rehanbgmi rehanbgmi commented May 21, 2025

This commit introduces the capability to swap hair along with the face from a source image to a target image/video or live webcam feed.

Key changes include:

  1. Hair Segmentation:

    • Integrated the isjackwild/segformer-b0-finetuned-segments-skin-hair-clothing model from Hugging Face using the transformers library.
    • Added modules/hair_segmenter.py with a segment_hair function to produce a binary hair mask from an image.
    • Updated requirements.txt with transformers.
  2. Combined Face-Hair Mask:

    • Implemented create_face_and_hair_mask in modules/processors/frame/face_swapper.py to generate a unified mask for both face (from landmarks) and segmented hair from the source image.
  3. Enhanced Swapping Logic:

    • Modified swap_face and related processing functions (process_frame, process_frame_v2, process_frames, process_image) to utilize the full source image (source_frame_full).
    • The swap_face function now performs the standard face swap and then:
      • Segments hair from the source_frame_full.
      • Warps the hair and its mask to the target face's position using an affine transformation estimated from facial landmarks. - Applies color correction (apply_color_transfer) to the warped hair. - Blends the hair onto the target frame, preferably using cv2.seamlessClone for improved realism.
    • Existing mouth mask logic is preserved and applied to the final composited frame.
  4. Webcam Integration:

    • Updated the webcam processing loop in modules/ui.py (create_webcam_preview) to correctly load and pass the source_frame_full to the frame processors.
    • This enables hair swapping in live webcam mode.
    • Added error handling for source image loading in webcam mode.

This set of changes addresses your request for more realistic face swaps that include hair. Further testing and refinement of blending parameters may be beneficial for optimal results across all scenarios.

Summary by Sourcery

Implement hair swapping and realism enhancements by integrating a Segformer-based hair segmentation model, updating the face swap pipeline to warp, color-correct, and blend hair in addition to faces, and extending this functionality to images, videos, and live webcam feeds.

New Features:

  • Integrate a hair segmentation model to generate hair masks from source images
  • Enable hair swapping alongside face swapping by warping, color-correcting, and blending hair onto targets
  • Extend live webcam mode to support hair swapping using the full source image

Enhancements:

  • Refactor face swapping functions to accept the full source frame and integrate hair blending logic
  • Implement seamlessClone-based blending with affine warping and color transfer for improved realism
  • Update frame processors and UI to correctly load and pass the full source image for hair processing
  • Add create_face_and_hair_mask utility to combine facial landmark masks with segmented hair

Build:

  • Add transformers dependency to requirements.txt

This commit introduces the capability to swap hair along with the face from a source image to a target image/video or live webcam feed.

Key changes include:

1.  **Hair Segmentation:**
    - Integrated the `isjackwild/segformer-b0-finetuned-segments-skin-hair-clothing` model from Hugging Face using the `transformers` library.
    - Added `modules/hair_segmenter.py` with a `segment_hair` function to produce a binary hair mask from an image.
    - Updated `requirements.txt` with `transformers`.

2.  **Combined Face-Hair Mask:**
    - Implemented `create_face_and_hair_mask` in `modules/processors/frame/face_swapper.py` to generate a unified mask for both face (from landmarks) and segmented hair from the source image.

3.  **Enhanced Swapping Logic:**
    - Modified `swap_face` and related processing functions (`process_frame`, `process_frame_v2`, `process_frames`, `process_image`) to utilize the full source image (`source_frame_full`).
    - The `swap_face` function now performs the standard face swap and then:
        - Segments hair from the `source_frame_full`.
        - Warps the hair and its mask to the target face's position using an affine transformation estimated from facial landmarks.
        - Applies color correction (`apply_color_transfer`) to the warped hair.
        - Blends the hair onto the target frame, preferably using `cv2.seamlessClone` for improved realism.
    - Existing mouth mask logic is preserved and applied to the final composited frame.

4.  **Webcam Integration:**
    - Updated the webcam processing loop in `modules/ui.py` (`create_webcam_preview`) to correctly load and pass the `source_frame_full` to the frame processors.
    - This enables hair swapping in live webcam mode.
    - Added error handling for source image loading in webcam mode.

This set of changes addresses your request for more realistic face swaps that include hair. Further testing and refinement of blending parameters may be beneficial for optimal results across all scenarios.
Copy link
Contributor

sourcery-ai bot commented May 21, 2025

Reviewer's Guide

This PR augments the existing face‐swap pipeline with hair swapping by integrating a Segformer hair segmentation model, combining face and hair masks, extending the core swap_face routine to warp, color‐correct and blend hair onto the target, and updating frame/Image/stream processing (including webcam UI) to carry through the full source image.

Sequence Diagram for Enhanced swap_face with Hair Processing

sequenceDiagram
    participant FPL as FrameProcessorLogic
    participant SFace as face_swapper.swap_face
    participant HSeg as hair_segmenter
    participant CV2 as OpenCV

    FPL->>+SFace: swap_face(source_face_obj, target_face, source_frame_full, temp_frame)
    SFace->>SFace: Perform standard face swap on temp_frame
    SFace->>+HSeg: segment_hair(source_frame_full)
    HSeg-->>-SFace: hair_only_mask_source
    SFace->>CV2: estimateAffinePartial2D(source_kps, target_kps)
    CV2-->>SFace: matrix
    SFace->>CV2: warpAffine(hair_only_mask_source_binary, matrix, dsize)
    CV2-->>SFace: warped_hair_mask
    SFace->>CV2: warpAffine(source_frame_full, matrix, dsize)
    CV2-->>SFace: warped_source_hair_image
    SFace->>SFace: apply_color_transfer(warped_source_hair_image, swapped_frame)
    SFace->>CV2: seamlessClone(color_corrected_warped_hair, swapped_frame, warped_hair_mask_binary, center)
    CV2-->>SFace: final_frame_with_hair
    SFace->>SFace: Apply mouth mask (if enabled)
    SFace-->>-FPL: final_swapped_frame
Loading

File-Level Changes

Change Details Files
Integrate hair segmentation model
  • Add standalone hair_segmenter module using Hugging Face Segformer
  • Install and import transformers in requirements
  • Provide segment_hair() to output a binary hair mask
modules/hair_segmenter.py
requirements.txt
Implement combined face+hair mask generator
  • Create create_face_and_hair_mask() to merge facial landmark mask with hair mask
  • Ensure mask dimensions align and apply blur for smooth edges
modules/processors/frame/face_swapper.py
Enhance swap_face() to warp & blend hair
  • Extend swap_face signature to accept full source frame
  • Segment, warp and color‐correct hair using affine from landmarks
  • Blend hair via cv2.seamlessClone with fallback to simple masking
  • Adjust mouth‐mask logic to apply on the final composited frame
modules/processors/frame/face_swapper.py
Propagate source_frame_full through processing pipelines
  • Update signatures of process_frame, process_frame_v2, process_frames, process_image to include source_frame_full
  • Modify all swap_face() invocations accordingly
  • Add error handling when reading source or temp frames
modules/processors/frame/face_swapper.py
Enable hair swapping in live webcam mode
  • Revise create_webcam_preview to load source image and detect source_face_obj_for_cam
  • Pass both face object and full source image into frame processors
  • Add error checks for missing source files or faces in UI loop
modules/ui.py

Possibly linked issues


Tips and commands

Interacting with Sourcery

  • Trigger a new review: Comment @sourcery-ai review on the pull request.
  • Continue discussions: Reply directly to Sourcery's review comments.
  • Generate a GitHub issue from a review comment: Ask Sourcery to create an
    issue from a review comment by replying to it. You can also reply to a
    review comment with @sourcery-ai issue to create an issue from it.
  • Generate a pull request title: Write @sourcery-ai anywhere in the pull
    request title to generate a title at any time. You can also comment
    @sourcery-ai title on the pull request to (re-)generate the title at any time.
  • Generate a pull request summary: Write @sourcery-ai summary anywhere in
    the pull request body to generate a PR summary at any time exactly where you
    want it. You can also comment @sourcery-ai summary on the pull request to
    (re-)generate the summary at any time.
  • Generate reviewer's guide: Comment @sourcery-ai guide on the pull
    request to (re-)generate the reviewer's guide at any time.
  • Resolve all Sourcery comments: Comment @sourcery-ai resolve on the
    pull request to resolve all Sourcery comments. Useful if you've already
    addressed all the comments and don't want to see them anymore.
  • Dismiss all Sourcery reviews: Comment @sourcery-ai dismiss on the pull
    request to dismiss all existing Sourcery reviews. Especially useful if you
    want to start fresh with a new review - don't forget to comment
    @sourcery-ai review to trigger a new review!

Customizing Your Experience

Access your dashboard to:

  • Enable or disable review features such as the Sourcery-generated pull request
    summary, the reviewer's guide, and others.
  • Change the review language.
  • Add, remove or edit custom review instructions.
  • Adjust other review settings.

Getting Help

Copy link
Author

@rehanbgmi rehanbgmi left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

i checked on my sysytem its wroking on nvidia

Copy link
Contributor

@sourcery-ai sourcery-ai bot left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Hey @rehanbgmi - I've reviewed your changes - here's some feedback:

  • Rather than loading the Segformer model on every call to segment_hair, initialize and cache the processor and model once (e.g. at module load) to avoid excessive latency and memory use.
  • The new create_face_and_hair_mask duplicates much of the existing face‐mask logic—consider refactoring to reuse create_face_mask and then simply merge in the hair mask to reduce code duplication.
  • Propagating source_frame_full through every processor method has significantly bloated the function signatures; consider encapsulating shared data in a context object or simplifying the API to keep calls more concise.
Here's what I looked at during the review
  • 🟡 General issues: 7 issues found
  • 🟢 Security: all looks good
  • 🟢 Testing: all looks good
  • 🟢 Documentation: all looks good

Sourcery is free for open source - if you like our reviews please consider sharing them ✨
Help me be more useful! Please click 👍 or 👎 on each comment and I'll use the feedback to improve your reviews.

from transformers import SegformerImageProcessor, SegformerForSemanticSegmentation
import cv2 # Imported for BGR to RGB conversion, though PIL can also do it.

def segment_hair(image_np: np.ndarray) -> np.ndarray:
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

issue (performance): Model reloaded on every call

Instantiating the processor and model inside segment_hair causes them to reload for every frame, resulting in significant performance overhead. Load them once at import or cache them globally to improve inference speed.

final_swapped_frame = swapped_frame.copy() # Initialize final_swapped_frame

# START of Hair Blending Logic
if source_face_obj.kps is not None and target_face.kps is not None and source_face_obj.kps.shape[0] >=2 and target_face.kps.shape[0] >=2 : # kps are 5x2 landmarks
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

issue (bug_risk): Keypoint count check is too permissive

Downstream affine estimation requires at least 3 non-collinear keypoints. Update the check to shape[0] >= 3 or add a collinearity check to prevent failures or degenerate transforms.

except cv2.error as e:
logging.warning(f"cv2.seamlessClone failed: {e}. Falling back to simple blending.")
# Fallback: Simple Blending (if seamlessClone fails)
warped_hair_mask_3ch = cv2.cvtColor(warped_hair_mask_binary, cv2.COLOR_GRAY2BGR) > 0 # boolean mask
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

issue (bug_risk): Boolean mask indexing may flatten the result

Indexing with a 3-channel boolean mask in NumPy can flatten the output or select data incorrectly. Use a single-channel mask (e.g., mask[:,:,0]) or expand dimensions (mask[:,:,None]) to ensure correct assignment.

)

final_swapped_frame = swapped_frame.copy() # Initialize final_swapped_frame
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

suggestion (performance): Clone at top-level may be unnecessary

Delay copying swapped_frame until just before the first in-place modification to avoid unnecessary frame buffer allocation when blending is skipped.

@@ -620,3 +745,113 @@ def apply_color_transfer(source, target):
source = (source - source_mean) * (target_std / source_std) + target_mean

return cv2.cvtColor(np.clip(source, 0, 255).astype("uint8"), cv2.COLOR_LAB2BGR)


def create_face_and_hair_mask(source_face: Face, source_frame: Frame) -> np.ndarray:
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

nitpick: Unused helper function

Consider removing create_face_and_hair_mask if it's not currently used, or integrate it into the logic if needed.

modules/ui.py Outdated
Comment on lines 941 to 944
else:
temp_frame = frame_processor.process_frame(source_image, temp_frame)
if source_face_obj_for_cam and source_frame_full_for_cam is not None:
temp_frame = frame_processor.process_frame(source_face_obj_for_cam, source_frame_full_for_cam, temp_frame)
# else: temp_frame remains unchanged if source isn't ready
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

suggestion: No fallback path if source loading fails

Consider disabling the swapper UI or displaying a persistent error when source data is missing, rather than silently skipping processing.

Suggested change
else:
temp_frame = frame_processor.process_frame(source_image, temp_frame)
if source_face_obj_for_cam and source_frame_full_for_cam is not None:
temp_frame = frame_processor.process_frame(source_face_obj_for_cam, source_frame_full_for_cam, temp_frame)
# else: temp_frame remains unchanged if source isn't ready
else:
if source_face_obj_for_cam and source_frame_full_for_cam is not None:
temp_frame = frame_processor.process_frame(source_face_obj_for_cam, source_frame_full_for_cam, temp_frame)
else:
update_status("Error: Missing source face or frame data; disabling face swapper UI")
modules.globals.fp_ui["face_swapper"] = False

Comment on lines 38 to 41
# Label 2 is for hair in this model
hair_mask = np.where(segmentation_map == 2, 255, 0).astype(np.uint8)

return hair_mask
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

suggestion (code-quality): Inline variable that is immediately returned (inline-immediately-returned-variable)

Suggested change
# Label 2 is for hair in this model
hair_mask = np.where(segmentation_map == 2, 255, 0).astype(np.uint8)
return hair_mask
return np.where(segmentation_map == 2, 255, 0).astype(np.uint8)

@@ -67,14 +68,93 @@ def get_face_swapper() -> Any:
return FACE_SWAPPER


def swap_face(source_face: Face, target_face: Face, temp_frame: Frame) -> Frame:
def swap_face(source_face_obj: Face, target_face: Face, source_frame_full: Frame, temp_frame: Frame) -> Frame:
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

issue (code-quality): We've found these issues:


Explanation

The quality score for this function is below the quality threshold of 25%.
This score is a combination of the method length, cognitive complexity and working memory.

How can you solve this?

It might be worth refactoring this function to make it shorter and more readable.

  • Reduce the function length by extracting pieces of functionality out into
    their own functions. This is the most important thing you can do - ideally a
    function should be less than 10 lines.
  • Reduce nesting, perhaps by introducing guard clauses to return early.
  • Ensure that variables are tightly scoped, so that code using related concepts
    sits together within the function rather than being scattered.


def process_frame_v2(temp_frame: Frame, temp_frame_path: str = "") -> Frame:
# process_frame_v2 needs to accept source_frame_full as well
def process_frame_v2(source_frame_full: Frame, temp_frame: Frame, temp_frame_path: str = "") -> Frame:
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

issue (code-quality): We've found these issues:


Explanation

The quality score for this function is below the quality threshold of 25%.
This score is a combination of the method length, cognitive complexity and working memory.

How can you solve this?

It might be worth refactoring this function to make it shorter and more readable.

  • Reduce the function length by extracting pieces of functionality out into
    their own functions. This is the most important thing you can do - ideally a
    function should be less than 10 lines.
  • Reduce nesting, perhaps by introducing guard clauses to return early.
  • Ensure that variables are tightly scoped, so that code using related concepts
    sits together within the function rather than being scattered.

if not source_face_obj:
logging.error(f"No face detected in source image {source_path}")
return
result = process_frame(source_face_obj, source_img, target_frame)
cv2.imwrite(output_path, result)
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

issue (code-quality): Hoist repeated code outside conditional statement (hoist-statement-from-if)

@hacksider
Copy link
Owner

Looks great! Awesome work but is very slow on live. Can you create a toggle to enable/disable the effects?

This commit introduces shell scripts to automate the setup process and provide convenient ways to run the application on macOS.

New files added:
- setup_mac.sh: Checks for Python 3.9+ and ffmpeg, creates a virtual environment, installs pip dependencies from requirements.txt.
- run_mac.sh: Runs the application with the CPU execution provider by default.
- run_mac_cpu.sh: Explicitly runs with the CPU execution provider.
- run_mac_coreml.sh: Runs with the CoreML execution provider.
- run_mac_mps.sh: Runs with the MPS execution provider.

The README.md has also been updated with a new section detailing how to use these scripts for macOS users.

These scripts aim to simplify the initial setup and execution of the project on macOS, similar to the .bat files available for Windows.
@rehanbgmi
Copy link
Author

rehanbgmi commented May 25, 2025

yes i made some changes, please let me know

Removes a duplicated 'if not modules.globals.map_faces:' line within the create_webcam_preview function. This error was causing a syntax issue and preventing the application from running correctly, particularly in webcam mode.
Replaces Python 3.10+ type hint syntax (e.g., Frame | None)
with Python 3.9 compatible syntax (e.g., Optional[Frame])
in modules/processors/frame/face_swapper.py.

This resolves a TypeError encountered when running on Python 3.9.
Specifically, the return type of _prepare_warped_source_material_and_mask
was updated.
…access.

I modified `modules/video_capture.py` so that it will explicitly try using `cv2.CAP_AVFOUNDATION` when initializing `cv2.VideoCapture` on macOS. If AVFoundation fails to open the camera, it will then fall back to the default OpenCV backend.

This adjustment should improve camera compatibility and stability on macOS, especially in situations where the default backend might not be working as expected.
This commit introduces automation scripts for Windows users and updates the README.md accordingly.

New/Modified Windows Scripts:
- setup_windows.bat: New script to automate Python checks, ffmpeg warning, virtual environment (.venv) creation, pip upgrade, and dependency installation.
- run_windows.bat: New script to run the application with CPU execution provider by default, activating .venv.
- run-cuda.bat: Updated to use .venv and pass arguments.
- run-directml.bat: Updated to use .venv and pass arguments.

README.md Changes:
- Updated the "For Windows:" section under "Installation (Manual)" to detail the new automated setup using setup_windows.bat and the revised run scripts.
- Recommended Python 3.10 for Windows for best compatibility.
- Provided updated manual setup notes for Windows, including a PowerShell command for ffmpeg installation and using .venv for consistency.
- Ensured the general Python recommendation in the manual setup prerequisites also mentions Python 3.10.
… done so far and provide feedback for Jules to continue.
This commit incorporates fixes based on a detailed code review, addressing several critical and code quality issues to improve application stability and maintainability.

Key changes include:

modules/hair_segmenter.py:
- Removed a redundant model initialization check to streamline logic.

modules/processors/frame/face_swapper.py:
- Added try-except error handling around calls to segment_hair(), create_face_mask(), and apply_color_transfer() to prevent crashes during the hair swapping process and allow for graceful fallbacks.
- Ensured safer access to the global 'enable_hair_swapping' flag by using getattr(), providing a default value to prevent AttributeErrors.
- Verified that a previously flagged redundant image read in process_image() was already addressed.

modules/ui.py:
- Corrected the function call signature for the Face Enhancer's process_frame_v2 method within the webcam preview when map_faces mode is active.
- Made the conditional check for source image objects (source_face_obj_for_cam and source_frame_full_for_cam) in the webcam preview more explicit.
- Reviewed and confirmed that other reported code quality issues (like a redundant conditional for the face enhancer toggle and webcam error handling consistency) were either not present in the current codebase or already adequately handled by previous modifications.

These changes aim to make the application more robust before tackling further functional improvements and performance optimizations for the hair swapping feature.
Addresses an IndentationError at the definition of the
_blend_material_onto_frame helper function in
modules/processors/frame/face_swapper.py.

The fix ensures that the function definition line starts at
column 0 (no leading whitespace) and that the preceding
function's structure does not cause misinterpretation by the
Python parser. Duplicated/malformed definitions of related
helper functions were also confirmed to be removed in prior steps.

This resolves a syntax error that prevented your application from
starting.
Resolves a SyntaxError ('(' was never closed) and potential
IndentationError related to the definition of helper functions
_prepare_warped_source_material_and_mask and _blend_material_onto_frame
in modules/processors/frame/face_swapper.py.

The fix ensures:
- The definition of _blend_material_onto_frame correctly starts at
  column 0 (no leading whitespace).
- Duplicated and malformed blocks defining these functions, which were
  causing parsing errors, have been removed.
- The file now contains only single, correct definitions for these
  helper functions.

This should allow your application to start and run without these
specific syntax/indentation issues.
refactor: Default "Swap Hair" toggle to OFF

I've changed the default initial state of the "Enable Hair Swapping" feature to OFF.
- I updated `modules/globals.py` so that `enable_hair_swapping = False`.
- I also updated `modules/ui.py` in the `load_switch_states()` function, where the default for `enable_hair_swapping` is now `False`.

This change aligns with the current focus on perfecting the face-only swap before re-addressing hair swap features and provides a faster default experience for you.
…per.py

Resolves a SyntaxError ('(' was never closed) and associated
IndentationErrors in modules/processors/frame/face_swapper.py.
These errors were caused by malformed and duplicated definitions of
the helper functions _prepare_warped_source_material_and_mask and
_blend_material_onto_frame.

The fix involved:
- Removing the entire erroneous duplicated/malformed function blocks.
- Ensuring that the single, correct definitions for these helper
  functions are properly indented at the top level of the module.

This critical fix addresses a major blocker that prevented the
application from starting and parsing the face_swapper.py module.
Modifies modules/face_analyser.py to set det_thresh=0.4
(down from the default of 0.5) when preparing the
insightface.app.FaceAnalysis model.

This change aims to make face detection more sensitive,
particularly for webcam feeds under varied conditions, to reduce
instances of "Face detection failed for target or source" errors.
@hacksider
Copy link
Owner

Will be reviewing this over the weekend! Thanks!

Deletes an erroneous `[end of modules/processors/frame/face_swapper.py]`
marker line from the end of the face_swapper.py file. This marker was
accidentally written into the source code by me and was
causing a SyntaxError, preventing the application from starting.

This commit ensures the file is syntactically correct.
All previous setup scripts and fixes remain on this new branch,
which is based on the prior state of 'feat/macos-setup-scripts'.
Modifies modules/video_capture.py to exclusively attempt using
cv2.CAP_AVFOUNDATION for camera initialization on macOS.
The fallback to the default backend for macOS within this specific
initialization block has been removed.

If AVFoundation fails to open the camera, an error is logged, and
the subsequent standard check in the function will raise a
RuntimeError, making it clearer that AVFoundation was the point of
failure.

This change aims to provide better diagnostics for macOS camera
issues and ensure the intended AVFoundation backend is prioritized
without immediate fallback to potentially problematic default backends
like OBSENSOR.
Optimizes webcam performance by running full face detection and
frame processing (face swap, enhancers) only every N frames
(currently N=3) in modules/ui.py (create_webcam_preview function).

For intermediate frames, the raw (but mirrored/resized) webcam
feed is displayed. This aims to improve UI responsiveness and reduce
overall CPU/GPU load during live webcam sessions, particularly when
resource-intensive operations like hair swapping or face enhancement
are active.

The actual swap/effect will appear at a reduced frame rate (FPS/N),
but the UI should remain smoother.
Reverts the Nth frame processing logic previously introduced in
modules/ui.py (create_webcam_preview function). Webcam frames
will now be processed by the full pipeline on every frame,
instead of skipping frames.

This change is based on your feedback requesting to focus on
optimizing the per-frame performance rather than using frame
skipping techniques at this stage.
Removes the leftover `detection_frame_counter += 1` line from the
`create_webcam_preview` function in modules/ui.py. This line was
erroneously kept after the rest of the Nth frame processing logic
was reverted, causing an UnboundLocalError as the counter was no
longer initialized.

This fix ensures the webcam preview can start correctly without this error.
I've completely removed the Nth frame processing logic (frame counter,
interval, and conditional execution) from the `create_webcam_preview`
function in `modules/ui.py`. The frame processing block has been
unindented to ensure it runs on every frame.

This resolves an `UnboundLocalError` for 'detection_frame_counter'
that occurred because the variable was being used after its
initialization was removed in a previous attempt to revert this logic.
The webcam preview will now process every frame as it did before the
Nth frame optimization was introduced.
Optimizes webcam performance for face swapping by introducing
Nth-frame full face detection and using a KCF tracker for
intermediate frames in modules/processors/frame/face_swapper.py.

Key changes:
- Full face analysis (get_one_face) now runs every N frames (default 3)
  or when tracking is lost in the process_frame function (for single
  face mode).
- For intermediate frames, a KCF tracker updates the target face bounding
  box, and keypoints are estimated by translating the last known good
  keypoints.
- The actual face swap (inswapper model) still runs on every frame if a
  face (either detected or tracked) is available.
- Experimental tracking logic added to _process_live_target_v2 for
  map_faces=True in live mode (non-many_faces path).
- Added robustness:
    - None checks for landmarks in mouth_mask and create_face_mask
      functions, with fallbacks for create_face_mask.
    - Division-by-zero check in apply_color_transfer.
- Reset tracker state in process_video for new video files.

This aims to significantly improve FPS by reducing the frequency of
costly full face analysis, while still providing a continuous swap.
Mouth masking will be less effective on tracked intermediate frames
due to the absence of full landmark data.
Introduces Nth-frame full face detection combined with KCF bounding
box tracking and Lucas-Kanade (LK) optical flow for keypoint (KPS)
tracking on intermediate frames. This is primarily for single-face
webcam mode to improve performance while maintaining per-frame swaps.

Key Changes:
- Modified `face_swapper.py` (`process_frame`):
    - Full `insightface.FaceAnalysis` runs every N frames (default 5)
      or if tracking is lost.
    - KCF tracker updates bounding box on intermediate frames.
    - Optical flow (`cv2.calcOpticalFlowPyrLK`) tracks the 5 keypoints
      from the previous frame to the current intermediate frame.
    - A `Face` object is constructed with tracked bbox and KPS for
      swapping on intermediate frames (detailed landmarks like
      `landmark_2d_106` are None for these).
    - Experimental similar logic added to `_process_live_target_v2`
      for `map_faces=True` live mode (non-many_faces path).
- Robustness:
    - Mouth masking and face mask creation functions in `face_swapper.py`
      now handle cases where `landmark_2d_106` is `None` (e.g., by
      skipping mouth mask or using bbox for face mask).
    - Added division-by-zero check in `apply_color_transfer`.
- State Management:
    - Introduced `reset_tracker_state()` in `face_swapper.py` to clear
      all tracking-related global variables.
    - `ui.py` now calls `reset_tracker_state()` at appropriate points
      (webcam start, mode changes, new source image selection) to ensure
      clean tracking for new sessions.
- `DETECTION_INTERVAL` in `face_swapper.py` increased to 5.

This aims to provide you with a smoother face swap experience with better FPS
by reducing the frequency of expensive full face analysis, while the
actual swap operation continues on every frame using tracked data.
I resolved an IndentationError in the create_lower_mouth_mask function
in modules/processors/frame/face_swapper.py by correcting the
indentation of the lower_lip_order list definition and the subsequent
try-except block.

Additionally, I updated the function's return type hint to use
typing.Tuple and typing.Optional for Python 3.9+ compatibility.

This fixes a crash that prevented your application from running.
Refactors the usage of the update_status function to break a
circular import dependency.

- In modules/processors/frame/face_swapper.py:
  - Removed direct import of update_status from modules.core.
  - Modified pre_start(), process_image(), and process_video()
    to accept update_status as a Callable parameter
    (status_fn_callback).
  - Internal calls now use this passed callback.
- In modules/core.py:
  - Updated the calls to pre_start(), process_image(), and
    process_video() for frame processors (specifically face_swapper)
    to pass the core.update_status function as the
    status_fn_callback argument.

This change ensures that face_swapper.py no longer needs to import
modules.core directly for status updates, resolving the ImportError.
@hacksider
Copy link
Owner

As per testing, I have encountered this

Traceback (most recent call last):
File "E:\Deep-Live-Cam\Deep-Live-Cam\run.py", line 6, in
core.run()
File "E:\Deep-Live-Cam\Deep-Live-Cam\modules\core.py", line 257, in run
if hasattr(frame_processor, 'pre_start') and not frame_processor.pre_start(status_fn_callback=update_status): # Pass callback here
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "E:\Deep-Live-Cam\Deep-Live-Cam\modules\processors\frame\face_swapper.py", line 68, in pre_start
status_fn_callback("Select an image for source path.", NAME)
File "E:\Deep-Live-Cam\Deep-Live-Cam\modules\core.py", line 176, in update_status
ui.update_status(message)
File "E:\Deep-Live-Cam\Deep-Live-Cam\modules\ui.py", line 573, in update_status
status_label.configure(text=_(text))
^^^^^^^^^^^^^^^^^^^^^^
AttributeError: 'NoneType' object has no attribute 'configure'

Replaces the create_lower_mouth_mask function in
modules/processors/frame/face_swapper.py with a version that has
corrected indentation. This resolves an "IndentationError: unexpected indent"
that was preventing the application from starting.

The replaced block also includes minor robustness improvements for
ROI calculations and Gaussian blur kernel sizes within this function
and implicitly updated other related utility functions that were part of the
provided code block.
@hacksider
Copy link
Owner

Same error. I also run your fork on my computer to ensure if i'm missing something and it have similar error. Will also investigate my environment on this. But the push 5 days ago works great (i'll just add toggle on that).

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants