This is the official codebase for our paper:

NFCMTL: Auto NailFold Capillaroscopy through a Multi-Task Learning Model

• Unified framework for NFC capillary segmentation, classification, and keypoint detection
• Uses Multiscale Vision Transformer (MViT) + FPN + Mask-RCNN
• Applies uncertainty-weighted multi-task loss to NFC tasks
• Outperforms prior SOTA in all major tasks

We address key challenges in nailfold capillaroscopy (NFC) by designing a joint learning model. Compared to traditional and fragmented pipelines, NFCMTL unifies them into a streamlined inference graph as shwon below.

• Backbone: Multiscale image visual encoding using MViTv2-T pretrained on MViTv2-T.
  • Input: 3-channel RGB image
  • Output: Multiscale feature maps at 1/4, 1/8, and 1/16 resolutions
• Neck: Feature Pyramid Network (FPN)
• Heads: Mask-RCNN-based architecture with 3 output heads:
  • Classification: Type of each capillary loop
  • Segmentation: Pixel-wise instance masks
  • Keypoints: 8 landmark predictions for vessel apex and limbs

We adopt Kendall et al.'s uncertainty-based formulation to dynamically balance task contributions:

Where:

• $\mathcal{L}_i$ is the task-specific loss
• $\sigma_i$ is the learned task uncertainty

This adaptive scheme improves training stability and robustness, especially under annotation noise or scale imbalance.

We use the public ANFC dataset ([Zhao et al., ISBI 2024]) with:

• 321 NFC images, high-res RGB
• Capillary instance masks, class labels, and 9 anatomical keypoints

Keypoints include apex (U, D), arterial limb (LL, LR), venous limb (RL, RR), bases (LB, RB), and optional conjunction point (X).

git clone <https this repo>
cd NFCMTL
conda create -n nfcmtl python=3.11
conda activate nfcmtl
pip install -r requirements.txt

Install Detectron2 based on your CUDA version.

1. Request dataset according to ANFC dataset.
2. Put COCO-formatted annotations in the desired directory as follows:

├── train
│   ├── *.jpg
│   ├── anfc_coco_train.json
├── val
│   ├── *.jpg
│   ├── anfc_coco_val.json

python main.py --data-path /path/to/data/ANFC-coco

To evaluate specific tasks, uncomment and run one of the following inside main.py:

nfcmtl_test(cfg, mode="segmentation")
nfcmtl_test(cfg, mode="classification")
nfcmtl_test(cfg, mode="keypoints")

Or visualize predictions:

visualize(cfg, data_path, output_dir)

If you find this work helpful, please cite:

bibtex coming soon.

Questions and issues are welcome via GitHub!