Abstract
Motivation
Method
Evaluation
Visual Results
Others

SVG ImageAuxOL

On-the-Fly Improving Segment Anything
for Medical Image Segmentation using Auxiliary Online Learning

Tianyu Huang1, Tao Zhou1, Weidi Xie4, Shuo Wang3, Qi Dou2, Yizhe Zhang1

1Nanjing University of Science and Technology 2Chinese University of Hong Kong 3Fudan University 4Shanghai Jiaotong University
ArXiv IEEE Xplore Github Code

Abstract

The current variants of the Segment Anything Model (SAM), which include the original SAM and Medical SAM, still lack the capability to produce sufficiently accu rate segmentation for medical images. In medical imag- ing contexts, it is not uncommon for human experts to rectify segmentations of specific test samples after SAM generates its segmentation predictions. These rectifica- tions typically entail manual or semi-manual corrections employing state-of-the-art annotation tools. Motivated by this process, we introduce a novel approach that leverages the advantages of online machine learning to enhance Segment Anything (SA) during test time. We employ rectified annotations to perform online learning, with the aim of improving the segmentation quality of SA on medical images. To ensure the effectiveness and efficiency of online learning when integrated with large-scale vision models like SAM, we propose a new method called Auxiliary Online Learning (AuxOL), which entails adaptive online-batch and adaptive segmentation fusion. Experiments conducted on eight datasets covering four medical imaging modalities validate the effectiveness of the proposed method. Our work proposes and validates a new, practical, and effective approach for enhancing SA on downstream segmentation tasks (e.g., medical image segmentation).

Motivation

training a medical SAM Training a medical SAM requires acquiring a large volume of labeled medical images. There exist practical challenges in building SAM for medical image data, including variations in modalities (e.g., CT, MRI, Ultrasound), subjects (e.g., organs, tissues, cells), scales, annotation quality, and tasks.

fine-tuning and/or adapting SAM Fine-tuning and/or adapting a general purpose SAM to a particular medical image segmentation task, although a reasonable approach, can lead to the possibility of SAM losing its generalization capability (to certain degree). For each distinct medical image segmentation task, one may need to apply full-session offline fine-tuning and/or adaptation. Although possible to do, it incurs significant computation and time costs, and being rigid as each task requires a new session of offline training.

AuxOL(Ours) Utilizes a much smaller (compared to SAM) auxiliary model during inference in conjunction with SAM. The auxiliary model adjusts SAM’s output while conducting online weight updates. We call this online learning method AuxOL (Auxiliary Online Learning).


Methodology

An overview of the main steps of our AuxOL with SAM: Improving Segment Anything (SA) for medical images via auxiliary learning in an online learning pipeline.

Evaluation

Online learning of AuxOL improves the performances of SAM and Medical SAM in polyp segmentation of endoscopic images (five datasets), breast cancer segmentation of ultrasound images, gland segmentation of histology images, and fluid region segmentation of OCT scans

Visual Results

Sensitivity to Imperfect Human Annotations and Prompts

Improvement of DSC on each sample

Citation

          @ARTICLE{10916782,
            author={Huang, Tianyu and Zhou, Tao and Xie, Weidi and Wang, Shuo and Dou, Qi and Zhang, Yizhe},
            journal={IEEE Transactions on Medical Imaging},
            title={On-the-Fly Improving Segment Anything for Medical Image Segmentation using Auxiliary Online Learning},
            year={2025},
            volume={},
            number={},
            pages={1-1},
            keywords={Image segmentation;Biomedical imaging;Adaptation models;Training;Computational modeling;Machine learning;Annotations;Three-dimensional displays;Foundation models;Data mining;Segment Anything Model;Online Machine Learning;Medical Image Segmentation;Auxiliary Online Learning;Rectified Annotations},
            doi={10.1109/TMI.2025.3548985}}