micro_sam

Segment Anything for Microscopy

Segment Anything for Microscopy implements automatic and interactive annotation for microscopy data. It is built on top of Segment Anything by Meta AI and specializes it for microscopy and other biomedical imaging data. Its core components are:

  • The micro_sam tools for interactive data annotation, built as napari plugin.
  • The micro_sam library to apply Segment Anything to 2d and 3d data or fine-tune it on your data.
  • The micro_sam models that are fine-tuned on publicly available microscopy data and that are available on BioImage.IO.

Based on these components micro_sam enables fast interactive and automatic annotation for microscopy data, like interactive cell segmentation from bounding boxes:

box-prompts

micro_sam is now available as stable version 1.0 and we will not change its user interface significantly in the foreseeable future. We are still working on improving and extending its functionality. The current roadmap includes:

  • Releasing more and better finetuned models for the biomedical imaging domain.
  • Integrating parameter efficient training and compressed models for efficient fine-tuning and faster inference.
  • Improving the 3D segmentation and tracking functionality.

If you run into any problems or have questions please open an issue or reach out via image.sc using the tag micro-sam.

Quickstart

You can install micro_sam via mamba:

$ mamba install -c conda-forge micro_sam

We also provide installers for Windows and Linux. For more details on the available installation options, check out the installation section.

After installing micro_sam, you can start napari from within your environment using

$ napari

After starting napari, you can select the annotation tool you want to use from Plugins -> SegmentAnything for Microscopy. Check out the quickstart tutorial video for a short introduction and the annotation tool section for details.

The micro_sam python library can be imported via

import micro_sam

It is explained in more detail here.

We provide different finetuned models for microscopy that can be used within our tools or any other tool that supports Segment Anything. See finetuned models for details on the available models. You can also train models on your own data, see here for details.

Citation

If you are using micro_sam in your research please cite

Installation

There are three ways to install micro_sam:

  • From mamba is the recommended way if you want to use all functionality.
  • From source for setting up a development environment to use the latest version and to change and contribute to our software.
  • From installer to install it without having to use mamba (supported platforms: Windows and Linux, supports only CPU).

You can find more information on the installation and how to troubleshoot it in the FAQ section.

We do not recommend installing micro-sam with pip.

From mamba

mamba is a drop-in replacement for conda, but much faster. The steps below may also work with conda, but we recommend using mamba, especially if the installation does not work with conda. You can follow the instructions here to install mamba.

IMPORTANT: Make sure to avoid installing anything in the base environment.

micro_sam can be installed in an existing environment via:

$ mamba install -c pytorch -c conda-forge micro_sam

or you can create a new environment (here called micro-sam) via:

$ mamba create -c pytorch -c conda-forge -n micro-sam micro_sam

if you want to use the GPU you need to install PyTorch from the pytorch channel instead of conda-forge. For example:

$ mamba create -c pytorch -c nvidia -c conda-forge -n micro-sam micro_sam pytorch pytorch-cuda=12.1

NOTE: If you create a new enviroment (eg. here called micro-sam), you must activate the environment using

$ mamba activate micro-sam

You may need to change this command to install the correct CUDA version for your system, see https://pytorch.org/ for details.

From source

To install micro_sam from source, we recommend to first set up an environment with the necessary requirements:

To create one of these environments and install micro_sam into it follow these steps

  1. Clone the repository:
$ git clone https://github.com/computational-cell-analytics/micro-sam
  1. Enter it:
$ cd micro-sam
  1. Create the GPU or CPU environment:
$ mamba env create -f <ENV_FILE>.yaml
  1. Activate the environment:
$ mamba activate sam
  1. Install micro_sam:
$ pip install -e .

From installer

We also provide installers for Linux and Windows:

The installers will not enable you to use a GPU, so if you have one then please consider installing micro_sam via mamba instead. They will also not enable using the python library.

Linux Installer:

To use the installer:

  • Unpack the zip file you have downloaded.
  • Make the installer executable: $ chmod +x micro_sam-1.0.0post0-Linux-x86_64.sh
  • Run the installer: ./micro_sam-1.0.0post0-Linux-x86_64.sh
    • You can select where to install micro_sam during the installation. By default it will be installed in $HOME/micro_sam.
    • The installer will unpack all micro_sam files to the installation directory.
  • After the installation you can start the annotator with the command .../micro_sam/bin/napari.
    • Proceed with the steps described in Annotation Tools
    • To make it easier to run the annotation tool you can add .../micro_sam/bin to your PATH or set a softlink to .../micro_sam/bin/napari.

Windows Installer:

  • Unpack the zip file you have downloaded.
  • Run the installer by double clicking on it.
  • Choose installation type: Just Me(recommended) or All Users(requires admin privileges).
  • Choose installation path. By default it will be installed in C:\Users\<Username>\micro_sam for Just Me installation or in C:\ProgramData\micro_sam for All Users.
    • The installer will unpack all micro_sam files to the installation directory.
  • After the installation you can start the annotator by double clicking on .\micro_sam\Scripts\micro_sam.annotator.exe or with the command .\micro_sam\Scripts\napari.exe from the Command Prompt.
  • Proceed with the steps described in Annotation Tools

Easybuild installation

There is also an easy-build recipe for micro_sam under development. You can find more information here.

Annotation Tools

micro_sam provides applications for fast interactive 2d segmentation, 3d segmentation and tracking. See an example for interactive cell segmentation in phase-contrast microscopy (left), interactive segmentation of mitochondria in volume EM (middle) and interactive tracking of cells (right).

The annotation tools can be started from the napari plugin menu, the command line or from python scripts. They are built as napari plugin and make use of existing napari functionality wherever possible. If you are not familiar with napari, we recommend to start here. The micro_sam tools mainly use the point layer, shape layer and label layer.

The annotation tools are explained in detail below. We also provide video tutorials.

The annotation tools can be started from the napari plugin menu:

You can find additional information on the annotation tools in the FAQ section.

HINT: If you would like to start napari to use micro-sam from the plugin menu, you must start it by activating the environment where micro-sam has been installed using:

$ mamba activate <ENVIRONMENT_NAME>
$ napari

Annotator 2D

The 2d annotator can be started by

The user interface of the 2d annotator looks like this:

It contains the following elements:

  1. The napari layers for the segmentations and prompts:
    • prompts: shape layer that is used to provide box prompts to Segment Anything. Prompts can be given as rectangle (marked as box prompt in the image), ellipse or polygon.
    • point_prompts: point layer that is used to provide point prompts to Segment Anything. Positive prompts (green points) for marking the object you want to segment, negative prompts (red points) for marking the outside of the object.
    • committed_objects: label layer with the objects that have already been segmented.
    • auto_segmentation: label layer with the results from automatic instance segmentation.
    • current_object: label layer for the object(s) you're currently segmenting.
  2. The embedding menu. For selecting the image to process, the Segment Anything model that is used and computing its image embeddings. The Embedding Settings contain advanced settings for loading cached embeddings from file or for using tiled embeddings.
  3. The prompt menu for changing whether the currently selected point is a positive or a negative prompt. This can also be done by pressing T.
  4. The menu for interactive segmentation. Clicking Segment Object (or pressing S) will run segmentation for the current prompts. The result is displayed in current_object. Activating batched enables segmentation of multiple objects with point prompts. In this case one object will be segmented per positive prompt.
  5. The menu for automatic segmentation. Clicking Automatic Segmentation will segment all objects n the image. The results will be displayed in the auto_segmentation layer. We support two different methods for automatic segmentation: automatic mask generation (supported for all models) and instance segmentation with an additional decoder (only supported for our models). Changing the parameters under Automatic Segmentation Settings controls the segmentation results, check the tooltips for details.
  6. The menu for commiting the segmentation. When clicking Commit (or pressing C) the result from the selected layer (either current_object or auto_segmentation) will be transferred from the respective layer to committed_objects. When commit_path is given the results will automatically be saved there.
  7. The menu for clearing the current annotations. Clicking Clear Annotations (or pressing Shift + C) will clear the current annotations and the current segmentation.

Point prompts and box prompts can be combined. When you're using point prompts you can only segment one object at a time, unless the batched mode is activated. With box prompts you can segment several objects at once, both in the normal and batched mode.

Check out the video tutorial for an in-depth explanation on how to use this tool.

Annotator 3D

The 3d annotator can be started by

The user interface of the 3d annotator looks like this:

Most elements are the same as in the 2d annotator:

  1. The napari layers that contain the segmentations and prompts.
  2. The embedding menu.
  3. The prompt menu.
  4. The menu for interactive segmentation in the current slice.
  5. The menu for interactive 3d segmentation. Clicking Segment All Slices (or pressing Shift + S) will extend the segmentation of the current object across the volume by projecting prompts across slices. The parameters for prompt projection can be set in Segmentation Settings, please refer to the tooltips for details.
  6. The menu for automatic segmentation. The overall functionality is the same as for the 2d annotator. To segment the full volume Apply to Volume needs to be checked, otherwise only the current slice will be segmented. Note that 3D segmentation can take quite long without a GPU.
  7. The menu for committing the current object.
  8. The menu for clearing the current annotations. If all slices is set all annotations will be cleared, otherwise they are only cleared for the current slice.

You can only segment one object at a time using the interactive segmentation functionality with this tool.

Check out the video tutorial for an in-depth explanation on how to use this tool.

Annotator Tracking

The tracking annotator can be started by

The user interface of the tracking annotator looks like this:

Most elements are the same as in the 2d annotator:

  1. The napari layers that contain the segmentations and prompts. Same as for the 2d segmentation application but without the auto_segmentation layer.
  2. The embedding menu.
  3. The prompt menu.
  4. The menu with tracking settings: track_state is used to indicate that the object you are tracking is dividing in the current frame. track_id is used to select which of the tracks after division you are following.
  5. The menu for interactive segmentation in the current frame.
  6. The menu for interactive tracking. Click Track Object (or press Shift + S) to segment the current object across time.
  7. The menu for committing the current tracking result.
  8. The menu for clearing the current annotations.

The tracking annotator only supports 2d image data with a time dimension, volumetric data + time is not supported. We also do not support automatic tracking yet.

Check out the video tutorial for an in-depth explanation on how to use this tool.

Image Series Annotator

The image series annotation tool enables running the 2d annotator or 3d annotator for multiple images that are saved in a folder. This makes it convenient to annotate many images without having to restart the tool for every image. It can be started by

When starting this tool via the plugin menu the following interface opens:

You can select the folder where your images are saved with Input Folder. The annotation results will be saved in Output Folder. You can specify a rule for loading only a subset of images via pattern, for example *.tif to only load tif images. Set is_volumetric if the data you want to annotate is 3d. The rest of the options are settings for the image embedding computation and are the same as for the embedding menu (see above). Once you click Annotate Images the images from the folder you have specified will be loaded and the annotation tool is started for them.

This menu will not open if you start the image series annotator from the command line or via python. In this case the input folder and other settings are passed as parameters instead.

Check out the video tutorial for an in-depth explanation on how to use the image series annotator.

Finetuning UI

We also provide a graphical inferface for fine-tuning models on your own data. It can be started by clicking Finetuning in the plugin menu after starting napari.

Note: if you know a bit of python programming we recommend to use a script for model finetuning instead. This will give you more options to configure the training. See these instructions for details.

When starting this tool via the plugin menu the following interface opens:

You can select the image data via Path to images. You can either load images from a folder or select a single image file. By providing Image data key you can either provide a pattern for selecting files from the folder or provide an internal filepath for HDF5, Zarr or similar fileformats.

You can select the label data via Path to labels and Label data key, following the same logic as for the image data. The label masks are expected to have the same size as the image data. You can for example use annotations created with one of the micro_sam annotation tools for this, they are stored in the correct format. See the FAQ for more details on the expected label data.

The Configuration option allows you to choose the hardware configuration for training. We try to automatically select the correct setting for your system, but it can also be changed. Details on the configurations can be found here.

Using the Python Library

The python library can be imported via

import micro_sam

This library extends the Segment Anything library and

You can import these sub-modules via

import micro_sam.prompt_based_segmentation
import micro_sam.instance_segmentation
# etc.

This functionality is used to implement the interactive annotation tools in micro_sam.sam_annotator and can be used as a standalone python library. We provide jupyter notebooks that demonstrate how to use it here. You can find the full library documentation by scrolling to the end of this page.

Training your Own Model

We reimplement the training logic described in the Segment Anything publication to enable finetuning on custom data. We use this functionality to provide the finetuned microscopy models and it can also be used to train models on your own data. In fact the best results can be expected when finetuning on your own data, and we found that it does not require much annotated training data to get significant improvements in model performance. So a good strategy is to annotate a few images with one of the provided models using our interactive annotation tools and, if the model is not working as good as required for your use-case, finetune on the annotated data. We recommend checking out our latest preprint for details on the results on how much data is required for finetuning Segment Anything.

The training logic is implemented in micro_sam.training and is based on torch-em. Check out the finetuning notebook to see how to use it. We also support training an additional decoder for automatic instance segmentation. This yields better results than the automatic mask generation of segment anything and is significantly faster. The notebook explains how to train it together with the rest of SAM and how to then use it.

More advanced examples, including quantitative and qualitative evaluation, can be found in the finetuning directory, which contains the code for training and evaluating our models. You can find further information on model training in the FAQ section.

Here is a list of resources, together with their recommended training settings, for which we have tested model finetuning:

Resource Name Capacity Model Type Batch Size Finetuned Parts Number of Objects
CPU 32GB ViT Base 1 all 10
CPU 64GB ViT Base 1 all 15
GPU (NVIDIA GTX 1080Ti) 8GB ViT Base 1 Mask Decoder, Prompt Encoder 10
GPU (NVIDIA Quadro RTX5000) 16GB ViT Base 1 all 10
GPU (Tesla V100) 32GB ViT Base 1 all 10
GPU (NVIDIA A100) 80GB ViT Tiny 2 all 50
GPU (NVIDIA A100) 80GB ViT Base 2 all 40
GPU (NVIDIA A100) 80GB ViT Large 2 all 30
GPU (NVIDIA A100) 80GB ViT Huge 2 all 25

NOTE: If you use the finetuning UI or micro_sam.training.training.train_sam_for_configuration you can specify the hardware configuration and the best settings for it will be set automatically. If your hardware is not in the settings we have tested choose the closest match. You can set the training parameters yourself when using micro_sam.training.training.train_sam. Be aware that the choice for the number of objects per image, the batch size, and the type of model have a strong impact on the VRAM needed for training and the duration of training. See the finetuning notebook for an overview of these parameters.

Finetuned Models

In addition to the original Segment Anything models, we provide models that are finetuned on microscopy data. They are available in the BioImage.IO Model Zoo and are also hosted on Zenodo.

We currently offer the following models:

  • vit_h: Default Segment Anything model with ViT Huge backbone.
  • vit_l: Default Segment Anything model with ViT Large backbone.
  • vit_b: Default Segment Anything model with ViT Base backbone.
  • vit_t: Segment Anything model with ViT Tiny backbone. From the Mobile SAM publication.
  • vit_l_lm: Finetuned Segment Anything model for cells and nuclei in light microscopy data with ViT Large backbone. (Zenodo) (idealistic-rat on BioImage.IO)
  • vit_b_lm: Finetuned Segment Anything model for cells and nuclei in light microscopy data with ViT Base backbone. (Zenodo) (diplomatic-bug on BioImage.IO)
  • vit_t_lm: Finetuned Segment Anything model for cells and nuclei in light microscopy data with ViT Tiny backbone. (Zenodo) (faithful-chicken BioImage.IO)
  • vit_l_em_organelles: Finetuned Segment Anything model for mitochodria and nuclei in electron microscopy data with ViT Large backbone. (Zenodo) (humorous-crab on BioImage.IO)
  • vit_b_em_organelles: Finetuned Segment Anything model for mitochodria and nuclei in electron microscopy data with ViT Base backbone. (Zenodo) (noisy-ox on BioImage.IO)
  • vit_t_em_organelles: Finetuned Segment Anything model for mitochodria and nuclei in electron microscopy data with ViT Tiny backbone. (Zenodo) (greedy-whale on BioImage.IO)

See the two figures below of the improvements through the finetuned model for LM and EM data.

You can select which model to use in the annotation tools by selecting the corresponding name in the Model: drop-down menu in the embedding menu:

To use a specific model in the python library you need to pass the corresponding name as value to the model_type parameter exposed by all relevant functions. See for example the 2d annotator example.

Choosing a Model

As a rule of thumb:

  • Use the vit_l_lm or vit_b_lm model for segmenting cells or nuclei in light microscopy. The larger model (vit_l_lm) yields a bit better segmentation quality, especially for automatic segmentation, but needs more computational resources.
  • Use the vit_l_em_organelles or vit_b_em_organelles models for segmenting mitochondria, nuclei or other roundish organelles in electron microscopy.
  • For other use-cases use one of the default models.
  • The vit_t_... models run much faster than other models, but yield inferior quality for many applications. It can still make sense to try them for your use-case if your working on a laptop and want to annotate many images or volumetric data.

See also the figures above for examples where the finetuned models work better than the default models. We are working on further improving these models and adding new models for other biomedical imaging domains.

Other Models

Previous versions of our models are available on Zenodo:

  • vit_b_em_boundaries: for segmenting compartments delineated by boundaries such as cells or neurites in EM.
  • vit_b_em_organelles: for segmenting mitochondria, nuclei or other organelles in EM.
  • vit_b_lm: for segmenting cells and nuclei in LM.
  • vit_h_em: for general EM segmentation.
  • vit_h_lm: for general LM segmentation.

We do not recommend to use these models since our new models improve upon them significantly. But we provide the links here in case they are needed to reproduce older segmentation workflows.

We provide additional models that were used for experiments in our publication on Zenodo:

FAQ

Here we provide frequently asked questions and common issues. If you encounter a problem or question not addressed here feel free to open an issue or to ask your question on image.sc with the tag micro-sam.

Installation questions

1. How to install micro_sam?

The installation for micro_sam is supported in three ways: from mamba (recommended), from source and from installers. Check out our tutorial video to get started with micro_sam, briefly walking you through the installation process and how to start the tool.

2. I cannot install micro_sam using the installer, I am getting some errors.

The installer should work out-of-the-box on Windows and Linux platforms. Please open an issue to report the error you encounter.

NOTE: The installers enable using micro_sam without mamba or conda. However, we recommend the installation from mamba / from source to use all its features seamlessly. Specifically, the installers currently only support the CPU and won't enable you to use the GPU (if you have one).

3. What is the minimum system requirement for micro_sam?

From our experience, the micro_sam annotation tools work seamlessly on most laptop or workstation CPUs and with > 8GB RAM. You might encounter some slowness for $\leq$ 8GB RAM. The resources micro_sam's annotation tools have been tested on are:

  • Windows:
    • Windows 10 Pro, Intel i5 7th Gen, 8GB RAM
    • Windows 10 Enterprise LTSC, Intel i7 13th Gen, 32GB RAM
    • Windows 10 Pro for Workstations, Intel Xeon W-2295, 128GB RAM

  • Linux:

    • Ubuntu 20.04, Intel i7 11th Gen, 32GB RAM
    • Ubuntu 22.04, Intel i7 12th Gen, 32GB RAM

  • Mac:

    • macOS Sonoma 14.4.1
      • M1 Chip, 8GB RAM
      • M3 Max Chip, 36GB RAM

Having a GPU will significantly speed up the annotation tools and especially the model finetuning.

micro_sam has been tested mostly with CUDA 12.1 and PyTorch [2.1.1, 2.2.0]. However, the tool and the library is not constrained to a specific PyTorch or CUDA version. So it should work fine with the standard PyTorch installation for your system.

5. I am missing a few packages (eg. ModuleNotFoundError: No module named 'elf.io). What should I do?

With the latest release 1.0.0, the installation from mamba and source should take care of this and install all the relevant packages for you. So please reinstall micro_sam, following the installation guide.

6. Can I install micro_sam using pip?

We do not recommend installing micro-sam with pip. It has several dependencies that are only avaoiable from conda-forge, which will not install correctly via pip.

Please see the installation guide for the recommended way to install micro-sam.

The PyPI page for micro-sam exists only so that the napari-hub can find it.

7. I get the following error: importError: cannot import name 'UNETR' from 'torch_em.model'.

It's possible that you have an older version of torch-em installed. Similar errors could often be raised from other libraries, the reasons being: a) Outdated packages installed, or b) Some non-existent module being called. If the source of such error is from micro_sam, then a) is most likely the reason . We recommend installing the latest version following the installation instructions.

Usage questions

1. I have some micropscopy images. Can I use the annotator tool for segmenting them?

Yes, you can use the annotator tool for:

  • Segmenting objects in 2d images (using automatic and/or interactive segmentation).
  • Segmenting objects in 3d volumes (using automatic and/or interactive segmentation for the entire object(s)).
  • Tracking objects over time in time-series data.
  • Segmenting objects in a series of 2d / 3d images.
  • In addition, you can finetune the Segment Anything / micro_sam models on your own microscopy data, in case the provided models do not suffice your needs. One caveat: You need to annotate a few objects before-hand (micro_sam has the potential of improving interactive segmentation with only a few annotated objects) to proceed with the supervised finetuning procedure.

2. Which model should I use for my data?

We currently provide three different kind of models: the default models vit_h, vit_l, vit_b and vit_t; the models for light microscopy vit_l_lm, vit_b_lm and vit_t_lm; the models for electron microscopy vit_l_em_organelles, vit_b_em_organelles and vit_t_em_organelles. You should first try the model that best fits the segmentation task your interested in, the lm model for cell or nucleus segmentation in light microscopy or the em_organelles model for segmenting nuclei, mitochondria or other roundish organelles in electron microscopy. If your segmentation problem does not meet these descriptions, or if these models don't work well, you should try one of the default models instead. The letter after vit denotes the size of the image encoder in SAM, h (huge) being the largest and t (tiny) the smallest. The smaller models are faster but may yield worse results. We recommend to either use a vit_l or vit_b model, they offer the best trade-off between speed and segmentation quality. You can find more information on model choice here.

3. I have high-resolution microscopy images, micro_sam does not seem to work.

The Segment Anything model expects inputs of shape 1024 x 1024 pixels. Inputs that do not match this size will be internally resized to match it. Hence, applying Segment Anything to a much larger image will often lead to inferior results, or sometimes not work at all. To address this, micro_sam implements tiling: cutting up the input image into tiles of a fixed size (with a fixed overlap) and running Segment Anything for the individual tiles. You can activate tiling with the tile_shape parameter, which determines the size of the inner tile and halo, which determines the size of the additional overlap.

  • If you are using the micro_sam annotation tools, you can specify the values for the tile_shape and halo via the tile_x, tile_y, halo_x and halo_y parameters in the Embedding Settings drop-down menu.
  • If you are using the micro_sam library in a python script, you can pass them as tuples, e.g. tile_shape=(1024, 1024), halo=(256, 256). See also the wholeslide annotator example.
  • If you are using the command line functionality, you can pass them via the options --tile_shape 1024 1024 --halo 256 256.

NOTE: It's recommended to choose the halo so that it is larger than half of the maximal radius of the objects you want to segment.

4. The computation of image embeddings takes very long in napari.

micro_sam pre-computes the image embeddings produced by the vision transformer backbone in Segment Anything, and (optionally) stores them on disc. I fyou are using a CPU, this step can take a while for 3d data or time-series (you will see a progress bar in the command-line interface / on the bottom right of napari). If you have access to a GPU without graphical interface (e.g. via a local computer cluster or a cloud provider), you can also pre-compute the embeddings there and then copy them over to your laptop / local machine to speed this up.

  • You can use the command micro_sam.precompute_embeddings for this (it is installed with the rest of the software). You can specify the location of the pre-computed embeddings via the embedding_path argument.
  • You can cache the computed embedding in the napari tool (to avoid recomputing the embeddings again) by passing the path to store the embeddings in the embeddings_save_path option in the Embedding Settings drop-down. You can later load the pre-computed image embeddings by entering the path to the stored embeddings there as well.

5. Can I use micro_sam on a CPU?

Most other processing steps are very fast even on a CPU, the automatic segmentation step for the default Segment Anything models (typically called as the "Segment Anything" feature or AMG - Automatic Mask Generation) however takes several minutes without a GPU (depending on the image size). For large volumes and time-series, segmenting an object interactively in 3d / tracking across time can take a couple of seconds with a CPU (it is very fast with a GPU).

HINT: All the tutorial videos have been created on CPU resources.

6. I generated some segmentations from another tool, can I use it as a starting point in micro_sam?

You can save and load the results from the committed_objects layer to correct segmentations you obtained from another tool (e.g. CellPose) or save intermediate annotation results. The results can be saved via File -> Save Selected Layers (s) ... in the napari menu-bar on top (see the tutorial videos for details). They can be loaded again by specifying the corresponding location via the segmentation_result parameter in the CLI or python script (2d and 3d segmentation). If you are using an annotation tool you can load the segmentation you want to edit as segmentation layer and rename it to committed_objects.

7. I am using micro_sam for segmenting objects. I would like to report the steps for reproducability. How can this be done?

The annotation steps and segmentation results can be saved to a Zarr file by providing the commit_path in the commit widget. This file will contain all relevant information to reproduce the segmentation.

NOTE: This feature is still under development and we have not implemented rerunning the segmentation from this file yet. See this issue for details.

8. I want to segment objects with complex structures. Both the default Segment Anything models and the micro_sam generalist models do not work for my data. What should I do?

micro_sam supports interactive annotation using positive and negative point prompts, box prompts and polygon drawing. You can combine multiple types of prompts to improve the segmentation quality. In case the aforementioned suggestions do not work as desired, micro_sam also supports finetuning a model on your data (see the next section on finetuning). We recommend the following: a) Check which of the provided models performs relatively good on your data, b) Choose the best model as the starting point to train your own specialist model for the desired segmentation task.

9. I am using the annotation tool and napari outputs the following error: While emmitting signal ... an error ocurred in callback ... This is not a bug in psygnal. See ... above for details.

These messages occur when an internal error happens in micro_sam. In most cases this is due to inconsistent annotations and you can fix them by clearing the annotations. We want to remove these errors, so we would be very grateful if you can open an issue and describe the steps you did when encountering it.

10. The objects are not segmented in my 3d data using the interactive annotation tool.

The first thing to check is: a) make sure you are using the latest version of micro_sam (pull the latest commit from master if your installation is from source, or update the installation from conda / mamba using mamba update micro_sam), and b) try out the steps from the 3d annotation tutorial video to verify if this shows the same behaviour (or the same errors) as you faced. For 3d images, it's important to pass the inputs in the python axis convention, ZYX. c) try using a different model and change the projection mode for 3d segmentation. This is also explained in the video.

11. I have very small or fine-grained structures in my high-resolution microscopic images. Can I use micro_sam to annotate them?

Segment Anything does not work well for very small or fine-grained objects (e.g. filaments). In these cases, you could try to use tiling to improve results (see Point 3 above for details).

12. napari seems to be very slow for large images.

Editing (drawing / erasing) very large 2d images or 3d volumes is known to be slow at the moment, as the objects in the layers are stored in-memory. See the related issue.

13. While computing the embeddings (and / or automatic segmentation), a window stating: "napari" is not responding pops up.

This can happen for long running computations. You just need to wait a bit longer and the computation will finish.

14. I have 3D RGB microscopy volumes. How does micro_sam handle these images?

micro_sam performs automatic segmentation in 3D volumes by first segmenting slices individually in 2D and merging the segmentations across 3D based on overlap of objects between slices. The expected shape of your 3D RGB volume should be (Z * Y * X * 3) (reason: Segment Anything is devised to consider 3-channel inputs, so while the user provides micro-sam with 1-channel inputs, we handle this by triplicating this to fit the requirement, or with 3-channel inputs, we use them in the expected RGB array structures as it is).

15. I want to use a model stored in a different directory than the micro_sam cache. How can I do this?

The micro-sam CLIs for precomputation of image embeddings and annotators (Annotator 2d, Annotator 3d, Annotator Tracking, Image Series Annotator) accept the argument -c / --checkpoint to pass model checkpoints. If you start a micro-sam annotator from the napari plugin menu, you can provide the path to model checkpoints in the annotator widget (on right) under Embedding Settings drop-down in the custom weights path option.

NOTE: It is important to choose the correct model type when you opt for the above recommendation, using the -m / --model_type argument or selecting it from the Model dropdown in Embedding Settings respectively. Otherwise you will face parameter mismatch issues.

Fine-tuning questions

1. I have a microscopy dataset I would like to fine-tune Segment Anything for. Is it possible using micro_sam?

Yes, you can fine-tune Segment Anything on your own dataset. Here's how you can do it:

2. I would like to fine-tune Segment Anything on open-source cloud services (e.g. Kaggle Notebooks), is it possible?

Yes, you can fine-tune Segment Anything on your custom datasets on Kaggle (and BAND). Check out our tutorial notebook for this.

3. What kind of annotations do I need to finetune Segment Anything?

Annotations are referred to the instance segmentation labels, i.e. each object of interests in your microscopy images have an individual id to uniquely identify all the segmented objects. You can obtain them by micro_sam's annotation tools. In micro_sam, it's expected to provide dense segmentations (i.e. all objects per image are annotated) for finetuning Segment Anything with the additional decoder, however it's okay to use sparse segmentations (i.e. few objects per image are annotated) for just finetuning Segment Anything (without the additional decoder).

4. I have finetuned Segment Anything on my microscopy data. How can I use it for annotating new images?

You can load your finetuned model by entering the path to its checkpoint in the custom_weights_path field in the Embedding Settings drop-down menu. If you are using the python library or CLI you can specify this path with the checkpoint_path parameter.

5. What is the background of the new AIS (Automatic Instance Segmentation) feature in micro_sam?

micro_sam introduces a new segmentation decoder to the Segment Anything backbone, for enabling faster and accurate automatic instance segmentation, by predicting the distances to the object center and boundary as well as predicting foregrund, and performing seeded watershed-based postprocessing to obtain the instances.

6. I want to finetune only the Segment Anything model without the additional instance decoder.

The instance segmentation decoder is optional. So you can only finetune SAM or SAM and the additional decoder. Finetuning with the decoder will increase training times, but will enable you to use AIS. See this example for finetuning with both the objectives.

NOTE: To try out the other way round (i.e. the automatic instance segmentation framework without the interactive capability, i.e. a UNETR: a vision transformer encoder and a convolutional decoder), you can take inspiration from this example on LIVECell.

7. I have a NVIDIA RTX 4090Ti GPU with 24GB VRAM. Can I finetune Segment Anything?

Finetuning Segment Anything is possible in most consumer-grade GPU and CPU resources (but training being a lot slower on the CPU). For the mentioned resource, it should be possible to finetune a ViT Base (also abbreviated as vit_b) by reducing the number of objects per image to 15. This parameter has the biggest impact on the VRAM consumption and quality of the finetuned model. You can find an overview of the resources we have tested for finetuning here. We also provide a the convenience function micro_sam.training.train_sam_for_configuration that selects the best training settings for these configuration. This function is also used by the finetuning UI.

8. I want to create a dataloader for my data, to finetune Segment Anything.

Thanks to torch-em, a) Creating PyTorch datasets and dataloaders using the python library is convenient and supported for various data formats and data structures. See the tutorial notebook on how to create dataloaders using torch-em and the documentation for details on creating your own datasets and dataloaders; and b) finetuning using the napari tool eases the aforementioned process, by allowing you to add the input parameters (path to the directory for inputs and labels etc.) directly in the tool.

NOTE: If you have images with large input shapes with a sparse density of instance segmentations, we recommend using sampler for choosing the patches with valid segmentation for the finetuning purpose (see the example for PlantSeg (Root) specialist model in micro_sam).

9. How can I evaluate a model I have finetuned?

To validate a Segment Anything model for your data, you have different options, depending on the task you want to solve and whether you have segmentation annotations for your data.

We provide an example notebook that shows how to use this evaluation functionality.

Contribution Guide

Discuss your ideas

We welcome new contributions! First, discuss your idea by opening a new issue in micro-sam. This allows you to ask questions, and have the current developers make suggestions about the best way to implement your ideas.

Clone the repository

We use git for version control.

Clone the repository, and checkout the development branch:

$ git clone https://github.com/computational-cell-analytics/micro-sam.git
$ cd micro-sam
$ git checkout dev

Create your development environment

We use conda to manage our environments. If you don't have this already, install miniconda or mamba to get started.

Now you can create the environment, install user and developer dependencies, and micro-sam as an editable installation:

$ mamba env create environment_gpu.yaml
$ mamba activate sam
$ python -m pip install requirements-dev.txt
$ python -m pip install -e .

Make your changes

Now it's time to make your code changes.

Typically, changes are made branching off from the development branch. Checkout dev and then create a new branch to work on your changes.

$ git checkout dev
$ git checkout -b my-new-feature

We use google style python docstrings to create documentation for all new code.

You may also find it helpful to look at this developer guide, which explains the organization of the micro-sam code.

Testing

Run the tests

The tests for micro-sam are run with pytest

To run the tests:

$ pytest

Writing your own tests

If you have written new code, you will need to write tests to go with it.

Unit tests

Unit tests are the preferred style of tests for user contributions. Unit tests check small, isolated parts of the code for correctness. If your code is too complicated to write unit tests easily, you may need to consider breaking it up into smaller functions that are easier to test.

Tests involving napari

In cases where tests must use the napari viewer, these tips might be helpful (in particular, the make_napari_viewer_proxy fixture).

These kinds of tests should be used only in limited circumstances. Developers are advised to prefer smaller unit tests, and avoid integration tests wherever possible.

Code coverage

Pytest uses the pytest-cov plugin to automatically determine which lines of code are covered by tests.

A short summary report is printed to the terminal output whenever you run pytest. The full results are also automatically written to a file named coverage.xml.

The Coverage Gutters VSCode extension is useful for visualizing which parts of the code need better test coverage. PyCharm professional has a similar feature, and you may be able to find similar tools for your preferred editor.

We also use codecov.io to display the code coverage results from our Github Actions continuous integration.

Open a pull request

Once you've made changes to the code and written some tests to go with it, you are ready to open a pull request. You can mark your pull request as a draft if you are still working on it, and still get the benefit of discussing the best approach with maintainers.

Remember that typically changes to micro-sam are made branching off from the development branch. So, you will need to open your pull request to merge back into the dev branch like this.

Optional: Build the documentation

We use pdoc to build the documentation.

To build the documentation locally, run this command:

$ python build_doc.py

This will start a local server and display the HTML documentation. Any changes you make to the documentation will be updated in real time (you may need to refresh your browser to see the changes).

If you want to save the HTML files, append --out to the command, like this:

$ python build_doc.py --out

This will save the HTML files into a new directory named tmp.

You can add content to the documentation in two ways:

  1. By adding or updating google style python docstrings in the micro-sam code.
    • pdoc will automatically find and include docstrings in the documentation.
  2. By adding or editing markdown files in the micro-sam doc directory.
    • If you add a new markdown file to the documentation, you must tell pdoc that it exists by adding a line to the micro_sam/__init__.py module docstring (eg: .. include:: ../doc/my_amazing_new_docs_page.md). Otherwise it will not be included in the final documentation build!

Optional: Benchmark performance

There are a number of options you can use to benchmark performance, and identify problems like slow run times or high memory use in micro-sam.

Run the benchmark script

There is a performance benchmark script available in the micro-sam repository at development/benchmark.py.

To run the benchmark script:

$ python development/benchmark.py --model_type vit_t --device cpu`

For more details about the user input arguments for the micro-sam benchmark script, see the help:

$ python development/benchmark.py --help

Line profiling

For more detailed line by line performance results, we can use line-profiler.

line_profiler is a module for doing line-by-line profiling of functions. kernprof is a convenient script for running either line_profiler or the Python standard library's cProfile or profile modules, depending on what is available.

To do line-by-line profiling:

  1. Ensure you have line profiler installed: python -m pip install line_profiler
  2. Add @profile decorator to any function in the call stack
  3. Run kernprof -lv benchmark.py --model_type vit_t --device cpu

For more details about how to use line-profiler and kernprof, see the documentation.

For more details about the user input arguments for the micro-sam benchmark script, see the help:

$ python development/benchmark.py --help

Snakeviz visualization

For more detailed visualizations of profiling results, we use snakeviz.

SnakeViz is a browser based graphical viewer for the output of Python’s cProfile module.

  1. Ensure you have snakeviz installed: python -m pip install snakeviz
  2. Generate profile file: python -m cProfile -o program.prof benchmark.py --model_type vit_h --device cpu
  3. Visualize profile file: snakeviz program.prof

For more details about how to use snakeviz, see the documentation.

Memory profiling with memray

If you need to investigate memory use specifically, we use memray.

Memray is a memory profiler for Python. It can track memory allocations in Python code, in native extension modules, and in the Python interpreter itself. It can generate several different types of reports to help you analyze the captured memory usage data. While commonly used as a CLI tool, it can also be used as a library to perform more fine-grained profiling tasks.

For more details about how to use memray, see the documentation.

Creating a new release

To create a new release you have to edit the version number in micro_sam/__version__.py in a PR. After merging this PR the release will automatically be done by the CI.

Using micro_sam on BAND

BAND is a service offered by EMBL Heidelberg that gives access to a virtual desktop for image analysis tasks. It is free to use and micro_sam is installed there. In order to use BAND and start micro_sam on it follow these steps:

Start BAND

  • Go to https://band.embl.de/ and click Login. If you have not used BAND before you will need to register for BAND. Currently you can only sign up via a Google account.
  • Launch a BAND desktop with sufficient resources. It's particularly important to select a GPU. The settings from the image below are a good choice.
  • Go to the desktop by clicking GO TO DESKTOP in the Running Desktops menu. See also the screenshot below.

image

Start micro_sam in BAND

  • Select Applications -> Image Analysis -> uSAM (see screenshot) image
  • This will open the micro_sam menu, where you can select the tool you want to use (see screenshot). Note: this may take a few minutes. image
  • For testing if the tool works, it's best to use the 2d annotator first.
    • You can find an example image to use here: /scratch/cajal-connectomics/hela-2d-image.png. Select it via Select image. (see screenshot) image
  • Then press 2d annotator and the tool will start.

Transfering data to BAND

To copy data to and from BAND you can use any cloud storage, e.g. ownCloud, dropbox or google drive. For this, it's important to note that copy and paste, which you may need for accessing links on BAND, works a bit different in BAND:

  • To copy text into BAND you first need to copy it on your computer (e.g. via selecting it + Ctrl + C).
  • Then go to the browser window with BAND and press Ctrl + Shift + Alt. This will open a side window where you can paste your text via Ctrl + V.
  • Then select the text in this window and copy it via Ctrl + C.
  • Now you can close the side window via Ctrl + Shift + Alt and paste the text in band via Ctrl + V

The video below shows how to copy over a link from owncloud and then download the data on BAND using copy and paste:

https://github.com/computational-cell-analytics/micro-sam/assets/4263537/825bf86e-017e-41fc-9e42-995d21203287

 1"""
 2.. include:: ../doc/start_page.md
 3.. include:: ../doc/installation.md
 4.. include:: ../doc/annotation_tools.md
 5.. include:: ../doc/python_library.md
 6.. include:: ../doc/finetuned_models.md
 7.. include:: ../doc/faq.md
 8.. include:: ../doc/contributing.md
 9.. include:: ../doc/band.md
10"""
11import os
12
13from .__version__ import __version__
14
15os.environ["PYTORCH_ENABLE_MPS_FALLBACK"] = "1"