File size: 8,965 Bytes
23c2fac 4e0112a 23c2fac 695dbab 23c2fac 695dbab 9f1c102 695dbab 9fefdcb 695dbab 9f1c102 695dbab 9f1c102 695dbab 5f6111d 695dbab 9fefdcb 23c2fac | 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99 100 101 102 103 104 105 106 107 108 109 110 111 112 113 114 115 116 117 118 119 120 121 122 123 124 125 126 127 128 129 130 131 132 133 134 | ---
library_name: pytorch
license: other
tags:
- foundation
- android
pipeline_tag: image-segmentation
---
# MobileSam: Optimized for Qualcomm Devices
Transformer based encoder-decoder where prompts specify what to segment in an image thereby allowing segmentation without the need for additional training. The image encoder generates embeddings and the lightweight decoder operates on the embeddings for point and mask based image segmentation.
This is based on the implementation of MobileSam found [here](https://github.com/facebookresearch/segment-anything).
This repository contains pre-exported model files optimized for Qualcomm® devices. You can use the [Qualcomm® AI Hub Models](https://github.com/qualcomm/ai-hub-models/blob/main/src/qai_hub_models/models/mobilesam) library to export with custom configurations. More details on model performance across various devices, can be found [here](#performance-summary).
Qualcomm AI Hub Models uses [Qualcomm AI Hub Workbench](https://workbench.aihub.qualcomm.com) to compile, profile, and evaluate this model. [Sign up](https://myaccount.qualcomm.com/signup) to run these models on a hosted Qualcomm® device.
## Getting Started
There are two ways to deploy this model on your device:
### Option 1: Download Pre-Exported Models
Below are pre-exported model assets ready for deployment.
| Runtime | Precision | Chipset | SDK Versions | Download |
|---|---|---|---|---|
| ONNX | float | Universal | QAIRT 2.42, ONNX Runtime 1.24.3 | [Download](https://qaihub-public-assets.s3.us-west-2.amazonaws.com/qai-hub-models/models/mobilesam/releases/v0.50.2/mobilesam-onnx-float.zip)
| QNN_DLC | float | Universal | QAIRT 2.43 | [Download](https://qaihub-public-assets.s3.us-west-2.amazonaws.com/qai-hub-models/models/mobilesam/releases/v0.50.2/mobilesam-qnn_dlc-float.zip)
| TFLITE | float | Universal | QAIRT 2.43, TFLite 2.19.1 | [Download](https://qaihub-public-assets.s3.us-west-2.amazonaws.com/qai-hub-models/models/mobilesam/releases/v0.50.2/mobilesam-tflite-float.zip)
For more device-specific assets and performance metrics, visit **[MobileSam on Qualcomm® AI Hub](https://aihub.qualcomm.com/models/mobilesam)**.
### Option 2: Export with Custom Configurations
Use the [Qualcomm® AI Hub Models](https://github.com/qualcomm/ai-hub-models/blob/main/src/qai_hub_models/models/mobilesam) Python library to compile and export the model with your own:
- Custom weights (e.g., fine-tuned checkpoints)
- Custom input shapes
- Target device and runtime configurations
This option is ideal if you need to customize the model beyond the default configuration provided here.
See our repository for [MobileSam on GitHub](https://github.com/qualcomm/ai-hub-models/blob/main/src/qai_hub_models/models/mobilesam) for usage instructions.
## Model Details
**Model Type:** Model_use_case.semantic_segmentation
**Model Stats:**
- Model checkpoint: vit_t
- Input resolution: 720p (720x1280)
- Number of parameters (sam_encoder): 6.95M
- Model size (sam_encoder) (float): 26.6 MB
- Number of parameters (sam_decoder): 6.16M
- Model size (sam_decoder) (float): 23.7 MB
## Performance Summary
| Model | Runtime | Precision | Chipset | Inference Time (ms) | Peak Memory Range (MB) | Primary Compute Unit
|---|---|---|---|---|---|---
| decoder | ONNX | float | Snapdragon® 8 Elite Gen 5 Mobile | 2.948 ms | 0 - 243 MB | NPU
| decoder | ONNX | float | Snapdragon® X2 Elite | 3.023 ms | 11 - 11 MB | NPU
| decoder | ONNX | float | Snapdragon® X Elite | 7.621 ms | 11 - 11 MB | NPU
| decoder | ONNX | float | Snapdragon® 8 Gen 3 Mobile | 5.008 ms | 5 - 350 MB | NPU
| decoder | ONNX | float | Qualcomm® QCS8550 (Proxy) | 7.11 ms | 0 - 15 MB | NPU
| decoder | ONNX | float | Qualcomm® QCS9075 | 8.436 ms | 4 - 7 MB | NPU
| decoder | ONNX | float | Snapdragon® 8 Elite For Galaxy Mobile | 3.655 ms | 0 - 222 MB | NPU
| decoder | QNN_DLC | float | Snapdragon® 8 Elite Gen 5 Mobile | 2.379 ms | 3 - 203 MB | NPU
| decoder | QNN_DLC | float | Snapdragon® X2 Elite | 2.849 ms | 4 - 4 MB | NPU
| decoder | QNN_DLC | float | Snapdragon® X Elite | 5.973 ms | 4 - 4 MB | NPU
| decoder | QNN_DLC | float | Snapdragon® 8 Gen 3 Mobile | 3.844 ms | 0 - 311 MB | NPU
| decoder | QNN_DLC | float | Qualcomm® QCS8275 (Proxy) | 12.991 ms | 1 - 283 MB | NPU
| decoder | QNN_DLC | float | Qualcomm® QCS8550 (Proxy) | 5.527 ms | 4 - 6 MB | NPU
| decoder | QNN_DLC | float | Qualcomm® SA8775P | 6.478 ms | 0 - 284 MB | NPU
| decoder | QNN_DLC | float | Qualcomm® QCS9075 | 6.913 ms | 4 - 10 MB | NPU
| decoder | QNN_DLC | float | Qualcomm® QCS8450 (Proxy) | 9.63 ms | 2 - 227 MB | NPU
| decoder | QNN_DLC | float | Qualcomm® SA7255P | 12.991 ms | 1 - 283 MB | NPU
| decoder | QNN_DLC | float | Qualcomm® SA8295P | 8.029 ms | 0 - 196 MB | NPU
| decoder | QNN_DLC | float | Snapdragon® 8 Elite For Galaxy Mobile | 2.801 ms | 0 - 267 MB | NPU
| decoder | TFLITE | float | Snapdragon® 8 Elite Gen 5 Mobile | 2.387 ms | 0 - 206 MB | NPU
| decoder | TFLITE | float | Snapdragon® 8 Gen 3 Mobile | 3.904 ms | 0 - 237 MB | NPU
| decoder | TFLITE | float | Qualcomm® QCS8275 (Proxy) | 13.052 ms | 0 - 201 MB | NPU
| decoder | TFLITE | float | Qualcomm® QCS8550 (Proxy) | 5.649 ms | 0 - 8 MB | NPU
| decoder | TFLITE | float | Qualcomm® SA8775P | 6.692 ms | 0 - 203 MB | NPU
| decoder | TFLITE | float | Qualcomm® QCS9075 | 6.943 ms | 0 - 19 MB | NPU
| decoder | TFLITE | float | Qualcomm® QCS8450 (Proxy) | 9.518 ms | 0 - 224 MB | NPU
| decoder | TFLITE | float | Qualcomm® SA7255P | 13.052 ms | 0 - 201 MB | NPU
| decoder | TFLITE | float | Qualcomm® SA8295P | 8.088 ms | 0 - 200 MB | NPU
| decoder | TFLITE | float | Snapdragon® 8 Elite For Galaxy Mobile | 2.939 ms | 0 - 205 MB | NPU
| encoder | ONNX | float | Snapdragon® 8 Elite Gen 5 Mobile | 84.094 ms | 20 - 1522 MB | NPU
| encoder | ONNX | float | Snapdragon® X2 Elite | 96.11 ms | 14 - 14 MB | NPU
| encoder | ONNX | float | Snapdragon® X Elite | 172.164 ms | 12 - 12 MB | NPU
| encoder | ONNX | float | Snapdragon® 8 Gen 3 Mobile | 125.916 ms | 20 - 3043 MB | NPU
| encoder | ONNX | float | Qualcomm® QCS8550 (Proxy) | 177.901 ms | 12 - 40 MB | NPU
| encoder | ONNX | float | Qualcomm® QCS9075 | 200.498 ms | 12 - 15 MB | NPU
| encoder | ONNX | float | Snapdragon® 8 Elite For Galaxy Mobile | 100.485 ms | 9 - 1390 MB | NPU
| encoder | QNN_DLC | float | Snapdragon® 8 Elite Gen 5 Mobile | 48.926 ms | 12 - 1324 MB | NPU
| encoder | QNN_DLC | float | Snapdragon® X2 Elite | 51.713 ms | 12 - 12 MB | NPU
| encoder | QNN_DLC | float | Snapdragon® X Elite | 100.189 ms | 12 - 12 MB | NPU
| encoder | QNN_DLC | float | Snapdragon® 8 Gen 3 Mobile | 66.047 ms | 10 - 1628 MB | NPU
| encoder | QNN_DLC | float | Qualcomm® QCS8275 (Proxy) | 209.31 ms | 0 - 1210 MB | NPU
| encoder | QNN_DLC | float | Qualcomm® QCS8550 (Proxy) | 97.222 ms | 12 - 15 MB | NPU
| encoder | QNN_DLC | float | Qualcomm® SA8775P | 103.074 ms | 1 - 1209 MB | NPU
| encoder | QNN_DLC | float | Qualcomm® QCS9075 | 121.864 ms | 12 - 30 MB | NPU
| encoder | QNN_DLC | float | Qualcomm® QCS8450 (Proxy) | 469.913 ms | 11 - 1740 MB | NPU
| encoder | QNN_DLC | float | Qualcomm® SA7255P | 209.31 ms | 0 - 1210 MB | NPU
| encoder | QNN_DLC | float | Qualcomm® SA8295P | 474.209 ms | 0 - 1258 MB | NPU
| encoder | QNN_DLC | float | Snapdragon® 8 Elite For Galaxy Mobile | 49.637 ms | 12 - 1215 MB | NPU
| encoder | TFLITE | float | Snapdragon® 8 Elite Gen 5 Mobile | 104.259 ms | 32 - 1233 MB | NPU
| encoder | TFLITE | float | Snapdragon® 8 Gen 3 Mobile | 167.58 ms | 17 - 1923 MB | NPU
| encoder | TFLITE | float | Qualcomm® QCS8275 (Proxy) | 360.435 ms | 29 - 1180 MB | NPU
| encoder | TFLITE | float | Qualcomm® QCS8550 (Proxy) | 216.954 ms | 33 - 37 MB | NPU
| encoder | TFLITE | float | Qualcomm® SA8775P | 240.406 ms | 33 - 1227 MB | NPU
| encoder | TFLITE | float | Qualcomm® QCS9075 | 289.564 ms | 32 - 109 MB | NPU
| encoder | TFLITE | float | Qualcomm® QCS8450 (Proxy) | 670.636 ms | 23 - 2068 MB | NPU
| encoder | TFLITE | float | Qualcomm® SA7255P | 360.435 ms | 29 - 1180 MB | NPU
| encoder | TFLITE | float | Qualcomm® SA8295P | 523.394 ms | 19 - 1181 MB | NPU
| encoder | TFLITE | float | Snapdragon® 8 Elite For Galaxy Mobile | 131.152 ms | 32 - 1235 MB | NPU
## License
* The license for the original implementation of MobileSam can be found
[here](https://github.com/facebookresearch/segment-anything/blob/main/LICENSE).
## References
* [Segment Anything](https://arxiv.org/abs/2306.14289)
* [Source Model Implementation](https://github.com/facebookresearch/segment-anything)
## Community
* Join [our AI Hub Slack community](https://aihub.qualcomm.com/community/slack) to collaborate, post questions and learn more about on-device AI.
* For questions or feedback please [reach out to us](mailto:ai-hub-support@qti.qualcomm.com).
|