infer_locate_anything_detect

Run NVIDIA LocateAnything-3B for open-vocabulary object detection with Ikomia.

LocateAnything is a vision-language grounding model that can locate objects from natural-language descriptions. This Ikomia plugin returns bounding boxes for object categories or natural-language phrases, and point coordinates for pointing prompts.

🚀 Use with Ikomia API

1. Install Ikomia API

We strongly recommend using a virtual environment. If you're not sure where to start, Ikomia provides a tutorial here.

pip install ikomia

2. Install plugin requirements

pip install -r requirements.txt

LocateAnything-3B is hosted on Hugging Face and uses custom Transformers code. The first run downloads the model weights into the plugin weights cache folder.

3. Create your workflow

from ikomia.dataprocess.workflow import Workflow
from ikomia.utils.displayIO import display


wf = Workflow()

detector = wf.add_task(name="infer_locate_anything_detect", auto_connect=True)

wf.run_on(url="https://github.com/Ikomia-dev/notebooks/blob/main/examples/img/img_people_workspace.jpg?raw=true")

display(detector.get_image_with_graphics())

☀️ Use with Ikomia Studio

Ikomia Studio offers a friendly UI with the same features as the API.

• If you haven't started using Ikomia Studio yet, download and install it from this page.
• For additional guidance on getting started with Ikomia Studio, check out this blog post.

📝 Set algorithm parameters

• model_name (str) - default nvidia/LocateAnything-3B: Hugging Face model identifier or local model path.
• mode (str) - default detect: LocateAnything localization mode. Supported values are detect, ground_single, ground_multi, ground_gui, and point.
• prompt (str) - default laptop, plant, people: object categories or phrase, depending on mode.
• input_size (int) - default 800: resize the input image so its longest side is this value before inference. Bounding boxes are scaled back to the original image size.
• cuda (bool) - default depends on CUDA availability: if True, run on GPU when available; otherwise CPU is used.
• device_map (str) - default none: set to auto to let Transformers/Accelerate place model layers automatically. This can reduce memory pressure during loading, but none keeps standard Ikomia device placement.
• generation_mode (str) - default hybrid: LocateAnything generation mode. Supported values are hybrid, fast, and slow.
• max_new_tokens (int) - default 8192: maximum number of tokens generated by the model.
• temperature (float) - default 0.7: sampling temperature. Set to 0.0 for greedy generation.
• top_p (float) - default 0.9: nucleus sampling probability.
• repetition_penalty (float) - default 1.1: penalty used to reduce repetitive generation.

from ikomia.dataprocess.workflow import Workflow
from ikomia.utils.displayIO import display


wf = Workflow()

detector = wf.add_task(name="infer_locate_anything_detect", auto_connect=True)
detector.set_parameters({
    "model_name": "nvidia/LocateAnything-3B",
    "mode": "detect",
    "prompt": "laptop, plant, people",
    "input_size": "800",
    "cuda": "True",
    "device_map": "none",
    "generation_mode": "hybrid",
    "max_new_tokens": "8192",
    "temperature": "0.7",
    "top_p": "0.9",
    "repetition_penalty": "1.1",
})

wf.run_on(url="https://github.com/Ikomia-dev/notebooks/blob/main/examples/img/img_people_workspace.jpg?raw=true")

if detector.get_param_object().mode == "point":
    img_output = detector.get_output(0)
    point_output = detector.get_output(2)
    display(img_output.get_image_with_graphics(point_output))
else:
    display(detector.get_image_with_graphics())

Mode behavior

The mode parameter selects the LocateAnything task prompt used before inference. It changes how the plugin interprets prompt, how detections are mapped to Ikomia classes, and whether the result is a box or a point.

• detect: split prompt as comma-separated categories. Each category becomes an Ikomia class and is queried with Locate all the instances that matches the following description: {category}.
• ground_single: use the full prompt as one phrase, query with Locate a single instance that matches the following description: {prompt}., and keep the first parsed box.
• ground_multi: use the full prompt as one phrase, query with Locate all the instances that match the following description: {prompt}., and return all parsed boxes.
• ground_gui: use the full prompt as one GUI phrase, query only with Locate the region that matches the following description: {prompt}., and return box outputs.
• point: use the full prompt as one phrase, query with Point to: {prompt}., and return parsed points in the keypoints output as CPointF coordinates.

Generation mode behavior

The generation_mode parameter controls LocateAnything's coordinate decoding strategy inside model.generate(). It does not change what the model searches for; it changes how the model generates the structured coordinate answer.

• hybrid: default mode recommended for balanced speed and output quality.
• fast: faster decoding, useful when throughput matters more than maximum coordinate quality.
• slow: slower decoding, useful when prioritizing output quality over speed.

Use mode to choose the detection or grounding prompt type. Use generation_mode to tune the model's decoding behavior for that prompt.

📚 References

• LocateAnything model card: https://huggingface.co/nvidia/LocateAnything-3B
• NVIDIA Eagle repository: https://github.com/NVlabs/Eagle
• LocateAnything worker: https://github.com/NVlabs/Eagle/blob/main/Embodied/locateanything_worker.py

⚠️ License

LocateAnything-3B is released under the NVIDIA License for research and development use. Review the model card and license terms before using the model.