Phrase Grounding

Enabling robots to identify and manipulate objects using natural language commands

Instead of programming robots to recognize specific objects, use natural language to direct them flexibly and intuitively.

Example commands:

• "Pick up the red mug on the left"
• "Move the largest box to the shelf"
• "Grab the wrench next to the screwdriver"

This makes robots more adaptable to changing environments and easier to control without reprogramming.

Overlaying information on specific real-world objects identified through natural language

Users can point at objects and ask questions or request information using natural descriptions rather than pre-programmed markers.

Example applications:

• "Show me information about the painting on the right wall"
• "Highlight the circuit board component labeled A3"
• "Display specs for the machine in the center"

Selecting and editing image regions using natural language instead of manual tools

Replace tedious manual selection with simple text descriptions to speed up editing workflows.

Example selections:

• "Select the person in the red shirt"
• "Highlight all the products on the top shelf"
• "Mask the car in the background"

This makes image editing more accessible to non-technical users who struggle with traditional selection tools.

Localizing traffic elements, pedestrians, or obstacles based on descriptive commands

Help autonomous systems identify and respond to specific objects in complex traffic scenarios.

Example detections:

• "The pedestrian crossing from the right"
• "The stop sign partially hidden by the tree"
• "The vehicle merging from the left lane"

This enables more nuanced understanding of traffic situations beyond simple category detection.

Identifying specific anatomical structures or anomalies described in radiology reports

Connect textual medical descriptions directly to image regions without manual annotation.

Example identifications:

• "The lesion in the upper right lobe"
• "The fracture near the distal end"
• "The enlarged lymph node adjacent to the trachea"

🚧

Important

Phrase grounding should assist, not replace, medical professionals. Always have qualified healthcare providers review and validate AI-generated localizations.

Locating specific products in warehouse images or store shelves

Find products quickly using natural descriptions rather than maintaining complex product databases.

Example searches:

• "The blue box on the third shelf"
• "The damaged package in the corner"
• "All products with red labels"

This speeds up inventory management, picking operations, and quality control.

Finding specific objects or people across video frames

Track and locate specific elements in video content using textual descriptions for surveillance or content indexing.

Example tracking:

• "The person wearing the yellow jacket"
• "The black sedan that entered from the north entrance"
• "The package placed on the counter at 3:15 PM"

Enables efficient video search without manually reviewing hours of footage.

Good phrases include distinguishing attributes:

✅ Good: "The red mug on the left side of the table" ❌ Too vague: "The mug"

✅ Good: "The person wearing a blue jacket and glasses" ❌ Ambiguous: "That person"

Include colors, positions, sizes, and relationships to help the VLM distinguish between similar objects.

Learn more about annotation best practices →

Leverage spatial relationships:

• Absolute positions: "on the left", "in the center", "at the top"
• Relative positions: "next to the door", "behind the car", "under the table"
• Ordinal positions: "the first chair", "the rightmost box", "the tallest building"

These help disambiguate when multiple similar objects are present.

Combine multiple characteristics:

✅ Better: "The small red box on the top shelf" ⚠️ Okay: "The red box"

Multiple attributes make it easier for the model to identify the exact target object, especially in cluttered scenes.

Train your VLM with varied scenarios:

• Different object types and categories
• Various phrasings for similar objects
• Multiple attributes (color, size, position, condition)
• Complex scenes with many objects
• Different lighting conditions and viewpoints

More diversity in your training dataset = better generalization to new descriptions and scenes.

Learn about dataset preparation →

Use consistent language across your dataset:

• Choose one term and stick with it ("damaged" vs. "defective" vs. "broken")
• Use consistent color names ("red" not "crimson" or "scarlet")
• Maintain consistent spatial descriptions ("left" not "left side" or "left-hand side")

Consistency helps the model learn more effectively and produce reliable results.

For good model performance, aim for:

• Minimum: 100 phrase-box pairs across at least 20 images
• Recommended: 500+ pairs across 100+ images
• Optimal: 1000+ pairs for production use

More diverse examples with varied phrases and object types lead to better performance on new descriptions. See dataset insights to track your annotation progress.

Learn how to upload annotations →

Yes! Vision-language models support zero-shot grounding, where you can describe objects without any training examples.

Zero-shot works well for:

• Quick experiments and prototypes
• Common objects with standard descriptions
• Scenarios where you don't have training data

For specialized domains or higher accuracy, fine-tuning with your own dataset is recommended. Use pre-trained models as a starting point.

Learn how to run zero-shot inference →

Phrase Grounding localizes objects with bounding boxes based on descriptions.

• Input: Image + description ("the red car")
• Output: Bounding box coordinates

Visual Question Answering answers questions about images with text.

• Input: Image + question ("What color is the car?")
• Output: Text answer ("red")

Use phrase grounding to find things, VQA to understand things.

Learn more about VQA →

Yes! You can provide multiple phrases to locate multiple objects in the same image during inference:

• "The red car" → Box 1
• "The blue bicycle" → Box 2
• "The person in yellow" → Box 3

Each phrase gets its own bounding box, allowing you to locate everything you need in one pass. Process the results to extract and visualize all detections.

Learn about handling multiple results →

Modern vision-language models handle complex expressions including:

• Negations: "The dog that is NOT wearing a collar"
• Relationships: "The cup next to the laptop"
• Multiple attributes: "The large red car parked behind the building"
• Conditions: "The damaged box on the bottom shelf"

The more context you provide, the more precisely the model can identify the target.

Absolutely! You can create multiple datasets with different task types and use them together in your application workflow.

For example:

1. Use phrase grounding to locate specific products
2. Use VQA to answer questions about those products
3. Use classification to categorize them

This creates powerful, flexible vision systems that go beyond single-task models. Train separate models for each task and combine them in your application.

Learn about VQA →