Annotate For Phrase Grounding

This workflow is ideal when you know what you want to describe before drawing boxes.

[GIF PLACEHOLDER: Type caption → highlight phrase → draw linked bounding box]

1. Write your caption using the Text tool (press T)

2. Highlight a phrase:

   • Press H or click the Highlight tool
   • Click and drag to select a phrase in your caption
   • The phrase becomes highlighted with a colored background

3. Draw a bounding box:

   • You'll be automatically prompted to draw a box
   • Press R or click the Rectangle tool
   • Click and drag on the image to draw a box around the object
   • The box will be linked to your highlighted phrase with a matching color

4. Repeat for other phrases:

   • Press H to highlight another phrase
   • Draw its corresponding bounding box
   • Continue until all key objects are annotated

Tip: Each phrase-box pair gets a unique color to help you quickly see which phrases link to which boxes.

⚠️

Highlighted phrases cannot overlap

Each highlighted phrase must be a distinct, non-overlapping text interval in your caption. You cannot highlight overlapping text regions. Plan your caption structure to ensure phrases are clearly separated.

This workflow is ideal when you want to identify objects visually before describing them.

[GIF PLACEHOLDER: Draw bounding box → link by highlighting phrase in caption]

1. Draw a bounding box:

   • Press R or click the Rectangle tool
   • Click and drag on the image to draw a box around an object
   • The box appears unlinked (no associated phrase yet)

2. Link to a phrase:

   • You'll be automatically prompted to select a phrase
   • Press H or click the Highlight tool
   • Highlight the phrase in your caption that describes this object
   • The box and phrase become linked with a matching color

3. Repeat for other objects:

   • Press R to draw another box
   • Highlight its corresponding phrase
   • Continue until all key objects are annotated

Alternative: Use IntelliScribe to auto-link phrases

If you have unlinked boxes:

1. Press P or click IntelliScribe Phrases
2. The AI will automatically highlight phrases in your caption that match your boxes
3. Review and adjust the highlighted phrases as needed

Prerequisites: You must have at least one bounding box drawn and a caption written.

Learn more about IntelliScribe →

1. Press T or click the Text tool

2. Click in the caption text area

3. Make your changes

4. Changes save automatically as you type

Use when: Fixing typos, adding details, rephrasing descriptions

1. Click on the bounding box you want to adjust

2. Drag the box to reposition it

3. Drag the corner or edge handles to resize it

4. The linked phrase remains connected

Use when: Adjusting box position or size while keeping the same phrase link

1. Press D or click the Delete tool

2. Click on the bounding box you want to remove

3. The box and its phrase link are deleted (the phrase text remains in the caption)

Use when: Removing incorrect boxes, duplicate annotations, or boxes on wrong objects

❗️

Deletions cannot be undone. Make sure you want to remove the annotation before clicking.

1. Press D or click the Delete tool

2. Click on the highlighted phrase in the caption

3. The highlight and box link are removed (the box and caption text remain)

Use when: Unlinking a phrase to connect it to a different box, or removing phrase highlights while keeping boxes

❗️

Deletions cannot be undone. You can re-highlight the phrase to link it again.

Efficient annotation pattern:

1. Press T → Type caption
2. Press H → Highlight phrase → Press R → Draw box
3. Repeat step 2 for other phrases
4. Press E to move to next image

Best for: Beginners, systematic annotation, descriptive-heavy tasks

Alternative pattern:

1. Press R → Draw box → Press H → Highlight phrase
2. Repeat for other objects
3. Press E to move to next image

Best for: Experienced annotators, visual identification, speed

Quick AI-assisted pattern:

1. Press C → Review generated caption → Press T to edit if needed
2. Press R → Draw boxes around objects
3. Press P → Review AI-highlighted phrases → Adjust if needed
4. Press E to move to next image

Best for: Large datasets, common objects, rapid annotation

Learn more about IntelliScribe →

Be descriptive and specific:

✅ Good:

• "A large black FPGA chip in the center, several white tactile push buttons on the left side, a blue VGA connector at the bottom"
• "Two red apples in a wooden bowl on the left, three green apples scattered on the right side of the table"
• "A worker wearing a yellow safety helmet and orange vest standing next to a forklift"

❌ Too vague:

• "Some electronics"
• "Fruits on a table"
• "A person at work"

Include spatial information:

Use positional terms to help the model understand object locations:

• Position: "in the center", "on the left edge", "at the top", "near the bottom"
• Relative position: "above the", "next to", "between the", "surrounding"
• Direction: "facing left", "pointing upward", "horizontal", "vertical"

Use consistent terminology:

• Choose specific object names and use them consistently across your dataset
• Define a vocabulary for your domain (e.g., "FPGA chip" vs. "integrated circuit")
• Standardize color descriptions (e.g., "blue" not "light blue" or "navy")
• Use the same format for similar objects

Draw tight boxes:

• Include the entire object but minimize empty space
• Align box edges with object boundaries
• For irregular objects, include the smallest rectangle that contains the whole object

Be consistent:

• Draw boxes at the same level of detail across your dataset
• If you include shadows or reflections for one object, do so for all similar objects
• Maintain consistent box tightness (not too tight on some, loose on others)

Handle overlapping objects:

• Draw separate boxes for each object, even if they overlap
• Ensure each box clearly defines one object's boundaries
• Link each box to its own phrase in the caption

Handle partially visible objects:

• Draw boxes around visible portions of partially occluded objects
• Include a phrase like "partially visible" in the caption if relevant
• Be consistent about whether you annotate partially visible objects

Phrases must not overlap:

Highlighted phrases must be distinct, non-overlapping text intervals in your caption. You cannot highlight the same words multiple times or create overlapping highlights.

✅ Good:

• "A large black FPGA chip in the center" and "several white tactile buttons on the left" (separate, non-overlapping phrases)
• "red apple" and "blue bowl" (completely distinct text regions)

❌ Invalid:

• Highlighting "black FPGA chip" and then trying to highlight "FPGA chip in the center" (overlapping words)
• Highlighting "red apple" and then highlighting "apple" again (duplicate highlight of same word)

Best practice: Write your caption with clearly separated descriptive phrases for each object you plan to annotate.

One phrase per box:

Each bounding box should link to exactly one phrase in the caption. Break complex descriptions into multiple linked annotations if needed.

✅ Good:

• Phrase: "large black FPGA chip" → Box: around the FPGA chip
• Phrase: "row of gold pin headers" → Box: around the pin headers
• Phrase: "7-segment LED display" → Box: around the LED display

❌ Too complex:

• Phrase: "large black FPGA chip with pin headers and an LED display" → Multiple boxes
• Better: Split into three separate phrase-box pairs

Match phrase specificity to box:

• If your phrase mentions color, size, or location, ensure these are accurate for the box
• Don't use generic phrases like "object" or "thing" unless unavoidable
• Include distinguishing features that help identify the specific object

Link all key objects:

• Annotate all prominent objects in the image
• Don't leave obvious objects without phrase-box links
• For complex scenes, prioritize the most important 5-10 objects

Balance detail with practicality:

• For simple scenes: Link most visible objects (5-8 phrase-box pairs)
• For complex scenes: Focus on key objects and representative examples (8-12 pairs)
• Avoid exhaustive annotation of every tiny detail

Possible causes and solutions:

1. Network connectivity — Check your internet connection. Annotations require an active connection to save.

2. Session timeout — If you've been idle, refresh the page and reopen the annotator.

3. Browser issues — Clear cache or try a different browser. Supported browsers: Chrome, Firefox, Safari, Edge.

4. Incomplete annotation — Ensure boxes are properly linked to phrases. Unlinked boxes may not save correctly.

Verify your annotations saved:

• Check your dataset Explorer Annotations tab for statistics
• Review the bottom thumbnail strip in the annotator for "Annotated" badges

Solution steps:

1. Activate Rectangle tool — Press R or click the Rectangle tool icon

2. Check tool mode — If another tool is active (H, T, D), press Esc to exit, then press R

3. Zoom level — If the image is too zoomed in/out, adjust zoom to see the area clearly

4. Browser permissions — Ensure your browser allows drawing on canvas elements

Performance optimization:

1. Large images — Images over 10 MB may load slowly. Upload optimized images when possible.

2. Many annotations — Images with 20+ boxes may be slower. Consider splitting complex scenes across multiple images.

3. Browser resources — Close unnecessary tabs and applications.

4. Use keyboard shortcuts — Keyboard navigation is faster than mouse clicks.

Solution:

1. Press D or click the Delete tool

2. Click on the highlighted phrase in the caption (not the box)

3. The phrase highlight and link are removed, but the bounding box remains

4. You can now highlight a different phrase to link to that box

Alternatively, delete the box and redraw it linked to the correct phrase.

Common issues:

1. IntelliScribe Caption (C) — Works on any image. If it fails, check your internet connection.

2. IntelliScribe Phrases (P) — Requires:

   • At least one bounding box drawn
   • A caption with text
   • Refresh the page if it still doesn't work

Learn more about IntelliScribe →

Unfortunately, deletions cannot be undone. Best practices to prevent this:

1. Be careful with Delete tool — Press D intentionally, not by accident

2. Exit Delete mode — Press Esc after deleting to prevent accidental deletions

3. Export backups — Regularly download your dataset as a backup

4. Review before navigating — Check your work before pressing E to move to the next image

Both approaches work:

Caption first (recommended for beginners):

• Write complete caption describing all objects
• Highlight phrases systematically
• Draw boxes for each highlighted phrase
• More organized and less likely to miss objects

Boxes first (faster for experienced annotators):

• Identify objects visually
• Draw boxes quickly
• Link to phrases in caption
• Use IntelliScribe Phrases to auto-highlight

Choose based on your preference and workflow. See Link phrases to bounding boxes for both workflows.

Balance detail with practicality:

Include:

• Main objects and their attributes (color, size, shape)
• Spatial relationships and positions
• Distinguishing features
• Counts when relevant ("three apples", "several buttons")

Avoid:

• Excessive detail about unimportant elements
• Subjective opinions ("beautiful", "interesting")
• Information not visible in the image
• Redundant descriptions

Good length: 10-30 words for simple scenes, 20-50 words for complex scenes

See Caption writing guidelines for examples.

Generally no, but there are exceptions:

Standard practice:

• Each object gets one bounding box
• Each box links to one phrase
• No duplicate boxes on the same object

Exceptions:

• Object has multiple distinct parts worth annotating separately
• Object appears in different contexts requiring different descriptions
• Dataset requires multiple annotation styles for comparison

Be consistent with your approach across the entire dataset.

Draw separate boxes for each object:

1. Draw a box around the first object (even if partially occluded)
2. Draw a box around the second object
3. Boxes can overlap in the interface
4. Link each box to its own phrase in the caption

Example:

• Caption: "A red apple behind a blue bowl on the left, a green apple visible in the bowl"
• Box 1: Around visible part of red apple → "red apple behind a blue bowl"
• Box 2: Around blue bowl → "blue bowl on the left"
• Box 3: Around green apple → "green apple visible in the bowl"

Yes! Multiple team members can annotate different images in the same dataset:

• Add team members to your organization
• Assign different images to different annotators
• Create annotation guidelines based on this documentation
• Reference Annotation guidelines for consistency

Pro tip: Create a style guide for your team covering:

• Vocabulary and terminology to use
• How tight to draw boxes
• How detailed captions should be
• Which objects to prioritize

Yes! You can upload pre-existing phrase grounding annotations:

• Upload phrase grounding annotations in supported formats
• This is useful if you have annotations from other tools or datasets
• You can then edit these annotations in the annotator

After importing, check your dataset Explorer to verify your uploaded annotations.

Use IntelliScribe to speed up annotation, but always review:

IntelliScribe strengths:

• Fast caption generation
• Good for common objects
• Useful for getting started quickly
• Helpful for large datasets

When to annotate manually:

• Domain-specific objects
• Precise terminology required
• Complex relationships between objects
• Quality over speed

Best approach: Combine both:

1. Use IntelliScribe Caption (press C) to generate initial caption
2. Edit caption manually (press T) to add domain-specific terms
3. Draw boxes manually (press R) for precision
4. Use IntelliScribe Phrases (press P) to auto-link phrases
5. Review and adjust links as needed

Learn more about IntelliScribe →

• Use the same terminology for similar objects across images
• Draw boxes at consistent tightness across your dataset
• Follow the same caption writing guidelines for all images
• Create a style guide if annotating collaboratively

• Accurate boxes are more valuable than many imprecise ones
• Take time to write descriptive captions
• Review and edit annotations when you spot errors
• Check the bottom thumbnail strip to review which images have been annotated

• Annotate objects at different scales (small, medium, large)
• Include objects in different positions and contexts
• Cover different lighting conditions and angles
• Include partially visible objects when relevant

• Learn keyboard shortcuts to speed up annotation
• Choose a workflow (caption-first vs. box-first) and stick with it
• Use IntelliScribe to speed up initial annotation
• Monitor your progress through the dataset Explorer Annotations tab
• Take breaks to maintain annotation quality