Configure Training Settings

For your first training run:

1. Use 1 GPU (typically 1× T4 or L4) to validate configuration
2. Enable Advanced Evaluation to understand model behavior
3. Set checkpointing to 1-2 epochs for frequent feedback
4. Monitor resource usage and training progress

After validation:

1. Scale to multi-GPU for faster training
2. Disable Advanced Evaluation if speed is priority
3. Adjust checkpoint frequency based on training duration
4. Optimize batch size for GPU memory utilization

Model size considerations:

Out of memory errors? Reduce batch size or upgrade to GPU with more VRAM.

View detailed GPU specifications →

Balance monitoring needs with training efficiency:

Frequent checkpoints (1-2 epochs):

• ✅ Detailed progress tracking
• ✅ More recovery points if training fails
• ✅ Better for short training runs (< 10 epochs)
• ❌ Slower training due to evaluation overhead

Infrequent checkpoints (5-10 epochs):

• ✅ Faster training with less overhead
• ✅ Better for long runs (50+ epochs)
• ❌ Less granular progress visibility
• ❌ Fewer recovery points

Configure checkpoint strategies →

Before starting training:

1. Estimate duration: Similar models with similar datasets provide baseline
2. Calculate credits: Usage Multiplier × Estimated Minutes = Total Credits
3. Add buffer: Include 20-30% buffer for unexpected training time
4. Verify budget: Ensure sufficient Compute Credits available

Example calculation:

Configuration: 4× A100 (40GB) → 16.0 credits/minute
Estimated time: 2 hours (120 minutes)
Buffer (25%): 30 minutes

Total credits needed: 16.0 × 150 = 2,400 credits

View pricing and examples →

No. Training configuration is locked when the run starts. If you need different settings:

1. Cancel the current run
2. Return to your workflow
3. Click Run Training to configure a new run with updated settings

Tip: Test configurations with short runs (few epochs) before committing to long training sessions.

Training will pause when your Compute Credits are depleted:

1. The run enters a "paused" state
2. You receive a notification
3. Training resumes automatically when credits are available
4. No progress is lost—training continues from the last checkpoint

To avoid interruptions:

• Monitor your resource usage before starting
• Purchase additional Compute Credits if needed
• Use smaller GPU configurations for longer availability

Use 1 GPU when:

• Testing new configurations or hyperparameters
• Training small models (< 3B parameters)
• Working with limited Compute Credit budget
• Dataset is small (< 1,000 images)

Use multiple GPUs when:

• Training large models (7B+ parameters)
• Working with large datasets (10,000+ images)
• Speed is critical for iteration velocity
• You have sufficient Compute Credit budget

Performance scaling: Multi-GPU training doesn't scale linearly. 4 GPUs are typically 2.5-3.5× faster than 1 GPU, not 4× faster.

Learn about distributed training →

Checkpointing:

• Occurs during training at specified intervals (every N epochs)
• Creates recovery points for interrupted training
• Generates evaluation metrics and visualizations
• Temporary until training completes

Final model saving:

• Occurs when training completes successfully
• Creates the deployable model artifact
• Includes final weights and configuration
• Permanent and available for deployment

Configure checkpoint behavior →

Maybe. GPU requirements depend on:

Model architecture size:

• 0.5-1B parameter models: T4 or L4 sufficient
• 3-7B parameter models: A10G or A100 (40GB) recommended
• 7-13B parameter models: A100 (80GB) or H100 required

Batch size:

• Larger batch sizes require more VRAM
• Reduce batch size if you encounter out-of-memory errors

Dataset complexity:

• High-resolution images need more memory
• Complex annotations increase memory usage

Rule of thumb: Start with the GPU configuration from similar successful runs. Upgrade if you encounter memory errors.

View GPU selection guide →

Training time estimates are based on historical data and may vary due to:

Factors that increase training time:

• Large batch sizes require more computation per step
• Complex model architectures have slower forward/backward passes
• Advanced Evaluation enabled adds checkpoint overhead
• High-resolution images increase processing time

Factors that decrease training time:

• Multi-GPU configurations (but not linearly)
• Smaller datasets finish epochs faster
• Disabled Advanced Evaluation reduces overhead

Tip: Track actual training time for your specific configurations to improve future estimates.

Issue: Cannot start training due to resource constraints.

Potential causes:

• Insufficient Compute Credits for selected GPU configuration
• All GPUs of selected type currently in use
• Account limits exceeded

Solutions:

1. Check Compute Credits:

   • View your resource usage
   • Calculate required credits: Usage Multiplier × Estimated Duration
   • Purchase additional credits or wait for monthly renewal

2. Try different GPU type:

   • Select an alternative GPU with similar capabilities
   • Consider using fewer GPUs (e.g., 1× instead of 4×)

3. Wait and retry:

   • GPU availability varies; retry after a few minutes
   • Schedule training during off-peak hours if possible

Issue: Validation step shows errors preventing training.

Common validation errors:

To resolve:

1. Note the specific error message from validation
2. Click Back to exit configuration
3. Fix the underlying issue in your dataset or workflow
4. Restart the configuration and validate again

Learn about dataset requirements →

Issue: Changes made in earlier steps are lost when navigating back.

Why this happens:

• Configuration wizard maintains session state
• Browser back button may cause state loss
• Session timeout resets configuration

To avoid:

• Use the Back button within the wizard, not browser back
• Complete configuration in one session
• Don't leave configuration dialog idle for extended periods

If configuration is lost:

• Your workflow settings are preserved
• Only the training run configuration needs to be redone
• Previous selections may auto-populate based on workflow defaults

Issue: Desired GPU model not available in dropdown.

Reasons for unavailability:

1. Plan restrictions: Some GPU types require specific subscription plans
2. Region limitations: Certain GPUs may be region-specific
3. Temporary unavailability: High-demand GPUs may be temporarily unavailable
4. Beta access: Cutting-edge GPUs (H200, B200) may require beta enrollment

Solutions:

• Upgrade plan: Contact sales for Enterprise plans with more GPU options
• Use available alternative: Select similar GPU with comparable performance
• Join waitlist: Request access to beta GPU types
• Check documentation: Verify GPU is generally available

Contact support for GPU access →