Manage Runs

When training with new model architectures, datasets, or system prompts:

1. Stay on the monitoring page during initial setup (first 5-10 minutes)
2. Watch training progress through preparation stages
3. Check metrics appear after first epoch completes
4. Review loss curves for expected patterns
5. Verify estimated completion time aligns with expectations

This helps catch configuration errors early and prevents wasting Compute Credits on failing runs.

If you notice problems during monitoring:

• Configuration errors — Wrong dataset split, incorrect model size
• Unexpected metrics — Loss not decreasing, NaN values appearing
• Resource issues — Out of memory errors, extremely slow progress
• Wrong workflow — Accidentally started wrong training configuration

Kill immediately to preserve Compute Credits rather than letting failed training run to completion.

Before deleting runs:

1. Identify successful runs with acceptable metrics
2. Download trained models you want to keep
3. Export evaluation results if needed for documentation
4. Save training configurations for reproducibility
5. Then delete runs to keep project organized

Remember: Deletion is permanent—models cannot be recovered after run deletion.

Maintain organized training projects:

• Weekly: Delete obvious test runs and failed experiments
• After milestones: Clean up superseded runs after successful training
• Before major work: Archive or delete old runs to reduce clutter
• Keep history: Preserve successful runs that produced deployed models

Regular cleanup makes it easier to find important runs and understand project history.

Workflow names appear in run listings, so use clear, descriptive names:

Good examples:

• PCB-Detection-Qwen-Small
• Defect-Classification-NVLM-Baseline
• Quality-VQA-InternVL-Production

Poor examples:

• Workflow 1
• Test
• New Training

Descriptive names make it easier to identify runs in listings and understand project history.

When runs fail, always:

1. View detailed logs to understand failure reason
2. Check error messages in the training progress section
3. Review configuration settings that might cause issues
4. Fix underlying problems before starting new runs
5. Delete failed runs after documenting issues

Understanding failures prevents repeating configuration mistakes.

The number of simultaneous active runs depends on your organization plan and available Compute Credits.

Typical limits:

• GPU availability — Limited by total GPU resources in your plan
• Concurrent training — Usually 1-5 active runs depending on plan tier
• Queued runs — No limit; runs wait for GPU resources to become available

Runs queue automatically when GPU resources are unavailable. They start when resources free up from completed runs.

View your resource usage →

No. Training runs cannot be paused and resumed. Your options are:

• Let training complete — Continue to successful completion
• Kill the run — Stop permanently (cannot resume)
• Start a new run — Create fresh training session with same or modified settings

Workaround for long training:

If you need to stop and continue later:

1. Configure workflows with shorter epoch counts
2. Train incrementally across multiple runs
3. Use checkpointing features (if available for your model architecture)

Compute Credits are consumed for actual GPU time used, regardless of whether training completes:

• Credits consumed: Time from run start until killed
• Credits saved: Remaining training time that would have been used
• No refunds: Credits for elapsed time cannot be recovered

Example:

• Training estimated: 60 minutes (10 credits)
• Killed after: 15 minutes
• Credits consumed: ~2.5 credits
• Credits saved: ~7.5 credits

Best practice: Monitor runs closely during initial stages to catch issues early and minimize wasted credits.

Learn about resource usage →

This safeguard prevents accidental data loss and ensures training completes cleanly:

• Prevents data corruption — Ensures training logs and metrics are properly saved
• Avoids wasted resources — Forces intentional decision to stop training
• Maintains data integrity — Training must reach a stopped state before removal

To delete an active run:

1. Kill the run first (stops training immediately)
2. Wait for status to change to "Killed"
3. Delete the run after it's stopped

The two-step process ensures you intentionally stop training before permanent deletion.

No. Training runs cannot be restarted. Instead:

For failed runs:

1. Review logs to identify failure cause
2. Fix configuration issues (dataset, model settings, prompts)
3. Start a new run from the corrected workflow

For killed runs:

1. Decide if you need to continue training
2. If yes, start a new run with same or modified settings
3. Delete the old run to clean up

Training state is not preserved between runs. Each run starts fresh from your configured workflow settings.

Training logs and metrics are stored permanently until you delete the run:

• Active runs: Logs and metrics update in real-time
• Completed runs: All data remains accessible indefinitely
• Failed runs: Error logs and partial metrics remain available
• Killed runs: Progress up to kill point is preserved

Storage includes:

• Training progress and status history
• Loss curves and evaluation metrics
• Detailed logs and error messages
• Resource usage data

After deletion: All run data is permanently removed and cannot be recovered.

Best practice: Download important models and export metrics before deleting runs.

Training runs inherit their names from their parent workflow, and individual runs cannot be renamed.

To improve run identification:

1. Rename the workflow — Updates future run names
2. Use descriptive workflow names — Helps identify runs in listings
3. Add run notes — Document run purpose or configuration changes (if available)

Run identification:

• Runs display workflow name plus session ID
• Started time helps distinguish multiple runs from same workflow
• Status and metrics provide additional context