Kill a Run

Before deciding to kill a run, verify it's actually problematic:

1. Monitor progress for at least 5-10 minutes
2. Check if stages advancing through preprocessing and setup
3. Review logs for actual errors vs. normal warnings
4. Wait for first metrics to appear if training just started

Don't kill prematurely:

• Initial stages can take several minutes
• Some warnings are normal and don't indicate failure
• First epoch often slower than subsequent epochs

Do kill immediately if:

• Clear configuration errors visible
• Out of memory errors appearing
• Training stuck at same point for 15+ minutes
• Wrong workflow started accidentally

Keep notes on killed runs to avoid repeating mistakes:

Record:

• What went wrong (configuration error, resource issue, etc.)
• Why you killed it instead of letting it complete
• What you changed before restarting
• Lessons learned for future training

Benefits:

• Avoid repeating same configuration errors
• Track patterns in training issues
• Share knowledge with team members
• Improve workflow configurations over time

Don't immediately restart after killing—fix underlying issues first:

Configuration problems:

1. Update workflow settings with corrections
2. Test configuration with shorter training if possible
3. Verify dataset is correct and accessible
4. Review model compatibility with your task

Resource problems:

1. Reduce batch size if out of memory
2. Select smaller model if architecture too large
3. Resize images in dataset if too high resolution
4. Check credit availability before long training

Validation:

• Test with minimal epochs (e.g., 5 epochs) to validate fixes
• Scale up gradually after confirming setup works
• Monitor closely during initial restart

Killing is useful for rapid workflow testing:

Strategy:

1. Start training with test configuration
2. Monitor first few minutes to verify setup correct
3. Kill after validation (don't need full training)
4. Iterate quickly on configurations

Good for:

• Testing new dataset preparations
• Validating system prompt formats
• Checking model architecture compatibility
• Confirming training parameters don't cause errors

Not good for:

• Actual model training (obviously)
• Generating metrics for evaluation
• Producing deployable models

Make informed decisions about whether to kill or continue:

Questions to ask:

1. How much training time elapsed?

   • Early (<10 min): Minimal credits lost if kill
   • Mid-training (30+ min): More credits invested
   • Late (>80% complete): Consider letting finish

2. Is training clearly failing?

   • Yes: Kill immediately regardless of time
   • Maybe: Review logs and metrics to decide
   • No: Let complete if making good progress

3. Can you fix issue mid-training?

   • No: Kill and restart with fixes
   • Yes but requires restart: Kill and fix
   • Issue resolves itself: Continue training

General principle: Kill early when certain of problems, but let complete if training progressing acceptably even if not optimal.

Learn about credit management →

No. Killed runs cannot be resumed or restarted from their stopped point.

Your options after killing:

1. Start a new run — Begins fresh training from scratch
2. Use same workflow — Reuses configuration from killed run
3. Modify workflow — Update settings before starting new run

Training always starts from beginning:

• No checkpoint resumption from killed runs
• Each run is independent training session
• Progress from killed run cannot be continued

Workaround for long training:

• Configure workflows with shorter epoch counts
• Train incrementally with multiple runs
• Save intermediate results after each run

No. Compute Credits are consumed based on actual GPU time used, regardless of whether training completes.

What you're charged for:

• GPU time from run start until kill
• Validation and metric computation before kill
• Resource allocation and setup overhead

What you're not charged for:

• Remaining estimated training time not executed
• Queue time waiting for GPU resources
• Time after kill action completes

Why no refunds:

• GPU resources were allocated and used
• Training computations were actually executed
• Infrastructure costs incurred for running time

Credit savings:

• Killing early saves remaining training time credits
• Early detection of issues minimizes wasted credits
• Quick kills (<5 min) minimize lost credits

Unfortunately, kills cannot be undone. Once you confirm the kill action:

• Training stops immediately and permanently
• Run cannot be resumed or restarted
• GPU resources are released

If you killed the wrong run:

1. Start a new run immediately with same configuration
2. Use same workflow to replicate settings exactly
3. Monitor closely to ensure correct run this time
4. Delete killed run after starting replacement

Prevention tips:

• Double-check run name in kill confirmation dialog
• Verify run status before killing (ensure it's the problematic one)
• Review run start time to confirm identity
• Read confirmation message carefully before clicking Confirm

In the future:

• Use descriptive workflow names for easy identification
• Review run details before accessing kill option
• Take a moment to verify before confirming kill

Currently, runs must be killed one at a time through the interface. There is no bulk kill functionality.

To kill multiple runs:

1. Navigate to the Runs tab
2. For each run you want to stop:
   • Click the three-dot menu (⋮)
   • Select Kill Run
   • Confirm the action
3. Repeat for each active run

Use cases for killing multiple runs:

• Wrong workflow configurations applied to multiple runs
• Need to free all GPU resources immediately
• Testing scenarios where all test runs should be stopped

Efficiency tip: Start from newest to oldest runs to ensure most recent (likely higher priority) runs are addressed first.

Killing is nearly instantaneous in most cases:

Typical sequence:

1. You click Confirm in kill dialog
2. Platform sends stop signal to training process (< 1 second)
3. GPU processing halts immediately
4. Status updates to "Killed" (< 5 seconds)
5. Resources freed for other uses

Possible delays:

• Training in middle of saving checkpoint: Wait for save to complete (< 30 seconds)
• Validation cycle in progress: Allow validation to finish (< 1 minute)
• Infrastructure communication delays: Rare, usually resolve within 1-2 minutes

If kill seems stuck:

• Refresh the page to see updated status
• Check the Logs tab for kill confirmation messages
• Wait 2-3 minutes for infrastructure to process
• Contact support if run still shows "Running" after 5 minutes

Resource availability:

• GPU resources usually available for new runs within 1-2 minutes after kill
• Queued runs start automatically once resources free up

No. Killing one run does not affect other active training runs:

Each run is isolated:

• Runs use dedicated GPU allocations
• Training processes are independent
• Killing one run doesn't impact others

Benefits:

• Kill problematic runs without disrupting successful training
• Free resources for specific high-priority runs
• Test configurations independently

Resource reallocation:

• GPU freed from killed run becomes available
• Queued runs may start using freed resources
• Other active runs continue unaffected

Project organization:

• Runs in different workflows completely independent
• Runs in same workflow don't depend on each other
• Project structure unaffected by kills

Kill immediately if:

• ❌ Configuration is clearly wrong (can't produce useful results)
• ❌ Out of memory errors occurring (won't complete anyway)
• ❌ Loss showing NaN or infinite values (training broken)
• ❌ You need the GPU resources for something else urgent

Let complete if:

• ✅ Training progressing slowly but correctly
• ✅ Metrics improving even if suboptimal
• ✅ Near completion (>80% done) with acceptable progress
• ✅ Want to analyze results before deciding next steps

Consider carefully if:

• ⚠️ Mid-training with significant credits invested
• ⚠️ Training isn't optimal but producing some results
• ⚠️ Unsure if issue is temporary or permanent
• ⚠️ Want to see what metrics look like at completion

Decision framework:

1. Review logs for error severity
2. Check metrics for improvement trends
3. Calculate credits consumed vs. remaining
4. Assess if results would be usable even if suboptimal

General principle: Kill early and decisively when certain of failure; let complete if there's value in seeing final results.