Agent

Documentation Index

Fetch the complete documentation index at: https://docs.tensorpool.dev/llms.txt

Use this file to discover all available pages before exploring further.

​

Target Failures

• GPU hardware faults: Xid errors (79, 63, 48, etc.)
• Distributed communication failures, NCCL errors
• Infrastructure problems: hardware failures, kernel panics
• Storage problems: I/O errors, checkpoint corruption, S3 timeouts
• Network problems: mounted object storage bucket issues
• GPU memory problems: CUDA out of memory, memory leaks, gradient explosion

​

How It Works

1. Registration: Provide credentials to access your job scheduler of choice (Slurm, K8s, or TensorPool Jobs) on the TensorPool Agent dashboard. Whitelist permissions you allow the agent to take on your behalf.
2. Monitoring: The training job is continuously monitored for failure.
3. Recovery (if job fails): The TensorPool Agent analyzes logs, attempts to diagnose and fix the issue. The job enters a recovering state.
4. Resolution: If recovery succeeds, monitoring resumes. You’re alerted about the failure, actions taken, and recovery status. If the TensorPool Agent lacks permissions, it provides a list of actions it attempted and would have tried.

​

TensorPool Agent Status Lifecycle

​

Failure Detection

​

Setup Requirements

apiVersion: v1
kind: Config
clusters:
  - name: my-cluster
    cluster:
      server: https://...
contexts:
  - name: my-context
    context:
      cluster: my-cluster
      user: my-user
users:
  - name: my-user
    user:
      token: ...

apiVersion: apps/v1
kind: Job  # Supported: Job, Deployment, StatefulSet, DaemonSet, Pod, PyTorchJob, TFJob, MPIJob
metadata:
  name: my-training-job  # generateName is NOT supported
  namespace: default
spec:
  # ...

​

Next Steps

• Set up the TensorPool Agent on the dashboard
• Learn about TensorPool Jobs for running training workloads