How to Set Up a Cloud GPU Instance (August 2026)

Renting a cloud GPU is easy. Getting it to actually run your workload without a driver mismatch, an out-of-memory error, or a surprise egress bill is where most people hit a wall. This guide walks you through the whole setup, from choosing a provider to running your first job.

TLDR:

• Match your GPU to your workload before provisioning: lighter GPUs like the A10G for inference, H100 for serious training runs.
• Use pre-configured AMIs with drivers bundled in to save 30-60 minutes of manual CUDA setup.
• Spot instances run 60-90% cheaper than on-demand but can be interrupted, so checkpoint your jobs.
• Data egress fees run $0.08-$0.12 per GB, so co-locate storage and compute in the same region.
• Thunder Compute Local runs GPU workloads via tnr run python train.py without managing cloud infrastructure.

choosing the right gpu cloud provider

Not every GPU cloud provider works the same way, and picking the wrong one can cost you hours of setup time or hundreds of dollars in wasted compute. The right choice depends on your workload type, budget, and how much infrastructure overhead you want to manage.

There are a few key factors worth weighing before you commit to a provider.

what to look for in a gpu cloud provider

• GPU availability: Some providers oversell capacity, especially for high-demand GPUs like the H100. Check whether the provider guarantees on-demand access or puts you in a queue.
• Pricing model: Hourly on-demand rates give you flexibility for short workloads, while reserved or spot instances lower costs for longer runs. Know which you need before comparing prices.
• Networking and storage: Interconnect speed matters for multi-GPU jobs. If you're training large models across multiple nodes, look for providers that support NVLink or InfiniBand fabrics.
• Software environment: Does the provider offer pre-configured machine images with CUDA, cuDNN, and popular frameworks already installed? Starting from a bare instance adds setup time.
• Geographic availability: Where your compute runs affects latency and, in some cases, compliance. European Azure regions, for example, run 10-25% lower than East US baseline rates.

the main provider categories

You broadly have three options: hyperscalers, specialized GPU clouds, and bare-metal rentals.

Hyperscalers like AWS, GCP, and Azure offer deep integration with managed services and strong SLAs, but their GPU pricing tends to carry a steep premium. AWS p4d.24xlarge instances, for example, run at $42.77/hr.

The cheapest GPU cloud providers focus exclusively on GPU workloads and typically undercut hyperscaler pricing by a wide margin. Thunder Compute falls into this category, with H100 access at rates well below what major cloud providers charge for equivalent hardware.

Bare-metal GPU rentals give you the most control and often the lowest per-hour cost, but they require more hands-on configuration and lack the managed tooling that speeds up deployment.

a quick comparison

For most developers and researchers running iterative ML experiments or inference workloads, a specialized GPU cloud hits the right balance between cost and convenience. Hyperscalers make more sense when your workload is tightly coupled with other managed services you already rely on.

understanding gpu instance types and specifications

Choosing the right GPU instance shapes everything that follows: cost, performance, and whether your workload even fits in memory. Before you spin anything up, it's worth understanding what the specs actually mean.

Cloud GPU instances vary across three core dimensions: compute power (measured in TFLOPS), memory capacity (VRAM), and interconnect bandwidth. Each of these affects a different class of workload. Inference jobs are often memory-bandwidth bound, training runs are compute bound, and multi-GPU workloads live or die by interconnect speed.

the main gpu tiers you'll encounter

Most cloud providers organize GPU offerings into a few recognizable tiers based on the underlying NVIDIA hardware:

• A100: The most widely available high-end GPU across major providers. Comes in 40GB and 80GB VRAM variants, with the 80GB version preferred for training large models or running multi-modal workloads. NVLink-connected multi-GPU setups are common here.
• H100 vs A100: Currently the fastest GPU available for cloud rental. Available in SXM5 and PCIe variants, with SXM5 offering higher memory bandwidth and better multi-GPU scaling via NVLink. Expect materially higher hourly rates compared to A100.

key specs to compare before provisioning

One thing to watch: providers sometimes list the same GPU model across different instance families with meaningfully different configurations. An H100 PCIe instance and an H100 SXM5 instance are not equivalent, even though both carry the H100 label. Always check the full spec sheet, beyond the GPU name alone.

Instance naming conventions also vary by provider. AWS uses families like p4d and p5, Google Cloud uses a2 and a3, and Azure uses NC and ND series. The naming tells you little about raw performance; the underlying GPU model and memory configuration are what matter.

creating and configuring your gpu instance

Once your provider account is ready, the actual instance setup takes only a few steps. The process varies slightly by provider, but the core workflow is consistent across AWS, Google Cloud, Azure, and most GPU cloud services.

choosing your gpu instance type

Before launching, you need to match the GPU to your workload. The wrong choice means either overpaying for unused capacity or hitting memory limits mid-run.

• For inference and lighter fine-tuning tasks, an NVIDIA A10G instance gives you solid throughput at a lower hourly rate. This is a good starting point if you are not running full model training.
• For serious training workloads, LLM fine-tuning, or multi-GPU jobs, you want an H100 or A100. The H100 SXM5 variant delivers meaningfully faster throughput on transformer workloads compared to its PCIe counterpart.
• For multi-GPU training, check whether the provider supports NVLink or InfiniBand interconnects. Interconnect bandwidth can be the bottleneck, not the GPU itself.

launching the instance

Most providers give you a web console, CLI, or API to spin up an instance. The CLI route is faster for repeat setups. On AWS, for example, you would use the EC2 GPU instances launch wizard or run an aws ec2 run-instances command with your chosen AMI and instance type. On Google Cloud, gcloud compute instances create handles the same job.

When configuring the instance, pay attention to:

• Region selection, since GPU availability varies by region and pricing can differ by 10-25% across geographic zones
• Storage volume size, since GPU instances often need larger root volumes to hold model weights and datasets
• Security groups or firewall rules, which should allow SSH access and any ports your training framework requires

connecting and verifying your gpu

After the instance starts, SSH into it and run nvidia-smi. This command outputs your GPU model, driver version, memory usage, and current utilization. If the GPU is not showing up, the driver may not be installed or may need a reboot to load properly.

A quick sanity check before running any workload:

nvidia-smi
python3 -c "import torch; print(torch.cuda.is_available())"

Both should return positive results before you proceed. If PyTorch reports False, your CUDA installation likely does not match the installed driver version. Reinstalling the matching CUDA toolkit resolves this in most cases.

installing nvidia drivers on your gpu instance

Once your instance is running, the first real task is getting NVIDIA drivers installed correctly. Skipping this step or rushing through it causes more failed GPU setups than almost anything else.

Most cloud providers offer GPU-optimized AMIs or images with drivers pre-installed, and you should use them when available. AWS offers the Deep Learning AMI, Google Cloud has Deep Learning VM Images, and Azure provides NC-series VM images with CUDA drivers bundled in. Starting from one of these saves you 30 to 60 minutes of manual setup and eliminates a common class of driver version conflicts.

If you are starting from a base Ubuntu image, you will need to install drivers manually.

manual driver installation on ubuntu

Before touching any driver packages, run a full system update to avoid dependency conflicts:

sudo apt-get update && sudo apt-get upgrade -y

Then install the NVIDIA driver. For most deep learning workloads on current-generation hardware, the 535 or 545 series drivers work reliably:

sudo apt-get install -y nvidia-driver-545
sudo reboot

After the instance restarts, verify the installation with:

nvidia-smi

A successful output shows your GPU model, driver version, and CUDA version. If you see "No devices found" or a command not found error, the driver did not install correctly, which usually points to a kernel mismatch or a missing dependency.

installing cuda toolkit

The CUDA toolkit is separate from the driver and required for any GPU-accelerated workloads. NVIDIA's official CUDA downloads page lets you select your OS, architecture, and preferred installation method. For Ubuntu 22.04 with a network installer:

wget https://developer.download.nvidia.com/compute/cuda/repos/ubuntu2204/x86_64/cuda-keyring_1.1-1_all.deb
sudo dpkg -i cuda-keyring_1.1-1_all.deb
sudo apt-get update
sudo apt-get install -y cuda-toolkit-12-3

Set your environment variables so the system can find CUDA binaries:

export PATH=/usr/local/cuda/bin:$PATH
export LD_LIBRARY_PATH=/usr/local/cuda/lib64:$LD_LIBRARY_PATH

Add those lines to your .bashrc or .zshrc to make them persist across sessions. Run nvcc --version to confirm the toolkit is accessible. At this point, your GPU instance is ready to run workloads.

installing and configuring cuda toolkit

The previous section walked through getting drivers installed and showed the basic CUDA toolkit commands for Ubuntu 22.04. Before running those commands, there's a decision point that trips people up: picking the right CUDA version and figuring out what to do about the driver that ships bundled with the CUDA installer.

matching cuda to your driver

NVIDIA maintains a forward-compatibility model where newer CUDA versions require a minimum driver version. If your driver is too old, CUDA will install without complaint but fail silently at runtime. Before downloading, check your installed driver version with nvidia-smi and cross-reference it against NVIDIA's CUDA compatibility table. As a rough guide:

• CUDA 12.0 requires driver 525 or newer, covering most standard workloads on older instances.
• CUDA 12.3 requires driver 545 or newer, which is common on mid-generation cloud images.
• CUDA 12.6 requires driver 560 or newer, needed for the latest framework releases.

Mismatches are common when using a pre-installed AMI that ships with an older driver. If your framework requires CUDA 12.6 but your driver only supports 12.3, update the driver before proceeding.

bundled driver: install or skip?

CUDA installers offer to install a driver alongside the toolkit. If you already have a working NVIDIA driver verified with nvidia-smi, skip the bundled driver. Installing it on top of an existing driver creates conflicts and usually requires a full driver wipe to recover.

When running the runfile-based CUDA installer, uncheck the driver option explicitly:

sudo sh cuda_12.3.0_545.23.06_linux.run --toolkit --silent --override

The --toolkit flag installs only the toolkit and leaves your existing driver untouched. For the apt-based method, the cuda-toolkit-12-3 package excludes the driver by default, so no extra flags are needed.

installing cudnn

Most deep learning frameworks need cuDNN on top of the CUDA toolkit. After CUDA is installed, download the cuDNN package matching your CUDA version from NVIDIA's developer portal (an account is required). For CUDA 12.x on Ubuntu:

sudo apt-get install -y libcudnn9-cuda-12

Confirm cuDNN is accessible by checking its version file:

cat /usr/include/cudnn_version.h | grep CUDNN_MAJOR -A 2

A clean output here means your full CUDA stack is in place: driver, toolkit, and cuDNN. At that point, frameworks like PyTorch or JAX can access GPU acceleration without any additional configuration.

setting up your development environment

With your CUDA stack confirmed, the next layer is the actual software you'll run workloads in. Choices about frameworks, notebooks, and containerization all affect how smoothly your GPU workload runs in practice.

installing pytorch and tensorflow

PyTorch is the default choice for most ML research and production fine-tuning work. Installing it via pip with the correct CUDA index URL is the safest way to get a GPU-compatible build:

pip install torch torchvision torchaudio --index-url https://download.pytorch.org/whl/cu121

Swap cu121 for your CUDA version (e.g., cu118 for CUDA 11.8). After installing, verify GPU access with:

python3 -c "import torch; print(torch.cuda.get_device_name(0))"

TensorFlow handles installation slightly differently. Google ships GPU support in the base package for TensorFlow 2.x:

pip install tensorflow[and-cuda]

This pulls in the required CUDA and cuDNN libraries as Python packages, which is useful on instances where you want to avoid system-level CUDA management. One trade-off: the packaged CUDA versions lag slightly behind official NVIDIA releases.

configuring jupyter for remote access

Running Jupyter on a remote GPU instance requires a small amount of configuration to make it accessible from your local browser. Start by installing Jupyter and setting a password:

pip install jupyterlab
jupyter lab --generate-config
jupyter lab password

Then launch Jupyter without opening a browser, bound to all interfaces:

jupyter lab --no-browser --ip=0.0.0.0 --port=8888

On your local machine, forward the port over SSH:

ssh -L 8888:localhost:8888 user@your-instance-ip

Open localhost:8888 in your browser and enter the password you set. For persistent sessions that survive disconnects, wrap the Jupyter process in tmux or screen before launching.

using docker for reproducibility

Docker is worth the setup cost when you need your environment to be reproducible across runs or shareable with teammates. NVIDIA's Container Toolkit lets Docker containers access the host GPU directly:

sudo apt-get install -y nvidia-container-toolkit
sudo systemctl restart docker

After that, pull a pre-built image from NVIDIA's NGC registry, which ships with CUDA, cuDNN, and framework versions already tested together:

docker run --gpus all nvcr.io/nvidia/pytorch:24.01-py3

This approach also makes moving workloads between instances much cleaner, since the container carries its own dependency graph instead of relying on what is installed at the system level.

optimizing costs and managing gpu resources

Cloud GPU costs can spiral fast if you're not paying attention. Pricing models vary across providers, instance types, and regions, and a few misconfigured or idle instances can quietly drain your budget over a billing cycle.

spot vs. on-demand instances

Most major providers offer two core pricing tiers. On-demand instances give you guaranteed availability at a fixed hourly rate. Cloud GPU spot instances let you bid on unused capacity at steep discounts, often 60 to 90% cheaper, but they can be interrupted with little warning.

For fault-tolerant workloads like distributed training jobs with checkpointing, spot instances are a practical way to cut costs. For inference serving or time-sensitive jobs where interruption is unacceptable, on-demand is the safer choice.

right-sizing your instance

One of the most common sources of wasted spend is over-provisioning. Running a multi-GPU instance for a workload that fits on a single GPU doubles or quadruples your hourly cost with no performance gain.

Before committing to an instance size, profile your workload:

• Run your job on a smaller instance first and check GPU memory utilization. If you're using less than 70% of available VRAM, consider ways to cut cloud GPU costs during prototyping.
• Use provider monitoring tools like AWS CloudWatch, GCP Cloud Monitoring, or Azure Monitor to track GPU utilization over time and catch consistently underused instances.
• For batch inference, consider whether a smaller, cheaper GPU can meet your latency requirements before defaulting to the largest available option.

storage and data transfer costs

Compute costs get most of the attention, but storage and egress fees add up. Keeping large datasets on high-performance attached storage when they're only accessed once per training run is a common oversight.

A few practical habits help here:

• Use object storage (S3, GCS, Azure Blob) for datasets and checkpoints between runs, and mount or copy data to local instance storage only when a job is actively running.
• Be aware that egress fees on cloud GPUs typically cost between $0.08 and $0.12 per GB on major providers. If you're moving large model artifacts or datasets frequently, those fees compound.
• Co-locate your storage and compute in the same region to avoid cross-region transfer charges entirely.

autoscaling and job scheduling

If you're running recurring training jobs or serving variable inference traffic, autoscaling can prevent you from paying for idle GPU time. Most cloud providers offer managed autoscaling groups or Kubernetes-based GPU node pools that spin resources up and down based on demand. Pairing autoscaling with a job scheduler like Ray or Slurm gives you tighter control over when GPUs are actually allocated versus sitting idle.

troubleshooting common setup issues

Even with careful setup, GPU cloud environments have a handful of failure modes that show up repeatedly. Most are fixable in minutes once you know what to look for.

cuda and driver version mismatches

This is the most common failure after a fresh setup. The symptom is usually PyTorch reporting False for CUDA availability, or a runtime error mentioning a CUDA version the driver does not support. Run nvidia-smi and check the "CUDA Version" field in the top right corner. That number reflects the maximum CUDA version your current driver supports. If your installed toolkit exceeds it, you need to either downgrade the toolkit or update the driver. NVIDIA's CUDA compatibility table maps driver versions to supported CUDA releases, and cross-referencing those two numbers resolves most mismatches.

out-of-memory errors

OOM errors during training or inference usually mean one of three things: your model is too large for the available VRAM, your batch size is too high, or memory fragmentation is accumulating across iterations.

• Reduce batch size first. Halving it often resolves OOM errors without touching the model architecture.
• Use gradient checkpointing if you are training. It trades compute for memory by recomputing activations on the backward pass instead of storing them.
• Call torch.cuda.empty_cache() between inference calls if you see memory grow over time. It releases cached allocations the framework is holding onto, though it does not free memory held by live tensors.
• Switch to mixed-precision training with torch.cuda.amp.autocast() if your workload tolerates fp16. It roughly halves VRAM usage for most transformer models.

networking and firewall issues

If you can SSH into your instance but cannot reach Jupyter or a model serving endpoint, a firewall rule is likely blocking the port. Check your security group (AWS), VPC firewall (GCP), or network security group (Azure) and confirm the relevant port is open to your IP. For Jupyter, that is typically port 8888. SSH port forwarding sidesteps this entirely for local browser access, which is worth using when you do not need a publicly exposed endpoint.

quota limits blocking instance launches

GPU instance quotas are set at the account level and are often zero by default for high-end hardware. If your launch fails with a quota or limit error, submit a quota increase request through your provider's console before you need the capacity. Approval can take anywhere from a few hours to a couple of days on major providers, so request headroom before a deadline, not during one.

set up gpu instances with thunder compute local

Thunder Compute Local gives you GPU access without the overhead of managing cloud infrastructure yourself. You get a dedicated GPU instance that behaves like a local machine, so your existing tools, workflows, and scripts work out of the box.

Getting started takes about five minutes. Here's the setup process:

• Install the Thunder Compute CLI with a single pip command: pip install thunder-compute
• Authenticate with your account credentials using tnr login
• Spin up a GPU instance by running tnr run followed by your usual command, such as tnr run python train.py
• Your code executes on a remote GPU, and output streams back to your terminal as if it were running locally

supported gpus and pricing

Thunder Compute supports a range of NVIDIA GPUs, including the H100, which covers most deep learning training and inference workloads. Pricing is pay-as-you-go with per-second billing, so you only pay for the compute you actually use.

when thunder compute local fits best

This setup works well in specific situations:

• You're iterating quickly on model code and want to test on real GPU hardware without committing to a long-running instance
• Your local machine lacks a GPU entirely and you need a frictionless way to run GPU-dependent scripts
• You want to avoid configuring CUDA drivers, virtual environments, or cloud console dashboards just to run a training job
• You're working with existing codebases where rewriting for a new API isn't practical

The key distinction from traditional cloud GPU providers is the local-first interface. You keep your development environment intact, and Thunder Compute handles instance provisioning, environment replication, and teardown automatically. There's no separate SSH session to manage or remote filesystem to sync.

final thoughts on setting up a gpu in the cloud

Most GPU setup headaches come down to version mismatches, over-provisioned instances, or idle compute you forgot to stop. Knowing what to check before you launch puts you ahead of most of those issues. From there, your focus can stay on the actual work. Check out Thunder Compute if you want to run GPU workloads without managing the infrastructure yourself.

FAQ

what's the fastest way to set up a gpu in the cloud without configuring cuda drivers yourself?

Use a provider that ships pre-built GPU images with CUDA, cuDNN, and framework dependencies already installed. AWS Deep Learning AMIs, Google Cloud Deep Learning VM Images, and Thunder Compute's pre-configured templates all skip 30 to 60 minutes of manual driver work and eliminate the most common class of version conflicts.

hyperscaler gpu cloud vs specialized gpu cloud: which should you use for ml training?

Specialized GPU clouds hit the better price-to-performance ratio for most ML training and inference workloads. AWS p4d.24xlarge instances run at $42.77/hr, while providers like Thunder Compute offer H100 access at meaningfully lower on-demand rates. Hyperscalers make more sense when your training job is tightly coupled to managed services you already depend on, like RDS or BigQuery.

how do i verify my gpu is working correctly after setting up a cloud instance?

Run nvidia-smi immediately after connecting to confirm your GPU model, driver version, and memory are visible. Follow that with python3 -c "import torch; print(torch.cuda.is_available())" to confirm your framework can access the GPU. If PyTorch returns False, your installed CUDA toolkit version likely exceeds what your driver supports, which you can check by cross-referencing the CUDA Version field in nvidia-smi output against NVIDIA's compatibility table.

spot vs on-demand gpu instances: when does each pricing model make sense?

Spot instances cut costs by 60 to 90% but can be interrupted mid-run, so they work best for distributed training jobs that checkpoint regularly. On-demand instances are the right call for inference serving, time-sensitive jobs, or any workload where an unexpected interruption causes real damage. If you're iterating quickly on model code, on-demand with per-second billing keeps costs predictable without the interruption risk.

can i run gpu workloads in the cloud without managing ssh sessions or remote filesystems?

Yes. Thunder Compute Local lets you run GPU workloads directly from your terminal with tnr run python train.py, streaming output back as if the job were running locally. Instance provisioning, environment replication, and teardown happen automatically, so there's no separate SSH session to manage or remote filesystem to sync.