Outpost VFX Accelerates AI Model Training with Multi-GPU AWS, Cutting Iteration Time and Client Delivery Schedule

Outpost VFX reduced AI face replacement model training time by up to 8x by migrating from single RTX 3090 GPUs to multi-GPU AWS EC2 P5 instances with NVIDIA H100 GPUs and PyTorch DDP, cutting client delivery from weeks to days.

What happened

Outpost VFX, a visual effects studio with locations in the UK, Canada, and India, had been training face swap models on single RTX 3090 GPUs. Each fine-tune took 1-2 weeks, creating bottlenecks in production timelines. The team collaborated with the AWS Generative AI Innovation Center to adapt their codebase for distributed training.

Over a six-week advisory period, AWS scientists converted the model to use PyTorch Distributed Data Parallel (DDP). This strategy copies model weights to each GPU, allowing the system to process more images per batch. The team ran training on EC2 P5 instances with NVIDIA H100 GPUs and NVLink, which provide significantly higher bandwidth for gradient synchronization compared to PCIe-based G-series instances.

The result: up to 8x faster training speeds. The baseline of 1-2 weeks per fine-tune on a single G5 instance dropped to days on P5 instances. Most importantly, v001 delivery to clients for initial review now takes 2 days, compared to the previous 1-2 week timeline.

Why AI builders should care

This case study demonstrates a repeatable pattern for teams stuck on single-GPU training. Moving to distributed multi-GPU training on cloud infrastructure can dramatically reduce iteration cycles, which is critical for any GPU-intensive model workflow.

The key enablers were:

• Higher VRAM: H100 GPUs offer 80GB of HBM3 memory vs. 24GB on RTX 3090, allowing larger batch sizes and higher-resolution inputs.
• Faster gradient sync: NVLink interconnects on P5 instances provide much higher bandwidth than PCIe, reducing communication overhead during distributed training.
• Managed parallelization: PyTorch DDP handled weight replication and gradient averaging across GPUs with minimal code changes.

For AI builders, the lesson is that a targeted migration from consumer GPUs to enterprise cloud GPUs, combined with a distributed training strategy, can unlock order-of-magnitude speedups without rewriting the entire model.

Practical implications

If you are considering a similar migration, here are actionable steps based on Outpost VFX's experience:

1. Audit your training bottleneck: Identify whether single-GPU VRAM or compute is the limiting factor. If you are waiting days or weeks for fine-tunes, distributed training is likely worth the investment.
2. Choose instances designed for distributed training: Look for GPUs with high-bandwidth interconnects (NVLink, NVSwitch) rather than PCIe-based setups. AWS P5 instances with H100 GPUs are one option.
3. Adopt PyTorch DDP or similar: DDP is well-supported and requires relatively small code changes. The AWS team converted Outpost VFX's codebase in a six-week advisory period.
4. Plan for security and integration: Outpost VFX ran training in a segregated, secure cloud environment that aligned with their existing AWS infrastructure. Plan your network and data policies upfront.
5. Consider future scaling: Higher-resolution outputs and newer instance generations are natural next steps. Outpost VFX sees potential in using Amazon SageMaker AI for managed training and model versioning.

Caveats

This is a single case study from an AWS blog post, so results may vary depending on model architecture, dataset size, and existing infrastructure. The reported 8x speedup was measured against a specific baseline (single GPU on a G5 instance) and may not generalize to all workloads.

The advisory period with AWS scientists was a dedicated engagement; teams without similar support may need more time to adapt their codebases. Future improvements like higher-resolution outputs and newer P5 generations are speculative and depend on cost and availability.

Finally, the cost of P5 instances is significantly higher than consumer GPUs. Teams should evaluate whether the speedup justifies the increased compute spend for their specific use case.

FAQs

How did AWS help Outpost VFX accelerate AI model training for visual effects?

AWS provided multi-GPU EC2 P5 instances with NVIDIA H100 GPUs and NVLink to enable distributed training. The AWS Generative AI Innovation Center collaborated to adapt the model code for PyTorch Distributed Data Parallel (DDP) training, allowing faster training times and shorter client review cycles.

What hardware and services were used (NVIDIA H100, EC2 P5) to train the model?

Outpost VFX used NVIDIA H100 GPUs on EC2 P5 instances with NVLink interconnects for distributed multi-GPU training. The environment was configured to align with Outpost VFX's security and AWS-based infrastructure.

What is PyTorch Distributed Data Parallel (DDP) and how was it applied?

PyTorch Distributed Data Parallel (DDP) is a parallelization technique that copies model weights to each GPU, enabling larger effective batch sizes and parallel processing. Outpost VFX's model codebase was converted to use PyTorch DDP during a six-week advisory period with AWS scientists.

What production improvements resulted from the AWS-enabled training (time savings, higher resolution outputs)?

Training time decreased from weeks on single-GPU to days on multi-GPU P5 instances. Direct client review delivery (v001) reduced to about 2 days from the previous 1-2 weeks. Output quality improved with the ability to handle higher-resolution images and larger datasets.