Software Engineer, AI Infrastructure - Palo Alto [252825]

$140,000 - $252,000 yearly

Job Description

Tesla’s AI Infrastructure team builds the software that turns GPUs and custom silicon into reliable, efficient training capacity for Autopilot, Optimus, Robotaxi, and Dojo. We own the systems layer between bare metal and researchers: cluster lifecycle, fleet automation, observability, scheduling interfaces, and the control-plane software that keeps multi-node training jobs healthy at scale.

In this role, you'll work as a software engineer in a high-performance environment, shipping production services and tooling that operate thousands of accelerated machines, not only configuring them by hand. You'll design, build, and operate software that provisions, monitors, heals, and optimizes our AI training fleets. You treat the cluster as a product: APIs, automation pipelines, reliability signals, and abstractions that make supercomputing capacity safe and fast for ML users. You’ll focus on distributed-systems and systems-software problems (hosts, accelerators, fabric, storage, schedulers), then encode fixes into software so the same failure class doesn’t require a hero page again. Scope and level scale with impact, from owning major automation surfaces to leading design across fleet-wide platforms.

The Role

  • Keep the training fleet up; Drive reliability and efficiency of GPU clusters so research and production training progress is not gated on infrastructure toil or silent degradation
  • Ship automation as product; Replace manual ops with reliable, testable software, APIs, job pipelines, operators/controllers, and libraries other engineers and on-call can depend on
  • Own observability and remediation paths; Build metrics, logging, alerting, and automated remediation (e.g. Prometheus/Grafana-based stacks) that detect bad nodes, sick GPUs, fabric/storage faults, and scheduler failures and fence or repair safely at scale
Anthony Antonucci

Qualifications

Relevant experience in Engineering & Information Technology; strong problem-solving skills; commitment to Tesla mission.

Immediate Fill

No