Network and Compute Engineer, AI Infrastructure
60 – 65 $в час
Tekfortune Inc.San Jose и ещё 1 городУдалённоот 7 летКонтракт
PythonAIKubernetesLinuxMachine LearningBashAnsibleinfrastructureArtificial IntelligenceCloud ComputingNVIDIAComputer Networks
Описание
This role involves designing, deploying, validating, and operating high-performance infrastructure for enterprise artificial intelligence and accelerated-computing workloads.
Responsibilities
- Deploy, configure, validate, and operate accelerated-compute clusters
- Perform server and cluster bring-up, including operating-system imaging, boot validation, firmware alignment, hardware discovery, and post-deployment health checks
- Support NVIDIA MGX-based infrastructure and understand the relationship between hardware architecture, firmware, operating systems, drivers, and the platform software stack
- Install and administer Linux-based compute environments
- Support Kubernetes installation, configuration, upgrades, and lifecycle operations
- Deploy and administer Slurm-based workload environments
- Assist with NVIDIA GB200 firmware updates, software-stack validation, sanity checks, and platform readiness testing
- Support NVLink configuration and partitioning, where applicable
- Troubleshoot GPU and system errors, including XID events, degraded nodes, boot failures, firmware inconsistencies, and workload-performance issues
- Develop standardized, repeatable build and validation procedures for compute nodes and clusters
- Design and support resilient network connectivity for accelerated-compute and AI cluster environments
- Configure and troubleshoot high-speed Ethernet and, where applicable, InfiniBand fabrics
- Support leaf-spine network architectures, network segmentation, routing, switching, addressing, name resolution, and time synchronization
- Diagnose packet loss, latency, congestion, interface errors, cabling faults, routing issues, and workload connectivity failures
- Validate host-to-host, node-to-storage, management-plane, and workload-plane connectivity
- Support network configuration required for Kubernetes, cluster provisioning, telemetry, and distributed workloads
- Work with security teams to implement network isolation, firewall rules, access controls, and secure administrative paths
- Maintain accurate network diagrams, port maps, addressing plans, and configuration records
- Install, configure, and operate NVIDIA Base Command Manager
- Support rack import, rack view, base view, node provisioning, and cluster lifecycle activities
- Work with bare-metal management systems for server discovery, power operations, hardware health, firmware administration, and remote troubleshooting
- Integrate cluster platforms with enterprise directory services
- Support NVIDIA Mission Control processes and related AI Factory operational practices
- Assist with the deployment and operation of Run:ai or equivalent workload-orchestration capabilities
- Automate repetitive provisioning, validation, configuration, and reporting activities
- Implement infrastructure monitoring for compute nodes, GPUs, network devices, interfaces, firmware, thermals, power, storage dependencies, and cluster services
- Establish actionable dashboards, telemetry, alerts, and operational health checks
- Lead technical diagnosis across hardware, firmware, Linux, Kubernetes, scheduler, network, and application layers
- Perform root-cause analysis and document corrective and preventive actions
- Develop operational runbooks for failure scenarios such as node loss, degraded network links, GPU errors, provisioning failures, and cluster-service disruption
- Participate in incident response, problem management, capacity planning, lifecycle planning, and controlled change implementation
- Identify automation and standardization opportunities that reduce manual effort, deployment defects, and recovery time
- Work with infrastructure, cloud, security, storage, application, vendor, and service-management teams
- Apply least-privilege access to management, compute, network, and orchestration platforms
- Integrate administrative access with approved enterprise identity and directory services
- Secure management interfaces, service accounts, automation credentials, application programming interfaces, and remote-access pathways
- Maintain separation between management, workload, storage, and out-of-band management networks
- Support firmware, operating-system, container, Kubernetes, and network vulnerability remediation
- Ensure administrative actions and infrastructure changes are traceable through logging and change records
- Protect sensitive cluster configuration, credentials, certificates, logs, and telemetry
- Follow secure configuration standards, change controls, and applicable customer or regulatory requirements
Requirements
- Strong administration and troubleshooting experience with Linux
- Strong knowledge of data-center networking, including Ethernet, Transmission Control Protocol/Internet Protocol, routing, switching, virtual local area networks, link aggregation, Domain Name System, Dynamic Host Configuration Protocol, and Network Time Protocol
- Hands-on experience installing or operating Kubernetes clusters
- Experience deploying and supporting physical compute infrastructure
- Understanding of server boot processes, operating-system imaging, firmware, device drivers, hardware management, and remote management controllers
- Experience diagnosing issues across server, operating-system, network, and application layers
- Working knowledge of Python, Bash, Ansible, or equivalent infrastructure-automation technologies
- Experience with monitoring, logging, metrics, dashboards, alerting, and telemetry
- Ability to create clear architecture diagrams, build documents, operating procedures, and troubleshooting runbooks
- Strong analytical skills and the ability to work effectively during complex infrastructure incidents
- Bachelor
- Recommended 7 or more years of experience in infrastructure, compute, network, data-center, cloud-platform, or high-performance-computing engineering
- Recommended 3 or more years of hands-on experience with Linux-based cluster platforms, Kubernetes, accelerated computing, or comparable large-scale infrastructure
- Demonstrated experience owning complex technical problems from initial diagnosis through resolution and prevention
Nice to have
- Experience with NVIDIA AI infrastructure, NVIDIA Mission Control, or NVIDIA Base Command Manager
- Experience with NVIDIA MGX or GB200-based systems
- Experience with Slurm and distributed-compute scheduling
- Experience with Run:ai
- Experience with InfiniBand, Remote Direct Memory Access, or Remote Direct Memory Access over Converged Ethernet
- Knowledge of NVLink and multi-GPU communication
- Experience interpreting NVIDIA XID errors and GPU telemetry
- Experience with bare-metal provisioning and Redfish-compatible management interfaces
- Experience with Prometheus, Grafana, OpenTelemetry, or equivalent observability platforms
- Exposure to container runtimes, container registries, Kubernetes networking, and graphics-processing-unit operators
- Experience supporting production AI, machine-learning, high-performance-computing, or large-scale data platforms
- NVIDIA Certified Associate or NVIDIA Certified Professional credentials relevant to AI infrastructure
- Red Hat Certified System Administrator or Red Hat Certified Engineer
- Certified Kubernetes Administrator
- Cisco Certified Network Professional, Data Center or Enterprise
- NVIDIA networking or InfiniBand training
- Linux Foundation certifications
- Security-focused credentials appropriate to the operating environment
We offer
- Client is not willing to sponsor any visa
- Contract W2
- Contract Corp To Corp
- 12 Months
- Travel Required
- $60 - $65/hr
Hiring process
- Log in or create an account to see job match score