Project Overview
- Built and operated clusters consisting of approximately
20 clusters (Management, Workload, HPC / Dev, Staging, Prod / Kakao, KEP, Public, Closed) with up to around 2,000 GPUs.
- Managed cluster construction and operations in various environments such as development, production, public, and closed networks, including clusters with hundreds of
GPU worker nodes.
- Resolved infrastructure-related issues, contributing to stable service operations.
What did I do
Kubernetes Cluster Tasks
- On-Premise / On-Cloud Kubernetes Cluster Setup
- Constructed around
20 Kubernetes clusters using a combination of physical machines from within the company, virtual machines from the company's provided cloud, and baremetal machines.
- Utilized the tool Kubespray for direct cluster setup and management.
- This allowed for stable operation of the MLOps platform and Inference operation tools.
- Closed Network Kubernetes Cluster Setup for External Business PoC
- Conducted cluster setup in a closed network for the purpose of
Proof of Concept (PoC) for Kakao i Machine Learning MLOps platform installation.
- This enabled the successful execution of
CJ(Korean Company) PoC by configuring the Kakao i Machine Learning MLOps platform service to work smoothly in a closed network environment.
- Large-Scale Infiniband Network HPC Cluster Operation
- Operated an HPC cluster using an
Infiniband network, consisting of over 500 A100 GPUs connected by an Infiniband network, actively used for Multi-Node distributed learning.
- Configured Kubernetes to work with Infiniband and GPU at the Kubernetes level and resolved issues occurring during operations.
- This allowed users of the MLOps platform to perform large-scale distributed learning stably.
- Kubernetes Cluster Operation
- Configured and operated GPU MIG settings and utilized MIG resources.
- Performed the addition and removal of additional GPU or CPU devices from the cluster.
- Handled issues such as Disk Faults and Memory failures on Cluster Worker Nodes.
- Conducted troubleshooting for various issues arising in Kubernetes.
Migration of Existing Training Platform's GPU Devices to Kakao i Machine Learning Platform
- Migrated approximately 200 GPUs to the Kakao i Machine Learning cluster.
- Collaborated with the infrastructure team to perform
device OS reinstallations and configuration setups and joined devices to the Kubernetes cluster.
- This provided a more pleasant GPU usage environment for Kakao i Machine Learning platform users.
Relocation of High-Powered Standalone GPU Devices
- Relocated around 200 GPUs due to power issues on the existing location.
- While the physical equipment relocation was handled by the infrastructure team, I planned and led the parts related to the MLOps platform and Inference platform services.
- Planned how to divide and relocate the equipment for a seamless transition without affecting the live environment.
- Addressed concerns in
Inference service during the NAS device migration, maintaining service continuity in Kubernetes despite IP changes and unmounted mount information.
- Successfully completed large-scale equipment relocation to resolve issues without affecting services.
Equipment Categorization and Operational Tasks
- Managed hundreds of
Physical Machines and Virtual Machines actively. Additionally, conducted an overall equipment categorization for operational convenience.
- This facilitated collaboration with the infrastructure team and made it easier for team members to check equipment statuses.
Challenges
I have documented detailed Troubleshooting processes for issues encountered during service development and operations on my blog. Below are some examples: