Platform Copilot

Work the DriftHigh runbook in order [1]: 1. Open the monitoring dashboard and confirm which input features breached PSI (> 0.2). 2. Check the feature-materialization job ran on schedule and did not error. 3. Compare the current input distribution against the training baseline. 4. If features are stale or wrong, fix ingestion first — do not retrain on bad data. If it is a genuine demand shift, trigger the weekly training DAG to refresh the model and confirm the new candidate clears the promotion gate before it serves traffic [2]. If the gate keeps rejecting candidates or error stays elevated, page the platform on-call and open an incident [3].

So one scheduler can run every workload — CPU APIs, GPU LLM serving, batch and distributed training, and Airflow — which is what makes GPU cost control (priority classes, preemption, quotas) possible at all [1]. It also gives a declarative topology, so disaster recovery becomes "re-apply config", and it runs KServe and Ray natively [1]. Managed serverless GPU endpoints were considered and rejected: weak multi-tenant GPU economics, vendor lock-in, and no shared substrate for training + Airflow + serving — and the shared substrate is precisely the platform's value [2].

Placement is derived from the workload, not requested by the caller [1]. The platform maps a model's framework to a compute pool: GPU-bound runtimes (vLLM, Transformers) go to the GPU pool; everything else (LightGBM, sklearn/ONNX) goes to the CPU pool [1]. The GPU nodes carry a nvidia.com/gpu=present:NoSchedule taint, so only workloads that actually need a GPU — and therefore tolerate that taint — ever land there; a cheap CPU model can't silently occupy a $3/hr GPU node [2].

Two properties decide it [1]: PagedAttention, which nearly eliminates KV-cache fragmentation so more sequences fit per GPU, and continuous batching, which lets new requests join the running batch instead of waiting for it to drain. Together that is roughly a 5–8× throughput gain over vanilla HuggingFace generation on the same GPU. The cost model shows that throughput gain is about a 7× reduction in cost per request — so it is an economics decision, not just a latency one [2]. Vanilla HF is kept only as the benchmark baseline; TGI and TensorRT-LLM are candidate backends behind the same interface [3].