Case 02 Shipped

Reliability for multi‑region model serving

A dual-registry image path forced by a platform limit, a deploy path for eight regions with no lost version commits, a recovery design with explicit targets, and the alerting chain behind it.

Context
AI platform of a global mobility company, 2026
My part
I extended the in-house Python SDK for SageMaker serving, designed its dual-registry image path, fixed a multi-region deploy race, wrote the recovery design and built the SLO alerting chain.
Team
A small platform team on two clouds.
Status
SDK, deploy and alerting work shipped. Recovery figures are design targets.

Stack: AWS SageMaker · Python SDK · MLflow · Unity Catalog · Harbor · Amazon ECR · OIDC · Lambda · SNS · CloudWatch · Grafana · Kafka

  • 8 serving regions targeted by the hardened deploy path Shipped
  • p99 91 ms reproduced real-time feature path, 99.4% of rows inside 500 ms Measured
  • 1 h recovery time objective in the DR design Target
On this page

Context

The AI platform serves models from SageMaker. The company’s in-house Python SDK takes a model from MLflow to a running endpoint, and one release targets eight serving regions.

Problem

Three problems sat on the serving path:

  1. Asynchronous inference could not pull images from the corporate Harbor registry: SageMaker rejects private-registry authentication for async endpoints.
  2. Parallel regional deploys raced to commit a shared version file, so some commits were lost. Endpoint creation also had no upper bound on time.
  3. The platform had no written answer to a harder question: what happens to serving, and how fast, if a region goes away.

Constraints

  • One release, repeated in every serving region.
  • Images start in Harbor, and CI may not hold long-lived registry keys.
  • Real-time endpoints already pulled from Harbor.
  • Standby capacity costs money in every region, every hour.

Options

For the image path:

  1. An ECR pull-through cache in front of Harbor. Rejected: ECR pull-through caches accept only a fixed list of upstream registries, and a self-hosted Harbor is not on it.
  2. A dual-registry path. CI copies images from Harbor to ECR, and the SDK picks the registry by endpoint type. Chosen.

For recovery, the usual shapes of standby are active capacity in a second region, a warm standby, or replicas that exist but are scaled to zero. Zero-scaled standby costs the least and recovers the slowest, because instances must start before traffic moves. The design uses zero-scaled standby replicas, with an RTO of one hour and an RPO equal to the latest release.

Engineering decisions

  1. CI copies images from Harbor to a central ECR, authenticated with OIDC. The SDK sends async endpoints to ECR and real-time endpoints to Harbor.
  2. The version-file write leaves the parallel regional deploys and moves into one post-deploy job, so no commit races another.
  3. Endpoint creation gets an upper bound on time.
  4. Recovery goes into an ADR: RTO of one hour, RPO equal to the latest release, standby replicas scaled to zero, release checks and a runbook.
  1. Build
    • Corporate Harbor registry serving images
  2. Mirror
    • Central Amazon ECR OIDC authentication, no static keys
  3. Deploy
    • Region 1 bounded endpoint creation
    • Region 2 bounded endpoint creation
    • Regions 3 to 8 same path
  4. Record
    • Post-deploy job writes the shared version file once, so no commit races another
  • Recovery Design targets from the ADR: standby replicas scaled to zero, RTO of one hour, RPO equal to the latest release.
One release, eight serving regions. Endpoint creation has an upper bound in every region, and the version file is written once, after all regions finish.Abstracted view. Names and counts are illustrative, not the real topology.

Implementation

The SDK gained an async deploy path, an S3-transport predictor client, MLflow integration and the IAM grants async inference needs. I also added deploys from the Unity Catalog model registry and proved the path end to end with an NLP model in dev, stage and prod. The run exposed four platform defects, including model settings that the MLflow deploy path had dropped without an error.

For alerting, I built the SLO chain for the data platform: a detector on a SQL warehouse, a Lambda webhook and an SNS FIFO topic, with a dedup key that keeps breach and resolve events for each SLO in order. The chain ends in PagerDuty, and an SLO breach reaches it end to end. Databricks platform metrics reach Grafana through CloudWatch custom metrics, and the SageMaker dashboards live in code, with each panel tested against the live data source.

To give the platform’s real-time proposal a measured basis, I benchmarked streaming feature ingestion:

Path Measurement Result
Kafka to Delta, micro-batch (production) latency floor over 43,726 events p50 8.1 s
Real-time path, reproduced per row latency p50 38 ms, p99 91 ms
Real-time path, reproduced per row rows inside the 500 ms target 99.4%

Production ran on the micro-batch path. The real-time figures come from a reproduction that isolated the four settings that decide its latency: checkpoint epoch, one Kafka task per partition, Arrow batch size and asynchronous progress tracking. The two rows describe two designs, so the table keeps them apart.

Validation

Release checks run on every deploy, and the runbook covers recovery step by step. The team approved the recovery design, and this page lists its figures as design targets.

Outcome

  • ShippedAsync endpoints that pull from ECR while real-time endpoints keep Harbor, with no static registry keys in CI.
  • ShippedSerialised version commits and bounded endpoint creation on the deploy path for the eight serving regions.
  • ShippedAn SLO alerting chain into PagerDuty, with ordered breach and resolve events, and platform metrics in Grafana.
  • MeasuredA p50 8.1 s floor on the production micro-batch path; p50 38 ms and p99 91 ms on the reproduced real-time path.
  • TargetRTO of one hour and RPO equal to the latest release, from the recovery ADR.