Case 05 Shipped
A migration built from a full inventory
A legacy Kubeflow estate, from 51 pipelines to more than 1,500 registered models, inventoried and scored before anything moved, then migrated to Databricks in waves.
Stack: Kubeflow · GCP · Databricks · Unity Catalog · Asset Bundles · MLflow · Delta Sharing · Harbor · Vault · Trivy · release-please · Terragrunt
- 51 Kubeflow pipelines inventoried and scored Measured
- ~25% of the estate retired instead of moved Shipped
- 141 catalogs moved to a new data-governance group through reviewed changes Shipped
On this page
Context
The platform’s ML workloads ran on a self-managed Kubeflow estate in GCP, and the target was Databricks on AWS. I audited the legacy estate, wrote the migration plan, built the platform groundwork it depended on, and ran the move in waves.
Problem
A migration plan is as good as its inventory. Nobody had a complete list of what ran on Kubeflow, what each pipeline depended on, or which of the registered models still mattered.
Constraints
- Two clouds during the move: GCP as the source, AWS as the target.
- Services load models by name, so model references had to survive the move.
- Teams in several workspaces needed the same data.
Options
For each asset there were two paths: migrate it to the target platform, or retire it. The complexity score set the order and the effort.
Engineering decisions
- Score every asset for migration complexity.
- Map Kubeflow Pipelines primitives to Databricks Jobs and Asset Bundles.
- Move more than 1,500 registered MLflow models to Unity Catalog model aliases, one mapping per model family, with Delta Sharing across workspaces.
- Retire about a quarter of the estate instead of moving it.
- Inventory
- 51 Kubeflow pipelines
- 41 image directories, about 22 unique
- 25 package directories, about 22 unique
- 1,500+ registered models, mostly per-city variants
- Connected through every assetScore
- Migration complexity one score per asset
- Decide
- Migrate mapped to the target platform
- Decommission about 25% of the estate
- Connected through migrate pathMap
- Databricks Jobs and Asset Bundles from Kubeflow Pipelines primitives
- Unity Catalog model aliases from MLflow models
- Delta Sharing data across workspaces
Implementation
The plan needed a target platform that teams could trust, so the groundwork shipped first.
Custom containers for Databricks jobs. A cluster policy accepts images from the corporate registry and nothing else, and registry credentials stay hidden from users. A versioned base-image line uses release-please for versions and Trivy for scanning, and a self-service guide walks teams through the flow.
Unity Catalog governance. I extended the account group model and created the production data-governance admin group. Reviewed Terragrunt changes moved ownership of 141 catalogs, across Unity Catalog, BigQuery federation and Delta Sharing, to the new group, with declarative import blocks bringing drifted resources under code. An artifact-allowlist module with input validation and destroy guards runs on five workspaces, and a governance watchdog detects access and ownership drift in dev, stage and prod, and corrects it in dev.
Validation
I built the first migration pilots as Databricks Asset Bundles. One of them collapsed 75 per-city Kubeflow jobs into one parameterised job. The pilots proved the mappings on real pipelines, and the complexity scores then set the order of the waves that followed.
Outcome
- MeasuredAn inventory of 51 Kubeflow pipelines and the images and packages behind them (about 22 unique of each), every asset scored for migration complexity.
- ShippedKubeflow Pipelines primitives moved to Databricks Jobs and Asset Bundles.
- ShippedMore than 1,500 registered MLflow models moved, family by family, to Unity Catalog model aliases.
- ShippedAbout 25% of the estate retired instead of moved.
- ShippedThe first migration pilots as Asset Bundles, with 75 per-city jobs collapsed into one parameterised job.
- ShippedCustom-container support with a Harbor-only policy, Vault-injected credentials and scanned base images.
- ShippedCatalog ownership moved to a data-governance group through reviewed changes, an artifact allowlist on five workspaces, and a governance watchdog.