Hadoop Migration Validation and Rollback Procedures

Every Bifrost migration is designed to be safe, repeatable, and reversible. This page describes the mechanisms that make that possible: the validation framework that confirms correctness at every step, the decision engine that turns validation results into go-or-no-go verdicts, and the rollback procedures available at each phase.

Validation Framework

Bifrost validates at multiple levels. The appropriate set of checks runs automatically for the current phase; nothing is skipped silently.

Row-count parity

For every migrated table, Bifrost compares source and target row counts per partition. The default tolerance is 0 % for immutable tables; for append-only or streaming tables the tolerance is configurable.

Value-level data diff

Row-count parity alone does not catch value drift. Bifrost uses hash-tree sampling for value-level validation:

• Partitions are hashed in a binary tree.
• Only divergent branches are expanded.
• Verification cost is O(log n) rather than O(n) for a full-table scan.

The default sample rate is 1 % of partitions. For business-critical tables, customers can raise this to 5 % or 10 % at the cost of additional run time. Sample rate is configurable per table or globally:

data_diff:
  row_count_tolerance: 0.0
  sample_rate: 0.01
  hash_algorithm: xxhash64
  partition_parallelism: 8
  timeout_per_partition: 300

Query parity

Validates that production queries return the same results, within the same time envelope, on the source engine and the target engine. Run with a representative query set:

bifrost modernize validate \
  --type query-parity \
  --query-file benchmark_queries.sql \
  --source-engine hive \
  --target-engine trino

The tolerance for latency regression is 1.3x (queries on Trino can be up to 30 % slower than on Hive before triggering a warning). Value-match tolerance is 0 % by default.

Schema comparison

Validates column names, types, nullability, partition specs, and sort orders match between source and target. Catches subtle drift that would not show up in row-count parity alone.

Plan Validation (before anything destructive)

Before the phase gates below ever run, bifrost classic validate-plan validates the migration plan itself against the discovered estate: offline, read-only, and repeatable. It closes the two blind spots that phase gates cannot: discovery completeness (every configuration key the estate actually runs must have a recorded disposition in the migration plan: carried, translated, denied for a stated reason, or explicitly reviewed) and projected drift (what the migrated cluster's configuration and endpoints will look like, compared against the estate's effective configuration, before a single service is stopped). An optional live re-probe confirms the captured baseline still matches the running cluster, so a plan computed days after discovery cannot silently go stale. Findings feed the same decision engine and the same PROCEED / WARN / ABORT verdicts described below; see the CLI reference for flags and report layout.

The report is written to extract/plan_validation.md and extract/plan_validation.json in the inventory. Findings are grouped into check families:

Family	Checks
PLAN-D	Configuration drift against the estate's effective configuration and the target defaults.
PLAN-C	Discovery completeness, including the freshness of the capture (--live).
PLAN-E	Endpoint and topology parity: ports, TLS endpoints, and service placement.
PLAN-S	The adoption chain: delivered packages, service definitions, security capabilities, and the canary registration of every migrated service.
PLAN-I	Identities.
PLAN-P	Stack advisor recommendations.
PLAN-T	Topology decisions.
PLAN-X	Secrets that appear in plain text in the configuration.
PLAN-W	Waivers and acknowledgements in force.

A standalone run reports the true verdict. In a chained run, --gate-mode soft caps an ABORT at WARN and --gate-mode report always exits 0; the report carries the true findings in every mode.

A finding is waived in inventories/<cluster>/plan_waivers.yml (or the file named by --waivers). Every waiver names a check, a key or service scope, and a reason; an unscoped waiver is refused. A waived failure stays visible as WARN with its reason, and a waiver that matches nothing is reported as its own finding.

waivers:
  - check_id: PLAN-D-1
    key: dfs.ha.fencing.methods
    reason: "Fencing method reviewed for this estate"
    owner: "[email protected]"

Before-and-After Canary

The canary compares the state and behaviour of every service before and after the migration, on the same objects. Signatures are recorded on the source before the window and on the migrated cluster after the management plane takeover; the two snapshots are then compared, and the comparison returns a verdict.

Step	Command	What it does
Prerequisites	bifrost classic canary prereqs	Provisions the canary identity and trust material on a secured cluster.
Seed	bifrost classic canary seed --execute	Writes a small deterministic dataset to the source under the canary namespace (HDFS paths, a Hive database, a ZooKeeper node).
Before	bifrost classic canary capture --snapshot cdp-before	Records the before-state signatures of every enabled service.
After	bifrost classic canary capture --snapshot post-takeover	Records the same signatures on the migrated cluster.
Compare	bifrost classic canary verify --from cdp-before --to post-takeover	Compares the snapshots, writes the report, and returns a verdict.
Report	bifrost classic canary report	Renders the report without changing the exit code.
Clean up	bifrost classic canary cleanup --execute	Removes the canary objects.

The signatures cover HDFS content checksums, Hive and Spark query results, HBase state, Kafka topics and offsets, Solr collections, Atlas entities, Oozie jobs, Hue functions, Ranger policies, KMS keys, and Livy sessions, among others. Captures are read-only by default; --workload adds jobs that write to HDFS scratch space and measure performance against the before-state.

• Gate modes. --gate-mode soft (the default) reports a mismatch as a warning, hard turns a critical mismatch into ABORT, and report is advisory. The presets --mode light (soft, signatures only) and --mode full (hard, with workloads) set both at once.
• Same run only. Snapshots from different migration runs are not compared unless --allow-cross-run is given, because a comparison with an earlier cutover proves nothing about the current one.
• Real tables. Besides the seeded dataset, selected production tables can be fingerprinted read-only (row count and content hash per table).
• After a rollback. A post-rollback snapshot compared against cdp-before confirms that a rolled-back cluster is back in its original state.

The report is written as Markdown, JSON, and a self-contained HTML page under the cluster's state directory.

Decision Engine Gates

At the end of each phase, the decision engine runs a set of checks and returns one of three verdicts:

• PROCEED: every critical check passes. The next phase is allowed. Production gates typically still require explicit human approval.
• WARN: a non-critical check failed. Results are logged and sent to the notification channels, but progression is not blocked.
• ABORT: a critical check failed, or two or more abort triggers fired. The rollback mechanism for the current phase is invoked automatically, without human intervention.

Classic gates

A Classic migration passes through these gates:

Gate	When	Effect of ABORT
validate-plan	Offline, after extraction	The plan is corrected before any destructive step.
validate-pre	Before the window	The window does not start.
Phase-7 gates (upgraded-version target)	During migrate	Automatic rollback.
Management plane takeover	Phase 9	The takeover stops; rollback is started by the operator.
validate-ambari	Optional, before or after the takeover	Reported; nothing is changed.
canary verify	After the takeover	With --gate-mode hard, a critical mismatch returns ABORT.
validate-post	After the takeover	Automatic rollback only with --rollback-on-abort.

For Classic, automatic rollback applies inside migrate --execute, after the package swap; after the takeover, a rollback is started by the operator (rollback --from-adopted) or by validate-post --rollback-on-abort. finalize requires a PROCEED from validate-post; a WARN does not qualify.

Every check carries a status and a severity. A skipped critical check is classified (NOT_APPLICABLE, NOT_ASKED, NOT_MEASURABLE, or UNCLASSIFIED), and a skipped critical check that is not NOT_APPLICABLE caps the verdict at WARN, so a gate never returns PROCEED over an unanswered question. See CLI reference: exit codes.

validate-post enforces the following critical checks:

Check	Effect if failed
HDFS fsck reports no corrupt blocks	ABORT
HDFS fsck reports no missing blocks	ABORT
Live DataNodes at least 95 % of expected	ABORT
HBase reports no dead region servers	ABORT
Hive Metastore table count matches baseline	ABORT
Policy database count matches baseline	ABORT
Kerberos authentication succeeds	ABORT
TLS handshake succeeds	ABORT

Warning checks (non-blocking):

• HDFS under-replicated blocks less than 1000.
• YARN reports no unhealthy nodes.
• Kafka reports no under-replicated partitions.
• TeraSort duration no more than 20 % slower than baseline.

Abort triggers (two or more cause automatic rollback):

• NameNode not active by the gate time.
• Safe mode not exited by the gate time.
• Two or more critical check failures.
• Any DataNode data loss detected.

Modernize and Direct gates

Critical checks at Modernize and Direct gates:

Check	Threshold
Table row-count parity	Must match exactly
Data-diff sample pass rate	> 99.99 %
Iceberg snapshot consistent	Yes
Trino query parity ratio	< 1.3x latency regression
Object storage cluster health	HEALTH_OK
Spark job completion rate	>= 99 %

Abort triggers:

• Any table row-count mismatch.
• Object storage cluster in HEALTH_ERR.
• Catalog service unreachable.
• Trino coordinator in crash loop.

Wave-level failure semantics

During Modernize and Direct wave execution, table failures are handled at the table level, not the wave level:

1. The failed table is quarantined (marked FAILED in the migration record).
2. The wave continues; other tables proceed normally.
3. A wave is marked COMPLETE only when every non-quarantined table passes validation.
4. Quarantined tables do not block the wave, but must be resolved before the cluster can be decommissioned.
5. Retry policy is 3 attempts with exponential back-off: 5 minutes, then 15 minutes, then 45 minutes.
6. After 3 failures, the table requires manual intervention.

Rollback: Path 1 (Classic)

Every phase is reversible until finalize. The rollback window remains open from the start of the migration through the pre-finalization soak and closes only when bifrost classic finalize --confirm-irreversible runs. Rollback reuses the same decision engine as forward migration, so the cluster is only considered healthy once post-rollback validation passes: the rollback closes with its own verdict, and the canary post-rollback snapshot is compared against the before-state.

Four rollback windows exist, in stage order:

Stage	When it applies	Mechanism	Typical duration
After package swap	Target distribution installed, services not yet started	Remove target packages, restore source distribution from local cache, restart cluster-manager agent	~2 hours
After services started	Services running on target distribution, post-migration validation failed	Stop services, run HDFS rollback (upgraded-version target), restore HMS and policy databases from pg_dump, reinstall source distribution, restart via cluster manager	~4 hours
After pre-finalization monitoring (5-day soak)	Soak period in progress; issue surfaces after initial validation passed	Same mechanism as "after services started"; the window stays open for the full soak	~4 hours
After finalization	bifrost classic finalize has run	Not reversible. On an upgraded-version target, hdfs dfsadmin -finalizeUpgrade has deleted the previous/ directory; the on-disk predecessor blocks are gone.	N/A

Bifrost waits at least 5 business days before finalizing specifically so operators can still choose not to finalize if the migrated cluster misbehaves during the soak. Finalize is a deliberate, manual step with a required --confirm-irreversible flag. Re-run validate-post at the end of the soak: finalize accepts only a PROCEED at most seven days old by default (BIFROST_FINALIZE_VALIDATE_POST_MAX_AGE_SECONDS).

On a version-aligned target no layout upgrade runs; -finalizeUpgrade is recorded as not applicable and rollback does not revert HDFS. Packages, configuration, ZooKeeper, databases, and KMS keys are restored. Because HDFS keeps its newer state, two acknowledgements are required for this run: that the restored metastore describes the earlier view of HDFS (--parity-db-divergence-ack), and that keys of encryption zones created after the backup are lost (--accept-orphaned-ez).

After package swap (services not started)

The fastest rollback window. The source distribution packages are still in the local cache from the backup phase; nothing started on the target distribution yet.

• Remove target-distribution packages.
• Restore source-distribution packages from the on-node cache.
• Restart the cluster-manager agent.
• Services come up on the source distribution as if the swap had not happened.

After services started (validation failed)

Services already started on the target distribution and post-migration validation did not pass.

• Stop target-distribution services.
• On an upgraded-version target, revert HDFS. For shrink-and-grow, run hdfs namenode -rollingUpgrade rollback. For stop-and-swap, restart the NameNode with the -rollback startup argument.
• Restore the Hive Metastore database from pg_dump.
• Restore the policy database from pg_dump.
• Reinstall the source-distribution packages from the local cache.
• Restart services via the cluster manager.

After pre-finalization monitoring (5-day soak)

Validation already passed but a production issue surfaces during the post-migration soak (typically 5 business days). The rollback window remains open for the entire soak and uses the same mechanism as "after services started". This is the window customers run against most often in practice, because real operational problems often appear only hours or days after cutover.

After finalization

Not reversible. Once bifrost classic finalize has run on an upgraded-version target, hdfs dfsadmin -finalizeUpgrade has removed the previous/ directory on the NameNode and every DataNode. Neither the NameNode -rollback startup argument nor -rollingUpgrade rollback is available; both depend on previous/ being present. The cluster-manager database and HMS/policy database backups Bifrost captured during the backup phase have also been removed.

This is the exact reason Bifrost requires the 5-day soak before offering the finalize command. Skipping or shortening the soak trades operational safety for cleanup time.

Partial rollback: shrink-and-grow variant

In shrink-and-grow runs, some DataNodes migrate while others remain on the source distribution. Any of the four stages above can be hit with only part of the cluster on the target distribution; the revert is scoped to the migrated nodes:

• Decommission the migrated nodes.
• Wait for HDFS replication to complete.
• Revert those nodes to the source distribution using the appropriate stage-specific mechanism.
• Recommission.

Rollback After the Management Plane Takeover

When the cluster has already been adopted by the management plane, bifrost classic rollback --from-adopted --ambari-host <host> reverses the adoption before the ordinary rollback runs:

1. The Ambari cluster definition, hosts, and configuration groups are captured for the record.
2. The Ambari cluster is removed (un-adopt).
3. The pre-adoption configuration snapshot is restored on every node.
4. The Ambari agents and server are uninstalled.
5. The group memberships and service accounts the adoption created are removed.

Each stage can be turned off for a single run. The ordinary rollback to the selected phase follows.

Rollback Modes and Acknowledgements

• Plan first. Without --execute, the rollback shows a preview.
• Preflight. --preflight runs only the pre-write checks and writes nothing, so a team can confirm before a window that a rollback would be possible.
• Verify. --verify runs only the post-rollback checks, for example after a manual repair.
• Resume. --resume finishes a rollback that stopped after Cloudera Manager came up; --from-the-start repeats it.
• Integrity. Every rollback asset is checked against the checksums in the rollback manifest before anything is written.
• Acknowledgements. Specific situations require an acknowledgement for the current run, for example --external-db-backup-ack for a metastore or Ranger database that is not on PostgreSQL and was snapshotted outside Bifrost.
• Verdict. The rollback ends with COMPLETE or INCOMPLETE; an incomplete rollback exits with code 5 and lists every step that was declined or could not be measured.
• Out of scope. The Hue and Cloudera Manager databases are restored from the database team's snapshot.

Rollback command

# Roll the whole cluster back to a specific phase
bifrost classic rollback --cluster prod01 --to-phase backup --execute

# Roll back an adopted cluster
bifrost classic rollback --cluster prod01 --from-adopted --ambari-host ambari.example.internal --execute

# Roll a single node back (shrink-and-grow)
bifrost classic rollback --cluster PROD01 --node dn-042.example.internal

Rollback: Paths 2 and 3 (Modernize / Direct)

Modernize and Direct rollbacks are fundamentally different from Classic rollbacks because the legacy environment is still running in parallel with the target during most of the migration.

Table-level rollback

Instantaneous. Iceberg metadata is swapped back; the table reverts to its pre-migration state in microseconds. Bifrost executes this automatically when validation fails, and the same revert can be triggered manually after the fact:

# Revert a single migrated table to its pre-migration state
bifrost modernize rollback --table production_db.customers

# Revert every table in a wave
bifrost modernize rollback --wave 3

Table redirection is disabled for the affected tables as part of the revert; queries resume against the legacy catalog until the table is migrated again.

Service-level rollback

Service rollbacks use Helm rollback. The previous Helm release of any Kubernetes service (Trino, Polaris, Spark Operator, Airflow, and so on) can be reinstated with a single helm rollback command.

Storage-level rollback

HDFS data is not deleted during Modernize; it remains intact until explicit decommission. During the dual-read bridge phase, the legacy path is always available as a fallback. Only bifrost modernize decommission (after the silence period) makes the legacy storage unavailable.

Per-phase rollback matrix (Modernize and Direct)

Phase	Failure symptom	Revert command	State implication	Time to recover
land fails part-way	Helm release partially applied	helm rollback <release> per component that rolled; re-run bifrost modernize land --status	Target platform returns to the previous Helm release; no source data is touched	Minutes per component
bridge fails	Trino table redirection misconfigured, or DistCp warm-sync loop erroring	bifrost modernize bridge --disable issues a Helm upgrade of the Trino catalog config to drop the redirection rules; a Trino coordinator reload follows. Fix the config and re-run bridge.	Queries fall back to the legacy catalog only	5 to 15 minutes (Helm upgrade + coordinator reload)
migrate-table fails validation	Data-diff or query-parity below threshold	Automatic: Bifrost reverts via rollback --table <name>. Manual: same command	Table redirection disabled; legacy table is authoritative again	Microseconds (Iceberg metadata swap)
migrate-wave partial failure	Some tables passed, some failed	Quarantined tables auto-revert; passed tables remain migrated	Wave marked INCOMPLETE until quarantined tables resolve	Per quarantined table
convert-workflow produces broken DAG	Airflow DAG import error, or runtime failure on cutover	Pause the Airflow DAG; keep Oozie workflow active on legacy cluster; re-run converter with corrected rulesets	No production impact; legacy workflow still authoritative	Minutes to hours
hue-import partial	Some queries or dashboards failed to import	Inspect hue-import migration report; re-run for specific documents with --user-mapping-file fixes	Legacy HUE still available	Minutes
decommission refused	Bifrost detects residual access during the silence period	Investigate the access source (reported in decommission log); re-run decommission --dry-run after remediation	No state change (decommission never executed)	N/A

All revert operations are recorded in the migration ledger and surfaced through bifrost modernize status.

Irreversible Finalize

Every migration has a single, irreversible final step that removes rollback assets and ends the ability to revert.

Classic finalize

bifrost classic finalize --cluster prod01 --confirm-irreversible --execute

Without --execute, finalize plans only. It refuses unless every HDFS encryption-zone key is proven decryptable on the target and a fresh validate-post returned PROCEED.

Finalize removes:

• The source distribution package cache on every node.
• LVM snapshots of NameNode metadata volumes.
• Baseline captures and backup databases.
• Temporary rollback keytabs and certificates.

Bifrost recommends a 5-business-day soak of clean operation before running finalize. Re-run validate-post at the end of the soak: finalize accepts only a PROCEED at most seven days old by default. The --confirm-irreversible flag is required and cannot be bypassed.

Modernize and Direct decommission

Modernize and Direct do not have a single "finalize" step. Instead, each legacy service is decommissioned individually after its silence period passes:

# Decommission HDFS after 30 days of confirmed silence
bifrost modernize decommission \
  --service hdfs \
  --cluster PROD01 \
  --after-silence 30d

The final irreversible step for a Direct migration is the Cloudera Manager shutdown:

bifrost direct decommission \
  --service cloudera-manager \
  --cm-host cm.example.internal \
  --confirm-irreversible

This step ends the Cloudera subscription requirement and cannot be reversed.

Summary: Verdict, Scope, and Rollback

The following table captures the decision engine's granularity across paths:

Scope	Verdict applies to	Rollback scope
Phase gate (Classic)	Entire cluster	Entire cluster
Table migration (Modernize / Direct)	Single table	Single table (Iceberg metadata swap)
Wave validation (Modernize / Direct)	All tables in the wave	Per-table (only failed tables revert)
Storage migration (Modernize / Direct)	DistCp job	Re-run from last checkpoint
Workflow conversion (Modernize / Direct)	Single workflow	No rollback needed (re-run converter)

Next Steps

• Operations: monitoring, capacity planning, and the production readiness checklist.
• Troubleshooting: common issues and their resolutions.
• CLI reference: every command, every flag, every option.