Migration 101: it's not a copy-paste

A small moment first

The first real migration I ran felt like the safest task on the list. The old system had the data; the new one needed it; my job was to carry it across. I exported, I imported, the totals matched, everyone moved on. It was only later — when someone went looking for a record that should have been there — that I learned "the totals matched" and "the data is correct" are two completely different claims.

That gap is the whole topic. So let me map the thing first, then spend the rest of the post on the part that gap lives in.

Why "copy-paste" is the wrong picture

When people hear migration they picture copying a folder: same files, new location, done. That picture is what gets migrations into trouble, because a migration is almost never a copy. It's a translation.

The old system and the new system were built by different people, at different times, with different ideas about how the world is shaped. They store the same customer differently. They allow things the other forbids. They mean different things by the word "required." Moving the data means re-expressing it in the new system's terms — and that's an act of rewriting, not copying.

Two more things make it harder than a copy, and they're the reason it deserves real care:

• It's a one-shot, high-stakes event. An integration runs every day; if one sync is wrong you fix it and the next run is fine. A migration usually happens once, on a cutover date, often with the old system getting switched off behind it. Get it wrong and there isn't a clean "next run."
• "It ran" doesn't mean "it's right." A migration can finish with zero errors and still be quietly broken — records dropped, relationships severed, values subtly mangled. Nothing turns red. That's the dangerous part.

The shape of every migration

Strip away the jargon and every migration is three steps:

Extract — pull the data out of the old system. (Read everything you're carrying across.)

Reshape — rewrite it to fit the new system: its fields, its rules, its model of the world. This is the translation, and it's where the work is.

Load — put the reshaped data into the new system. (Write it in.)

Picture it left to right: pull it out, rewrite it to fit, put it in. People expect extract and load to be the hard parts. They're the easy parts. The middle — and a fourth step almost everyone skips — is where migrations are won or lost.

Extract and load: real, but mostly solved

Pulling data out of the old system and pushing it into the new one is genuine work, but it's generic work. Each system has an export, an import, an API — and each one documents its own. The old tool's export is not the new tool's import, but they both exist and the steps are written down for you. Change the systems and this part changes, and you follow their docs and move on.

There's little for me to teach here that the vendor hasn't already written better. The interesting part is the middle.

Reshape: where the translation happens

Reshaping is taking what the old system gave you and making it true in the new system. If you've read the integration posts, the moves will feel familiar — mapping fields, converting types, handling the values that don't fit. The shapes are the same. What's different in a migration is the stakes and the once-ness: you don't get to be sloppy and fix it tomorrow, because there often is no tomorrow for this data.

A few things bite migrations specifically:

• The new system is stricter than the old one. Old systems accumulate junk: blank required fields, two formats for the same phone number, a "country" field with forty spellings of "USA." The old system tolerated it for years. The new system, set up clean, rejects it on import — and now you're cleaning a decade of mess against a cutover deadline. The data you're moving is older and dirtier than you think.
• Relationships are the part that silently breaks. A single record is easy. The hard part is the links between them — this contact belongs to that company; this order belongs to that customer. Migrate the records but lose the links, and every individual row is technically present while the thing they formed together is gone. The counts still match. The meaning doesn't.
• History and "who/when" rarely survive. Created dates, the original owner, the audit trail — the new system wants to stamp everything with today and with you. Decide on purpose what history you carry across, because the default is to quietly lose it.
• Same value, new model. The old system had one "Status" field; the new one splits it into "Stage" and "Status." Or the old free-text field has to become a fixed dropdown. Reshaping isn't always one-field-to-one-field — sometimes you're deciding how an old idea even exists in the new world.

There's one more thing to do during reshape that costs almost nothing and saves the whole verify step. Carry the old record's ID across with it.

The golden rule: carry the source's ID across

Every record in the old system has an identifier — a unique ID the source uses to mean "this exact record, and no other." When you reshape a record, bring that ID with it and store it in a field on the destination, something like legacy_id or source_id. You're not using it as the new system's own ID; you're keeping it as a tag that says "this row came from that row over there."

It costs you one extra field and one line in the mapping. In return it gives you the one thing a migration otherwise doesn't have: a way to point at a record on each side and know, for certain, that they're the same record. That's the hook the entire verify step hangs on — which is why I call it the golden rule. Do this and verifying becomes fast and exact. Skip it and you're left guessing with counts.

None of this is exotic. It's the honest admission that the two systems were never going to line up, and reshaping is where you absorb the difference — deliberately, with a decision for every messy case, not just the clean ones.

The fourth step nobody puts on the diagram: verify

Here's the step that separates a migration that ran from one that's right: after you load, you prove it landed correctly — before you switch the old system off.

This is the part people skip, because the import said "success" and success feels like done. It isn't. But how you verify matters as much as that you do — because there are two obvious ways to check, and both are bad.

Counting is too vague. "The old system had 10,000 records and the new one has 10,000" feels like proof, and it's almost worthless. A matching total can hide an equal number of duplicates created and originals dropped — 500 lost, 500 doubled, total unchanged. Counts can only ever tell you the quantity looks right, never that the right records are the ones that made it.

Comparing every value of every record is too slow. The opposite extreme — open every record on both sides and check every field matches — is genuinely thorough and almost never practical. On any real dataset it takes forever and you'll never actually finish it before the cutover.

The golden rule gives you the middle path that's both fast and exact. Because every destination record now carries its source_id, you can verify existence cheaply and completely:

• Check the IDs, not the values. Take the full list of source IDs from the old system and confirm each one is present in the destination's source_id field. This is a fast, complete check — it covers every record, not a sample — and it answers the question counts can't: did these specific records make it across, or did something quietly drop and something else quietly duplicate? Any source ID missing on the other side is a dropped record. Any source ID showing up twice is a duplicate. The discrepancy isn't hidden in a total anymore — it's named.
• Let reshape own correctness. You don't need to re-check every value at verify time, because that's reshape's job. If the reshape rules are right, the values are right by construction — you proved the transformation once, for the field, instead of re-proving it ten thousand times, for each record. Verify confirms the records arrived; reshape already guaranteed they arrived correct.
• Then a handful of records by hand. Finish with a small, deliberate spot-check: open ten or so real records you know — including the ugly ones from reshape, the blank fields and the orphaned links — and look at them whole in the new system. This is the human sanity check that catches a reshape rule you got wrong. A few records, looked at properly, not all of them skimmed.

So: IDs prove everything arrived, reshape proves it's correct, and a few hand-picked records prove you didn't fool yourself. Fast, complete, and honest — without the impossible task of comparing every value.

"Don't verify a migration by counting records or by comparing every value. Carry the source's ID into every destination record, then check those IDs all arrived. Reshape guarantees the values; the IDs guarantee the records; a handful checked by hand keeps you honest."

What "basic" actually means

So a basic migration — the 101 version — is four steps, not three:

1. Extract: read the data out of the old system (follow its docs for the export or API).
2. Reshape: rewrite it to fit the new system's fields, rules, and model — handling the dirty data, the relationships, and the history on purpose, not by default — and carry each record's source ID across (the golden rule).
3. Load: write it into the new system (again, follow that platform's docs).
4. Verify: check that every source ID made it across (fast and complete), trust reshape for value correctness, and hand-check a handful of real records — all before you retire the old system.

Get reshape and verify right and the migration genuinely holds. The concept is small once you stop picturing a copy and start respecting the translation.

What this post is deliberately not covering

The basic version above is the shape, not the full playbook.

The moment a migration is real and large, more shows up: you can't move millions of records in one go, so you do it in batches; you don't trust the cutover blind, so you do a full dry run against a copy first; you plan how to roll back if the cutover goes wrong; you keep the old system readable for a while as a safety net. Each of those deserves its own post, and I'll link them here as I write them — starting with the one that matters most: you never migrate once.

For how this differs from an ongoing sync — a migration happens once, an integration runs forever — see Integration 101: the bridge is the whole job. Confusing the two is its own kind of expensive mistake.

What I'd do now

Before I extract anything, I write down two things on one page. First, the reshape plan: for every field, what it becomes in the new system, what I'll do with the records that don't fit — and the source_id field that rides along on every record. Second, the verification plan: the source-ID check that proves everything arrived, plus the specific real records I'll open by hand. I write the verification plan before the migration, on purpose, so "looks done" can't talk me out of it on the day. And I never let the source ID get dropped in reshape — it's one field, and it's the difference between verifying in minutes and guessing with counts.

If you take one thing from this: a migration isn't moving data, it's rebuilding it somewhere new — and it isn't finished when it runs, it's finished when you've proven it's right.

Why I'm sharing this

I've carried data between systems enough times to have learned the hard way that the scary part isn't the moving — it's everything that fails without making a sound. The migrations that taught me most are the ones that "succeeded" and were wrong, and I only found out later. I'm writing it down for the people a few steps behind me, who are about to trust a green checkmark — and for the ones ahead, who can tell me what I'm still getting wrong.