Adiom

One Database Is a Liability: Splitting Tenant Data Across Clusters Without Losing Your Mind

Alexander Komyagin — Tue, 09 Jun 2026 22:40:06 GMT

A single shared database is the easiest way to start a multi-tenant product - and one of the fastest ways to back yourself into a corner. As you scale, four forces pull your data apart: compliance, locality, quality of service, and blast radius. This is how to architect for that split, and how to move tenants between clusters when - not if - you have to.

Why One Cluster Stops Working

In the beginning, every tenant lives happily in one database. It’s simple, it’s cheap, and it’s exactly the right call. But growth has a way of turning that simplicity into a constraint. Sooner or later, four distinct pressures show up - usually all at once - and none of them are solved by buying a bigger box.

Compliance	A healthcare customer needs their data isolated under HIPAA. An EU customer's records can't leave the EU under GDPR. A government tenant needs a sovereign environment entirely. "Just add a tenant_id column" doesn't satisfy an auditor who wants hard isolation boundaries.
Locality	Your tenant in Sydney shouldn't pay a 200ms round-trip to a database in Virginia. Latency is a product feature, and the only way to fix it is to put data physically close to the users who read and write it.
Quality of Service (QoS)	One enterprise tenant runs a runaway analytical query and suddenly everyone shares the pain. Noisy neighbors are inevitable in a shared cluster. Dedicated or pooled clusters let you guarantee performance tiers and stop one tenant from degrading the rest.
Blast Radius	A bad migration, a corrupt index, a failover gone wrong - in a single cluster, that's an incident for every customer at once. Partitioning tenants across clusters means a failure is contained to a slice of your business, not all of it.

The goal isn’t more databases for their own sake. It’s the ability to draw isolation boundaries on purpose, where the business actually needs them.

The Architecture: Five Components That Make It Work

A regional, multi-cluster, multi-tenant system isn’t one big piece of software - it’s a handful of components that each do one job well. Get these five right and the rest is detail.

The five components of a regional multi-tenant data platform

1. Application-level routing

The router is the brain. Every request carries a tenant identity, and the application layer maps that identity to the right cluster - tenant → cluster - before any query is issued. This map is the source of truth for where a tenant’s data lives. It needs to be fast, cached close to the application, and - critically - updatable at runtime, because the day you move a tenant, the router is what makes the cutover real. Routing at the application layer, often as a service side car (rather than relying on a database-side proxy), keeps the logic explicit, testable, and tied to your domain’s notion of a tenant.

2. Database clusters

These are the homes for your data: a fleet of clusters partitioned by region, compliance regime, or service tier. Some are pooled (many small tenants sharing a cluster for cost efficiency), some are dedicated (one large or sensitive tenant per cluster). The architecture has to treat clusters as interchangeable destinations - provisionable, decommissionable, and addressable by the router - rather than hand-tuned pets.

3. Data mover

This is the component most teams underestimate, and it’s the reason this post exists. Tenants don’t stay put. They outgrow their pooled cluster, change compliance requirements, or need to move to a new region. The data mover is what relocates a tenant’s data from one cluster to another - ideally live, with the tenant’s application still serving traffic the whole time. Without it, every one of the four pressures above eventually becomes a wall you can’t get past.

4. Observability

With data spread across many clusters, “is the system healthy?” becomes a per-cluster, per-tenant question. You need unified visibility into cluster load, per-tenant resource consumption, replication lag, and the progress and integrity of any in-flight migration. Observability is also what tells you when to move a tenant - it’s the signal that a cluster is getting hot or a neighbor is getting noisy.

5. Control panel

The control panel is the operator’s cockpit: provision a cluster, place a tenant, kick off a migration, watch it land, update the routing map, and decommission what’s empty. It turns a set of powerful-but-dangerous primitives into repeatable, auditable operations. The better the control panel, the less every tenant move depends on one engineer remembering the runbook.

“But MongoDB Already Has Sharding” - Why That’s Not the Same Thing

If you’re on MongoDB, the database can shard for you. Why run twelve independent replica sets behind an application-level router when you could run one 12-shard cluster and let the balancer spread tenants across shards automatically (using sharding tags)? Both spread data across twelve sets of hardware. They are not the same architecture.

A single sharded cluster is one logical database. That’s the whole point of it - and also the catch. A 12-shard cluster shares config servers, a balancer, a version, and a maintenance window across all twelve shards. Twelve replica sets with app-level routing are twelve genuinely independent databases that happen to be coordinated by your application.

	12-shard cluster	12 replica sets + app routing
Blast radius	Shared config servers and balancer - a control-plane problem can affect the whole cluster	Each replica set fails on its own; one going down touches only its tenants
Compliance / locality	One cluster, typically one region and one trust boundary	Put each replica set in its own region or compliance regime, independently
Upgrades & maintenance	Version and maintenance windows are cluster-wide	Upgrade or patch one tenant's database without touching the rest
Tenant placement	The balancer decides and moves, by shard key and tags - not by your business rules	You place tenants explicitly, by tier, region, or contract
Moving a tenant	Chunk migration within one cluster; can't move a tenant to a different region or trust boundary	A data-mover job relocates a tenant to any other cluster, anywhere

Native sharding is excellent at what it’s for: scaling a single dataset horizontally when you don’t need isolation between the rows. But the four pressures in this post - compliance, locality, QoS, and blast radius - are all about drawing boundaries, and a sharded cluster is deliberately built to erase them. App-level routing over independent clusters keeps you in control of where each boundary sits. The trade is that you now own the routing map and the tenant moves yourself - which is exactly why the data mover matters.

Sharding spreads one database wider. App-level routing across independent clusters gives you many databases you can place, isolate, and move on purpose.

The Hard Part: Migrating Big Tenants, Live

Here’s the irony at the center of multi-tenant operations. You almost never need to move a tenant when they’re small and easy. You need to move them precisely when they’ve become big - hundreds of gigabytes or more - and important, and busy. The migration you can’t avoid is also the one that’s hardest to pull off.

And the constraints stack up fast. A tenant migration in a live multi-tenant system has to satisfy all of these at once:

Minimal interruption of service. The tenant is in production. You don’t get to take them offline for a weekend.
Automated or semi-automated. If every move requires an engineer babysitting a UI for eight hours, you’ll never do it often enough to keep up.
Filtered at the source. You’re moving one tenant out of a shared cluster - not the whole database. The mover has to select just that tenant’s data, cleanly.
Into a non-empty destination. The target cluster already hosts other tenants. You’re merging into a live dataset, not restoring into a blank slate.
Carefully throttled. The source cluster is still serving every other tenant. A migration that saturates I/O turns one tenant’s move into everyone else’s outage - exactly the blast radius you were trying to avoid.

Now look at what most data migration tooling actually assumes:

Most migration tools expect…	Multi-tenant reality
Dedicated infrastructure you provision and babysit	You want it to run as a job on infra you already have
A one-shot bulk copy, no live changes without complicated setup	The tenant keeps writing throughout
An operator clicking through a UI	You need it scripted and repeatable
An empty destination	The target already has tenants on it
Migrate the whole dataset	You need to filter to one or several tenants at the source
Run flat-out, as fast as possible	You must throttle to protect production neighbors

Standard migration tools are built for the empty-to-empty, take-it-offline, click-the-button case. Multi-tenant migrations are none of those things.

How We Solved It: Dsync as the Data Mover

For one of our customers running exactly this kind of regional multi-tenant platform, we used Dsync as the data mover - and it lines up against the constraints above point for point. Dsync was built for live production migrations, which is precisely what a tenant move demands:

Live migration. Initial sync plus change data capture, so the tenant keeps serving traffic while their data moves. The cutover is a routing-map flip, not a downtime window.
No specialized infrastructure. Dsync runs as Kubernetes jobs on the cluster you already operate - nothing extra to provision, secure, and tear down for every move.
Source-side filtering. Move exactly one tenant’s data out of a shared cluster, instead of the entire dataset.
Merges into non-empty destinations. Land a tenant on a cluster that’s already serving other tenants - the realistic case, not the demo case.
Load-level throttling. Cap the migration’s footprint so the source cluster keeps its performance promises to every other tenant on it.
Fast and resumable. Parallelized copy for hundreds of gigabytes, and if something interrupts a run, it picks up where it left off instead of starting over.
Observable and secure, with embedded validation. You can watch progress and lag in real time, and Dsync verifies data integrity as part of the move - so you cut over on evidence, not hope.

The result is a tenant migration that behaves like a routine operation instead of a heart-surgery event: filtered at the source, throttled for the neighbors, merged into a live destination, validated end to end, and finished with a routing flip - all without standing up a parallel migration stack.

Building a regional multi-tenant system?

If you’re building - or planning to build - a regional, multi-tenant data platform, the data mover is the component that decides whether your architecture can actually evolve. Check out Dsync for tenant data mobility, and reach out if you want help with tenant migrations - we’ve done this in production and can help you do it too.

[

Talk to the Dsync Team

](https://www.adiom.io/contact)

Migrating to Azure Cosmos DB: Why, How, and What to Expect

Alexander Komyagin — Wed, 03 Jun 2026 05:42:00 GMT

If you’re optimizing your existing database stack or modernizing it to keep up with the times, you have probably already felt the pull toward a fully managed, AI-ready, and globally distributed database on Azure.

Before going further, it is worth being precise about names, because Azure’s branding has been evolving over the years and can be somewhat confusing. Two distinct products are in play:

Azure Cosmos DB for NoSQL - the native Cosmos DB engine and document API. This was originally launched as the “Cosmos DB SQL API” (and Cosmos DB itself was briefly called “DocumentDB” back in 2014). The “SQL API” name persisted for years and still turns up in old docs and Stack Overflow answers - it refers to the same thing now called Cosmos DB for NoSQL. It is not a relational database; the “SQL” only ever meant a SQL-like query dialect over JSON.
Azure DocumentDB (aka vCore) - a MongoDB-compatible document database, powered by the open-source DocumentDB engine (a PostgreSQL-based engine, MIT-licensed, open-sourced by Microsoft in January 2025) with full MongoDB wire-protocol compatibility. Microsoft now markets this as “Azure DocumentDB” in its own right. It is unrelated to Amazon’s separate “DocumentDB” product.

These are not interchangeable, and they compete with different things. This post keeps them separate throughout: why teams migrate, how each product works and stacks up, how to run the migration, and who has done it successfully.

1. Why Migrate to Azure Cosmos DB

The motivations split into three major categories: costs, removing operational burden, and unlocking capabilities that are impractical to build yourself.

Eliminating operational toil. Self-managed NoSQL means owning sharding, replica-set failover, patching, backup verification, and capacity planning. Both products are fully managed - automatic maintenance, patching, and updates - with no application changes required.

Global distribution. Cosmos DB for NoSQL offers turnkey global distribution: you add regions with a button, with active-active multi-region writes and automatic failover. This is the single hardest thing to replicate with a self-managed cluster. It is also the workload pattern OpenAI relies on - running ChatGPT’s product data on Cosmos DB with multi-region replication across dozens of regions (more on that below).

SLAs you can put in a contract. Cosmos DB for NoSQL provides financially-backed SLAs covering availability, latency, throughput, and consistency. Multi-region accounts are guaranteed 99.999% availability; Azure DocumentDB carries a 99.99% SLA. Cosmos DB for NoSQL delivers single-digit-millisecond response times and offers five well-defined consistency levels, letting teams tune the latency/consistency tradeoff explicitly.

Elastic scalability and cost control. The NoSQL API supports three throughput models - standard provisioned, autoscale, and serverless (pay-per-request). There is a lifetime free tier (1000 RU/s + 25 GB), and Reserved Capacity offers up to 63% savings. One caveat: serverless accounts are single-region only.

Thesdsdds DocumentDB path for MongoDB shops. For lift-and-shift from MongoDB, Azure DocumentDB is usually the better target than the older RU-based MongoDB API: predictable per-vCore pricing rather than the harder-to-estimate RU model, rich aggregation-pipeline fidelity, and multi-cloud portability via standard MongoDB drivers and tooling.

The AI era - one database for operational data and vectors. Microsoft now positions Cosmos DB explicitly as a “unified AI database”: document, vector, key-value, graph, and table data in one store. Integrated DiskANN vector and hybrid similarity search lets teams keep embeddings next to operational data for RAG, AI agents, and LLM caching - avoiding a separate vector database.

What’s new - Azure Cosmos DB Conf 2026 (April 28, 2026)

The 2026 Cosmos DB conference headlined several developments worth factoring into a migration decision:

Azure Cosmos DB Agent - a built-in AI assistant with a growing catalog of skills for partition-key selection, data modeling, index design, and RU-consumption troubleshooting. This directly targets the hardest parts of a migration.
MultiCloudDB SDK (preview) - a portable Java SDK (“write once, run anywhere”) spanning Azure Cosmos DB, Amazon DynamoDB, and Google Cloud Spanner.
Azure DocumentDB - “one codebase, any cloud” - deploy the open-source engine on-premises via Kubernetes or fully managed on Azure with no code rewrites.
Azure RBAC integration for Cosmos DB entered private preview.
AMD EPYC v7 infrastructure - up to 35% more performance and performance-per-dollar on the newest generation.

OpenAI, Vercel, and Office Depot/ODP all presented; OpenAI’s Jonathan Lee described running “thousands of product tables on Azure Cosmos DB” with multi-region replication across dozens of regions.

2. Technical Details and How It Compares

Shared architecture concepts

Cosmos DB for NoSQL is built on a few core ideas:

Request Units (RUs). CPU, memory, and IOPS are abstracted into a single throughput currency. Every operation costs a measurable number of RUs; billing follows RU/s plus storage. (Azure DocumentDB / vCore does not use RUs = it bills on provisioned vCore tiers, which is part of its appeal.)
Partitioning. Data is split into logical partitions by a chosen partition key, mapped onto physical partitions (each ~50 GB and 10,000 RU/s). Picking a high-cardinality, evenly-accessed key is the single most important design decision. Hierarchical (sub-)partition keys handle high-cardinality and multi-tenant workloads.
Five consistency levels - Strong, Bounded Staleness, Session, Consistent Prefix, Eventual. Session is the default; Strong gives RPO 0.
Change feed - an ordered, persistent log of item changes (latest-version and all-versions-and-deletes modes), used for event-driven architectures and replication.
- Note that Change Stream in Azure Document DB is independent from Cosmos NoSQL change feed. It’s implemented on top of PostgreSQL WAL replication and still has some rough edges as of this writing - the Cosmos DB team is actively working on that.

Comparison A - Azure DocumentDB (vCore) vs. MongoDB Atlas and AWS DocumentDB

This is the relevant comparison if you run MongoDB today or planning to replatform a legacy database to MongoDB. All three databases speak the MongoDB wire protocol; the differences are price, access to premium storage, and native integrations.

	Azure DocumentDB (vCore)	MongoDB Atlas	Amazon DocumentDB
Pricing model	Predictable per-vCore tiers + storage	Compute + storage, premium at scale	Instance-based + storage I/O
Premium / high-perf storage	NVMe-class tiers; DiskANN vector search	Workload-tiered storage	I/O-Optimized storage class
MongoDB feature parity	High and improving; open-source engine	Reference implementation - full parity	Trails real MongoDB on version/features
Native integrations	Deep Azure (AI Foundry, Functions, AI Search, Entra ID)	Cloud-agnostic; owns its own ecosystem	Deep AWS (IAM, Lambda, etc.)
Availability SLA	99.995%	Tier-dependent	Tier-dependent
Lock-in	Engine open-source (MIT); service on Azure, multi-cloud on K8s, or on-prem	Low - runs on any cloud or self-hosted	High - AWS only

The deciding factors are usually price and integration. DocumentDB’s predictable per-vCore pricing is easier to forecast and often cheaper than Atlas, it gives you NVMe-class premium storage and integrated vector search, and it plugs natively into the Azure AI stack. Atlas wins on full MongoDB fidelity and cloud portability; AWS DocumentDB mainly makes sense if you are already all-in on AWS, though it lags on MongoDB feature parity.

Comparison B - Azure Cosmos DB for NoSQL vs. DynamoDB and Cassandra

This is the relevant comparison for greenfield, high-scale key-value/document workloads where you are not tied to MongoDB.

	Cosmos DB for NoSQL	AWS DynamoDB	Apache Cassandra
Model	Document, rich query/indexing	Key-value + document, thinner query model	Wide-column, CQL
Pricing	RU/s (provisioned, autoscale, serverless) + storage	Capacity units / on-demand	Self-managed infra cost
Global distribution	Turnkey, multi-region multi-write	Global Tables	Multi-datacenter, manual ops
Consistency	5 tunable levels	Eventual or strong	Tunable per-query (quorum)
Operational model	Fully managed	Fully managed	Self-managed unless using a managed offering (e.g. Astra DB, Azure Cassandra MI)
Lock-in	Azure only	AWS only	None (open source)

Against DynamoDB, Cosmos DB for NoSQL’s distinguishing strengths are its richer query and indexing model and a five-level consistency spectrum (DynamoDB offers only two). Against self-managed Cassandra, the win is eliminating the operational burden of multi-datacenter clusters while keeping tunable consistency. Cosmos DB’s main downsides are Azure lock-in and an RU cost model that is notoriously hard to estimate up front. (Cosmos DB also offers a wire-compatible Cassandra API - a common zero-downtime landing zone for existing Cassandra workloads; see the Symantec story below.)

3. How to Migrate

For brevity, in this post we frame a migration as three phases - pre-migration (assessment and planning), migration (moving data), and post-migration (cutover and optimization). Well-executed planning is the single biggest predictor of a smooth migration - you can read more about that in our 5-series post on migration planning.

Step 1 - Plan

Assess. Inventory every database and collection with its name and data size. The Azure Cosmos DB Migration extension for VS Code assesses a MongoDB workload and flags unsupported features before you move anything.
Choose the target. Azure DocumentDB for lift-and-shift from MongoDB; Cosmos DB for NoSQL for high-scale work.
Choose the partition/shard key. This is the most important and immutable decision - pick a key that distributes both storage and request volume evenly, and don’t blindly reuse your existing MongoDB shard key. (The new Cosmos DB Agent can help here.)
Capacity-plan. For the NoSQL API, size RU/s with the capacity calculator or measure real query charges against sample data; pre-provision enough RU/s so Cosmos DB creates partitions ahead of ingestion. For vCore, pick the node tier.

Step 2 - Choose tooling and migrate

The right tool depends on dataset size, downtime tolerance, and how hard it is to set up - an underrated dimension, since complex tooling is itself a source of migration risk and delay.

Tool	Mode	Ease of setup	Best for
Adiom Dsync	Online (continuous)	Easy - even with horizontal scalability	Production NoSQL migrations with minimal-downtime cutover
Native MongoDB tools (mongodump/mongorestore)	Offline	Easy	< 10 GB
Azure Database Migration Service (DMS)	Online or offline	Moderate - Premium tier often required	< 1 TB
Azure Data Factory	Offline	Moderate	> 1 TB
Azure Databricks + Spark	Online or offline	Hard - requires custom code	Large datasets, custom transforms
Cosmos DB Desktop Data Migration Tool	Offline	Easy	Cross-platform CLI use

For production workloads - where you cannot afford an extended downtime window - Adiom’s Dsync is the leading purpose-built tool for production NoSQL migrations. It performs an initial bulk copy followed by continuous change-data-capture replication, keeping source and target in sync so you can validate the target under real traffic and then cut over with minimal downtime. Critically, it is easy to set up - and stays easy even when you scale it horizontally to move large datasets quickly. You can run it on your laptop to migrate 100’s of GB’s, or in your Kubernetes cluster to migrate 100’s of TBs. That combination of low setup friction and horizontal scalability is what sets it apart from the heavier built-in options, which trade setup complexity for capability.

The built-in Azure options have constraints worth knowing: DMS online migrations require DMS instances in the Premium tier, and are not supported against serverless Cosmos DB accounts. For DocumentDB/vCore specifically, RU-to-vCore migration is GA and free from the Azure portal.

Step 3 - Online migration and cutover

For minimal downtime, bulk-copy a snapshot, then continuously replicate changes until source and target converge - exactly the pattern Dsync automates. When replication lag is near zero, stop writes briefly, drain remaining changes, switch the application’s connection string, and cut over. Offline migration, by contrast, incurs downtime for the entire copy duration.

Step 4 - Post-migration optimization

The only mandatory step is repointing your connection string - but cutting over before optimizing causes an immediate price/performance hit. Recommended:

Tune indexing. Cosmos DB for MongoDB 3.6+ indexes only _id by default; the NoSQL API indexes everything automatically. Disable or minimize indexing during bulk load, add indexes after.
Validate data, configure global distribution (≥ 2 regions for HA), and set the consistency level (default: session).
Monitor RU consumption via Azure Monitor and tune iteratively.

4. Success Stories

OpenAI / ChatGPT. The headline reference: “OpenAI relies on Cosmos DB to dynamically scale their ChatGPT service – one of the fastest-growing consumer apps ever – enabling high reliability and low maintenance” (Satya Nadella). At Cosmos DB Conf 2026, OpenAI detailed running thousands of product tables on Cosmos DB with multi-region replication across dozens of regions - a proof point that the database scales with one of the most demanding consumer workloads on the planet.

Novo Nordisk — migrating off a relational database (2025). A textbook cost-migration story: Novo Nordisk moved an application from an RDBMS setup costing ~$280/month to Azure Cosmos DB at under $1/month — while improving the end-user experience and reducing its carbon footprint. Presented at Cosmos DB Conf 2025.

Next - re-architecting for cost (2026). The UK retailer faced escalating RU costs on a production workload. By redesigning partitioning, indexing, and query shape — and moving from querying shared mutable state to event capture - Next cut its Cosmos DB costs by more than 60% with no loss of performance or reliability. The lesson, per the presenter: “the issue wasn’t scale, it was design.” (Conf 2026 production recap)

Office Depot / The ODP Group - AI agent memory (2026). ODP built an enterprise AI Personal Assistant on Azure Cosmos DB, using it as the agent memory layer - per-user partitioned profiles and analytics. Employee adoption nearly doubled year-over-year, with database scaling described as “hands-off” - no manual re-architecture as load grew. (Conf 2026 production recap)

KPMG - agentic AI audit (2025). KPMG Clara AI uses Cosmos DB for chat history, session data, and agent memory. It serves 95,000 auditors across 140+ countries and processes petabytes of data annually - scale KPMG states would be impossible without Cosmos DB’s ability to deploy globally without re-architecting per jurisdiction.

Takeaways

Don’t get lost in the terminology: Cosmos DB for NoSQL (the engine formerly branded the Cosmos DB SQL API) versus Azure DocumentDB (vCore), built on the open-source DocumentDB engine.
Compare Azure DocumentDB against MongoDB Atlas and AWS DocumentDB - the deciding factors are price, premium storage, and native integrations. Compare Cosmos DB for NoSQL against DynamoDB and Cassandra - the wins are richer querying, tunable consistency, and managed global distribution.
The partition/shard key is immutable - get key design and capacity planning right before moving a single document.
For production migrations, prefer continuous replication with a minimal-downtime cutover, and weigh ease of setup alongside capability. Adiom’s Dsync is purpose-built for this: easy to set up, horizontally scalable, and free of the offline-copy downtime that the built-in tools incur.

Cassandra and New AKS-native Enterprise Offering

Alexander Komyagin — Thu, 21 May 2026 20:22:02 GMT

New Control Dashboard in Dsync Enterprise

We’re excited to share two releases that go hand in hand: a new Cassandra connector in private preview, and a fully AKS-native edition of Dsync Enterprise, now available on the Azure Marketplace.

The Cassandra connector

The Cassandra connector supports both source and sink roles, so you can migrate into and out of Cassandra-family databases. It works with Apache Cassandra, DataStax Enterprise, and Astra DB.

That opens up a set of migration paths customers have been asking us for:

- Apache Cassandra / Astra DB / DataStax Enterprise → Azure Managed Instance for Apache Cassandra - a homogeneous lift to a managed service.

- Apache Cassandra / Astra DB / DataStax Enterprise → MongoDB Atlas - a heterogeneous move to a document model.

As with the rest of Dsync, these are live migrations: data flows continuously from source to destination, so you can cut over on your schedule instead of the migration’s. And with Dsync Enterprise the migration scales linearly with the number of workers, so you can migrate 10TB with the same ease as 10GB.

Why an AKS-native offering

Cassandra deployments aren’t small. A typical cluster holds terabytes of data and billions of records, and that scale is exactly where migrations get hard. For workloads like these, customers reach for Dsync Enterprise, which is built to scale horizontally.

The challenge was deployment. Standing up and operating a scalable migration tool shouldn’t be a project of its own. So we built a fully AKS-native offering and published it on the Azure Marketplace. It deploys straight into Kubernetes, regular or Automatic, - and if you’re not on Azure, the same release ships as a Helm chart for any Kubernetes cluster.

What’s new in Dsync Enterprise

The AKS-native release also introduces a web control dashboard for setting up and operating migrations:

- Manage multiple concurrent flows from a single deployment - no redeploying to add a migration.

- Dynamically allocate worker nodes to each flow, and scale them up and down natively in Kubernetes as the workload changes.

- Test your flows and transformations ahead of time, so you can validate your configuration before moving production data.

You control worker allocation per flow from the Web UI, which means one Dsync deployment can run a fleet of migrations side by side.

More than Cassandra

The AKS-native offering features Dsync Enterprise and includes several private-preview connectors as a bonus. Beyond the Cassandra paths above, common routes include:

- MongoDB Atlas / Community → Azure DocumentDB

- Cosmos DB → MongoDB Atlas

- AWS DocumentDB → Azure DocumentDB or MongoDB Atlas

- DynamoDB → Cosmos DB NoSQL or MongoDB Atlas

- Apache HBase → MongoDB, DocumentDB, or Cosmos DB NoSQL

- PostgreSQL → DocumentDB or MongoDB Atlas

- SQL Server → MongoDB, DocumentDB, or Cosmos DB NoSQL

- DB2 → DocumentDB or MongoDB Atlas

- Oracle → DocumentDB or MongoDB Atlas

For the full list of supported connectors, see our documentation.

Why teams choose Dsync

Adiom Dsync is the fastest, easiest, and most reliable way to move production workloads at scale — for RDBMS and NoSQL databases, in homogeneous or heterogeneous configurations. It’s purpose-built for mission-critical migrations:

- Accelerated timelines - complete online migrations in minutes to hours, not weeks or months, with native horizontal scaling in AKS.

- Enterprise-grade reliability - built-in resiliency and data validation reduce risk and prevent interruptions.

- Zero-storage transfer - data flows directly from source to destination with network encryption and no intermediate storage.

- Effortless deployment - no extra infrastructure to manage; monitor progress via CLI or web UI with full operational transparency.

- Expert guidance - tap into our deep experience with the process and the technology for a smooth, predictable migration.

Getting started

Dsync can deploy into your existing AKS cluster (regular or Automatic), or you can create a new cluster as part of the offer setup. All you need is a VNet from which both the source and destination databases are reachable.

- Get the offer: Adiom Dsync on the Azure Marketplace

- Enterprise deployment docs: https://docs.adiom.io/enterprise/scalable-deployment

- AKS-native deployment details: https://docs.adiom.io/enterprise/running-dsynct/azure-marketplace

The Cassandra connector is in private preview today. Contact us for a migration assessment, help with setup, and a customized private offer.

Mission-Critical DB Migrations Are Heart Surgery. Treat Them That Way.

Alexander Komyagin — Thu, 16 Apr 2026 22:13:37 GMT

Most engineering teams wildly under- or overestimate what a large database migration actually takes. Here’s the honest picture - and why the right tooling is the difference between months and years.

Two Wrong Opinions, One Real Problem

Ask an engineering or business unit leader what a database migration involves and you’ll almost always land on one of two extremes. Either they think it’s trivially easy - “just run it once in production” - or they treat it as some mythical, unknowable risk that gets endlessly deferred. In both cases, the result is the same: the can gets kicked down the road.

“Just run it once in production” is not a migration strategy. It’s a hope dressed up as a plan.

The misconception is partly understandable. Most leaders have only witnessed a handful of migrations in their careers, if any. And the range of what “a migration” can mean is enormous - which is exactly where the confusion starts.

Small Migrations vs. Mission-Critical Ones

Not every migration is equal. Small migrations - think under 100GB for non-critical internal services like dashboards or analytics tools - are genuinely fast and low-risk. They can often be completed in a single day with minimal preparation and even less organizational anxiety. These are routine procedures.

Large migrations for revenue-generating, customer-facing systems are something else entirely. They are, without exaggeration, like heart surgery. The stakes are high. Precision is non-negotiable. And the confidence to execute comes not from luck or seniority, but from the ability to consistently and repeatably run the process in production - with the right instruments in hand.

There’s a reason surgical teams don’t operate with improvised tools. A scalpel alone, versus access to a da Vinci Xi robotic system, represents not just a difference in capability but a difference in the outcomes that are even possible. The same logic applies here.

Production migrations get complicated because of feedback loops between activities

What a Live Production Migration Actually Requires

To execute a live database migration in production without taking your system offline, you need to coordinate a surprising number of moving parts simultaneously:

Parallelize data copy - copying terabytes sequentially is impractical; concurrency is essential.
Coordinate initial sync and CDC - Change Data Capture must seamlessly bridge the gap between the initial snapshot and live writes.
Maintain data integrity - every row must land correctly, every relationship intact, every constraint respected.
Be able to rollback - if something breaks, you need a clean, practiced path back. No improvising at 2am.
Account for source and destination specifics - edge cases, encoding differences, type mismatches, and engine-specific behaviors all need handling.
Handle legacy and “ghost” records - production databases accumulate years of orphaned rows, deprecated schemas, and soft-deleted data that can break assumptions.

None of this is optional. Miss any one of them and you risk data loss, extended downtime, or a failed cutover window with the entire business watching.

What Good Tooling Provides

✓ Resumability	✓ Data integrity checks
✓ Observability	✓ Transformations
✓ Security	✓ Ease of use
✓ Scalability	✓ Extensibility for schema changes

Repeatability Is the Real Metric

Perhaps the least intuitive part of a major migration project is how many times you need to run the full process before you actually touch production for real. The answer is: a lot.

First, the migration runs repeatedly in lower environments - dev, staging, UAT - to calibrate the process, catch edge cases, and establish baseline performance metrics. These runs let you extrapolate how long the production migration will take and what can go wrong. Then come production dry runs: full end-to-end rehearsals, everything except the final cutover flip.

Cutovers themselves are tightly choreographed events. They happen during off-peak windows, require all hands on deck, and are followed by stabilization periods of at least one to two weeks where the team monitors closely for regressions. Business units can only offer these windows a few times a year. There’s no wasted attempts budget.

The repeatability and predictability of your migration tooling essentially define how long the project takes - and when you can actually cut over.

Why We Built Dsync

Dsync is our answer to the question: what does the robotic surgical platform look like for database migrations? It’s not a one-click migration service - the expertise and decision-making still lives with your team. But it meaningfully changes the probability of a successful outcome and compresses the timeline from something measured in calendar years to something measured in weeks.

Before you can commit to a large migration project, you need months of feasibility assessment just to answer whether it’s even possible and what risks you’re taking on. Dsync shortens that runway significantly, giving teams the observability, resumability, and integrity checks needed to build confidence fast - and the extensibility to handle whatever surprises your production schema has been hiding for the last decade.

We’re working toward a more automated future. But even today, having the right platform under you is the difference between a migration that drags on for a year of organizational anxiety and one that’s done before the next planning cycle.

Ready to treat your next migration like the surgery it is?

[

Talk to the Dsync team →

](https://www.adiom.io/contact)

New SQL Connectors in Public Preview

Alexander Komyagin — Mon, 09 Mar 2026 18:02:55 GMT

We’re excited to share that support for SQL Server, Oracle, and PostgreSQL via our SQLBatch connector is now in public preview. The SQLBatch connector supports both initial sync and CDC (all query-based), and is available in our Open Source Dsync. Check it out on GitHub.

Usage

The SQLBatch connector accepts a YAML configuration file with user-defined SQL queries declaring “virtual namespaces” to be migrated:

driver: sqlserver  # or oracle, postgres
connectionstring: sqlserver://user:password@host:port?database=DbName
mappings:
  - namespace: db.users
    query: "SELECT id, name, email FROM users"
    partitionquery: "SELECT id FROM users WHERE id % 4 = 0 ORDER BY id"
    cols: [id]
    limit: 1000
    changes:
      - initialcursorquery: "SELECT MAX(updated_at) FROM users"
        query: "SELECT id, 'U', updated_at FROM users WHERE updated_at > $1 ORDER BY updated_at LIMIT 1000"
        interval: 5s

Each virtual namespace has the following key properties:

Unique name
Query to get the full count
Query to partition the dataset
Query to retrieve the objects
One or more Change Tracking queries
- One for the main table and one per each embedded object

To start Dsync with the SQLBatch connector and the config saved in cfg.yml:

dsync sqlbatch --config=cfg.yml /dev/null --log-json

A complete example for migrating or syncing the sample Microsoft AdventureWorks database from SQL Server to Cosmos DB for NoSQL is available in the AdventureWorks sample repository.

Read more about the connector usage in the docs.

To help create or bootstrap the config file you can use your favorite AI coding agent (like Claude Code, Codex, Gemini or Factory.AI Droid) with the Skills framework, or by converting MongoDB’s Relational Migrator mappings using our AI Agent. Both assets will be published soon but feel free to contact us in the meantime if you need them.

Note that the Open Source version of Dsync only supports a single namespace for CDC. For CDC with multiple namespace consider our Enterprise version.

How it works

Dialects

In the SQLBatch connector, dialects or drivers abstract away the differences between SQL database vendors (SQL Server, PostgreSQL, DB2, etc.). Each dialect handles:

1. SQL syntax variations - Different databases use different syntax for pagination (LIMIT vs TOP), string concatenation, date functions, etc.

2. Data type mappings - How native types (e.g., NVARCHAR, BIGINT, DATETIME2) translate to the target document store

3. CDC mechanisms - Change Data Capture works differently across databases (SQL Server uses CT/CDC tables, PostgreSQL uses logical replication slots, DB2 uses journals)

4. Query generation - The dialect generates vendor-specific queries for initial loads and incremental syncs

When you configure a SQLBatch source, you specify the dialect (e.g., driver: sqlserver or driver: postgres), and the connector uses the appropriate SQL generation and type conversion logic for that database platform.

Denormalization

The connector uses a user-defined query to denormalize the data at the source. Most recent versions of major SQL database vendors do a great job at supporting JSON, and their query engines are able to perform complex JOINs much more efficiently than any streaming processor.

For additional transformations beyond what RDMBS can do, such as converting JSON into native BSON types, Dsync supports extensible transformers.

Change Tracking / CDC

The SQLBatch connector relies on a change tracking (CT) query and a configurable polling interval to determine what objects have changed at the source. Since the final JSON objects are often composed from several tables, the connector supports multiple CT queries per namespace - one for each of the source tables that are embedded in the object.

The modified or deleted objects are then efficiently refetched using the main query in batches - hence the name “SQLBatch”. Compared to WAL-based log parsing, our approach allows us to eliminate a whole class of possible data integrity issues, require no custom software on database servers, and only needing regular read access.

At Adiom, we’re making usually complex and messy data migration and replication easy. You can check the SQLBatch connector out on GitHub and read more about its usage in the docs.

We also have DB2 support for SQLBatch available in Private Preview. Contact us to request access.

[Part 5] Production Database Migration or Modernization: A Comprehensive Planning Guide

Alexander Komyagin — Wed, 18 Feb 2026 17:20:50 GMT

In this fifth and the final post in our guide, we will cover a sample database migration project plan and will discuss the key lessons learned based on our extensive experiences with 10GB to 100TB+ migrations.

We highly recommend reading the previous posts in the series:

Part 1 - Migration Readiness and Key Factors Influencing Timeline and Risk

Part 2 - Downtime Strategy Options and Migration Tools

Part 3 - Migration Planning and Post-Migration Validation Strategy

Part 4 - Common Complications and Best Practices

9. Sample Migration Plan: RDBMS to NoSQL

To “ground” the concepts that we discussed in past posts, let’s describe a realistic migration example:

Scenario Overview

A financial services company operates a backend API service with these characteristics:

220GB relational database with 12 tables
Java monolith using Hibernate with tight database coupling in places
Several dozen API endpoints supporting web and mobile clients
Nightly batch jobs and Databricks data pipelines
Event streaming via Kafka for some of the operations

Data must be denormalized for the target NoSQL database, no stored procedures or triggers currently in use.

The service is busy during banking hours (6am-6pm), otherwise mostly idle except for heavy batch jobs around 3am. Of course, this being a banking client, there’s exactly zero tolerance for data integrity issues.

They’ve selected a NoSQL database for better horizontal scalability and chosen Adiom’s Dsync for migration based on its RDBMS to NoSQL transformation capabilities and CDC support.

Phase 1: Planning and Design (2-4 weeks, extended to 6-7 weeks)

Initial timeline: 2-4 weeks.

The team begins by mapping the current relational schema to a denormalized NoSQL model, identifying which tables should become embedded documents versus separate collections.

Architecture review board approval, vendor consultant coordination, and executive stakeholder presentations generally consume more time than anticipated.

Network constraints surface during planning - the source database sits in an on-premises data center while the target will be Cloud-based, requiring careful private link configuration and security review.

Based on the initial testing, Dsync can migrate their 220GB database in approximately 4 hours, enabling a straightforward weekend migration window with a sufficient user notification. Dsync’s CDC capabilities mean they don’t need to worry about capturing changes during migration.

The team decides to implement reverse sync for critical tables as a risk mitigation strategy - changes on the new database will flow back to the old system during a two-week validation period after cutover, allowing them to revert if issues arise. Batch jobs need to be accounted for separately in the rollback scenario. The team evaluates complete replay as well as capturing the changes as part of the Dsync reverse flow.

Due to lack of documentation and numerous undocumented dependencies discovered in the codebase, this phase extends an additional 2-3 weeks beyond the original estimate.

Phase 2: Environment Setup and Tooling Preparation (1-3 weeks)

The team provisions the target NoSQL cluster in their Cloud environment, configures Dsync with access to both source and target databases, and establishes monitoring and alerting.

DNS issues with private links consume several days of troubleshooting. Security team reviews and permission grants take longer than expected, but by week 3 all infrastructure is operational.

Phase 3: Code Changes and Dev Testing (2-3 months)

This phase involves substantial parallel engineering effort across four streams:

Stream 1: Data Access Layer Refactoring - Engineers create a clean DAL interface abstracting all database operations. They systematically refactor the monolithic codebase to route all data access through this interface rather than making direct database calls. They then build a NoSQL implementation of the DAL. This work represents the largest engineering investment in the migration.

Stream 2: Batch Jobs and Pipeline Updates - The team updates nightly batch jobs and Databricks pipelines to work with the denormalized NoSQL schema. Several “surprises” emerge - aggregations that were simple SQL queries now require multiple steps, and some join patterns need complete rethinking.

Stream 3: Forward Data Migration and Transformation - Using Dsync in the dev environment, the team tests data migration and transformation logic repeatedly. They refine mappings, test performance, and optimize the target database configuration. Initial migrations take 6+ hours; after tuning they achieve 3-hour migration times in dev.

Stream 4: Reverse Sync Testing - The team configures and tests reverse transformations, ensuring changes made in the NoSQL database can flow back to the relational source. This provides confidence in their safety net strategy. Rollback of batch jobs is tested separately.

Phase 4: UAT Testing (1-2 months)

UAT environment contains roughly half of production data volume (110GB). The team conducts extensive functional testing with business users, performance testing under realistic load, and capacity planning.

They discover that their initial target database configuration is undersized for peak load and adjust accordingly. Batch jobs complete successfully, though some require optimization to meet their time windows.

Several data model artifacts surface in UAT that weren’t present in dev’s smaller dataset -older records with different encoding, legacy status values no longer in use, and some foreign key relationships that exist in data but aren’t properly constrained. These findings drive updates to transformation logic.

Phase 5: Production Dry Run (1 month, with issues extending timeline further)

The team executes a complete migration using full production data (220GB) with Dsync’s CDC maintaining continuous sync. Application servers run against the migrated data in parallel with production traffic going to the old database.

This reveals critical issues that didn’t appear in smaller environments. Old production records - some dating back 15+ years - contain data type inconsistencies and encoding problems that break transformation assumptions. In a handful of records, several fields that are expected to be dates contain text values. The target schema must be adjusted to handle these cases, which requires updating transformation logic, batch jobs, and the application DAL implementation. These changes propagate back through dev and UAT environments for validation.

The dry run validates migration timing and infrastructure capacity - actual production migration takes 4.5 hours, which is within acceptable bounds. It also reveals that one less-frequently-used API endpoint has 10x higher latency with the new database - investigation shows a missing index that was implicit in the relational schema.

Issues discovered in the dry run extend this phase and push back the planned cutover date by two weeks.

Phase 6: Cutover Preparation (1-2 weeks)

With all dry run issues resolved, the team builds a minute-by-minute cutover plan for a Saturday night/Sunday morning window. They assign specific responsibilities, prepare rollback procedures, and draft user communications. The plan includes checkpoints at 30-minute intervals with go/no-go decisions.

Internal users receive two weeks’ notice, external users receive one week’s notice of the planned maintenance window. Customer support receives detailed talking points and FAQs.

Phase 7: Production Migration and Cutover (2 days)

Friday evening, the team begins final preparations. They check the health of all the services and endpoints, the source and the destination. No surprises.

Saturday at 11pm, they initiate the migration following their detailed runbook:

10.45pm - Final health-check before initiating the migration

11:00pm - Enable read-only mode on the source database and pos

11:05pm - Begin full data migration with Dsync

3:20am - Data migration completes (4h 15m actual time)

3:25am - Automated validation scripts run

4:10am - Validation completes with complete match accounting for expected differences in concurrent data from batch jobs

4:15am - Switch application configuration to new database and deploy new batch process

4:30am - Smoke tests pass

4.30am - GO/NO-GO Decision

4:40am - Enable reverse sync from NoSQL to RDBMS (safety measure)

4:45am - Begin gradual traffic ramp-up

6:00am - Full traffic on new database

8:00am - Business users begin validation

Monday morning passes smoothly with banking hours traffic. By Monday evening, the migration is declared successful. Next Saturday afternoon, after a week of operation, the team disables reverse sync and formally decommissions the legacy system.

Phase 8: Post-Migration Monitoring and Optimization (1 month)

During the first week, monitoring reveals one forgotten index needed by a weekly report job - easily remedied. The team makes several configuration optimizations based on production query patterns. By week four, all performance metrics meet or exceed targets, and the team transitions to normal operational monitoring.

Timeline Summary:

Planning and Design: 6-7 weeks (2-4 weeks initially planned)
Environment Setup: 3 weeks
Code Changes and Dev Testing: 3 months
UAT Testing: 1 month
Production Dry Run: 1.5 months (1 month initially planned, extended due to issues)
Cutover Prep: 2 weeks
Cutover Execution: 2 days
Post-Migration: 1 month active monitoring and optimization

Total: Approximately 8 months from kickoff to completion

10. Lessons Learned and Conclusion

Biggest Bottlenecks

Migration tooling
Code changes
Overhead of communication and scheduling

Even with a tool like Dsync and minimum required code changes, it’s not uncommon for migration timelines to extend when coordination between multiple teams is needed. In most cases it can’t be helped, but it pays off to be realistic in planning the budget and setting expectations with broader stakeholders.

Common Pitfalls to Avoid

Underestimating the time required for dependency discovery and stakeholder alignment - add buffer to early phases
Assuming test data represents production reality - data quality issues lurk in old records
Skipping the production dry run to save time - this is where the most expensive issues are caught
Inadequate monitoring and alerting - you can’t fix what you can’t see
Poor communication with stakeholders - silence creates anxiety and distrust

Success Factors

Executive sponsorship and clear prioritization ensure teams can focus on migration work without competing demands
Comprehensive testing at each phase catches issues when they’re cheaper to fix
Realistic timeline estimates with built-in buffer accommodate inevitable surprises
Strong collaboration between engineering, operations, and business teams keeps everyone aligned
The right tools for the job - whether vendor solutions like Dsync or custom scripts -dramatically affect success probability.

Next Steps and Resources

If you’re planning a database migration, start with thorough assessment. Understand your requirements, constraints, and risk tolerance. Engage stakeholders early. Consider proof-of-concept migrations with subsets of data to validate your approach.

For complex migrations, especially RDBMS to NoSQL transformations, specialized tools can significantly reduce risk and timeline. Adiom’s Dsync, for example, was purpose-built for these challenging scenarios.

How Dsync Helps

Dsync addresses many of the pain points in database migrations through seamless integration between initial data copy and real-time CDC, resumability, observability, sophisticated transformation capabilities that handle the complexity of relational to NoSQL mapping, embedded validation tools that automate data integrity verification, and expert support from teams who’ve executed hundreds of migrations.

For migrations where data transformation is required, downtime is unacceptable, and data integrity is critical - exactly the scenarios where in-house solutions fall short - specialized tools like Dsync transform a months-long, high-risk project into a manageable, well-supported process.

Database migration is challenging, but with careful planning, the right tools, and realistic expectations, it’s an achievable goal that unlocks new capabilities for your applications and business. Contact us for help with production migrations.

You can download and try Dsync for your migration on GitHub. We built it to make your migration experience seamless.

[Part 4] Production Database Migration or Modernization: A Comprehensive Planning Guide

Alexander Komyagin — Mon, 09 Feb 2026 05:05:31 GMT

The fourth part of our multi-post guide discusses common complications in production migrations as well as general best practices to ensure success of your project.

If you haven’t read the first three parts, read them here:

Part 1 - Migration Readiness and Key Factors Influencing Timeline and Risk

Part 2 - Downtime Strategy Options and Migration Tools

Part 3 - Migration Planning and Post-Migration Validation Strategy

7. Common Complications and Mitigation

Expect the unexpected. These complications appear in nearly every migration.

Legacy Code Dependencies and Technical Debt

Codebases evolve over years, accumulating assumptions about database behavior. Stored procedures that seemed convenient become migration blockers. Undocumented business logic embedded in database triggers must be reimplemented. Tackle technical debt incrementally during the dev phase rather than attempting a big-bang refactor.

Lack of Code Documentation and Dependencies

The engineer who wrote critical data access code left years ago. Documentation is sparse or outdated. Untangling dependencies becomes archaeological work. Budget extra time for discovery and invest in documenting what you learn. For complex environments, try to execute a production dry-run as early as possible - that’s where you’ll learn the most.

Database or Network Issues During Ingestion

Network partitions, connection timeouts, and resource exhaustion can interrupt migration. Design for resilience and choose tools with retry logic, checkpointing to resume from failure points, rate limiting to avoid overwhelming source or target, and monitoring to detect issues quickly.

Tool Permissions and Access Control

Migration tools need appropriate permissions on both source and target databases. In enterprise environments, obtaining these permissions through security review processes can take weeks. Address access requirements early.

Data Model Incompatibilities and Transformations

Migrating from relational to NoSQL requires fundamental thinking shifts. Joins become embedded documents or separate queries. Transactions have different semantics. Carefully design your target schema and test transformations thoroughly with realistic data. When performing a dry run, test all the critical workflows to ensure that queries are executed efficiently.

Integration Dependencies

Your database isn’t isolated. Data pipelines, reporting systems, third-party integrations, and internal tools all depend on database structure and availability. Catalog all dependencies early and coordinate changes across teams.

Performance Degradation During Migration

Migration processes can saturate network bandwidth, max out CPU on source or target databases, and impact production workloads. Rate-limit your migration, schedule intensive operations during low-traffic periods, and monitor impact continuously.

Unexpected Data Quality Issues

Old databases accumulate data quality problems: inconsistent formats, orphaned records, violated constraints that aren’t enforced, and edge cases in data types. These issues commonly surface during production dry runs when you encounter the full diversity of production data. Build data filtering and transformation logic to handle these scenarios.

DNS

If nothing else, it’s almost certain that DNS issues will haunt you in your production environment during the migration. Rehearse and test all the related changes ahead of time.

8. Best Practices

Learn from our decade-long experience as well as the collective experience of the industry.

Comprehensive Testing in Non-Production Environments

Test everything, repeatedly, in environments that mirror production. Automate tests so they can run frequently without manual effort. Include not just happy paths but failure scenarios and edge cases.

Incremental Migration When Possible

Reduce risk by breaking large migrations into smaller chunks. Migrate less critical data or services first to refine your process. Each successful increment builds team confidence and improves procedures.

Complete Production Dry-runs

There’s nothing closer to your production than… your production. Do a dry-run, or two. A dry-run is a complete execution of your migration, including application re-deployment, but without the cut over. Don’t skip steps. Turn this into a well-planned and rehearsed activity, to avoid rollercoasters on the day of.

Automated Validation Scripts

Manual data comparison doesn’t scale. Choose tools or build robust automation that can validate millions of records quickly and accurately. These scripts become invaluable assets for future migrations and ongoing data quality monitoring.

Clear Rollback Procedures

Hope for the best, plan for the worst. Everyone involved should understand exactly what triggers a rollback and what steps to execute. Practice rollback procedures during dry runs.

Documentation and Runbooks

Document your architecture, data transformations, migration procedures, and operational playbooks. Document the step-by-step project plan. This documentation helps during the migration and becomes institutional knowledge for operating the new system.

Post-Migration Performance Tuning

Your initial migration focuses on correctness. After cutover, optimize based on real production query patterns. Add indexes, adjust configurations, and refine your data model based on actual performance metrics.

In our next and the last post in this series we will showcase a sample project plan and will discuss the key lessons.

Download and try Dsync for your migration on GitHub. We built it to make your migration experience seamless.

[Part 3] Production Database Migration or Modernization: A Comprehensive Planning Guide

Alexander Komyagin — Wed, 04 Feb 2026 15:53:49 GMT

In this third part of our multi-post guide we will discuss migration planning and post-migration validation strategy. It might even be the most important post in the series!

If you haven’t read the first two parts, read them here:

Part 1 - Migration Readiness and Key Factors Influencing Timeline and Risk

Part 2 - Downtime Strategy Options and Migration Tools

“Plan for what is difficult while it is easy, do what is great while it is small”, Sun Tzu

5. Migration Plan and Timeline

A structured, phased approach minimizes risk and ensures that the right resources are involved. Use the phases given below as a common framework, and feel free to adapt it to your unique environment as needed.

Phase 1: Planning and Design

Develop comprehensive architecture diagrams showing data flows, define detailed input schema and schema mappings including all transformations, create a migration runbook documenting every step, identify dependencies across systems, and establish rollback criteria and procedures. Review business constraints, choose your migration strategy and tooling. Do a small POC if necessary.

This phase typically takes 2-4 weeks but can extend significantly with complex stakeholder landscapes, architectural review boards, and infrastructure constraints.

Phase 2: Environment Setup and Tooling Preparation

Provision target database infrastructure with appropriate capacity, configure migration tools and establish connectivity, set up monitoring and alerting for both source and target, ensure availability of non-production environments mirroring production, and implement security controls and access policies.

Expect 1-3 weeks, with network configuration - private links, DNS, firewall rules - often being the source of unexpected delays.

Phase 3: Dev Testing

This critical phase typically runs 2-3 months with parallel work streams:

Stream 1: Code Changes - Create or refactor your data access layer (DAL) to abstract database interactions. Rewrite application code to use the new interface and implement the DAL for your target database. This stream represents the bulk of engineering effort.

Stream 2: Integration and Batch Job Changes - Update all batch jobs, ETL processes, and data pipelines to work with the new schema and database paradigm. Surprises often emerge here as dormant edge cases surface. Often batch jobs lack test coverage. This is a perfect opportunity to fix that.

Stream 3: Data Migration Testing - Test your migration process repeatedly in dev environments. Test both initial data copy and change-data-capture (CDC). Validate forward transformations that reshape data for the new model. Iterate on performance tuning for both migration speed and target database configuration. The goal is to know how fast you could go (and what that would require), and how fast you need to go in order for the migration to be successful.

Stream 4: Reverse Sync Testing - If your strategy includes reverse sync as a safety measure (writing changes from the new database back to the old), test these reverse transformations thoroughly.

Phase 4: UAT Testing

With a UAT environment approximating production size (often 30-50% of production data), spend 1-2 months conducting comprehensive functional testing, performance testing under realistic load, capacity planning for the target database, and migration timing validation.

UAT often reveals data model artifacts that weren’t present in smaller dev datasets. Test data that’s obsolete or doesn’t reflect current usage patterns can create misleading results, so work with real or recently sanitized production data when possible.

Phase 5: Production Dry Run

This crucial phase - often 3-4 weeks or longer - involves executing the full migration against production data without cutting over traffic. Go all the way, and run your application and batch jobs against the migrated data to validate correctness. Conduct complete integration testing with all dependent systems.

Perform extensive data validation comparing source and target. Measure impact on the source database during migration, migration speed and bottlenecks, and target database performance under production query patterns.

For high-risk migrations, consider an extended dry run where the target remains continuously in sync with the source for weeks or months while you compare API responses between systems to build confidence.

Production data often contains artifacts from old records, inconsistent data types, and edge cases that never appeared in testing. Discovering these issues in the dry run is expensive - requiring changes to propagate back through dev and UAT - but far better than finding them during the actual cutover.

Ensure to document all the steps, their timings and performance observations. Map out various edge cases (e.g. the destination or the source become unavailable for a short period of time, migration tool is interrupted, etc), test them and their impact on the process. For example, if your migration tooling or process isn’t resumable, requires downtime, and fails toward the end of your downtime window, you may as well call it. Don’t just test the happy path only.

Phase 6: Cutover Preparation

Based on dry run learnings, build a detailed cutover plan specifying every action down to the minute. Allocate resources ensuring the right people are available during the cutover window. Be wary of holidays and other high-risk periods. Pick the quietest time possible based on traffic patterns. Prepare user notifications for internal and external stakeholders.

Note from experience: when final cutover steps are dependent on DNS updates, it’s best to rehearse that in advance. DNS updates bring the Internet down.

This phase typically requires 1-2 weeks. For complex migrations it’s prudent to prepare a backup cutover window.

Phase 7: Production Migration and Cutover

When everything is properly planned, the actual cutover is often anticlimactic - a 2-day window (or even less) of executing well-rehearsed steps. Continuous communication and pre-defined escalation paths ensure the team can respond quickly to any unexpected issues.

Phase 8: Post-Migration Monitoring and Optimization

For at least a month after cutover, maintain heightened monitoring. Watch for performance degradation, data inconsistencies, unexpected errors, and capacity issues. Commonly forgotten items - like indexes needed by infrequent batch jobs or new application endpoints - often surface during this period. Continue optimization based on real production traffic patterns.

6. Validation Strategy

Data integrity is paramount. Your validation strategy should include multiple layers of verification.

Data Integrity Checks

Implement automated validation scripts that compare row counts across tables, calculate and compare checksums for data blocks, sample random records for detailed field-by-field comparison, and validate referential integrity and constraints. Run these checks repeatedly - after initial migration in Dev and UAT, during the dry run, and again after cutover.

Application Testing

Beyond data validation, verify that your application functions correctly: functional testing of all critical user journeys, performance testing to ensure response times meet SLAs, integration testing with all dependent systems, and end-to-end testing of business workflows. Automated test suites accelerate this process and provide regression protection.

Rollback Procedures and Criteria

Define clear criteria for success or rollback: What data inconsistency rate is acceptable? What level of performance degradation triggers rollback? How long can resolution take before you must roll back?

Document and rehearse rollback procedures. In a dual-write scenario, rollback might mean redirecting read traffic back to the old database. For migrations with downtime, you may need to rely on reverse sync. Without reverse sync, you would need to accept a certain loss if you go back.

From 100’s of migrations that we’ve been involved in, we only saw a handful where a rollback was actually triggered - if all previous steps were properly followed, the rollback scenario is merely a theoretical possibility. However, if the conditions for it are not well defined, it can add very real stress during the critical cutover time window.

Monitoring and Alerting Setup

Establish comprehensive monitoring before cutover including database performance metrics, application error rates, API latency percentiles, data consistency checks, and business metrics like transaction volumes. Configure alerts with appropriate thresholds to catch issues quickly.

Configure monitoring for migration tooling - you should have full visibility into its progress, ETA for completion, and any errors.

In our next post in this series we will discuss common complications that affect even the best planned migrations, and migration best practices.

Download and try Dsync for your migration on GitHub. We built it to make your migration experience seamless.

[Part 2] Production Database Migration or Modernization: A Comprehensive Planning Guide

Alexander Komyagin — Sat, 31 Jan 2026 03:31:29 GMT

This is a second part of our multi-post guide that walks through the essential components of planning and executing a successful production database migration for large-scale backend services.

If you haven’t read the first part, where we cover Migration Readiness Assessment and the Six Key Factors Influencing Timeline and Risk, you can find it in Part 1.

Migration performance and risks need to be carefully balanced through proper downtime strategy and optimal tool selection

3. Downtime Strategy Options

Your downtime strategy fundamentally shapes your migration approach and timeline.

Some Downtime (Planned Maintenance Window)

This traditional approach involves scheduling a maintenance window - typically several hours to a full day - during which services are completely or partially unavailable (e.g. read-only mode) while you migrate data and cut over to the new database.

Pros: Simplest to implement, lowest risk of data inconsistencies, easier validation, straightforward rollback if issues arise.

Cons: Service interruption impacts users, requires careful timing and communication, may not be acceptable for globally distributed services.

Timeline: Shortest overall project timeline but concentrated downtime during cutover.

Best for: Applications with natural low-traffic periods or off-hours (e.g. insurance, banking), internal tools, or when business requirements permit scheduled maintenance.

For large databases or network-constrained environments, it’s important to ensure that the migration window covers at least 1.5-2x actual data migration time to account for variability in data ingestion and minor deviations from the original plan, such as transient network or database issues.

Although most companies immediately discount this old-school approach because they expect the downtime period to be long, many modern databases coupled with highly-parallelized and scalable migration tooling can easily make migrating many TBs and Billions of records a matter of hours rather than days.

Minimal Downtime (Quick Cutover)

Migrate the bulk of data, continuously sync the deltas (starting from the point in time right before the initial data copy to avoid losing data), then execute a brief cutover window (minutes to an hour) to sync final changes and switch traffic to the new database.

Pros: Significantly reduced user impact, most data migrated without affecting production, maintains business continuity.

Cons: Requires CDC or careful change tracking, more complex orchestration, higher stakes during cutover window.

Timeline: Medium complexity; bulk migration can happen over days or even weeks while services run normally.

Best for: Services that can only tolerate brief interruptions during low-traffic periods.

It’s worth noting that blending CDC (change-data-capture) and initial data copy requires two distinct data processing paths for the migration. They need to be well coordinated and individually tested from the data integrity and performance perspectives. The CDC mechanism should be fast enough to catch up with all the changes within a reasonable and predictable timeframe, and continue replicating them in near-real-time. The initial data copy should be fast enough to reduce the catch up period and give the CDC part the best odds to succeed.

Some modern database migration tools implement the above already, but the burden of proper testing and evaluation always falls on the end user.

Zero Downtime (Dual Writes, Gradual Migration)

Implement dual writes via a dedicated DAL (Database Abstraction Layer) where all changes go to both old and new databases simultaneously, then gradually shift read traffic to the new database after validating data consistency. The bulk of data is backfilled in the background via a separate process.

Pros: No user-facing downtime, extensive validation possible before cutover, gradual rollout reduces risk.

Cons: Most complex implementation requiring significant code changes, increased operational complexity and monitoring during transition, potential for data inconsistencies between systems, extended timeline with both systems running in parallel.

Timeline: Longest overall timeline but eliminates concentrated downtime.

Best for: Mission-critical services where any downtime is unacceptable, services with global user bases across time zones.

Key challenges specific to this approach:

Consistency: Dual writes can cause data drift between the source and the destination unless each write has a transactional semantic and the writes order is strictly maintained for each record across both systems. Even if the business side is ok with some drift, no guarantees can be made here - the only practical approach is testing and measuring the drift.
Race Conditions: Unless the write workload is insert-only, the abstraction layer needs logic to prevent backfills from overwriting fresh data, and the destination write path should be able to handle possible conflicts.
Data Anomalies: Production data often has “surprises” that only show up after the move unless the application code has been tested with the post-migration data - this is best done as part of the production dry-run.

Migration in Batches

For complex systems, consider migrating in phases (“blue-green” approach) rather than all at once.

Multi-tenant Systems: Group tenants logically (by size, activity level, or business relationship) and migrate each group separately. This requires specialized middleware and feature flags to route requests to the appropriate database based on the tenant. Starting with smaller, lower-risk tenants allows you to refine your process before migrating larger customers.

Multi-service Systems: In a microservices architecture, identify independent groups of services and migrate them separately. Services with fewer dependencies make ideal starting points. This incremental approach reduces blast radius and allows learning from each migration.

A variation of this approach is to split the migration by logically separate and independent groups of schemas/tables/collections.

4. Migration Approaches and Tools

Selecting the right tooling is critical for success as tools underpin the execution of any of the chosen approaches.

Vendor-Provided Tools

Cloud providers and database vendors often offer native migration services: MongoDB Atlas Migrations, AWS Database Migration Services from Azure, AWS and Google.

These tools integrate seamlessly with their respective cloud ecosystems and support a range of source and target database combinations.

Vendors like MongoDB offer specialized tooling like mongosync and MongoDB’s Relational Migrator.

Pros: Easy to set up, vendor support.

Cons: The tools have very limited scope and only cater to the lowest 50th percentile of workloads. Most of them are SaaS-based and move data over the public internet.

Best for: Workloads of average or below-average size and complexity, with little or no special requirements.

Specialized Third-Party Solutions

Tools like Adiom’s Dsync, Oracle’s Golden Gate and Fivetran provide advanced capabilities for complex migrations. Dsync, for example, excels at NoSQL and RDBMS to NoSQL migrations with sophisticated transformation capabilities, modular extensions, and real-time CDC for minimal downtime.

Pros: Optimized tooling for repeatable and scalable execution of migrations with minimum or no-downtime. Easy to operationalize. Specialized expertise and support.

Cons: Your company needs to onboard another vendor unless the solution is available in Open Source or the tool provider has a partnership with the destination database vendor.

Best for: Large-scale and mission-critical migrations. Complex transformations (especially RDBMS to NoSQL). Scenarios where specialized support and expertise add significant value.

Custom Scripts and Open-Source Tools

Apache Spark for data processing, Debezium for CDC from various databases, Kafka Connect for streaming data pipelines, and custom ETL scripts give you maximum flexibility. Modern AI-tools like Copilot and Claude Code can help you to quickly create data migration scripts.

Pros: Full control of the process with no overhead of working with vendors.

Cons: Limited observability and scalability. Reliability and resumability are hard to implement without dedicated design reviews and testing.

Best for: Straightforward one-way migrations for small datasets (10’s of GBs). Can be used for larger and more complex migrations when your company has the necessary expertise and can allocate resources to the project.

Selection Criteria

Choose tools based on your source and target database types, data volume and transformation complexity, downtime tolerance, team expertise, budget, and whether you need ongoing support. In many cases, a hybrid approach combining vendor tools or scripts for smaller or less-critical databases or schemas along with specialized tooling for large critical ones yields the best results.

At Adiom, our goal is to give customers the best of both worlds. We offer specialized database migration solution Dsync and we closely partner with database vendors like Microsoft and MongoDB to reduce friction and risk, and to accelerate timelines for workload migrations.

In our next post in this series we will discuss migration planning, timelines and common complications.

Download and try Dsync for your migration on GitHub. We built it to make your migration experience seamless.

[Part 1] Production Database Migration or Modernization: A Comprehensive Planning Guide

Alexander Komyagin — Mon, 26 Jan 2026 05:00:28 GMT

Migrating a production database supporting critical backend API services is one of the most challenging undertakings in software engineering. Whether you’re modernizing from a legacy relational database to a NoSQL database like MongoDB, moving to a cloud-native solution like Azure Cosmos DB or AWS DynamoDB, or simply upgrading your database to a newer version, the stakes are high. A poorly executed migration at its worst can result in data loss, extended downtime, revenue impact, and erosion of customer trust. Not even mentioning frustration of internal stakeholders!

Database migrations are often a nail-biter - but they don’t have to be.

Commonly, migration timelines extend 4-6x longer than originally anticipated due to poor preparation, planning and internal coordination. This extension drives up not only costs, but also uncertainty and risks associated with other projects that are impacted by the migration.

This multi-post guide walks through the essential components of planning and executing a successful production database migration for large-scale backend services.

1. Migration Requirements & Assessment

Before diving into technical implementation, establish a clear foundation for your migration project.

Business Requirements and Success Criteria

Define what success looks like. Are you migrating to reduce costs, improve performance, enable new features, or meet compliance requirements? Establish measurable criteria such as maximum acceptable downtime, target performance metrics, data integrity requirements, and budget constraints. These objectives will guide every subsequent decision.

Current Database Assessment

Conduct a thorough analysis of your existing database including total data volume, number of tables and relationships, read and write query patterns and hotspots, current performance baselines, backup and recovery procedures, and existing monitoring and alerting. Understanding your starting point is critical for accurate planning.

Target Database Selection and Justification

Choose your target database based on your specific requirements rather than trends. Consider factors like workload characteristics (OLTP vs. OLAP), scalability requirements, consistency vs. availability tradeoffs, operational complexity, team expertise, and total cost of ownership. Document your rationale for stakeholder buy-in.

Codebase Assessment

Analyze how your application interacts with the database, including the code embedded server-side in the database in the form of stored procedures and triggers. Map your data models, access patterns, and API endpoints. Identify which endpoints are read-heavy versus write-heavy, which queries are most complex, and where performance bottlenecks exist. Understanding the coupling between your application and database will reveal the scope of code changes required.

Integrations and Batch Jobs

Review and document integration with other systems, data pipelines and other in-flows and out-flows. Map the workflows, workload types, and dependencies on the data model.

Infrastructure and Networking

Carefully review the infrastructure required for the migration: available headroom on the source database, possible bottlenecks on the network path between the source and the destination, as well as the type of the connection that will be used (Public IP/VPN/PrivateLink). Additionally, investigate elasticity in the destination database to accommodate the large ingestion job for initial data copy. Ideally, it can be temporarily upscaled.

Stakeholder Alignment and Communication Plan

Database migrations affect multiple teams. Identify all the relevant stakeholders and establish a communication plan that keeps engineering teams, DevOps, product management, customer support, and executive leadership informed throughout the process. Regular updates on progress, risks, and timeline changes are essential for maintaining alignment.

2. Six Key Factors Influencing Timeline and Risk

While many factors might affect a migration, we identified six as the most critical impacting both the duration and risk profile of your migration.

1. Data Volume and Complexity

A 10GB database might migrate in less than an hour, while a multi-terabyte database could take weeks. Long data migration jobs increase risks exponentially, as they prolong your exposure and make testing and migration attempts more expensive. Data complexity - including data types and constraints - compounds the challenge beyond raw size, especially when moving to a different vendor where not all of them may be supported. Last but not least, large data sets often hide legacy records from an old application version or production testing. Those legacy records are sometimes malformed and cause unexpected problems during the production migration.

2. Existing Abstraction Layers

The presence and quality of your Object-Relational Mapping (ORM) layer or Data Access Layer (DAL) significantly affects migration difficulty. A well-designed abstraction layer allows you to swap database implementations with minimal application code changes. Conversely, SQL queries scattered throughout your codebase create a massive refactoring challenge.

3. Change Data Capture Capabilities

CDC is crucial for minimizing downtime and enabling zero-downtime migrations. If your source database supports CDC, you can continuously replicate changes during the migration process. Without CDC, you’re limited to snapshot-based approaches that require longer downtime windows. Note that some databases don’t support native CDC capabilities at all (e.g. Cassandra) and some support multiple options (e.g. LogMiner or XStream in Oracle). It is possible to get CDC-like capabilities with Change Tracking - whether native like in SQL Server or via a special version or lastModified field in each record. Different CDC approaches have specific requirements like installing software on a database server, maintaining triggers, and they all offer different performance profiles. It’s critical to establish that your CDC mechanism offers at least 2-3x your maximum data velocity.

4. Database Schema Complexity and Data Model Changes

Migrating from a normalized relational model to a denormalized NoSQL model requires careful planning. Every join that previously happened in the database must be rethought. Consider how you’ll handle transactions, maintain data consistency, and model relationships in the new paradigm. Review required data transformations from the feasibility perspective, and analyze the changes required in side jobs.

5. Application Architecture and Database Coupling

Monolithic applications tightly coupled to their databases present greater challenges than microservices with clean separation of concerns. Assess how deeply database-specific features (stored procedures, triggers, custom types) are embedded in your architecture.

Monolithic applications generally require longer testing cycles and more coordination for changes. Notably, if the code in a monolithic application isn’t frozen, it’s common for migration-related changes to pile-up behind active feature and bug-fix work.

6. Data Migration Tooling

Data migration is the core activity and the foundation for any database migration or modernization project. A solid foundation is built on tooling that enables repeatable, reliable, fast and predictable execution. For a small database, a simple script might suffice. But large and mission-critical databases require purpose-built and supported solutions, that have flexibility to accommodate your specific requirements.

Look forward to the next post in the series! There we will discuss migration approaches and downtime strategies.

Download and try Dsync for your migration on GitHub. We built it to make your migration experience seamless.

Cost-effective vector embeddings storage with AWS S3 Vectors

Alexander Komyagin — Wed, 14 Jan 2026 17:16:35 GMT

Vector pipeline (courtesy of Vercel)

Ever since vector stores and databases took the database market by storm in 2023-24, we ended up with a plethora of specialized vector databases such as Pinecone, Weaviate, Milvus, Qdrant. Many of OLTP databases quickly jumped on the train and added support for vector indexes - notably MongoDB and PostgreSQL - but the actual embeddings are still stored along with other operational data, competing for CPU, RAM and storage resources.

Since then it became apparent that the embeddings workload is significantly different from the regular OLTP. Vectors are stored en-masse, and most of them are accessed relatively infrequently or never at all. Storing that kind of data and serving it from your OLTP system is prohibitively expensive - it’s not enough to just build a great vector index, but the cost/performance ratio has to be good.

As the market is adapting, a new category of players and open formats has emerged - those based on object storage. Turbopuffer, LanceDB, and now native S3 Vector Indexes. We’re especially excited about the latter, as it dramatically simplifies the operational stack, and we hope that it will live up to the expectations.

S3 Vector bucket

For users wanting to try it out and export their existing vectors away from pgVector and other solutions, Dsync now supports S3 Vectors as a destination. The feature is currently in “Public Preview”. Please feel free to share your feedback, ideas, and requests for help via our Contact Form or our Discord Channel.

Dsync is the most effective and seamless solution for data mobility between different databases and data stores (check out the full list of supported connectors in the Dsync docs!). To use Dsync to migrate your vectors from pgVector to S3 Vectors, you need to create a new vector bucket and then a new vector index:

Creating a new S3 vector bucket

Creating a new S3 vector index

Make sure to set the same number of dimensions as in your source embeddings. In my test setup, I have 3-dimensional vectors in the “embedding” column of the “public.items” table:

zsh% psql postgresql://XX:YY@localhost:5432

postgres=# select * from items;

 id | embedding

----+-----------

  1 | [1,2,3]

  2 | [4,5,6]

(2 rows)

Once the vector index is set up, migrating the embeddings to your new S3 Vector Index with Dsync is as easy as a single command, provided you already logged into AWS in your CLI via “aws sso login”, and downloaded and built Dsync from our repo:

./dsync --mode=InitialSync --ns "public.items:alex-s3vec-index" postgresql://XX:YY@localhost:5432 s3vector --bucket=alex-s3vec-test --vector-key=embedding

Lastly, the AWS S3 Vectors is still a new feature, so please take a minute to go over the

limitations, especially the rate limits. The Dsync S3 Vectors connector supports “–rate-limit” and “–batch-size” options that you can adjust:

./dsync --mode=InitialSync --ns "public.items:alex-s3vec-index" postgresql://XX:YY@localhost:5432 s3vector --bucket=alex-s3vec-test --vector-key=embedding --rate-limit 3000 --batch-size 500

Dsync can also retrieve live changes from the source database and update S3 vectors in real-time via CDC, rather than just doing a one-time sync (skip the “–mode=InitialSync option”):

./dsync --ns "public.items:alex-s3vec-index" postgresql://XX:YY@localhost:5432 s3vector --bucket=alex-s3vec-test --vector-key=embedding

Happy migrating!

Dsync Docs

Contact Form

Discord Channel

Announcing DocumentDB support: an Open Source MongoDB compatible Database

Alexander Komyagin — Tue, 16 Dec 2025 04:55:03 GMT

At Adiom, we firmly believe in the power of true Open Source Software to advance the ecosystem and enable unhindered innovation. We’re excited to announce full Azure DocumentDB support as a source and as a destination, including the managed and the self-hosted versions.

Azure DocumentDB is built on PostgreSQL and uses custom extensions for enabling MongoDB compatibility. It lacks certain features, namely Change Streams, and in general can be considered as lagging several versions behind MongoDB’s main branch, but with AWS and Microsoft joining forces on this one, we believe that the gap will start shrinking quickly.

While AWS already has a product with the same name that has MongoDB compatibility, Microsoft took a very different approach and invested in their solution becoming a contribution to OSS and Linux Foundation.

If you want to replicate or migrate your database to DocumentDB from MongoDB or another database, the process is as seamless as a single command, whether you have 100 documents or 1 billion:

dsync $SOURCE_URI $DOCUMENTDB_URI

If you want to see progress in your CLI rather than the Web version:

dsync --progress --logfile dsync.log $SOURCE_URI $DOCUMENTDB_URI

More options and supported sources can be found in our documentation.

Dsync takes care of parallelization, write batch optimization, reporting and automatically coordinating real-time CDC data capture with the initial data snapshot while maintaining data integrity. For large (>100GB) and mission-critical deployments, Dsync has a horizontally-scalable Enterprise version that features resumability and OpenTelemetry-compatible observability.

Note that for a MongoDB source to work with CDC, it needs to be a replica set or a sharded cluster. Otherwise you can pass the “–mode InitialSync” option to dsync to skip CDC.

https://documentdb.io/docs

https://www.youtube.com/watch?v=vdB7fZvQC-Y

Try dsync:

https://github.com/adiom-data/dsync/

Introducing S3 Support in dsync: Lightning-Fast Direct Transfer

Alexander Komyagin — Wed, 10 Dec 2025 00:26:50 GMT

We’re excited to announce that dsync now supports Amazon S3 as both a source and destination. This opens up powerful new possibilities for data migration workflows.

What This Means for You

With S3 support, you can now export data directly from any supported connector - including DynamoDB, Azure Cosmos DB, MongoDB, and PostgreSQL - straight to S3, or import it back just as easily. The process is:

Direct: No intermediate storage required
Fast: Fully parallelized operations
Simple: Clean, intuitive command structure

Your data is stored in S3 as organized JSON arrays within .json files, using a namespace-based directory structure that’s both human-readable and machine-friendly.

Quick Example

Exporting two DynamoDB tables to S3 is as simple as:

./dsync --mode InitialSync --namespace "table1,table2" dynamodb s3://test

This creates the following structure in your s3://test bucket:

table1/
├── .metadata.json
├── xxxxxx.json
├── yyyyyy.json
└── ...

table2/
├── .metadata.json
├── aaaaaa.json
├── bbbbbb.json
└── ...

The .metadata.json file tracks record counts in each JSON file, enabling progress reporting during imports and quick validation of your data.

Flexible Imports

When importing from S3, dsync reads .json files from your specified path. The folder structure automatically maps to namespaces, even with nested directories. Best of all, imports work with any valid JSON array files - not just those created by dsync.

./dsync --mode InitialSync s3://test dynamodb

Want to validate your data without writing anywhere? Use the special /dev/null connector:

./dsync --mode InitialSync s3://test /dev/null --log-json

Why We Built This

This feature emerged from a real need: we required a robust solution for retaining large volumes of data from our internal deployments. After evaluating existing S3 export/import tools, we found them universally inadequate. Common issues included:

Requiring local disk dumps as intermediaries
Painfully slow performance
Complex, brittle automation

Whereas native DynamoDB export to S3 using AWS console took close to 9 hours, dsync took less than 10 minutes!

We built the S3 dsync connector to solve these problems for ourselves, and we’re thrilled to share it with our customers and the community.

Fine-Tuning Your Exports

The S3 connector offers several options to customize behavior:

./dsync --mode InitialSync --namespace "table1,table2" dynamodb s3://test [options]

Available options:

--pretty-json – Format output files for readability
--max-total-memory – Control memory usage (default: 100MB)
--max-file-size – Set maximum file size in S3 (default: 10MB)

Prefer CLI progress output? Add the –progress and –logfile flags:

./dsync --progress --logfile /tmp/dsync.log --mode InitialSync \
  --namespace "table1,table2" dynamodb s3://test

Learn More

For complete documentation on dsync and its capabilities, visit our documentation.

Happy exporting-importing!

Accelerating Migrations to Cosmos DB NoSQL: White Paper

Alexander Komyagin — Fri, 07 Nov 2025 19:46:59 GMT

Azure Cosmos DB for NoSQL is a general JSON database with a rich SQL-like interface, multi-region deployments (including multi-region writes), and strong availability guarantees. Cosmos DB has a usage-based billing model and has high elasticity - the database can dynamically scale or auto-scale from next-to-0 to ChatGPT.

One of the most common migration paths to Cosmos DB is from AWS DynamoDB. DynamoDB is a massively scalable key-value store used internally in AWS for many vital services. While both DynamoDB and Cosmos DB are serverless NoSQL databases, they have a lot of differences: data model, access patterns, multi-region deployment options, consistency guarantees.

For users wanting to explore Cosmos DB or migrate their production workloads from DynamoDB or other databases, Dsync is a perfect solution that makes the process seamless. You can migrate 100’s of GB’s of data in a few hours with one simple command. For smaller databases, you can use our Open Source single-binary version. For large migrations and production deployments you can use our scalable Enterprise Dsync (Dsynct).

In this white paper we review our approach for online database migrations to Azure Cosmos DB for NoSQL, and we specifically examine the migration path from AWS DynamoDB. The paper covers technical details and the architecture of the migration solution with Dsync, as well as its practical advantages that accelerate and derisk migration projects.

Migrate RDBMS to MongoDB

Alexander Komyagin — Tue, 28 Oct 2025 22:43:02 GMT

We’re excited to announce the newest addition to our connector family: the “SQL batch” connector. Dsync with the SQL batch connector makes RDBMS migration to NoSQL significantly easier, while upholding Dsync’s standards of consistency and reliability at scale. By leveraging our Migration AI agent, it streamlines data transformation and the migration process.

The SQL batch connector is now available in Private Preview. Contact us to try it out.

Dsync sets a new standard for online database migrations and real-time replication, combining scalability, reliability, ease of use, and unmatched flexibility.

Connector

The SQL batch connector lets you specify a custom SQL query on the source database, similar to creating a virtual table. This is especially useful for migrating relational data to MongoDB or other NoSQL databases, where you may need to reshape data to take advantage of the document model.

Key features:

Supports custom SQL queries
Supports JSON aggregation in modern SQL engines
Utilizes advanced JOIN capabilities
Compatible with both SQL Server and PostgreSQL as sources (including Initial Sync and CDC)
Seamlessly integrates with the Dsync platform and transformer
Offers task-based parallelization and resumability
Users can use MongoDB Relational Migrator for schema mapping and import it into the AI workflow.

AI-enabled database migration from SQL to MongoDB

Example: Denormalizing TPC-H Dataset

Source: PostgreSQL with 4 CPU and 34 GB RAM (AlloyDB on GCP)

Destination: MongoDB v8 Sharded Cluster

Dataset: 30GB, ~100 million rows (TPC-H dataset)

The TPC-H dataset is highly normalized, containing customer, order, part, and supplier data. For our MongoDB target schema, we denormalize the data to optimize queries for customer orders and specific parts.

TPC-H schema mapping

Generate schema mapping with MongoDB’s Relational Migrator.

Schema mapping in MongoDB Relational Migrator

Export the project as a .relmig file and feed it to our Migration AI Agent (based on Factory AI’s Droid - help us name it! Does Migroid sound good?).

Migration AI Agent

The AI agent generates a ready-to-use YAML config for the SQL batch connector, which you can use with Dsync (single-binary or Enterprise) or customize further:

dsync -ns "orders:tpch_dsync.orders,customer:tpch_dsync.customer,part:tpch_dsync.part" sqlbatch --config tpch_migration.yaml <MONGODB_URI> <TRANSFORMER>

Dsync Web Progress

The migration produces the desired document structures in MongoDB. For example, here is the “orders” collection:

Target order document in MongoDB

Performance analysis

Test Setup:

Source: PostgreSQL (4 CPU, 34 GB RAM, AlloyDB on GCP)
Destination: MongoDB v8 Replica Set (4 CPU, 16 GB RAM)
Dsync VM: 4 CPU, 16 GB RAM and 8 CPU, 32 GB RAM

Results:

Initial sync with Dsync completes in 40 minutes or less, using 20–25% of source PostgreSQL CPU (takes 20 minutes with 8 CPU Dsync VM, using 60% of source CPU).
For production, we recommend using a read replica or running migrations during off-hours.
Relational Migrator took 6–8 hours and failed with a connection error at the very end in our runs, despite the source being available and operational.

Comparison of run times and status

MongoDB Relational Migrator Error

Dsync proved to be 10–20x faster and more reliable than MongoDB Relational Migrator. Despite our initial concerns about running aggregations on the source, this approach has proven to have a manageable impact and offers high flexibility - especially when used with a transformer. It is also more performant than alternatives like constructing final objects on the destination and more practical than building a streaming join engine. Additionally, it allows us to maintain the reliability and integrity standards of the Dsync platform.

Give it a try

The SQL batch connector is now available in Private Preview. Contact us to request access. In upcoming posts, we’ll cover CDC functionality for the SQL batch connector.

Dsync Now Available on Microsoft Azure Marketplace

Alexander Komyagin — Fri, 17 Oct 2025 16:32:17 GMT

We’re thrilled to announce that Dsync is now available in the Microsoft Azure Marketplace, making it easier than ever for Azure customers to deploy, manage, and scale their data migrations and replications.

Dsync helps enterprises eliminate the complexity and risk of database migrations, offering a secure, high-performance solution that seamlessly moves production workloads between NoSQL and SQL databases - including to Azure Cosmos DB. With built-in resiliency, resumability, and end-to-end data integrity, Dsync transforms what was once a painful, manual process into a fast, reliable, and repeatable experience.

“Making Dsync available in the Azure Marketplace enables us to reach more organizations looking to simplify and accelerate their data migrations,” said Alexander Komyagin, CEO of Adiom. “Our mission is to make data movement effortless, reliable, and secure - no matter where it runs.”

This milestone strengthens our collaboration with Microsoft and expands the options available to enterprises building on the Azure platform.

👉 Get started today — Visit Dsync in the Azure Marketplace and learn how Adiom can help your organization move data faster, safer, and smarter.

Full Press Release

Documentation

100TB HBase to MongoDB in Under 48 Hours: Building Support for Large-Scale Online Production Migration

Alexander Komyagin — Tue, 09 Sep 2025 18:55:52 GMT

Modern enterprises often find themselves needing to migrate massive production datasets between different database technologies. Recently, we tackled one of the most challenging scenarios: migrating over 100TB of data containing 100+ billion records from HBase to MongoDB while maintaining active production traffic. In this article we explore how we built the technical foundation to make this possible.

The Challenge: Scale Meets Complexity

The environment presented several unique challenges:

Massive scale: 100+ TB of data across billions of records in an active HBase cluster
Timeline: Migration should complete in a reasonable amount of time
Zero tolerance for extended downtime: Production systems can’t wait months for migration
Complex data transformation: Converting HBase key-value pairs to MongoDB’s Extended JSON format
Production reliability requirements: The migration solution itself must be enterprise-grade

HBase Architecture Primer

HBase Architecture Diagram

HBase is a horizontally scalable column-oriented key-value store. Records are organized into tables. Each record or row is identified by its row key, which has to be unique. The data for that row is represented as key-value pairs, where the key is a column name, and the value is arbitrary binary data (byte[]) - it can be a primitive data type like string, an encoded JSON document, an image, or something else. The columns are grouped into column families which are stored into separate files in the underlying HDFS cluster.

For scalability, HBase uses Region Servers, where each Region Server hosts data for a set of particular ranges of row keys that are called Regions. For example, a production HBase cluster can have 32 Region Servers with 256 total Regions, meaning that each Region Server is responsible for 8 Regions.

MongoDB Architecture Primer

MongoDB Sharded Cluster and JSON example

MongoDB is a horizontally scalable document-oriented database. MongoDB documents are records in Extended JSON (or BSON), uniquely identified by the _id key. The documents are organized in collections that are logically grouped into databases.

MongoDB supports high availability with the Replica Set architecture. For horizontal scalability MongoDB leverages sharding and Sharded Clusters, where the data is distributed across two or more Replica Sets (Shards) based on ranges (“chunks”) for a particular Shard Key.

MongoDB Atlas is a fully managed cloud database service that simplifies deploying, scaling, and managing MongoDB databases in the cloud.

Technical Requirements for HBase to MongoDB Migration

Building a system capable of handling this scale required addressing seven critical requirements:

Minimize Downtime: We implemented a CDC (Change Data Capture) stream coordinated with initial data copy to ensure no changes are lost during migration.

Scale and Performance: With billions of records, every inefficiency scales exponentially. The migration data path required optimization down to the microsecond level.

Repeatability: Large modernization efforts involve iterations across development, UAT, and production environments, demanding consistent, predictable outcomes.

Resumability and Resilience: Production migrations must handle failures gracefully without restarting from scratch.

Reverse Sync: A cooling-off period after migration requires syncing changes back to HBase with fallback capability.

Observability: Real-time progress tracking, ETAs, and performance bottleneck identification are essential for production planning.

Throttling: Impact on the source system serving live traffic must be carefully controlled.

Solving the HBase CDC Challenge

HBase’s architecture presented a unique technical hurdle: it doesn’t support pull-based change streams and only offers push-based replication. Our solution leveraged HBase’s existing replication mechanism with a creative twist:

We created a “fake” HBase cluster with a modified write path that writes to Kafka instead of storage
The original HBase cluster replicates to this fake cluster using standard HBase replication
This architecture transforms HBase’s push-based replication into Kafka’s pull-based change stream
Multiple Kafka topics with partitioned streams support high CDC throughput

The Dsync Architecture

Our solution centers on a scalable enterprise-grade architecture built with several key components:

Scalable Dsync Architecture

Temporal Workflow Engine: Provides durable workflow orchestration for complex, long-running migration processes.

Task-Based Parallelization: Work splits into tasks based on source HBase regions, with multiple dsync workers processing tasks in parallel.

YAML-Based Transformations: Data transformations and filtering are defined in YAML using CEL (Common Expression Language), making complex data model adjustments manageable at scale.

Automatic CDC Switching: After initial data copy completion, the system automatically switches to replicating CDC changes from the pre-copy timestamp, ensuring zero data loss.

Lightweight deployment: Even when deployed at scale, Dsync only uses compute (CPU + RAM) and doesn’t require persistent storage, making it easily deployable on commodity VMs or containers.

Observability and Reporting: Dsync provides an intuitive web-based real-time progress report along with granular metrics via OpenTelemetry.

Dsync progress report

Detailed metrics in SigNoz via OpenTelemetry

Performance Results

Our technical development has achieved impressive benchmarks:

2 million writes/second replication capability to a 7xM200 destination Atlas cluster
24-48 hour migration window for 100 billion records
Efficient resource utilization: 10 worker nodes (16 CPUs, 64GB RAM each) plus 1 coordinator node (8 CPUs, 32GB RAM)

Best Practices for Large-Scale Migration

Through this development process, we’ve identified key practices for successful large-scale database migrations:

Plan for Iterations: Design for multiple runs across Dev, UAT, and production environments with several dry runs before go-live.

Prioritize Repeatability: Use migration solutions that enable fast, consistent execution across different environments.

Balance Accuracy and Speed: Develop data reconciliation strategies that acknowledge the trade-offs inherent in large-scale operations.

Modularize Workflows: Split large tables into separate workflows for easier operational management.

Size Appropriately: Revise production sizing based on actual object sizes discovered during development and testing phases.

Follow MongoDB Ingestion Optimization: Overprovision destinations, disable backups during migration, set oplog to fixed size, and build indexes during the CDC phase rather than initial load.

Looking Forward

The HBase connector is now available in preview - a major breakthrough for large-scale database migrations. This technology helps companies modernize their databases on a short and predictable timeline, and without compromising operational stability.

By combining Temporal workflows, intelligent CDC, and highly parallel processing, we’ve built a platform that can handle even the toughest migration challenges. We’re continuing to improve dsync, making migrations that were once impossible now achievable and reliable.

Make sure to check out the demo video of the migration process:

The HBase connector is available in preview. For more information about the scalable deployment and dsync, please consult our documentation. If you need assistance with HBase or other database migrations, please contact us.

Database migrations. Done Right

Alexander Komyagin — Wed, 11 Jun 2025 21:40:50 GMT

We recently visited Microsoft Reactor to talk about database migration paths, common pain points and best practices. Our CEO, Alex, shared his firsthand experiences and customer stories. While the session was focused on migrations to Azure Cosmos DB, most learnings are transferable to other migrations paths:

Pick a tool that makes the process easy, fast and repeatable
- Paradox: a 10-hour migration is much longer than 10x 1-hour migrations
Assign a migration “champion”
- One technical person responsible for the planning and project execution
Get a one-time prod data load as soon as possible to the destination for testing
- Can be a redacted sample. But needs to be truly representative in terms of size and form of the data that you will be migrating
Keep things as simple as possible with few moving parts in the process.
- Doing something complex might be exciting and is good material for a blog post, but in production best things are usually boring.
Do dry-runs. Lots of them.
- If your dry run is a long and painful process with an uncertain outcome, your production run will be even worse.
Over-provision on the destination as much as possible for the migration.
- Write throughput on the destination is typically the biggest bottleneck for large datasets.

Adiom is Microsoft’s trusted partner for migration solutions that enable seamless online database migrations from MongoDB to Cosmos DB vCore, and DynamoDB to Cosmos DB for NoSQL.

For more information on these migrations, visit the Quickstart page in our documentation: https://docs.adiom.io/getting-started/quickstart

Session recording and slides:

Migrating MongoDB Data to FerretDB with dsync

Alexander Komyagin — Sat, 19 Apr 2025 02:05:44 GMT

This is a slightly modified version of the article on FerretDB’s blog.

FerretDB is an open-source alternative and a drop-in replacement for MongoDB. FerretDB implements MongoDB’s wire protocol as a proxy layer over PostgreSQL with DocumentDB extension as the backend. You can read more about it in the GitHub repo.

For users looking to migrate data from MongoDB to FerretDB, ensuring a smooth transition is crucial.

Traditional migration workflows often rely on static dumps and manual restoration steps, which can often lead to complications. Skipped collections. Metadata mismatches. Data loss. And worst of all – no clue where things went wrong.

dsync by Adiom is a tool that connects directly to both the source and destination of MongoDB-compatible services and streams data in real time. It seamlessly handles both the initial sync and live replication, continuously monitoring for any changes and updating the destination database accordingly. Using dsync, users can easily migrate their data from MongoDB to FerretDB - as easy as running a command in the terminal.

In this post, we’ll review how to use dsync to migrate the data from MongoDB to a running FerretDB instance.

Prerequisites

Make sure to have the following ready before you start:

Running FerretDB instance.
Running MongoDB instance (local or remote).

Download dsync

Download the latest release of dsync from the GitHub Releases page. For Mac users, you may need to configure a security exception to execute the binary by following these steps.

Alternatively, you can build dsync from the source code.

git clone https://github.com/adiom-data/dsync.git
cd dsync
go build

Run dsync to migrate into FerretDB

To migrate data from your local MongoDB instance to FerretDB, simply specify the source and destination connection strings.

For example:

MongoDB is running locally at mongodb://localhost:27018/
FerretDB is running at mongodb://:@localhost:27017

Set the following environment variables for the source and destination connection strings and run dsync to migrate all data:

export MDB_SRC='mongodb://localhost:27018/'
export FERRETDB_DEST='mongodb://<username>:<password>@localhost:27017/'
./dsync --progress --logfile dsync.log "$MDB_SRC" "$FERRETDB_DEST"

Replace and with your FerretDB credentials. When running FerretDB without authentication enabled, they can be omitted.

When dsync is running, it opens a live-change monitoring session in the terminal to track the progress of the migration.

Dsync Progress Report : ChangeStream
Time Elapsed: 00:12:50        1/1 Namespaces synced
Processing change stream events

The session will remain open for as long as dsync is running. So even if new data is added to the source MongoDB instance, dsync will keep track of it and replicate it to FerretDB.

Lastly, confirm that the data has been migrated successfully by connecting to the FerretDB instance and checking the data, or running dsync with the verify option.

Have any questions about the migration process? Contact us – we’re happy to help.

MongoDB + Weaviate = ♥

Alexander Komyagin — Tue, 18 Mar 2025 03:13:14 GMT

In today’s data-driven world, implementing advanced search functionality can give your business a significant competitive advantage. Let’s explore how combining MongoDB with Weaviate creates a powerful solution for modern search needs.

Understanding the Technologies

MongoDB is a flexible NoSQL database that excels at storing document-oriented data with dynamic schemas, making it ideal for diverse applications.

Weaviate is an open-source vector database specialized in storing and querying high-dimensional vectors, enabling semantic search capabilities.

Vector indexes organize embeddings (mathematical representations of data) that capture semantic meaning, allowing for similarity-based searches rather than exact matching.

Why Vector Search Matters

Vector search powers several key applications across industries:

Semantic search for context-based document retrieval
Personalized recommendation systems
Anomaly detection for security and quality control
Enhanced conversational AI and chatbots

MongoDB vs. Weaviate: 5 Key Differences

Deployment: Weaviate offers easier self-hosting, while MongoDB’s vector capabilities are primarily available through Atlas
Performance and Features: Weaviate is optimized specifically for vector operations and offers important features right out of the box, such as embedding service and hybrid search
Integration: MongoDB simplifies management by storing operational and vector data together
Cost Structure: Weaviate’s open-source nature may offer cost advantages
Support Ecosystem: MongoDB has a larger community, while Weaviate has specialized vector search expertise

Why Use Both Together?

Rather than viewing this as an either/or decision, consider the benefits of integration:

Complementary Strengths: MongoDB handles diverse data structures and complex queries, while Weaviate excels at semantic search and vector operations. The combination enables vector search on self-managed or community MongoDB versions.

Performance Isolation: Separating workloads allows each system to focus on its strengths, improving overall system availability and cost-efficiency.

Cost Benefits: Despite managing two systems, the specialized capabilities can optimize resource utilization and reduce development time.

Future-Proofing: A combined approach provides flexibility to adapt to evolving data requirements and technological advancements.

Replication Options

To integrate these systems, consider these data replication methods:

Kafka-Debezium: Robust but infrastructure-heavy
Custom Python Scripts: Flexible but introduces maintenance risks
SaaS Tools (Airbyte, Fivetran): Convenient but potentially clunky
Dsync: A lightweight solution offering simple, fast, and reliable real-time replication with minimal overhead. We recently added a preview version for Weaviate sink – contact us if you’re interested in giving it a try!

By combining MongoDB and Weaviate, you can create a data architecture that leverages the best of both worlds for more powerful, efficient, and future-ready search capabilities.

Migrate from Cosmos DB RU for MongoDB: vCore or Atlas?

Alexander Komyagin — Fri, 28 Feb 2025 20:30:47 GMT

In today’s cloud-native world, choosing the right database infrastructure is critical for application performance, cost management, and operational efficiency. Microsoft’s Cosmos DB has been a popular choice for organizations needing globally distributed, multi-model database services. However, many users have faced challenges with performance and the Request Unit (RU) pricing model, leading to increased interest in alternative solutions.

In this post we explore the migration path from Cosmos DB RU for MongoDB to the newer Cosmos DB vCore for MongoDB option, including common challenges, alternative solutions, and a streamlined migration approach using dsync.

A typical Cosmos DB migration planning session

Common Challenges with the RU Model for MongoDB

While Cosmos DB’s RU for MongoDB model excels at elastic scalability and point reads, it presents several key challenges:

Unpredictable Pricing: The consumption-based RU model can make costs difficult to forecast, especially for workloads with variable traffic patterns. Many organizations find themselves over-provisioning to avoid performance issues, resulting in higher costs than necessary. Cosmos RU supports auto-scaling that can help to mitigate some of the performance impacts, but it doesn’t scale down to 0 and the lower bound depends on the data size and the maximum allowed RU set.

Performance Limitations: For data-intensive operations like large-scale data ingestion or scan-heavy workloads, the RU model often requires substantial provisioning, which can become prohibitively expensive. It also requires knowing in advance which collections need to be sharded with what shard key.

Limited Change Stream (CDC) Support: Cosmos DB RU only supports creating a change stream for individual collections, and it doesn’t generate change events for delete operations. This is often a major limiting factor for event-driven architectures.

Alternative Options

When considering alternatives to Cosmos DB RU for MongoDB, several options emerge:

Self-Managed MongoDB

Pros:

Complete control over deployment and configuration
Potential cost savings for certain workloads

Cons:

Lacks many enterprise-grade features
Requires MongoDB-specific operational expertise
Higher maintenance overhead

MongoDB Atlas

Pros:

Advanced features including encryption (with queryable encryption)
Robust streaming capabilities and vector search functionality
Excellent performance characteristics
Native Document engine and feature-rich queries

Cons:

Limited deployment flexibility (minimum of 3 nodes required)
Introduces a new vendor into your infrastructure ecosystem if you’re not already using MongoDB

Cosmos DB vCore for MongoDB (New Option)

Pros:

PostgreSQL-based architecture
Open-source engine with available repository
Integrated vector database capabilities
Seamless integration with other Azure products and services
Supports single-node deployment for cost optimization
Familiar MongoDB compatibility

Cons:

Not all MongoDB features and query operators are supported
Some performance optimizations are still a work in progress

Seamless Online Cosmos DB Migration with dsync

The most straightforward path to migrate from Cosmos DB RU to vCore for MongoDB or MongoDB is using the dsync tool. This approach simplifies and accelerates what would otherwise be a lengthy and complex migration process. What would take many hours with dump-restore, now takes minutes!

Basic migration syntax:

dsync $COSMOS_RU $COSMOS_VCORE

You can find additional details, examples and configuration options in dsync documentation.

Developed by Adiom, dsync is an open-source tool designed to make online NoSQL database migrations remarkably simple for developers and DevOps teams. Inspired by the simplicity of the familiar “rsync” tool for file operations, dsync brings that same ease of use to online database migrations.

Key Features of dsync:

Lightweight: Minimal resource requirements
Highly Parallelized: Optimized for performance during migrations
Reliable: Includes resumability and robust data validation checks
Portable: Runs anywhere - from your laptop to a VM or Docker container
User-Friendly: Zero learning curve for anyone familiar with basic bash commands

The dsync tool automates both initial data synchronization and change data capture (CDC), managing the coordination and transition between these phases without manual intervention.

Migration Considerations

Even when migrating between MongoDB-compatible APIs, several factors require special attention:

Indexes

Ensure your index strategy is properly transferred and optimized for the new environment. Different MongoDB implementations have subtle differences in indexing behavior.

Performance Validation

Thoroughly test query and write performance under your expected load patterns. The vCore model’s performance characteristics will differ from your RU-based deployment.

Data Integrity

Decide what approach to use for validating data integrity post-migration. This is especially critical for online migrations. Our CEO’s post on Medium has a good summary of common methods. Dsync already includes embedded validation checks based on counts and document hash (docs).

NOTE for Cosmos vCore destinations: the collection.estimatedDocumentCount() method may return incorrect counts after migrating a large data set. We recommend using using the count() or countDocuments() methods that return accurate information but may take a little longer.

Delete Operations

Note that Cosmos DB doesn’t expose deletes in its change stream, which can present a challenge during migration if you’re looking to minimize downtime (so-called live or online migration). Ideally, you should pause or queue delete operations at the application level during migration. If that’s not possible and you are migrating to Cosmos DB vCore for MongoDB, the Adiom team offers specialized support.

Best Practice

For a successful migration, replicate the entire migration process across all environments, starting with development moving through your testing environments before tackling production. This approach helps identify and resolve any migration-specific issues early in the process.

The portability of dsync makes it particularly well-suited for this progressive migration strategy, allowing you to establish and refine your migration playbook with minimal effort.

Conclusion

As organizations look to optimize their database infrastructure for performance and cost, the migration from Cosmos DB RU to vCore for MongoDB represents an attractive option. With the right tools and approach, this migration can be executed with minimal disruption to your applications and operations.

The dsync tool simplifies this journey, bringing rsync-like simplicity to what would otherwise be a complex database migration process. Whether you’re looking to reduce costs, improve performance for specific workloads, or gain more deployment flexibility, the new vCore option deserves serious consideration.

If you need assistance with a migration, help with evaluating options, or simply want more information about dsync, please reach out to the our team as we are continuing to enhance this open-source tool for the benefit of the broader database community.

Online migration from DynamoDB to CosmosDB NoSQL

Alexander Komyagin — Tue, 18 Feb 2025 06:08:51 GMT

We’re excited to announce the public preview of DynamoDB and Cosmos DB NoSQL support in dsync, a powerful tool that revolutionizes database migration workflows. This new capability enables seamless, online migration from AWS DynamoDB to Azure Cosmos DB NoSQL, eliminating the traditional hurdles of cross-cloud database transitions.

DynamoDB vs Cosmos DB NoSQL: Why Consider Migration?

While both DynamoDB and Cosmos DB NoSQL are powerful NoSQL databases, each offers unique advantages that might better suit your evolving needs. DynamoDB is a key-value store on steroids. It can handle massive scale (a lot of internal AWS services are using it), but it uses its own specific query language and the number of supported features is rather limited.

Azure Cosmos DB NoSQL provides multi-region writes, automatic indexing, and more flexible consistency levels, making it particularly attractive for globally distributed applications. It also supports a flavor of SQL as a query language, and it integrates seamlessly with other Azure services, potentially simplifying your cloud infrastructure if you’re already invested in the Azure ecosystem or planning to move from AWS to Azure.

The Old Way: Offline Migration Headaches

Traditionally, migrating from DynamoDB to CosmosDB involved a complex, time-consuming export-import process with cloud storage and a data processing framework. For example, it could be the following:

Export data from DynamoDB to S3
Set up and configure Spark clusters
Transform and load data into Cosmos DB
Plan for significant application downtime

This approach required extensive planning, specialized expertise, and often resulted in prolonged service interruptions.

The New Way: Streamlined Online Migration from DynamoDB to CosmosDB with dsync

Dsync transforms this entire process into a straightforward operation that any developer or DevOps engineer can perform. Here’s what makes it special:

Zero Infrastructure Requirements: Runs directly on your local machine or VM
No Data Storage: Processes data in-memory without persistent storage
Familiar Unix-like Syntax: If you know basic shell commands, you already know how to use dsync
Online Migration: Minimal to no downtime required
Cross-Cloud Security: Simplified authentication process

Here’s how simple it becomes:

## Start Cosmos DB Sink Connector
cosmos-sink 8089 $URL $KEY &

## Login to AWS CLI
aws sso login

## Run dsync
dsync --progress --logfile dsync.log --namespace <TABLENAME>:<DB>.<CONTAINER> dynamodb grpc://localhost:8089 --insecure

Learn more

You can learn more in our QuickStart guide or by watching the demo video on YouTube.

Ready to simplify your database migration? Try dsync today and experience the future of seamless cross-cloud data synchronization.

Online vs. Offline Database Migrations: Which Path is Right for You?

Alexander Komyagin — Mon, 23 Dec 2024 01:41:33 GMT

Database migrations are a necessary evil in the life of any growing application or a company. They’re often complex, stressful, and, if not handled correctly, can lead to major headaches. One of the first critical decisions you’ll face is choosing between an offline or online migration strategy. Both have their pros and cons, and the best choice will depend on your specific needs, resources, and risk tolerance.

Let’s break down the key considerations and when each approach shines.

The Essentials: Shared Concerns for All Migrations

Before we dive into online vs. offline, let’s cover common ground. These are things to think about no matter which migration method you choose:

Speed and Throughput: How long d? This depends on the number of records, their size, latency between systems, network bandwidth, write capacity of your destination database, and the parallelism of the migration tooling that you will choose. Migration speed and throughput aren’t simply linear calculations. Think of it like moving houses – the time it takes depends not just on how much stuff you have, but also on the size of the moving truck, the distance between houses, and how quickly you can unpack at the new location.

Pro-tip: Put your migration host in the same region as your databases, and over-provision your destination to handle the initial surge of writes.
Careful Planning: Always start with your lower environments (dev, QA). Run dry runs on production to simulate real-world data migration scenarios and identify any hidden issues. You don’t want surprises when you’re moving your most important data. Differences between production and lower environments are common, so be vigilant about schema and data type discrepancies.
Testing, Testing, 1-2-3: Don’t just verify functionality in the new database! Run performance tests to see how your queries and operations are performing. Changes in database technology – even version upgrades – can drastically impact performance.
The Escape Hatch: Rollback Planning: While it’s rare to actually roll back a migration, having a fallback plan is crucial for risk mitigation. It’s like having a parachute – you may never use it, but you’re darn glad it’s there.
Data Integrity Validation: Validate your data with checks like count comparisons, full database/table hashes, and spot checks. Find that balance between risk, downtime and migration duration. Sometimes, a combination of methods is need. Note that most off-the-shelf tools don’t help you validate, placing this burden on the user.
Migration Resumability: Can you stop and resume the process, or do you need to start over? This has massive implications for how you plan your migration window, so think about this carefully.
Indexes: A Hidden Time Sink: Migrations can be a fantastic opportunity to optimize your indexes. It may even be required, as some databases need different indexing strategies, e.g. Cosmos DB and MongoDB. Should you create indexes before or after the data copy? For small datasets, it doesn’t matter as much. But for larger datasets, you are best to build them after the initial copy to avoid write amplification and make sure that index sizes stay on the smaller side.
Cost: Mission-critical migrations take time, effort, and money. While it may be tempting to consider them a one-off expense, it’s a project with many variables, that can easily stretch into months. The more variables you can identify and control, the smoother the project will run. We recommend opting for an optimized self-service migration tool that can help you iterate fast. In our experience, this helps reduce project timelines from months into weeks.

Offline Migrations: The “Big Bang” Approach

Offline migrations involve taking your source system offline, moving all the data, and then bringing the new system online. Just like backup and restore. It’s straightforward in concept but comes with some caveats.

When Offline Migrations Work Best:

Lower Environments: Ideal for development, testing and QA environments.
Tolerance for Downtime: You can afford a service interruption.
Minimizing Variables: You want to avoid the added complexity of online migrations.
Smaller Datasets: Typically suitable for under 100GB or under a million records/documents.

Online Migrations: The Seamless Transition

Online migrations strive for zero or minimal downtime by migrating the data while the source system remains active. Typically they involve an initial data copy followed by a streaming changes in real-time from the source to the destination. This is ideal for applications that can’t tolerate service disruptions.

When Online Migrations Work Best:

Minimize Downtime: Essential for mission-critical applications, SaaS products, e-commerce platforms, or anywhere a prolonged outage has severe financial implications.
Large Data Sets: Suitable for 100s of GB or 10s of millions of records or more.
Testing In Parallel: Allows running the old and new systems concurrently for pre-cutover validation.

Reference: Common Migration Tooling for Migrating MongoDB, Azure Cosmos DB, or AWS DocumentDB

mongodump + mongorestore: A reliable offline MongoDB solution that can process multiple namespaces in parallel.
MongoDB Live Migrate: A good option for live migrations to MongoDB Atlas, but with limitations.
MongoDB’s mongosync: A newer tool for MongoDB replication and online migrations. It offers a high level of parallelization and a REST interface, and only works for native MongoDB databases 6.0+.
MongoDB’s mongomirror: As an predecessor of mongosync, it supports online migrations for native MongoDB databases, and supports older MongoDB versions going back to 2.6, but it has many limitations such as single threaded CDC and it doesn’t work with sharded clusters.
Azure Data Factory & AWS DMS: Easy-to-use web-based solutions for Azure and AWS, respectively. As managed solutions, they are not available for local deployments, require specific firewall configuration, have performance and other limitations.
Kafka + Debezium: A very flexible solution that requires custom code, orchestration and Kafka infrastructure.
3rd Party ETL Solutions (Airbyte, NiFi, Fivetran, etc.): Generally complex, not optimized for database migrations. They carry performance, cost and data security limitations.
Dsync: A single binary that enables online migrations between different databases by seamlessly incorporating both initial data copy and CDC (change data capture) for replicating changes from the source to the destination. Dsync parallel design also helps with accelerating the initial data copy by 10x for large data sets, thus allowing users to execute a simple offline migration where otherwise they would have to do it online.

Making the choice

Your migration strategy should align with your specific circumstances and requirements. Consider online migrations when downtime must be minimized and your dataset is substantial. Opt for offline migrations when simplicity and risk reduction take precedence, particularly with smaller datasets or non-production environments. If you’re not sure, try the offline approach first.

The landscape of migration tools continues to evolve, with solutions ranging from cloud provider offerings like Azure Data Factory and AWS Data Migration Service to specialized tools like dsync. Each brings its own strengths and limitations, making it crucial to evaluate them against your specific needs, infrastructure constraints, and performance requirements.

Remember that successful migrations aren’t just about moving data – they’re about ensuring your application continues to perform optimally while maintaining data integrity throughout the process. Taking time to plan, test, and validate your migration strategy will pay dividends in the form of a smoother, more reliable transition.

Benchmarking Data Migration Tools

Alexander Komyagin — Fri, 22 Nov 2024 19:30:04 GMT

This is a slightly modified version of the blog post that our intern, Grace, wrote recently. The original blog post can be found on Medium.

For this blog post, we benchmark dsync, an open-source data migration tool designed for NoSQL databases, against popular, industry-standard tools, to see how it stacks up in terms of speed, reliability, and ease of use.

We’ll walk through the methodology, our experiences with each tool, and the lessons we learned about their strengths, weaknesses, and nuances in data migration.

Data Migration Tools

We evaluated three frequently used data migration tools alongside dsync. Here’s an overview of each tool:

Airbyte

Airbyte is a data integration platform that boasts a vast library of over 300 connectors, supporting a wide range of sources and destinations. Its open-source architecture makes it highly customizable, while the Airbyte Cloud offering provides a managed solution for users. Designed to address common data synchronization challenges, Airbyte emphasizes flexibility and extensibility, making it suitable for complex use cases.

Fivetran

Fivetran is a fully managed data integration service known for its “set-it-and-forget-it” philosophy. It automates data pipeline maintenance and schema updates, providing a highly reliable and low-maintenance solution. Fivetran excels in syncing data from traditional databases, SaaS applications, and event streams into centralized data warehouses. With its user-friendly interface and robust error handling, it is often favored by organizations prioritizing stability and minimal manual intervention.

Estuary.dev

Estuary.dev focuses on real-time data movement and transformation, providing a lightweight solution for integrating sources and destinations. The platform is particularly suited for scenarios requiring low-latency data flows. Its minimalistic design makes setup simple, but the trade-off is reduced visibility and control during migration tasks. Estuary.dev is best for small to medium workloads that don’t require extensive monitoring or fine-grained tuning.

Executive Summary

Through our tests, we found out that dsync performs about 5.75x faster than Estuary.dev, while Airbyte and Fivetran did not work in our tests at all.

Here is a table with the final results:

Benchmarking Task

The benchmarking task consisted of comparing the performance of data migration tools by using a Cosmos DB instance as the source and a MongoDB instance as the destination. We used YCSB (Yahoo! Cloud Serving Benchmark) to generate realistic data loads for testing.

The specific benchmarking steps were as follows:

Download YCSB
Generate data using YCSB: We created a dataset of 10 million documents in a single collection using Workload A. This took a couple of hours to complete.
Initial sync test: We performed the initial data sync with the migration tools.
Generate additional data using YCSB: We generated an additional 1 million documents to simulate a real-world scenario where new data arrives during an ongoing migration.
Catch-up test: We measured how well each tool handled the “catch-up” process, which is effectively the maximum rate of CDC (change data capture) or incremental replication.

This is the environment that we used for the tests:

Source Database: Azure Cosmos DB with MongoDB API

- Provisioning: Configured with 4,000 RUs and Autoscale

- Indexing: Default indexing settings for MongoDB API
Destination Database: Self-managed MongoDB instance on GCP

- CPU: 4 vCPUs

- RAM: 16 GB

- Single-node replica set configuration
Virtual Machine: GCP

- CPU: 4 vCPUs

- RAM: 16 GB

Tool 1: Airbyte

Setup:

We started with the Airbyte Cloud version, which we were able to access through their website by just creating an account. Adding Cosmos DB as the source via the MongoDB connector worked after some trial and error, but we faced a lot of issues when configuring the destination.

Challenges:

We encountered a format specifier error with the destination configurations. When we traced back to the source code on the Github repository, it seemed to be from a missing argument.

This issue seemed to be a known bug in the Airbyte GitHub repository, and some people had solved the issue by changing the version of destination to 0.1.9 in settings.

Unfortunately, the Airbyte Cloud version does not allow direct changes to connector versions. To apply the fix, we had to deploy Airbyte locally — a process that took approximately 45 minutes. Once deployed, we adjusted the settings, toggled TLS encryption, and changed the connector version. Despite these efforts, the syntax error persisted.

Further complicating matters, we encountered repeated network HTTP errors where Airbyte disconnected with a “temporarily down” message. These errors were frustratingly difficult to debug due to inadequate logging and unclear error messages.

Verdict:

Airbyte is often praised for its wide range of connectors (400+), and for good reason — it supports many data sources and destinations, making it a go-to tool for integration tasks. However, the tool’s setup complexity, poor error logging, and instability makes it a challenging choice for data migration.

Tool 2: Fivetran

Setup:

Fivetran did not have an open source code version and we were able to quickstart the application by just creating an account. We were able to successfully configure the source using the “Azure Cosmos DB for MongoDB” connector.

Challenges:

Despite supporting MongoDB as a source, we were disappointed to find that Fivetran didn’t offer MongoDB as a destination. This made it unsuitable for our specific test setup, which required a MongoDB destination.

Verdict:

While Fivetran excels at handling integrations with many sources and destinations, its limited support for MongoDB left it unfit for our specific use case.

Tool 3: Estuary.dev

Setup:

Estuary was undoubtedly the easiest to set up among all the SaaS tools that we tested. The source and destination setup worked on the first try without any significant issues.

Results:

Initial Sync: 1 hour 9 minutes

Challenges:

Efficiency: While Estuary.dev successfully completed the migration, its speed lagged behind other tools.
Lack of Visibility: One of the major pain points with Estuary was the lack of feedback during the migration. There were no throughput metrics, error logs, or progress indicators. This made it difficult to know if the migration had started or if it had completed successfully.
No Pause/Resume: There was no option to pause the migration, so we couldn’t measure how well Estuary handled the catch-up time after generating 1 million more documents.

Verdict:

Estuary was the most straightforward tool in terms of setup, but its lack of visibility and control over the migration process made it difficult to evaluate performance in detail. It’s an option for smaller, simpler migrations but lacks the sophistication needed for complex or real-time workloads.

Tool 4: Dsync

Setup:

Finally, we tested dsync, the open-source migration tool that we develop at Adiom. Given that it’s distributed as a simple standalone binary that can be downloaded in the Github repo, there’s no setup difficulties. Testing was conducted on a virtual machine. We tried the default load-level as well as the “Beast” level.

./dsync -s SOURCE -d DEST -load-level Beast

Results:

Initial Sync Time: 12:40 min
Catch-up Time: 7:54 min

Challenges:

Catch-Up Efficiency: We noticed that the catch-up time for dsync did not decrease even when higher load settings were applied. This is something that the team is investigating and it will be addressed in a future release.

Verdict:

Dsync showed promising performance, especially in terms of efficiency as it was more than 5x faster than Estuary. Further, dsync’s performance and metrics were easily observable, making it easy to track the progress and estimate the time to completion.

Final Thoughts: Benchmarked Data Migration Tools Comparison and Takeaways

After testing these tools, we have a few key takeaways:

Airbyte offers a massive library of connectors, but its error-prone setup and lack of clear diagnostics made it challenging to use.
Fivetran is limited in its support for specific use cases like MongoDB.
Estuary.dev is a convenient option for simple migrations with the easiest setup, but lacks efficiency and key features like logging and pausing/resumability, which makes it difficult to assess and manage large-scale migrations.
Dsync is a highly customizable and performant tool, but currently only supports Cosmos DB and MongoDB. We have a preview of DynamoDB support, and are working on other connectors as well. If you would like to give it a try, or talk about the future roadmap, you can get in contact with the team on the contact page.

Ultimately, the right tool depends on the specific requirements of the project, whether it’s ease of use, flexibility, performance under load, or visibility into the migration process.

About the Author

Grace is a junior at UC Berkeley, studying Computer Science and Economics. She interned as a software developer at Adiom, where as one of her projects she benchmarked dsync, an open-source data migration tool designed for NoSQL databases, against popular, industry-standard tools, to see how it stacks up in terms of speed, reliability, and ease of use.

MongoDB Atlas vs Cosmos DB - A Comparative Guide

Alexander Komyagin — Mon, 21 Oct 2024 05:06:02 GMT

Introduction to MongoDB Atlas vs Cosmos DB

In the realm of NoSQL databases, MongoDB and Azure Cosmos DB are two powerhouse solutions catering to diverse application needs. Here are brief overviews and recent developments for each:

MongoDB: The leading document-oriented NoSQL database, MongoDB has just unveiled MongoDB 8.0, a landmark release focusing on enhanced performance, security, and scalability. Key highlights include:

Improved Query Performance: Enhanced query optimization and indexing capabilities.
Client-Side Field Level Encryption (FLE) 2.0: Strengthened data protection with more seamless encryption.
Time-Series Data Support: Native support for efficient time-series data storage and querying.
Enhanced Aggregation Framework: More powerful data processing with improved aggregation capabilities.

Azure Cosmos DB: Microsoft’s globally distributed, multi-model database service. Recent updates include the general availability of vCore provisioning for more flexible compute resource allocation, alongside enhancements to its change feed and analytics capabilities.

Key Differences, Strengths, and Pitfalls

MongoDB

Strengths:
- Maturity and wide adoption.
- Robust query language and indexing capabilities, further enhanced in MongoDB 8.0.
- Flexible deployment options (self-managed, MongoDB Atlas).
- Native Time-Series Support
- Embedded Vector Search capabilities
- Queryable Encryption
Pitfalls:
- Global distribution and multi-region writes can be somewhat complex to setup and manage unless you’re familiar with sharding.
- May require additional tools for full-stack monitoring and security, though Client-Side FLE 2.0 enhances security posture.

Azure Cosmos DB (Provisioned RUs vs vCore)

Provisioned RUs (Request Units)

Strengths:
- Simplified pricing for predictable workloads.
- Works for point queries.
- Auto-scaling for throughput (RUs) with Azure Cosmos DB’s autoscale feature.
- Comprehensive security and monitoring within Azure.
Pitfalls:
- Can be costly for workloads with variable throughput.
- Really poor performance for large ingestion or bulk reads.
- No way to develop against it locally.
- Over-provisioning to ensure peak performance can lead to wasted resources.
- Has notable limitations when it comes to MongoDB API support, some of them are not well documented (my personal favorite is $sample stage in aggregation not returning uniformly random results)

vCore (Virtual Core) Model

Strengths:
- Better suited for compute-intensive, variable workloads.
- More cost-effective for applications with high storage needs but lower throughput demands.
- Enhanced flexibility with separate scaling for compute, storage, and I/O.
Pitfalls:
- Pricing complexity due to separate billing for compute, storage, and I/O.
- Requires more precise resource planning to avoid under/over-provisioning.
- Has broader functionality limitations as of now than RU (e.g. ChangeStreams are still not supported) although the gap is closing quickly.

5 Simple Guidelines for Choosing the Right Database Solution

Predictable Throughput, Simple Scalability Needs:
- Choose MongoDB (especially with MongoDB 8.0’s enhancements) if you prefer a self-managed or serverless MongoDB Atlas setup for its straightforward scalability and robust query capabilities.
- Opt for Cosmos DB with Provisioned RUs if your workload fits well within predictable throughput boundaries and you’re deeply integrated with the Azure ecosystem.
Variable/Compute-Intensive Workloads:
- Consider Cosmos DB vCore for its flexibility in scaling compute resources independently, ideal for applications with fluctuating demands.
- Evaluate MongoDB (with its enhanced performance in MongoDB 8.0) if your primary concerns are query complexity and data model flexibility, and you can manage scalability through sharding or third-party tools.
Multi-Region Writes and Conflict Resolution are Key:
- Azure Cosmos DB (either RUs or vCore) is preferable due to its built-in, customizable conflict resolution mechanisms and seamless global distribution capabilities.
- MongoDB may require more application-level logic for conflict resolution, but can still be suitable if your team is experienced with MongoDB’s ecosystem.
Time-Series Data and Enhanced Security are Priorities:
- MongoDB 8.0 is an attractive choice with its native time-series support and enhanced Client-Side Field Level Encryption (FLE) 2.0 and Queryable Encryption for strengthened data protection.
Ecosystem and Operational Preferences:
- Azure Ecosystem: If deeply invested in Azure services, Cosmos DB (either provisioning model) integrates seamlessly.
- Cross-Cloud or On-Premises Flexibility: MongoDB offers more deployment flexibility across various environments.

Choosing Between Azure Cosmos DB’s API for MongoDB vs Native MongoDB Atlas

Use Azure Cosmos DB’s API for MongoDB if:
- You’re already invested in the Azure ecosystem.
- Need global distribution with automatic conflict resolution.
- Want to leverage Azure’s security and monitoring tools.
- Are comfortable with potential feature lag compared to native MongoDB.
Stick with Native MongoDB if:
- You prefer more control over database configuration.
- Are deeply invested in the MongoDB ecosystem (e.g., MongoDB Atlas, MongoDB Compass).
- Require specific MongoDB features not yet supported by Azure Cosmos DB’s API for MongoDB, such as the latest enhancements in MongoDB 8.0.

Conclusion

Selecting between MongoDB, Cosmos DB with Provisioned RUs, or Cosmos DB vCore involves weighing your application’s unique needs, scalability patterns, ecosystem preferences, and the latest feature enhancements, such as those introduced in MongoDB 8.0. By understanding the strengths and pitfalls of each, you can make an informed decision that optimizes both performance and cost for your NoSQL database solution.

Next Steps

Evaluate your application’s specific requirements against the criteria outlined above.
Test and prototype with the chosen database solution to ensure optimal performance and cost alignment.
Stay informed about the latest updates and features from both MongoDB and Azure Cosmos DB to future-proof your database strategy.
We will be periodically revising our stance and sharing new updates as they come in. Stay tuned!

If the choice is not obvious for you, no need to worry. Now you have a way to switch or test out a different solution later on with 0 effort! Adiom’s dsync supports online migrations from Cosmos DB to MongoDB Atlas, from MongoDB Atlas to Cosmos DB, and from Cosmos DB with provisioned RUs to Cosmos DB vCore.

At Adiom, we built dsync to help developers and DevOps perform live migrations and real-time replication easily. Dsync is a fast, reliable, easy-to-use and Open Source solution. Using dsync helps to accelerate and derisk projects, and allows teams to get their applications and services onto a new database in minutes instead of months.

Give dsync a try by downloading it from GitHub. It’s distributed as a binary that you can just run anywhere, including your laptop, and it doesn’t require specialized infrastructure or any complex setup.

Adiom offers help and commercial support on terms that suit your projects best. Get in touch with us on the contact page.

Migrate from Azure Cosmos DB to MongoDB Atlas Using Dsync

Alexander Komyagin — Tue, 15 Oct 2024 04:46:30 GMT

Migrating data between databases can be a complex and time-consuming process, especially for large-scale, mission-critical workloads. One increasingly common scenario is the online or live migration from Azure Cosmos DB to MongoDB Atlas, a high-performance, cloud-native NoSQL database with a wide range of advanced features. This use case presents unique challenges, including compatibility, downtime minimization, and ensuring data consistency throughout the process - see our previous post for more details.

While existing solutions like custom scripts, mongodump/mongorestore, and third-party tools are available, they often fall short in scalability, resiliency, or ease of use. Enter dsync—our open-source tool specifically designed to handle these complex migrations faster and with less risk.

Dsync: Built to Solve the Pain of Data Migration

Dsync is a powerful data migration and synchronization tool developed by Adiom, Inc. to address the needs of developers and DevOps teams dealing with large-scale migrations. We built dsync after seeing the limitations of other tools firsthand—either they required too much manual intervention, or they couldn’t scale effectively for production-level workloads. Dsync offers a simple, resilient, and highly performant solution that’s built with both ease of use and enterprise-grade reliability in mind.

Key Advantages of Dsync:

Ease of Use: Deploy dsync with a single binary—no additional infrastructure required. Run it on VMs or Docker and manage migrations via CLI or web-based monitoring. All complexity of orchestrating a highly parallelized live migration is abstracted away from the user.
Resiliency: Migrations can be interrupted and resumed without losing progress. Dsync ensures data consistency and integrity at every step.
Speed: It significantly accelerates migration timelines, reducing hours-long processes to just minutes compared to traditional methods. What could’ve been a 2 hour wait becomes a 10-minute thing.
Security: Dsync doesn’t store data or send it anywhere except the designated destination. It also supports network encryption to keep data secure.
No Hidden Costs: As an open-source tool, dsync eliminates the need to onboard expensive SaaS solutions or purchase licenses in early stages of the process.

Prerequisites for Migration

Before you start migrating from Azure CosmosDB to MongoDB Atlas using dsync, make sure you have the following in place:

Provision the Source (Cosmos DB) and Destination (MongoDB Atlas) databases. For optimal performance and to avoid request timeouts, make sure to provision at least 4000 RU/s on your source Cosmos DB collections. On the Atlas side, we recommend M50 or better at least for the migration itself. Dsync supports load level configuration to control how fast it works.
Connection strings for both your Cosmos DB and MongoDB Atlas instances, including the necessary credentials and configuration. Don’t worry, dsync doesn’t transfer, store or log your credentials anywhere.
1. Instructions for Cosmos DB
2. Instructions for Atlas
Provision a VM for dsync. The most common VM configurations are 4 CPUs, 16 GB of RAM or 8 CPUs, 32 GB of RAM. Ensure that the network is properly configured - dsync should be able to connect to both the Cosmos DB instance and MongoDB Atlas from that host.

Steps for a Live or Online Migration Using Dsync

Once the prerequisites are set, follow these steps to execute your migration seamlessly. You can also find these steps and additional details in our documentation.

1. Download dsync

Head over to the dsync GitHub repository and download the latest release. Unpack the zip and choose the binary that matches your host.

2. Run the Migration

Use the CLI to start your migration by providing the necessary connection strings and any configuration settings (such as the databases and collections you want to migrate). Dsync will initiate the transfer of data from your Cosmos DB source to MongoDB Atlas destination.

./dsync -s <Cosmos> -d <Atlas> --progress --logfile dsync.log

By default, dsync will use the “adiom-internal” database in your MongoDB Atlas instance to store the metadata, but it’s configurable.

3. Monitor the Process

You can monitor your migration in real-time via the web-based interface (accessible via localhost:8080/progress) or directly in the CLI. Dsync provides detailed logs and metrics on throughput, latency, and any potential errors, allowing you to track progress and troubleshoot as needed.

After the initial data copy, dsync will automatically switch to CDC-based incremental change replication.

When the reported replication lag (“Events to catch up”) gets close to 0 and stays there, it means Cosmos DB and Atlas are synchronized.

You can initiate the cutover by stopping the writes on your source. After that, the lag should go down to 0 quickly. At this point, the data migration is complete.

4. Validate the Data

Dsync includes built-in validation mechanisms to ensure the data is transferred accurately. You can run post-migration checks to verify that all data matches between Cosmos DB and MongoDB Atlas, minimizing the risk of inconsistencies. All you need is to restart the original dsync process and add one new parameter at the end:

./dsync -s <Cosmos> -d <Atlas> --progress --logfile dsync.log --verify-quick-count

dsync will display the verification result in the CLI.

5. Cutover

Once the migration is complete and validated, you can cut over your applications to point to the new MongoDB Atlas database. The switchover can be done with minimal downtime, ensuring a smooth transition without disrupting your business operations.

Next Steps: Unlock the Power of MongoDB Atlas

With your data successfully migrated to MongoDB Atlas, you can now leverage its advanced features. MongoDB Atlas offers significant performance improvements with the new 8.0 release, including faster queries and enhanced scalability. You can also take advantage of native vector search for AI-driven applications and robust security features for compliance and data protection. Whether you’re scaling globally or improving operational resilience, Atlas provides the tools to make the most of your data.

Conclusion

Migrating from Azure CosmosDB to MongoDB Atlas may seem daunting, but with dsync, the process becomes manageable, resilient, and fast. By following these steps, you can ensure your data is moved with minimal disruption and be ready to unlock the full potential of MongoDB Atlas’s advanced features. If you’re ready to modernize your database infrastructure, dsync is here to help you accelerate the transition and keep your data safe along the way.

You can see dsync in action in our live demo on Youtube.

Adiom offers help and commercial support on terms that suit your projects best. Get in touch with us on the contact page.

A Repeatable Solution for Cosmos DB to MongoDB Migration

Alexander Komyagin — Mon, 14 Oct 2024 02:59:13 GMT

This is an updated version of the original article published on Medium by our CEO, Alexander Komyagin. In this article we’ll explore the challenges of building a solution for live migration from Cosmos DB to MongoDB. There are many reasons for users to want to migrate - we will save them for a future article. Here we will focus on technical challenges of what might look like an easy migration between two databases with the same API.

The Reality of Compatibility

Cosmos DB for MongoDB is compatible with a limited subset of MongoDB’s wire protocol, suitable only for the simplest MongoDB applications. This limited compatibility can lead to higher costs and unexpected challenges. Standard MongoDB migration tools (mongomirror, mongosync, etc.) don’t work. Mongodump and mongorestore turn out to be painfully slow and can take hours on a relatively small (<100GB) dataset.

Key Concerns for Migration

Migrating production workloads involves addressing several critical concerns:

Avoiding Excessive Downtime
Speeding Up the Migration
Preserving Data Integrity
Implementing a Rollback Plan

Avoiding Downtime

For larger datasets, avoiding downtime during migration is crucial. A live migration approach typically involves the following stages:

Initial Data Copy: Bulk copy the data from the source to the destination.
Capturing Changes: Use Change Data Capture (CDC) to capture changes from the beginning of the data copy process and apply them after the copy is done.
Replicating Changes: Continue replicating changes until the lag between the destination and the source is small enough to cut over.

However, Cosmos DB’s limited support for Change Streams complicates the approach:

No Delete Events: It’s recommended that users implement safe-deletes in their applications using a special field to mark deleted documents and set the TTL field for automatic cleanup. They will need to clean up those documents on the MongoDB side as well. In any case, users will likely need to change their application to make it all work. In certain cases, it’s possible to compare the source and destination collections side-by-side to determine what was deleted - that’s the approach we implemented in Adiom’s dsync.
Per-Collection Change Streams: There’s no global or per-database Change Stream support. Each collection needs to have its own Change Stream, requiring writing scripts or coordinating parallel Change Streams. In our tests, we saw performance degradation with more than 10–15 parallel Change Streams on Cosmos DB.
Specific Pipeline Options: Cosmos DB requires using the exact Change Stream pipeline options as specified; otherwise, it won’t work. They also always return the full document, so a replace is needed on the destination. This creates serious risks for gradual cutovers where writes on the destination are allowed early assuming no conflicts. To mitigate these risks, a reliable way to check data integrity is required.
No Timestamps: Timestamps are normally used to calculate the replication lag or delay. Knowing the lag is extremely important to properly time the cutover in live migrations. Since Cosmos DB Change Stream events don’t have timestamps, a different method is required. In Adiom’s dsync we use event sequencing to calculate the lag.

Lastly, it’s important to ensure proper coordination between Change Streams and initial data copy to avoid data loss due to race conditions. In Adiom’s dsync, this is done automatically behind the scenes, without requiring any input from the user.

Speeding Up the Migration

Migration speed depends on:

Latency
Level of Parallelization
Provisioned RU/s on the source
Capacity on the destination (CPU, RAM, disk IO)

Network latency to Cosmos DB can significantly impact throughput. In dsync, we parallelize reads from different namespaces. Further, we employ intelligent partitioning strategies to identify sensible split point(s) in the range of _id values, and use parallel range queries for large namespaces to increase speed. Sadly, Cosmos DB’s implementation of the aggregation $sample command does not work correctly, so we can’t do statistical sampling to determine split points programmatically.

Resumability

Dsync tracks synced namespaces and stores resume tokens from Change Streams to avoid restarting the migration process from scratch. This is particularly important during the initial testing phase but also for ensuring reliability.

Data Integrity

With no dbHash support in Cosmos DB, dsync relies on heuristics like count and size, as well as comprehensive per-namespace checksum comparison.

Backout Plan

In Enterprise and mission-critical applications it’s more than just a checkbox item. Having an out-of-the-box ability to reverse the synchronization process helps to accelerate planning and testing.

Observability

It’s important to expose and monitor metrics to identify bottlenecks and ensure direct visibility into the migration process. Bottlenecks can differ between clusters, environments, and sometimes between runs. Dsync offers a CLI- and Web-based progress output, and detailed logs.

Conclusion

Migrating from Cosmos DB to MongoDB is challenging but manageable with the right approach. Addressing key concerns around downtime, speed, data integrity, and backout plans ensures a smooth transition.

Adiom offers help and commercial support on terms that suit your projects best. Get in touch with us on the contact page.

Powering Universal Data Exchange with Adiom

Alexander Komyagin — Tue, 08 Oct 2024 19:36:48 GMT

In today’s data-driven world, the need for seamless data exchange and synchronization between Enterprise databases and Edge devices is more crucial than ever. This is where Adiom comes into play, providing innovative solutions to bridge the gap between large heterogeneous databases and the Edge, making data mobility easier and more efficient.

Adiom’s vision of establishing a universal data fabric for data object-based communication, management, and exchange is truly groundbreaking. By offering data synchronization tools that are simple, fast, reliable, and Open Source, the company is setting a new standard in the industry.

With a focus on expanding its customer base and enhancing user experience, Adiom’s website serves as a platform to showcase the company’s cutting-edge features and functionalities. While the business history is currently undisclosed, the emphasis remains on driving innovation in data mobility. By leveraging Adiom, Enterprises can streamline their data exchange processes and optimize communication between different systems. Whether it’s syncing data between Enterprise databases or facilitating real-time interactions at the Edge, Adiom’s solutions empower organizations to stay ahead in today’s competitive landscape.

In conclusion, Adiom is at the forefront of revolutionizing data mobility, making it easier for businesses to exchange and manage data effectively. With a commitment to simplicity, speed, reliability, and Open Source principles, Adiom is paving the way for a more connected and data-driven future.

Simplify Enterprise Database Communication with Adiom

Alexander Komyagin — Tue, 08 Oct 2024 19:36:48 GMT

In the fast-paced world of Enterprise database communication, efficiency is key. Adiom Solutions, a leading tech company, is on a mission to simplify data mobility for businesses dealing with large and critical databases. With a vision to enable seamless data exchange between Enterprise databases and the Edge, Adiom is revolutionizing the way data is managed and communicated.

At the core of Adiom’s offerings are their data synchronization tools, designed to be the simplest, fastest, most reliable, and Open Source. These tools empower businesses to streamline their database communication processes, ultimately enhancing productivity and efficiency. Whether it’s syncing data between databases, managing data objects, or facilitating communication between different systems, Adiom’s solutions are tailored to meet the needs of modern enterprises.

One of the key highlights of Adiom’s approach is its emphasis on user experience and customer satisfaction. By providing intuitive tools and resources, Adiom aims to improve the overall experience for its users and expand its customer base. Adiom’s focus on innovation and simplicity speaks volumes about its commitment to driving progress in the tech industry. By offering Open Source solutions and prioritizing user experience, Adiom is setting itself apart as a leader in the field of Enterprise database communication. In a world where data is king, Adiom is paving the way for a more connected and efficient future. With their cutting-edge tools and commitment to simplicity, businesses can trust Adiom to simplify their database communication processes and streamline their operations.