Multiple replicas: controlled gradual updates
Rolling updates, blue-green, canary, compatibility, and recovery with multiple replicas.
Updated: November 14, 2026
Two or more copies of the API serve traffic behind a load balancer. Deployment replaces copies gradually, without removing all of them at once. Replicas on different machines can also protect against the loss of a machine, provided the other components are resilient.
Two application replicas do not mean two databases. In this example, both use a shared database. A failure there can affect all of them.
Before you begin
- All copies receive the same identified version; persistent data does not depend on disposable local disks.
- The load balancer detects readiness and allows destinations to be removed from new requests.
- The remaining replicas can handle the load, or an additional copy can be started before removal.
- Sessions, caches, files, and messages support coexistence of versions.
- Migration is a single, controlled task; each replica does not try to run it at startup.
If B cannot handle the load alone, start an extra replica before removing A. The rollout advances only after checks.
Complete workflow
- Receive the version that passed acceptance testing and record configuration, migration, and stopping criteria.
- Prepare the database compatibly and check health before replacing processes.
- Remove A from new requests and wait for ongoing work. If extra capacity is available, start the new instance before this removal.
- Install/start A's new version and test the replica directly. Keep it out of service until ready.
- Return A to the load balancer. Check metrics per version and replica, in addition to global metrics.
- Observe for an appropriate duration and volume of operations. Low traffic requires synthetic tests and longer observation; no errors without usage does not prove safety.
- Repeat for B and then other groups. Degradation stops expansion.
- Check the final set, connections, and task queue; record completion and retain versions and evidence for recovery.
Rolling, blue-green, and canary in this scenario
| Technique | Difference |
|---|---|
| Rolling update | Replace groups progressively while preserving minimum capacity. This is the basis of this example. |
| Blue-green | Prepare a complete new set, test, and transfer traffic; requires more temporary resources. |
| Canary | Direct a representative portion of traffic/users to the new version before expanding. One of two replicas does not guarantee exactly half the requests. |
These are variations for controlling the transition, not a requirement to use Kubernetes. Percentages and replica counts do not replace evidence of health.
Updating the database and contracts
During the update, old and new replicas may operate simultaneously. The database, API contracts, and message consumers must remain compatible throughout this window. For a complete staged migration example, see One server: releases, current, and switching processes.
- Adding a required field without a value for existing records may fail. Changing a type, semantics, or format can cause breakage even without removing columns.
- Changing shared schema may affect all replicas at once; an application canary does not isolate that impact.
- New APIs must continue serving old frontends in open tabs or caches. New frontends must not require an API still missing from part of the set.
- Measure locking and migration cost; online does not mean an absolute absence of locks. A migration may require maintenance even with many replicas.
Failures and recovery by stage
| Failure | Action |
|---|---|
| Preparation or migration fails | Stop before replacing replicas. If it affected the database, contain the shared impact and recover according to the actual state. |
| A does not become ready | It receives no traffic. B keeps serving; recover A and capacity before advancing. |
| A fails after receiving traffic | Remove A, keep old B if healthy and capable, check data produced by A, and roll back/fix. |
| B fails while new A is serving | Do not advance or blindly remove A. Ensure capacity with a healthy compatible version or add another replica. |
| All are already new and a defect appears | Gradually return to the previous version if compatible; otherwise, disable the feature, fix forward, or use maintenance. |
| Incorrect data in the database | Contain writers, identify records, and correct selectively. Rolling back replicas does not undo data. |
| Load balancer, database, or shared cache fails | Activate recovery for that component. More API replicas do not resolve the shared failure. |
Queue workers: their own procedure
- Start new consumers compatible with messages and the database.
- Stop old consumers from fetching new tasks and allow already started ones to finish.
- Acknowledge completion according to the queue protocol; stop only after draining or an explicit recovery decision.
- If interrupted, use the system's redelivery/visibility policy. Design idempotent processing: repeating a task must not charge or perform external effects twice.
- Version messages when necessary and check producers, consumers, and old jobs. Control retries and a failure queue for tasks that cannot be processed.
Canceling consumption does not automatically mean returning all in-progress tasks; behavior depends on the broker and library. An external effect may have occurred before the task was acknowledged, requiring deduplication or reconciliation.
Bugfixes, hotfixes, and completion criteria
Regular fixes follow a new release and gradual rollout. During an incident, consider removing the problematic version or disabling the feature before writing the patch. Hotfixes receive priority review and testing; reducing observation requires explicit ownership and risk. The fix must also be applied to affected versions that are still maintained.
- Errors and latency within defined limits, with representative volume.
- Correct business journeys; a live process is not enough.
- Healthy capacity, queues, connections, and database.
- Versions actually running checked on each replica.
- Rollback window maintained and removal of old structures postponed until safe.
Sources and further reading
- Kubernetes — Rolling updates
- Kubernetes — Termination and endpoints
- RabbitMQ — Acknowledgments and redelivery
- AWS — Rollback safety
- MySQL — Online DDL
References consulted on November 2, 2026. The procedures are conceptual models: parameters, commands, and guarantees depend on the technology and need to be rehearsed.
Other articles in the series
Share this article
Sources & further reading
This note is a practical introduction, not a substitute for the original documentation. Check source dates and implementation details before making technical decisions.