Vault 📚

01_coordination_services

7 min read

Coordination Services in Distributed Systems

In large-scale distributed architectures, multiple processes, microservices, or nodes must operate in concert to achieve consistency, fault tolerance, and robust state management. Coordination services address these challenges by offering primitives like distributed locks, leader election, and configuration storage. This document provides an overview of common coordination problems, introduces popular coordination systems, and outlines best practices for integrating them into a distributed environment. ASCII diagrams and command examples highlight practical usage scenarios.

In a distributed setup, systems are physically and/or logically separated across multiple machines or datacenters. They need ways to keep certain states or tasks in sync, manage membership (who is up or down), and resolve conflicts. Coordination services such as Apache ZooKeeper, etcd, and HashiCorp Consul provide a consistent data store and specialized APIs to solve these problems.

Typical challenges they address include:

  1. Leader Election: Electing a single process as “leader” for certain tasks (like updating a shared resource).
  2. Distributed Locking: Ensuring only one node modifies a resource or performs a job at a time.
  3. Configuration Management: Storing cluster-wide configurations in a consistent, fault-tolerant manner.
  4. Service Discovery: Announcing and discovering running service instances dynamically.

By offloading these tasks to a trusted coordination service, developers avoid implementing and debugging low-level consensus and concurrency logic themselves.

Common Coordination Problems

Leader Election

Selecting a node to act as primary in a cluster ensures that only one node handles tasks like cluster management, partition rebalancing, or job scheduling. Other nodes remain followers or standbys. If the leader fails, a new one is chosen.

+-------------------+
|  Node A (Leader)  |
+-------------------+
         ^
         |
         | (Leader election)
         |
+-------------------+    +-------------------+
| Node B (Follower) |    | Node C (Follower) |
+-------------------+    +-------------------+

When Node A fails, B or C is elected as new leader.

Distributed Locking

A distributed lock ensures that only one instance can perform a specific operation at a time (e.g., updating a shared file). If multiple nodes attempt to grab the same lock, the coordination service grants it to one while making others wait or fail.

Service Discovery

Services register themselves (their IPs, ports, health states) with the coordination store, letting clients query or watch for updates. This eliminates hard-coded addresses. For example, a web service can discover available database instances through a registry rather than a static config file.

Configuration Management

Storing application settings, feature flags, or runtime parameters in a central key-value store ensures that all instances read a unified configuration. Changes can be watched in real time by applications to adapt on the fly.

Apache ZooKeeper

One of the earliest widely adopted coordination services, ZooKeeper implements a replicated, in-memory hierarchical data store. Clients create, read, update, or watch “znodes,” which store data and can notify watchers on changes.

  • ZNodes: Nodes in a tree-like structure; can hold data and child znodes.
  • Ephemeral ZNodes: Automatically removed when the client’s session ends, useful for ephemeral membership or locks.
  • Watches: Clients set watches on znodes to get callbacks when data or children change.
  • Quorum Writes: ZooKeeper is typically deployed as a cluster of 3 or 5 servers to achieve majority consensus.

Example (zkCli command snippet):

# Connect to a ZooKeeper server
zkCli.sh -server 127.0.0.1:2181
 
# Create a znode
create /mylock "some-data"
 
# List children
ls /
 
# Set data
set /mylock "updated"

etcd

A key-value store from the CoreOS project, etcd uses the Raft consensus algorithm for strong consistency and fault tolerance. It’s known for powering Kubernetes’ cluster state.

  • Key-Value: Data is organized in a flat namespace of keys (e.g., /app/config/db_url).
  • Raft: Leader-based consensus ensuring writes are replicated to a majority of the etcd cluster.
  • Watch: Clients can subscribe to changes on certain keys or prefixes.
  • Transactions: etcd supports atomic compare-and-swap updates, enabling distributed locks and concurrency control.

Example (etcdctl snippet):

# Set a key
etcdctl put /myapp/config "config_data"
 
# Retrieve a key
etcdctl get /myapp/config
 
# Watch for changes
etcdctl watch /myapp/

HashiCorp Consul

Consul integrates service discovery, health checks, key-value storage, and DNS-based queries.

  • Agent Model: Each node runs a Consul agent that can register services, perform health checks, and communicate with the Consul server cluster.
  • Catalog & DNS: Services register with the catalog, which can be queried via DNS or HTTP.
  • Key-Value Storage: A simple hierarchical key-value interface for config or locks.
  • ACL System: Access control for controlling read/write operations.

Example (Consul CLI snippet):

# Set a KV pair
consul kv put myapp/config "{\"max_retries\": 5}"
 
# Retrieve a KV pair
consul kv get myapp/config
 
# Register a service
consul services register -name="web" -port=8080

Typical Coordination Patterns

Leader Election Pattern

Services that require exactly one active instance use ephemeral znodes (ZooKeeper) or a compare-and-swap key (etcd) to claim leadership. Others watch that key; if the leader node fails, a new node claims leadership.

+--------------+   +---------+  +---------+
|  ZK/etcd/etc |---| Node A  |--| Node B  |
|  (KV store)  |   +---------+  +---------+
       ^
       |   ephemeral node or CAS check
       v
   Leader = Node A

Locking / Mutex

An application obtains a lock by creating a znode or updating an etcd key with a unique token. If the key is already taken, it waits or fails. Upon completion, it deletes the key or ephemeral znode to release the lock.

Distributed Configuration

Store config items under paths (e.g., /app/config/db). Clients watch for changes, so updating /app/config/db triggers watchers to reload new settings without a redeploy.

 Key: /app/config/db_url
 Value: "jdbc:mysql://dbhost:3306/mydb"

Membership and Heartbeats

Nodes create ephemeral keys signifying they’re alive. On failure or network partition, the ephemeral key goes away (session ends), alerting other components that the node is offline.

. Architecture and Deployment

Cluster Setup

Coordination services typically run on multiple nodes (3 or 5 recommended). Each node keeps a copy of the state, and they apply consensus algorithms (like ZooKeeper’s Zab or etcd’s Raft) to maintain consistency.

+----------+  +----------+  +----------+
| Node1    |  | Node2    |  | Node3    |
| (Leader) |  | (Follower|  | (Follower|
+----------+  +----------+  +----------+
         ^^^  Replication/Consensus ^^^

An odd number of nodes avoids ties. A majority must be reachable for writes to succeed.

Failover and Quorum

If the current leader fails, the remaining nodes elect a new leader. In a 3-node cluster, any 2 form a quorum to keep operating. Under deeper network splits, partial partitions lose their quorum and become read-only or unavailable.

High Availability Considerations

  • Separate from App Infrastructure: Run the coordination cluster on dedicated or robust nodes to avoid frequent restarts or resource contention.
  • Monitoring and Alerts: Track metrics like leader changes, cluster health, and latencies.
  • Network Reliability: Provide stable, low-latency links between coordination nodes.

Implementation Tips and Commands

ZooKeeper Basic Lock Example (Pseudo-code)

import org.apache.curator.framework.CuratorFramework;
import org.apache.curator.framework.recipes.locks.InterProcessMutex;
 
CuratorFramework client = ... // connect to ZooKeeper
InterProcessMutex lock = new InterProcessMutex(client, "/mylock");
try {
    lock.acquire();
    // Perform critical operation
} finally {
    lock.release();
}

etcd Lock/Lease with Command Line

# Create a lease for 10 seconds
LEASE_ID=$(etcdctl lease grant 10 | grep ID | awk '{print $3}')
 
# Lock key with that lease
etcdctl lock --lease=$LEASE_ID mylock "Performing critical section"
 
# On success, it prints that we hold the lock. The lock is released after 10s or when we exit.

Consul Leader Election Example

# Consul CLI: Acquire a session-based lock on a key
SESSION_ID=$(consul session create -name "my-leader-session" -ttl=10s | grep ID | awk '{print $2}')
 
# Attempt to acquire lock
consul kv put myapp/leader "node1" -acquire=$SESSION_ID
 
# If success, we are leader. Renew session periodically, or it expires releasing the key.

Best Practices

  1. Keep Clusters Small: 3 or 5 nodes typically suffice. More nodes slow down consensus.
  2. Dedicated Resources: Place ZooKeeper/etcd/Consul on stable hardware or VMs with minimal interference.
  3. Security: Enable TLS for inter-node traffic, use authentication for client requests, and isolate the coordination network segments.
  4. Watch Complexity: Minimally rely on watchers for heavily changing data; watchers can cause high load if triggers are frequent.
  5. Quorum Awareness: Ensure you have an odd number of nodes and that your environment can handle network partitions.
  6. Use Official Client Libraries: Avoid re-inventing concurrency logic; established recipes or built-in function calls are typically more robust.

Graph View