Lecture 17: COPS, Causal Consistency

What this lecture covers

The lecture asks whether a replicated, multi-data-center storage system can serve both reads and writes purely from the local data center, without waiting on other sites, while still avoiding the confusing behavior that pure eventual consistency allows. It uses the COPS paper as a case study, building up from a naive eventually-consistent design through two straw-man improvements to COPS's actual mechanism: tracking causal dependencies per client and shipping them with each write.

You'll see why a plausible-looking "insert a photo, then add it to a list" example can break under eventual consistency, how systems like Dynamo and Cassandra live with that weakness, and how COPS instead has each write carry the version numbers it depends on so remote data centers can delay applying it until those dependencies are visible. The lecture also covers the honest limitations of causal consistency, including cascading delays and its blindness to causality it never observed directly.

Key ideas

• Local reads and local writes: the lecture's goal system lets clients read and write only their local data center, unlike Spanner (writes need a Paxos majority) or Facebook's memcached scheme (writes go to one primary site).
• Eventual consistency: a weak guarantee that all replicas converge to the same value if writes stop, with no ordering promises in between, which lets simple straw-man designs replicate asynchronously with excellent performance.
• Anomalies: intuitive-looking code, such as inserting a photo then referencing it in a list, can fail under eventual consistency because writes can arrive at other data centers out of order.
• Version numbers and Lamport clocks: writes are timestamped so data centers agree on the "latest" value; Lamport clocks (max of local clock and one more than the highest version seen) prevent a fast clock at one server from locking out other writers.
• Last-writer-wins conflicts: concurrent writes to the same key are resolved by picking the higher timestamp, which is deterministic but can lose information, such as a counter increment overwriting another increment instead of combining them.
• Dependencies and client context: a COPS client library silently tracks which prior gets and puts a put depends on, and attaches that dependency list to the put so remote shard servers can enforce the same order.
• Causal consistency: if operation B depends on operation A, any client that observes B is guaranteed to also be able to observe A; unrelated operations have no ordering requirement, which preserves parallelism.
• Cascading delays: a shard server holding a dependent write may have to wait, possibly indefinitely under a network partition, for its dependencies to arrive, and those dependencies can themselves be waiting on further dependencies.

Walkthrough

The goal: local reads and local writes (0:03)

The lecture contrasts Spanner and the Facebook memcached design, both of which require cross-data-center coordination for writes, with the goal for today: a system where a client's writes and reads both stay local, for performance and to avoid depending on remote data centers being reachable.

Straw man one: eventual consistency (6:07)

A first design replies to a write immediately and streams it asynchronously to other data centers, giving excellent local performance but no ordering guarantees between different keys. The lecture works through a photo-and-photo-list example where a reader can see a photo referenced in a list before the photo itself has arrived, an anomaly that isn't a bug against the eventual-consistency spec but is confusing to program against. Deciding which of two concurrent writes "wins" is handled with timestamped version numbers, refined with Lamport clocks to stop a fast clock from starving other writers, and resolved via last-writer-wins, which can silently drop information like a lost counter increment.

Straw man two: sync as a barrier (32:26)

To fix the photo-list anomaly, the lecture adds a sync operator that blocks until a given version of a key is visible at every data center, so a writer can force the photo to be globally visible before adding it to the list. This works but makes writes slow and fragile to a data center being down, motivating a logging alternative where all writes at a data center pass through one ordered log server that streams changes to other sites in order. That approach restores fast local writes but funnels every write through a single log server, which becomes a bottleneck as the system grows.

COPS: dependencies instead of waiting (46:35)

COPS clients accumulate a "context" of the versions they have read or written, and attach relevant dependencies to each put rather than blocking on them. A remote shard server receiving a put with dependencies (for example, that a new value of Z depends on specific versions of X and Y) will not apply that put locally until those dependency versions are already visible in its data center, querying its local shard servers for X and Y if needed. This gives causal consistency: a client that sees a later, dependent write is guaranteed to also be able to see everything it depended on, while writes with no causal relationship proceed fully in parallel.

Formal definition and an optimization (1:02:56)

The lecture states the causal dependency rules precisely: a client's own sequential operations create dependencies, and a get depends transitively on the put that produced the value it read. It also explains an optimization: after a put, COPS can discard earlier dependencies from the client's context and keep only the put's own version number, because a remote data center that waits for that version to appear will already be waiting, transitively, for everything that version itself depended on.

Limits: cascading delays, invisible causality, and multi-object reads (57:44 and 1:15:08)

If a dependency never arrives, perhaps due to a network partition, the waiting write can block indefinitely, and dependencies on dependencies can cause long cascading waits even with no failures. COPS also only knows about causality it observed directly through gets and puts; if two users coordinate outside the system, for example by phone, COPS has no way to preserve that ordering. The lecture closes with an access-control-list example where no single order of reading two related objects guarantees a consistent combination, which motivates COPS-GT's approach of returning full dependency information on gets so a client can detect and repair such mismatches.

Before you watch

• Review the Spanner and Facebook memcached lectures in this course, since this lecture repeatedly contrasts COPS's local writes against both designs.
• Be comfortable with the general idea of eventual consistency and version numbers before this lecture, since it builds directly on top of those concepts rather than reintroducing them from scratch.

Check your understanding

1. Why can the "insert photo, then add it to the photo list" example fail under simple eventual consistency, and how does COPS prevent that failure?
2. What problem does the sync-based straw man solve, and what specific cost does it introduce compared to COPS's approach?
3. Why doesn't a COPS put need to carry every dependency in a client's full context, only the most recent one?
4. What is a cascading dependency delay, and why can it happen even without any node failing?
5. Give an example of causality that COPS cannot track, and explain why its consistency guarantee doesn't cover it.