Replica groups

Rama’s replication guarantees that two replicas of the same task thread will never be placed on the same node. This protects a module from the failure of any individual node. However, nodes don’t always fail independently. A whole rack, data center, or cloud availability zone can go down at once, and if all replicas of a task thread were placed on nodes in that failure domain the module would lose availability for those tasks.

Rama’s "replica group" system lets you tell Rama about these failure domains so it can spread the replicas of each task thread across them. Replica groups were added in version 1.10.0.

A supervisor’s replica group is defined in its rama.yaml file under the supervisor.replica.group key. Here’s an example rama.yaml using this:

zookeeper.servers:
  - "1.2.3.4"
  - "5.6.7.8"
conductor.host: "9.10.11.12"
local.dir: "/data/rama"
supervisor.port.range:
  - 2500
  - 3500
supervisor.replica.group: "az1"

For example, if your cluster spans three availability zones you would set supervisor.replica.group on each supervisor to the availability zone in which it’s running.

Modules can then enforce replica groups when being deployed by passing the --enforceReplicaGroups flag to the deploy command like this:

rama deploy \
--action launch \
--jar target/my-application.jar \
--module com.mycompany.MyModule \
--tasks 64 \
--threads 16 \
--workers 8 \
--replicationFactor 3 \
--enforceReplicaGroups

When enforced, the replicas of each task thread will be placed on supervisors in different replica groups. If there aren’t enough supervisors across replica groups to satisfy this constraint, the deploy will fail. --enforceReplicaGroups is optional.

The --enforceReplicaGroups flag must also be passed on module update, similar to config overrides and target labels. This allows replica group enforcement to be removed by simply not passing it on an update. The flag does not have to be passed when scaling a module, however. Scaling maintains whatever replica group setting the module currently has.