Loquat: A Design For Large-Scale Distributed Applications
Scaling applications built on Distributed Erlang is a difficult task. Distributed Erlang relies on a global process registry, supported by a full-mesh overlay network, which restricts scalability due to the replication overhead of the registry, and the connection and failure detection overhead required by each participating node in the system. Recently, attempts have been made to resolve the problems of Distributed Erlang: SD-Erlang and Ericsson’s planned port of global to the Kademlia DHT are focused on reducing the amount of information that each node in the global registry needs to replicate, but that only temporarily mitigates the problem. These approaches to reduce the cost of maintaining the global registry only temporarily solve the problem: these techniques are insufficient for building geo-distributed, large-scale actor systems, or any applications where nodes in the system will experience high partitioning and members of the DHT become unreachable. Additionally, the ability to explicitly name each process, and select a member as a target for a message, becomes extremely problematic at large-scales with thousands of nodes, and hundreds of thousands of processes. We present Loqaut, a design for large-scale actor programming on the Erlang virtual machine. Loquat is comprised of two components: a highly-scalable, resilient, distribution backend for actor systems, with a novel programming model that generalizes the Erlang actor model, for building, large-scale, distributed applications.
Highlight the design decisions of Erlang and Distributed Erlang and present a new design, focused on scalability and fault-tolerance.
Distributed Erlang users.
Christopher Meiklejohn is a Erasmus Mundus doctoral student at Université catholique de Louvain and Instituto Superior Técnico.