soootaleb/ddappsctl repository overview

⁠DDAPPS

DDAPPS (Deno Distributed APPlicationS) is a framework to build distributed applications.

• Docker Hub - Server⁠
• Docker Hub - CLI⁠
• Binaries⁠ are store on CircleCI in the DenoBuildServer & DenoBuildCLI jobs ("artifacts" panel)

⁠Introduction

I've used the async nature of JavaScript in order to build distributed systems (see the examples, there are a distributed key-value store like etcd and a blockchain). Deno has great conventional web frameworks based on request/response. I've developed ddapps as an async, distributed, messages oriented framework allowing to build server applications that run and behave without any connected user. This framework is designed with heavy typing in order to help the developer, and provides base components required to build distributed systems.

Those components are:

• Client (SDK) to build your own integration
• CLI to interact with the system
• Networking to allow cluster communication
• Monitoring to observe the system
• API for clients' requests handling an routing
• Logging to known what's happening
• Peer as a core component

This document offers a high level view of the framework, its philosophy and how to use it. There is a technical documentation⁠ in the sources, along with concrete examples⁠ on how ddapps can be used for complex applications.

⁠Getting Started

⁠Interacting with a server

Ddapps works out of the box, even if it currently doesn't expose any meaningful feature by itself. This section uses the compiled binary⁠, but you can find the associated Docker image⁠. Here is the CLI Docker image⁠.

Start the server by executing ddapps. By default it uses the port 8080.

# --console-messages activates console logging
# --debug will display all messages (only clients' requests if ommited)
$ ddapps --console-messages --debug

Interact with the server using the CLI. By default it targets localhost:8080

$ ddappsctl ping

You should get a response like

{
  type: "ClientResponse",
  source: "localhost",
  destination: "Client",
  payload: { token: "px2vydgrs7", type: "Pong", payload: 3, timestamp: 1647610732123 }
}

You can see the server logs

🔄 12   DDAPPS              Messenger           InitialMessage           null
🔄 3    Net                 Peer                DiscoveryResult          {"success":false,"result":"Discovery is not activated","source":"discovery_disabled"}
🔄 2    Peer                Logger              LogMessage               {"message":"Node is ready after discovery result"}
🔄 4571 Net                 Logger              ClientConnectionOpen     {"_conn":{},"_sock":{"readyState":1,"protocol":null},"_latency":0}
🔵 0    127.0.0.1-7         Api                 ClientRequest            {"token":"aghl4yd21p","type":"Ping","payload":null,"timestamp":1647611045333}
🟢 3    Api                 127.0.0.1-7         ClientResponse           {"token":"aghl4yd21p","type":"Pong","payload":3,"timestamp":1647611045336}
🔄 0    Net                 Api                 ClientConnectionClose    "127.0.0.1-7"
🔄 0    Net                 Logger              ClientConnectionClose    "127.0.0.1-7"

⁠Adding you own feature

We will use ddapps to get a remote machine's hostname using the ddapps CLI.

Create a file hostname.ddapps.ts

import { DDAPPS, Api, EMType, Message, EOpType, state as base } from "https://deno.land/x/[email protected]/mod.ts";

// Extend from the Api component to receive the client message
class HostnameApi extends Api {

  // Override the EMType.ClientRequest handler
  protected [EMType.ClientRequest](message: Message<EMType.ClientRequest>) {

    // Call the parent method for core mechanics
    super.ClientRequest(message)

    // Send a response of type Any with the machine hostname as a payload
    this.response(message.payload.token, EOpType.Any, Deno.hostname());
  }
}

// Bootstrap you DDAPPS application
new DDAPPS()
  .use(HostnameApi) // Use your custom API instead of the base one
  .run(base); // Start the application listening on the network

You don't have to compile it for now, just start the application with Deno CLI.

# --unstable is needed to call Deno.hostname()
$ deno run -A --unstable hostname.ddapps.ts

In another terminal, you can use the ddapps CLI to send a message Any to the server.

$ deno run -A --unstable cli/ddapps.ts any

You should get a response of which the payload is you machine's hostname

{
  type: "ClientResponse",
  source: "localhost",
  destination: "Client",
  payload: {
    token: "hkth7g1fca",
    type: "Any",
    payload: "MacBook-Pro-de-Sofiane.local",
    timestamp: 1647612904435
  }
}

This basic example showed you how to bootstrap a ddapps application, extend a base component in order to add some logic, and use the CLI in order to interact with your application. So far we've only used pre-existing structures for types, enums and components. The rest of this document will show you how to create a more advanced ddapps application in order to interact with multiple components, and execute ddapps on multiple machines in order to leverage its distributed and message oriented nature.

⁠Concepts

This section will often use examples from the example directory containing a key-value store (KV store) and a blockchain.

⁠Naming

Some structures' names in ddapps are prefixed depending on their nature (not all are however)

• [I] Interfaces (e.g IMessage)
• [E] Enumerations (e.g EComponent)

In the example, the KV store structures are also prefexied with KV (e.g KVPeer), and blockchains' ones are prefixed with C or Chain (e.g ChainPeer).

⁠Messages

Messages are the building block allowing entities to exchange information (between components, but also remote peers or clients). In ddapps, messages have a type, a payload (which is typed depending on the type of the message) a source and a destination, much like a TCP packet. The response you got in the Getting Started is a typical message of type ClientResponse and a payload containing the information you requested.

interface IMessage<T> {
  type: T;
  source: string;
  destination: EComponent | string;
  payload: MPayload[T]; // MessagePayload
}

At its core, ddapps relies on (async) events using Deno.CustomEvent, disptachEvent, addEventListener & removeEventListener. CustomEvent is used for its capability to embed a payload. This payload is always a message matching the interface IMessage. User is not supposed to deal with CustomEvent since it's only a core transport object. Instead the framework exposes messages transparently.

⁠Components

A component is a singleton object that handles (non exclusively) messages. You can create as many as you need, in order to respect the separation of concerns. However there is a minimal set of components that are created out of the box by the DDAPPS factory (see section below).

• Net for network connectivity
• Peer as the main logical component
• Logger to make messages readable
• Monitor to handle cluster operations & observability
• Api to accept clients' requests & route them

In a ddapps, there can only be at least and only ONE instance of each of those base components or inherited versions. If you need to tune their behavior, extend the class and use yours. If you need to add unrelated components, inherit from the Messenger class (see section below).

You can extend any and all of those components for your own needs. They don't necessarily offer specific features but handle some important aspects of ddapps that you may want to fine tune. For example, if you want to extend the logic of adding a new node to the cluster, you may extend Peer or Net, but extending Api won't allow you (out of the box) to handle the related events.

⁠Messenger base class

All components inherit directly or indirectly from the Messenger base class. This base class subscribes the component to the correct messages, calls the correct handler, and exposes the Messenger#send method in order to allow components to send typed messages.

⁠Receiving messages

Components receive a message when they are the destination of the message (i.e the class name is exactly the same as the destination value of the message). They handle the message only if they have an associated handler (i.e a class method with a name matching the message type).

// Peer is a core component provided by ddapps, used here as an example
class Peer extends Messenger {
  // Message handlers are class methods accepting a typed message
  protected [EMType.Ping](message: IMessage<EMType.Ping>) {
    console.log("Received a ping");
  }
}

The typing of messages helps you on the payload manipulation. The TypeScript signature is reliable and can tell you what is the exact nature of the payload (properties and their types).

⁠Sending messages

To leverage the heavy typing used by ddapps, a .send() method is accessible by all components (inherited from the Messenger base class). The first argument must be a message typed derived from from EMType. The second argument is the payload and will only compile if its signature matches the message type. The third argument is the destination of the message.

class Peer extends Messenger {
  // Message handlers are class methods accepting a typed message
  protected [EMType.Ping](message: IMessage<EMType.Ping>) {
    this.send(EMType.LogMessage, {
      message: "Messages of type Any are for demo purpose and should be avoided"
    }, Logger)
  }
}

Here again, ddapps typing will help you build a safe payload by making you specify all its properties with their correct nature.

Messages can be sent to peers and clients by specifying an IP as the destination. Ddapps doesn't make much difference between a remote component and a local one. This allows you always use the same approach whether you exchange messages localy or remotely.

⁠Shared state

All components have access a shared object named state. It stores the application data such as configuration, networking resources, and any business logic you need. You can access it simply via this.state.

class Peer extends Messenger {
  // The state is passed to any messenger, no need to override the constructor
  constructor(protected state: IState) {
    this.state.ready = true;
  }
}

⁠Usage

⁠Tooling

This section does not offer a complete technical reference but instead describes some peripheral utilities at your disposal when building with ddapps.

⁠DDAPPS factory

DDAPPS class⁠ allows you to bootstrap your application without worrying of various subtlties.

The factory handles

• Instanciating and registering all necessary components, only once, in the correct order
• Performing dependency injection of the correct state in all components
• Starting the web server and accepting requests
• Sending an InitialMessage for components initialisation

You will mainly use DDAPPS.use() in order to specify what components you want to instanciate in your application.

In the getting start, the last three lines perform this action. DDAPPS relies on method chaining to let you register multiple components (see the example). You must call DDAPPS.run() at the end since it actually creates objects and starts the web server (so it's blocking).

// Bootstrap you DDAPPS application
new DDAPPS()
  .use(HostnameApi) // Use your custom API instead of the base one
  .run(base); // Start the application listening on the network

When running the application, you must provide a shared state. The Getting Started uses the default one, but you can also extend it to add your own properties. Pass it to the .run() method and the state will be injected in all components you registered.

As mentioned in the "Concepts::Components" section, there can only be at least and only ONE of each base component or its derivative. You can create and use any number of custom components that inherit from Messenger but it will still be a singleton. If you use a component that inherits one of the base components (Net, Api, Peer, Monitor, Logger), the factory will use the one provided and not create the basic one. You should not loose their capabilities since you inherit their behavior.

⁠ddappsctl CLI

⁠Typing

⁠Testing

⁠Dependencies

I try to use as few as possible dependencies. For now ddapps relies on

• Cliffy⁠ to develop the ddappsctl CLI
• Standard Library⁠ for various utilities such as HTTP server or logging colors