adexnetwork/invictus-rpc repository overview

⁠Summary

Invictus RPC is a reverse proxy for Ethereum/EVM RPC providers, that aims to provide a single resilliant endpoint that aggregates multiple providers, and allows failover and quorum.

Originally discussed in this issue⁠ which outlines it's goals and requirements.

⁠Configuration

Configuration looks like this:

{
  "check_interval_seconds": 30,
  "networks": {
    "ethereum": {
      "chain_id": 1,
      "providers": [
        { "url": "https://eth.llamarpc.com", "priority": 2, "tiebreak_weight": 1 }
      ],
      "block_majority_divider": 1
    }
  }
}

The configuration is dynamically loaded from the main branch of this repo. In case this fails, we fallback to the configuration file as of when the binary of invictus-rpc was built.

⁠Priority match

Priority match allows you to set a higher/lower priority for a provider if the request body string matches a certain sub-string.

For example, in providers:

    { "url": "https://eth.best-logs-provider.com", "priority_match": { "eth_getLogs": 99 }, "priority": 2, "tiebreak_weight": 1 }

⁠Inner error detection

On the top level of the config, you can provide an array of strings:

  "inner_errors": [
    "\"error\":{\"code\":-32601",
    "\"error\":{\"code\":-32603",
    "\"error\":{\"code\":-32000",
    "\"we can't execute this request\""
  ],

If any of those are detected in the first 1024 bytes of the response body string and there are remaining retries, we will fallback to another provider.

⁠How does it work

All providers are periodically checked for quorum - only the ones in the quorum are considered for serving each request. From those, we take the ones with the highest priority, and if there's more than 1 with the highest priority, we choose randomly based on tiebreak_weight (which defaults to 1). For example, if there are 5 providers, but one is down, 4 will be considered. Let's say that two of them have the highest priority - they will serve all the requests, but one has tiebreak_weight of 2, the other the default of 1, so the first one will serve 2x more requests than the other.

This system allows us to fine-tune almost everything: we can implement fallback, but we can prioritize specific providers and load-balance at the same time.

⁠Features

• failover: automatically excludes RPC providers which are down based on a quorum
• priorities: you can prioritize some RPC providers to be preferred over others
• load balancing: if more than one RPC provider has the same priority, we load-balance between them. We can control how much traffic to be sent via tiebreak_weight
• remote reading of the configuration: configuration can be changed without re-deploying
• periodic quorum selection: the provider quorum is re-determined once every N seconds (configurable)
• failure detection: even if the RPC provider successfully responds to rpc_getBlock (what we use to determine quorum), it can still be excluded if it's failing often (FailingOften); furthermore, once a provider fails during the quorum selection once, it can be, based on configuration, excluded from the next N quorum selections (FailedRecently); if a provider frequently returns a good response but is at a different block than the quorum, it will be excluded from selection for some time as well
• quorum fallback: even if there is no majority group (no agreement on block), the highest block will be selected, since the design goal is resilliance
• performance tracking: we track the performance of different providers over real load
• GET mode: GET mode allows to make JSON-RPC requests using GET, which is useful to configure NGINX/Cloudflare caching for said GET requests

⁠Pitfalls

⁠Config suggestion

For chains with quick block times, please do not add more than 2 RPC providers. The reason for this is that with more than 2, they can all be at different blocks, leading to an inability to reach consensus. In case we can't reduce them to 2, 4 is better than 3 because it's more likely that at least 2 will be on the same block.

NOTE: we fixed this with the quorum fallback mode⁠, but it's still a good practice to avoid 3 and either use 2 or 4.

⁠No quorum for network after startup

When Invictus starts, there's a small period of time during which it might respond with "no quorum for network" - this only happens if it takes more than 10 seconds to get a quorum for a network OR if there's legitimately no good quorum for the network (all providers down).

Theoretically, the first case should never hapepn because each provider check is limited to 7 seconds.

⁠Routes

• POST /:network: JSON-RPC proxy for the given network. For example, https://invictus.ambire.com/ethereum is an Ethereum JSON-RPC proxy
• GET /:network/get/:method/:params: proxy for the given network but JSON-RPC requests are encoded entirely in the URL
• GET /:network/status: show the configuration and current quorum for the underlying providers for the given network
• PUT /:network/configuratione-reficite: hot reload of the configuration; this will immediately re-set the quorum based on the new config
• GET /picturam: return global data structure of current invictus instance

⁠Reasons to exclude an RPC (failure modes)

There are many reasons to exclude an RPC:

• Failed the last quorum due to returning an error
• Failed the last quorum recently due to returning an error - after returning one error, it's excluded for N more quorums (FailedRecently)
• Didn't make the last quorum because the block is not the dominant block (eg RPC is unusually behind on unusually ahead)
• A certain % of the last proxy requests ended in a non-200 HTTP code (FailingOften)
• A certain % of the last quorum cycles ended up excluding this RPC (FailingOften)

⁠How to use GET mode

curl 'http://localhost:1337/ethereum/get/eth_getBalance/\["0x77777777789A8BBEE6C64381e5E89E501fb0e4c8","latest"\]' | jq
{
  "jsonrpc": "2.0",
  "id": 1,
  "result": "0x66f1b821a9adda16"
}

⁠How to use traditional POST mode

curl -H "Content-Type: application/json" --data "{\"jsonrpc\":\"2.0\",\"method\":\"net_version\",\"params\":[],\"id\":1}" http://localhost:1337/ethereum

NOTE: this is a bit fiddly because the URL has to be correct, for example no whitespaces are allowed

NOTE: In GET mode, it's easier too pass invalid JSON. In this case, Ankr returns good feedback (tells you what's wrong about the JSON), while Infura/Alchemy do not (just say invalid JSON). Llama RPC returns feedback but not in JSON form.

⁠How to run custom tests

Receipts:
cargo test --test receipt_checks -- --ignored --nocapture