---
title: AccountsDB Replication for RPC Services
---

## Problem

Validators fall behind the network when bogged down by heavy RPC load. This
seems to be due to a combination of CPU load and lock contention caused by
serving RPC requests. The most expensive RPC requests involve account scans.

## Solution Overview

AccountsDB `replicas` that run separately from the main validator can be used to
offload account-scan requests. Replicas would only: request and pull account
updates from the validator, serve client account-state RPC requests, and manage
AccountsDb and AccountsBackgroundService clean + shrink.

The replica communicates to the main validator via a new RPC mechanism to fetch
metadata information about the replication and the accounts update from the validator.
The main validator supports only one replica node. A replica node can relay the
information to 1 or more other replicas forming a replication tree.

At the initial start of the replica node, it downloads the latest snapshot
from a validator and constructs the bank and AccountsDb from it. After that, it queries
its main validator for new slots and requests the validator to send the updated
accounts for that slot and update to its own AccountsDb.

The same RPC replication mechanism can be used between a replica to another replica.
This requires the replica to serve both the client and server in the replication model.

On the client RPC serving side, `JsonRpcAccountsService` is responsible for serving
the client RPC calls for accounts related information similar to the existing
`JsonRpcService`.

The replica will also take snapshots periodically so that it can start quickly after
a restart if the snapshot is not too old.

## Detailed Solution
The following sections provide more details of the design.

### Consistency Model
The AccountsDb information is replicated asynchronously from the main validator to the replica.
When a query against the replica's AccountsDb is made, the replica may not have the latest
information of the latest slot. In this regard, it will eventually be consistent. However, for
a particular slot, the information provided is consistent with that of its peer validator
for commitment levels confirmed and finalized. For V1, we only support queries at these two
levels.

### Solana Replica Node
A new node named solana-replica-node will be introduced whose main responsibility is to maintain
the AccountsDb replica. The RPC node or replica node is used interchangeably in this document.
It will be a separate executable from the validator.

The replica consists of the following major components:

The `ReplicaSlotConfirmationRequestor`: this service is responsible for periodically sending the
request `ReplicaSlotConfirmationRequest` to its peer validator or replica for the latest slots.
It specifies the latest slot (last_replicated_slot) for which the replica has already
fetched the accounts information for. This maintains the ReplWorkingSlotSet and manages
the lifecycle of BankForks, BlockCommitmentCache (for the highest confirmed slot) and
the optimistically confirmed bank.

The `ReplicaSlotConfirmationServer`: this service is responsible for serving the
`ReplicaSlotConfirmationRequest` and sends the `ReplicaSlotConfirmationResponse` back to the requestor.
The response consists of a vector of new slots the validator knows of which is later than the
specified last_replicated_slot. This service also runs in the main validator. This service
gets the slots for replication from the BankForks, BlockCommitmentCache and OptimisticallyConfirmedBank.

The `ReplicaAccountsRequestor`: this service is responsible for sending the request
`ReplicaAccountsRequest` to its peer validator or replica for the `ReplicaAccountInfo` for a
slot for which it has not completed accounts db replication. The `ReplicaAccountInfo` contains
the `ReplicaAccountMeta`, Hash and the AccountData. The `ReplicaAccountMeta` contains info about
the existing `AccountMeta` in addition to the account data length in bytes.

The `ReplicaAccountsServer`: this service is responsible for serving the `ReplicaAccountsRequest`
and sends `ReplicaAccountsResponse` to the requestor. The response contains the count of the
ReplAccountInfo and the vector of ReplAccountInfo. This service runs both in the validator
and the replica relaying replication information. The server can stream the account information
from its AccountCache or from the storage if already flushed. This is similar to how a snapshot
package is created from the AccountsDb with the difference that the storage does not need to be
flushed to the disk before streaming to the client. If the account data is in the cache, it can
be directly streamed. Care must be taken to avoid account data for a slot being cleaned while
serving the streaming. When attempting a replication of a slot, if the slot is already cleaned
up with accounts data cleaned as result of update in later rooted slots, the replica should
forsake this slot and try the later uncleaned root slot.

During replication we also need to replicate the information of accounts that have been cleaned
up due to zero lamports, i.e. we need to be able to tell the difference between an account in a
given slot which was not updated and hence has no storage entry in that slot, and one that
holds 0 lamports and has been cleaned up through the history. We may record this via some
"Tombstone" mechanism -- recording the dead accounts cleaned up for a slot. The tombstones
themselves can be removed after exceeding the retention period expressed as epochs. Any
attempt to replicate slots with tombstones removed will fail and the replica should skip
this slot and try later ones.

The `JsonRpcAccountsService`: this is the RPC service serving client requests for account
information. The existing JsonRpcService serves other client calls than AccountsDb ones.
The replica node only serves the AccountsDb calls.

The existing JsonRpcService requires `BankForks`, `OptimisticallyConfirmedBank` and
`BlockCommitmentCache` to load the Bank. The JsonRpcAccountsService will need to use
information obtained from ReplicaSlotConfirmationResponse to construct the AccountsDb.

The `AccountsBackgroundService`: this service also runs in the replica which is responsible
for taking snapshots periodically and shrinking the AccountsDb and doing accounts cleaning.
The existing code also uses BankForks which we need to keep in the replica.

### Compatibility Consideration

For protocol compatibility considerations, all the requests have the replication version which
is initially set to 1. Alternatively, we can use the validator's version. The RPC server side
shall check the request version and fail if it is not supported.

### Replication Setup
To limit adverse effects on the validator and the replica due to replication, they can be
configured with a list of replica nodes which can form a replication pair with it. And the
replica node is configured with the validator which can serve its requests.


### Fault Tolerance
The main responsibility of making sure the replication is tolerant of faults lies with the
replica. In case of request failures, the replica shall retry the requests.


### Interface

Following are the client RPC APIs supported by the replica node in JsonRpcAccountsService.

- getAccountInfo
- getBlockCommitment
- getMultipleAccounts
- getProgramAccounts
- getMinimumBalanceForRentExemption
- getInflationGovernor
- getInflationRate
- getEpochSchedule
- getRecentBlockhash
- getFees
- getFeeCalculatorForBlockhash
- getFeeRateGovernor
- getLargestAccounts
- getSupply
- getStakeActivation
- getTokenAccountBalance
- getTokenSupply
- getTokenLargestAccounts
- getTokenAccountsByOwner
- getTokenAccountsByDelegate

Following APIs are not included:

- getInflationReward
- getClusterNodes
- getRecentPerformanceSamples
- getGenesisHash
- getSignatureStatuses
- getMaxRetransmitSlot
- getMaxShredInsertSlot
- sendTransaction
- simulateTransaction
- getSlotLeader
- getSlotLeaders
- minimumLedgerSlot
- getBlock
- getBlockTime
- getBlocks
- getBlocksWithLimit
- getTransaction
- getSignaturesForAddress
- getFirstAvailableBlock
- getBlockProduction


Action Items

1. Build the replica framework and executable
2. Integrate snapshot restore code for bootstrap the AccountsDb.
3. Develop the ReplicaSlotConfirmationRequestor and ReplicaSlotConfirmationServer interface code
4. Develop the ReplicaSlotConfirmationRequestor and ReplicaSlotConfirmationServer detailed implementations: managing the ReplEligibleSlotSet lifecycle: adding new roots and deleting root to it. And interfaces managing ReplWorkingSlotSet interface: adding and removing. Develop component synthesising information from BankForks, BlockCommitmentCache and OptimistcallyConfirmedBank on the server side and maintaining information on the client side.
5. Develop the interface code for ReplicaAccountsRequestor and ReplicaAccountsServer
6. Develop detailed implementation for ReplicaAccountsRequestor and ReplicaAccountsServer and develop the replication account storage serializer and deserializer.
7. Develop the interface code JsonRpcAccountsService
8. Detailed Implementation of JsonRpcAccountsService, refactor code to share with part of JsonRpcService.
9. Integrate with the AccountsBackgroundService in the replica for shrinking, cleaning, snapshotting.
10. Metrics and performance testing
