docs: Query proxy server operations manual (#3916)

* Doc: Query proxy server operations manual

* Review rework

cspell-custom-words.txt

@@ -120,6 +120,7 @@ optin
 Optin
 parachain
 pdas
+permissioned
 permissionless
 permissionlessly
 Polkachu

docs/query_proxy.md

@@ -0,0 +1,201 @@
+# Operating the Wormhole Queries Proxy Server
+
+The Wormhole queries proxy server (sometimes referred to as the CCQ proxy) is a server that listens on a
+REST endpoint for Wormhole query requests. It validates those requests and forwards them to the guardian
+CCQ P2P network for processing by the guardians. It then accumulates the responses from the guardians,
+verifies quorum and forwards the response to the client.
+
+## Building the Proxy Server
+
+The proxy server runs as another instance of the `guardiand` process, similar to the spy. It is built exactly
+the same as the spy, and requires the same dependencies. Please see the [Operations Guide](operations.md#building-guardiand) for
+details on how to build `guardiand`.
+
+## Deploying the Proxy Server
+
+The proxy server can be deployed just like the spy, including potentially running in a container. Note that it
+requires a public IP address to listen for REST requests, and it needs to be able to reach the guardian P2P network.
+
+The proxy is not particularly resource intensive, so should run successfully on a reasonable size VM.
+
+## Configuring the Proxy Server
+
+There are two main parts to configuring the proxy server. The first is setting up the command line arguments,
+which generally will not change after initial setup. The second part of the configuration is the permissions file,
+which will change as the requirements of integrators change.
+
+### Proxy Server Command Line Arguments
+
+The following is a sample command line for running the proxy server in mainnet.
+
+```shell
+wormhole $build/bin/guardiand query-server \
+      --env "mainnet" \
+      --nodeKey /home/ccq/data/ccq_server.nodeKey \
+      --permFile "/home/ccq/data/ccq_server.perms.json" \
+      --signerKey "/home/ccq/data/ccq_server.signerKey" \
+      --listenAddr "[::]:8080" \
+      --ethRPC https://eth.drpc.org \
+      --ethContract "0x98f3c9e6E3fAce36bAAd05FE09d375Ef1464288B" \
+      --logLevel=info \
+      --telemetryLokiURL $LOKI_URL \
+      --telemetryNodeName "Mainnet CCQ server 1" \
+      --promRemoteURL $PROM_URL
+```
+
+- The `env` can be mainnet, testnet or devnet.
+- The `nodeKey` should point to the file containing the P2P key. The first time the proxy runs, if the
+  file does not exist, it will be created. You can look in the proxy server logs to get the generated key.
+- The `permFile` is the JSON permissions file, which is documented below.
+- The `signerKey` should point to an armored file containing a key that will be used to sign requests received
+  from integrators who are configured to support auto signing and opt not to sign a request. Please see below
+  for how to generate this file.
+- The `listenAddr` specifies the port on which the proxy listens for REST requests.
+- The `ethRPC` and `ethContract` are used to read the wormhole guardian set on start up. The address
+  above is for mainnet. If you are running in testnet, you should point to Holesky and use `0xa10f2eF61dE1f19f586ab8B6F2EbA89bACE63F7a`.
+  (You can confirm these addresses [here](https://docs.wormhole.com/wormhole/reference/constants#contract-addresses).)
+  Note that using a public endpoint should be fine, since the proxy only does a single read of the guardian set.
+- The `telemetryLokiURL`, `telemetryNodeName` and `promRemoteURL` are used for telemetry purposes and
+  the values will be provided by Wormhole Foundation personnel if appropriate.
+
+#### Creating the Signing Key File
+
+Do the following to create the signing key file. Note that the `block-type` must exactly match what is specified below,
+but the `desc` can be anything you want.
+
+```shell
+wormhole$ build/bin/guardiand keygen --desc "Your CCQ proxy server" --block-type "CCQ SERVER SIGNING KEY" /home/ccq/data/ccq_server.signerKey
+```
+
+### Guardian Support for a New Proxy Server
+
+The Queries P2P network is permissioned. The guardians will ignore P2P traffic from sources that are not in their configuration.
+Additionally, they will only honor query requests signed using a key in their configured list. Before you can begin publishing
+requests from your proxy, you must get a quorum (preferably all) of the guardians to add your values for the following to their
+configurations:
+
+- P2P key (the value from `nodeKey` file, logged on proxy start up).
+- The public key associated with the signing key. See the `signerKey` file.
+
+Please work with foundation personnel to get your proxy server added to the guardian configurations.
+
+### Permissions Configuration
+
+The file specified by the `permFile` parameter contains JSON that defines the set of allowed queries users, along with the
+sets of requests they are allowed to make.
+
+#### File Format
+
+The simplest file would look something like this
+
+```json
+{
+  "permissions": [
+    {
+      "userName": "Monitor",
+      "apiKey": "insert_generated_api_key_here",
+      "allowUnsigned": true,
+      "allowedCalls": [
+        {
+          "ethCall": {
+            "note:": "Name of WETH on Ethereum",
+            "chain": 2,
+            "contractAddress": "0xC02aaA39b223FE8D0A0e5C4F27eAD9083C756Cc2",
+            "call": "0x06fdde03"
+          }
+        }
+      ]
+    }
+}
+```
+
+This creates a single user called "Monitor", who will use the specified API key (more on API keys below).
+This user is allowed to submit unsigned requests (which will be signed using the configured signing key).
+
+This sample user is only allowed to make a single `ethCall` request on Ethereum (Wormhole chain ID 2),
+which allows them to call the `name` method on the contract that resides at `0xC02aaA39b223FE8D0A0e5C4F27eAD9083C756Cc2`.
+The `call` parameter is the first four bytes of the hash of the ABI encoded function call to be allowed.
+
+A given user can have any number of allowed calls (at least one), but they can only make calls that are configured here.
+
+#### Supported Call Types
+
+The proxy server supports all of the query types supported by the Wormhole Queries protocol. For details on those calls,
+please see the [Wormhole Queries Whitepaper](../whitepapers/0013_ccq.md).
+
+The following are the EVM call types, all of which require the `chain`, `contractAddress` and `call` arguments.
+
+- `ethCall`
+- `ethCallByTimestamp`
+- `ethCallWithFinality`
+
+The following are the Solana call types. Both require the `chain` parameter plus the extra parameter listed below.
+
+- `solAccount`, requires the `account` parameter.
+- `solPDA`, requires the `programAddress` parameter.
+
+The Solana account and and program address can be expressed as either a 32 byte hex string starting with "0x" or as a base 58 value.
+
+#### Creating New API Keys
+
+Each user must have an API key. These keys only have meaning to the proxy server. They are not passed to the guardians.
+The proxy requires that a key be present in each query request, and that the specified key exists in the permissions file.
+Beyond that, the API keys have no special meaning. They can be generated using a site like [this](https://www.uuidgenerator.net/version4).
+
+#### Updating the Permissions File
+
+The proxy server monitors the permissions file for changes. Whenever a change is detected, it reads the file, validates it, and if
+it passes validation, switches to the new version. Care should be taken when editing the file while the proxy server is running, because
+as soon as you save the file, the changes will be picked up (whether they are logically complete or not).
+
+## Telemetry
+
+The proxy server provides two types of telemetry data, logs and metrics.
+
+### Logging
+
+The proxy server uses the same logging mechanism as the guardian. It will write to a local file, but can also be configured to
+publish logs to Grafana using the Loki protocol. If you will be running your proxy server in mainnet, you should contact foundation
+personnel about getting a Grafana ID to be used for logging and use it to set the `--telemetryLokiURL` command line argument.
+
+If you set the log level to `info`, the proxy server logs information on all incoming requests and output bound responses. This can
+be helpful for determining when requests reach quorum, but may be too chatty as the level of queries traffic grows. If that is the
+case, you can set the log level to `warn`.
+
+### Metrics
+
+The proxy server uses Prometheus to track various activity and can publish them to Grafana. If you will be running your proxy server in mainnet,
+you should contact foundation personnel about getting a Grafana ID and use it to set the `--promRemoteURL` command line argument.
+
+For the set of available metrics, see [here](../node/cmd/ccq/metrics.go).
+
+## Troubleshooting
+
+### P2P Health
+
+If you think you are having trouble with your access to the P2P network, you can add `--monitorPeers` to the command line arguments,
+which will cause the proxy server to periodically check its connectivity to the P2P bootstrap peers, and attempt to reconnect if necessary.
+
+### Invalid Requests
+
+If the proxy server determines that a request is invalid, it does the following:
+
+- Logs an error message using the user name (not the API Key).
+- Increments the appropriate Prometheus metric.
+- Sends a failure response to the user.
+
+Note that if the proxy server thinks a request is valid, but the guardians do not, the guardians silently drop the request, so it will look
+like a timeout. This is to avoid a denial of service attack on the guardians. This can happen if the proxy server is not properly permissioned
+on the guardians.
+
+### Logging Request Detail.
+
+If a given integrator is reporting problems with their queries, you may find it useful to add the following to their permissions config
+(at the same level as the API Key, etc).
+
+```json
+"logResponses": true,
+```
+
+This will cause the proxy server to log every response received for that user, along with the number of responses and how many are
+still needed to meet quorum.