A production decentralised application is not one system. It is a set of failure domains joined around an onchain state machine. Contracts hold authoritative state. RPC endpoints transport access to that state. Indexers build disposable projections. The front end maintains temporary user state. Wallets control identity and signing. On L2 networks, a sequencer may add another availability dependency. Treating those components as interchangeable is how a provider outage becomes a transaction incident.
The architectural rule is simple: canonical state must remain distinguishable from cached, indexed, optimistic, and user-local state at all times. An RPC response is not an indexer response. A submitted transaction is not an included transaction. An included transaction is not necessarily final. A healthy HTTP endpoint is not necessarily a healthy chain. Ethereum’s RPC interface explicitly distinguishes latest, safe, finalized, and pending block contexts, and the execution specification notes that latest can still be reorganised under normal conditions.
A useful reference topology looks like this:
text
Copy
+----------------------+
| Monitoring plane |
| RPC, indexer, DB, UI |
| tx, contract, L2 |
+----------+-----------+
|
v
+---------+ +-----------------+-----------------+
| User | -----> | Front end state machine |
+---------+ | confirmed, pending, stale, error |
+----------+----------------+--------+
| |
read path signing path
| |
v v
+----------+------+ +-----+---------+
| RPC router | | Wallet |
| health gates | | EIP-1193 |
| fallback/quorum | | user control |
+-----+-----------+ +-----+---------+
| |
+--------+--------+ |
| | |
v v v
+-----------+ +-----------+ transaction
| RPC A | | RPC B | submission
+-----+-----+ +-----+-----+
| |
+--------+--------+
|
v
+-----------+
| Chain/L2 |
+-----+-----+
|
+-----------+-----------+
| |
v v
+-------------+ +-------------+
| Contracts | --logs->| Indexer |
| state/views | | projections |
+-------------+ +------+------+
|
v
+-------------+
| DB / cache |
+-------------+
The important boundary is the split between the read path and the signing path. A dApp should be capable of reading without a connected wallet, and losing an application RPC endpoint should not cause the application to start manipulating the user’s wallet provider as an emergency substitute. The EIP-1193 provider interface is designed for RPC access plus wallet and chain state changes, while the wallet retains responsibility for account and private-key management.
Contain
1. Contain.
When behaviour is uncertain, stop expanding the blast radius. Do not restart every service. Do not swap contract addresses. Do not deploy a speculative hotfix. Do not send users through repeated transaction attempts.
First establish which failure domain is actually unavailable.
| Failure domain | Immediate application behaviour |
|---|---|
| One RPC provider | Circuit-break that endpoint. Keep the application on already tested endpoints. Verify chain ID, head freshness, and agreement before restoring it. Ethereum exposes eth_chainId, eth_blockNumber, eth_syncing, and related RPC methods that can be used as health evidence rather than relying only on HTTP status. |
| All application RPC providers | Enter degraded or unavailable read mode. Disable transaction preparation that depends on fresh state. Do not silently switch to an unknown public endpoint. |
| Indexer | Preserve direct contract reads that are safe to perform through RPC. Disable or mark stale any aggregate, history, search, portfolio, leaderboard, or relationship view that requires the index. The Graph exposes indexer health, indexed height, chain head, and fatal error information specifically so lag and failure can be distinguished. Graph Node health signals |
| Wallet | Stay operational in read-only mode. Disable signing controls until a wallet is available. A user rejection is not an infrastructure incident. EIP-1193 assigns 4001 to user rejection, 4900 to provider disconnection, and 4901 to loss of the requested chain. EIP-1193 provider contract |
| L2 sequencer | Mark the network degraded and stop assuming ordinary transaction submission is available. An OP Stack sequencer outage can prevent normal L2 transaction submission, while a separate failure mode can allow unsafe blocks to advance even though safe and finalised heights stop progressing. OP Stack sequencer outage model |
| Contract capability | Disable the affected application action. Verify address, bytecode, proxy implementation, pause state, recent upgrades, and actual call behaviour before touching infrastructure. A front-end disable switch protects users from the application’s interface, but it does not change what an independently submitted onchain transaction can do. |
| Front-end deployment | Stop the rollout. Pin or roll back to the last known-good static build and deployment manifest before debugging secondary symptoms. |
For L2 outages, protocol escape hatches require special caution. OP Stack documentation describes direct L1 submission through OptimismPortal as a way to bypass the sequencer, but it also documents important timing and derivation caveats. That is a censorship-resistance mechanism, not a reason for a dApp to automatically reroute ordinary user transactions during an incident.
Incident owner: freeze unrelated deployments.
Application owner: expose the correct degraded mode.
RPC owner: prove endpoint health.
Indexer owner: report exact lag and last successfully processed block.
Contract owner: verify deployed state.
No random changes until those facts agree.
Stabilize
2. Stabilize.
Build the normal architecture so an outage produces reduced functionality, not ambiguous functionality.
Separate authoritative state from projections
Contracts are the business state machine. Events are a replication interface. The indexer converts those logs into a query-friendly model. The front end consumes both, but it must know which one it is showing. The Graph’s own model is explicitly an extraction and processing layer over blockchain data rather than a replacement for the underlying chain.
For every material indexed entity, retain enough provenance to answer:
text
Copy
chainId
contractAddress
blockNumber
blockHash
transactionHash
logIndex
indexerSchemaVersion
indexedAt
That gives an operator a path from a displayed row back to a specific canonical candidate event.
Events should describe the state transitions that downstream systems genuinely need. Do not make the indexer reconstruct critical business meaning from transaction calldata when a stable event can express it. Equally, do not make an indexed event projection the only way to retrieve safety-critical current state. Expose appropriate contract views so balances, permissions, pause flags, positions, nonces, limits, or equivalent state can be verified directly when necessary.
Reorganisations make this separation operationally important. Ethereum’s RPC model provides safe and finalized block tags precisely because the most recently observed canonical head carries different certainty from safer chain history.
Make RPC redundancy deliberate
Production reads should normally have at least two independently operated RPC paths, with a deliberate failure policy. A third path, including a self-operated node where justified by the service’s reliability requirements, can reduce common-provider dependency.
Do not confuse redundancy with blind round-robin load balancing.
A provider belongs in the healthy pool only after checks such as:
text
Copy
expected chainId
acceptable head age
acceptable safe/finalized lag
acceptable error rate
acceptable latency
agreement on recent block hashes
required RPC methods available
not actively syncing, where applicable
Ethereum’s JSON-RPC reference standardises the application-facing node methods, including block context, eth_call, gas estimation, transaction retrieval, receipts, node sync status, and chain identity.
For low-risk reads, fast failover is usually enough. For high-value reads used to construct transactions, calculate limits, or authorise user actions, compare responses or otherwise verify the relevant block context before trusting a replacement provider. Modern client libraries can implement fallback, priority, timeout, and provider-health strategies, but the application’s semantic checks still matter because two responsive endpoints can be at different chain heads.
Protect providers from your own application as well. Coalesce identical reads, cache immutable history, bound concurrency, apply exponential backoff with jitter, enforce method-specific timeouts, and stop retry storms when an upstream begins returning rate-limit or server errors. EIP-1193 itself recommends rate limiting at the wallet or client boundary because a provider exposed to a web application should be treated as potentially adversarial.
Treat the wallet as a separate state machine
A wallet connection is not application login state and it is not an RPC health probe.
The application must respond deterministically to connect, disconnect, chainChanged, and accountsChanged. A chain change invalidates chain-specific cached state. An account change invalidates account-specific balances, allowances, permissions, nonces, positions, and pending intent assumptions. EIP-1193 standardises those events for precisely this class of state transition.
Where several browser wallets may coexist, EIP-6963 provides event-based multi-provider discovery to avoid providers overwriting or racing around a single injected-provider slot. Its specification also warns that wallet metadata such as reverse-domain identifiers is self-attested and should not be treated as a security identity.
Do not reconnect in an infinite loop. Do not translate a 4001 user rejection into “network error”. Do not retain transaction forms under the assumption that the previous account and chain are still active.
Wallet unavailable means read-only, not application unavailable.
Simulate before requesting a signature
The transaction pipeline should be explicit:
text
Copy
load current state
|
v
validate account + chain
|
v
build exact calldata
|
v
simulate
|
v
estimate resource requirements
|
v
re-check time-sensitive constraints
|
v
request wallet confirmation
|
v
obtain transaction hash
|
v
track inclusion
|
v
track required confirmation/finality
|
v
reconcile indexer + UI
eth_call executes a message call against a specified chain state without creating an onchain transaction, while the current Ethereum Execution API also defines eth_simulateV1 for sequences of calls on simulated state. Provider support must be checked before depending on optional or newer methods.
Simulation is evidence, not a guarantee. State can change between simulation and inclusion. Another transaction can consume liquidity, alter an allowance, increment a nonce, change a price, fill an order, or modify contract state before the user’s transaction executes. Protect the actual contract call with appropriate slippage limits, expiries, expected-state checks, nonces, replay protection, or equivalent invariants rather than asking simulation to provide guarantees it cannot provide.
After the wallet returns a transaction hash, that hash becomes the primary reconciliation key. Do not show success merely because the signing request resolved. Query the transaction and receipt, classify replacement or failure correctly, and move optimistic UI state to confirmed state only at the application’s defined confirmation level. Ethereum exposes transaction and receipt lookup through the standard RPC interface.
Version deployments as one release
The production unit is not just a front-end commit.
It should be a manifest binding:
text
Copy
chain ID
contract addresses
proxy addresses, where applicable
implementation version
ABI hash/version
indexer deployment/schema version
front-end build
feature flags
supported wallet capabilities
RPC configuration version
minimum compatible backend version
release timestamp
That manifest makes “recent changes” answerable during an incident.
If upgradeable contracts are part of the design, make upgrade safety a deployment gate, not an emergency procedure. OpenZeppelin’s current upgrade tooling validates proposed implementations and compatibility before performing supported proxy upgrades, and its documentation recommends preserving network deployment metadata used by the tooling.
Do not assume upgradeability is automatically safer than immutability. It creates a recovery mechanism and an administrative attack surface at the same time. The correct design depends on governance, custody, storage-layout discipline, user expectations, and the application’s threat model.
Diagnose
3. Diagnose.
During an incident, the first operator comment should force evidence into the thread:
Current impact: Which chains, actions, regions, users, or API paths are failing? Reads, writes, or both?
Recent changes: What deployed, migrated, upgraded, rotated, scaled, or reconfigured immediately before impact?
Health signals: What do RPC head age, safe/finalised lag, provider errors, indexer lag, database health, queue depth, wallet error codes, transaction inclusion, and contract calls show?
Rollback options: What is the last known-good front end, indexer deployment, service image, configuration, or contract implementation? Which rollback is already tested?
Backup status: What database snapshot, indexer checkpoint, node snapshot, deployment manifest, and configuration backup is verified recoverable?
Ownership: One incident commander. One owner per fault domain. Post evidence, not theories.
Never post secrets, seed phrases, private keys, signing material, RPC credentials, database credentials, or privileged tokens into an incident thread.
Then establish the fault boundary.
| Layer | Signals worth collecting | What they distinguish |
|---|---|---|
| RPC | Request success, p50/p95/p99 latency, timeout rate, 429/5xx rate, eth_chainId, head age, block height, recent block hash, safe/finalised heights |
Provider outage, throttling, stale node, wrong network, partial sync, chain-level problem. Ethereum exposes chain identity, peer and sync information through RPC methods, although details of sync status vary by client. |
| Indexer | chainHeadBlock, indexed latestBlock, lag, synced, health, fatalError, query latency, DB saturation |
Query service failure versus indexing lag versus deterministic mapping failure. These fields are available through Graph Node’s indexing status interface. |
| Database | Connection saturation, CPU, memory, disk latency, replication lag, locks, slow queries, storage growth | Indexer failure caused by downstream persistence rather than chain ingestion. |
| Queues/workers | Oldest message age, queue depth, retries, dead-letter volume, consumer throughput | Backpressure versus dead worker versus poison message. |
| Wallet | 4001, 4100, 4200, 4900, 4901, chainChanged, accountsChanged |
User cancellation, permission failure, unsupported method, full disconnect, wrong-chain disconnect, or legitimate session change. |
| Transactions | Simulation failures, submission failures, returned hashes, pending age, receipt status, revert data, confirmation age | Preflight problem, wallet/provider problem, mempool issue, onchain revert, or delayed inclusion. |
| Contract | Address, deployed bytecode, implementation, pause state, authorised upgrade history, relevant event stream, view-call behaviour | Application mismatch versus actual contract-state change. |
| L2 | Unsafe head advancement, safe head advancement, finalised head advancement, transaction submission behaviour | Ordinary RPC degradation versus sequencer downtime versus failure to publish batches upstream. OP Stack explicitly distinguishes sequencer downtime from transaction-submission outages using these behaviours. |
| Front end | Build/version manifest, JS errors, API error rate, stale-cache incidence, feature flags | Chain failure versus a client release regression. |
A useful cross-check is to ask two healthy RPC providers for the same recent block by number and compare its hash. Height alone is weak evidence. Two providers can both answer quickly while one follows stale or divergent data. For a safety-sensitive workflow, also record which block context was used for the decisive read. Ethereum’s distinction between latest, safe, and finalized makes that context observable instead of implicit.
For an indexer discrepancy, reduce the problem to one entity and one chain transition:
text
Copy
UI value
|
v
indexer entity
|
v
event(s) that produced it
|
v
transaction receipt
|
v
block hash / height
|
v
direct contract read
Find the first layer that disagrees. Fix that layer.
Do not edit all five.
Recover
4. Recover.
Recovery proceeds from the smallest reversible change to the largest irreversible one.
RPC failure
Remove the unhealthy provider from routing. Validate the replacement against expected chainId, acceptable head age, required methods, and recent canonical block hashes. Restore traffic gradually. Do not place the failed endpoint back at full weight because one health request succeeded.
For an in-house node, check sync progress and peer health before declaring it serviceable. Ethereum’s RPC API exposes eth_syncing and net_peerCount for this purpose, while noting that client-specific sync details differ.
If a wallet already returned a transaction hash before the RPC incident, query that hash through healthy providers before asking the user to submit again.
Never create a new replacement transaction merely because the status screen is uncertain.
First determine whether the original transaction is pending, included, replaced, dropped, or absent and confirm the account nonce state. Duplicate user actions are often worse than a delayed status update.
Indexer failure
Stop schema changes and nonessential writers.
Snapshot the current database before destructive repair. Record the last known-good indexed block and its hash. Restore the previous indexer release or database backup when that is the tested recovery path. Replay deterministically from a verified checkpoint. Compare a small set of high-value entities against direct chain reads before directing production traffic back to the recovered index. The Graph’s health interface exposes chain head, indexed head and fatal indexing errors specifically to make this state measurable.
A blockchain index is derived state. It should therefore be rebuildable from canonical data plus versioned indexing code. A database backup still matters because replaying months or years of history may be operationally expensive.
Wallet failure
There is nothing to repair on the server merely because one user’s wallet disconnected.
Keep public data available. Stop wallet-dependent controls. React to EIP-1193 state events. Let the user reconnect deliberately. Re-read account, chain, balance, allowance, nonce, and application state after reconnection rather than resuming the previous transaction flow from memory.
If several injected wallets are expected, use standards-based provider discovery rather than assuming whichever extension most recently wrote a global provider object is the user’s intended wallet. EIP-6963 was designed to remove that collision class.
Sequencer failure
Do not “repair” a chain-level sequencer outage by repeatedly changing your dApp’s RPC configuration.
Confirm block progression and the chain’s documented outage signals. Put the product into the correct degraded state. Preserve reads when they remain trustworthy, with a visible freshness indication. Disable ordinary transaction actions whose inclusion path is unavailable.
Some rollups provide protocol-level delayed or forced inclusion paths. OP Stack documents L1 submission through OptimismPortal, and Arbitrum documents a delayed-inbox force-inclusion mechanism that reduces dependence on sequencer availability. Those mechanisms are chain-specific and carry different delay, ordering, cost, and operational assumptions.
Treat them as documented recovery or censorship-resistance procedures, not generic automatic RPC fallback.
Contract failure
This is the point where controlled urgency matters most.
If an upgrade caused the incident, first determine:
text
Copy
Is the proxy still administrable?
Is the previous implementation known?
Is storage layout compatible with rollback?
Did the failed version mutate storage?
Is a pause mechanism available?
What governance path is required?
Has the proposed recovery been simulated?
OpenZeppelin’s upgrade tooling performs upgrade-safety and compatibility checks before supported upgrades, but a rollback after state has already changed is not automatically equivalent to deploying the old bytecode again. Storage and semantic changes must be understood first.
If a tested, governance-approved rollback is demonstrably safe, use it.
If not, contain the affected contract capability and prepare a validated forward repair.
Do not trade a reversible pause for a clever onchain shortcut while state is unstable.
Front-end failure
A front-end regression is usually the cheapest rollback in the stack. Restore the previous static deployment together with its matching manifest. Do not roll back the UI alone to a version whose ABI, contract address, indexer schema, or backend assumptions no longer match production.
After recovery, reconcile pending transactions and optimistic front-end state before announcing normal operation. Users who submitted immediately before the outage may have successful onchain transactions even if your acknowledgement path failed.
Prevent
5. Prevent.
The architecture is finished only when the failure modes have owners, thresholds, and rehearsed recovery paths.
Define separate service objectives for read availability, read freshness, transaction-preparation availability, transaction-submission success, inclusion latency, and indexer freshness. A single “site uptime” percentage cannot tell you that pages are loading while every transaction is failing.
Monitor at least these control points:
| Control plane | Alerts that matter |
|---|---|
| RPC | Error ratio, timeout ratio, rate limiting, head age, cross-provider block divergence, safe/finalised lag, method availability |
| Indexer | Indexed-head lag, fatal mapping errors, query latency, database saturation, replay rate |
| Front end | Release-specific JS failures, API error rate, stale-data mode frequency, transaction funnel failures |
| Wallet | Provider discovery failure and disconnect rates, while keeping user rejection separate from infrastructure failure |
| Transactions | Simulation failure, wallet submission failure, pending age, revert rate, time to inclusion, time to required confirmation |
| Contracts | Pause changes, upgrades, ownership/admin changes, unexpected revert growth, critical event-rate anomalies |
| L2 | Block age, unsafe versus safe/finalised progression, sequencer submission behaviour |
| Backups | Backup age, restore verification, recovery-point objective, restore-time objective |
The Graph’s documented health model shows why a binary endpoint probe is insufficient for indexing: the service can answer queries while its indexed height lags the chain, and Graph Node therefore publishes both heights plus health and fatal-error information.
Test failures before production supplies them for you. Inject RPC timeouts. Return 429s. Make one provider several blocks stale. Stop the indexer. Delay database queries. Break WebSocket subscriptions and require catch-up over HTTP. Disconnect the wallet. Change accounts. Change chains. Deploy the previous front-end build. Restore an indexer database into an isolated environment. Exercise the contract pause and upgrade procedure where those mechanisms exist.
For subscriptions, assume disconnects will happen. A reconnecting WebSocket listener should resume from a durable or locally recorded block cursor and query the missed range rather than assuming the subscription delivered every event while connectivity was interrupted. This follows directly from treating logs as reconstructable chain data rather than an ephemeral notification stream. Ethereum provides historical log queries with explicit block ranges and block tags through JSON-RPC.
Back up the pieces the chain cannot reconstruct for you: deployment manifests, ABI and source artefacts, indexer definitions and migrations, database checkpoints, application configuration, observability configuration, release metadata, and operational runbooks. For derived databases and node data, verify that the recovery procedure can actually restore and then catch up.
Change management also belongs in the architecture. During normal operation, deployments should be versioned and reversible. During an active incident, unrelated changes stop. A system with ten potential recovery actions but no record of which one ran first is harder to restore than a simpler system with disciplined ownership.
Primary research question: In a controlled production-like fault test where one RPC provider remains responsive but lags the canonical head by at least three blocks, does a health-gated quorum or agreement-based read strategy reduce incorrect user-visible state and transaction preflight mismatches compared with simple round-robin or first-success failover, without breaching the application’s p95 latency target?
That question is worth measuring because it tests a central assumption behind multi-provider architecture. More RPC providers do not automatically create resilience. The mechanism that matters is whether the application can distinguish availability from freshness and agreement. Record incorrect-state incidence, simulation disagreement, transaction failure rate, p95/p99 latency, RPC cost, provider ejection time, and recovery time across repeated injected faults.
The operating conclusion is therefore stronger than “use redundant infrastructure”:
A production dApp must preserve truth boundaries when dependencies fail.
Contracts remain authoritative. Indexers remain reconstructable. Front-end state remains disposable. Wallets remain user-controlled. RPC providers remain replaceable transports. L2 sequencing remains an explicit chain dependency. Upgrades remain governed changes, not improvised recovery tools.
When any one of those components fails, the application should become less capable, but not less truthful.