Hyperlane Troubleshooting Guide

General Troubleshooting

Logging

Logging levels and formats can be configured for Hyperlane tooling using the following two environment variables:

• LOG_LEVEL: The log level to filter to. Defaults to info. Choices: debug | info | warn | error | off
• LOG_FORMAT: The format of the log output. Defaults to pretty. Choices: pretty | json

The Hyperlane CLI also allows configuring logging via the --log and --verbosity flags.

Chain Configuration

Within your working directory, you may find a chains/ yaml files organized by chain name. These metadata.yaml files describe the information needed to use the chain in Hyperlane deployments and apps.

You can define a full configuration for any new chain in this file. The metadata that can be configured is defined in this example configuration. You can also find the chain metadata schema at chainMetadataTypes.ts.

Here's an example configuration for two local anvil chains:

Example: Two Local Anvil Chains

anvil1:
  chainId: 31337
  domainId: 31337
  name: anvil1
  protocol: ethereum
  rpcUrls:
    - http: http://localhost:8545
  nativeToken:
    name: Ether
    symbol: ETH
    decimals: 18
anvil2:
  chainId: 31338
  domainId: 31338
  name: anvil2
  protocol: ethereum
  rpcUrls:
    - http: http://localhost:8555

You can also extend a core chain config by providing the fields to be overridden:

sepolia:
  blocks:
    confirmations: 2

Override RPC URLs

You can override the RPC urls by extending the core chain config.

Example: Define RPC URLs array and adjust retry settings

demochain:
  name: demochain
  chainId: 123456
  domainId: 123456
  protocol: ethereum
  rpcUrls:
    - http: https://rpc-testnet.demochain.gg
sepolia:
  rpcUrls:
    - http: https://rpc2.sepolia.org
    - http: https://some-other-sepolia-rpc.gg
      retry:
        maxRequests: 10

Override transaction settings

Transaction overrides are any properties to include when forming transaction requests. For example:

• gasPrice: number | string
• maxFeePerGas: number | string
• maxPriorityFeePerGas: number | string
• nonce: number | string
• type: number
• ccipReadEnabled: boolean

In the example below we're using a gas price of 7 gwei, hardcoding the nonce, and setting a maximum value for the base and priority fees.

sepolia:
  transactionOverrides:
    gasPrice: 7000000000 # 7 gwei
    maxFeePerGas: 150000000000 # 150 gwei
    maxPriorityFeePerGas: 40000000000 # 40 gwei
    nonce: 1337

warning

If you are overriding the nonce in the chain configuration, ensure you are updating the value on successful transactions.

Message Delivery Issues

Long Message Delivery Times or Timeouts

Delays or timeouts in message delivery often result from RPC issues, such as overloaded endpoints, or from chains that produce blocks only on demand.

To identify the issue, check for logs indicating RPC-related problems such as: Deprioritizing an inner provider in FallbackProvider

To address these delays:

• Check RPC Health: Ensure RPC endpoints are responsive.
• Configure Chains with On-Demand Blocks: For these chains, set the reorgPeriod to 0. This ensures that agents always look at the tip of the chain, avoiding delays caused by block finalization processes.

Agent Troubleshooting

Validator Troubleshooting

• Validator has not announced any storage locations Warning:

  • Reason: This occurs when the relayer cannot retrieve validator signatures required to process a message.
  • Solution:
    • Ensure the validators have announced their storage locations. Check validator logs for a message such as: Validator has announced signature storage location, locations: ["file:///tmp/hyperlane-validator-signatures-local"]
    • Verify that each validator has a unique signature storage path (--checkpointSyncer.path) to prevent overwriting.
    • Confirm that the relayer has read access to the storage paths.

• No ISM Found for Origin Error

  • Reason: This occurs when the Interchain Security Module (ISM) does not recognize the origin chain.
  • Solution:
    • Ensure the ISM configuration includes the Validators for the origin chain.
    • If it doesn't, add validators for the origin chain to your ISM.
    • Restart the relayer after updating the ISM configuration.

Relayer Troubleshooting

Checking Pending Messages in the Queue

If messages are stuck in the relayer queue or not being processed, use the list_operations endpoint to inspect the relayer's queue:

curl http://0.0.0.0:9090/list_operations?destination_domain=<destination_domain_id>

This endpoint provides the status of messages in the queue and can help identify why they are not being delivered.

Debugging with Logs

For detailed insights, enable debug logging and monitor the relayer's activity: HYP_LOG_LEVEL=debug. Once logs are captured, you can use them to locate issues with specific message IDs.

Retrying Messages

After reviewing the logs, you can trigger an immediate retry of all pending messages using the message_retry endpoint:

curl --location 'http://127.0.0.1:9090/message_retry' \
--header 'Content-Type: application/json' \
--data '[{"messageid": "*", "origindomain": "*", "senderaddress": "*", "destinationdomain": "*", "recipientaddress": "*"}]'

Make sure that you use the most recent version of the relayer and capture all logs for debugging.

Relayer Selecting Incorrect ISM

Ensure the Interchain Security Module (ISM) on your chain is configured correctly. An incorrect ISM can result in messages not being processed. Verify the ISM address in your configuration and ensure it matches your deployment setup.

Advanced Troubleshooting

eth_getStorageAt() Compatibility

Some chains do not natively support the eth_getStorageAt() API. If you're deploying on one of these chains and encounter issues, review the changes made to the Hyperlane codebase in this commit.

To deploy Hyperlane on these chains follow the steps:

1. Clone the repo
2. Apply the changes in the commit.
3. Build the project from the root directory: yarn build
4. Run the Hyperlane CLI from the root directory: yarn workspace @hyperlane-xyz/cli hyperlane