Problem
We're currently running a SubQuery project to index data on the XDC Apothem network, but the indexer appears to get stuck around block height 74655418 and does not progress further.
Here are some relevant logs from the indexer:
subquery-node-1 | 2025-05-13T07:31:41.951Z <subql-node> INFO Node started on port: 3000
subquery-node-1 | 2025-05-13T07:31:41.954Z <WorkerBlockDispatcherService> INFO Enqueueing blocks 74654819...74654918, total 100 blocks
subquery-node-1 | 2025-05-13T07:31:41.958Z <WorkerBlockDispatcherService> INFO Enqueueing blocks 74654919...74655018, total 100 blocks
subquery-node-1 | 2025-05-13T07:31:41.962Z <WorkerBlockDispatcherService> INFO Enqueueing blocks 74655019...74655118, total 100 blocks
subquery-node-1 | 2025-05-13T07:31:41.968Z <WorkerBlockDispatcherService> INFO Enqueueing blocks 74655119...74655218, total 100 blocks
subquery-node-1 | 2025-05-13T07:31:42.101Z <WorkerBlockDispatcherService> INFO Enqueueing blocks 74655219...74655318, total 100 blocks
subquery-node-1 | 2025-05-13T07:31:42.111Z <WorkerBlockDispatcherService> INFO Enqueueing blocks 74655319...74655418, total 100 blocks
subquery-node-1 | 2025-05-13T07:31:42.117Z <WorkerBlockDispatcherService> INFO Enqueueing blocks 74655419...74655518, total 100 blocks
subquery-node-1 | 2025-05-13T07:31:50.294Z <WorkerBlockDispatcherService> INFO {"threadId":1,"fetchedBlocks":152,"toFetchBlocks":200,"isIndexing":false}
subquery-node-1 | 2025-05-13T07:31:50.295Z <WorkerBlockDispatcherService> INFO {"threadId":2,"fetchedBlocks":72,"toFetchBlocks":200,"isIndexing":false}
subquery-node-1 | 2025-05-13T07:31:50.295Z <WorkerBlockDispatcherService> INFO {"threadId":3,"fetchedBlocks":48,"toFetchBlocks":200,"isIndexing":false}
subquery-node-1 | 2025-05-13T07:31:50.295Z <WorkerBlockDispatcherService> INFO {"threadId":4,"fetchedBlocks":64,"toFetchBlocks":198,"isIndexing":false}
After this point, block processing halts.
RPC Endpoints Tried
We've tested with the following public RPCs:
https://earpc.apothem.networkhttps://erpc.apothem.networkhttps://rpc.apothem.network
Each of these eventually returned intermittent 502 Bad Gateway or 503 Service Unavailable errors. Example:
subquery-node-1 | 2025-05-13T07:04:02.932Z <WorkerBlockDispatcherService> INFO {"threadId":1,"fetchedBlocks":194,"toFetchBlocks":200,"isIndexing":false}
subquery-node-1 | 2025-05-13T07:04:05.425Z <FetchService> ERROR Having a problem when getting best block Error: bad response (status=502, headers={"date":"Tue, 13 May 2025 07:04:05 GMT","content-type":"text/html","transfer-encoding":"chunked","connection":"keep-alive","server":"cloudflare","nel":"{\"report_to\":\"cf-nel\",\"success_fraction\":0.0,\"max_age\":604800}","cf-cache-status":"DYNAMIC","report-to":"{\"group\":\"cf-nel\",\"max_age\":604800,\"endpoints\":[{\"url\":\"https://a.nel.cloudflare.com/report/v4?s=t7uC21m4GgnJDqeI44aL0s23WRvUQunZgFDpuQCps%2FOhfo9ASEAnGZCsfZCpVo3z%2FbI3AubthSF3EIfrTsSe6oqy74QIbacSTx5oCSMgc19CY6Ii7ZaOO4ko7tnfq%2BfSX7BbyS%2F8wvE%3D\"}]}","cf-ray":"93f04cb8db2cef4e-LHR","alt-svc":"h3=\":443\"; ma=86400"}, body="<html>\r\n<head><title>502 Bad Gateway</title></head>\r\n<body>\r\n<center><h1>502 Bad Gateway</h1></center>\r\n<hr><center>nginx/1.18.0 (Ubuntu)</center>\r\n</body>\r\n</html>\r\n", requestBody="[{\"method\":\"eth_getBlockByNumber\",\"params\":[\"latest\",false],\"id\":68,\"jsonrpc\":\"2.0\"},{\"method\":\"eth_getBlockByNumber\",\"params\":[\"latest\",false],\"id\":69,\"jsonrpc\":\"2.0\"}]", requestMethod="POST", url="https://earpc.apothem.network", code=SERVER_ERROR, version=web/5.7.1)
subquery-node-1 | 2025-05-13T07:04:05.426Z <FetchService> ERROR Having a problem when getting finalized block Error: bad response (status=502, headers={"date":"Tue, 13 May 2025 07:04:05 GMT","content-type":"text/html","transfer-encoding":"chunked","connection":"keep-alive","server":"cloudflare","nel":"{\"report_to\":\"cf-nel\",\"success_fraction\":0.0,\"max_age\":604800}","cf-cache-status":"DYNAMIC","report-to":"{\"group\":\"cf-nel\",\"max_age\":604800,\"endpoints\":[{\"url\":\"https://a.nel.cloudflare.com/report/v4?s=t7uC21m4GgnJDqeI44aL0s23WRvUQunZgFDpuQCps%2FOhfo9ASEAnGZCsfZCpVo3z%2FbI3AubthSF3EIfrTsSe6oqy74QIbacSTx5oCSMgc19CY6Ii7ZaOO4ko7tnfq%2BfSX7BbyS%2F8wvE%3D\"}]}","cf-ray":"93f04cb8db2cef4e-LHR","alt-svc":"h3=\":443\"; ma=86400"}, body="<html>\r\n<head><title>502 Bad Gateway</title></head>\r\n<body>\r\n<center><h1>502 Bad Gateway</h1></center>\r\n<hr><center>nginx/1.18.0 (Ubuntu)</center>\r\n</body>\r\n</html>\r\n", requestBody="[{\"method\":\"eth_getBlockByNumber\",\"params\":[\"latest\",false],\"id\":68,\"jsonrpc\":\"2.0\"},{\"method\":\"eth_getBlockByNumber\",\"params\":[\"latest\",false],\"id\":69,\"jsonrpc\":\"2.0\"}]", requestMethod="POST", url="https://earpc.apothem.network", code=SERVER_ERROR, version=web/5.7.1)
Debugging Attempts
We also tried enabling unfinalized block indexing:
--unfinalized-block=true
Which showed:
<UnfinalizedBlocks> INFO Unfinalized blocks is enabled
Attempting to set a higher startBlock (e.g. 75000000 [2days ago]) causes a fatal error:
<subql-node> ERROR undefined Error: Node failed to start
Cause: Error: Value at height 74654719 was undefined
But this eventually fails.
What We've Tried
Switching between all public Apothem RPCs
Enabling unfinalized blocks with --unfinalized-block=true
Changing startBlock to a higher value
Restarting the indexer container
Repo reference: https://github.com/subquery/ethereum-subql-starter/tree/main/XDC/xdc-apothem-starter
Discussion (5)
Hi, I'm the lead SubQuery SDK dev.
The issues you're having look to be related to the RPC rather than SubQuery. I have tried to reproduce this but not had any luck, indexing seems fine for me.
What version of the SDK are you using?
subquerynetwork/subql-node-ethereum:v6.0.5
subquerynetwork/subql-query:v2.22.0
"@subql/cli": "5.9.1",
"@subql/node-ethereum": "6.0.5"
yes, it seems like RPC issue we are facing too many unfinalized blocks error and slow indexing.
Errors Screenshot Reference:
docs.google.com/document/d/1Oe6OeU...
Are you still encountering this issue?
docs.google.com/document/d/1BjwJov...
I have attache the screenshots for reference.
yes we are facing same issue now on different blocks. and the indexing stucks at 0.0 blocks/s or "ERROR Having a problem when getting finalized block"