Investigate errors from logs - relayer Kusama to BHP finality

[bkontur]

https://grafana.teleport.parity.io/explore?schemaVersion=1&panes=%7B%227ff%22:%7B%22datasource%22:%22P8781AF333E0457B4%22,%22queries%22:%5B%7B%22refId%22:%22A%22,%22expr%22:%22%7Bpod%3D~%5C%22bridges-common-relay%2Fbridges-common-relay-headers-a-76f49567fc-m5r8w%5C%22%7D%20%7C%3D%20%60%60%22,%22queryType%22:%22range%22,%22datasource%22:%7B%22type%22:%22loki%22,%22uid%22:%22P8781AF333E0457B4%22%7D,%22editorMode%22:%22builder%22%7D%5D,%22range%22:%7B%22from%22:%22now-2d%22,%22to%22:%22now%22%7D%7D%7D&orgId=1

	2024-10-17 23:20:54.377	
2024-10-17 21:20:54 +00 ERROR bridge: Finality sync proof submission tx to BridgeHubPolkadot has failed with error: ProofSubmissionTxLost.    
2024-10-17 23:20:45.049	
2024-10-17 21:20:45 +00  INFO bridge: Synced 25385581 of 25385586 headers    
	
2024-10
	2024-10-17 23:20:20.609	
2024-10-17 21:20:20 +00  INFO bridge: Synced 25385581 of 25385582 headers    
2024-10-17 23:20:07.897	
2024-10-17 21:20:07 +00  INFO bridge: Synced 25385569 of 25385581 headers    
2024-10-17 23:19:54.868	
2024-10-17 21:19:54 +00 ERROR bridge: Finality sync loop iteration has failed with error: Target(FailedToSubmitTransaction { chain: "BridgeHubPolkadot", error: RpcError(Call(ErrorObject { code: ServerError(1010), message: "Invalid Transaction", data: Some(RawValue("Transaction has a bad signature")) })) })    
	
2024-10-17 23:19:54.868	
2024-10-17 21:19:54 +00 ERROR bridge: Failed to send transaction to BridgeHubPolkadot node: Call(ErrorObject { code: ServerError(1010), message: "Invalid Transaction", data: Some(RawValue("Transaction has a bad signature")) })    
	
2024-10-17 23:19:53.828	
2024-10-17 21:19:53 +00  INFO bridge: Synced 25385569 of 25385579 headers    

and after few minutes later, there are no errors, but syncing stopped on 25385726:

	2024-10-17 23:36:36.368	
2024-10-17 21:36:36 +00  INFO bridge: Synced 25385726 of 25385727 headers    
2024-10-17 23:36:24.262	
2024-10-17 21:36:24 +00  INFO bridge: Synced 25385726 of 25385727 headers    
2024-10-17 23:36:12.157	
2024-10-17 21:36:12 +00  INFO bridge: Synced 25385726 of 25385727 headers    
2024-10-17 23:36:00.005	
2024-10-17 21:36:00 +00  INFO bridge: Synced 25385726 of 25385727 headers    
2024-10-17 23:35:48.731	
2024-10-17 21:35:48 +00  INFO bridge: Synced 25385726 of 25385727 headers    
2024-10-17 23:35:36.624	
2024-10-17 21:35:36 +00  INFO bridge: Synced 25385726 of 25385727 headers    
2024-10-17 23:35:24.384	
2024-10-17 21:35:24 +00  INFO bridge: Synced 25385726 of 25385727 headers    
2024-10-17 23:35:10.203	
2024-10-17 21:35:10 +00  INFO bridge: Synced 25385721 of 25385726 headers    
2024-10-17 23:34:57.629	
2024-10-17 21:34:57 +00  INFO bridge: Synced 25385721 of 25385723 headers    

[bkontur]

hmm, "of header" 25385727 is not changing Synced 25385726 of 25385727 headers

[bkontur]

when I ran that locally now, the "of header" is advancing 25412898

2024-10-19 19:50:11 +00  INFO bridge: Synced 25385726 of 25412898 headers

[bkontur]

• investigate why "of header" stopped advancing

[bkontur]

• also wondering what those ProofSubmissionTxLost are:

2024-10-20 19:29:00 +00 ERROR bridge: Finality sync proof submission tx to BridgeHubPolkadot has failed with error: ProofSubmissionTxLost.
2024-10-20 19:29:13 +00  INFO bridge: Synced 25426767 of 25426769 headers
2024-10-20 19:29:26 +00  INFO bridge: Synced 25426767 of 25426771 headers
2024-10-20 19:29:39 +00  INFO bridge: Synced 25426767 of 25426771 headers
2024-10-20 19:29:51 +00  INFO bridge: Synced 25426767 of 25426775 headers
2024-10-20 19:30:04 +00  INFO bridge: Synced 25426775 of 25426777 headers
2024-10-20 19:30:16 +00  INFO bridge: Synced 25426775 of 25426779 headers
2024-10-20 19:30:29 +00  INFO bridge: Synced 25426775 of 25426782 headers
2024-10-20 19:30:42 +00  INFO bridge: Synced 25426782 of 25426783 headers
2024-10-20 19:30:54 +00  INFO bridge: Synced 25426782 of 25426787 headers
2024-10-20 19:31:07 +00  INFO bridge: Synced 25426787 of 25426789 headers
2024-10-20 19:31:20 +00  INFO bridge: Synced 25426787 of 25426791 headers
2024-10-20 19:31:32 +00  INFO bridge: Synced 25426787 of 25426792 headers
2024-10-20 19:31:37 +00 ERROR bridge: Finality sync proof submission tx to BridgeHubPolkadot has failed with error: ProofSubmissionTxLost.
2024-10-20 19:31:49 +00  INFO bridge: Synced 25426787 of 25426796 headers
2024-10-20 19:32:02 +00  INFO bridge: Synced 25426787 of 25426796 headers
2024-10-20 19:32:14 +00  INFO bridge: Synced 25426796 of 25426800 headers
2024-10-20 19:32:26 +00  INFO bridge: Synced 25426796 of 25426802 headers
2024-10-20 19:32:40 +00  INFO bridge: Synced 25426802 of 25426805 headers
2024-10-20 19:32:52 +00  INFO bridge: Synced 25426802 of 25426806 headers
2024-10-20 19:33:13 +00  INFO bridge: Synced 25426807 of 25426810 headers
2024-10-20 19:33:26 +00  INFO bridge: Synced 25426807 of 25426812 headers
2024-10-20 19:33:39 +00  INFO bridge: Synced 25426812 of 25426814 headers
2024-10-20 19:33:51 +00  INFO bridge: Synced 25426812 of 25426816 headers
2024-10-20 19:34:10 +00  INFO bridge: Synced 25426817 of 25426819 headers
2024-10-20 19:34:22 +00  INFO bridge: Synced 25426817 of 25426821 headers
2024-10-20 19:34:34 +00  INFO bridge: Synced 25426817 of 25426823 headers
2024-10-20 19:34:47 +00  INFO bridge: Synced 25426823 of 25426825 headers
2024-10-20 19:34:59 +00  INFO bridge: Synced 25426823 of 25426827 headers
2024-10-20 19:35:12 +00  INFO bridge: Synced 25426823 of 25426829 headers
2024-10-20 19:35:18 +00 ERROR bridge: Finality sync proof submission tx to BridgeHubPolkadot has failed with error: ProofSubmissionTxLost.
2024-10-20 19:35:31 +00  INFO bridge: Synced 25426831 of 25426833 headers
2024-10-20 19:35:44 +00  INFO bridge: Synced 25426831 of 25426834 headers
2024-10-20 19:35:56 +00  INFO bridge: Synced 25426831 of 25426836 headers
2024-10-20 19:36:06 +00 ERROR bridge: Finality sync proof submission tx to BridgeHubPolkadot has failed with error: ProofSubmissionTxLost.
2024-10-20 19:36:19 +00  INFO bridge: Synced 25426831 of 25426841 headers
2024-10-20 19:36:31 +00  INFO bridge: Synced 25426838 of 25426843 headers
2024-10-20 19:36:36 +00 ERROR bridge: Finality sync proof submission tx to BridgeHubPolkadot has failed with error: ProofSubmissionTxLost.
2024-10-20 19:36:48 +00 ERROR bridge: Finality sync proof submission tx to BridgeHubPolkadot has failed with error: ProofSubmissionTxLost.

[RomarQ]

Any update on this?

Yesterday, on a bridge we have between Moonbeam and Moonriver, the finality relayer between (Kusama and Moonbeam) also stopped transmitting the headers with the same exact logs you report in this issue Transaction has a bad signature. We also see Transaction has an ancient birth block.

A relayer I have locally worked flawlessly, with only some ProofSubmissionTxLost from time to time, which is related to forks.

[bkontur]

Sorry, no update on this so far :(