Checkout failure: "Git fetch failed with exit code 128"

[carlossanlop]

Build

https://dev.azure.com/dnceng/public/_build/results?buildId=1706886&view=results

Build leg reported

Libraries Test Run release coreclr Linux_musl x64 Release

Pull Request

Rolling build for runtime

Action required for the engineering services team

To triage this issue (First Responder / @dotnet/dnceng):

• Open the failing build above and investigate
• Add a comment explaining your findings

If this is an issue that is causing build breaks across multiple builds and would get benefit from being listed on the build analysis check, follow the next steps:

1. Add the label "Known Build Error"
2. Edit this issue and add an error string in the Json below that can help us match this issue with future build breaks. You should use the known issues documentation

{
   "ErrorMessage" : "Git fetch failed with exit code: 128",
   "BuildRetry": true
}

Additional information about the issue reported

The Checkout dotnet/runtime@main to s step is the one that failed. Here is the end of the log output:

remote: Compressing objects:  29% (30070/103687)        
remote: Compressing objects:  30% (31107/103687)        
remote: Compressing objects:  31% (32143/103687)        
remote: Compressing objects:  31% (32322/103687)        
remote: Compressing objects:  32% (33180/103687)        
remote: Compressing objects:  33% (34217/103687)        
remote: Compressing objects:  33% (35247/103687)        
remote: Compressing objects:  34% (35254/103687)        
remote: Compressing objects:  35% (36291/103687)        
remote: Compressing objects:  36% (37328/103687)        
remote: Compressing objects:  37% (38365/103687)        
remote: Compressing objects:  37% (38926/103687)        
remote: Compressing objects:  38% (39402/103687)        
error: RPC failed; curl 18 transfer closed with outstanding read data remaining
fatal: protocol error: bad pack header
##[error]Git fetch failed with exit code: 128
Finishing: Checkout dotnet/runtime@main to s

Report

Build	Definition	Step Name	Console log	Pull Request
1294676	dotnet/runtime	Checkout dotnet/runtime@refs/pull/124407/merge to s	Log	dotnet/runtime#124407
1293674	maestro-auth-test/helix-queue-insights-test	Checkout maestro-auth-test/helix-queue-insights-test@refs/pull/1124/merge to s	Log	maestro-auth-test/helix-queue-insights-test#1124
1292965	dotnet/runtime	Checkout dotnet/runtime@refs/pull/124309/merge to s	Log	dotnet/runtime#124309
1292606	dotnet/sdk	Checkout dotnet/sdk@refs/pull/53022/merge to s	Log	dotnet/sdk#53022
1292603	dotnet/sdk	Checkout dotnet/sdk@refs/pull/53021/merge to s	Log	dotnet/sdk#53021
1292503	dotnet/aspnetcore	Checkout dotnet/aspnetcore@refs/pull/65413/merge to s	Log	dotnet/aspnetcore#65413
1292514	dotnet/sdk	Checkout dotnet/sdk@refs/pull/52468/merge to s	Log	dotnet/sdk#52468
1292504	dotnet/aspnetcore	Checkout dotnet/aspnetcore@refs/pull/65413/merge to s	Log	dotnet/aspnetcore#65413
1292506	dotnet/aspnetcore	Checkout dotnet/aspnetcore@refs/pull/65413/merge to s	Log	dotnet/aspnetcore#65413
1292351	dotnet/runtime	Checkout dotnet/runtime@refs/pull/124356/merge to s	Log	dotnet/runtime#124356
1291521	dotnet/runtime	Checkout dotnet/runtime@refs/pull/124338/merge to s	Log	dotnet/runtime#124338
1291615	dotnet/arcade-services	Checkout dotnet/arcade-services@refs/pull/5941/merge to s	Log	dotnet/arcade-services#5941
1291484	dotnet/runtime	Checkout dotnet/runtime@refs/pull/124340/merge to s	Log	dotnet/runtime#124340
1291228	dotnet/runtime	Checkout dotnet/runtime@refs/pull/124310/merge to s	Log	dotnet/runtime#124310
1290913	dotnet/runtime	Checkout dotnet/runtime@refs/pull/124312/merge to s	Log	dotnet/runtime#124312
1290974	dotnet/runtime	Checkout dotnet/runtime@refs/pull/124314/merge to s	Log	dotnet/runtime#124314
1290967	dotnet/runtime	Checkout dotnet/runtime@refs/pull/124316/merge to s	Log	dotnet/runtime#124316
1290853	dotnet/runtime	Checkout dotnet/runtime@refs/pull/124246/merge to s	Log	dotnet/runtime#124246
1290802	maestro-auth-test/helix-queue-insights-test	Checkout maestro-auth-test/helix-queue-insights-test@refs/pull/1123/merge to s	Log	maestro-auth-test/helix-queue-insights-test#1123
1290916	dotnet/runtime	Checkout dotnet/runtime@refs/pull/124313/merge to s	Log	dotnet/runtime#124313
1288532	dotnet/aspnetcore	Checkout dotnet/aspnetcore@refs/pull/65381/merge to s	Log	dotnet/aspnetcore#65381
1288438	dotnet/sdk	Checkout dotnet/sdk@refs/pull/48879/merge to s	Log	dotnet/sdk#48879
1287920	dotnet/runtime	Checkout dotnet/runtime@refs/pull/124095/merge to s	Log	dotnet/runtime#124095
1287634	dotnet/sdk	Checkout dotnet/sdk@refs/pull/52909/merge to s	Log	dotnet/sdk#52909
1286359	dotnet/sdk	Checkout dotnet/sdk@refs/pull/52915/merge to s	Log	dotnet/sdk#52915
1286337	dotnet/runtime	Checkout dotnet/runtime@refs/pull/123860/merge to s	Log	dotnet/runtime#123860
1286020	dotnet/maui	Checkout dotnet/maui@inflight/candidate to s	Log
1286077	dotnet/runtime	Checkout dotnet/runtime@release/8.0 to s	Log
1286413	dotnet/runtime	Checkout dotnet/runtime@refs/pull/123379/merge to s	Log	dotnet/runtime#123379
1286060	dotnet/runtime	Checkout dotnet/runtime@release/10.0 to s	Log
1286350	dotnet/aspnetcore	Checkout dotnet/aspnetcore@refs/pull/65374/merge to s	Log	dotnet/aspnetcore#65374
1286019	dotnet/maui	Checkout dotnet/maui@inflight/candidate to s	Log
1286329	dotnet/aspnetcore	Checkout dotnet/aspnetcore@release/10.0 to s	Log
1286437	dotnet/runtime	Checkout dotnet/runtime@main to s	Log
1286298	dotnet/runtime	Checkout dotnet/runtime@refs/pull/123323/merge to s	Log	dotnet/runtime#123323
1286323	dotnet/aspnetcore	Checkout dotnet/aspnetcore@refs/pull/65372/merge to s	Log	dotnet/aspnetcore#65372
1286428	dotnet/runtime	Checkout dotnet/runtime@release/9.0-staging to s	Log
1285925	dotnet/runtime	Checkout dotnet/runtime@refs/pull/124130/merge to s	Log	dotnet/runtime#124130
1286440	dotnet/runtime	Checkout dotnet/runtime@refs/pull/124189/merge to s	Log	dotnet/runtime#124189
1286438	dotnet/runtime	Checkout dotnet/runtime@refs/pull/124189/merge to s	Log	dotnet/runtime#124189
1286281	dotnet/runtime	Checkout dotnet/runtime@refs/pull/124055/merge to s	Log	dotnet/runtime#124055
1286286	dotnet/runtime	Checkout dotnet/runtime@refs/pull/124186/merge to s	Log	dotnet/runtime#124186
1286328	dotnet/runtime	Checkout dotnet/runtime@main to s	Log
1286310	dotnet/performance	Checkout dotnet/performance@refs/pull/4974/merge to s/performance	Log	dotnet/performance#4974
1286139	dotnet/sdk	Checkout dotnet/sdk@refs/pull/52911/merge to s	Log	dotnet/sdk#52911
1286300	dotnet/runtime	Checkout dotnet/runtime@refs/pull/123323/merge to s	Log	dotnet/runtime#123323
1286412	dotnet/runtime	Checkout dotnet/runtime@refs/pull/123379/merge to s	Log	dotnet/runtime#123379
1286417	dotnet/runtime	Checkout dotnet/runtime@refs/pull/124133/merge to s	Log	dotnet/runtime#124133
1286017	dotnet/maui	Checkout dotnet/maui@refs/pull/33914/merge to s	Log	dotnet/maui#33914
1286416	dotnet/runtime	Checkout dotnet/runtime@refs/pull/124133/merge to s	Log	dotnet/runtime#124133
1286415	dotnet/runtime	Checkout dotnet/runtime@refs/pull/124133/merge to s	Log	dotnet/runtime#124133
1286065	dotnet/runtime	Checkout dotnet/runtime@release/10.0 to s	Log
1286420	dotnet/roslyn	Checkout dotnet/roslyn@refs/pull/82315/merge to s	Log	dotnet/roslyn#82315
1286419	dotnet/roslyn	Checkout dotnet/roslyn@refs/pull/82315/merge to s	Log	dotnet/roslyn#82315
1286424	dotnet/runtime	Checkout dotnet/runtime@refs/pull/124125/merge to s	Log	dotnet/runtime#124125
1286425	dotnet/runtime	Checkout dotnet/runtime@refs/pull/124125/merge to s	Log	dotnet/runtime#124125
1286423	dotnet/runtime	Checkout dotnet/runtime@refs/pull/124125/merge to s	Log	dotnet/runtime#124125
1285833	dotnet/maui	Checkout dotnet/maui@net11.0 to s	Log
1286135	dotnet/runtime	Checkout dotnet/runtime@refs/pull/122761/merge to s	Log	dotnet/runtime#122761
1286196	dotnet/runtime	Checkout dotnet/runtime@refs/pull/124183/merge to s	Log	dotnet/runtime#124183
1286142	dotnet/performance	Checkout dotnet/performance@refs/pull/5093/merge to s/performance	Log	dotnet/performance#5093
1286194	dotnet/runtime	Checkout dotnet/runtime@refs/pull/124183/merge to s	Log	dotnet/runtime#124183
1286217	dotnet/runtime	Checkout dotnet/runtime@refs/pull/124176/merge to s	Log	dotnet/runtime#124176
1286083	dotnet/maui	Checkout dotnet/maui@refs/pull/33843/merge to s	Log	dotnet/maui#33843
1286200	dotnet/aspnetcore	Checkout dotnet/aspnetcore@refs/pull/65184/merge to s	Log	dotnet/aspnetcore#65184
1285775	dotnet/runtime	Checkout dotnet/runtime@refs/pull/124174/merge to s	Log	dotnet/runtime#124174
1286201	dotnet/aspnetcore	Checkout dotnet/aspnetcore@refs/pull/65184/merge to s	Log	dotnet/aspnetcore#65184
1286210	dotnet/runtime	Checkout dotnet/runtime@refs/pull/124184/merge to s	Log	dotnet/runtime#124184
1286212	dotnet/runtime	Checkout dotnet/runtime@refs/pull/124184/merge to s	Log	dotnet/runtime#124184
1285939	dotnet/runtime	Checkout dotnet/runtime@refs/pull/123907/merge to s	Log	dotnet/runtime#123907
1278685	dotnet/runtime	Checkout dotnet/runtime@refs/pull/123966/merge to s	Log	dotnet/runtime#123966
1286211	dotnet/runtime	Checkout dotnet/runtime@refs/pull/124184/merge to s	Log	dotnet/runtime#124184
1286121	dotnet/runtime	Checkout dotnet/runtime@refs/pull/124138/merge to s	Log	dotnet/runtime#124138
1286209	dotnet/aspire	Checkout dotnet/aspire@refs/pull/14401/merge to s	Log	dotnet/aspire#14401
1286147	dotnet/maui	Checkout dotnet/maui@inflight/candidate to s	Log
1285897	dotnet/runtime	Checkout dotnet/runtime@refs/pull/124132/merge to s	Log	dotnet/runtime#124132
1285913	dotnet/runtime	Checkout dotnet/runtime@refs/pull/124172/merge to s	Log	dotnet/runtime#124172
1286054	dotnet/runtime	Checkout dotnet/runtime@refs/pull/124179/merge to s	Log	dotnet/runtime#124179
1286014	dotnet/runtime	Checkout dotnet/runtime@refs/pull/124048/merge to s	Log	dotnet/runtime#124048
1286208	dotnet/runtime	Checkout dotnet/runtime@refs/pull/123702/merge to s	Log	dotnet/runtime#123702
1286207	dotnet/runtime	Checkout dotnet/runtime@refs/pull/123702/merge to s	Log	dotnet/runtime#123702
1286205	dotnet/runtime	Checkout dotnet/runtime@refs/pull/123702/merge to s	Log	dotnet/runtime#123702
1286092	dotnet/runtime	Checkout dotnet/runtime@refs/pull/124177/merge to s	Log	dotnet/runtime#124177
1286177	dotnet/runtime	Checkout dotnet/runtime@refs/pull/124180/merge to s	Log	dotnet/runtime#124180
1286079	dotnet/runtime	Checkout dotnet/runtime@release/8.0 to s	Log
1286094	dotnet/runtime	Checkout dotnet/runtime@refs/pull/124177/merge to s	Log	dotnet/runtime#124177
1286070	dotnet/runtime	Checkout dotnet/runtime@release/9.0 to s	Log
1286175	dotnet/runtime	Checkout dotnet/runtime@refs/pull/124180/merge to s	Log	dotnet/runtime#124180
1286097	dotnet/sdk	Checkout dotnet/sdk@refs/pull/52909/merge to s	Log	dotnet/sdk#52909
1285802	dotnet/runtime	Checkout dotnet/runtime@refs/pull/123251/merge to s	Log	dotnet/runtime#123251
1285855	dotnet/runtime	Checkout dotnet/runtime@refs/pull/122093/merge to s	Log	dotnet/runtime#122093
1286080	dotnet/runtime	Checkout dotnet/runtime@release/8.0 to s	Log
1286061	dotnet/runtime	Checkout dotnet/runtime@release/10.0 to s	Log
1286195	dotnet/runtime	Checkout dotnet/runtime@refs/pull/124183/merge to s	Log	dotnet/runtime#124183
1286149	dotnet/maui	Checkout dotnet/maui@inflight/candidate to s	Log
1285978	dotnet/aspnetcore	Checkout dotnet/aspnetcore@refs/pull/65306/merge to s	Log	dotnet/aspnetcore#65306
1286046	dotnet/runtime	Checkout dotnet/runtime@refs/pull/124178/merge to s	Log	dotnet/runtime#124178
1286078	dotnet/runtime	Checkout dotnet/runtime@release/8.0 to s	Log
1286076	dotnet/runtime	Checkout dotnet/runtime@release/8.0 to s	Log
1286053	dotnet/runtime	Checkout dotnet/runtime@refs/pull/124179/merge to s	Log	dotnet/runtime#124179
Displaying 100 of 185 results

Summary

24-Hour Hit Count	7-Day Hit Count	1-Month Count
3	116	185

Known issue validation

Build: 🔎 https://dev.azure.com/dnceng-public/public/_build/results?buildId=323960
Error message validated: Git fetch failed with exit code: 128
Result validation: ✅ Known issue matched with the provided build.
Validation performed at: 6/28/2023 9:15:14 PM UTC

[ChadNedzlek]

So this is... interesting. I agree this seems like essentially a random network failure that is not the fault of the code. Also... it doesn't seem like there is anything to be done other than retry the build.

@ulisesh and @AlitzelMendez. This seems like another instance of a "known issue" that would really benefit from the ability to get retried automatically. I think Carlos did the right thing by opening the known issue, but his PR (and any future PR that hits this) is still going to be blocked, since the build doesn't run at all in this case, because the checkout failed. And it would be great to help folks that hit this.

[ulisesh]

I created the issue to track how often it was happening and measure the impact of the problem. It might be hard to get traction on networking problems but having the number of impacted builds has great value in my opinion.

In the last 24 hours, we had 37 builds failed with the same error. cc @markwilkie who might find this data interesting.

I think automatic retries for known issues will be the next feature after we are done with analyzing builds retroactively, we can discuss this if people think we should change priorities.

[markwilkie]

So 37 builds were caught with the known issues feature @ulisesh ? Could I get the query please? (super exciting)

[ulisesh]

I saw 37 hits in the office but now I only see 33 🤔. There are 33 failed jobs in 2 different builds. Sorry if I misused the terms but I still think it is great that we can track these build breaks 😄

This is the query:

KnownIssues
| where IssueId == 9009 and IssueRepository == "dotnet/arcade"

[ViktorHofer]

This happened 36 times in the last 24 hours. Should we add a retry (assuming that's possible)?