build: build failure on darwin builders since CL 536399 [consistent failure] #70402
Assignees
Labels
Milestone
Comments
gopherbot
added
the
NeedsInvestigation
|
Found new dashboard test flakes for: 2024-11-17 14:32 gotip-darwin-amd64-nocgo go@f9a95b1b [build] (log)2024-11-17 14:32 gotip-darwin-amd64-nocgo go@44d4b699 [build] (log)2024-11-17 20:17 gotip-darwin-amd64-nocgo go@63f762bc [build] (log)2024-11-17 20:21 gotip-darwin-amd64-nocgo go@a867e5e5 [build] (log) |
|
Found new dashboard test flakes for: 2024-11-17 21:09 gotip-darwin-amd64-nocgo go@01e1e5c2 [build] (log) |
|
Found new dashboard test flakes for: 2024-11-17 20:16 gotip-darwin-amd64-nocgo go@04807d3a [build] (log) |
cherrymui
changed the title
|
cc @aclements |
This was referenced Nov 18, 2024
|
Found new dashboard test flakes for: 2024-11-17 14:32 gotip-darwin-amd64-longtest go@f9a95b1b [build] (log)2024-11-17 14:32 gotip-darwin-amd64_13 go@f9a95b1b [build] (log)2024-11-17 14:32 gotip-darwin-amd64_14 go@f9a95b1b [build] (log)2024-11-17 14:32 gotip-darwin-arm64_13 go@f9a95b1b [build] (log)2024-11-17 14:32 gotip-darwin-amd64-longtest go@44d4b699 [build] (log)2024-11-17 14:32 gotip-darwin-amd64_13 go@44d4b699 [build] (log)2024-11-17 14:32 gotip-darwin-amd64_14 go@44d4b699 [build] (log)2024-11-17 14:32 gotip-darwin-arm64_13 go@44d4b699 [build] (log)2024-11-17 20:16 gotip-darwin-amd64_13 go@04807d3a [build] (log)2024-11-17 20:16 gotip-darwin-amd64_14 go@04807d3a [build] (log)2024-11-17 20:16 gotip-darwin-arm64_13 go@04807d3a [build] (log)2024-11-17 20:17 gotip-darwin-amd64-longtest go@63f762bc [build] (log)2024-11-17 20:17 gotip-darwin-amd64_13 go@63f762bc [build] (log)2024-11-17 20:17 gotip-darwin-amd64_14 go@63f762bc [build] (log)2024-11-17 20:17 gotip-darwin-arm64_13 go@63f762bc [build] (log)2024-11-17 20:21 gotip-darwin-amd64-longtest go@a867e5e5 [build] (log)2024-11-17 20:21 gotip-darwin-amd64_13 go@a867e5e5 [build] (log)2024-11-17 20:21 gotip-darwin-amd64_14 go@a867e5e5 [build] (log)2024-11-17 20:21 gotip-darwin-arm64_13 go@a867e5e5 [build] (log)2024-11-17 21:09 gotip-darwin-amd64-longtest go@01e1e5c2 [build] (log)2024-11-17 21:09 gotip-darwin-amd64_13 go@01e1e5c2 [build] (log)2024-11-17 21:09 gotip-darwin-amd64_14 go@01e1e5c2 [build] (log)2024-11-17 21:09 gotip-darwin-arm64_13 go@01e1e5c2 [build] (log)2024-11-18 02:08 gotip-darwin-amd64_13 go@d1180dbd [build] (log)2024-11-18 02:08 gotip-darwin-arm64_13 go@d1180dbd [build] (log) |
|
Looking closely at https://ci.chromium.org/ui/p/golang/builders/ci/gotip-darwin-amd64_13/b8730968502183490337/overview I think what's going on is that we have this one new line in the middle of the And this new "build-output" action is confusing the LUCI test reader: Prior to this change (example), the dist JSON output just had text embedded in it: |
|
Found new dashboard test flakes for: 2024-11-18 02:08 gotip-darwin-amd64-nocgo go@d1180dbd [build] (log)2024-11-18 02:08 gotip-darwin-amd64_14 go@d1180dbd [build] (log)2024-11-18 02:09 gotip-darwin-amd64-nocgo go@90b1dc01 [build] (log)2024-11-18 02:09 gotip-darwin-amd64_13 go@90b1dc01 [build] (log)2024-11-18 02:09 gotip-darwin-arm64_13 go@90b1dc01 [build] (log) |
|
Change https://go.dev/cl/628955 mentions this issue: |
|
Marking as a release blocker because we either need to re-enable build JSON from |
|
Found new dashboard test flakes for: 2024-11-18 02:09 gotip-darwin-amd64-longtest go@90b1dc01 [build] (log)2024-11-18 02:09 gotip-darwin-amd64_14 go@90b1dc01 [build] (log) |
gopherbot
pushed a commit
that referenced
this issue
Nov 18, 2024
Unfortunately, this is tripping up the LUCI test output processor, so we need to disable it until we can figure that out. For #70402. Updates #62067. Cq-Include-Trybots: luci.golang.try:gotip-darwin-arm64_13,gotip-linux-amd64-longtest Change-Id: I9ae722218e98b8060b8b4c46358f23381ac8537a Reviewed-on: https://go-review.googlesource.com/c/go/+/628955 LUCI-TryBot-Result: Go LUCI <[email protected]> Reviewed-by: Cherry Mui <[email protected]>
|
Found new dashboard test flakes for: 2024-11-18 02:08 gotip-darwin-amd64-longtest go@d1180dbd [build] (log) |
|
Updating this with some details that I'm seeing so far. The For TestEvent objects, the test ID was constructed from the Package and Test fields. The new BuildEvent objects populate the ImportPath field instead of a Package field, and so they end up having an empty ID associated, which get rejected by the server. The fix for this particular problem should be, when the Package field is empty, to consider the new ImportPath field and use it to generate the ID. As @mknyszek pointed out, result_adapter already intends to ignore unknown actions, so those might not need any special handling to get the baseline behavior back. I'll look more into that after handling the test IDs. |
|
Change https://go.dev/cl/629335 mentions this issue: |
I'm not sure if that's the right fix. The If a build event never matches up with a test, then it might make sense to construct a result for it independently using the |
|
Change https://go.dev/cl/629375 mentions this issue: |
gopherbot
pushed a commit
to golang/build
that referenced
this issue
Nov 18, 2024
Two reasons: to buy more time to work on the result_adapter/ResultDB side (beyond just the bare minimum behavior of not regressing beyond what we had before), and to test that using a GODEBUG for this works as we expect. It already uncovered the need to merge multiple GODEBUG settings that we hadn't run into earlier. For golang/go#70402. Change-Id: I9b86f6a5462637eef84f3efca9fc31d11657d756 Reviewed-on: https://go-review.googlesource.com/c/build/+/629375 Auto-Submit: Dmitri Shuralyov <[email protected]> LUCI-TryBot-Result: Go LUCI <[email protected]> Reviewed-by: Dmitri Shuralyov <[email protected]> Reviewed-by: Michael Knyszek <[email protected]>
|
Updated plan, which takes into account https://pkg.go.dev/cmd/go@master#hdr-Build__json_encoding and #70402 (comment), is to determine that a given JSON object is a BuildEvent rather than a TestEvent by checking its Action field and seeing a "build-" prefix, and then attach the build output to the test record corresponding to the package. Out of the options currently available at https://pkg.go.dev/go.chromium.org/luci/resultdb/proto/v1#TestStatus, TestStatus_FAIL will do. So the main observable difference will be that the package build error output will be attached to the package's test record, instead of only being visible in the non-structured output. I have a working prototype of that, need to clean it up a bit before mailing to handle edge cases. Edit: Filed #70435 to track the LUCI side of this work. |
dmitshur
added
NeedsFix
|
Change https://go.dev/cl/634237 mentions this issue: |
gopherbot
pushed a commit
to golang/build
that referenced
this issue
Dec 7, 2024
We'll drop it from tip and the new release-branch.go1.24 branch as soon as result_adapter is ready to handle events that contain ImportPath but not Package fields, but that's not the case this week. For golang/go#70402. For golang/go#70435. Change-Id: Ibd5ce399e2b8e072eb9066a1d304964b41f6cc1b Reviewed-on: https://go-review.googlesource.com/c/build/+/634237 Reviewed-by: Michael Knyszek <[email protected]> Reviewed-by: Dmitri Shuralyov <[email protected]> LUCI-TryBot-Result: Go LUCI <[email protected]>
|
Found new dashboard test flakes for: 2024-12-06 21:58 go1.24-darwin-amd64_13 release-branch.go1.24@26682773 [build] (log) |
|
Found new dashboard test flakes for: 2024-12-06 21:58 go1.24-darwin-arm64_13 release-branch.go1.24@26682773 [build] (log) |
|
Failures above were on release-branch.go1.24 prior to CL 634237. |
|
Found new dashboard test flakes for: 2024-12-06 21:58 go1.24-darwin-amd64-nocgo release-branch.go1.24@26682773 [build] (log) |
|
Found new dashboard test flakes for: 2024-12-06 21:58 go1.24-darwin-amd64_14 release-branch.go1.24@26682773 [build] (log) |
|
Change https://go.dev/cl/635899 mentions this issue: |
gopherbot
pushed a commit
that referenced
this issue
Dec 13, 2024
go test -json has two new effects in Go 1.24: it implies go build -json, and it adds a FailedBuild field in test events. For compatibility, CL 629335 added gotestjsonbuildtext=1, which disables the implicit go build -json, but that CL didn't affect the FailedBuild field. In principle this shouldn't matter because it's just another JSON field, but just so we don't have to worry about some intermediate behavior, this CL makes gotestjsonbuildtext=1 disable the FailedBuild field as well. Updates #62067 Updates #70402 Change-Id: I006d1ea0468980ee2564495324e8b4ed082898af Reviewed-on: https://go-review.googlesource.com/c/go/+/635899 Auto-Submit: Austin Clements <[email protected]> Reviewed-by: Dmitri Shuralyov <[email protected]> LUCI-TryBot-Result: Go LUCI <[email protected]> Reviewed-by: Dmitri Shuralyov <[email protected]>
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Issue created automatically to collect these failures.
Example (log):
— watchflakes
The text was updated successfully, but these errors were encountered: