8380011: Path-to-gcroots search should not trigger stack overflows

[tstuefe]

This backports a fix to JFR path-to-roots-heap-search using DFS that would lead to crashes if object graphs are too deep and we run out of stack. The fix is very simple and low-risk.

I had to manually resolve (even though Skara applied the clean label? What gives?) due to

https://bugs.openjdk.org/browse/JDK-8283710 (JVMTI: Use BitSet for object marking)

but the change was trivial, and my code not affected by missing 8283710.

---

Progress

• Change must not contain extraneous whitespace
• JDK-8380011 needs maintainer approval
• Commit message must refer to an issue

Issue

• JDK-8380011: Path-to-gcroots search should not trigger stack overflows (Bug - P3 - Requested)

Reviewers

• Andrew John Hughes (@gnu-andrew - Reviewer)

Reviewing

Using git

Checkout this PR locally:
$ git fetch https://git.openjdk.org/jdk17u-dev.git pull/4306/head:pull/4306
$ git checkout pull/4306

Update a local copy of the PR:
$ git checkout pull/4306
$ git pull https://git.openjdk.org/jdk17u-dev.git pull/4306/head

Using Skara CLI tools

Checkout this PR locally:
$ git pr checkout 4306

View PR using the GUI difftool:
$ git pr show -t 4306

Using diff file

Download this PR as a diff file:
https://git.openjdk.org/jdk17u-dev/pull/4306.diff

Using Webrev

Link to Webrev Comment

[bridgekeeper]

👋 Welcome back stuefe! A progress list of the required criteria for merging this PR into master will be added to the body of your pull request. There are additional pull request commands available for use with this pull request.

[openjdk]

❗ This change is not yet ready to be integrated.
See the Progress checklist in the description for automated requirements.

[openjdk]

This backport pull request has now been updated with issue from the original commit.

[openjdk]

⚠️ @tstuefe This change is now ready for you to apply for maintainer approval. This can be done directly in each associated issue or by using the /approval command.

[tstuefe]

/approval request This fixes a crash in JFR when DFS object graph search gets too deep. Patch is small and low-risk. Tested: GHAs, and ran the contained regression test on Linux x64 manually.

[openjdk]

@tstuefe
8380011: The approval request has been created successfully.

[mlbridge]

Webrevs

• 00: Full (e408323a)

[gnu-andrew]

Patch is near-clean, just thrown off by the slightly different types in the DFSClosure constructor signature. Testing looks good and risk seems low, just adding a space check to avoid overflow.

[gnu-andrew]

I think the clean label does more than simply trying to apply the patch and includes some fuzzing; see https://github.com/openjdk/skara/blob/master/vcs/src/main/java/org/openjdk/skara/vcs/DiffComparator.java It's another reason I'm not a fan of allowing it to be used to skip review.

[tstuefe]

Thank you @gnu-andrew !

/integrate

[openjdk]

@tstuefe This pull request has not yet been marked as ready for integration.

[tstuefe]

@GoeLin Would you mind approving this one?