Backport bump jruby 9.4.13#19188
Conversation
🤖 GitHub commentsJust comment with:
|
* upgrade jruby to 9.4.11.0 * follow up cleanups * follow up cleanups * follow up cleanups * follow up cleanups * add licenses * Update rakelib/plugins-metadata.json * attempt to fix rubygems * attempt to fix rubygems * attempt to fix rubygems * tweaks for jruby 9.4.12.0 and sha256 validation * attempt at fixing rubygems * attempt at fixing rubygems * attempt at fixing rubygems * attempt at fixing rubygems * Bump jruby to 9.4.12.1 * Explicitly require 'set' to make `to_set` available Something appears to have changed in code loading where this require is no longer in the call chain. Explicitly require it here. * Fix up dependency mapper * Update bundler code with latest rubygems release This commit makes 2 changes: 1. We were sending a `Gem::Platform` instance to Thor parser instead of a string. Convert to string version of a platform name. With newly added checks in Thor, we were getting `noMethodError` for `Gem::Platform`. 2. It looks like there was a patch to get around a bug in bundler that was reaching out to the network for local gems. This appears to have been fixed upstream, this removes the patch. * Restore patch * Remove patch (TODO, link to full writeup) * SPIKE: Take up jruby 9.4.13.0 * Patch bundler cache Still validating this. Essentially ensure we check local gems when calling cache using the updated rubygems class. * WIP: test patch for acceptance tests * Skip when unable to cache * Cache every gem * Fix acceptance test * Force another route through builtin * Fixup idea (it returns nil, not raise) * Fix missing version update * Fix qatest filter plugin Without including a gemspec with a locally built gem logstash-plugin list will not show it in the output. * Use vendored gem instead of reaching out to rubygems We vendor an udpated test gem for local install. Use that instead of reaching out to rubygems. * Document and comment on the patched bundler code * Fix test instead of downloading the "broken" one from rubygems (downloading the `.gem` file directly does show up in plugin-list), using a vendored one polutes the cache. Use a new vendored gem --------- Co-authored-by: João Duarte <jsvduarte@gmail.com> Co-authored-by: João Duarte <jsvd@users.noreply.github.com>
Without this change the Bundler shipped with JRuby 9.4.13 (2.6.3) checks the existing lockfile version (2.3.26), given the mismatching version, Bundler tries to restarts itself but with half backed configuration, not setting a HOME properly and makes failing the `:bootstrap`. ./gradlew bootstrap runs Bundler inside Gradle’s embedded JRuby (jruby-core), not via vendor/jruby/bin/jruby. Because that embedded runtime has no JRuby home set, Gem.ruby points to a non-existent $TMPDIR/bin/jruby, and when Bundler tries to auto-switch to the lockfile’s Bundler version (2.3.26 vs 2.6.3), it fails with Errno::ENOENT trying to exec that path.
* Pin jar-dependencies to match jruby 9.4.13.0 When default and bundler dont agree we can see errors about activated versions. * Revert "fix jars installer for new maven and pin psych to 5.2.2 (elastic#16919)" This reverts commit 52b7fb0.
…regenerated NOTICES.TXT
andsel
force-pushed
the
backport_bump_jruby_9.4.13
branch
from
79fdc60 to
116e076
Compare
|
run exhaustive tests |
…' which exists starting from v9
|
run exhaustive tests |
Co-authored-by: Rye Biesemeyer <yaauie@users.noreply.github.com>
TL;DRThe failed Ruby, X-Pack unit, and X-Pack integration jobs all have the same root cause: Remediation
Investigation detailsRoot CauseThis is a dependency/configuration regression introduced by the PR. The PR diff for @@ -62,14 +62,9 @@ Gem::Specification.new do |gem|
gem.add_runtime_dependency "treetop", "~> 1" #(MIT license)
- gem.add_runtime_dependency "i18n", "~> 1" #(MIT license)
-
gem.add_runtime_dependency "thwait"But Logstash still requires Evidence
The X-Pack integration log has the same stack at No existing VerificationNot run locally; the local checkout is not this PR branch. The diagnosis is based on the Buildkite logs plus the PR diff showing the removed runtime dependency. What is this? | From workflow: PR Buildkite Detective Give us feedback! React with 🚀 if perfect, 👍 if helpful, 👎 if not. |
andsel
force-pushed
the
backport_bump_jruby_9.4.13
branch
from
d6b5c16 to
5ea4d35
Compare
|
run exhaustive tests |
donoghuc
left a comment
donoghuc
left a comment
There was a problem hiding this comment.
Overall:
For the most part this looks correct, though I do have a question below that is gating an approval. I focused mainly on manual testing the pluginmanager (as this was the hardest part to get working when i worked on the original 9.x upgrade). Everything seems to be in order there! I did find #19235 but that is not related to this work.
Change request
Can you double check or walk me through the updates to the Gemfile.jruby-3.1.lock.release? It seems like there are some test/dev deps that have slipped in. How was this generated/modified?
When i tried locally I came up with this diff (against what you have):
diff --git a/Gemfile.jruby-3.1.lock.release b/Gemfile.jruby-3.1.lock.release
index 3a46e2453..ced91b3ca 100644
--- a/Gemfile.jruby-3.1.lock.release
+++ b/Gemfile.jruby-3.1.lock.release
@@ -104,7 +104,6 @@ GEM
dalli (3.2.8)
date (3.3.3-java)
diff-lcs (1.6.2)
- digest-crc (0.5.1)
docile (1.4.1)
domain_name (0.6.20240107)
dotenv (2.8.1)
@@ -146,20 +145,11 @@ GEM
pleaserun (~> 0.0.29)
rexml
stud
- ftw (0.0.49)
- addressable (>= 2.4)
- backports (>= 2.6.2)
- cabin (> 0)
- http_parser.rb (~> 0.6)
- gelf (3.1.0)
- json
gelfd2 (0.4.1)
gem_publisher (1.5.0)
gems (1.3.0)
gene_pool (1.5.0)
concurrent-ruby (>= 1.0)
- gmetric (0.1.3)
- gserver (0.0.1)
hashdiff (1.2.1)
hitimes (1.3.1-java)
http (3.3.0)
@@ -773,13 +763,8 @@ GEM
rack (>= 3.0.0)
rack-test (2.2.0)
rack (>= 1.3)
- rackup (2.0.0)
- rack (>= 3)
- webrick
rainbow (3.1.1)
rake (13.4.2)
- random-port (0.8.0)
- tago (~> 0.0)
redis (4.8.1)
regexp_parser (2.12.0)
reline (0.6.3)
@@ -789,8 +774,6 @@ GEM
rspec-core (~> 3.13.0)
rspec-expectations (~> 3.13.0)
rspec-mocks (~> 3.13.0)
- rspec-collection_matchers (1.2.1)
- rspec-expectations (>= 2.99.0.beta1)
rspec-core (3.13.6)
rspec-support (~> 3.13.0)
rspec-expectations (3.13.5)
@@ -799,9 +782,6 @@ GEM
rspec-mocks (3.13.8)
diff-lcs (>= 1.2.0, < 2.0)
rspec-support (~> 3.13.0)
- rspec-sequencing (0.1.1)
- concurrent-ruby
- rspec (>= 3.0.0)
rspec-support (3.13.7)
rspec-wait (1.0.2)
rspec (>= 3.4)
@@ -818,14 +798,10 @@ GEM
unicode-display_width (>= 2.4.0, < 4.0)
rubocop-ast (1.42.0)
parser (>= 3.3.7.2)
- ruby-kafka (1.5.0)
- digest-crc
ruby-progressbar (1.13.0)
rubyzip (1.3.0)
rufus-scheduler (3.0.9)
tzinfo
- rumbster (1.1.0)
- mail
sawyer (0.9.3)
addressable (>= 2.3.5)
faraday (>= 0.17.3, < 3)
@@ -850,18 +826,13 @@ GEM
rack-protection (= 4.2.1)
rack-session (>= 2.0.0, < 3)
tilt (~> 2.0)
- snappy (0.5.1-java)
- snappy-jars (~> 1.1.0)
- snappy-jars (1.1.0.1.2-java)
spoon (0.0.6)
ffi
stud (0.0.23)
- tago (0.7.0)
thread_safe (0.3.6-java)
thwait (0.2.0)
e2mmap
tilt (2.7.0)
- timecop (0.9.11)
timeout (0.6.1)
treetop (1.6.18)
polyglot (~> 0.3)
@@ -890,7 +861,6 @@ GEM
addressable (>= 2.8.0)
crack (>= 0.3.2)
hashdiff (>= 0.4.0, < 2.0.0)
- webrick (1.9.2)
xml-simple (1.1.9)
rexml
@@ -901,26 +871,18 @@ DEPENDENCIES
belzebuth
benchmark-ips
bigdecimal (~> 3.1)
- cabin (~> 0.6)
cgi (~> 0.3.7)
- childprocess
+ childprocess (~> 4)
ci_reporter_rspec (~> 1)
date (= 3.3.3)
- digest-crc (~> 0.5.1)
- elasticsearch
fileutils (~> 1.7)
flores (~> 0.0.8)
fpm (~> 1, >= 1.14.1)
- ftw (~> 0.0.42)
- gelf (= 3.1.0)
gems (~> 1)
- gmetric
- gserver
- insist
jar-dependencies (= 0.5.4)
json-schema (~> 2)
logstash-codec-avro (~> 3.4)
- logstash-codec-cef
+ logstash-codec-cef (~> 6.2)
logstash-codec-collectd (~> 3.1)
logstash-codec-dots (~> 3.0)
logstash-codec-edn (~> 3.1)
@@ -1018,41 +980,26 @@ DEPENDENCIES
logstash-output-tcp (~> 6.2)
logstash-output-udp (~> 3.2)
logstash-output-webhdfs (~> 3.1)
- logstash-patterns-core
- manticore
minitar (~> 1)
murmurhash3 (= 0.1.6)
octokit (~> 4.25)
paquet (~> 0.2)
pleaserun (~> 0.0.28)
polyglot
- pry
rack-test
- rackup (< 2.1.0)
rake (~> 13)
- random-port
- rspec
- rspec-collection_matchers
- rspec-sequencing
- rspec-wait
+ rspec (~> 3.5)
rubocop
rubocop-ast (= 1.42.0)
- ruby-kafka
ruby-progressbar (~> 1)
rubyzip (~> 1)
- rufus-scheduler (~> 3.0.9)
- rumbster
simplecov (~> 0.22.0)
simplecov-json
- sinatra
- snappy
- stud
+ stud (~> 0.0.22)
thwait
- timecop
treetop
uri (~> 0.12.3)
- webmock
- webrick
+ webmock (~> 3)
BUNDLED WITH
2.6.3
|
It was regenerated with commands: rm Gemfile.lock
./gradlew clean installDefaultGems
cp Gemfile.lock Gemfile.jruby-3.1.lock.releaseIn commit 387eb08. Then I spotted that many plugins were resolved to newer minor version, so limit the surface of changes I rolled back the changes on plugin versions in 493fdbb. I think these are leftover of that upgrade and rollback some versions. The idea is, in this step, to reduce the surface of changes to plugins, that will be a followup update on the a next version bump. For example in this PR I don't want to switch elasticsearch client from |
|
run exhaustive tests |
|
I would like to experiment with updating the lockfile such that we only take up minor bumps for gems when required. I see some bumps that i'm not convinced are strictly necessary. I would like to investigate and come up with a robust/repeatable way to generate the lock file to minimize the change set. I have assigned myself this PR and will work on this asap. |
Bring the JRuby 9.4.13 backport up to date with the latest 8.19 (puma 8.x, 8.19.18, i18n pin revert, ERB/jwt lock updates).
…nse artifacts psych pin 5.2.2 -> 5.2.3 to match the gem JRuby 9.4.13 ships by default (the only lock<default mismatch; otherwise Bundler.setup raises an activation conflict). Regenerate NOTICE.TXT and restore the JRuby stdlib entries in licenseMapping.csv that the 8.19 merge had dropped. No third-party gem version changes beyond what the JRuby bump requires.
|
run exhaustive tests |
|
UDPATES:
NOTES: You may be wondering why there are so many license additions: This is because the default set of gems shipped with jruby changed. This requires additions/updates to the license mapping workflows. |
💚 Build Succeeded
History
|
donoghuc
left a comment
donoghuc
left a comment
There was a problem hiding this comment.
With minimal gem set and passing exhaustive/PR tests I think this is G2G. Will wait for a proper release cycle point for merge.
3 participants
Release notes
Update jruby to 9.4.13.0 #19188
What does this PR do?
Backports #17787 and #17696 to bundle JRuby
9.4.13to Logstahs8.19branch, fixing conflicts.Why is it important/What is the impact to the user?
Updates bundled JRuby to a more recent version.
Checklist
[ ] I have made corresponding changes to the documentation[ ] I have made corresponding change to the default configuration files (and/or docker env variables)[ ] I have added tests that prove my fix is effective or that my feature worksAuthor's Checklist
How to test this PR locally
Related issues
How to review
56 files of the 77 are NOTICE files or the license mapping CSV file.
Of the remaining changed files are mostly updates from the original PR, except:
run_command_in_pathinstead ofrun_sudo_command_in_pathbecause the method is not present.I've left some comments highlighting these places.