Mailing List Archive

Apr 6, 2022, 1:39 PM

Post #2 of 13 (1013 views)

OK, this also happens with Oracle's JDK17. Now I'm confused

On Wed, Apr 6, 2022 at 4:28 PM Michael Sokolov <msokolov@gmail.com> wrote:
>
> Hi, locally I failed to run spotlessCheck/spotlessApply on main (10.x)
> branch. I assume it's because of a JVM difference; here's the error:
>
>
> Step 'google-java-format' found problem in
> 'lucene/core/src/java/module-info.java':
> null
> java.lang.reflect.InvocationTargetException
> at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native
> Method)
>
> [giant gradle stack trace omitted]
>
> Caused by: java.lang.IllegalAccessError: class
> com.google.googlejavaformat.java.JavaInput (in unnamed module
> @0x1430dc06) cannot access class
> com.sun.tools.javac.parser.Tokens$TokenKind (in module jdk.compiler)
> because module jdk.compiler does not export com.sun.tools.javac.parser
> to unnamed module @0x1430dc06
> at com.google.googlejavaformat.java.JavaInput.buildToks(JavaInput.java:349)
> at com.google.googlejavaformat.java.JavaInput.buildToks(JavaInput.java:334)
> at com.google.googlejavaformat.java.JavaInput.<init>(JavaInput.java:276)
> at com.google.googlejavaformat.java.Formatter.getFormatReplacements(Formatter.java:280)
> at com.google.googlejavaformat.java.Formatter.formatSource(Formatter.java:267)
> at com.google.googlejavaformat.java.Formatter.formatSource(Formatter.java:233)
>
> It seems like a no-no for Google to be relying on Sun packages, but IDK
>
> I tried changing
>
> > id 'com.diffplug.spotless' version "6.4.2" apply false
>
> in build.gradle
>
> and
>
> > googleJavaFormat('1.15.0')
>
> in spotless.gradle
>
> (the newest versions), because why not, but it didn't seem to help.
>
> Any idea how to fix? If it's a problem with my JDK, is there a thing I
> can ask to have changed in that? Re-export the com.sun.tools.javac
> package I guess?
>
> -Mike

---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: dev-help@lucene.apache.org

Re: spotless targets [ In reply to ]

mdrob at mdrob

Apr 6, 2022, 1:55 PM

Post #3 of 13 (1013 views)

Save a copy of your gradle.properties and then rerun. There are add opens
options in the jvm opts we generate that were not there before

On Wed, Apr 6, 2022 at 3:46 PM Michael Sokolov <msokolov@gmail.com> wrote:

> OK, this also happens with Oracle's JDK17. Now I'm confused
>
> On Wed, Apr 6, 2022 at 4:28 PM Michael Sokolov <msokolov@gmail.com> wrote:
> >
> > Hi, locally I failed to run spotlessCheck/spotlessApply on main (10.x)
> > branch. I assume it's because of a JVM difference; here's the error:
> >
> >
> > Step 'google-java-format' found problem in
> > 'lucene/core/src/java/module-info.java':
> > null
> > java.lang.reflect.InvocationTargetException
> > at
> java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native
> > Method)
> >
> > [giant gradle stack trace omitted]
> >
> > Caused by: java.lang.IllegalAccessError: class
> > com.google.googlejavaformat.java.JavaInput (in unnamed module
> > @0x1430dc06) cannot access class
> > com.sun.tools.javac.parser.Tokens$TokenKind (in module jdk.compiler)
> > because module jdk.compiler does not export com.sun.tools.javac.parser
> > to unnamed module @0x1430dc06
> > at
> com.google.googlejavaformat.java.JavaInput.buildToks(JavaInput.java:349)
> > at
> com.google.googlejavaformat.java.JavaInput.buildToks(JavaInput.java:334)
> > at
> com.google.googlejavaformat.java.JavaInput.<init>(JavaInput.java:276)
> > at
> com.google.googlejavaformat.java.Formatter.getFormatReplacements(Formatter.java:280)
> > at
> com.google.googlejavaformat.java.Formatter.formatSource(Formatter.java:267)
> > at
> com.google.googlejavaformat.java.Formatter.formatSource(Formatter.java:233)
> >
> > It seems like a no-no for Google to be relying on Sun packages, but IDK
> >
> > I tried changing
> >
> > > id 'com.diffplug.spotless' version "6.4.2" apply false
> >
> > in build.gradle
> >
> > and
> >
> > > googleJavaFormat('1.15.0')
> >
> > in spotless.gradle
> >
> > (the newest versions), because why not, but it didn't seem to help.
> >
> > Any idea how to fix? If it's a problem with my JDK, is there a thing I
> > can ask to have changed in that? Re-export the com.sun.tools.javac
> > package I guess?
> >
> > -Mike
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
> For additional commands, e-mail: dev-help@lucene.apache.org
>
>

Re: spotless targets [ In reply to ]

Apr 6, 2022, 2:05 PM

Post #4 of 13 (1013 views)

Thanks, Mike. So I removed gradle.properties (I never edited anything
in there), and re-run, and I can see that I now have

# We also open up internal compiler modules for spotless/ google jaa format.
org.gradle.jvmargs=-Xmx3g \
--add-exports jdk.compiler/com.sun.tools.javac.api=ALL-UNNAMED \
--add-exports jdk.compiler/com.sun.tools.javac.file=ALL-UNNAMED \
--add-exports jdk.compiler/com.sun.tools.javac.parser=ALL-UNNAMED \
--add-exports jdk.compiler/com.sun.tools.javac.tree=ALL-UNNAMED \
--add-exports jdk.compiler/com.sun.tools.javac.util=ALL-UNNAMED

which seems as if it ought to fix, but I got the same error -- OH --
but not the second time??! wacky, but (crossing fingers) I hope it
will stay fixed now

On Wed, Apr 6, 2022 at 4:56 PM Mike Drob <mdrob@mdrob.com> wrote:
>
> Save a copy of your gradle.properties and then rerun. There are add opens options in the jvm opts we generate that were not there before
>
> On Wed, Apr 6, 2022 at 3:46 PM Michael Sokolov <msokolov@gmail.com> wrote:
>>
>> OK, this also happens with Oracle's JDK17. Now I'm confused
>>
>> On Wed, Apr 6, 2022 at 4:28 PM Michael Sokolov <msokolov@gmail.com> wrote:
>> >
>> > Hi, locally I failed to run spotlessCheck/spotlessApply on main (10.x)
>> > branch. I assume it's because of a JVM difference; here's the error:
>> >
>> >
>> > Step 'google-java-format' found problem in
>> > 'lucene/core/src/java/module-info.java':
>> > null
>> > java.lang.reflect.InvocationTargetException
>> > at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native
>> > Method)
>> >
>> > [giant gradle stack trace omitted]
>> >
>> > Caused by: java.lang.IllegalAccessError: class
>> > com.google.googlejavaformat.java.JavaInput (in unnamed module
>> > @0x1430dc06) cannot access class
>> > com.sun.tools.javac.parser.Tokens$TokenKind (in module jdk.compiler)
>> > because module jdk.compiler does not export com.sun.tools.javac.parser
>> > to unnamed module @0x1430dc06
>> > at com.google.googlejavaformat.java.JavaInput.buildToks(JavaInput.java:349)
>> > at com.google.googlejavaformat.java.JavaInput.buildToks(JavaInput.java:334)
>> > at com.google.googlejavaformat.java.JavaInput.<init>(JavaInput.java:276)
>> > at com.google.googlejavaformat.java.Formatter.getFormatReplacements(Formatter.java:280)
>> > at com.google.googlejavaformat.java.Formatter.formatSource(Formatter.java:267)
>> > at com.google.googlejavaformat.java.Formatter.formatSource(Formatter.java:233)
>> >
>> > It seems like a no-no for Google to be relying on Sun packages, but IDK
>> >
>> > I tried changing
>> >
>> > > id 'com.diffplug.spotless' version "6.4.2" apply false
>> >
>> > in build.gradle
>> >
>> > and
>> >
>> > > googleJavaFormat('1.15.0')
>> >
>> > in spotless.gradle
>> >
>> > (the newest versions), because why not, but it didn't seem to help.
>> >
>> > Any idea how to fix? If it's a problem with my JDK, is there a thing I
>> > can ask to have changed in that? Re-export the com.sun.tools.javac
>> > package I guess?
>> >
>> > -Mike
>>
>> ---------------------------------------------------------------------
>> To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
>> For additional commands, e-mail: dev-help@lucene.apache.org
>>

---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: dev-help@lucene.apache.org

Re: spotless targets [ In reply to ]

Apr 6, 2022, 11:10 PM

Post #5 of 13 (1013 views)

Hi Mike,

Thanks, Mike. So I removed gradle.properties (I never edited anything
> in there), and re-run, and I can see that I now have
>

This file is generated and contains some defaults optimized for a local
machine. Yes, it may change from time to time and these changes will not be
picked up automatically.

> which seems as if it ought to fix, but I got the same error -- OH --
> but not the second time??! wacky, but (crossing fingers) I hope it

will stay fixed now
>

Here's why. The first time you run gradle after you remove the properties
file, the build scripts see the file is not there and regenerate the
defaults. However, the gradle instance doesn't automatically pick up the
just-generated defaults (it can't, they are JVM-settings) so any subsequent
task may fail as if that file wasn't present at all. A subsequent run will
pick up the new settings from the gradle.properties file and that's why it
worked for you.

Perhaps it should be better explained here - if you can find a wording that
works, please modify it:
https://github.com/apache/lucene/blob/main/help/localSettings.txt

In theory we could interrupt (stop) gradle execution if gradle.properties
hasn't been generated yet but in practice this would cause more confusion
than help. I don't think there is a clean way around it.

Dawid

Re: spotless targets [ In reply to ]

michael at michaelgibney

Apr 7, 2022, 7:47 AM

Post #6 of 13 (1013 views)

Thanks to you all! I had been having the same issue.

I wonder if something as simple as putting a circuit breaker
[props_version] property in `gradle.properties` could be helpful? For
changes that could indicate the need to regenerate gradle.properties, the
"magic" property could be incremented. That would simplify detection of
potential error cases, and allow to print more meaningful
warning/instructions.

Instructions/failure message could be something like: "[something has
changed]. You might want to consider re-generating your `gradle.properties`
file (and re-applying any custom modifications). to ignore this warning,
you can manually update [props_version] property in your current
`gradle.properties` file".

It'd be a little hacky; but it would be effective in proactively putting
this information in front of users who need it.

IMO it'd be ok to fail the first run if gradle.properties hadn't yet been
generated. The static check could fail the first run, but with a more
transparent message; e.g.: "prerequisite `gradle.properties` was
successfully generated, but is not available for this run. Simply re-run
the last command; the newly-generated `gradle.properties` file will be
picked up on subsequent runs". The potential confusion I guess would arise
from the fact that there are tasks that _don't_ require a pre-generated
`gradle.properties` file, and and these would fail unnecessarily?

Michael

On Thu, Apr 7, 2022 at 2:11 AM Dawid Weiss <dawid.weiss@gmail.com> wrote:

>
> Hi Mike,
>
> Thanks, Mike. So I removed gradle.properties (I never edited anything
>> in there), and re-run, and I can see that I now have
>>
>
> This file is generated and contains some defaults optimized for a local
> machine. Yes, it may change from time to time and these changes will not be
> picked up automatically.
>
>
>> which seems as if it ought to fix, but I got the same error -- OH --
>> but not the second time??! wacky, but (crossing fingers) I hope it
>
> will stay fixed now
>>
>
> Here's why. The first time you run gradle after you remove the properties
> file, the build scripts see the file is not there and regenerate the
> defaults. However, the gradle instance doesn't automatically pick up the
> just-generated defaults (it can't, they are JVM-settings) so any subsequent
> task may fail as if that file wasn't present at all. A subsequent run will
> pick up the new settings from the gradle.properties file and that's why it
> worked for you.
>
> Perhaps it should be better explained here - if you can find a wording
> that works, please modify it:
> https://github.com/apache/lucene/blob/main/help/localSettings.txt
>
> In theory we could interrupt (stop) gradle execution if gradle.properties
> hasn't been generated yet but in practice this would cause more confusion
> than help. I don't think there is a clean way around it.
>
> Dawid
>

Re: spotless targets [ In reply to ]

Apr 7, 2022, 10:31 AM

Post #7 of 13 (1013 views)

> I wonder if something as simple as putting a circuit breaker
> [props_version] property in `gradle.properties` could be helpful? For
> changes that could indicate the need to regenerate gradle.properties, the
> "magic" property could be incremented. That would simplify detection of
> potential error cases, and allow to print more meaningful
> warning/instructions.
>

This can be made in a number of ways, I guess. I just didn't have time/
care for it much. Typically everything would run just fine - the JVM
options that open up modules for spotless is the first thing that caused
serious issues...

The problem with "magic" property bumps is people who switch between
branches (like I do) - for them, this would cause a problem and be a
workflow-blocker. So maybe we should just detect whether the right settings
are set for spotless (and any other task that needs it) and leave out
everything else.

> Instructions/failure message could be something like: "[something has
> changed]. You might want to consider re-generating your `gradle.properties`
> file (and re-applying any custom modifications). to ignore this warning,
> you can manually update [props_version] property in your current
> `gradle.properties` file".
>

It's a fairly basic property file. You could even suggest what needs to be
changed (additions/ removals/ changed properties).

> IMO it'd be ok to fail the first run if gradle.properties hadn't yet been
> generated.
>

I don't know. It should really work on the first run. On the other hand,
it's really slow then (not taking advantage of multiple cpus, etc.).

D.

>

Re: spotless targets [ In reply to ]

rcmuir at gmail

Apr 7, 2022, 10:47 AM

Post #8 of 13 (1013 views)

On Thu, Apr 7, 2022 at 1:32 PM Dawid Weiss <dawid.weiss@gmail.com> wrote:
>
>
> This can be made in a number of ways, I guess. I just didn't have time/ care for it much. Typically everything would run just fine - the JVM options that open up modules for spotless is the first thing that caused serious issues...
>

Does spotless have an option to fork (like the old ant fork=true),
e.g. passing the very special list of module system exports/opens that
it needs? This way these hacks wouldn't need to be in this particular
file passed to the entire gradle JVM? IMO, that would improve the
situation (less stuff in the file, etc).

Are all gradle spotless users having to do these things? Can they be
somewhere other than this file?

---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: dev-help@lucene.apache.org

Re: spotless targets [ In reply to ]

Apr 7, 2022, 12:26 PM

Post #9 of 13 (1013 views)

>
> Does spotless have an option to fork (like the old ant fork=true),
>

I don't know but I don't think so. It's a plain plugin for gradle, with its
own set of dependencies. Gradle plugins are not forked - they run within
the process (much like ant tasks).

> Are all gradle spotless users having to do these things? Can they be
> somewhere other than this file?

I don't think there is any other way. The problem is not even spotless
itself - it's the google formatting code that is a dependency of spotless
that is causing problems here.

https://github.com/diffplug/spotless/tree/main/plugin-gradle#google-java-format
https://github.com/diffplug/spotless/issues/834

You could run this plugin via methods other than the gradle integration
layer (for example via forked ant). This has drawbacks though - no
incremental inputs, worse reporting, etc.

Dawid

Re: spotless targets [ In reply to ]

Apr 8, 2022, 12:04 AM

Post #10 of 13 (1013 views)

Maybe a check like this?
https://github.com/apache/lucene/pull/802

On Thu, Apr 7, 2022 at 9:26 PM Dawid Weiss <dawid.weiss@gmail.com> wrote:

> Does spotless have an option to fork (like the old ant fork=true),
>>
>
> I don't know but I don't think so. It's a plain plugin for gradle, with
> its own set of dependencies. Gradle plugins are not forked - they run
> within the process (much like ant tasks).
>
>
>> Are all gradle spotless users having to do these things? Can they be
>> somewhere other than this file?
>
>
> I don't think there is any other way. The problem is not even spotless
> itself - it's the google formatting code that is a dependency of spotless
> that is causing problems here.
>
>
> https://github.com/diffplug/spotless/tree/main/plugin-gradle#google-java-format
> https://github.com/diffplug/spotless/issues/834
>
> You could run this plugin via methods other than the gradle integration
> layer (for example via forked ant). This has drawbacks though - no
> incremental inputs, worse reporting, etc.
>
> Dawid
>

Re: spotless targets [ In reply to ]

Apr 8, 2022, 8:07 AM

Post #11 of 13 (1013 views)

I guess this is related to the use of Java modules that now hide symbols?

On Fri, Apr 8, 2022 at 3:05 AM Dawid Weiss <dawid.weiss@gmail.com> wrote:
>
>
> Maybe a check like this?
> https://github.com/apache/lucene/pull/802
>
> On Thu, Apr 7, 2022 at 9:26 PM Dawid Weiss <dawid.weiss@gmail.com> wrote:
>>>
>>> Does spotless have an option to fork (like the old ant fork=true),
>>
>>
>> I don't know but I don't think so. It's a plain plugin for gradle, with its own set of dependencies. Gradle plugins are not forked - they run within the process (much like ant tasks).
>>
>>>
>>> Are all gradle spotless users having to do these things? Can they be
>>> somewhere other than this file?
>>
>>
>> I don't think there is any other way. The problem is not even spotless itself - it's the google formatting code that is a dependency of spotless that is causing problems here.
>>
>> https://github.com/diffplug/spotless/tree/main/plugin-gradle#google-java-format
>> https://github.com/diffplug/spotless/issues/834
>>
>> You could run this plugin via methods other than the gradle integration layer (for example via forked ant). This has drawbacks though - no incremental inputs, worse reporting, etc.
>>
>> Dawid

---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: dev-help@lucene.apache.org

Re: spotless targets [ In reply to ]

uwe at thetaphi

Apr 8, 2022, 8:30 AM

Post #12 of 13 (1013 views)

The reason for this is a change in Java 16 that disallows to access packages in modules which are not exported. The compiler internals are not exported. Originally this was already planned for Java 9 but changed shortly before release. Instead a warning was printed that a classpath based application tries to access non-exported packages. It was controllable by parameter "--illegal-access=....". In Jdk 16 the default was changed from "warn/permit" to "deny". In Jdk 17 the setting was completely removed (https://openjdk.java.net/jeps/403).

So you have to tell java to export the affected packages (each one separately listed) also to classpath applications (unnamed module).

Uwe

Am 8. April 2022 15:07:44 UTC schrieb Michael Sokolov <msokolov@gmail.com>:
>I guess this is related to the use of Java modules that now hide symbols?
>
>On Fri, Apr 8, 2022 at 3:05 AM Dawid Weiss <dawid.weiss@gmail.com> wrote:
>>
>>
>> Maybe a check like this?
>> https://github.com/apache/lucene/pull/802
>>
>> On Thu, Apr 7, 2022 at 9:26 PM Dawid Weiss <dawid.weiss@gmail.com> wrote:
>>>>
>>>> Does spotless have an option to fork (like the old ant fork=true),
>>>
>>>
>>> I don't know but I don't think so. It's a plain plugin for gradle, with its own set of dependencies. Gradle plugins are not forked - they run within the process (much like ant tasks).
>>>
>>>>
>>>> Are all gradle spotless users having to do these things? Can they be
>>>> somewhere other than this file?
>>>
>>>
>>> I don't think there is any other way. The problem is not even spotless itself - it's the google formatting code that is a dependency of spotless that is causing problems here.
>>>
>>> https://github.com/diffplug/spotless/tree/main/plugin-gradle#google-java-format
>>> https://github.com/diffplug/spotless/issues/834
>>>
>>> You could run this plugin via methods other than the gradle integration layer (for example via forked ant). This has drawbacks though - no incremental inputs, worse reporting, etc.
>>>
>>> Dawid
>
>---------------------------------------------------------------------
>To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
>For additional commands, e-mail: dev-help@lucene.apache.org
>

--
Uwe Schindler
Achterdiek 19, 28357 Bremen
https://www.thetaphi.de

Re: spotless targets [ In reply to ]

Apr 8, 2022, 10:14 AM

Post #13 of 13 (1013 views)