Mailing List Archive

FWIW: I don't remember it ever being randomizable -- because it has to be
set as a commandline arg on the JVM. Users have always been able to set
it to either true|false from the ant/gradle command line however (maybe
Mike was thinking of jenkins jobs that randomize it? ... i thought uwe's
jenkins box use to randomize it as part of the VM randomization but I
don't think that's currently true.

-Hoss
http://www.lucidworks.com/

---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: dev-help@lucene.apache.org

Run, for example:

gradlew -p lucene testOpts

to see the options available. Notice this one:

tests.asserts = true # Enables or disables assertions mode.

and run your tests with:

gradlew -p lucene -Ptests.asserts=false test

From IDE level you can disable assertions with -Dtests.asserts=false
(and not adding -ea).

D.

On Thu, Feb 18, 2021 at 3:45 AM Gautam Worah <worah.gautam@gmail.com> wrote:
>
> Hi Folks,
>
> I was working on PR LUCENE-9476 when we found cases where `asserts` in test cases were always enabled. To verify whether this was true, I wrote a simple test case that called assert 1000 times, modified a variable and then checked its value. This test case always passed because `assert` was always enabled.
>
> Mike McCandless mentioned in the PR that Lucene earlier had the capability to randomly disable `asserts` so that there were no accidental cases of developers relying on `asserts` always being enabled. We may have lost this feature when the project transitioned to Gradle.
> When I change the default value of `tests.asserts` in randomization.gradle to false, the test fails promptly.
>
> Has anyone else noticed this/knows more about this?
>
> Thanks,
> Gautam Worah.

---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: dev-help@lucene.apache.org

Ahh, you are right, Dawid! I thought our test infra had the randomization
of enabling or disabling assertions, but it looks like it is our Jenkins
jobs that contain the randomization.

Could we somehow move the randomness into our test infrastructure? E.g. it
would have helped Gautam (and other developers in the future) catch that
subtle but nasty bug in his PR during private development rather than later
in random Jenkins runs. If I remember right, we originally added this
randomization precisely because we had such a bug, and the bug got released
and later caught and fixed in a subsequent release.

Or is that a bad idea for some reason :)

Mike McCandless

http://blog.mikemccandless.com


On Fri, Feb 19, 2021 at 3:44 AM Dawid Weiss <dawid.weiss@gmail.com> wrote:

> Run, for example:
>
> gradlew -p lucene testOpts
>
> to see the options available. Notice this one:
>
> tests.asserts = true # Enables or disables assertions
> mode.
>
> and run your tests with:
>
> gradlew -p lucene -Ptests.asserts=false test
>
> From IDE level you can disable assertions with -Dtests.asserts=false
> (and not adding -ea).
>
> D.
>
> On Thu, Feb 18, 2021 at 3:45 AM Gautam Worah <worah.gautam@gmail.com>
> wrote:
> >
> > Hi Folks,
> >
> > I was working on PR LUCENE-9476 when we found cases where `asserts` in
> test cases were always enabled. To verify whether this was true, I wrote a
> simple test case that called assert 1000 times, modified a variable and
> then checked its value. This test case always passed because `assert` was
> always enabled.
> >
> > Mike McCandless mentioned in the PR that Lucene earlier had the
> capability to randomly disable `asserts` so that there were no accidental
> cases of developers relying on `asserts` always being enabled. We may have
> lost this feature when the project transitioned to Gradle.
> > When I change the default value of `tests.asserts` in
> randomization.gradle to false, the test fails promptly.
> >
> > Has anyone else noticed this/knows more about this?
> >
> > Thanks,
> > Gautam Worah.
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
> For additional commands, e-mail: dev-help@lucene.apache.org
>
>

We sure can - you'd have to derive the default option for
tests.asserts from the seed, much like it's done for file encoding:

https://github.com/apache/lucene-solr/blob/master/gradle/testing/randomization.gradle#L94-L97

This has to be done on the build side, not at runtime, because once
the JVM starts, it's too late (assertions are either on or off).

D.

On Fri, Feb 19, 2021 at 1:08 PM Michael McCandless
<lucene@mikemccandless.com> wrote:
>
> Ahh, you are right, Dawid! I thought our test infra had the randomization of enabling or disabling assertions, but it looks like it is our Jenkins jobs that contain the randomization.
>
> Could we somehow move the randomness into our test infrastructure? E.g. it would have helped Gautam (and other developers in the future) catch that subtle but nasty bug in his PR during private development rather than later in random Jenkins runs. If I remember right, we originally added this randomization precisely because we had such a bug, and the bug got released and later caught and fixed in a subsequent release.
>
> Or is that a bad idea for some reason :)
>
> Mike McCandless
>
> http://blog.mikemccandless.com
>
>
> On Fri, Feb 19, 2021 at 3:44 AM Dawid Weiss <dawid.weiss@gmail.com> wrote:
>>
>> Run, for example:
>>
>> gradlew -p lucene testOpts
>>
>> to see the options available. Notice this one:
>>
>> tests.asserts = true # Enables or disables assertions mode.
>>
>> and run your tests with:
>>
>> gradlew -p lucene -Ptests.asserts=false test
>>
>> From IDE level you can disable assertions with -Dtests.asserts=false
>> (and not adding -ea).
>>
>> D.
>>
>> On Thu, Feb 18, 2021 at 3:45 AM Gautam Worah <worah.gautam@gmail.com> wrote:
>> >
>> > Hi Folks,
>> >
>> > I was working on PR LUCENE-9476 when we found cases where `asserts` in test cases were always enabled. To verify whether this was true, I wrote a simple test case that called assert 1000 times, modified a variable and then checked its value. This test case always passed because `assert` was always enabled.
>> >
>> > Mike McCandless mentioned in the PR that Lucene earlier had the capability to randomly disable `asserts` so that there were no accidental cases of developers relying on `asserts` always being enabled. We may have lost this feature when the project transitioned to Gradle.
>> > When I change the default value of `tests.asserts` in randomization.gradle to false, the test fails promptly.
>> >
>> > Has anyone else noticed this/knows more about this?
>> >
>> > Thanks,
>> > Gautam Worah.
>>
>> ---------------------------------------------------------------------
>> To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
>> For additional commands, e-mail: dev-help@lucene.apache.org
>>

---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: dev-help@lucene.apache.org

I think it has a downside: having a bug in an assert is really more of a
corner case. This is the kind of thing jenkins is for?

Locally it would be nice for the compile-test-debug loop for the asserts to
always be working. Tying it into random seeds is gonna result in tests that
behave in crazy ways: I'm thinking about first time contributors and stuff.
Not every test is some crazy random one...

On Fri, Feb 19, 2021 at 7:08 AM Michael McCandless <
lucene@mikemccandless.com> wrote:

> Ahh, you are right, Dawid! I thought our test infra had the randomization
> of enabling or disabling assertions, but it looks like it is our Jenkins
> jobs that contain the randomization.
>
> Could we somehow move the randomness into our test infrastructure? E.g.
> it would have helped Gautam (and other developers in the future) catch that
> subtle but nasty bug in his PR during private development rather than later
> in random Jenkins runs. If I remember right, we originally added this
> randomization precisely because we had such a bug, and the bug got released
> and later caught and fixed in a subsequent release.
>
> Or is that a bad idea for some reason :)
>
> Mike McCandless
>
> http://blog.mikemccandless.com
>
>
> On Fri, Feb 19, 2021 at 3:44 AM Dawid Weiss <dawid.weiss@gmail.com> wrote:
>
>> Run, for example:
>>
>> gradlew -p lucene testOpts
>>
>> to see the options available. Notice this one:
>>
>> tests.asserts = true # Enables or disables assertions
>> mode.
>>
>> and run your tests with:
>>
>> gradlew -p lucene -Ptests.asserts=false test
>>
>> From IDE level you can disable assertions with -Dtests.asserts=false
>> (and not adding -ea).
>>
>> D.
>>
>> On Thu, Feb 18, 2021 at 3:45 AM Gautam Worah <worah.gautam@gmail.com>
>> wrote:
>> >
>> > Hi Folks,
>> >
>> > I was working on PR LUCENE-9476 when we found cases where `asserts` in
>> test cases were always enabled. To verify whether this was true, I wrote a
>> simple test case that called assert 1000 times, modified a variable and
>> then checked its value. This test case always passed because `assert` was
>> always enabled.
>> >
>> > Mike McCandless mentioned in the PR that Lucene earlier had the
>> capability to randomly disable `asserts` so that there were no accidental
>> cases of developers relying on `asserts` always being enabled. We may have
>> lost this feature when the project transitioned to Gradle.
>> > When I change the default value of `tests.asserts` in
>> randomization.gradle to false, the test fails promptly.
>> >
>> > Has anyone else noticed this/knows more about this?
>> >
>> > Thanks,
>> > Gautam Worah.
>>
>> ---------------------------------------------------------------------
>> To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
>> For additional commands, e-mail: dev-help@lucene.apache.org
>>
>>

On Fri, Feb 19, 2021 at 8:07 AM Robert Muir <rcmuir@gmail.com> wrote:

I think it has a downside: having a bug in an assert is really more of a
> corner case. This is the kind of thing jenkins is for?
>

Ahh, that is indeed a really good point. I would want/expect asserts to
always work correctly when running local tests ... if we randomly disabled
them in our checkouts it can cause a false sense of security, too soon.

OK, I agree, let's leave it as randomization in Jenkins! How do we know
that Jenkins job/s are still randomizing assertions? Who tests the tester?

Mike McCandless

http://blog.mikemccandless.com

I don't argue either way - just stating it's possible to randomize this.


> I think it has a downside: having a bug in an assert is really more of a
> corner case. This is the kind of thing jenkins is for?
>

This may be the reason why it's fixed by default (it is/was in ant too).

I don't think it is enabled (at least in policeman jenkins). perhaps it
didn't work correctly when the build was cutover to gradle. Take a look at
any old build such as
https://jenkins.thetaphi.de/view/Lucene-Solr/job/Lucene-Solr-master-Linux/29491/
. You can see the variables it randomizes right there.

You can confirm by clicking console->full log and it prints exact gradle
command that it runs:
https://jenkins.thetaphi.de/view/Lucene-Solr/job/Lucene-Solr-master-Linux/29491/consoleFull

Let's look into it, in a couple weeks or so?

On Fri, Feb 19, 2021 at 8:32 AM Michael McCandless <
lucene@mikemccandless.com> wrote:

> On Fri, Feb 19, 2021 at 8:07 AM Robert Muir <rcmuir@gmail.com> wrote:
>
> I think it has a downside: having a bug in an assert is really more of a
>> corner case. This is the kind of thing jenkins is for?
>>
>
> Ahh, that is indeed a really good point. I would want/expect asserts to
> always work correctly when running local tests ... if we randomly disabled
> them in our checkouts it can cause a false sense of security, too soon.
>
> OK, I agree, let's leave it as randomization in Jenkins! How do we know
> that Jenkins job/s are still randomizing assertions? Who tests the tester?
>
> Mike McCandless
>
> http://blog.mikemccandless.com
>

Hi,

I don’t fully remember what the setup previously was, but at least for master and 8.x it does not automatically enable/disable asserts. We can of course do this together with the other settings like GC or compressed OOPs, its just a few more lines in the Groovy file.



I was also thinking that we have Security Manager enabled/disabled from time to time. But recently, I see no randomization for this on Jenkins, unless it’s part of the Gradle build.



Uwe



-----

Uwe Schindler

Achterdiek 19, D-28357 Bremen

https://www.thetaphi.de

eMail: uwe@thetaphi.de



From: Robert Muir <rcmuir@gmail.com>
Sent: Friday, February 19, 2021 3:13 PM
To: dev@lucene.apache.org
Subject: Re: Random disabling of asserts in tests is not working



I don't think it is enabled (at least in policeman jenkins). perhaps it didn't work correctly when the build was cutover to gradle. Take a look at any old build such as https://jenkins.thetaphi.de/view/Lucene-Solr/job/Lucene-Solr-master-Linux/29491/ . You can see the variables it randomizes right there.



You can confirm by clicking console->full log and it prints exact gradle command that it runs: https://jenkins.thetaphi.de/view/Lucene-Solr/job/Lucene-Solr-master-Linux/29491/consoleFull



Let's look into it, in a couple weeks or so?



On Fri, Feb 19, 2021 at 8:32 AM Michael McCandless <lucene@mikemccandless.com <mailto:lucene@mikemccandless.com> > wrote:

On Fri, Feb 19, 2021 at 8:07 AM Robert Muir <rcmuir@gmail.com <mailto:rcmuir@gmail.com> > wrote:



I think it has a downside: having a bug in an assert is really more of a corner case. This is the kind of thing jenkins is for?



Ahh, that is indeed a really good point. I would want/expect asserts to always work correctly when running local tests ... if we randomly disabled them in our checkouts it can cause a false sense of security, too soon.



OK, I agree, let's leave it as randomization in Jenkins! How do we know that Jenkins job/s are still randomizing assertions? Who tests the tester?



Mike McCandless

http://blog.mikemccandless.com

Hi Uwe,

No, it's not randomized - always runs with the security manager enabled.
All the options are here:

https://github.com/apache/lucene-solr/blob/master/gradle/testing/randomization.gradle#L68-L103

When the value says "random" we pick the random value at runtime (so that
it also works within IDEs). We could pick security manager at build-time
(derive from project seed). This is a no-brainer to do. As Robert said -
perhaps we should keep some things more strict for developers and just
shuffle on the CI-only. This requires passing -Ptests.*=... flags but is
simple, I think.

Dawid

On Fri, Feb 19, 2021 at 7:45 PM Uwe Schindler <uwe@thetaphi.de> wrote:

> Hi,
>
> I don’t fully remember what the setup previously was, but at least for
> master and 8.x it does not automatically enable/disable asserts. We can of
> course do this together with the other settings like GC or compressed OOPs,
> its just a few more lines in the Groovy file.
>
>
>
> I was also thinking that we have Security Manager enabled/disabled from
> time to time. But recently, I see no randomization for this on Jenkins,
> unless it’s part of the Gradle build.
>
>
>
> Uwe
>
>
>
> -----
>
> Uwe Schindler
>
> Achterdiek 19, D-28357 Bremen
>
> https://www.thetaphi.de
>
> eMail: uwe@thetaphi.de
>
>
>
> *From:* Robert Muir <rcmuir@gmail.com>
> *Sent:* Friday, February 19, 2021 3:13 PM
> *To:* dev@lucene.apache.org
> *Subject:* Re: Random disabling of asserts in tests is not working
>
>
>
> I don't think it is enabled (at least in policeman jenkins). perhaps it
> didn't work correctly when the build was cutover to gradle. Take a look at
> any old build such as
> https://jenkins.thetaphi.de/view/Lucene-Solr/job/Lucene-Solr-master-Linux/29491/
> . You can see the variables it randomizes right there.
>
>
>
> You can confirm by clicking console->full log and it prints exact gradle
> command that it runs:
> https://jenkins.thetaphi.de/view/Lucene-Solr/job/Lucene-Solr-master-Linux/29491/consoleFull
>
>
>
> Let's look into it, in a couple weeks or so?
>
>
>
> On Fri, Feb 19, 2021 at 8:32 AM Michael McCandless <
> lucene@mikemccandless.com> wrote:
>
> On Fri, Feb 19, 2021 at 8:07 AM Robert Muir <rcmuir@gmail.com> wrote:
>
>
>
> I think it has a downside: having a bug in an assert is really more of a
> corner case. This is the kind of thing jenkins is for?
>
>
>
> Ahh, that is indeed a really good point. I would want/expect asserts to
> always work correctly when running local tests ... if we randomly disabled
> them in our checkouts it can cause a false sense of security, too soon.
>
>
>
> OK, I agree, let's leave it as randomization in Jenkins! How do we know
> that Jenkins job/s are still randomizing assertions? Who tests the tester?
>
>
>
> Mike McCandless
>
> http://blog.mikemccandless.com
>
>

Hi Dawid,



cool thanks for the pointer. I really like this list ????

So I am not sure at which point we ever had randomization of security manager and/or asserts. I assume, Policeman Jenkins never had that.

I have the feeling that Elastic did this on their build servers, but that’s also not proved. I suggested that change in one of my talks at BerlinBuzzwords, but may have never implemented it.



Anyways: I am open to add randomization on Jenkins, it’s just 2 lines of code in the randomize-java-groovy file on Policeman Jenkins. Maybe disable asserts/and or SecurityManager in 1/5th of all cases.



Uwe



-----

Uwe Schindler

Achterdiek 19, D-28357 Bremen

https://www.thetaphi.de

eMail: uwe@thetaphi.de



From: Dawid Weiss <dawid.weiss@gmail.com>
Sent: Friday, February 19, 2021 7:52 PM
To: Lucene Dev <dev@lucene.apache.org>
Subject: Re: Random disabling of asserts in tests is not working





Hi Uwe,



No, it's not randomized - always runs with the security manager enabled. All the options are here:



https://github.com/apache/lucene-solr/blob/master/gradle/testing/randomization.gradle#L68-L103



When the value says "random" we pick the random value at runtime (so that it also works within IDEs). We could pick security manager at build-time (derive from project seed). This is a no-brainer to do. As Robert said - perhaps we should keep some things more strict for developers and just shuffle on the CI-only. This requires passing -Ptests.*=... flags but is simple, I think.



Dawid



On Fri, Feb 19, 2021 at 7:45 PM Uwe Schindler <uwe@thetaphi.de <mailto:uwe@thetaphi.de> > wrote:

Hi,

I don’t fully remember what the setup previously was, but at least for master and 8.x it does not automatically enable/disable asserts. We can of course do this together with the other settings like GC or compressed OOPs, its just a few more lines in the Groovy file.



I was also thinking that we have Security Manager enabled/disabled from time to time. But recently, I see no randomization for this on Jenkins, unless it’s part of the Gradle build.



Uwe



-----

Uwe Schindler

Achterdiek 19, D-28357 Bremen

https://www.thetaphi.de

eMail: uwe@thetaphi.de <mailto:uwe@thetaphi.de>



From: Robert Muir <rcmuir@gmail.com <mailto:rcmuir@gmail.com> >
Sent: Friday, February 19, 2021 3:13 PM
To: dev@lucene.apache.org <mailto:dev@lucene.apache.org>
Subject: Re: Random disabling of asserts in tests is not working



I don't think it is enabled (at least in policeman jenkins). perhaps it didn't work correctly when the build was cutover to gradle. Take a look at any old build such as https://jenkins.thetaphi.de/view/Lucene-Solr/job/Lucene-Solr-master-Linux/29491/ . You can see the variables it randomizes right there.



You can confirm by clicking console->full log and it prints exact gradle command that it runs: https://jenkins.thetaphi.de/view/Lucene-Solr/job/Lucene-Solr-master-Linux/29491/consoleFull



Let's look into it, in a couple weeks or so?



On Fri, Feb 19, 2021 at 8:32 AM Michael McCandless <lucene@mikemccandless.com <mailto:lucene@mikemccandless.com> > wrote:

On Fri, Feb 19, 2021 at 8:07 AM Robert Muir <rcmuir@gmail.com <mailto:rcmuir@gmail.com> > wrote:



I think it has a downside: having a bug in an assert is really more of a corner case. This is the kind of thing jenkins is for?



Ahh, that is indeed a really good point. I would want/expect asserts to always work correctly when running local tests ... if we randomly disabled them in our checkouts it can cause a false sense of security, too soon.



OK, I agree, let's leave it as randomization in Jenkins! How do we know that Jenkins job/s are still randomizing assertions? Who tests the tester?



Mike McCandless

http://blog.mikemccandless.com

> cool thanks for the pointer. I really like this list ????
>

The list is sort of internal detail... what I really wanted to have is a
list of options and current "values" computed for a particular run of
options and seed - this is the "testOpts" task that you can run for any
project. Compare the output of these (note the flags - "C" for computed
value, "!" for non-default value, etc.):

gradlew -p lucene/core testOpts
gradlew -p lucene/solr testOpts
gradlew -p lucene/core testOpts -Ptests.asserts=false

This can be improved even more... and I have written a plugin that cleans
up management of such options, but haven't had the time to port Lucene's
build yet, eh.

Dawid




> So I am not sure at which point we ever had randomization of security
> manager and/or asserts. I assume, Policeman Jenkins never had that.
>
> I have the feeling that Elastic did this on their build servers, but
> that’s also not proved. I suggested that change in one of my talks at
> BerlinBuzzwords, but may have never implemented it.
>
>
>
> Anyways: I am open to add randomization on Jenkins, it’s just 2 lines of
> code in the randomize-java-groovy file on Policeman Jenkins. Maybe disable
> asserts/and or SecurityManager in 1/5th of all cases.
>
>
>
> Uwe
>
>
>
> -----
>
> Uwe Schindler
>
> Achterdiek 19, D-28357 Bremen
>
> https://www.thetaphi.de
>
> eMail: uwe@thetaphi.de
>
>
>
> *From:* Dawid Weiss <dawid.weiss@gmail.com>
> *Sent:* Friday, February 19, 2021 7:52 PM
> *To:* Lucene Dev <dev@lucene.apache.org>
> *Subject:* Re: Random disabling of asserts in tests is not working
>
>
>
>
>
> Hi Uwe,
>
>
>
> No, it's not randomized - always runs with the security manager enabled.
> All the options are here:
>
>
>
>
> https://github.com/apache/lucene-solr/blob/master/gradle/testing/randomization.gradle#L68-L103
>
>
>
> When the value says "random" we pick the random value at runtime (so that
> it also works within IDEs). We could pick security manager at build-time
> (derive from project seed). This is a no-brainer to do. As Robert said -
> perhaps we should keep some things more strict for developers and just
> shuffle on the CI-only. This requires passing -Ptests.*=... flags but is
> simple, I think.
>
>
>
> Dawid
>
>
>
> On Fri, Feb 19, 2021 at 7:45 PM Uwe Schindler <uwe@thetaphi.de> wrote:
>
> Hi,
>
> I don’t fully remember what the setup previously was, but at least for
> master and 8.x it does not automatically enable/disable asserts. We can of
> course do this together with the other settings like GC or compressed OOPs,
> its just a few more lines in the Groovy file.
>
>
>
> I was also thinking that we have Security Manager enabled/disabled from
> time to time. But recently, I see no randomization for this on Jenkins,
> unless it’s part of the Gradle build.
>
>
>
> Uwe
>
>
>
> -----
>
> Uwe Schindler
>
> Achterdiek 19, D-28357 Bremen
>
> https://www.thetaphi.de
>
> eMail: uwe@thetaphi.de
>
>
>
> *From:* Robert Muir <rcmuir@gmail.com>
> *Sent:* Friday, February 19, 2021 3:13 PM
> *To:* dev@lucene.apache.org
> *Subject:* Re: Random disabling of asserts in tests is not working
>
>
>
> I don't think it is enabled (at least in policeman jenkins). perhaps it
> didn't work correctly when the build was cutover to gradle. Take a look at
> any old build such as
> https://jenkins.thetaphi.de/view/Lucene-Solr/job/Lucene-Solr-master-Linux/29491/
> . You can see the variables it randomizes right there.
>
>
>
> You can confirm by clicking console->full log and it prints exact gradle
> command that it runs:
> https://jenkins.thetaphi.de/view/Lucene-Solr/job/Lucene-Solr-master-Linux/29491/consoleFull
>
>
>
> Let's look into it, in a couple weeks or so?
>
>
>
> On Fri, Feb 19, 2021 at 8:32 AM Michael McCandless <
> lucene@mikemccandless.com> wrote:
>
> On Fri, Feb 19, 2021 at 8:07 AM Robert Muir <rcmuir@gmail.com> wrote:
>
>
>
> I think it has a downside: having a bug in an assert is really more of a
> corner case. This is the kind of thing jenkins is for?
>
>
>
> Ahh, that is indeed a really good point. I would want/expect asserts to
> always work correctly when running local tests ... if we randomly disabled
> them in our checkouts it can cause a false sense of security, too soon.
>
>
>
> OK, I agree, let's leave it as randomization in Jenkins! How do we know
> that Jenkins job/s are still randomizing assertions? Who tests the tester?
>
>
>
> Mike McCandless
>
> http://blog.mikemccandless.com
>
>

I agree with Rob -- let's leave this sort of thing for Jenkins. It's
really an edge case and I'd prefer consistently knowing the asserts always
work locally.

~ David Smiley
Apache Lucene/Solr Search Developer
http://www.linkedin.com/in/davidwsmiley


On Sat, Feb 20, 2021 at 1:14 PM Dawid Weiss <dawid.weiss@gmail.com> wrote:

>
> cool thanks for the pointer. I really like this list ????
>>
>
> The list is sort of internal detail... what I really wanted to have is a
> list of options and current "values" computed for a particular run of
> options and seed - this is the "testOpts" task that you can run for any
> project. Compare the output of these (note the flags - "C" for computed
> value, "!" for non-default value, etc.):
>
> gradlew -p lucene/core testOpts
> gradlew -p lucene/solr testOpts
> gradlew -p lucene/core testOpts -Ptests.asserts=false
>
> This can be improved even more... and I have written a plugin that cleans
> up management of such options, but haven't had the time to port Lucene's
> build yet, eh.
>
> Dawid
>
>
>
>
>> So I am not sure at which point we ever had randomization of security
>> manager and/or asserts. I assume, Policeman Jenkins never had that.
>>
>> I have the feeling that Elastic did this on their build servers, but
>> that’s also not proved. I suggested that change in one of my talks at
>> BerlinBuzzwords, but may have never implemented it.
>>
>>
>>
>> Anyways: I am open to add randomization on Jenkins, it’s just 2 lines of
>> code in the randomize-java-groovy file on Policeman Jenkins. Maybe disable
>> asserts/and or SecurityManager in 1/5th of all cases.
>>
>>
>>
>> Uwe
>>
>>
>>
>> -----
>>
>> Uwe Schindler
>>
>> Achterdiek 19, D-28357 Bremen
>>
>> https://www.thetaphi.de
>>
>> eMail: uwe@thetaphi.de
>>
>>
>>
>> *From:* Dawid Weiss <dawid.weiss@gmail.com>
>> *Sent:* Friday, February 19, 2021 7:52 PM
>> *To:* Lucene Dev <dev@lucene.apache.org>
>> *Subject:* Re: Random disabling of asserts in tests is not working
>>
>>
>>
>>
>>
>> Hi Uwe,
>>
>>
>>
>> No, it's not randomized - always runs with the security manager enabled.
>> All the options are here:
>>
>>
>>
>>
>> https://github.com/apache/lucene-solr/blob/master/gradle/testing/randomization.gradle#L68-L103
>>
>>
>>
>> When the value says "random" we pick the random value at runtime (so that
>> it also works within IDEs). We could pick security manager at build-time
>> (derive from project seed). This is a no-brainer to do. As Robert said -
>> perhaps we should keep some things more strict for developers and just
>> shuffle on the CI-only. This requires passing -Ptests.*=... flags but is
>> simple, I think.
>>
>>
>>
>> Dawid
>>
>>
>>
>> On Fri, Feb 19, 2021 at 7:45 PM Uwe Schindler <uwe@thetaphi.de> wrote:
>>
>> Hi,
>>
>> I don’t fully remember what the setup previously was, but at least for
>> master and 8.x it does not automatically enable/disable asserts. We can of
>> course do this together with the other settings like GC or compressed OOPs,
>> its just a few more lines in the Groovy file.
>>
>>
>>
>> I was also thinking that we have Security Manager enabled/disabled from
>> time to time. But recently, I see no randomization for this on Jenkins,
>> unless it’s part of the Gradle build.
>>
>>
>>
>> Uwe
>>
>>
>>
>> -----
>>
>> Uwe Schindler
>>
>> Achterdiek 19, D-28357 Bremen
>>
>> https://www.thetaphi.de
>>
>> eMail: uwe@thetaphi.de
>>
>>
>>
>> *From:* Robert Muir <rcmuir@gmail.com>
>> *Sent:* Friday, February 19, 2021 3:13 PM
>> *To:* dev@lucene.apache.org
>> *Subject:* Re: Random disabling of asserts in tests is not working
>>
>>
>>
>> I don't think it is enabled (at least in policeman jenkins). perhaps it
>> didn't work correctly when the build was cutover to gradle. Take a look at
>> any old build such as
>> https://jenkins.thetaphi.de/view/Lucene-Solr/job/Lucene-Solr-master-Linux/29491/
>> . You can see the variables it randomizes right there.
>>
>>
>>
>> You can confirm by clicking console->full log and it prints exact gradle
>> command that it runs:
>> https://jenkins.thetaphi.de/view/Lucene-Solr/job/Lucene-Solr-master-Linux/29491/consoleFull
>>
>>
>>
>> Let's look into it, in a couple weeks or so?
>>
>>
>>
>> On Fri, Feb 19, 2021 at 8:32 AM Michael McCandless <
>> lucene@mikemccandless.com> wrote:
>>
>> On Fri, Feb 19, 2021 at 8:07 AM Robert Muir <rcmuir@gmail.com> wrote:
>>
>>
>>
>> I think it has a downside: having a bug in an assert is really more of a
>> corner case. This is the kind of thing jenkins is for?
>>
>>
>>
>> Ahh, that is indeed a really good point. I would want/expect asserts to
>> always work correctly when running local tests ... if we randomly disabled
>> them in our checkouts it can cause a false sense of security, too soon.
>>
>>
>>
>> OK, I agree, let's leave it as randomization in Jenkins! How do we know
>> that Jenkins job/s are still randomizing assertions? Who tests the tester?
>>
>>
>>
>> Mike McCandless
>>
>> http://blog.mikemccandless.com
>>
>>