Mailing List Archive

BadApple report
If people don’t know about: http://fucit.org/solr-jenkins-reports/suspicious-failure-report.html, I strongly recommend you periodically check it. It reports tests that have changed their failure rates lately. There are three currently:

"org.apache.solr.search.TestIndexSearcher","testSearcherListeners"
"org.apache.solr.update.processor.DocExpirationUpdateProcessorFactoryTest","testAutomaticDeletes"
"org.apache.solr.cloud.PackageManagerCLITest","testPackageManager

Short form:

Raw fail count by week totals, most recent week first (corresponds to bits):
Week: 0 had 128 failures
Week: 1 had 68 failures
Week: 2 had 113 failures
Week: 3 had 103 failures


********Failures in Hoss' reports for the last 4 rollups.

There were 298 unannotated tests that failed in Hoss' rollups. Ordered by the date I downloaded the rollup file, newest->oldest. See above for the dates the files were collected
These tests were NOT BadApple'd or AwaitsFix'd

Failures in the last 4 reports..
Report Pct runs fails test
0123 0.4 1461 5 DeleteReplicaTest.deleteReplicaAndVerifyDirectoryCleanup
0123 0.7 1464 9 MetricTriggerIntegrationTest.testMetricTrigger
0123 1.6 1377 29 MultiThreadedOCPTest.test
0123 0.7 1455 5 NodeMarkersRegistrationTest.testNodeMarkersRegistration
0123 2.1 1481 17 RollingRestartTest.test
0123 0.4 1537 55 ScheduledTriggerIntegrationTest.testScheduledTrigger
0123 7.7 98 6 ShardSplitTest.testSplitWithChaosMonkey
0123 0.4 1455 9 SystemCollectionCompatTest.testBackCompat
0123 0.7 1456 14 TestPackages.testPluginLoading
0123 1.1 1460 9 TestQueryingOnDownCollection.testQueryToDownCollectionShouldFailFast
0123 0.7 1498 13 TestSimScenario.testSuggestions
********************************************
I took the SuppressWarnings count section out, it’s ridiculously big.
Re: BadApple report [ In reply to ]
Thanks for keeping an eye Erick. I took a quick look at the
"TestIndexSearcher" failures and I think they're related to SOLR-14525.
Should be fixed after this[1] commit by Noble.

[1] https://gitbox.apache.org/repos/asf?p=lucene-solr.git;h=5827ddf

On Mon, Jun 8, 2020 at 7:52 AM Erick Erickson <erickerickson@gmail.com>
wrote:

> If people don’t know about:
> http://fucit.org/solr-jenkins-reports/suspicious-failure-report.html, I
> strongly recommend you periodically check it. It reports tests that have
> changed their failure rates lately. There are three currently:
>
> "org.apache.solr.search.TestIndexSearcher","testSearcherListeners"
>
> "org.apache.solr.update.processor.DocExpirationUpdateProcessorFactoryTest","testAutomaticDeletes"
> "org.apache.solr.cloud.PackageManagerCLITest","testPackageManager
>
> Short form:
>
> Raw fail count by week totals, most recent week first (corresponds to
> bits):
> Week: 0 had 128 failures
> Week: 1 had 68 failures
> Week: 2 had 113 failures
> Week: 3 had 103 failures
>
>
> ********Failures in Hoss' reports for the last 4 rollups.
>
> There were 298 unannotated tests that failed in Hoss' rollups. Ordered by
> the date I downloaded the rollup file, newest->oldest. See above for the
> dates the files were collected
> These tests were NOT BadApple'd or AwaitsFix'd
>
> Failures in the last 4 reports..
> Report Pct runs fails test
> 0123 0.4 1461 5
> DeleteReplicaTest.deleteReplicaAndVerifyDirectoryCleanup
> 0123 0.7 1464 9
> MetricTriggerIntegrationTest.testMetricTrigger
> 0123 1.6 1377 29 MultiThreadedOCPTest.test
> 0123 0.7 1455 5
> NodeMarkersRegistrationTest.testNodeMarkersRegistration
> 0123 2.1 1481 17 RollingRestartTest.test
> 0123 0.4 1537 55
> ScheduledTriggerIntegrationTest.testScheduledTrigger
> 0123 7.7 98 6
> ShardSplitTest.testSplitWithChaosMonkey
> 0123 0.4 1455 9
> SystemCollectionCompatTest.testBackCompat
> 0123 0.7 1456 14 TestPackages.testPluginLoading
> 0123 1.1 1460 9
> TestQueryingOnDownCollection.testQueryToDownCollectionShouldFailFast
> 0123 0.7 1498 13 TestSimScenario.testSuggestions
> ********************************************
> I took the SuppressWarnings count section out, it’s ridiculously big.
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
> For additional commands, e-mail: dev-help@lucene.apache.org
BadApple report [ In reply to ]
The number of chronically failing tests dropped considerably this past week, whether that’s an anomaly or not is a good question.

I’ve finished the SuppressWarnings annotations, so next week I _should_ be able to include how many new SuppressWarnings have been added to the code and have it mean something. I _strongly_ urge people to see if they can remove these annotations when they’re working on the area of code anyway.

The second thing I urge people to do is use their IDE well. IntelliJ does a series of automatic “inspections” for instance that can point to issues. It’ll highlight C-style array declarations which isn’t really a bug, but... I’m _not_ saying we should fix everything the inspections highlight, for instance it doesn’t like

if (a == false)

want’s to “simplify” it to

if (!a)

That’s one inspection I want to turn off; I find it too easy to overlook the “!”.

However, another thing that’s highlighted is something like

if (object.getName().someMethod)

where getName may return null. Again, I’m not saying each and every one of these should be changed. Just look at it and see if it’s really something that could happen and guard if so (how many NPEs have we had to be fixed later?).

Oh, and do be aware that IntelliJ can annotate inspections, but don’t do that. There’s no reason to pollute the code with IntelliJ-specific annotations.

OK, here’s the regular report.

Raw fail count by week totals, most recent week first (corresponds to bits):
Week: 0 had 34 failures
Week: 1 had 128 failures
Week: 2 had 68 failures
Week: 3 had 113 failures


********Failures in Hoss' reports for the last 4 rollups.

There were 264 unannotated tests that failed in Hoss' rollups. Ordered by the date I downloaded the rollup file, newest->oldest. See above for the dates the files were collected
These tests were NOT BadApple'd or AwaitsFix'd

Failures in the last 4 reports..
Report Pct runs fails test
0123 1.7 1186 15 RollingRestartTest.test
0123 0.9 1161 9 SystemCollectionCompatTest.testBackCompat
0123 0.9 1190 11 TestSimScenario.testSuggestions
********************************************
BadApple report [ In reply to ]
Not a bad week all told, but something seems a little odd, I remember a lot more e-mails going by, but perhaps it’s just these 26 tests failing repeatedly.


Raw fail count by week totals, most recent week first (corresponds to bits):
Week: 0 had 26 failures
Week: 1 had 34 failures
Week: 2 had 128 failures
Week: 3 had 68 failures


********Failures in Hoss' reports for the last 4 rollups.

There were 208 unannotated tests that failed in Hoss' rollups. Ordered by the date I downloaded the rollup file, newest->oldest. See above for the dates the files were collected
These tests were NOT BadApple'd or AwaitsFix'd

Failures in the last 4 reports..
Report Pct runs fails test
0123 2.7 893 15 RollingRestartTest.test
0123 1.8 872 9 SystemCollectionCompatTest.testBackCompat
********************************************

Full report attached (less suppresswarnings data).
BadApple report [ In reply to ]
Holding fairly steady.

Raw fail count by week totals, most recent week first (corresponds to bits):
Week: 0 had 26 failures
Week: 1 had 26 failures
Week: 2 had 34 failures
Week: 3 had 128 failures

This week’s report includes the SuppressWarnings summary. This is really the baseline, I added a few more that are counted in this as part of getting clean compiles, included here so people can see what they look like.

Only one test has failed every week over the last 4:
Failures in the last 4 reports..
Report Pct runs fails test
0123 4.7 639 17 RollingRestartTest.test
********************************************

Full report attached:
BadApple report [ In reply to ]
Holding fairly steady, but IDK whether Hoss’ scraping is getting data from Uwe’s machines, thought I saw an e-mail go by about that.

this is the first report where the suppresswarnings stats mean anything.

Full report attached:
Re: BadApple report [ In reply to ]
Hi Erick,

I'm wondering what is meant by "DO NOT ANNOTATE LIST" at the start of the
report? Better yet, can you please link to the scraping tool used to
generate the report?

Thank you!
Megan

On Mon, Jul 6, 2020 at 8:07 AM Erick Erickson <erickerickson@gmail.com>
wrote:

> Holding fairly steady, but IDK whether Hoss’ scraping is getting data from
> Uwe’s machines, thought I saw an e-mail go by about that.
>
> this is the first report where the suppresswarnings stats mean anything.
>
> Full report attached:
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
> For additional commands, e-mail: dev-help@lucene.apache.org
BadApple report [ In reply to ]
Actaully, pretty good. The attached file has a lot of noise in it that’s a listing of the files that have more or less SuppressWarnings annotations than last week, the delta is -19. It’s a crude measure, I can replace N SuppressWarnings in a class with one for the entire class, but it’s also easy to count. Down is the right direction though.

NamedList accounts for a huge number of SuppressWarnings. I do wonder if we can figure out better ways to avoid warnings with that class. Other than replace it. Wholesale surgery to replace it just to avoid warnings is a pretty bad idea of course….


SuppressWarnings count: last week: 5,372, this week: 5,353, delta -19



Processing file (History bit 3): HOSS-2020-07-13.csv
Processing file (History bit 2): HOSS-2020-07-06.csv
Processing file (History bit 1): HOSS-2020-06-29.csv
Processing file (History bit 0): HOSS-2020-06-22.csv


Number of AwaitsFix: 46 Number of BadApples: 4


**Annotated tests that didn't fail in the last 4 weeks.

**Tests removed from the next two lists because they were specified in 'doNotEnable' in the properties file
MoveReplicaHDFSTest.testNormalFailedMove

**Annotations can be removed from the following tests because they haven't failed in the last 4 rollups.

**Methods: 0


Raw fail count by week totals, most recent week first (corresponds to bits):
Week: 0 had 19 failures
Week: 1 had 24 failures
Week: 2 had 26 failures
Week: 3 had 26 failures


********Failures in Hoss' reports for the last 4 rollups.

There were 71 unannotated tests that failed in Hoss' rollups. Ordered by the date I downloaded the rollup file, newest->oldest. See above for the dates the files were collected
These tests were NOT BadApple'd or AwaitsFix'd

Failures in the last 4 reports..
Report Pct runs fails test
0123 0.9 447 5 TestInPlaceUpdatesDistrib.test
********************************************
BadApple report [ In reply to ]
Well, that’s one way to reduce the number of SuppressWarnings… cut out massive amounts of code ;)….

SuppressWarnings count: last week: 5,353, this week: 4,835, delta -518

We had quite a spike in the raw number of tests that have failed at least once in the last week:

Raw fail count by week totals, most recent week first (corresponds to bits):
Week: 0 had 502 failures
Week: 1 had 19 failures
Week: 2 had 24 failures
Week: 3 had 26 failures

IDK whether this reflects a temporary glitch or whether we’re now scanning more builds. At any rate we’ll see what next week brings.

This bit is encouraging, very few tests have failed every week for the last 4.

********Failures in Hoss' reports for the last 4 rollups.

There were 536 unannotated tests that failed in Hoss' rollups. Ordered by the date I downloaded the rollup file, newest->oldest. See above for the dates the files were collected
These tests were NOT BadApple'd or AwaitsFix'd

Failures in the last 4 reports..
Report Pct runs fails test
0123 57.1 25 21 SharedFSAutoReplicaFailoverTest.test
0123 4.4 741 33 TestContainerPlugin.testApiFromPackage
0123 2.3 732 13 TestInPlaceUpdatesDistrib.test
********************************************
BadApple report [ In reply to ]
Short form:

Processing file (History bit 3): HOSS-2020-07-27.csv
Processing file (History bit 2): HOSS-2020-07-20.csv
Processing file (History bit 1): HOSS-2020-07-13.csv
Processing file (History bit 0): HOSS-2020-07-06.csv


Number of AwaitsFix: 33 Number of BadApples: 4


**Annotated tests that didn't fail in the last 4 weeks.

**Tests removed from the next two lists because they were specified in 'doNotEnable' in the properties file
MoveReplicaHDFSTest.testNormalFailedMove

**Annotations can be removed from the following tests because they haven't failed in the last 4 rollups.

**Methods: 0


Raw fail count by week totals, most recent week first (corresponds to bits):
Week: 0 had 94 failures
Week: 1 had 502 failures
Week: 2 had 19 failures
Week: 3 had 24 failures


********Failures in Hoss' reports for the last 4 rollups.

There were 553 unannotated tests that failed in Hoss' rollups. Ordered by the date I downloaded the rollup file, newest->oldest. See above for the dates the files were collected
These tests were NOT BadApple'd or AwaitsFix'd

Failures in the last 4 reports..
Report Pct runs fails test
0123 93.3 30 26 SharedFSAutoReplicaFailoverTest.test
0123 6.0 1141 59 TestContainerPlugin.testApiFromPackage
0123 1.6 1000 17 TestInPlaceUpdatesDistrib.test
********************************************


Full results attached: