Mailing List Archive

Badapple report
We’re backsliding some. I encourage people to look at: http://fucit.org/solr-jenkins-reports/failure-report.html, we have a number of ill-behaved tests, particularly TestRequestRateLimiter, TestBulkSchemaConcurrent, TestConfig, SchemaApiFailureTest and TestIndexingSequenceNumbers…


Raw fail count by week totals, most recent week first (corresponds to bits):
Week: 0 had 100 failures
Week: 1 had 82 failures
Week: 2 had 94 failures
Week: 3 had 502 failures


********Failures in Hoss' reports for the last 4 rollups.

There were 585 unannotated tests that failed in Hoss' rollups. Ordered by the date I downloaded the rollup file, newest->oldest. See above for the dates the files were collected
These tests were NOT BadApple'd or AwaitsFix'd

Failures in the last 4 reports..
Report Pct runs fails test
0123 4.4 1583 37 BasicDistributedZkTest.test
0123 4.3 1727 77 CloudExitableDirectoryReaderTest.test
0123 2.5 8598 248 CloudExitableDirectoryReaderTest.testCreepThenBite
0123 1.9 1712 36 CloudExitableDirectoryReaderTest.testWhitebox
0123 0.5 1587 11 DocValuesNotIndexedTest.testGroupingDVOnlySortLast
0123 2.2 1679 82 HttpPartitionOnCommitTest.test
0123 0.5 1592 16 HttpPartitionTest.test
0123 1.0 1578 9 HttpPartitionWithTlogReplicasTest.test
0123 1.3 1569 13 LeaderFailoverAfterPartitionTest.test
0123 7.4 1643 59 MultiThreadedOCPTest.test
0123 0.3 1567 8 ReplaceNodeTest.test
0123 0.2 1588 6 ShardSplitTest.testSplitShardWithRule
0123 100.0 38 33 SharedFSAutoReplicaFailoverTest.test
0123 2.1 818 19 TestCircuitBreaker.testBuildingMemoryPressure
0123 2.6 818 13 TestCircuitBreaker.testResponseWithCBTiming
0123 6.2 1848 104 TestContainerPlugin.testApiFromPackage
0123 2.5 1662 33 TestDistributedGrouping.test
0123 0.4 1448 6 TestDynamicLoading.testDynamicLoading
0123 6.4 1614 74 TestExportWriter.testExpr
0123 8.6 1356 70 TestHdfsCloudBackupRestore.test
0123 9.1 1697 136 TestLocalFSCloudBackupRestore.test
0123 0.5 1607 26 TestPackages.testPluginLoading
0123 0.7 1596 15 TestQueryingOnDownCollection.testQueryToDownCollectionShouldFailFast
0123 1.5 1610 59 TestReRankQParserPlugin.testMinExactCount
0123 0.3 1552 4 TestReplicaProperties.test
0123 0.3 1556 5 TestSolrCloudWithDelegationTokens.testDelegationTokenRenew
0123 0.3 1565 9 TestSolrConfigHandlerCloud.test
********************************************
Re: Badapple report [ In reply to ]
Hi Erick,
I've introduced and later fixed a bug in TestConfig. It hasn't failed
since, so please don't annotate it.

On Mon, Aug 10, 2020 at 7:47 AM Erick Erickson <erickerickson@gmail.com>
wrote:

> We’re backsliding some. I encourage people to look at:
> http://fucit.org/solr-jenkins-reports/failure-report.html, we have a
> number of ill-behaved tests, particularly TestRequestRateLimiter,
> TestBulkSchemaConcurrent, TestConfig, SchemaApiFailureTest and
> TestIndexingSequenceNumbers…
>
>
> Raw fail count by week totals, most recent week first (corresponds to
> bits):
> Week: 0 had 100 failures
> Week: 1 had 82 failures
> Week: 2 had 94 failures
> Week: 3 had 502 failures
>
>
> ********Failures in Hoss' reports for the last 4 rollups.
>
> There were 585 unannotated tests that failed in Hoss' rollups. Ordered by
> the date I downloaded the rollup file, newest->oldest. See above for the
> dates the files were collected
> These tests were NOT BadApple'd or AwaitsFix'd
>
> Failures in the last 4 reports..
> Report Pct runs fails test
> 0123 4.4 1583 37 BasicDistributedZkTest.test
> 0123 4.3 1727 77 CloudExitableDirectoryReaderTest.test
> 0123 2.5 8598 248
> CloudExitableDirectoryReaderTest.testCreepThenBite
> 0123 1.9 1712 36
> CloudExitableDirectoryReaderTest.testWhitebox
> 0123 0.5 1587 11
> DocValuesNotIndexedTest.testGroupingDVOnlySortLast
> 0123 2.2 1679 82 HttpPartitionOnCommitTest.test
> 0123 0.5 1592 16 HttpPartitionTest.test
> 0123 1.0 1578 9 HttpPartitionWithTlogReplicasTest.test
> 0123 1.3 1569 13 LeaderFailoverAfterPartitionTest.test
> 0123 7.4 1643 59 MultiThreadedOCPTest.test
> 0123 0.3 1567 8 ReplaceNodeTest.test
> 0123 0.2 1588 6 ShardSplitTest.testSplitShardWithRule
> 0123 100.0 38 33 SharedFSAutoReplicaFailoverTest.test
> 0123 2.1 818 19
> TestCircuitBreaker.testBuildingMemoryPressure
> 0123 2.6 818 13
> TestCircuitBreaker.testResponseWithCBTiming
> 0123 6.2 1848 104 TestContainerPlugin.testApiFromPackage
> 0123 2.5 1662 33 TestDistributedGrouping.test
> 0123 0.4 1448 6 TestDynamicLoading.testDynamicLoading
> 0123 6.4 1614 74 TestExportWriter.testExpr
> 0123 8.6 1356 70 TestHdfsCloudBackupRestore.test
> 0123 9.1 1697 136 TestLocalFSCloudBackupRestore.test
> 0123 0.5 1607 26 TestPackages.testPluginLoading
> 0123 0.7 1596 15
> TestQueryingOnDownCollection.testQueryToDownCollectionShouldFailFast
> 0123 1.5 1610 59
> TestReRankQParserPlugin.testMinExactCount
> 0123 0.3 1552 4 TestReplicaProperties.test
> 0123 0.3 1556 5
> TestSolrCloudWithDelegationTokens.testDelegationTokenRenew
> 0123 0.3 1565 9 TestSolrConfigHandlerCloud.test
> ********************************************
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
> For additional commands, e-mail: dev-help@lucene.apache.org
Re: Badapple report [ In reply to ]
I investigated testRequestRateLimiters and hardened the tests up:

https://github.com/apache/lucene-solr/pull/1736

This will stop testConcurrentRequests from failing and should
hopefully stop testSlotBorrowing as well. If testSlotBorrowing
continues to fail, I will have to rethink the test.

On Mon, Aug 10, 2020 at 8:17 PM Erick Erickson <erickerickson@gmail.com> wrote:
>
> We’re backsliding some. I encourage people to look at: http://fucit.org/solr-jenkins-reports/failure-report.html, we have a number of ill-behaved tests, particularly TestRequestRateLimiter, TestBulkSchemaConcurrent, TestConfig, SchemaApiFailureTest and TestIndexingSequenceNumbers…
>
>
> Raw fail count by week totals, most recent week first (corresponds to bits):
> Week: 0 had 100 failures
> Week: 1 had 82 failures
> Week: 2 had 94 failures
> Week: 3 had 502 failures
>
>
> ********Failures in Hoss' reports for the last 4 rollups.
>
> There were 585 unannotated tests that failed in Hoss' rollups. Ordered by the date I downloaded the rollup file, newest->oldest. See above for the dates the files were collected
> These tests were NOT BadApple'd or AwaitsFix'd
>
> Failures in the last 4 reports..
> Report Pct runs fails test
> 0123 4.4 1583 37 BasicDistributedZkTest.test
> 0123 4.3 1727 77 CloudExitableDirectoryReaderTest.test
> 0123 2.5 8598 248 CloudExitableDirectoryReaderTest.testCreepThenBite
> 0123 1.9 1712 36 CloudExitableDirectoryReaderTest.testWhitebox
> 0123 0.5 1587 11 DocValuesNotIndexedTest.testGroupingDVOnlySortLast
> 0123 2.2 1679 82 HttpPartitionOnCommitTest.test
> 0123 0.5 1592 16 HttpPartitionTest.test
> 0123 1.0 1578 9 HttpPartitionWithTlogReplicasTest.test
> 0123 1.3 1569 13 LeaderFailoverAfterPartitionTest.test
> 0123 7.4 1643 59 MultiThreadedOCPTest.test
> 0123 0.3 1567 8 ReplaceNodeTest.test
> 0123 0.2 1588 6 ShardSplitTest.testSplitShardWithRule
> 0123 100.0 38 33 SharedFSAutoReplicaFailoverTest.test
> 0123 2.1 818 19 TestCircuitBreaker.testBuildingMemoryPressure
> 0123 2.6 818 13 TestCircuitBreaker.testResponseWithCBTiming
> 0123 6.2 1848 104 TestContainerPlugin.testApiFromPackage
> 0123 2.5 1662 33 TestDistributedGrouping.test
> 0123 0.4 1448 6 TestDynamicLoading.testDynamicLoading
> 0123 6.4 1614 74 TestExportWriter.testExpr
> 0123 8.6 1356 70 TestHdfsCloudBackupRestore.test
> 0123 9.1 1697 136 TestLocalFSCloudBackupRestore.test
> 0123 0.5 1607 26 TestPackages.testPluginLoading
> 0123 0.7 1596 15 TestQueryingOnDownCollection.testQueryToDownCollectionShouldFailFast
> 0123 1.5 1610 59 TestReRankQParserPlugin.testMinExactCount
> 0123 0.3 1552 4 TestReplicaProperties.test
> 0123 0.3 1556 5 TestSolrCloudWithDelegationTokens.testDelegationTokenRenew
> 0123 0.3 1565 9 TestSolrConfigHandlerCloud.test
> ********************************************
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
> For additional commands, e-mail: dev-help@lucene.apache.org

--
Regards,

Atri
Apache Concerted

---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: dev-help@lucene.apache.org
BadApple report [ In reply to ]
********Failures in Hoss' reports for the last 4 rollups.

There were 242 unannotated tests that failed in Hoss' rollups. Ordered by the date I downloaded the rollup file, newest->oldest. See above for the dates the files were collected
These tests were NOT BadApple'd or AwaitsFix'd

Failures in the last 4 reports..
Report Pct runs fails test
0123 6.4 1757 94 CloudExitableDirectoryReaderTest.test
0123 5.3 8740 325 CloudExitableDirectoryReaderTest.testCreepThenBite
0123 2.6 1734 42 CloudExitableDirectoryReaderTest.testWhitebox
0123 9.5 1688 107 HttpPartitionOnCommitTest.test
0123 2.7 1604 18 HttpPartitionTest.test
0123 1.8 1580 14 HttpPartitionWithTlogReplicasTest.test
0123 0.3 1567 10 LeaderFailoverAfterPartitionTest.test
0123 3.6 1639 57 MultiThreadedOCPTest.test
0123 0.3 1564 5 ReplaceNodeTest.test
0123 0.3 1584 4 ShardSplitTest.testSplitShardWithRule
0123 93.3 46 43 SharedFSAutoReplicaFailoverTest.test
0123 2.3 837 18 TestCircuitBreaker.testBuildingMemoryPressure
0123 0.9 837 12 TestCircuitBreaker.testResponseWithCBTiming
0123 3.6 1853 101 TestContainerPlugin.testApiFromPackage
0123 2.8 1683 37 TestDistributedGrouping.test
0123 4.2 1629 89 TestExportWriter.testExpr
0123 11.7 1326 87 TestHdfsCloudBackupRestore.test
0123 9.3 1672 121 TestLocalFSCloudBackupRestore.test
0123 1.2 1623 25 TestPackages.testPluginLoading
0123 0.3 1586 9 TestQueryingOnDownCollection.testQueryToDownCollectionShouldFailFast
0123 8.3 1629 82 TestReRankQParserPlugin.testMinExactCount
0123 0.3 1556 4 TestReplicaProperties.test
0123 0.3 1557 5 TestSolrCloudWithDelegationTokens.testDelegationTokenRenew
0123 1.5 1564 10 TestSolrConfigHandlerCloud.test
********************************************


Full report attached:
BadApple report [ In reply to ]
We have some pretty frequent failures, see:

http://fucit.org/solr-jenkins-reports/failure-report.html

I’m pretty sure LBSolrClientTest has been addressed. I’m looking at what commit caused TestConfigOverlay to start failing…

This can be a little hard to interpret since it includes tests that have been fixed over the last week, not to mention that many of them are intermittent.

The raw count of SupressAnnotations hasn’t changed, one was removed and one added.

Raw fail count by week totals, most recent week first (corresponds to bits):
Week: 0 had 119 failures
Week: 1 had 113 failures
Week: 2 had 100 failures
Week: 3 had 82 failures


********Failures in Hoss' reports in every one of the last 4 rollups.

There were 257 unannotated tests that failed in Hoss' rollups. Ordered by the date I downloaded the rollup file, newest->oldest. See above for the dates the files were collected
These tests were NOT BadApple'd or AwaitsFix'd

Failures in the last 4 reports..
Report Pct runs fails test
0123 3.2 1719 86 CloudExitableDirectoryReaderTest.test
0123 1.8 8552 297 CloudExitableDirectoryReaderTest.testCreepThenBite
0123 1.9 1700 41 CloudExitableDirectoryReaderTest.testWhitebox
0123 9.8 1687 125 HttpPartitionOnCommitTest.test
0123 0.6 1571 19 HttpPartitionTest.test
0123 3.5 1565 25 HttpPartitionWithTlogReplicasTest.test
0123 0.3 1604 54 MultiThreadedOCPTest.test
0123 2.0 825 8 SearchRateTriggerTest.testWaitForElapsed
0123 0.3 1556 4 ShardSplitTest.testSplitShardWithRule
0123 3.2 839 16 TestCircuitBreaker.testResponseWithCBTiming
0123 6.2 1824 100 TestContainerPlugin.testApiFromPackage
0123 2.3 1677 42 TestDistributedGrouping.test
0123 3.4 1590 88 TestExportWriter.testExpr
0123 6.8 1302 96 TestHdfsCloudBackupRestore.test
0123 6.8 1646 128 TestLocalFSCloudBackupRestore.test
0123 0.6 1591 21 TestPackages.testPluginLoading
0123 0.6 1550 9 TestQueryingOnDownCollection.testQueryToDownCollectionShouldFailFast
0123 1.7 1538 9 TestReplicaProperties.test
0123 0.3 1524 5 TestSolrCloudWithDelegationTokens.testDelegationTokenRenew
0123 0.6 1534 10 TestSolrConfigHandlerCloud.test
********************************************


Full output:
RE: BadApple report [ In reply to ]
Hi Erick,

The teste-only jobs @ ASF and Policeman Jenkins jobs of master branch were all converted to Gradle. It should have no effect on the Hossman Badapples analysis, but maybe have an extra look next week to find outlyers. The statistics about failed jobs in the XML output should be the same.

Uwe

-----
Uwe Schindler
Achterdiek 19, D-28357 Bremen
https://www.thetaphi.de
eMail: uwe@thetaphi.de

> -----Original Message-----
> From: Erick Erickson <erickerickson@gmail.com>
> Sent: Monday, August 24, 2020 3:59 PM
> To: dev@lucene.apache.org
> Subject: BadApple report
>
> We have some pretty frequent failures, see:
>
> http://fucit.org/solr-jenkins-reports/failure-report.html
>
> I’m pretty sure LBSolrClientTest has been addressed. I’m looking at what
> commit caused TestConfigOverlay to start failing…
>
> This can be a little hard to interpret since it includes tests that have been fixed
> over the last week, not to mention that many of them are intermittent.
>
> The raw count of SupressAnnotations hasn’t changed, one was removed and
> one added.
>
> Raw fail count by week totals, most recent week first (corresponds to bits):
> Week: 0 had 119 failures
> Week: 1 had 113 failures
> Week: 2 had 100 failures
> Week: 3 had 82 failures
>
>
> ********Failures in Hoss' reports in every one of the last 4 rollups.
>
> There were 257 unannotated tests that failed in Hoss' rollups. Ordered by the
> date I downloaded the rollup file, newest->oldest. See above for the dates the
> files were collected
> These tests were NOT BadApple'd or AwaitsFix'd
>
> Failures in the last 4 reports..
> Report Pct runs fails test
> 0123 3.2 1719 86 CloudExitableDirectoryReaderTest.test
> 0123 1.8 8552 297
> CloudExitableDirectoryReaderTest.testCreepThenBite
> 0123 1.9 1700 41 CloudExitableDirectoryReaderTest.testWhitebox
> 0123 9.8 1687 125 HttpPartitionOnCommitTest.test
> 0123 0.6 1571 19 HttpPartitionTest.test
> 0123 3.5 1565 25 HttpPartitionWithTlogReplicasTest.test
> 0123 0.3 1604 54 MultiThreadedOCPTest.test
> 0123 2.0 825 8 SearchRateTriggerTest.testWaitForElapsed
> 0123 0.3 1556 4 ShardSplitTest.testSplitShardWithRule
> 0123 3.2 839 16 TestCircuitBreaker.testResponseWithCBTiming
> 0123 6.2 1824 100 TestContainerPlugin.testApiFromPackage
> 0123 2.3 1677 42 TestDistributedGrouping.test
> 0123 3.4 1590 88 TestExportWriter.testExpr
> 0123 6.8 1302 96 TestHdfsCloudBackupRestore.test
> 0123 6.8 1646 128 TestLocalFSCloudBackupRestore.test
> 0123 0.6 1591 21 TestPackages.testPluginLoading
> 0123 0.6 1550 9
> TestQueryingOnDownCollection.testQueryToDownCollectionShouldFailFast
> 0123 1.7 1538 9 TestReplicaProperties.test
> 0123 0.3 1524 5
> TestSolrCloudWithDelegationTokens.testDelegationTokenRenew
> 0123 0.6 1534 10 TestSolrConfigHandlerCloud.test
> ********************************************
>
>
> Full output:


---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: dev-help@lucene.apache.org
BadApple report [ In reply to ]
You can probably ignore this week’s report, on the surface it’s pretty horrible but I suspect that just reflects some timing issues when code was moved and/or we switched to Gradle for trunk builds.

We has 2,627 tests fail over the last 7 days, with 150+ failing 100% of the time. However, only 10 failed in the last 24 hours with 1 failing 100% of the time which only ran once. And even that test works for me.

For comparison, 119 tests failed the previous 7 days, so you can see why when I saw Hoss’ rollup my eyes got big ;)

It’s not too surprising to have some temporary glitches like this, just growing pains. This’ll warp the reports for the next 4 weeks until it ages out.

Full report attached
Erick
BadApple report [ In reply to ]
We’re in kind of a weird state. In any given week, due to a _lot_ of things changing the results can look really awful due to a lot of tests failing when something screwey happens.

To reduce noise, though, I’d encourage people to be diligent about running “gradlew check” every time before checking in code changes. The actual tests that have failed each week for the last 4 is actually quite small…..

Raw fail count by week totals, most recent week first (corresponds to bits):
Week: 0 had 94 failures
Week: 1 had 2627 failures
Week: 2 had 119 failures
Week: 3 had 113 failures


********Failures in Hoss' reports in every one of the last 4 rollups.

There were 2760 unannotated tests that failed in Hoss' rollups. Ordered by the date I downloaded the rollup file, newest->oldest. See above for the dates the files were collected
These tests were NOT BadApple'd or AwaitsFix'd

Failures in the last 4 reports..
Report Pct runs fails test
0123 2.1 1643 115 HttpPartitionOnCommitTest.test
0123 0.5 1537 19 HttpPartitionTest.test
0123 0.7 1552 29 HttpPartitionWithTlogReplicasTest.test
0123 1.9 1543 31 MultiThreadedOCPTest.test
0123 0.3 1006 5 OverseerTest.testShardLeaderChange
0123 0.2 1498 4 StatsReloadRaceTest.testParallelReloadAndStats
0123 4.2 1684 78 TestContainerPlugin.testApiFromPackage
0123 6.4 1555 67 TestExportWriter.testExpr
0123 1.4 1260 67 TestHdfsCloudBackupRestore.test
0123 0.0 9300 10 TestJsonFacets.testBigger
0123 0.0 9300 10 TestJsonFacets.testStatsDistrib
0123 1.2 1560 75 TestLocalFSCloudBackupRestore.test
0123 0.2 1521 11 TestReplicaProperties.test
********************************************

Full report:
BadApple report [ In reply to ]
Raw fail count by week totals, most recent week first (corresponds to bits):
Week: 0 had 94 failures
Week: 1 had 82 failures
Week: 2 had 2627 failures
Week: 3 had 119 failures


********Failures in Hoss' reports in every one of the last 4 rollups.

There were 2757 unannotated tests that failed in Hoss' rollups. Ordered by the date I downloaded the rollup file, newest->oldest. See above for the dates the files were collected
These tests were NOT BadApple'd or AwaitsFix'd

Failures in the last 4 reports..
Report Pct runs fails test
0123 0.2 1593 4 ChaosMonkeySafeLeaderWithPullReplicasTest.test
0123 2.1 1717 118 HttpPartitionOnCommitTest.test
0123 0.5 1593 12 HttpPartitionTest.test
0123 0.7 1638 25 HttpPartitionWithTlogReplicasTest.test
0123 1.9 1602 24 MultiThreadedOCPTest.test
0123 0.3 1105 6 OverseerTest.testShardLeaderChange
0123 0.4 1607 7 SolrCloudExampleTest.testLoadDocsIntoGettingStartedCollection
0123 4.2 1738 87 TestContainerPlugin.testApiFromPackage
0123 6.4 1695 110 TestExportWriter.testExpr
0123 1.4 1309 31 TestHdfsCloudBackupRestore.test
0123 1.2 1604 39 TestLocalFSCloudBackupRestore.test
0123 0.2 1603 16 TestReplicaProperties.test
0123 0.2 1588 4 UnloadDistributedZkTest.test
********************************************

Full report:
BadApple report [ In reply to ]
The 2,722 unannotated tests is scary, but not really concerning, several weeks ago we had a bad week. Next week that should fall off the end of the 4 week window and we’ll be back to normal.

Raw fail count by week totals, most recent week first (corresponds to bits):
Week: 0 had 94 failures
Week: 1 had 51 failures
Week: 2 had 82 failures
Week: 3 had 2627 failures


********Failures in Hoss' reports in every one of the last 4 rollups.

There were 2722 unannotated tests that failed in Hoss' rollups. Ordered by the date I downloaded the rollup file, newest->oldest. See above for the dates the files were collected
These tests were NOT BadApple'd or AwaitsFix'd

Failures in the last 4 reports..
Report Pct runs fails test
0123 2.1 1673 80 HttpPartitionOnCommitTest.test
0123 0.7 1638 13 HttpPartitionWithTlogReplicasTest.test
0123 1.9 1642 29 MultiThreadedOCPTest.test
0123 0.3 1138 6 OverseerTest.testShardLeaderChange
0123 100.0 7 4 SharedFSAutoReplicaFailoverTest.test
0123 4.2 1684 64 TestContainerPlugin.testApiFromPackage
0123 6.4 1709 99 TestExportWriter.testExpr
0123 1.4 1334 16 TestHdfsCloudBackupRestore.test
0123 1.2 1605 20 TestLocalFSCloudBackupRestore.test
********************************************


Full report:
BadApple report [ In reply to ]
Mostly for historical context for a while, It includes the reference impl so the stats will be skewed from now until we integrate it all.

Short form:
Raw fail count by week totals, most recent week first (corresponds to bits):
Week: 0 had 142 failures
Week: 1 had 153 failures
Week: 2 had 51 failures
Week: 3 had 82 failures


********Failures in Hoss' reports in every one of the last 4 rollups.

There were 301 unannotated tests that failed in Hoss' rollups. Ordered by the date I downloaded the rollup file, newest->oldest. See above for the dates the files were collected
These tests were NOT BadApple'd or AwaitsFix'd

Failures in the last 4 reports..
Report Pct runs fails test
0123 4.8 1808 93 HttpPartitionOnCommitTest.test
0123 0.5 1748 11 HttpPartitionWithTlogReplicasTest.test
0123 5.2 1789 51 MultiThreadedOCPTest.test
0123 50.0 8 4 SharedFSAutoReplicaFailoverTest.test
0123 1.5 1829 102 TestExportWriter.testExpr
0123 0.3 1435 15 TestHdfsCloudBackupRestore.test
0123 1.0 1716 9 TestInPlaceUpdatesDistrib.test
0123 0.2 1721 16 TestLocalFSCloudBackupRestore.test
0123 1.0 1731 12 TestSolrConfigHandlerCloud.test
********************************************

Failures over the last 4 weeks, but not every week. Ordered most-recent first:


Full report:
BadApple report [ In reply to ]
The BadApple report remains skewed as the results include the reference impl so this is mostly in case people are curious….

I expect next week to see an uptick in the number of tests that have failed each of the last 4 weeks, that’ll be when the reference-impl parts of the report kick in. We’ll see how things progress after that.

There were 354 unannotated tests that failed in Hoss' rollups. Ordered by the date I downloaded the rollup file, newest->oldest. See above for the dates the files were collected
These tests were NOT BadApple'd or AwaitsFix'd

Failures in the last 4 reports..
Report Pct runs fails test
0123 1.4 1662 54 HttpPartitionOnCommitTest.test
0123 4.8 1624 25 HttpPartitionWithTlogReplicasTest.test
0123 0.3 1608 4 LBSolrClientTest.testServerIteratorTimeAllowed
0123 2.7 1684 53 MultiThreadedOCPTest.test
0123 50.0 8 4 SharedFSAutoReplicaFailoverTest.test
0123 5.0 1350 27 TestHdfsCloudBackupRestore.test
0123 4.8 1604 29 TestLocalFSCloudBackupRestore.test
0123 0.6 1610 10 TestSolrConfigHandlerCloud.test
********************************************


Full results:
BadApple report [ In reply to ]
Still working through the failures on the reference impl, so AFAIK, the tests failing large percentages of the time are on that branch.

Processing file (History bit 3): HOSS-2020-10-26.csv
Processing file (History bit 2): HOSS-2020-10-19.csv
Processing file (History bit 1): HOSS-2020-10-12.csv
Processing file (History bit 0): HOSS-2020-10-05.csv


Number of AwaitsFix: 31 Number of BadApples: 3


**Annotated tests that didn't fail in the last 4 weeks.

**Tests removed from the next two lists because they were specified in 'doNotEnable' in the properties file
MoveReplicaHDFSTest.testNormalFailedMove

**Annotations can be removed from the following tests because they haven't failed in the last 4 rollups.

**Methods: 0


Raw fail count by week totals, most recent week first (corresponds to bits):
Week: 0 had 150 failures
Week: 1 had 174 failures
Week: 2 had 142 failures
Week: 3 had 153 failures


********Failures in Hoss' reports in every one of the last 4 rollups.

There were 397 unannotated tests that failed in Hoss' rollups. Ordered by the date I downloaded the rollup file, newest->oldest. See above for the dates the files were collected
These tests were NOT BadApple'd or AwaitsFix'd

Failures in the last 4 reports..
Report Pct runs fails test
0123 100.0 35 35 AssignTest.classMethod
0123 100.0 255 255 AsyncCallRequestStatusResponseTest.classMethod
0123 0.8 1916 13 CachingDirectoryFactoryTest.stressTest
0123 100.0 155 155 CollectionsAPIDistributedZkTest.classMethod
0123 3.8 1960 51 CollectionsAPIDistributedZkTest.testBadActionNames
0123 3.8 1960 51 CollectionsAPIDistributedZkTest.testMissingNumShards
0123 3.8 1960 51 CollectionsAPIDistributedZkTest.testMissingRequiredParameters
0123 3.8 1960 51 CollectionsAPIDistributedZkTest.testNoConfigSetExist
0123 3.8 1960 51 CollectionsAPIDistributedZkTest.testZeroNumShards
0123 100.0 205 205 CollectionsAPISolrJTest.classMethod
0123 100.0 250 250 ConcurrentUpdateSolrClientMultiCollectionTest.classMethod
0123 100.0 205 205 DeleteNodeTest.classMethod
0123 1.8 1565 56 HttpPartitionOnCommitTest.test
0123 1.1 1546 33 HttpPartitionTest.test
0123 100.0 250 250 JsonRequestApiHeatmapFacetingTest.classMethod
0123 100.0 250 250 JsonRequestApiTest.classMethod
0123 2.2 1576 53 MultiThreadedOCPTest.test
0123 100.0 205 205 OverseerModifyCollectionTest.classMethod
0123 1.7 988 11 TestCircuitBreaker.testResponseWithCBTiming
0123 0.7 1521 6 TestCustomStream.testDynamicLoadingCustomStream
0123 1.3 1256 25 TestHdfsCloudBackupRestore.test
0123 1.1 1509 26 TestLocalFSCloudBackupRestore.test
0123 1.4 1513 25 TestPackages.testPluginLoading
0123 13.7 1982 233 TestSTUniformSplitPostingFormat.testCheckIntegrityReadsAllBytes
0123 26.0 1981 451 TestSynonymFilterFactory.testFormat
0123 26.0 1981 451 TestSynonymFilterFactory.testSynonyms
0123 26.1 1983 452 TestSysoutsLimits.OverHardLimit
0123 26.1 1983 452 TestSysoutsLimits.testOverSoftLimit
0123 0.4 1519 6 TestSystemCollAutoCreate.testAutoCreate
0123 13.7 1982 233 TestUniformSplitPostingFormat.testCheckIntegrityReadsAllBytes
0123 100.0 250 250 UsingSolrJRefGuideExamplesTest.classMethod
0123 100.0 250 250 ZkConfigFilesTest.classMethod
********************************************
BadApple report [ In reply to ]
Not much change this week, still getting considerable noise from the reference impl.

Raw fail count by week totals, most recent week first (corresponds to bits):
Week: 0 had 110 failures
Week: 1 had 150 failures
Week: 2 had 174 failures
Week: 3 had 142 failures


********Failures in Hoss' reports in every one of the last 4 rollups.

There were 368 unannotated tests that failed in Hoss' rollups. Ordered by the date I downloaded the rollup file, newest->oldest. See above for the dates the files were collected
These tests were NOT BadApple'd or AwaitsFix'd

Failures in the last 4 reports..
Report Pct runs fails test
0123 100.0 265 265 AsyncCallRequestStatusResponseTest.classMethod
0123 0.6 1765 14 CachingDirectoryFactoryTest.stressTest
0123 100.0 185 185 CollectionsAPIDistributedZkTest.classMethod
0123 3.3 1826 61 CollectionsAPIDistributedZkTest.testBadActionNames
0123 3.3 1826 61 CollectionsAPIDistributedZkTest.testMissingNumShards
0123 3.3 1826 61 CollectionsAPIDistributedZkTest.testMissingRequiredParameters
0123 3.3 1826 61 CollectionsAPIDistributedZkTest.testNoConfigSetExist
0123 3.3 1826 61 CollectionsAPIDistributedZkTest.testZeroNumShards
0123 100.0 300 300 ConcurrentUpdateSolrClientMultiCollectionTest.classMethod
0123 100.0 300 300 JsonRequestApiHeatmapFacetingTest.classMethod
0123 100.0 300 300 JsonRequestApiTest.classMethod
0123 0.4 1330 20 ShardSplitTest.testSplitMixedReplicaTypesLink
0123 3.0 1043 19 TestCircuitBreaker.testResponseWithCBTiming
0123 0.9 1076 21 TestHdfsCloudBackupRestore.test
0123 0.8 1295 23 TestLocalFSCloudBackupRestore.test
0123 1.1 1327 27 TestPackages.testPluginLoading
0123 1.0 1338 11 TestPullReplicaErrorHandling.testCantConnectToLeader
0123 2.6 1338 17 TestPullReplicaErrorHandling.testPullReplicaDisconnectsFromZooKeeper
0123 14.4 1844 276 TestSTUniformSplitPostingFormat.testCheckIntegrityReadsAllBytes
0123 26.6 1842 530 TestSynonymFilterFactory.testFormat
0123 26.6 1842 530 TestSynonymFilterFactory.testSynonyms
0123 26.6 1844 531 TestSysoutsLimits.OverHardLimit
0123 26.6 1844 531 TestSysoutsLimits.testOverSoftLimit
0123 14.4 1844 276 TestUniformSplitPostingFormat.testCheckIntegrityReadsAllBytes
0123 100.0 300 300 UsingSolrJRefGuideExamplesTest.classMethod
0123 100.0 300 300 ZkConfigFilesTest.classMethod
********************************************
BadApple report [ In reply to ]
Still seeing quite a bit of noise due to the reference impl. That said, we do have a reproducible error for TestRandomDVFaceting both 8x and master, see SOLR-14990.

Meanwhile, here’s the report for this week.

Raw fail count by week totals, most recent week first (corresponds to bits):
Week: 0 had 112 failures
Week: 1 had 110 failures
Week: 2 had 150 failures
Week: 3 had 174 failures


********Failures in Hoss' reports in every one of the last 4 rollups.

There were 342 unannotated tests that failed in Hoss' rollups. Ordered by the date I downloaded the rollup file, newest->oldest. See above for the dates the files were collected
These tests were NOT BadApple'd or AwaitsFix'd

Failures in the last 4 reports..
Report Pct runs fails test
0123 0.4 1656 10 CachingDirectoryFactoryTest.stressTest
0123 100.0 159 159 CollectionsAPIDistributedZkTest.classMethod
0123 2.1 1709 53 CollectionsAPIDistributedZkTest.testBadActionNames
0123 2.1 1709 53 CollectionsAPIDistributedZkTest.testMissingNumShards
0123 2.1 1709 53 CollectionsAPIDistributedZkTest.testMissingRequiredParameters
0123 2.1 1709 53 CollectionsAPIDistributedZkTest.testNoConfigSetExist
0123 2.1 1709 53 CollectionsAPIDistributedZkTest.testZeroNumShards
0123 100.0 260 260 ConcurrentUpdateSolrClientMultiCollectionTest.classMethod
0123 100.0 260 260 JsonRequestApiHeatmapFacetingTest.classMethod
0123 100.0 260 260 JsonRequestApiTest.classMethod
0123 0.6 1510 6 ManagedSchemaRoundRobinCloudTest.testAddFieldsRoundRobin
0123 2.4 1504 29 MoveReplicaTest.test
0123 0.3 1185 19 TestCircuitBreaker.testResponseWithCBTiming
0123 0.7 1024 22 TestHdfsCloudBackupRestore.test
0123 0.9 1232 25 TestLocalFSCloudBackupRestore.test
0123 0.6 1259 28 TestPackages.testPluginLoading
0123 1.3 1409 16 TestPullReplicaErrorHandling.testCantConnectToLeader
0123 2.1 1409 23 TestPullReplicaErrorHandling.testPullReplicaDisconnectsFromZooKeeper
0123 0.7 1506 10 TestQueryingOnDownCollection.testQueryToDownCollectionShouldFailFast
0123 13.1 1726 246 TestSTUniformSplitPostingFormat.testCheckIntegrityReadsAllBytes
0123 28.7 1723 482 TestSynonymFilterFactory.testFormat
0123 28.7 1723 482 TestSynonymFilterFactory.testSynonyms
0123 28.6 1726 483 TestSysoutsLimits.OverHardLimit
0123 28.6 1726 483 TestSysoutsLimits.testOverSoftLimit
0123 13.1 1726 246 TestUniformSplitPostingFormat.testCheckIntegrityReadsAllBytes
0123 100.0 260 260 UsingSolrJRefGuideExamplesTest.classMethod
0123 100.0 260 260 ZkConfigFilesTest.classMethod
********************************************
BadApple report [ In reply to ]
Unfortunately, the reference impl is creating quite a bit of noise in Hoss’ rollups. That said, I have a mail filter for test failures that puts the reference impl tests in a different mail folder and my sense is that the regular branch is getting an increasing number of failures.

If I have the energy, I’ll try to collect some of them.


Raw fail count by week totals, most recent week first (corresponds to bits):
Week: 0 had 210 failures
Week: 1 had 112 failures
Week: 2 had 110 failures
Week: 3 had 150 failures


********Failures in Hoss' reports in every one of the last 4 rollups.

There were 390 unannotated tests that failed in Hoss' rollups. Ordered by the date I downloaded the rollup file, newest->oldest. See above for the dates the files were collected
These tests were NOT BadApple'd or AwaitsFix'd

Failures in the last 4 reports..
Report Pct runs fails test
0123 0.5 1745 10 CachingDirectoryFactoryTest.stressTest
0123 100.0 159 159 CollectionsAPIAsyncDistributedZkTest.classMethod
0123 2.9 1759 49 CollectionsAPIAsyncDistributedZkTest.testAsyncIdRaceCondition
0123 3.0 1744 50 CollectionsAPIDistributedZkTest.testDeleteNonExistentCollection
0123 3.0 1772 53 CollectionsAPIDistributedZkTest.testNoConfigSetExist
0123 100.0 209 209 JsonRequestApiHeatmapFacetingTest.classMethod
0123 100.0 209 209 JsonRequestApiTest.classMethod
0123 0.4 1708 7 ManagedSchemaRoundRobinCloudTest.testAddFieldsRoundRobin
0123 3.1 1737 32 MoveReplicaTest.test
0123 1.9 1366 23 TestCircuitBreaker.testResponseWithCBTiming
0123 8.5 1275 94 TestContainerPlugin.testApi
0123 2.5 1368 36 TestDistributedStatsComponentCardinality.test
0123 0.3 1069 8 TestHdfsCloudBackupRestore.test
0123 0.2 1277 9 TestLocalFSCloudBackupRestore.test
0123 1.9 1313 17 TestPackages.testPluginLoading
0123 1.7 1575 20 TestPullReplicaErrorHandling.testCantConnectToLeader
0123 1.9 1575 31 TestPullReplicaErrorHandling.testPullReplicaDisconnectsFromZooKeeper
0123 100.0 209 209 UsingSolrJRefGuideExamplesTest.classMethod
0123 100.0 192 192 ZkConfigFilesTest.classMethod
********************************************