Mailing List Archive

Hi,

I have a question about the score produced by TFIDFSimilarity.
https://lucene.apache.org/core/8_5_1/core/org/apache/lucene/search/similarities/TFIDFSimilarity.html
says
that the IDF factor should be squared in the final score. However, looking
at the code, I see this:

public TFIDFScorer(float boost, Explanation idf, float[] normTable) {

// TODO: Validate?

this.idf = idf;

this.boost = boost;

this.queryWeight = boost * idf.getValue().floatValue();

this.normTable = normTable;

}


@Override

public float score(float freq, long norm) {

final float raw = tf(freq) * queryWeight; // compute tf(f)*weight

float normValue = normTable[(int) (norm & 0xFF)];

return raw * normValue; // normalize for field

}


Where does the second idf.getValue() factor come from? In Lucene 6.6.6,
before the patch for https://issues.apache.org/jira/browse/LUCENE-7368 was
applied, the code looked like this:

TFIDFSimScorer(IDFStats stats, NumericDocValues norms) throws
IOException {

this.stats = stats;

this.weightValue = stats.value;

this.norms = norms;

}



@Override

public float score(int doc, float freq) {

final float raw = tf(freq) * weightValue; // compute tf(f)*weight



return norms == null ? raw : raw * decodeNormValue(norms.get(doc)); //
normalize for field

}


...


public IDFStats(String field, Explanation idf) {

// TODO: Validate?

this.field = field;

this.idf = idf;

normalize(1f, 1f);

}


@Override

public void normalize(float queryNorm, float boost) {

this.boost = boost;

this.queryNorm = queryNorm;

queryWeight = queryNorm * boost * idf.getValue();

value = queryWeight * idf.getValue(); // idf for document

}

Did we lose an idf.getValue() factor in this patch? Or was it moved
somewhere else? Could you please point me to the code location where the
score is multiplied by the IDF value a second time?

Thanks!
Dumitru