Mailing List Archive

Developers,
Attached is the diff for MultiSearcher which seems to correct these
bugs. I have not yet found any problems caused by these changes in
testing........but we will keep you informed!

Thanks,
Rasik Pandey

-----Message d'origine-----
De : Rasik Pandey [mailto:rasik.pandey@ajlsm.com]
Envoyé : jeudi 12 septembre 2002 19:01
À : lucene-dev@jakarta.apache.org
Objet : TR : Possible Bug with MultiSearcher?


Developers,
Ok this is the latest test program to reproduce the original problem
reported. When I submitted the first test program, I was unable to
reproduce the original, however this new test program reproduces both
the original error and the second error which I reported in my last
mail. I believe the problem lies in the second line of the
SegmentsReader.readerIndex(int n) method. It doesn't compensate for the
fact that if "readers.length" returns an int of value "0", the "hi"
variable will be assigned a value of "-1", which is then returned to any
calling methods!


Hope this is helpful,
Rasik Pandey

-----Message d'origine-----
De : Rasik Pandey [mailto:rasik.pandey@ajlsm.com]
Envoyé : lundi 9 septembre 2002 18:46
À : 'Lucene Developers List'
Objet : RE : Possible Bug with MultiSearcher?


Lucene Developers,
A test program is attached. I reiterate that the searcher for the empty
index must be added first. The exception thrown at the end of the
program is an ArrayIndexOutOfBoundsException. I will continue trying to
recreate the orginal exception which I reported, but I think this one
may be related. I was unable to recreate the exception using
RAMDirectory stores, but only two FSDirectory stores. It maybe necessary
to delete these directories from time to time???? Please let me know if
I can provide any more information.

Rasik Pandey


-----Message d'origine-----
De : Doug Cutting [mailto:cutting@lucene.com]
Envoyé : jeudi 5 septembre 2002 23:22
À : Lucene Developers List
Objet : Re: Possible Bug with MultiSearcher?


Can you please submit a complete, self-contained test program that
demonstrates the problem? That will make it much easier for someone to
debug and fix it.

Thanks,

Doug

Rasik Pandey wrote:
> Hello,
> I am getting the following exception when searching using a
> MultiSearcher and the first Searchable added is an empty IndexSearcher

> with a segments file, the second is a normal IndexSearcher from a
> populated Index . I am able to reproduce this without fail. The
> exception is thrown when trying to retrieve a document from the Hits.
> Is the exception really waranted, given there are hits in the second
> index, I don't think the search should fail completely? When searching

> an individual index, adding it to a Multisearcher, and performing the
> search causes no problems. Any help or suggestions would be
> appreciated.
>
>
> Please see org.apache.lucene.store.InputStream.java 192:46 for the
> location where the Exception is generated.
>
>
> Rasik Pandey
>
>
> java.io.IOException: read past EOF
> at org.apache.lucene.store.InputStream.refill(Unknown Source) at
> org.apache.lucene.store.InputStream.readByte(Unknown Source) at
> org.apache.lucene.store.InputStream.readInt(Unknown Source) at
> org.apache.lucene.store.InputStream.readLong(Unknown Source) at
> org.apache.lucene.index.FieldsReader.doc(Unknown Source) at
> org.apache.lucene.index.SegmentReader.document(Unknown Source) at
> org.apache.lucene.search.IndexSearcher.doc(Unknown Source) at
> org.apache.lucene.search.MultiSearcher.doc(Unknown Source) at
> org.apache.lucene.search.Hits.doc(Unknown Source)
>
>
>
>
>
> --
> To unsubscribe, e-mail:
<mailto:lucene-dev-unsubscribe@jakarta.apache.org>
> For additional commands, e-mail:
> <mailto:lucene-dev-help@jakarta.apache.org>
>


--
To unsubscribe, e-mail:
<mailto:lucene-dev-unsubscribe@jakarta.apache.org>
For additional commands, e-mail:
<mailto:lucene-dev-help@jakarta.apache.org>

If you could write a unit test class (see src/test/.... classes for
examples) that first demonstrates the problem, that would be great.
Then we can see that the code really is broken, apply your fix to
MultiSearcher, and see that the tests pass after you fix.
Then we can feel good about committing the patched MultieSearcher :)

Thanks,
Otis


--- Rasik Pandey <rasik.pandey@ajlsm.com> wrote:
> Developers,
> Attached is the diff for MultiSearcher which seems to correct these
> bugs. I have not yet found any problems caused by these changes in
> testing........but we will keep you informed!
>
> Thanks,
> Rasik Pandey
>
> -----Message d'origine-----
> De : Rasik Pandey [mailto:rasik.pandey@ajlsm.com]
> Envoyé : jeudi 12 septembre 2002 19:01
> À : lucene-dev@jakarta.apache.org
> Objet : TR : Possible Bug with MultiSearcher?
>
>
> Developers,
> Ok this is the latest test program to reproduce the original problem
> reported. When I submitted the first test program, I was unable to
> reproduce the original, however this new test program reproduces both
> the original error and the second error which I reported in my last
> mail. I believe the problem lies in the second line of the
> SegmentsReader.readerIndex(int n) method. It doesn't compensate for
> the
> fact that if "readers.length" returns an int of value "0", the "hi"
> variable will be assigned a value of "-1", which is then returned to
> any
> calling methods!
>
>
> Hope this is helpful,
> Rasik Pandey
>
> -----Message d'origine-----
> De : Rasik Pandey [mailto:rasik.pandey@ajlsm.com]
> Envoyé : lundi 9 septembre 2002 18:46
> À : 'Lucene Developers List'
> Objet : RE : Possible Bug with MultiSearcher?
>
>
> Lucene Developers,
> A test program is attached. I reiterate that the searcher for the
> empty
> index must be added first. The exception thrown at the end of the
> program is an ArrayIndexOutOfBoundsException. I will continue trying
> to
> recreate the orginal exception which I reported, but I think this one
> may be related. I was unable to recreate the exception using
> RAMDirectory stores, but only two FSDirectory stores. It maybe
> necessary
> to delete these directories from time to time???? Please let me know
> if
> I can provide any more information.
>
> Rasik Pandey
>
>
> -----Message d'origine-----
> De : Doug Cutting [mailto:cutting@lucene.com]
> Envoyé : jeudi 5 septembre 2002 23:22
> À : Lucene Developers List
> Objet : Re: Possible Bug with MultiSearcher?
>
>
> Can you please submit a complete, self-contained test program that
> demonstrates the problem? That will make it much easier for someone
> to
> debug and fix it.
>
> Thanks,
>
> Doug
>
> Rasik Pandey wrote:
> > Hello,
> > I am getting the following exception when searching using a
> > MultiSearcher and the first Searchable added is an empty
> IndexSearcher
>
> > with a segments file, the second is a normal IndexSearcher from a
> > populated Index . I am able to reproduce this without fail. The
> > exception is thrown when trying to retrieve a document from the
> Hits.
> > Is the exception really waranted, given there are hits in the
> second
> > index, I don't think the search should fail completely? When
> searching
>
> > an individual index, adding it to a Multisearcher, and performing
> the
> > search causes no problems. Any help or suggestions would be
> > appreciated.
> >
> >
> > Please see org.apache.lucene.store.InputStream.java 192:46 for the
> > location where the Exception is generated.
> >
> >
> > Rasik Pandey
> >
> >
> > java.io.IOException: read past EOF
> > at org.apache.lucene.store.InputStream.refill(Unknown Source) at
> > org.apache.lucene.store.InputStream.readByte(Unknown Source) at
> > org.apache.lucene.store.InputStream.readInt(Unknown Source) at
> > org.apache.lucene.store.InputStream.readLong(Unknown Source) at
> > org.apache.lucene.index.FieldsReader.doc(Unknown Source) at
> > org.apache.lucene.index.SegmentReader.document(Unknown Source) at
> > org.apache.lucene.search.IndexSearcher.doc(Unknown Source) at
> > org.apache.lucene.search.MultiSearcher.doc(Unknown Source) at
> > org.apache.lucene.search.Hits.doc(Unknown Source)
> >
> >
> >
> >
> >
> > --
> > To unsubscribe, e-mail:
> <mailto:lucene-dev-unsubscribe@jakarta.apache.org>
> > For additional commands, e-mail:
> > <mailto:lucene-dev-help@jakarta.apache.org>
> >
>
>
> --
> To unsubscribe, e-mail:
> <mailto:lucene-dev-unsubscribe@jakarta.apache.org>
> For additional commands, e-mail:
> <mailto:lucene-dev-help@jakarta.apache.org>
> > C:\Program Files\GNU\WinCvs 1.2\cvs.exe diff -r 1.4 -w
> MultiSearcher.java
> Index: MultiSearcher.java
> ===================================================================
> RCS file:
>
/home/cvspublic/jakarta-lucene/src/java/org/apache/lucene/search/MultiSearcher.java,v
> retrieving revision 1.4
> diff -w -r1.4 MultiSearcher.java
> 96,98c100,105
> < public final Document doc(int n) throws IOException {
> < int i = searcherIndex(n); // find searcher index
> < return searchers[i].doc(n - starts[i]); // dispatch to
> searcher
> ---
> > public Document doc(int n) throws IOException {
> > int i = searcherIndex(n);
> > if (i < 0 || i > searchables.length) //added this to ensure we
> have a valid index-rbp
> > return null; //added this to ensure we have a valid
> index-rbp // find searcher index
> > else
> > return searchables[i].doc(n - starts[i]); // dispatch to
> searcher
> 113c120
> < else if (n > midValue)
> ---
> > else if (n >= midValue) //made this a ">=", to ensure that a
> valid index is found
>
> CVS command finished execution
>
> > --
> To unsubscribe, e-mail:
> <mailto:lucene-dev-unsubscribe@jakarta.apache.org>
> For additional commands, e-mail:
<mailto:lucene-dev-help@jakarta.apache.org>


__________________________________________________
Do you Yahoo!?
Yahoo! News - Today's headlines
http://news.yahoo.com

--
To unsubscribe, e-mail: <mailto:lucene-dev-unsubscribe@jakarta.apache.org>
For additional commands, e-mail: <mailto:lucene-dev-help@jakarta.apache.org>

Thanks for the code. I converted it to a unit test and put in the CVS.
The test fails, looks like a bug in MultiSearcher.

Otis

--- Rasik Pandey <rasik.pandey@ajlsm.com> wrote:
> Developers,
> Ok this is the latest test program to reproduce the original problem
> reported. When I submitted the first test program, I was unable to
> reproduce the original, however this new test program reproduces both
> the original error and the second error which I reported in my last
> mail. I believe the problem lies in the second line of the
> SegmentsReader.readerIndex(int n) method. It doesn't compensate for
> the
> fact that if "readers.length" returns an int of value "0", the "hi"
> variable will be assigned a value of "-1", which is then returned to
> any
> calling methods!
>
>
> Hope this is helpful,
> Rasik Pandey
>
> -----Message d'origine-----
> De : Rasik Pandey [mailto:rasik.pandey@ajlsm.com]
> Envoyé : lundi 9 septembre 2002 18:46
> À : 'Lucene Developers List'
> Objet : RE : Possible Bug with MultiSearcher?
>
>
> Lucene Developers,
> A test program is attached. I reiterate that the searcher for the
> empty
> index must be added first. The exception thrown at the end of the
> program is an ArrayIndexOutOfBoundsException. I will continue trying
> to
> recreate the orginal exception which I reported, but I think this one
> may be related. I was unable to recreate the exception using
> RAMDirectory stores, but only two FSDirectory stores. It maybe
> necessary
> to delete these directories from time to time???? Please let me know
> if
> I can provide any more information.
>
> Rasik Pandey
>
>
> -----Message d'origine-----
> De : Doug Cutting [mailto:cutting@lucene.com]
> Envoyé : jeudi 5 septembre 2002 23:22
> À : Lucene Developers List
> Objet : Re: Possible Bug with MultiSearcher?
>
>
> Can you please submit a complete, self-contained test program that
> demonstrates the problem? That will make it much easier for someone
> to
> debug and fix it.
>
> Thanks,
>
> Doug
>
> Rasik Pandey wrote:
> > Hello,
> > I am getting the following exception when searching using a
> > MultiSearcher and the first Searchable added is an empty
> IndexSearcher
>
> > with a segments file, the second is a normal IndexSearcher from a
> > populated Index . I am able to reproduce this without fail. The
> > exception is thrown when trying to retrieve a document from the
> Hits.
> > Is the exception really waranted, given there are hits in the
> second
> > index, I don't think the search should fail completely? When
> searching
>
> > an individual index, adding it to a Multisearcher, and performing
> the
> > search causes no problems. Any help or suggestions would be
> > appreciated.
> >
> >
> > Please see org.apache.lucene.store.InputStream.java 192:46 for the
> > location where the Exception is generated.
> >
> >
> > Rasik Pandey
> >
> >
> > java.io.IOException: read past EOF
> > at org.apache.lucene.store.InputStream.refill(Unknown Source)
> > at org.apache.lucene.store.InputStream.readByte(Unknown Source)
> > at org.apache.lucene.store.InputStream.readInt(Unknown Source)
> > at org.apache.lucene.store.InputStream.readLong(Unknown Source)
> > at org.apache.lucene.index.FieldsReader.doc(Unknown Source)
> > at org.apache.lucene.index.SegmentReader.document(Unknown Source)
> > at org.apache.lucene.search.IndexSearcher.doc(Unknown Source)
> > at org.apache.lucene.search.MultiSearcher.doc(Unknown Source)
> > at org.apache.lucene.search.Hits.doc(Unknown Source)
> >
> >
> >
> >
> >
> > --
> > To unsubscribe, e-mail:
> <mailto:lucene-dev-unsubscribe@jakarta.apache.org>
> > For additional commands, e-mail:
> > <mailto:lucene-dev-help@jakarta.apache.org>
> >
>
>
> --
> To unsubscribe, e-mail:
> <mailto:lucene-dev-unsubscribe@jakarta.apache.org>
> For additional commands, e-mail:
> <mailto:lucene-dev-help@jakarta.apache.org>
>

> ATTACHMENT part 2 application/octet-stream
name=TestMultiSearcher.java
> --
> To unsubscribe, e-mail:
> <mailto:lucene-dev-unsubscribe@jakarta.apache.org>
> For additional commands, e-mail:
<mailto:lucene-dev-help@jakarta.apache.org>


__________________________________________________
Do you Yahoo!?
New DSL Internet Access from SBC & Yahoo!
http://sbc.yahoo.com

--
To unsubscribe, e-mail: <mailto:lucene-dev-unsubscribe@jakarta.apache.org>
For additional commands, e-mail: <mailto:lucene-dev-help@jakarta.apache.org>

The TestMultiSearcher tests still fails for me, even after I apply your
changes to MultiSearcher.

Are you running 'ant test-unit' to verify that this is a valid fix?

Otis

--- Rasik Pandey <rasik.pandey@ajlsm.com> wrote:
> Developers,
> Attached is the diff for MultiSearcher which seems to correct these
> bugs. I have not yet found any problems caused by these changes in
> testing........but we will keep you informed!
>
> Thanks,
> Rasik Pandey
>
> -----Message d'origine-----
> De : Rasik Pandey [mailto:rasik.pandey@ajlsm.com]
> Envoyé : jeudi 12 septembre 2002 19:01
> À : lucene-dev@jakarta.apache.org
> Objet : TR : Possible Bug with MultiSearcher?
>
>
> Developers,
> Ok this is the latest test program to reproduce the original problem
> reported. When I submitted the first test program, I was unable to
> reproduce the original, however this new test program reproduces both
> the original error and the second error which I reported in my last
> mail. I believe the problem lies in the second line of the
> SegmentsReader.readerIndex(int n) method. It doesn't compensate for
> the
> fact that if "readers.length" returns an int of value "0", the "hi"
> variable will be assigned a value of "-1", which is then returned to
> any
> calling methods!
>
>
> Hope this is helpful,
> Rasik Pandey
>
> -----Message d'origine-----
> De : Rasik Pandey [mailto:rasik.pandey@ajlsm.com]
> Envoyé : lundi 9 septembre 2002 18:46
> À : 'Lucene Developers List'
> Objet : RE : Possible Bug with MultiSearcher?
>
>
> Lucene Developers,
> A test program is attached. I reiterate that the searcher for the
> empty
> index must be added first. The exception thrown at the end of the
> program is an ArrayIndexOutOfBoundsException. I will continue trying
> to
> recreate the orginal exception which I reported, but I think this one
> may be related. I was unable to recreate the exception using
> RAMDirectory stores, but only two FSDirectory stores. It maybe
> necessary
> to delete these directories from time to time???? Please let me know
> if
> I can provide any more information.
>
> Rasik Pandey
>
>
> -----Message d'origine-----
> De : Doug Cutting [mailto:cutting@lucene.com]
> Envoyé : jeudi 5 septembre 2002 23:22
> À : Lucene Developers List
> Objet : Re: Possible Bug with MultiSearcher?
>
>
> Can you please submit a complete, self-contained test program that
> demonstrates the problem? That will make it much easier for someone
> to
> debug and fix it.
>
> Thanks,
>
> Doug
>
> Rasik Pandey wrote:
> > Hello,
> > I am getting the following exception when searching using a
> > MultiSearcher and the first Searchable added is an empty
> IndexSearcher
>
> > with a segments file, the second is a normal IndexSearcher from a
> > populated Index . I am able to reproduce this without fail. The
> > exception is thrown when trying to retrieve a document from the
> Hits.
> > Is the exception really waranted, given there are hits in the
> second
> > index, I don't think the search should fail completely? When
> searching
>
> > an individual index, adding it to a Multisearcher, and performing
> the
> > search causes no problems. Any help or suggestions would be
> > appreciated.
> >
> >
> > Please see org.apache.lucene.store.InputStream.java 192:46 for the
> > location where the Exception is generated.
> >
> >
> > Rasik Pandey
> >
> >
> > java.io.IOException: read past EOF
> > at org.apache.lucene.store.InputStream.refill(Unknown Source) at
> > org.apache.lucene.store.InputStream.readByte(Unknown Source) at
> > org.apache.lucene.store.InputStream.readInt(Unknown Source) at
> > org.apache.lucene.store.InputStream.readLong(Unknown Source) at
> > org.apache.lucene.index.FieldsReader.doc(Unknown Source) at
> > org.apache.lucene.index.SegmentReader.document(Unknown Source) at
> > org.apache.lucene.search.IndexSearcher.doc(Unknown Source) at
> > org.apache.lucene.search.MultiSearcher.doc(Unknown Source) at
> > org.apache.lucene.search.Hits.doc(Unknown Source)
> >
> >
> >
> >
> >
> > --
> > To unsubscribe, e-mail:
> <mailto:lucene-dev-unsubscribe@jakarta.apache.org>
> > For additional commands, e-mail:
> > <mailto:lucene-dev-help@jakarta.apache.org>
> >
>
>
> --
> To unsubscribe, e-mail:
> <mailto:lucene-dev-unsubscribe@jakarta.apache.org>
> For additional commands, e-mail:
> <mailto:lucene-dev-help@jakarta.apache.org>
> > C:\Program Files\GNU\WinCvs 1.2\cvs.exe diff -r 1.4 -w
> MultiSearcher.java
> Index: MultiSearcher.java
> ===================================================================
> RCS file:
>
/home/cvspublic/jakarta-lucene/src/java/org/apache/lucene/search/MultiSearcher.java,v
> retrieving revision 1.4
> diff -w -r1.4 MultiSearcher.java
> 96,98c100,105
> < public final Document doc(int n) throws IOException {
> < int i = searcherIndex(n); // find searcher index
> < return searchers[i].doc(n - starts[i]); // dispatch to
> searcher
> ---
> > public Document doc(int n) throws IOException {
> > int i = searcherIndex(n);
> > if (i < 0 || i > searchables.length) //added this to ensure we
> have a valid index-rbp
> > return null; //added this to ensure we have a valid
> index-rbp // find searcher index
> > else
> > return searchables[i].doc(n - starts[i]); // dispatch to
> searcher
> 113c120
> < else if (n > midValue)
> ---
> > else if (n >= midValue) //made this a ">=", to ensure that a
> valid index is found
>
> CVS command finished execution
>
> > --
> To unsubscribe, e-mail:
> <mailto:lucene-dev-unsubscribe@jakarta.apache.org>
> For additional commands, e-mail:
<mailto:lucene-dev-help@jakarta.apache.org>


__________________________________________________
Do you Yahoo!?
New DSL Internet Access from SBC & Yahoo!
http://sbc.yahoo.com

--
To unsubscribe, e-mail: <mailto:lucene-dev-unsubscribe@jakarta.apache.org>
For additional commands, e-mail: <mailto:lucene-dev-help@jakarta.apache.org>

Otis,
There were two problems with the TestMultiSearcher test case you adapted
from the test program I sent. Please see the attached diff....The test
is sucessful if you make the necessary changes.

Thanks,
Rasik

-----Message d'origine-----
De : Otis Gospodnetic [mailto:otis_gospodnetic@yahoo.com]
Envoyé : jeudi 19 septembre 2002 07:58
À : Lucene Developers List
Objet : Re: RE : TR : Possible Bug with MultiSearcher?


The TestMultiSearcher tests still fails for me, even after I apply your
changes to MultiSearcher.

Are you running 'ant test-unit' to verify that this is a valid fix?

Otis

--- Rasik Pandey <rasik.pandey@ajlsm.com> wrote:
> Developers,
> Attached is the diff for MultiSearcher which seems to correct these
> bugs. I have not yet found any problems caused by these changes in
> testing........but we will keep you informed!
>
> Thanks,
> Rasik Pandey
>
> -----Message d'origine-----
> De : Rasik Pandey [mailto:rasik.pandey@ajlsm.com]
> Envoyé : jeudi 12 septembre 2002 19:01
> À : lucene-dev@jakarta.apache.org
> Objet : TR : Possible Bug with MultiSearcher?
>
>
> Developers,
> Ok this is the latest test program to reproduce the original problem
> reported. When I submitted the first test program, I was unable to
> reproduce the original, however this new test program reproduces both
> the original error and the second error which I reported in my last
> mail. I believe the problem lies in the second line of the
> SegmentsReader.readerIndex(int n) method. It doesn't compensate for
> the fact that if "readers.length" returns an int of value "0", the
> "hi" variable will be assigned a value of "-1", which is then returned

> to any
> calling methods!
>
>
> Hope this is helpful,
> Rasik Pandey
>
> -----Message d'origine-----
> De : Rasik Pandey [mailto:rasik.pandey@ajlsm.com]
> Envoyé : lundi 9 septembre 2002 18:46
> À : 'Lucene Developers List'
> Objet : RE : Possible Bug with MultiSearcher?
>
>
> Lucene Developers,
> A test program is attached. I reiterate that the searcher for the
> empty index must be added first. The exception thrown at the end of
> the program is an ArrayIndexOutOfBoundsException. I will continue
> trying to
> recreate the orginal exception which I reported, but I think this one
> may be related. I was unable to recreate the exception using
> RAMDirectory stores, but only two FSDirectory stores. It maybe
> necessary
> to delete these directories from time to time???? Please let me know
> if
> I can provide any more information.
>
> Rasik Pandey
>
>
> -----Message d'origine-----
> De : Doug Cutting [mailto:cutting@lucene.com]
> Envoyé : jeudi 5 septembre 2002 23:22
> À : Lucene Developers List
> Objet : Re: Possible Bug with MultiSearcher?
>
>
> Can you please submit a complete, self-contained test program that
> demonstrates the problem? That will make it much easier for someone
> to
> debug and fix it.
>
> Thanks,
>
> Doug
>
> Rasik Pandey wrote:
> > Hello,
> > I am getting the following exception when searching using a
> > MultiSearcher and the first Searchable added is an empty
> IndexSearcher
>
> > with a segments file, the second is a normal IndexSearcher from a
> > populated Index . I am able to reproduce this without fail. The
> > exception is thrown when trying to retrieve a document from the
> Hits.
> > Is the exception really waranted, given there are hits in the
> second
> > index, I don't think the search should fail completely? When
> searching
>
> > an individual index, adding it to a Multisearcher, and performing
> the
> > search causes no problems. Any help or suggestions would be
> > appreciated.
> >
> >
> > Please see org.apache.lucene.store.InputStream.java 192:46 for the
> > location where the Exception is generated.
> >
> >
> > Rasik Pandey
> >
> >
> > java.io.IOException: read past EOF
> > at org.apache.lucene.store.InputStream.refill(Unknown Source) at
> > org.apache.lucene.store.InputStream.readByte(Unknown Source) at
> > org.apache.lucene.store.InputStream.readInt(Unknown Source) at
> > org.apache.lucene.store.InputStream.readLong(Unknown Source) at
> > org.apache.lucene.index.FieldsReader.doc(Unknown Source) at
> > org.apache.lucene.index.SegmentReader.document(Unknown Source) at
> > org.apache.lucene.search.IndexSearcher.doc(Unknown Source) at
> > org.apache.lucene.search.MultiSearcher.doc(Unknown Source) at
> > org.apache.lucene.search.Hits.doc(Unknown Source)
> >
> >
> >
> >
> >
> > --
> > To unsubscribe, e-mail:
> <mailto:lucene-dev-unsubscribe@jakarta.apache.org>
> > For additional commands, e-mail:
> > <mailto:lucene-dev-help@jakarta.apache.org>
> >
>
>
> --
> To unsubscribe, e-mail:
> <mailto:lucene-dev-unsubscribe@jakarta.apache.org>
> For additional commands, e-mail:
> <mailto:lucene-dev-help@jakarta.apache.org>
> > C:\Program Files\GNU\WinCvs 1.2\cvs.exe diff -r 1.4 -w
> MultiSearcher.java
> Index: MultiSearcher.java
> ===================================================================
> RCS file:
>
/home/cvspublic/jakarta-lucene/src/java/org/apache/lucene/search/MultiSe
archer.java,v
> retrieving revision 1.4
> diff -w -r1.4 MultiSearcher.java
> 96,98c100,105
> < public final Document doc(int n) throws IOException {
> < int i = searcherIndex(n); // find
searcher index
> < return searchers[i].doc(n - starts[i]); // dispatch to
> searcher
> ---
> > public Document doc(int n) throws IOException {
> > int i = searcherIndex(n);
> > if (i < 0 || i > searchables.length) //added this to ensure we
> have a valid index-rbp
> > return null; //added this to ensure we have a valid
> index-rbp // find searcher index
> > else
> > return searchables[i].doc(n - starts[i]); // dispatch to
> searcher
> 113c120
> < else if (n > midValue)
> ---
> > else if (n >= midValue) //made this a ">=", to ensure that a
> valid index is found
>
> CVS command finished execution
>
> > --
> To unsubscribe, e-mail:
> <mailto:lucene-dev-unsubscribe@jakarta.apache.org>
> For additional commands, e-mail:
<mailto:lucene-dev-help@jakarta.apache.org>


__________________________________________________
Do you Yahoo!?
New DSL Internet Access from SBC & Yahoo!
http://sbc.yahoo.com

--
To unsubscribe, e-mail:
<mailto:lucene-dev-unsubscribe@jakarta.apache.org>
For additional commands, e-mail:
<mailto:lucene-dev-help@jakarta.apache.org>

Ah, yes, mea culpa, thanks.

Otis

--- Rasik Pandey <rasik.pandey@ajlsm.com> wrote:
> Otis,
> There were two problems with the TestMultiSearcher test case you
> adapted
> from the test program I sent. Please see the attached diff....The
> test
> is sucessful if you make the necessary changes.
>
> Thanks,
> Rasik
>
> -----Message d'origine-----
> De : Otis Gospodnetic [mailto:otis_gospodnetic@yahoo.com]
> Envoyé : jeudi 19 septembre 2002 07:58
> À : Lucene Developers List
> Objet : Re: RE : TR : Possible Bug with MultiSearcher?
>
>
> The TestMultiSearcher tests still fails for me, even after I apply
> your
> changes to MultiSearcher.
>
> Are you running 'ant test-unit' to verify that this is a valid fix?
>
> Otis
>
> --- Rasik Pandey <rasik.pandey@ajlsm.com> wrote:
> > Developers,
> > Attached is the diff for MultiSearcher which seems to correct these
>
> > bugs. I have not yet found any problems caused by these changes in
> > testing........but we will keep you informed!
> >
> > Thanks,
> > Rasik Pandey
> >
> > -----Message d'origine-----
> > De : Rasik Pandey [mailto:rasik.pandey@ajlsm.com]
> > Envoyé : jeudi 12 septembre 2002 19:01
> > À : lucene-dev@jakarta.apache.org
> > Objet : TR : Possible Bug with MultiSearcher?
> >
> >
> > Developers,
> > Ok this is the latest test program to reproduce the original
> problem
> > reported. When I submitted the first test program, I was unable to
> > reproduce the original, however this new test program reproduces
> both
> > the original error and the second error which I reported in my last
>
> > mail. I believe the problem lies in the second line of the
> > SegmentsReader.readerIndex(int n) method. It doesn't compensate for
>
> > the fact that if "readers.length" returns an int of value "0", the
> > "hi" variable will be assigned a value of "-1", which is then
> returned
>
> > to any
> > calling methods!
> >
> >
> > Hope this is helpful,
> > Rasik Pandey
> >
> > -----Message d'origine-----
> > De : Rasik Pandey [mailto:rasik.pandey@ajlsm.com]
> > Envoyé : lundi 9 septembre 2002 18:46
> > À : 'Lucene Developers List'
> > Objet : RE : Possible Bug with MultiSearcher?
> >
> >
> > Lucene Developers,
> > A test program is attached. I reiterate that the searcher for the
> > empty index must be added first. The exception thrown at the end of
>
> > the program is an ArrayIndexOutOfBoundsException. I will continue
> > trying to
> > recreate the orginal exception which I reported, but I think this
> one
> > may be related. I was unable to recreate the exception using
> > RAMDirectory stores, but only two FSDirectory stores. It maybe
> > necessary
> > to delete these directories from time to time???? Please let me
> know
> > if
> > I can provide any more information.
> >
> > Rasik Pandey
> >
> >
> > -----Message d'origine-----
> > De : Doug Cutting [mailto:cutting@lucene.com]
> > Envoyé : jeudi 5 septembre 2002 23:22
> > À : Lucene Developers List
> > Objet : Re: Possible Bug with MultiSearcher?
> >
> >
> > Can you please submit a complete, self-contained test program that
> > demonstrates the problem? That will make it much easier for
> someone
> > to
> > debug and fix it.
> >
> > Thanks,
> >
> > Doug
> >
> > Rasik Pandey wrote:
> > > Hello,
> > > I am getting the following exception when searching using a
> > > MultiSearcher and the first Searchable added is an empty
> > IndexSearcher
> >
> > > with a segments file, the second is a normal IndexSearcher from a
>
> > > populated Index . I am able to reproduce this without fail. The
> > > exception is thrown when trying to retrieve a document from the
> > Hits.
> > > Is the exception really waranted, given there are hits in the
> > second
> > > index, I don't think the search should fail completely? When
> > searching
> >
> > > an individual index, adding it to a Multisearcher, and performing
> > the
> > > search causes no problems. Any help or suggestions would be
> > > appreciated.
> > >
> > >
> > > Please see org.apache.lucene.store.InputStream.java 192:46 for
> the
> > > location where the Exception is generated.
> > >
> > >
> > > Rasik Pandey
> > >
> > >
> > > java.io.IOException: read past EOF
> > > at org.apache.lucene.store.InputStream.refill(Unknown Source) at
> > > org.apache.lucene.store.InputStream.readByte(Unknown Source) at
> > > org.apache.lucene.store.InputStream.readInt(Unknown Source) at
> > > org.apache.lucene.store.InputStream.readLong(Unknown Source) at
> > > org.apache.lucene.index.FieldsReader.doc(Unknown Source) at
> > > org.apache.lucene.index.SegmentReader.document(Unknown Source) at
>
> > > org.apache.lucene.search.IndexSearcher.doc(Unknown Source) at
> > > org.apache.lucene.search.MultiSearcher.doc(Unknown Source) at
> > > org.apache.lucene.search.Hits.doc(Unknown Source)
> > >
> > >
> > >
> > >
> > >
> > > --
> > > To unsubscribe, e-mail:
> > <mailto:lucene-dev-unsubscribe@jakarta.apache.org>
> > > For additional commands, e-mail:
> > > <mailto:lucene-dev-help@jakarta.apache.org>
> > >
> >
> >
> > --
> > To unsubscribe, e-mail:
> > <mailto:lucene-dev-unsubscribe@jakarta.apache.org>
> > For additional commands, e-mail:
> > <mailto:lucene-dev-help@jakarta.apache.org>
> > > C:\Program Files\GNU\WinCvs 1.2\cvs.exe diff -r 1.4 -w
> > MultiSearcher.java
> > Index: MultiSearcher.java
> > ===================================================================
> > RCS file:
> >
>
/home/cvspublic/jakarta-lucene/src/java/org/apache/lucene/search/MultiSe
> archer.java,v
> > retrieving revision 1.4
> > diff -w -r1.4 MultiSearcher.java
> > 96,98c100,105
> > < public final Document doc(int n) throws IOException {
> > < int i = searcherIndex(n); // find
> searcher index
> > < return searchers[i].doc(n - starts[i]); // dispatch to
> > searcher
> > ---
> > > public Document doc(int n) throws IOException {
> > > int i = searcherIndex(n);
> > > if (i < 0 || i > searchables.length) //added this to ensure
> we
> > have a valid index-rbp
> > > return null; //added this to ensure we have a valid
> > index-rbp // find searcher index
> > > else
> > > return searchables[i].doc(n - starts[i]); // dispatch to
> > searcher
> > 113c120
> > < else if (n > midValue)
> > ---
> > > else if (n >= midValue) //made this a ">=", to ensure that
> a
> > valid index is found
> >
> > CVS command finished execution
> >
> > > --
> > To unsubscribe, e-mail:
> > <mailto:lucene-dev-unsubscribe@jakarta.apache.org>
> > For additional commands, e-mail:
> <mailto:lucene-dev-help@jakarta.apache.org>
>
>
> __________________________________________________
> Do you Yahoo!?
> New DSL Internet Access from SBC & Yahoo!
> http://sbc.yahoo.com
>
> --
> To unsubscribe, e-mail:
> <mailto:lucene-dev-unsubscribe@jakarta.apache.org>
> For additional commands, e-mail:
> <mailto:lucene-dev-help@jakarta.apache.org>
> > C:\Program Files\GNU\WinCvs 1.2\cvs.exe diff -r 1.2 -w
> TestMultiSearcher.java
> Index: TestMultiSearcher.java
> ===================================================================
> RCS file:
>
/home/cvspublic/jakarta-lucene/src/test/org/apache/lucene/search/TestMultiSearcher.java,v
> retrieving revision 1.2
> diff -w -r1.2 TestMultiSearcher.java
> 182c182
> < assertEquals(4, hits.length());
> ---
> > assertEquals(4, hits2.length());
> 224c224
> < assertEquals(3, hits.length());
> ---
> > assertEquals(3, hits3.length());
>
> CVS command finished execution
>
> > --
> To unsubscribe, e-mail:
> <mailto:lucene-dev-unsubscribe@jakarta.apache.org>
> For additional commands, e-mail:
<mailto:lucene-dev-help@jakarta.apache.org>


__________________________________________________
Do you Yahoo!?
New DSL Internet Access from SBC & Yahoo!
http://sbc.yahoo.com

--
To unsubscribe, e-mail: <mailto:lucene-dev-unsubscribe@jakarta.apache.org>
For additional commands, e-mail: <mailto:lucene-dev-help@jakarta.apache.org>

Rasik Pandey wrote:
> Developers,
> Attached is the diff for MultiSearcher which seems to correct these
> bugs. I have not yet found any problems caused by these changes in
> testing........but we will keep you informed!

[ ... ]

diff -w -r1.4 MultiSearcher.java
96,98c100,105
< public final Document doc(int n) throws IOException {
< int i = searcherIndex(n); // find searcher index
< return searchers[i].doc(n - starts[i]);
---
> public Document doc(int n) throws IOException {
> int i = searcherIndex(n);
> if (i < 0 || i > searchables.length)
> return null;
> else
> return searchables[i].doc(n - starts[i]);

I don't agree with this change. If you pass in a bogus document number
then an exception should be thrown here. ArrayIndexOutOfBounds is not
the friendliest of exceptions, but it's better than returning null.

> 113c120
> < else if (n > midValue)
> ---
>
>> else if (n >= midValue) //made this a ">=", to ensure that a valid index is found

This is the standard binary search algorithm, and should not be changed
lightly (according to Knuth, no less). The problem is when there are
multiple entries with the same value then this algorithm doesn't always
return the index of the largest one, which is required in this case. So
I think the fix is to scan for the largest value:

--- MultiSearcher.java.~1.5.~ 2002-07-17 10:28:56.000000000 -0700
+++ MultiSearcher.java 2002-09-19 15:02:02.000000000 -0700
@@ -116,9 +116,13 @@
hi = mid - 1;
else if (n > midValue)
lo = mid + 1;
- else
+ else {
+ while (mid+1 < searchables.length && starts[mid+1] == midValue) {
+ mid++; // return last match
+ }
return mid;
}
+ }
return hi;
}

That passes the test suite just checked in too!

Any objections to this?

Doug


--
To unsubscribe, e-mail: <mailto:lucene-dev-unsubscribe@jakarta.apache.org>
For additional commands, e-mail: <mailto:lucene-dev-help@jakarta.apache.org>

Doug,
Understood. I made the second change,in MulitSeacher, and it works on
this end. Do you think this change needs to be made in other places of
the lucene code, such as the SegmentsReader.readerIndex(int n) method,
as it uses what looks to be the same algorithm?


Thanks,
Rasik

-----Message d'origine-----
De : Doug Cutting [mailto:cutting@lucene.com]
Envoyé : vendredi 20 septembre 2002 00:05
À : Lucene Developers List
Objet : Re: RE : TR : Possible Bug with MultiSearcher?




Rasik Pandey wrote:
> Developers,
> Attached is the diff for MultiSearcher which seems to correct these
> bugs. I have not yet found any problems caused by these changes in
> testing........but we will keep you informed!

[ ... ]

diff -w -r1.4 MultiSearcher.java
96,98c100,105
< public final Document doc(int n) throws IOException {
< int i = searcherIndex(n); // find searcher index
< return searchers[i].doc(n - starts[i]);
---
> public Document doc(int n) throws IOException {
> int i = searcherIndex(n);
> if (i < 0 || i > searchables.length)
> return null;
> else
> return searchables[i].doc(n - starts[i]);

I don't agree with this change. If you pass in a bogus document number
then an exception should be thrown here. ArrayIndexOutOfBounds is not
the friendliest of exceptions, but it's better than returning null.

> 113c120
> < else if (n > midValue)
> ---
>
>> else if (n >= midValue) //made this a ">=", to ensure that a
>> valid index is found

This is the standard binary search algorithm, and should not be changed
lightly (according to Knuth, no less). The problem is when there are
multiple entries with the same value then this algorithm doesn't always
return the index of the largest one, which is required in this case. So

I think the fix is to scan for the largest value:

--- MultiSearcher.java.~1.5.~ 2002-07-17 10:28:56.000000000 -0700
+++ MultiSearcher.java 2002-09-19 15:02:02.000000000 -0700
@@ -116,9 +116,13 @@
hi = mid - 1;
else if (n > midValue)
lo = mid + 1;
- else
+ else {
+ while (mid+1 < searchables.length && starts[mid+1] == midValue)
{
+ mid++; // return last match
+ }
return mid;
}
+ }
return hi;
}

That passes the test suite just checked in too!

Any objections to this?

Doug


--
To unsubscribe, e-mail:
<mailto:lucene-dev-unsubscribe@jakarta.apache.org>
For additional commands, e-mail:
<mailto:lucene-dev-help@jakarta.apache.org>


--
To unsubscribe, e-mail: <mailto:lucene-dev-unsubscribe@jakarta.apache.org>
For additional commands, e-mail: <mailto:lucene-dev-help@jakarta.apache.org>

Rasik Pandey wrote:
> Understood. I made the second change,in MulitSeacher, and it works on
> this end. Do you think this change needs to be made in other places of
> the lucene code, such as the SegmentsReader.readerIndex(int n) method,
> as it uses what looks to be the same algorithm?

I was wondering the same thing. The fix is still correct for
SegmentsReader, but I don't think the bug could ever be triggered. The
only way you could ever have an empty segment before a non-empty one is
if you explicitly passed an empty index to IndexWriter.addIndexes(), but
this method never calls SegmentsReader.readerIndex(). Still, it's worth
making the patch just to be safe in case it ever does call it.

I just checked in the patch for both files.

Doug


--
To unsubscribe, e-mail: <mailto:lucene-dev-unsubscribe@jakarta.apache.org>
For additional commands, e-mail: <mailto:lucene-dev-help@jakarta.apache.org>