Mailing List Archive

Ok. I've dug through all the code and although this isn't too pretty, it
looks to my untrained eye that this is probably the best way to implement a
corrected version that takes into consideration the deleted list.
Basically, it checks the "segments" file as before, then checks the
"segment.del" file in each segment, and takes the last time any of those
files were modified:

public static long lastModified(final Directory directory) throws
IOException {
synchronized (directory) { // in- & inter-process
sync
Long lastMod = (Long)new Lock.With(directory.makeLock("commit.lock"))
{
public Object doBody() throws IOException {
long last = directory.fileModified("segments"); // last add time
SegmentInfos infos = new SegmentInfos();
infos.read(directory);
for (int i = 0; i < infos.size(); i++) {
String delFile = infos.info(i).name + ".del";
if (directory.fileExists(delFile)) {
long lastDel = directory.fileModified(delFile); // last
delete time (per segment)
if (lastDel > last) last = lastDel;
}
}
return new Long(last);
}
}.run();
return lastMod.longValue();
}
}

Please let me know if I've missed any shortcuts or if you know of a better
way...

Scott

> -----Original Message-----
> From: Scott Ganyo [mailto:scott.ganyo@eTapestry.com]
> Sent: Wednesday, August 07, 2002 9:27 AM
> To: Lucene-Dev (E-mail)
> Subject: IndexReader.lastModified() not correct
>
>
> The current implementation of IndexReader.lastModified() does
> not return the
> results I am expecting. Here is the implementation:
>
> /** Returns the time the index in the named directory was
> last modified.
> */
> public static long lastModified(File directory) throws IOException {
> return FSDirectory.fileModified(directory, "segments");
> }
>
> The problem is that the "segments" file is apparently not
> updated when just
> doing a delete from an index. This is causing me problems
> because I am
> attempting to rely on on the lastModified() command for IndexSearcher
> caching. The only solution that I have thought of so far
> (without changing
> other parts of Lucene) is to make the lastModified() command
> look at all the
> files in the last segment for the last modified date.
>
> Thoughts?
>
> Thanks,
> Scott
>

Scott!
I don't know whether you code is good or not. My question is: can this problem cause my exception?
> java.io.IOException: read past EOF
> at java.lang.Throwable.fillInStackTrace(Native Method)
> at java.lang.Throwable.fillInStackTrace(Compiled Code)
> at java.lang.Throwable.<init>(Compiled Code)
> at java.lang.Exception.<init>(Compiled Code)
> at java.io.IOException.<init>(Compiled Code)
> at
> org.apache.lucene.store.FSInputStream.readInternal(Compiled Code)
> at
> org.apache.lucene.store.InputStream.readBytes(Compiled Code)
>

I also cache the searcher. I think the searcher was not reopened and tried to read something non-existing stuff

peter

> -----Original Message-----
> From: Scott Ganyo [mailto:scott.ganyo@eTapestry.com]
> Sent: Wednesday, August 07, 2002 6:32 PM
> To: 'Lucene Developers List'
> Subject: RE: IndexReader.lastModified() not correct
>
>
> Ok. I've dug through all the code and although this isn't
> too pretty, it
> looks to my untrained eye that this is probably the best way
> to implement a
> corrected version that takes into consideration the deleted list.
> Basically, it checks the "segments" file as before, then checks the
> "segment.del" file in each segment, and takes the last time
> any of those
> files were modified:
>
> public static long lastModified(final Directory directory) throws
> IOException {
> synchronized (directory) { //
> in- & inter-process
> sync
> Long lastMod = (Long)new
> Lock.With(directory.makeLock("commit.lock"))
> {
> public Object doBody() throws IOException {
> long last = directory.fileModified("segments"); //
> last add time
> SegmentInfos infos = new SegmentInfos();
> infos.read(directory);
> for (int i = 0; i < infos.size(); i++) {
> String delFile = infos.info(i).name + ".del";
> if (directory.fileExists(delFile)) {
> long lastDel =
> directory.fileModified(delFile); // last
> delete time (per segment)
> if (lastDel > last) last = lastDel;
> }
> }
> return new Long(last);
> }
> }.run();
> return lastMod.longValue();
> }
> }
>
> Please let me know if I've missed any shortcuts or if you
> know of a better
> way...
>
> Scott
>
> > -----Original Message-----
> > From: Scott Ganyo [mailto:scott.ganyo@eTapestry.com]
> > Sent: Wednesday, August 07, 2002 9:27 AM
> > To: Lucene-Dev (E-mail)
> > Subject: IndexReader.lastModified() not correct
> >
> >
> > The current implementation of IndexReader.lastModified() does
> > not return the
> > results I am expecting. Here is the implementation:
> >
> > /** Returns the time the index in the named directory was
> > last modified.
> > */
> > public static long lastModified(File directory) throws
> IOException {
> > return FSDirectory.fileModified(directory, "segments");
> > }
> >
> > The problem is that the "segments" file is apparently not
> > updated when just
> > doing a delete from an index. This is causing me problems
> > because I am
> > attempting to rely on on the lastModified() command for
> IndexSearcher
> > caching. The only solution that I have thought of so far
> > (without changing
> > other parts of Lucene) is to make the lastModified() command
> > look at all the
> > files in the last segment for the last modified date.
> >
> > Thoughts?
> >
> > Thanks,
> > Scott
> >
>

--
To unsubscribe, e-mail: <mailto:lucene-dev-unsubscribe@jakarta.apache.org>
For additional commands, e-mail: <mailto:lucene-dev-help@jakarta.apache.org>

Scott,
why do you lock the index while iterating del files?

if you don't lock it, you can calculate wrong time but has it effect on caching? one can modify the index between checking the lastModified value and creating new searcher.

peter

> -----Original Message-----
> From: Scott Ganyo [mailto:scott.ganyo@eTapestry.com]
> Sent: Wednesday, August 07, 2002 6:32 PM
> To: 'Lucene Developers List'
> Subject: RE: IndexReader.lastModified() not correct
>
>
> Ok. I've dug through all the code and although this isn't
> too pretty, it
> looks to my untrained eye that this is probably the best way
> to implement a
> corrected version that takes into consideration the deleted list.
> Basically, it checks the "segments" file as before, then checks the
> "segment.del" file in each segment, and takes the last time
> any of those
> files were modified:
>
> public static long lastModified(final Directory directory) throws
> IOException {
> synchronized (directory) { //
> in- & inter-process
> sync
> Long lastMod = (Long)new
> Lock.With(directory.makeLock("commit.lock"))
> {
> public Object doBody() throws IOException {
> long last = directory.fileModified("segments"); //
> last add time
> SegmentInfos infos = new SegmentInfos();
> infos.read(directory);
> for (int i = 0; i < infos.size(); i++) {
> String delFile = infos.info(i).name + ".del";
> if (directory.fileExists(delFile)) {
> long lastDel =
> directory.fileModified(delFile); // last
> delete time (per segment)
> if (lastDel > last) last = lastDel;
> }
> }
> return new Long(last);
> }
> }.run();
> return lastMod.longValue();
> }
> }
>
> Please let me know if I've missed any shortcuts or if you
> know of a better
> way...
>
> Scott
>
> > -----Original Message-----
> > From: Scott Ganyo [mailto:scott.ganyo@eTapestry.com]
> > Sent: Wednesday, August 07, 2002 9:27 AM
> > To: Lucene-Dev (E-mail)
> > Subject: IndexReader.lastModified() not correct
> >
> >
> > The current implementation of IndexReader.lastModified() does
> > not return the
> > results I am expecting. Here is the implementation:
> >
> > /** Returns the time the index in the named directory was
> > last modified.
> > */
> > public static long lastModified(File directory) throws
> IOException {
> > return FSDirectory.fileModified(directory, "segments");
> > }
> >
> > The problem is that the "segments" file is apparently not
> > updated when just
> > doing a delete from an index. This is causing me problems
> > because I am
> > attempting to rely on on the lastModified() command for
> IndexSearcher
> > caching. The only solution that I have thought of so far
> > (without changing
> > other parts of Lucene) is to make the lastModified() command
> > look at all the
> > files in the last segment for the last modified date.
> >
> > Thoughts?
> >
> > Thanks,
> > Scott
> >
>

--
To unsubscribe, e-mail: <mailto:lucene-dev-unsubscribe@jakarta.apache.org>
For additional commands, e-mail: <mailto:lucene-dev-help@jakarta.apache.org>

I lock the index because I copied the code from the open() method... and
because it seemed to be the right thing to do. I was concerned that the
index files not change during my iterating over them. For example, if the
index segments were merged into a new one between when I read the "segments"
file and when I iterated over the segment's deleted files, I wouldn't
necessarily get the right answer. On the other hand, I could check the last
modified date on the "segments" file *after* checking the individual
segments. Maybe I'll do that instead.

At any rate, though, you're right. It isn't truly safe unless the lock is
held during the entire process...

Scott

P.S. I don't know if this could have caused your Exception. I actually
can't think of a scenario off-hand where you would get that Exception except
in a corrupt index... is your index intact?

> -----Original Message-----
> From: Halácsy Péter [mailto:halacsy.peter@axelero.com]
> Sent: Wednesday, August 07, 2002 12:23 PM
> To: Lucene Developers List
> Subject: RE: IndexReader.lastModified() not correct
>
>
> Scott,
> why do you lock the index while iterating del files?
>
> if you don't lock it, you can calculate wrong time but has it
> effect on caching? one can modify the index between checking
> the lastModified value and creating new searcher.
>
> peter
>
> > -----Original Message-----
> > From: Scott Ganyo [mailto:scott.ganyo@eTapestry.com]
> > Sent: Wednesday, August 07, 2002 6:32 PM
> > To: 'Lucene Developers List'
> > Subject: RE: IndexReader.lastModified() not correct
> >
> >
> > Ok. I've dug through all the code and although this isn't
> > too pretty, it
> > looks to my untrained eye that this is probably the best way
> > to implement a
> > corrected version that takes into consideration the deleted list.
> > Basically, it checks the "segments" file as before, then checks the
> > "segment.del" file in each segment, and takes the last time
> > any of those
> > files were modified:
> >
> > public static long lastModified(final Directory directory) throws
> > IOException {
> > synchronized (directory) { //
> > in- & inter-process
> > sync
> > Long lastMod = (Long)new
> > Lock.With(directory.makeLock("commit.lock"))
> > {
> > public Object doBody() throws IOException {
> > long last = directory.fileModified("segments"); //
> > last add time
> > SegmentInfos infos = new SegmentInfos();
> > infos.read(directory);
> > for (int i = 0; i < infos.size(); i++) {
> > String delFile = infos.info(i).name + ".del";
> > if (directory.fileExists(delFile)) {
> > long lastDel =
> > directory.fileModified(delFile); // last
> > delete time (per segment)
> > if (lastDel > last) last = lastDel;
> > }
> > }
> > return new Long(last);
> > }
> > }.run();
> > return lastMod.longValue();
> > }
> > }
> >
> > Please let me know if I've missed any shortcuts or if you
> > know of a better
> > way...
> >
> > Scott
> >
> > > -----Original Message-----
> > > From: Scott Ganyo [mailto:scott.ganyo@eTapestry.com]
> > > Sent: Wednesday, August 07, 2002 9:27 AM
> > > To: Lucene-Dev (E-mail)
> > > Subject: IndexReader.lastModified() not correct
> > >
> > >
> > > The current implementation of IndexReader.lastModified() does
> > > not return the
> > > results I am expecting. Here is the implementation:
> > >
> > > /** Returns the time the index in the named directory was
> > > last modified.
> > > */
> > > public static long lastModified(File directory) throws
> > IOException {
> > > return FSDirectory.fileModified(directory, "segments");
> > > }
> > >
> > > The problem is that the "segments" file is apparently not
> > > updated when just
> > > doing a delete from an index. This is causing me problems
> > > because I am
> > > attempting to rely on on the lastModified() command for
> > IndexSearcher
> > > caching. The only solution that I have thought of so far
> > > (without changing
> > > other parts of Lucene) is to make the lastModified() command
> > > look at all the
> > > files in the last segment for the last modified date.
> > >
> > > Thoughts?
> > >
> > > Thanks,
> > > Scott
> > >
> >
>
> --
> To unsubscribe, e-mail:
> <mailto:lucene-dev-unsubscribe@jakarta.apache.org>
> For additional commands, e-mail:
> <mailto:lucene-dev-help@jakarta.apache.org>
>

Ok. Here's a new, simplified version that doesn't lock the index:

/** Returns the time the index in this directory was last modified. */
public static long lastModified(final Directory directory) throws
IOException {
long last = 0;
SegmentInfos infos = new SegmentInfos();
infos.read(directory);
for (int i = 0; i < infos.size(); i++) {
String delFile = infos.info(i).name + ".del";
if (directory.fileExists(delFile)) {
long lastDel = directory.fileModified(delFile); // last delete
time (per segment)
if (lastDel > last) last = lastDel;
}
}
long lastAdd = directory.fileModified("segments"); // last add time
return (lastAdd > last) ? lastAdd : last;
}

Comments?

Scott

> -----Original Message-----
> From: Scott Ganyo [mailto:scott.ganyo@eTapestry.com]
> Sent: Wednesday, August 07, 2002 12:41 PM
> To: 'Lucene Developers List'
> Subject: RE: IndexReader.lastModified() not correct
>
>
> I lock the index because I copied the code from the open()
> method... and
> because it seemed to be the right thing to do. I was
> concerned that the
> index files not change during my iterating over them. For
> example, if the
> index segments were merged into a new one between when I read
> the "segments"
> file and when I iterated over the segment's deleted files, I wouldn't
> necessarily get the right answer. On the other hand, I could
> check the last
> modified date on the "segments" file *after* checking the individual
> segments. Maybe I'll do that instead.
>
> At any rate, though, you're right. It isn't truly safe
> unless the lock is
> held during the entire process...
>
> Scott
>
> P.S. I don't know if this could have caused your Exception.
> I actually
> can't think of a scenario off-hand where you would get that
> Exception except
> in a corrupt index... is your index intact?
>
> > -----Original Message-----
> > From: Halácsy Péter [mailto:halacsy.peter@axelero.com]
> > Sent: Wednesday, August 07, 2002 12:23 PM
> > To: Lucene Developers List
> > Subject: RE: IndexReader.lastModified() not correct
> >
> >
> > Scott,
> > why do you lock the index while iterating del files?
> >
> > if you don't lock it, you can calculate wrong time but has it
> > effect on caching? one can modify the index between checking
> > the lastModified value and creating new searcher.
> >
> > peter
> >
> > > -----Original Message-----
> > > From: Scott Ganyo [mailto:scott.ganyo@eTapestry.com]
> > > Sent: Wednesday, August 07, 2002 6:32 PM
> > > To: 'Lucene Developers List'
> > > Subject: RE: IndexReader.lastModified() not correct
> > >
> > >
> > > Ok. I've dug through all the code and although this isn't
> > > too pretty, it
> > > looks to my untrained eye that this is probably the best way
> > > to implement a
> > > corrected version that takes into consideration the deleted list.
> > > Basically, it checks the "segments" file as before, then
> checks the
> > > "segment.del" file in each segment, and takes the last time
> > > any of those
> > > files were modified:
> > >
> > > public static long lastModified(final Directory
> directory) throws
> > > IOException {
> > > synchronized (directory) { //
> > > in- & inter-process
> > > sync
> > > Long lastMod = (Long)new
> > > Lock.With(directory.makeLock("commit.lock"))
> > > {
> > > public Object doBody() throws IOException {
> > > long last = directory.fileModified("segments"); //
> > > last add time
> > > SegmentInfos infos = new SegmentInfos();
> > > infos.read(directory);
> > > for (int i = 0; i < infos.size(); i++) {
> > > String delFile = infos.info(i).name + ".del";
> > > if (directory.fileExists(delFile)) {
> > > long lastDel =
> > > directory.fileModified(delFile); // last
> > > delete time (per segment)
> > > if (lastDel > last) last = lastDel;
> > > }
> > > }
> > > return new Long(last);
> > > }
> > > }.run();
> > > return lastMod.longValue();
> > > }
> > > }
> > >
> > > Please let me know if I've missed any shortcuts or if you
> > > know of a better
> > > way...
> > >
> > > Scott
> > >
> > > > -----Original Message-----
> > > > From: Scott Ganyo [mailto:scott.ganyo@eTapestry.com]
> > > > Sent: Wednesday, August 07, 2002 9:27 AM
> > > > To: Lucene-Dev (E-mail)
> > > > Subject: IndexReader.lastModified() not correct
> > > >
> > > >
> > > > The current implementation of IndexReader.lastModified() does
> > > > not return the
> > > > results I am expecting. Here is the implementation:
> > > >
> > > > /** Returns the time the index in the named directory was
> > > > last modified.
> > > > */
> > > > public static long lastModified(File directory) throws
> > > IOException {
> > > > return FSDirectory.fileModified(directory, "segments");
> > > > }
> > > >
> > > > The problem is that the "segments" file is apparently not
> > > > updated when just
> > > > doing a delete from an index. This is causing me problems
> > > > because I am
> > > > attempting to rely on on the lastModified() command for
> > > IndexSearcher
> > > > caching. The only solution that I have thought of so far
> > > > (without changing
> > > > other parts of Lucene) is to make the lastModified() command
> > > > look at all the
> > > > files in the last segment for the last modified date.
> > > >
> > > > Thoughts?
> > > >
> > > > Thanks,
> > > > Scott
> > > >
> > >
> >
> > --
> > To unsubscribe, e-mail:
> > <mailto:lucene-dev-unsubscribe@jakarta.apache.org>
> > For additional commands, e-mail:
> > <mailto:lucene-dev-help@jakarta.apache.org>
> >
>

> -----Original Message-----
> From: Scott Ganyo [mailto:scott.ganyo@eTapestry.com]
> Sent: Wednesday, August 07, 2002 7:41 PM
> To: 'Lucene Developers List'
> Subject: RE: IndexReader.lastModified() not correct

> P.S. I don't know if this could have caused your Exception.
> I actually
> can't think of a scenario off-hand where you would get that
> Exception except
> in a corrupt index... is your index intact?
>

I implemented the worst solution. I've checked the modification time of the directory containing my index. It's good on windows but not on unix. now I'll use your code.

I have only one question:
is it faster to iterate on segments:
SegmentInfos infos = new SegmentInfos();
infos.read(directory);

or you could use directory.list() and check the file extensions?

peter











--
To unsubscribe, e-mail: <mailto:lucene-dev-unsubscribe@jakarta.apache.org>
For additional commands, e-mail: <mailto:lucene-dev-help@jakarta.apache.org>

Looks correct to me, but it's a pain to have to do so much work to check
this.

I wonder if we should instead add a Directory.lastModified() method.
Both the "segments" file and the .del files are installed with
Directory.renameFile(). On both Unix and Win32, file renaming updates
the directory's last modified time, so FSDirectory could implement this
with File.lastModified() on its directory, and it wouldn't be hard to
implement this for RAMDirectory.

Thoughts?

Doug


--
To unsubscribe, e-mail: <mailto:lucene-dev-unsubscribe@jakarta.apache.org>
For additional commands, e-mail: <mailto:lucene-dev-help@jakarta.apache.org>

> -----Original Message-----
> From: Doug Cutting [mailto:cutting@lucene.com]
> Sent: Wednesday, August 07, 2002 8:09 PM
> To: Lucene Developers List
> Subject: Re: IndexReader.lastModified() not correct
>
>
> Looks correct to me, but it's a pain to have to do so much
> work to check
> this.
>
> I wonder if we should instead add a Directory.lastModified() method.
> Both the "segments" file and the .del files are installed with
> Directory.renameFile(). On both Unix and Win32, file
> renaming updates
> the directory's last modified time, so FSDirectory could
> implement this
> with File.lastModified() on its directory, and it wouldn't be hard to
> implement this for RAMDirectory.
>
> Thoughts?
if this is true then my
new File(pathToIndex).lastModified() should return good value. But it doesn't.


peter

--
To unsubscribe, e-mail: <mailto:lucene-dev-unsubscribe@jakarta.apache.org>
For additional commands, e-mail: <mailto:lucene-dev-help@jakarta.apache.org>

> I have only one question:
> is it faster to iterate on segments:
> SegmentInfos infos = new SegmentInfos();
> infos.read(directory);
>
> or you could use directory.list() and check the file extensions?

This is very fast, I think. If you dig into the code, there is very little
that it needs to do to determine the segments. I admit, though, that I
haven't done any performance comparisons between this version and doing a
directory list()...

Scott

> -----Original Message-----
> From: Halácsy Péter [mailto:halacsy.peter@axelero.com]
> Sent: Wednesday, August 07, 2002 12:52 PM
> To: Lucene Developers List
> Subject: RE: IndexReader.lastModified() not correct
>
>
>
>
> > -----Original Message-----
> > From: Scott Ganyo [mailto:scott.ganyo@eTapestry.com]
> > Sent: Wednesday, August 07, 2002 7:41 PM
> > To: 'Lucene Developers List'
> > Subject: RE: IndexReader.lastModified() not correct
>
> > P.S. I don't know if this could have caused your Exception.
> > I actually
> > can't think of a scenario off-hand where you would get that
> > Exception except
> > in a corrupt index... is your index intact?
> >
>
> I implemented the worst solution. I've checked the
> modification time of the directory containing my index. It's
> good on windows but not on unix. now I'll use your code.
>
> I have only one question:
> is it faster to iterate on segments:
> SegmentInfos infos = new SegmentInfos();
> infos.read(directory);
>
> or you could use directory.list() and check the file extensions?
>
> peter
>
>
>
>
>
>
>
>
>
>
>
> --
> To unsubscribe, e-mail:
<mailto:lucene-dev-unsubscribe@jakarta.apache.org>
For additional commands, e-mail: <mailto:lucene-dev-help@jakarta.apache.org>

I don't care much either way. I don't believe that the directory's modified
time is updated on rename of an enclosing file, though...

Scott

> -----Original Message-----
> From: Doug Cutting [mailto:cutting@lucene.com]
> Sent: Wednesday, August 07, 2002 1:09 PM
> To: Lucene Developers List
> Subject: Re: IndexReader.lastModified() not correct
>
>
> Looks correct to me, but it's a pain to have to do so much
> work to check
> this.
>
> I wonder if we should instead add a Directory.lastModified() method.
> Both the "segments" file and the .del files are installed with
> Directory.renameFile(). On both Unix and Win32, file
> renaming updates
> the directory's last modified time, so FSDirectory could
> implement this
> with File.lastModified() on its directory, and it wouldn't be hard to
> implement this for RAMDirectory.
>
> Thoughts?
>
> Doug
>
>
> --
> To unsubscribe, e-mail:
> <mailto:lucene-dev-unsubscribe@jakarta.apache.org>
> For additional commands, e-mail:
> <mailto:lucene-dev-help@jakarta.apache.org>
>

Halácsy Péter wrote:
> I implemented the worst solution. I've checked the modification time of the directory containing my index. It's good on windows but not on unix. now I'll use your code.

Why doesn't this work on unix?

> I have only one question:
> is it faster to iterate on segments:
> SegmentInfos infos = new SegmentInfos();
> infos.read(directory);
>
> or you could use directory.list() and check the file extensions?

I think it should be faster to iterate on segments.

Doug


--
To unsubscribe, e-mail: <mailto:lucene-dev-unsubscribe@jakarta.apache.org>
For additional commands, e-mail: <mailto:lucene-dev-help@jakarta.apache.org>

Scott Ganyo wrote:
> I don't care much either way. I don't believe that the directory's modified
> time is updated on rename of an enclosing file, though...

Hmmm... It is for me when I use 'mv' from the shell...

Doug



--
To unsubscribe, e-mail: <mailto:lucene-dev-unsubscribe@jakarta.apache.org>
For additional commands, e-mail: <mailto:lucene-dev-help@jakarta.apache.org>

Sorry. I meant to say "on windows." I've verified that you are correct on
Solaris and OS X.

Scott

> -----Original Message-----
> From: Doug Cutting [mailto:cutting@lucene.com]
> Sent: Wednesday, August 07, 2002 1:23 PM
> To: Lucene Developers List
> Subject: Re: IndexReader.lastModified() not correct
>
>
> Scott Ganyo wrote:
> > I don't care much either way. I don't believe that the
> directory's modified
> > time is updated on rename of an enclosing file, though...
>
> Hmmm... It is for me when I use 'mv' from the shell...
>
> Doug
>
>
>
> --
> To unsubscribe, e-mail:
> <mailto:lucene-dev-unsubscribe@jakarta.apache.org>
> For additional commands, e-mail:
> <mailto:lucene-dev-help@jakarta.apache.org>
>

-1. Now that we've verified that Windows won't work with this proposal
without additional checks (and I *really* hate OS-specific checks in Java),
I don't think this would be a good idea.

What about the solution that I posted? It doesn't seem like so all that
much work now that I'm not using the locks...

Scott

> -----Original Message-----
> From: Doug Cutting [mailto:cutting@lucene.com]
> Sent: Wednesday, August 07, 2002 1:09 PM
> To: Lucene Developers List
> Subject: Re: IndexReader.lastModified() not correct
>
>
> Looks correct to me, but it's a pain to have to do so much
> work to check
> this.
>
> I wonder if we should instead add a Directory.lastModified() method.
> Both the "segments" file and the .del files are installed with
> Directory.renameFile(). On both Unix and Win32, file
> renaming updates
> the directory's last modified time, so FSDirectory could
> implement this
> with File.lastModified() on its directory, and it wouldn't be hard to
> implement this for RAMDirectory.
>
> Thoughts?
>
> Doug
>
>
> --
> To unsubscribe, e-mail:
> <mailto:lucene-dev-unsubscribe@jakarta.apache.org>
> For additional commands, e-mail:
> <mailto:lucene-dev-help@jakarta.apache.org>
>

Scott,
IndexReader.lasModified() returns wrong value if and only if documents were deleted. if you don't want to reopen searcher on deletes you can use the old code.

can't we mix the two solutions?
IndexReader.lastModified(boolean dels)

if dels is true -> returns the time when a document was added or deleted
if dels is false -> return the time when the last document was added

peter

> -----Original Message-----
> From: Scott Ganyo [mailto:scott.ganyo@eTapestry.com]
> Sent: Thursday, August 08, 2002 4:04 PM
> To: 'Lucene Developers List'
> Subject: RE: IndexReader.lastModified() not correct
>
>
> -1. Now that we've verified that Windows won't work with
> this proposal
> without additional checks (and I *really* hate OS-specific
> checks in Java),
> I don't think this would be a good idea.
>
> What about the solution that I posted? It doesn't seem like
> so all that
> much work now that I'm not using the locks...
>
> Scott
>
> > -----Original Message-----
> > From: Doug Cutting [mailto:cutting@lucene.com]
> > Sent: Wednesday, August 07, 2002 1:09 PM
> > To: Lucene Developers List
> > Subject: Re: IndexReader.lastModified() not correct
> >
> >
> > Looks correct to me, but it's a pain to have to do so much
> > work to check
> > this.
> >
> > I wonder if we should instead add a
> Directory.lastModified() method.
> > Both the "segments" file and the .del files are installed with
> > Directory.renameFile(). On both Unix and Win32, file
> > renaming updates
> > the directory's last modified time, so FSDirectory could
> > implement this
> > with File.lastModified() on its directory, and it wouldn't
> be hard to
> > implement this for RAMDirectory.
> >
> > Thoughts?
> >
> > Doug
> >
> >
> > --
> > To unsubscribe, e-mail:
> > <mailto:lucene-dev-unsubscribe@jakarta.apache.org>
> > For additional commands, e-mail:
> > <mailto:lucene-dev-help@jakarta.apache.org>
> >
>

--
To unsubscribe, e-mail: <mailto:lucene-dev-unsubscribe@jakarta.apache.org>
For additional commands, e-mail: <mailto:lucene-dev-help@jakarta.apache.org>

I believe that would only needlessly complicate the API and confuse
developers. The truth is that the method is broken. Arguing that the
method is only broken under certain circumstances is just trying to avoid
that reality. Why do you want to do that?

Scott

> -----Original Message-----
> From: Halácsy Péter [mailto:halacsy.peter@axelero.com]
> Sent: Thursday, August 08, 2002 9:11 AM
> To: Lucene Developers List
> Subject: RE: IndexReader.lastModified() not correct
>
>
> Scott,
> IndexReader.lasModified() returns wrong value if and only if
> documents were deleted. if you don't want to reopen searcher
> on deletes you can use the old code.
>
> can't we mix the two solutions?
> IndexReader.lastModified(boolean dels)
>
> if dels is true -> returns the time when a document was added
> or deleted
> if dels is false -> return the time when the last document was added
>
> peter
>
> > -----Original Message-----
> > From: Scott Ganyo [mailto:scott.ganyo@eTapestry.com]
> > Sent: Thursday, August 08, 2002 4:04 PM
> > To: 'Lucene Developers List'
> > Subject: RE: IndexReader.lastModified() not correct
> >
> >
> > -1. Now that we've verified that Windows won't work with
> > this proposal
> > without additional checks (and I *really* hate OS-specific
> > checks in Java),
> > I don't think this would be a good idea.
> >
> > What about the solution that I posted? It doesn't seem like
> > so all that
> > much work now that I'm not using the locks...
> >
> > Scott
> >
> > > -----Original Message-----
> > > From: Doug Cutting [mailto:cutting@lucene.com]
> > > Sent: Wednesday, August 07, 2002 1:09 PM
> > > To: Lucene Developers List
> > > Subject: Re: IndexReader.lastModified() not correct
> > >
> > >
> > > Looks correct to me, but it's a pain to have to do so much
> > > work to check
> > > this.
> > >
> > > I wonder if we should instead add a
> > Directory.lastModified() method.
> > > Both the "segments" file and the .del files are installed with
> > > Directory.renameFile(). On both Unix and Win32, file
> > > renaming updates
> > > the directory's last modified time, so FSDirectory could
> > > implement this
> > > with File.lastModified() on its directory, and it wouldn't
> > be hard to
> > > implement this for RAMDirectory.
> > >
> > > Thoughts?
> > >
> > > Doug
> > >
> > >
> > > --
> > > To unsubscribe, e-mail:
> > > <mailto:lucene-dev-unsubscribe@jakarta.apache.org>
> > > For additional commands, e-mail:
> > > <mailto:lucene-dev-help@jakarta.apache.org>
> > >
> >
>
> --
> To unsubscribe, e-mail:
<mailto:lucene-dev-unsubscribe@jakarta.apache.org>
For additional commands, e-mail: <mailto:lucene-dev-help@jakarta.apache.org>

Scott Ganyo wrote:
> -1. Now that we've verified that Windows won't work with this proposal
> without additional checks (and I *really* hate OS-specific checks in Java),
> I don't think this would be a good idea.

Here's another approach: have FSDirectory.renameFile() call
File.setLastModified() on its directory. It doesn't need to check what
OS is running--this always a correct thing to do.

Then we can just use the directory's modified time to see when the index
has changed, keeping that operation fast and cheap.

Doug


--
To unsubscribe, e-mail: <mailto:lucene-dev-unsubscribe@jakarta.apache.org>
For additional commands, e-mail: <mailto:lucene-dev-help@jakarta.apache.org>

I'd be all for that, but that doesn't work on Winblows either (I just wrote
a test program and checked).

How about we just have the renameFile() method update the "segments" file
lastModified date and then we won't have to change the lastModified() method
at all...

Scott

> -----Original Message-----
> From: Doug Cutting [mailto:cutting@lucene.com]
> Sent: Thursday, August 08, 2002 11:17 AM
> To: Lucene Developers List
> Subject: Re: IndexReader.lastModified() not correct
>
>
> Scott Ganyo wrote:
> > -1. Now that we've verified that Windows won't work with
> this proposal
> > without additional checks (and I *really* hate OS-specific
> checks in Java),
> > I don't think this would be a good idea.
>
> Here's another approach: have FSDirectory.renameFile() call
> File.setLastModified() on its directory. It doesn't need to
> check what
> OS is running--this always a correct thing to do.
>
> Then we can just use the directory's modified time to see
> when the index
> has changed, keeping that operation fast and cheap.
>
> Doug
>
>
> --
> To unsubscribe, e-mail:
> <mailto:lucene-dev-unsubscribe@jakarta.apache.org>
> For additional commands, e-mail:
> <mailto:lucene-dev-help@jakarta.apache.org>
>

Scott Ganyo wrote:
> How about we just have the renameFile() method update the "segments" file
> lastModified date and then we won't have to change the lastModified() method
> at all...

I like that.

Or, better yet, add a setLastModified method to Directory, and have the
deletion code call it on the segments file when it writes a .del file.
That makes things explicit and doesn't overload renameFile(). If
someone calls renameFile() to do something that doesn't alter the index,
it won't break things. And if someone adds other, non-index, files into
the directory, it won't break things either (as it would if we used the
directory's last modified time).

Doug


--
To unsubscribe, e-mail: <mailto:lucene-dev-unsubscribe@jakarta.apache.org>
For additional commands, e-mail: <mailto:lucene-dev-help@jakarta.apache.org>

Here are the diffs for my latest proposal.

If there are no objections, I will check these changes in.

Doug

It looks fine to me.

The lack of encapsulation breaks my heart, though. ;)

Scott

> -----Original Message-----
> From: Doug Cutting [mailto:cutting@lucene.com]
> Sent: Thursday, August 08, 2002 12:26 PM
> To: Lucene Developers List
> Subject: Re: IndexReader.lastModified() not correct
>
>
> Here are the diffs for my latest proposal.
>
> If there are no objections, I will check these changes in.
>
> Doug
>