Mailing List Archive: Trying to create NEAR option in QueryParser -- ideas/help

Trying to create NEAR option in QueryParser -- ideas/help

Jan 12, 2002, 7:04 PM

Post #1 of 3 (804 views)

Hi,

I am trying to implement a NEAR option in the QueryParser.jj.

First I have to say, Brian Goetz has done an awesome job putting this
together.
I rely on it all the time, and it's solid and very complex especially if
you don't know JavaCC

Now to my question.

Overview of how I think the QueryParser works:
The basis for the QueryParser is to break up everything into the
appropriate type of Query (TermQuery, PhraseQuery, ...), by the matching
the query pattern. Then to combine these queries into a collection of
BooleanClauses and finally BooleanQueries.

Now the NEAR option will only work with phrase searches since that is
the only place where you can set the slop factor.

So I had two thoughts.
1) Create a new pattern <TERM> "NEAR"<NUM>+ <TERM>
The problem I have with this, is that I don't think it will work since
the parser will see <TERM> and never look for the "NEAR" option.

2) Retroactively make a TermQuery a PhraseQuery with a set slop.
This is somewhat how the AND conjunction works. I am proposing taking
the previous BooleanClause and, check to see if the query is a
TermQuery. If so then extract the Term and replace the query in the
BooleanClause with the new PhraseQuery. However, in trying to do this I
find that I cannot extract the Term from the TermQuery because there is
no getTerm() from the TermQuery. I don't think it would be difficult to
add, but I there might be other issues.

Any thoughts or idea will be helpful.

Thanks

--Peter

--
To unsubscribe, e-mail: <mailto:lucene-user-unsubscribe@jakarta.apache.org>
For additional commands, e-mail: <mailto:lucene-user-help@jakarta.apache.org>

Re: Trying to create NEAR option in QueryParser -- ideas/help [ In reply to ]

brian at quiotix

Jan 13, 2002, 6:11 PM

Post #2 of 3 (785 views)

Permalink

>First I have to say, Brian Goetz has done an awesome job putting this
>together.

A great way to get my attention, thanks! :)

>Overview of how I think the QueryParser works:
>The basis for the QueryParser is to break up everything into the
>appropriate type of Query (TermQuery, PhraseQuery, ...), by the matching
>the query pattern. Then to combine these queries into a collection of
>BooleanClauses and finally BooleanQueries.

That's the theory. Unfortunately, we cheat a little bit.

>Now the NEAR option will only work with phrase searches since that is the
>only place where you can set the slop factor.

Right.

>So I had two thoughts.
>1) Create a new pattern <TERM> "NEAR"<NUM>+ <TERM>
>The problem I have with this, is that I don't think it will work since the
>parser will see <TERM> and never look for the "NEAR" option.

That's one of the real problems with writing two-level parsers with tools
like JavaCC -- the tokenizer is completly separate from the parser. With
the current parser, it recognizes the tokens AND and OR and NOT, only in
uppercase, because (a) most users type in lower case and/or mixed case, and
(b) these are likely to be stop words (or should be) anyway. But NEAR is
more dangerous, since its a useful word, but the upper-case rule might be
enough.

>2) Retroactively make a TermQuery a PhraseQuery with a set slop.

This is a better strategy.

I had been planning on overhauling this myself, but basically that got
stalled because I didn't want to take it on until we had a clear statement
of what the goal and functionality of the query parser should be and were
looking for a syntax for all the various features that made sense.

--
Brian Goetz
Quiotix Corporation
brian@quiotix.com Tel: 650-843-1300 Fax: 650-324-8032

http://www.quiotix.com

--
To unsubscribe, e-mail: <mailto:lucene-user-unsubscribe@jakarta.apache.org>
For additional commands, e-mail: <mailto:lucene-user-help@jakarta.apache.org>

Re: Trying to create NEAR option in QueryParser -- ideas/help [ In reply to ]

carlson at bookandhammer

Jan 14, 2002, 8:20 AM

Post #3 of 3 (779 views)

Permalink

Thanks for the help Brian,

On Sunday, January 13, 2002, at 05:11 PM, Brian Goetz wrote:

>
>> First I have to say, Brian Goetz has done an awesome job putting this
>> together.
>
> A great way to get my attention, thanks! :)
>
>> Overview of how I think the QueryParser works:
>> The basis for the QueryParser is to break up everything into the
>> appropriate type of Query (TermQuery, PhraseQuery, ...), by the
>> matching the query pattern. Then to combine these queries into a
>> collection of BooleanClauses and finally BooleanQueries.
>
> That's the theory. Unfortunately, we cheat a little bit.
>
>> Now the NEAR option will only work with phrase searches since that is
>> the only place where you can set the slop factor.
>
> Right.
>
>> So I had two thoughts.
>> 1) Create a new pattern <TERM> "NEAR"<NUM>+ <TERM>
>> The problem I have with this, is that I don't think it will work since
>> the parser will see <TERM> and never look for the "NEAR" option.
>
> That's one of the real problems with writing two-level parsers with
> tools like JavaCC -- the tokenizer is completly separate from the
> parser. With the current parser, it recognizes the tokens AND and OR
> and NOT, only in uppercase, because (a) most users type in lower case
> and/or mixed case, and (b) these are likely to be stop words (or should
> be) anyway. But NEAR is more dangerous, since its a useful word, but
> the upper-case rule might be enough.
>
>> 2) Retroactively make a TermQuery a PhraseQuery with a set slop.
>
> This is a better strategy.
>
> I had been planning on overhauling this myself, but basically that got
> stalled because I didn't want to take it on until we had a clear
> statement of what the goal and functionality of the query parser should
> be and were looking for a syntax for all the various features that made
> sense.
>
>
>
> --
> Brian Goetz
> Quiotix Corporation
> brian@quiotix.com Tel: 650-843-1300 Fax:
> 650-324-8032
>
> http://www.quiotix.com
>
>
> --
> To unsubscribe, e-mail: <mailto:lucene-user-
> unsubscribe@jakarta.apache.org>
> For additional commands, e-mail: <mailto:lucene-user-
> help@jakarta.apache.org>
>
>

--
To unsubscribe, e-mail: <mailto:lucene-user-unsubscribe@jakarta.apache.org>
For additional commands, e-mail: <mailto:lucene-user-help@jakarta.apache.org>