Mailing List Archive

Hello Rajib

To extract tokens as strings from a Lucene TokenStream, you can use the
CharTermAttribute class. This attribute holds the current token's text.
Here's how you can modify your code:

ArrayList<String> hotTokens = new ArrayList<>();
TokenStream stream = analyzer.tokenStream(field, new StringReader(query));
stream.reset();

CharTermAttribute charTermAttribute =
stream.addAttribute(CharTermAttribute.class);

while (stream.incrementToken()) {
String termText = charTermAttribute.toString();
hotTokens.add(termText);
}

In this code, we first add the CharTermAttribute to the TokenStream. Then,
inside the loop where we iterate over the tokens, we call
charTermAttribute.toString(), which gives us the current token as a string.
We then add this string to our list of hot tokens.


On Thu, Jan 25, 2024 at 2:26?PM Saha, Rajib <rajib.saha@sap.com.invalid>
wrote:
>
> Hi Experts,
>
> I need to get the Token as String from TokenStream for some further
processing.
> I have similar code as below.
> Can somebody please help me, how I can get the Token as String[termText]
in below code?
>
>
> ArrayList<String> hotTokens = new ArrayList();
> TokenStream stream = analyzer.tokenStream(field, new StringReader(query));
> stream.reset();
>
> while (stream.incrementToken()) {
> String termText = //Extract the Term from the stream. Need Help
here, how to extract it.
> hotTokens.add(termText);
> }
>
>
> Please let me know, if needed any further information from my side.
>
> Thanks In Advance.
>
> Regards
> Rajib
>
>


--
Sincerely yours
Mikhail Khludnev