Mailing List Archive: Working a Solr Model

Working a Solr Model

Sep 29, 2013, 12:54 AM

Post #1 of 6 (2606 views)

I'm commencing a project that is going to use Solr because it will be
search intensive.

My research hasn't turned up a lot in terms of discussion and how tos
for either Solr and Perl or Solr and Catalyst. There are two CPAN
modules which I've been experimenting with, WebService::Solr and
Apache::Solr.
WebService::Solr only implements the LWP agent part of working with
Solr, which still leaves one to extract the actual JSON what LWP gives
you, plus it has fatal wide-character issues that "use utf8::all" isn't
able to fix, and also does not support xml and csv (other formats solr
can provide). Apache::Solr is much nicer in that it provides something
resumbling a dbic result set, but I've already encountered a few things
that seem like bugs and sending the resultsetlike object to the stash in
the same way as a dbic resultset doesn't seem to work the same way.

Are other people using Solr based models? if so are you using either of
the two CPAN modules or something else? Do you have examples and or
notes you can share?

_______________________________________________
List: Catalyst@lists.scsys.co.uk
Listinfo: http://lists.scsys.co.uk/cgi-bin/mailman/listinfo/catalyst
Searchable archive: http://www.mail-archive.com/catalyst@lists.scsys.co.uk/
Dev site: http://dev.catalyst.perl.org/

Re: Working a Solr Model [ In reply to ]

rob at intelcompute

Sep 29, 2013, 1:33 AM

Post #2 of 6 (2578 views)

Permalink

I've always rolled my own for due to the reasons you mention with what's
on CPAN, tho I last did this more than 2 years ago, so even my examples
need cleaning up and re-factoring again :-/

On 09/29/2013 08:54 AM, John Karr wrote:
> I'm commencing a project that is going to use Solr because it will be
> search intensive.
>
> My research hasn't turned up a lot in terms of discussion and how tos
> for either Solr and Perl or Solr and Catalyst. There are two CPAN
> modules which I've been experimenting with, WebService::Solr and
> Apache::Solr.
> WebService::Solr only implements the LWP agent part of working with
> Solr, which still leaves one to extract the actual JSON what LWP gives
> you, plus it has fatal wide-character issues that "use utf8::all"
> isn't able to fix, and also does not support xml and csv (other
> formats solr can provide). Apache::Solr is much nicer in that it
> provides something resumbling a dbic result set, but I've already
> encountered a few things that seem like bugs and sending the
> resultsetlike object to the stash in the same way as a dbic resultset
> doesn't seem to work the same way.
>
> Are other people using Solr based models? if so are you using either
> of the two CPAN modules or something else? Do you have examples and or
> notes you can share?
>
> _______________________________________________
> List: Catalyst@lists.scsys.co.uk
> Listinfo: http://lists.scsys.co.uk/cgi-bin/mailman/listinfo/catalyst
> Searchable archive:
> http://www.mail-archive.com/catalyst@lists.scsys.co.uk/
> Dev site: http://dev.catalyst.perl.org/

_______________________________________________
List: Catalyst@lists.scsys.co.uk
Listinfo: http://lists.scsys.co.uk/cgi-bin/mailman/listinfo/catalyst
Searchable archive: http://www.mail-archive.com/catalyst@lists.scsys.co.uk/
Dev site: http://dev.catalyst.perl.org/

Re: Working a Solr Model [ In reply to ]

drew at drewtaylor

Sep 29, 2013, 5:58 PM

Post #3 of 6 (2563 views)

Permalink

I've been using WebService::Solr. I have a wrapper on top of WS::S, which
puts together the queries and handles the response. It's not perfect, but
it's worked fine for me so far. We don't have any utf-8 data, so I haven't
been bitten by wide-character issues. Good to know for the future though!

Thanks,
Drew

On Sun, Sep 29, 2013 at 5:54 PM, John Karr <brainbuz@brainbuz.org> wrote:

> I'm commencing a project that is going to use Solr because it will be
> search intensive.
>
> My research hasn't turned up a lot in terms of discussion and how tos for
> either Solr and Perl or Solr and Catalyst. There are two CPAN modules which
> I've been experimenting with, WebService::Solr and Apache::Solr.
> WebService::Solr only implements the LWP agent part of working with Solr,
> which still leaves one to extract the actual JSON what LWP gives you, plus
> it has fatal wide-character issues that "use utf8::all" isn't able to fix,
> and also does not support xml and csv (other formats solr can provide).
> Apache::Solr is much nicer in that it provides something resumbling a dbic
> result set, but I've already encountered a few things that seem like bugs
> and sending the resultsetlike object to the stash in the same way as a dbic
> resultset doesn't seem to work the same way.
>
> Are other people using Solr based models? if so are you using either of
> the two CPAN modules or something else? Do you have examples and or notes
> you can share?
>
> ______________________________**_________________
> List: Catalyst@lists.scsys.co.uk
> Listinfo: http://lists.scsys.co.uk/cgi-**bin/mailman/listinfo/catalyst<http://lists.scsys.co.uk/cgi-bin/mailman/listinfo/catalyst>
> Searchable archive: http://www.mail-archive.com/**
> catalyst@lists.scsys.co.uk/<http://www.mail-archive.com/catalyst@lists.scsys.co.uk/>
> Dev site: http://dev.catalyst.perl.org/
>

Re: Working a Solr Model [ In reply to ]

marcus at nordaaker

Sep 29, 2013, 10:55 PM

Post #4 of 6 (2581 views)

Permalink

We have been using SolarBeam - https://github.com/judofyr/solarbeam

It uses Mojo::UserAgent, and will allow parallel requests, even if you use
catalyst. We found this very useful when writing search intensive apps.

Marcus.

On Sunday, September 29, 2013, John Karr wrote:

> I'm commencing a project that is going to use Solr because it will be
> search intensive.
>
> My research hasn't turned up a lot in terms of discussion and how tos for
> either Solr and Perl or Solr and Catalyst. There are two CPAN modules which
> I've been experimenting with, WebService::Solr and Apache::Solr.
> WebService::Solr only implements the LWP agent part of working with Solr,
> which still leaves one to extract the actual JSON what LWP gives you, plus
> it has fatal wide-character issues that "use utf8::all" isn't able to fix,
> and also does not support xml and csv (other formats solr can provide).
> Apache::Solr is much nicer in that it provides something resumbling a dbic
> result set, but I've already encountered a few things that seem like bugs
> and sending the resultsetlike object to the stash in the same way as a dbic
> resultset doesn't seem to work the same way.
>
> Are other people using Solr based models? if so are you using either of
> the two CPAN modules or something else? Do you have examples and or notes
> you can share?
>
> ______________________________**_________________
> List: Catalyst@lists.scsys.co.uk
> Listinfo: http://lists.scsys.co.uk/cgi-**bin/mailman/listinfo/catalyst<http://lists.scsys.co.uk/cgi-bin/mailman/listinfo/catalyst>
> Searchable archive: http://www.mail-archive.com/**
> catalyst@lists.scsys.co.uk/<http://www.mail-archive.com/catalyst@lists.scsys.co.uk/>
> Dev site: http://dev.catalyst.perl.org/
>

--
---
Marcus Ramberg
Chief Yak Shaver
Nordaaker Consulting
+47-93417508

Re: Working a Solr Model [ In reply to ]

neil at mylunn

Sep 30, 2013, 12:53 AM

Post #5 of 6 (2573 views)

Permalink

On 29/09/2013 5:54 PM, John Karr wrote:
> WebService::Solr only implements the LWP agent part of working with
> Solr, which still leaves one to extract the actual JSON what LWP gives
> you, plus it has fatal wide-character issues that "use utf8::all"
> isn't able to fix, and also does not support xml and csv (other
> formats solr can provide).
Actually will parse a response as a WebService::Solr::Document (or
actually a collection of) with accessors to fields and a to_xml method,
which I have used in production.

The LWP agent part can be set to what you want. Say,
AnyEvent::HTTP::LWP::UserAgent or other if you wish, which is good when
running Catalyst in an event environment. And as other format loaders
are just HTTP Post interfaces then you can just intropect the Agent (
->agent ) from WebService::Solr.

Have used the above to create 100MB XML documents and post those to Solr
server.

> Apache::Solr is much nicer in that it provides something resumbling a
> dbic result set, but I've already encountered a few things that seem
> like bugs and sending the resultsetlike object to the stash in the
> same way as a dbic resultset doesn't seem to work the same way.
>
> Are other people using Solr based models? if so are you using either
> of the two CPAN modules or something else? Do you have examples and or
> notes you can share?
>
> _______________________________________________
> List: Catalyst@lists.scsys.co.uk
> Listinfo: http://lists.scsys.co.uk/cgi-bin/mailman/listinfo/catalyst
> Searchable archive:
> http://www.mail-archive.com/catalyst@lists.scsys.co.uk/
> Dev site: http://dev.catalyst.perl.org/

_______________________________________________
List: Catalyst@lists.scsys.co.uk
Listinfo: http://lists.scsys.co.uk/cgi-bin/mailman/listinfo/catalyst
Searchable archive: http://www.mail-archive.com/catalyst@lists.scsys.co.uk/
Dev site: http://dev.catalyst.perl.org/

Re: Working a Solr Model [ In reply to ]

brainbuz at brainbuz

Sep 30, 2013, 7:13 PM

Post #6 of 6 (2570 views)

Permalink

Thanks for the suggestion, but I just spent some time looking at
SolarBeam, and found that the example code did not work.

A little time debugging it confirmed that it internally constructed a
valid query url string which when pasted into a browser resulted in
multiple found documents, but SolarBeam was getting 0 documents found.
Since the last release was a year ago, it is quite possible that
something changed to break it (I'm using SOLR 4.4, was the author
working with the 3.x series?). I also noticed that you (Marcus R)
started a fork.

Since I've been able to work-around the issues with Apache::Solr I'm
starting my project with it.

> We have been using SolarBeam - https://github.com/judofyr/solarbeam
>
> It uses Mojo::UserAgent, and will allow parallel requests, even if you
> use catalyst. We found this very useful when writing search intensive
> apps.
>
> Marcus.
>
> On Sunday, September 29, 2013, John Karr wrote:
>
> I'm commencing a project that is going to use Solr because it will
> be search intensive.
>
> My research hasn't turned up a lot in terms of discussion and how
> tos for either Solr and Perl or Solr and Catalyst. There are two
> CPAN modules which I've been experimenting with, WebService::Solr
> and Apache::Solr.
> WebService::Solr only implements the LWP agent part of working
> with Solr, which still leaves one to extract the actual JSON what
> LWP gives you, plus it has fatal wide-character issues that "use
> utf8::all" isn't able to fix, and also does not support xml and
> csv (other formats solr can provide). Apache::Solr is much nicer
> in that it provides something resumbling a dbic result set, but
> I've already encountered a few things that seem like bugs and
> sending the resultsetlike object to the stash in the same way as a
> dbic resultset doesn't seem to work the same way.
>
> Are other people using Solr based models? if so are you using
> either of the two CPAN modules or something else? Do you have
> examples and or notes you can share?
>
> _______________________________________________
> List: Catalyst@lists.scsys.co.uk
> Listinfo: http://lists.scsys.co.uk/cgi-bin/mailman/listinfo/catalyst
> Searchable archive:
> http://www.mail-archive.com/catalyst@lists.scsys.co.uk/
> Dev site: http://dev.catalyst.perl.org/
>
>
>
> --
> ---
> Marcus Ramberg
> Chief Yak Shaver
> Nordaaker Consulting
> +47-93417508
>
>
>
> _______________________________________________
> List: Catalyst@lists.scsys.co.uk
> Listinfo: http://lists.scsys.co.uk/cgi-bin/mailman/listinfo/catalyst
> Searchable archive: http://www.mail-archive.com/catalyst@lists.scsys.co.uk/
> Dev site: http://dev.catalyst.perl.org/