Mailing List Archive

On Feb 12, 2007, at 8:57 AM, Erik Hatcher wrote:
> i gots plenty of ideas but they all need to be tempered with real
> implementations.

- add faceting examples to the wiki (is there good faceting support
in solr.rb?)
- make solrb more prominent in the wiki (right now it's buried in
SolRuby)
- automatic iteration across solr paged responses
- distribute on rubyforge

Honestly I would prefer to keep solrb lean and mean...instead of it
becoming a huge kitchen sink of features that are difficult to wade
through.

//Ed

Hi:

I been testing Solrb for a while now. I would love to
see "Dynamic Faceting over the whole index" as I
explained in my earlier mail. This is the only thing
right now I am missing. I am sure this can be done at
the moment but I haven't gotten around to it yet.

Other then that I am looking forward to see Flare grow
with foklsonomy/tagging support etc. all the cool
stuff that wiki promises.

Regards



--- Edward Summers <ehs@pobox.com> skrev:

>
> On Feb 12, 2007, at 8:57 AM, Erik Hatcher wrote:
> > i gots plenty of ideas but they all need to be
> tempered with real
> > implementations.
>
> - add faceting examples to the wiki (is there good
> faceting support
> in solr.rb?)
> - make solrb more prominent in the wiki (right now
> it's buried in
> SolRuby)
> - automatic iteration across solr paged responses
> - distribute on rubyforge
>
> Honestly I would prefer to keep solrb lean and
> mean...instead of it
> becoming a huge kitchen sink of features that are
> difficult to wade
> through.
>
> //Ed
>






_________________________________________________________
Flyger tiden iväg? Fånga dagen med Yahoo! Mails inbyggda
kalender. Dessutom 250 MB gratis, virusscanning och antispam. Få den på: http://se.mail.yahoo.com

On 2/12/07, Erik Hatcher <erik@ehatchersolutions.com> wrote:
> so, anyone making use of solrb (and maybe even flare) yet?

We're on the *verge* of deploying the first round of user-facing Solr
functionality at my work (using solrb), and it's amazing. Almost a
million records and we can still run stupid-complicated queries ("find
me anything with at least 4 of these 10 tags") in milliseconds. solrb
is *really* working well, especially since it's so light-weight.

On 2/12/07, Edward Summers <ehs@pobox.com> wrote:
> Honestly I would prefer to keep solrb lean and mean...instead of it
> becoming a huge kitchen sink of features that are difficult to wade
> through.

I totally agree. One of the things that got me using solrb was that it
let me define how to bind it to my own application, instead of being
an all-singing, all-dancing, acts_as_everything_everywhere-style
plugin.

Some changes I'd like to see (and will be willing to help out with):
* gem packaging and distribution on rubyforge (along with a quick
release schedule)
* add XML parsing capability to libxml-ruby code
* better documentation, both on the method level and on the integration level
* more modular request/response design
* better security -- make sure Solr gets passed sanitary data
* query builder DSL?

I'm sure once we roll out more functionality based on solrb I'll have
a lot more ideas.

--
Coda Hale
http://blog.codahale.com

Thanks Erik,

(This is a mostly a plus-one, but also a go-go-go! email.)

> let me define how to bind it to my own application, instead of being
> an all-singing, all-dancing, acts_as_everything_everywhere-style
> plugin.
A resounding +1.

> * gem packaging and distribution on rubyforge (along with a quick
> release schedule)
> * add XML parsing capability to libxml-ruby code
> * better documentation, both on the method level and on the
> integration level
> * more modular request/response design
> * better security -- make sure Solr gets passed sanitary data
> * query builder DSL?
>
> I'm sure once we roll out more functionality based on solrb I'll have
> a lot more ideas.

codas_list.each {|feature| feature.vote_for }

(I'm also interested in fiddling with the source. So much fun, so
little time.)

Just to say it aloud: http://wiki.apache.org/solr/solrb/BrainStorming
gets me all giddy with anticipation.

best regards,
Olle

PS Erik: (Mr. Wettin says hi. He explained solr and solrb to me. I'm
sold.)

On Feb 13, 2007, at 2:51 AM, Coda Hale wrote:
> We're on the *verge* of deploying the first round of user-facing Solr
> functionality at my work (using solrb), and it's amazing. Almost a
> million records and we can still run stupid-complicated queries ("find
> me anything with at least 4 of these 10 tags") in milliseconds. solrb
> is *really* working well, especially since it's so light-weight.

Fantastic!

> Some changes I'd like to see (and will be willing to help out with):
> * gem packaging and distribution on rubyforge (along with a quick
> release schedule)

I personally am currently -0 to -1 on releasing solrb at rubyforge.
I know its the "Ruby Way", but we're at Apache and I'd like to do
this the "Apache Way". Making releases available elsewhere is
frowned upon.

I have no objections, for the time being, to us checking in packages
(.gem/.tar/etc) under the pkg directory as "release candidates" and
allowing folks to "gem install -source ... " them. I think the
additional -source switch gives Ed heartburn, though I really want to
make sure we do this the Apache way as much as possible.

I can set up nightly builds and test runs and publish nightly
packages too if we want.

Ideally solrb will stay lock-step with Solr releases so you can
always rely on a particular solrb version to work with a particular
Solr version.

> * add XML parsing capability to libxml-ruby code

She's all yours, Coda :)

> * better documentation, both on the method level and on the
> integration level

My bad. I was just reading the rdoc documentation in the Pickaxe
book. We can all chip in on this, but I'll take a stab in the near
future at improving this myself.

> * more modular request/response design

Something I'd love to see also. I think the request and response
code doesn't need to be so separate.

> * better security -- make sure Solr gets passed sanitary data

Care to elaborate on this?

> * query builder DSL?

Query Builder *UI* is what I'm after.

> I'm sure once we roll out more functionality based on solrb I'll have
> a lot more ideas.

Thanks Coda!

Erik

On Feb 13, 2007, at 3:21 AM, Olle Jonsson wrote:
>> let me define how to bind it to my own application, instead of being
>> an all-singing, all-dancing, acts_as_everything_everywhere-style
>> plugin.
> A resounding +1.

However, I do think we should fairly soon roll in a "acts_as_solr"
ActiveRecord hook. I don't personally use ActiveRecord in this way,
but it would be a huge draw for many. And its not all that hard to
implement. Though field naming conventions are tricky... should the
Solr schema.xml be generic enough to adapt to any model? Or should
the schema be more rigidly defined towards a specific model?

> (I'm also interested in fiddling with the source. So much fun, so
> little time.)

Jump on in!

> Just to say it aloud: http://wiki.apache.org/solr/solrb/
> BrainStorming gets me all giddy with anticipation.

This was a just a quick off-the-cuff idea. How we tie Solr to a Ruby
object model is the trick that I've not yet experienced a clean
solution for.

My work with Flare is going to lead to some UI-centric DSL stuff,
very much like Streamlined. And that will in turn have some
direction for the underlying solrb layer as well.

> PS Erik: (Mr. Wettin says hi. He explained solr and solrb to me.
> I'm sold.)

"Hi" back to Karl!

Erik

My work with Flare is going to lead to some UI-centric DSL stuff,
very much like Streamlined. And that will in turn have some
direction for the underlying solrb layer as well.

Interesting. The project was somewhat "slow" for a while but now that I checked it's all up and running with 0.07. Do you have any interest of using Streamlined in Flare?

Furthermore are there more rails plugin that you envision using? if so what?..... would be nice to get a head start :-)

Cheers



---------------------------------

Stava rätt! Stava lätt! Yahoo! Mails stavkontroll tar hand om tryckfelen och mycket mer! Få den på http://se.mail.yahoo.com

On Feb 13, 2007, at 8:53 AM, Antonio Eggberg wrote:

>
>
> My work with Flare is going to lead to some UI-centric DSL stuff,
> very much like Streamlined. And that will in turn have some
> direction for the underlying solrb layer as well.
>
> Interesting. The project was somewhat "slow" for a while but now
> that I checked it's all up and running with 0.07. Do you have any
> interest of using Streamlined in Flare?

I don't actually envision using Streamlined as-is, but rather being
Streamlined-like. Streamlined is very ActiveRecord dependent. While
I've yet to settle on exactly what Flare is, my current thinking is
that it'll be a Rails plugin that becomes the "Streamlined of Solr".
I think there will be more to it than just that, and being Rails-
centric is not the goal, but rather one nice integration point to
have. I think what we build will be applicable to Camping and the
next cool Ruby web framework too.

> Furthermore are there more rails plugin that you envision using? if
> so what?..... would be nice to get a head start :-)

Currently I don't foresee using any other Rails plugins as Flare
dependencies. Though in a similar vein as Streamlined, we ought to
roll in the acts_as_solr feature similar to the acts_as_ferret plugin.

Anyone out there integrating an ActiveRecord model and Solr/solrb?
If so, would acts_as_solr make your life easier?

Erik

On Feb 13, 2007, at 8:30 AM, Erik Hatcher wrote:
> I have no objections, for the time being, to us checking in
> packages (.gem/.tar/etc) under the pkg directory as "release
> candidates" and allowing folks to "gem install -source ... " them.
> I think the additional -source switch gives Ed heartburn, though I
> really want to make sure we do this the Apache way as much as
> possible.

That's fine...it just isn't optimal. I also don't like being reduced
to submitting patches all the time. Honestly, that's limited my
degree of involvement. Perhaps that's the apache way too :-) And that
isn't a plea to get commit rights btw--just an observation. I'm not
sure I really want commit rights and may perhaps just develop my own
minimalist solr gem over at rubyforge since the space is already there.

//Ed

On 2/13/07, Erik Hatcher <erik@ehatchersolutions.com> wrote:
> I personally am currently -0 to -1 on releasing solrb at rubyforge.
> I know its the "Ruby Way", but we're at Apache and I'd like to do
> this the "Apache Way". Making releases available elsewhere is
> frowned upon.
>
> I have no objections, for the time being, to us checking in packages
> (.gem/.tar/etc) under the pkg directory as "release candidates" and
> allowing folks to "gem install -source ... " them. I think the
> additional -source switch gives Ed heartburn, though I really want to
> make sure we do this the Apache way as much as possible.

Hosting gems outside of Rubyforge means that I don't get to type 'gem
update' and have the latest version of solrb installed. So I'm at a
loss as to why "the Apache way" should mean more complicated systems
maintenance for me. I mean, I don't go to apache.org and download,
compile, and install a src tarball every time there's an update to the
HTTP server -- I type 'apt-get upgrade'.

If it were complicated to add gems to Rubyforge, I could understand,
but it's the work of a few minutes every release.

--
Coda Hale
http://blog.codahale.com

On Feb 13, 2007, at 2:15 PM, Edward Summers wrote:
> On Feb 13, 2007, at 8:30 AM, Erik Hatcher wrote:
>> I have no objections, for the time being, to us checking in
>> packages (.gem/.tar/etc) under the pkg directory as "release
>> candidates" and allowing folks to "gem install -source ... "
>> them. I think the additional -source switch gives Ed heartburn,
>> though I really want to make sure we do this the Apache way as
>> much as possible.
>
> That's fine...it just isn't optimal.

I spoke with Tom Copeland and Chad Fowler a bit about this topic at
the Rails Edge event a few weeks ago. Serving gems from the Apache
mirroring environment would work just fine, and once it records a
source, it doesn't need to be specified for "gem update" I don't
believe.

I don't really know much about gems myself, so I'm mainly going on
gut feel here. I'm not the final say in this matter, I'm merely
expressing my opinion. What I want is for us to discuss all the pros/
cons to how to deploy gems and plugins. RubyForge is cool, for sure,
but I would like to see us work towards Apache being a first class
open source Ruby citizen, and having Solr and solrb/Flare all version
and release together has a lot of appeal to me.

> I also don't like being reduced to submitting patches all the time.
> Honestly, that's limited my degree of involvement. Perhaps that's
> the apache way too :-)

Indeed it is. The bar is certainly high for committership, but quite
open to contributors. And I've applied every patch you've submitted,
typically within less than a day. So you have had one of the
smoothest possible contributor roles ever. You're used to a lot more
control on the repository than you've currently got with Solr's
svn. That can change in good time. No hurry though - your patches
are given consideration ASAP and have to-date always been applied
promptly.

Knowing what you're up to professionally, I'd tend to think you're
busy with other things and not fiddling with solrb daily, so that
might have more to do with your slowing down on it. It'd be a shame
for you to feel that submitting patches is a drag... but it is a
necessary step to have community vetting of contributions before they
get committed. Look at how well that has worked for Solr itself.
Ryan's recent refactoring of request handlers and paths iterated ad
naseum via JIRA with very solid and detailed discussions from both
committers and contributors, and the design improved thanks to those
conversations.

The Apache Way, more than anything, is about community over code.
Just like the agile manifesto says, its not that the items on the
right are not important, its that we value the ones on the left more.

And on that note...

> And that isn't a plea to get commit rights btw--just an
> observation. I'm not sure I really want commit rights and may
> perhaps just develop my own minimalist solr gem over at rubyforge
> since the space is already there.

What a shame to have you fork at this early stage because of "--
source" or having to submit patches.

Minimal API is here: <http://wiki.apache.org/solr/SolRuby> Just talk
Net::HTTP... but let's work together to make solrb be the preferred
Ruby way to speak to Solr. And all the other details of code,
deployment, etc can continue to be discussion. I'm not dogmatic on
not releasing via RubyForge, or any other technical details (heh, I
committed your code after all :).

Erik

On Feb 13, 2007, at 9:10 PM, Erik Hatcher wrote:
> It'd be a shame for you to feel that submitting patches is a
> drag... but it is a necessary step to have community vetting of
> contributions before they get committed. Look at how well that has
> worked for Solr itself. Ryan's recent refactoring of request
> handlers and paths iterated ad naseum via JIRA with very solid and
> detailed discussions from both committers and contributors, and the
> design improved thanks to those conversations.

And for the record, I fought with the Ant committers fairly
vigorously to get my first patches committed, and continued to make a
nuisance of myself by fixing bugs, documentation, and prodding for
things to get committed when they sat in Bugzilla. And the same
thing happened with me and Lucene and Tapestry.

We're a meritocracy - do things of merit and you earn karma. But, it
does require perseverance.

Erik

Isn't rubyforge in some sense the equivalent of the maven repositories
at ... wherever they are? Or was the issue access prior release?

Rails plugins are a lot easier (pronounced "scarier"?) It'll scrape any
site you want for interesting things that look like plugins or you can
just give it an svn url.

On Feb 13, 2007, at 8:11 PM, Coda Hale wrote:
> On 2/13/07, Erik Hatcher <erik@ehatchersolutions.com> wrote:
>> I personally am currently -0 to -1 on releasing solrb at rubyforge.
>> I know its the "Ruby Way", but we're at Apache and I'd like to do
>> this the "Apache Way". Making releases available elsewhere is
>> frowned upon.
>>
>> I have no objections, for the time being, to us checking in packages
>> (.gem/.tar/etc) under the pkg directory as "release candidates" and
>> allowing folks to "gem install -source ... " them. I think the
>> additional -source switch gives Ed heartburn, though I really want to
>> make sure we do this the Apache way as much as possible.
>
> Hosting gems outside of Rubyforge means that I don't get to type 'gem
> update' and have the latest version of solrb installed.

Really? If you install a gem from another source, "gem update"
doesn't work? Is this documented somewhere I can learn more about?

We have a direct line to the rubygems developers, so if we need to
get rubygems changed to work with other sources more naturally, and
even have a built-in search path for apache.org then we can do it.

> So I'm at a
> loss as to why "the Apache way" should mean more complicated systems
> maintenance for me.

It won't, ideally. I wouldn't want it to be more complicated. If it
is (and again I plead a lot of ignorance when it comes to rubygems)
then let's fix it somehow. Having RubyForge as the only blessed open
source rubygems site smells bad to me.

> I mean, I don't go to apache.org and download,
> compile, and install a src tarball every time there's an update to the
> HTTP server -- I type 'apt-get upgrade'.

Good point, for sure. But, let's just assume we get the
infrastructure at apache.org set up to serve gems including the
worldwide mirroring capabilities, and have rubygems look at the gem
index at apache.org... "gem install solrb" should just work then, eh?

Again, I'm open to doing this all sorts of ways, and I play devil's
advocate in discussions often to explore the other side of
arguments. I'm interested in exploring more about making apache.org
be as simple as rubyforge with rubygems. What's needed that isn't
already there?

> If it were complicated to add gems to Rubyforge, I could understand,
> but it's the work of a few minutes every release.

Cool enough. And if we can automate the pushing of the gems to
wherever we deploy them to from the build process then even better.
hoe does this, right?

Ok, speaking of releases... is solrb ready for a release? What's
missing?

Getting a release out before Feb. 26 would be nice for me, and
perhaps we can deploy both to apache.org as a release candidate
(either just checking it into pkg, or pushing it to a nightly build
area) and to RubyForge and experiment with "gem install" both ways
and see what the hurdles are.

Erik

On Feb 13, 2007, at 9:10 PM, Erik Hatcher wrote:
> What a shame to have you fork at this early stage because of "--
> source" or having to submit patches.

I'm actually trying to make a point more than a threat. What's
stopping someone from uploading something called 'solr' to rubyforge
and thus creating confusion about what the appropriate gem to use is?

I can understand wanting to have code and packages hosted at apache.
But I don't seem what the problem is with at the same time
distributing the gem in a way that is familiar to the ruby community
at large. This is how rails is distributed for example [1,2].

//Ed

[1] http://rubyforge.org/projects/rails/
[2] http://rubyonrails.org/down

On Feb 13, 2007, at 9:25 PM, Steven Parkes wrote:
> Isn't rubyforge in some sense the equivalent of the maven repositories
> at ... wherever they are? Or was the issue access prior release?

Good point. Along that analogy, Maven supports multiple
repositories, just as rubygems does (and again if there are issues
with it working nicely with Apache infrastructure we can fix it).

> Rails plugins are a lot easier (pronounced "scarier"?) It'll scrape
> any
> site you want for interesting things that look like plugins or you can
> just give it an svn url.

Flare will be packaged as Rails plugins eventually.

Ok, so let's do an experiment with the first solrb release and see
how much pain is felt from an Apache hosted "release candidate" (we
have to be careful calling it a "Release" as that implies more
oversight from Solr and the Lucene PMC).

Erik

On Feb 13, 2007, at 9:43 PM, Edward Summers wrote:
> I can understand wanting to have code and packages hosted at
> apache. But I don't seem what the problem is with at the same time
> distributing the gem in a way that is familiar to the ruby
> community at large. This is how rails is distributed for example
> [1,2].

Certainly a good point about Rails itself.

Again, I'm not trying to make things more complicated or unfamiliar
to anyone. Let me spell out my ideal:

gem install solrb

and have that come from apache.org. If rubygems cannot currently
handle this, then I'd like to push in that direction. It would
require us to work on some infrastructure issues and get gems and
their manifests deployed properly to be mirrored.

But, let's move forward with the release planning rather than going
back and forth with handwaving over the hassles apache.org deployment
may present to Ruby developers.

With what we've got, how do we deploy a gem to RubyForge? Who would
be in charge of that and how would that coordinate with the official
building of the release? Can we do this straight from rake? "rake
release"? I'm all for us releasing to RubyForge given this
discussion. And I'll try the parallel track of pushing the Apache
infrastructure and rubygems modifications.

Erik

On 2/13/07, Erik Hatcher <erik@ehatchersolutions.com> wrote:

> Again, I'm not trying to make things more complicated or unfamiliar
> to anyone. Let me spell out my ideal:
>
> gem install solrb
>
> and have that come from apache.org. If rubygems cannot currently
> handle this, then I'd like to push in that direction. It would
> require us to work on some infrastructure issues and get gems and
> their manifests deployed properly to be mirrored.

I'm not entirely sure why this requires a change in the way rubyforge
or gem works. I think the previous analogy of 'apt-get apache2' is
spot on. This is not pulling a dpkg from apache.org, is it? Redhat's
up2date isn't pulling rpms from there, are they?

Somebody is bundling the sources from apache.org and putting them into
formats and package repositories that particular distributions use (be
it rpm, dpkg, whatever the hell suse uses, etc.) so the various update
mechanisms can handle them best. Why wouldn't the 'gem' for solrb,
etc. live on RubyForge while the sources live at apache.org?

I guess I don't see how the 'apache way' excludes Solr hosting gems at
RubyForge when its flagship project is hosted in every repository
system out there.

-Ross.

On Feb 13, 2007, at 10:07 PM, Ross Singer wrote:
> I guess I don't see how the 'apache way' excludes Solr hosting gems at
> RubyForge when its flagship project is hosted in every repository
> system out there.

It doesn't exclude it at all... I'm simply exploring ways in which we
can use what we have here at Apache.

For more on the official Apache release policies, read here: <http://
www.apache.org/dev/release.html>

Erik

For what it's worth I would consider hosting a gem at rubyforge more
analogous to posting something to CPAN, which is certainly okay (at
least according to those pesky mod_perl/httpd hackers who have apreq2
in CPAN).

-Brian

On Feb 14, 2007, at 3:13 AM, Erik Hatcher wrote:

>
> On Feb 13, 2007, at 10:07 PM, Ross Singer wrote:
>> I guess I don't see how the 'apache way' excludes Solr hosting
>> gems at
>> RubyForge when its flagship project is hosted in every repository
>> system out there.
>
> It doesn't exclude it at all... I'm simply exploring ways in which
> we can use what we have here at Apache.
>
> For more on the official Apache release policies, read here:
> <http://www.apache.org/dev/release.html>
>
> Erik
>
>

On 2/13/07, Erik Hatcher <erik@ehatchersolutions.com> wrote:
> I spoke with Tom Copeland and Chad Fowler a bit about this topic at
> the Rails Edge event a few weeks ago. Serving gems from the Apache
> mirroring environment would work just fine, and once it records a
> source, it doesn't need to be specified for "gem update" I don't
> believe.

If that's the case (and those two would be the ones to know), then my
only remaining objection is that --source is ugly. Which, really,
isn't *that* big of a deal. ;-)

--
Coda Hale
http://blog.codahale.com