Mailing List Archive

It seems that Nutch and Solr would benefit from a shared index serving
infrastructure. Other Lucene-based projects might also benefit from
this. So perhaps we should start a new project to build such a thing.
This could start either in java/contrib, or as a separate sub-project,
depending on interest.

Here are some quick ideas about how this might work.

An RPC mechanism would be used to communicate between nodes (probably
Hadoop's). The system would be configured with a single master node
that keeps track of where indexes are located, and a number of slave
nodes that would maintain, search and replicate indexes. Clients would
talk to the master to find out which indexes to search or update, then
they'll talk directly to slaves to perform searches and updates.

Following is an outline of how this might look.

We assume that, within an index, a file with a given name is written
only once. Index versions are sets of files, and a new version of an
index is likely to share most files with the prior version. Versions
are numbered. An index server should keep old versions of each index
for a while, not immediately removing old files.

public class IndexVersion {
String Id; // unique name of the index
int version; // the version of the index
}

public class IndexLocation {
IndexVersion indexVersion;
InetSocketAddress location;
}

public interface ClientToMasterProtocol {
IndexLocation[] getSearchableIndexes();
IndexLocation getUpdateableIndex(String id);
}

public interface ClientToSlaveProtocol {
// normal update
void addDocument(String index, Document doc);
int[] removeDocuments(String index, Term term);
void commitVersion(String index);

// batch update
void addIndex(String index, IndexLocation indexToAdd);

// search
SearchResults search(IndexVersion i, Query query, Sort sort, int n);
}

public interface SlaveToMasterProtocol {
// sends currently searchable indexes
// recieves updated indexes that we should replicate/update
public IndexLocation[] heartbeat(IndexVersion[] searchableIndexes);
}

public interface SlaveToSlaveProtocol {
String[] getFileSet(IndexVersion indexVersion);
byte[] getFileContent(IndexVersion indexVersion, String file);
// based on experience in Hadoop, we probably wouldn't really use
// RPC to send file content, but rather HTTP.
}

The master thus maintains the set of indexes that are available for
search, keeps track of which slave should handle changes to an index and
initiates index synchronization between slaves. The master can be
configured to replicate indexes a specified number of times.

The client library can cache the current set of searchable indexes and
periodically refresh it. Searches are broadcast to one index with each
id and return merged results. The client will load-balance both
searches and updates.

Deletions could be broadcast to all slaves. That would probably be fast
enough. Alternately, indexes could be partitioned by a hash of each
document's unique id, permitting deletions to be routed to the
appropriate slave.

Does this make sense? Does it sound like it would be useful to Solr?
To Nutch? To others? Who would be interested and able to work on it?

Doug