otis 2002/10/30 12:43:45
Modified: docs/lucene-sandbox/larm overview.html
Log:
- Updated.
Revision Changes Path
1.2 +16 -15 jakarta-lucene/docs/lucene-sandbox/larm/overview.html
Index: overview.html
===================================================================
RCS file: /home/cvs/jakarta-lucene/docs/lucene-sandbox/larm/overview.html,v
retrieving revision 1.1
retrieving revision 1.2
diff -u -r1.1 -r1.2
--- overview.html 30 Oct 2002 18:33:43 -0000 1.1
+++ overview.html 30 Oct 2002 20:43:45 -0000 1.2
@@ -361,15 +361,15 @@
<p>
The LARM web crawler is a result of experiences with the errors as
mentioned above, connected with a lot of monitoring to get the maximum out
- of the given system ressources. It was designed with several different
+ of the given system resources. It was designed with several different
aspects in mind:
</p>
<ul>
<li>Speed. This involves balancing the resources to prevent
- bottlenecks. The crawler is multithreaded. A lot of work went in avoiding
+ bottlenecks. The crawler is multi-threaded. A lot of work went in avoiding
synchronization between threads, i.e. by rewriting or replacing the standard
- Java classes, which slows down multithreaded programs a lot
+ Java classes, which slows down multi-threaded programs a lot
</li>
@@ -384,13 +384,13 @@
<li>Scalability. The crawler was supposed to be able to crawl <i>large
intranets</i> with hundreds of servers and hundreds of thousands of
- documents within a reasonable amount of time. It was not ment to be
+ documents within a reasonable amount of time. It was not meant to be
scalable to the whole Internet.</li>
<li>Java. Although there are many crawlers around at the time when I
started to think about it (in Summer 2000), I couldn't find a good
available implementation in Java. If this crawler would have to be integrated
- in a Java search engine, a homogenous system would be an advantage. And
+ in a Java search engine, a homogeneous system would be an advantage. And
after all, I wanted to see if a fast implementation could be done in
this language.
</li>
@@ -452,7 +452,7 @@
class to the pipeline.
</li><li>The storage mechanism is also pluggable. One of the next
issues would be to include this storage mechanism into the pipeline, to
- allow a seperation of logging, processing, and storage
+ allow a separation of logging, processing, and storage
</li>
</ul>
<p>
@@ -468,7 +468,7 @@
Still, there's a relatively high memory overhead <i>per server</i> and
also some overhead <i>per page</i>, especially since a map of already
crawled pages is held in memory at this time. Although some of the
- in-memory structures are already put on disc, memory consumption is still
+ in-memory structures are already put on disk, memory consumption is still
linear to the number of pages crawled. We have lots of ideas on how to
move that out, but since then, as an example with 500 MB RAM, the crawler
scales up to some 100.000 files on some 100s of hosts.
@@ -772,7 +772,7 @@
that dots "." were not supported in mapped properties indexes. As with
the new version (1.5 at the time of this writing) this is supposed to be
removed, but I have not tried yet. Therefore, the comma "," was made a
- synonymon for dots. Since "," is not allowed in domain names, you can
+ synonym for dots. Since "," is not allowed in domain names, you can
still use (and even mix) them if you want, or if you only have an older
BeanUtils version available.
</p>
@@ -784,7 +784,7 @@
<p>
LARM currently provides a very simple LuceneStorage that allows for
- integrating the crawler with Lucene. It's ment to be a working example on
+ integrating the crawler with Lucene. It's meant to be a working example on
how this can be accomplished, not a final implementation. If you like
to volunteer on that part, contributions are welcome.</p>
@@ -893,7 +893,7 @@
Probably it is copied through several buffers until it is complete.
This will take some CPU time, but mostly it will wait for the next packet
to arrive. The network transfer by itself is also affected by a lot of
- factors, i.e. the speed of the web server, acknowledgement messages,
+ factors, i.e. the speed of the web server, acknowledgment messages,
resent packages etc. so 100% network utilization will almost never be
reached.</li>
<li>The document is processed, which will take up the whole CPU. The
@@ -941,7 +941,7 @@
this until you have read the standard literature and have made at least
10 mistakes (and solved them!).</p>
<p>
- Multithreading doesn't come without a cost, however. First, there is
+ Multi-threading doesn't come without a cost, however. First, there is
the cost of thread scheduling. I don't have numbers for that in Java, but
I suppose that this should not be very expensive. MutExes can affect
the whole program a lot . I noticed that they should be avoided like
@@ -1171,7 +1171,7 @@
<p>
In the first implementation the fetcher would simply distribute the
incoming URLs to the threads. The thread pool would use a simple queue to
- store the remaining tasks. But this can lead to a very "unpolite"
+ store the remaining tasks. But this can lead to a very "impolite"
distribution of the tasks: Since ¾ of the links in a page point to the same
server, and all links of a page are added to the message handler at
once, groups of successive tasks would all try to access the same server,
@@ -1210,7 +1210,7 @@
already resolved host names.</li>
<li>the crawler itself was designed to crawl large local networks, and
- not the internet. Thus, the number of hosts is very limited.</li>
+ not the Internet. Thus, the number of hosts is very limited.</li>
</ul>
</blockquote>
@@ -1326,8 +1326,9 @@
</li>
</ul>
<p>
- One thing to keep in mind is that the number of URLs transferred to
- other nodes should be as large as possible. </p>
+ One thing to keep in mind is that the transfer of URLs to
+ other nodes should be done in batches with hundreds or
+ thousands or more URLs per batch.</p>
<p>
The next thing to be distributed is the storage mechanism. Here, the
number of pure crawling nodes and the number of storing (post processing)
--
To unsubscribe, e-mail: <mailto:lucene-dev-unsubscribe@jakarta.apache.org>
For additional commands, e-mail: <mailto:lucene-dev-help@jakarta.apache.org>
Modified: docs/lucene-sandbox/larm overview.html
Log:
- Updated.
Revision Changes Path
1.2 +16 -15 jakarta-lucene/docs/lucene-sandbox/larm/overview.html
Index: overview.html
===================================================================
RCS file: /home/cvs/jakarta-lucene/docs/lucene-sandbox/larm/overview.html,v
retrieving revision 1.1
retrieving revision 1.2
diff -u -r1.1 -r1.2
--- overview.html 30 Oct 2002 18:33:43 -0000 1.1
+++ overview.html 30 Oct 2002 20:43:45 -0000 1.2
@@ -361,15 +361,15 @@
<p>
The LARM web crawler is a result of experiences with the errors as
mentioned above, connected with a lot of monitoring to get the maximum out
- of the given system ressources. It was designed with several different
+ of the given system resources. It was designed with several different
aspects in mind:
</p>
<ul>
<li>Speed. This involves balancing the resources to prevent
- bottlenecks. The crawler is multithreaded. A lot of work went in avoiding
+ bottlenecks. The crawler is multi-threaded. A lot of work went in avoiding
synchronization between threads, i.e. by rewriting or replacing the standard
- Java classes, which slows down multithreaded programs a lot
+ Java classes, which slows down multi-threaded programs a lot
</li>
@@ -384,13 +384,13 @@
<li>Scalability. The crawler was supposed to be able to crawl <i>large
intranets</i> with hundreds of servers and hundreds of thousands of
- documents within a reasonable amount of time. It was not ment to be
+ documents within a reasonable amount of time. It was not meant to be
scalable to the whole Internet.</li>
<li>Java. Although there are many crawlers around at the time when I
started to think about it (in Summer 2000), I couldn't find a good
available implementation in Java. If this crawler would have to be integrated
- in a Java search engine, a homogenous system would be an advantage. And
+ in a Java search engine, a homogeneous system would be an advantage. And
after all, I wanted to see if a fast implementation could be done in
this language.
</li>
@@ -452,7 +452,7 @@
class to the pipeline.
</li><li>The storage mechanism is also pluggable. One of the next
issues would be to include this storage mechanism into the pipeline, to
- allow a seperation of logging, processing, and storage
+ allow a separation of logging, processing, and storage
</li>
</ul>
<p>
@@ -468,7 +468,7 @@
Still, there's a relatively high memory overhead <i>per server</i> and
also some overhead <i>per page</i>, especially since a map of already
crawled pages is held in memory at this time. Although some of the
- in-memory structures are already put on disc, memory consumption is still
+ in-memory structures are already put on disk, memory consumption is still
linear to the number of pages crawled. We have lots of ideas on how to
move that out, but since then, as an example with 500 MB RAM, the crawler
scales up to some 100.000 files on some 100s of hosts.
@@ -772,7 +772,7 @@
that dots "." were not supported in mapped properties indexes. As with
the new version (1.5 at the time of this writing) this is supposed to be
removed, but I have not tried yet. Therefore, the comma "," was made a
- synonymon for dots. Since "," is not allowed in domain names, you can
+ synonym for dots. Since "," is not allowed in domain names, you can
still use (and even mix) them if you want, or if you only have an older
BeanUtils version available.
</p>
@@ -784,7 +784,7 @@
<p>
LARM currently provides a very simple LuceneStorage that allows for
- integrating the crawler with Lucene. It's ment to be a working example on
+ integrating the crawler with Lucene. It's meant to be a working example on
how this can be accomplished, not a final implementation. If you like
to volunteer on that part, contributions are welcome.</p>
@@ -893,7 +893,7 @@
Probably it is copied through several buffers until it is complete.
This will take some CPU time, but mostly it will wait for the next packet
to arrive. The network transfer by itself is also affected by a lot of
- factors, i.e. the speed of the web server, acknowledgement messages,
+ factors, i.e. the speed of the web server, acknowledgment messages,
resent packages etc. so 100% network utilization will almost never be
reached.</li>
<li>The document is processed, which will take up the whole CPU. The
@@ -941,7 +941,7 @@
this until you have read the standard literature and have made at least
10 mistakes (and solved them!).</p>
<p>
- Multithreading doesn't come without a cost, however. First, there is
+ Multi-threading doesn't come without a cost, however. First, there is
the cost of thread scheduling. I don't have numbers for that in Java, but
I suppose that this should not be very expensive. MutExes can affect
the whole program a lot . I noticed that they should be avoided like
@@ -1171,7 +1171,7 @@
<p>
In the first implementation the fetcher would simply distribute the
incoming URLs to the threads. The thread pool would use a simple queue to
- store the remaining tasks. But this can lead to a very "unpolite"
+ store the remaining tasks. But this can lead to a very "impolite"
distribution of the tasks: Since ¾ of the links in a page point to the same
server, and all links of a page are added to the message handler at
once, groups of successive tasks would all try to access the same server,
@@ -1210,7 +1210,7 @@
already resolved host names.</li>
<li>the crawler itself was designed to crawl large local networks, and
- not the internet. Thus, the number of hosts is very limited.</li>
+ not the Internet. Thus, the number of hosts is very limited.</li>
</ul>
</blockquote>
@@ -1326,8 +1326,9 @@
</li>
</ul>
<p>
- One thing to keep in mind is that the number of URLs transferred to
- other nodes should be as large as possible. </p>
+ One thing to keep in mind is that the transfer of URLs to
+ other nodes should be done in batches with hundreds or
+ thousands or more URLs per batch.</p>
<p>
The next thing to be distributed is the storage mechanism. Here, the
number of pure crawling nodes and the number of storing (post processing)
--
To unsubscribe, e-mail: <mailto:lucene-dev-unsubscribe@jakarta.apache.org>
For additional commands, e-mail: <mailto:lucene-dev-help@jakarta.apache.org>