Mailing List Archive: crawling protected pages

crawling protected pages

Sep 11, 2005, 1:16 PM

Post #1 of 2 (1820 views)

Hi,

I need to crawl a site that is protected. Is there currently any way to do
this with nutch (i.e. providing a username/password) ?

Thanks,

Ed.

Re: crawling protected pages [ In reply to ]

otis_gospodnetic at yahoo

Sep 11, 2005, 9:45 PM

Post #2 of 2 (1669 views)

Permalink

I don't think there is. I assume you are considering using Nutch. If
so, use nutch-user mailing list instead. You'll get more help there.

Otis

--- Edward Quick <edwardquick@hotmail.com> wrote:

> Hi,
>
> I need to crawl a site that is protected. Is there currently any way
> to do
> this with nutch (i.e. providing a username/password) ?
>
> Thanks,
>
> Ed.
>
>
>