Mailing List Archive

Moshe Zadka <moshez@math.huji.ac.il> wrote:
> Whom do I contact about adding the following code to urllib:
> -------------->8-------------->8-------------------------
> class QuietFancyURLOpener(FancyURLOpener):
> def prompt_user_password(self, host, realm):
> return None, None
> -------------->8-------------->8-------------------------
> (Yes, I realize I can put it in my own program. But it seems a generally
> useful thing for non-interactive downloaders)

While we're at it... :-) - It might also be nice to have a FancyURLOpener
that raises an exception on an authentication mismatch. And one that takes
a user-defined user/password table dictionary or a callback function.

Yours,
Markus.

--
"That isn't the future I have in mind for the information highway: I want to
access more diverse connections and deeper thoughts, not cooler commercials
and new tolls." -- Geoff Dutton, http://photo.net/philg/humor/media-lab.html

I've been working on-and-off on a replacement for urllib that is more
easily customizable; they basic philosophy has been to avoid
inheritance as the customization mechanism. Instead, a single
director class coordinates the activities of many small handlers that
implement one feature/protocol.

The resulting architecture makes it much easier to create a UrlOpener
that handles redirects (which is probably the crucial feature of
FancyUrlOpener) without getting all of the other cruft (like prompting
the user to enter a password on stdin).

I've been delayed a bit because I wanted to add better support for
proxies, e.g. using a proxy only for URLs that match some pattern.
Since people are asking about this, I'll expedite my release plans
:-). I'll try to make a useful release available early next week.

Jeremy

FWIW, I've put a fairly complete spidering class together as part of a
project... may be useful for somebody out there...


http://www.mnot.net/cacheability/download.html


----- Original Message -----
From: Jeremy Hylton <jeremy@cnri.reston.va.us>
Newsgroups: comp.lang.python
To: <python-list@cwi.nl>
Sent: Sunday, June 13, 1999 3:24
Subject: Re: urllib: an alternative URLOpener


> I've been working on-and-off on a replacement for urllib that is more
> easily customizable; they basic philosophy has been to avoid
> inheritance as the customization mechanism. Instead, a single
> director class coordinates the activities of many small handlers that
> implement one feature/protocol.
>
> The resulting architecture makes it much easier to create a UrlOpener
> that handles redirects (which is probably the crucial feature of
> FancyUrlOpener) without getting all of the other cruft (like prompting
> the user to enter a password on stdin).
>
> I've been delayed a bit because I wanted to add better support for
> proxies, e.g. using a proxy only for URLs that match some pattern.
> Since people are asking about this, I'll expedite my release plans
> :-). I'll try to make a useful release available early next week.
>
> Jeremy
>
>
>
>
>

Ok. There is now an alpha version of urllib2 available from my Web
page: http://the-tech.mit.edu/~jeremy/python/

I wrote this module because the current urllib is too hard to
customize. It was difficult to use only a subset of the basic
functionality in FancyURLopener and the error reporting was
problematic. I hope to have addressed both these issues with urllib2.

This code has not received extensive testing. I expect that there are
a number of lurking bugs. There is too much diversity on the Web for
my own testing to have covered all the possible cases. I'd appreciate
bug reports, comments on the design, requests for new features, etc.
Most of all, I need suggestions for a name; urllib2 is a clunker.

I expect to make one or two more alpha releases, as I finalize the
feature set. I've got no timetable in mind, although I have a
vacation in August, which would be an appropriate time to make a final
release.

Jeremy

On Mon, 14 Jun 1999 11:42:14 -0400 (EDT), Jeremy Hylton
<jeremy@cnri.reston.va.us> wrote:

>Ok. There is now an alpha version of urllib2 available from my Web
>page: http://the-tech.mit.edu/~jeremy/python/

Please forgive the naive question, but how does one get the "realm" info
from a site?

I want to connect to a page that requires logging in with username and
password and then retrieve the data there, but it seems I have to know the
realm info as well?

Dave Cook

In article <slrn7mef1g.1b0.davecook@sputnik.escnd1.sdca.home.com>,
David M. Cook <davecook@home.com> wrote:
>
>Please forgive the naive question, but how does one get the "realm" info
>from a site?

IIRC, the challenge contains that info. urllib should be handling this
automatically; if you want to see what it is directly, I don't know if
there's a Python mechanism, but you can do something like this:

% telnet www.foo.com 80
GET / HTTP/1.0\n
\n
--
--- Aahz (@netcom.com)

Hugs and backrubs -- I break Rule 6 <*> http://www.rahul.net/aahz/
Androgynous poly kinky vanilla queer het

"That doesn't stop me from wanting to work all three of them over with
the clue stick for while, with no safewords allowed." --abostick