Ok, I have been working on a website which is now available at
http://www.biddin.com/ that performs a search of multiple auction sites
all at once, and then returns the results. The problem is that it is
very slow, and it appears that the primary bottleneck is in waiting on
the search engines at the various sites to begin reading data.
Well, this makes sense because the code to implement the search is
currently pure sequential with no threading or non-blocking IO
whatsoever. This means that a search request on Amazon can only occur
after a successful completion of a search on EBay (which is notoriously
slow).
What I'm getting around to asking (slowly<g>) is, what is the best way
to implement an algorithm that can retrieve the data from ALL 3 sites
simultaneously, so that the code is never blocked up waiting on one
particular bad site? Note that it should also be possible to do
timeouts if search site is failing, and it should be easily extensible
to handle more than 3 simultaneous requests in the future. Also, keep
in mind that threading is my LAST possible choice as the Python
implementation at my presence provider doesn't support Threading. :)
Suggestions as to where I can read up on implementing this as a series
of forked processes, and whether or not this would be practical are
certainly welcome. Also, if anyone has any brilliant ideas for how it
could easily be done with the Select mechanism, I'd love to hear it.
---------------
Jesse D. Sightler
http://www3.pair.com/jsight/
PS - Not that it matters, but much of this code will probably be
released under GPL eventually. :)
http://www.biddin.com/ that performs a search of multiple auction sites
all at once, and then returns the results. The problem is that it is
very slow, and it appears that the primary bottleneck is in waiting on
the search engines at the various sites to begin reading data.
Well, this makes sense because the code to implement the search is
currently pure sequential with no threading or non-blocking IO
whatsoever. This means that a search request on Amazon can only occur
after a successful completion of a search on EBay (which is notoriously
slow).
What I'm getting around to asking (slowly<g>) is, what is the best way
to implement an algorithm that can retrieve the data from ALL 3 sites
simultaneously, so that the code is never blocked up waiting on one
particular bad site? Note that it should also be possible to do
timeouts if search site is failing, and it should be easily extensible
to handle more than 3 simultaneous requests in the future. Also, keep
in mind that threading is my LAST possible choice as the Python
implementation at my presence provider doesn't support Threading. :)
Suggestions as to where I can read up on implementing this as a series
of forked processes, and whether or not this would be practical are
certainly welcome. Also, if anyone has any brilliant ideas for how it
could easily be done with the Select mechanism, I'd love to hear it.
---------------
Jesse D. Sightler
http://www3.pair.com/jsight/
PS - Not that it matters, but much of this code will probably be
released under GPL eventually. :)