Das deutsche Python-Forum

Because of the GIL, thread pools are not as useful in Python as you
might expect -- they execute one at a time and do not take advantage of
hyper-threading or multiple processors. If that kind of efficiency is
what you are after, then take a look at PyLinda which can coordinate
communication between multiple instances of Python running in separate
threads

#!/usr/bin/env python
# -*- coding: iso-8859-15 -*-

import urllib2
import threading
import time


lock = threading.Lock()


def download(url, retvals):
   
    try:
        html = urllib2.urlopen(url).read()
    except urllib2.URLError, exception:
        html = "ERROR: %s" % str(exception)
    lock.acquire()
    retvals.append((url, html))
    lock.release()


def main():
    urls = (
        "http://gerold.bcom.at/",
        "http://halvar.at/",
        "http://lugt.at/",
        "http://microsoft.de/",
        "http://www.python-forum.de/",
        "http://python.org/",
        "http://wallerforum.com/",
        #"http://log-in.fachdid.fu-berlin.de/Archiv/2001/2/Thema/", # braucht zu lange
    )
    retvals = []
    threads = []
    for url in urls:
        threads.append(
            threading.Thread(
                target = download, kwargs = dict(url = url, retvals = retvals)
            )
        )
        threads[-1].start()
        time.sleep(0.2)
   
    # Warten bis alle fertig sind
    for thread in threads:
        thread.join()
       
    for url, html in retvals:
        print url
        if html.startswith("ERROR"):
            print "Achtung Fehler!"
        print html[:100]
        print


if __name__ == "__main__":
    main()

Rebecca hat geschrieben:Multithreading und Multiprocessing ist sinnvoll fuer CPU-lastige nebenlaeufige Programme, fuer IO-lastige Programme aber eventuell nicht optimal. Es gibt noch eine andere Moeglichkeit! Ich zitiere mal die Doku vom asyncore-Modul:

There are only two ways to have a program on a single processor do ``more than one thing at a time.'' Multi-threaded programming is the simplest and most popular way to do it, but there is another very different technique, that lets you have nearly all the advantages of multi-threading, without actually using multiple threads. It's really only practical if your program is largely I/O bound. If your program is processor bound, then pre-emptive scheduled threads are probably what you really need. Network servers are rarely processor bound, however.

If your operating system supports the select() system call in its I/O library (and nearly all do), then you can use it to juggle multiple communication channels at once; doing other work while your I/O is taking place in the ``background.'' Although this strategy can seem strange and complex, especially at first, it is in many ways easier to understand and control than multi-threaded programming. The asyncore module solves many of the difficult problems for you, making the task of building sophisticated high-performance network servers and clients a snap. For ``conversational'' applications and protocols the companion asynchat module is invaluable.

debian75 hat geschrieben:im moment starte ich gleich viele threads wie ich urls habe

In Example #3, we're using the urllib module, which uses blocking sockets under the covers to make HTTP connections. The socket module releases the GIL, so both threads in example 3b can run simultaneously. Therefore, the single-threaded version (3a) is going to run slower in this case.

Leonidas hat geschrieben:`urlopen` killt da nicht den GIL (wie bist du auf die Formulierung gekommen), sondern es benutzt das `sockets`-Modul, welches in C geschrieben ist. Der GIL betrifft nur Python-Module, keine C-Module.

BlackJack hat geschrieben:Der "spawnt" danach aber $GOTT sei Dank wieder. Ohne den gäb's nämlich ganz schönes Chaos.