Datenübergabe an STDIN eines neuen Prozesses

[jens]

Ach doch... jetzt scheint es zu gehen... Hab mal aus newdata nur data gemacht und einen counter eingebaut... Zumindest in der Konsole geht's richtig:

Writing data to subprocess: ['Counter1:', 0, 'Counter2:', 1]
Subprocess started!
Received in subprocess: ['Counter1:', 0, 'Counter2:', 1]
Got data from subprocess: ['Counter1:', 1, 'Counter2:', 1]
Writing data to subprocess: ['Counter1:', 1, 'Counter2:', 2]
Received in subprocess: ['Counter1:', 1, 'Counter2:', 2]
Got data from subprocess: ['Counter1:', 2, 'Counter2:', 2]
Writing data to subprocess: ['Counter1:', 2, 'Counter2:', 3]
Received in subprocess: ['Counter1:', 2, 'Counter2:', 3]
Got data from subprocess: ['Counter1:', 3, 'Counter2:', 3]
Writing data to subprocess: ['Counter1:', 3, 'Counter2:', 4]
Received in subprocess: ['Counter1:', 3, 'Counter2:', 4]
Got data from subprocess: ['Counter1:', 4, 'Counter2:', 4]
Writing data to subprocess: ['Counter1:', 4, 'Counter2:', 5]
Received in subprocess: ['Counter1:', 4, 'Counter2:', 5]
Got data from subprocess: ['Counter1:', 5, 'Counter2:', 5]
Writing data to subprocess: ['Counter1:', 5, 'Counter2:', 6]
Received in subprocess: ['Counter1:', 5, 'Counter2:', 6]
Got data from subprocess: ['Counter1:', 6, 'Counter2:', 6]
Writing data to subprocess: ['Counter1:', 6, 'Counter2:', 7]
Received in subprocess: ['Counter1:', 6, 'Counter2:', 7]
Got data from subprocess: ['Counter1:', 7, 'Counter2:', 7]
Writing data to subprocess: ['Counter1:', 7, 'Counter2:', 8]
Received in subprocess: ['Counter1:', 7, 'Counter2:', 8]
Got data from subprocess: ['Counter1:', 8, 'Counter2:', 8]
Writing data to subprocess: ['Counter1:', 8, 'Counter2:', 9]
Received in subprocess: ['Counter1:', 8, 'Counter2:', 9]
Got data from subprocess: ['Counter1:', 9, 'Counter2:', 9]
Writing data to subprocess: ['Counter1:', 9, 'Counter2:', 10]
Received in subprocess: ['Counter1:', 9, 'Counter2:', 10]
Got data from subprocess: ['Counter1:', 10, 'Counter2:', 10]
Parent has closed connection. Terminating.
Closed stdin and out for subprocess. Terminating.

In SciTE kommt reproduzierbar dieses:

Writing data to subprocess: ['Counter1:', 0, 'Counter2:', 1]
Got data from subprocess: ['Counter1:', 0, 'Counter2:', 1]
Writing data to subprocess: ['Counter1:', 0, 'Counter2:', 2]
Got data from subprocess: ['Counter1:', 0, 'Counter2:', 2]
Writing data to subprocess: ['Counter1:', 0, 'Counter2:', 3]
Got data from subprocess: ['Counter1:', 0, 'Counter2:', 3]
Writing data to subprocess: ['Counter1:', 0, 'Counter2:', 4]
Got data from subprocess: ['Counter1:', 0, 'Counter2:', 4]
Writing data to subprocess: ['Counter1:', 0, 'Counter2:', 5]
Got data from subprocess: ['Counter1:', 0, 'Counter2:', 5]
Writing data to subprocess: ['Counter1:', 0, 'Counter2:', 6]
Got data from subprocess: ['Counter1:', 0, 'Counter2:', 6]
Writing data to subprocess: ['Counter1:', 0, 'Counter2:', 7]
Got data from subprocess: ['Counter1:', 0, 'Counter2:', 7]
Writing data to subprocess: ['Counter1:', 0, 'Counter2:', 8]
Got data from subprocess: ['Counter1:', 0, 'Counter2:', 8]
Writing data to subprocess: ['Counter1:', 0, 'Counter2:', 9]
Got data from subprocess: ['Counter1:', 0, 'Counter2:', 9]
Writing data to subprocess: ['Counter1:', 0, 'Counter2:', 10]
Got data from subprocess: ['Counter1:', 0, 'Counter2:', 10]
Closed stdin and out for subprocess. Terminating.
Subprocess started!
Received in subprocess: ['Counter1:', 0, 'Counter2:', 1]
Received in subprocess: ['Counter1:', 0, 'Counter2:', 2]
Received in subprocess: ['Counter1:', 0, 'Counter2:', 3]
Received in subprocess: ['Counter1:', 0, 'Counter2:', 4]
Received in subprocess: ['Counter1:', 0, 'Counter2:', 5]
Received in subprocess: ['Counter1:', 0, 'Counter2:', 6]
Received in subprocess: ['Counter1:', 0, 'Counter2:', 7]
Received in subprocess: ['Counter1:', 0, 'Counter2:', 8]
Received in subprocess: ['Counter1:', 0, 'Counter2:', 9]
Received in subprocess: ['Counter1:', 0, 'Counter2:', 10]
Parent has closed connection. Terminating.

Hier mal der modizifierte Code:
test1.py

Code: Alles auswählen

import sys, pickle

# Daten lesen und gleich wieder dumpen.
sys.stderr.write("Subprocess started!\n")
while True:
    try:
        data = pickle.load(sys.stdin)
        sys.stderr.write("Received in subprocess: %r\n" % data)
    except EOFError:
        sys.stderr.write("Parent has closed connection. Terminating.\n")
        break

    data[1] = data[1] + 1 # verändern des Counter1

    pickle.dump(data,sys.stdout, protocol=pickle.HIGHEST_PROTOCOL)
    sys.stdout.flush()

test2.py

Code: Alles auswählen

import sys, pickle, subprocess

proc = subprocess.Popen(["python","test1.py"],0,None,
                        subprocess.PIPE,subprocess.PIPE,None,
                        universal_newlines=True)

data = ["Counter1:",0,"Counter2:",0]
for in range(10):

    data[3] = data[3] + 1 # verändern des Counter2

    print "Writing data to subprocess:", data
    pickle.dump(data,proc.stdin, protocol=pickle.HIGHEST_PROTOCOL)
    data = pickle.load(proc.stdout)
    print "Got data from subprocess:", data

proc.stdin.close()
proc.stdout.close()

print "Closed stdin and out for subprocess. Terminating."

[jens]

Informationen dazu gibts unter:

http://www.python.org/doc/faq/windows.h ... t-or-win95

Das betrifft allerdings nur Win9x + NT... Ich denke mal unter 2000/XP sieht's anders aus

[modelnine]

Das sollte eigentlich nicht gehen, aber egal. Und: unter XP/2000 sieht's nicht anders aus, die Verhalten sich genau wie NT.

Der wichtige Unterschied zwischen meinem letzten Post und dem ersten war dass man python mit "-u" aufgerufen hat als Parameter, siehe dazu auch den geposteten Link. Der bewirkt nämlich dass der Unterprozess (also python) stdin und out im binary mode aufmacht, sonst sind die nämlich text-mode, und dann kommt so scheiße wie \n -> \r\n o.Ä. rein.

universal_newlines braucht man dann logischerweise nicht, darf man sogar nicht setzen.

jens: guck Dir also noch mal genau mein Beispiel an und modifizier dann Deines soweit, das ist die richtige Lösung.

--- Heiko.

[jens]

Also ich hab es jetzt nochmal getestet... Es ist egal ob man dem Interpreter mit -u startet und/oder universal_newlines=True ist

Ist kein -u und kein universal_newlines=True dann kommt es zu den Fehlern!

EDIT: Anders sieht es mit SciTE aus... Wenn das das "-u" fehlt, dann kommen die Ausgaben durcheinader, wie ich es gepostet hab... Mit "-u" ist alles genau so, wie es soll... Somit ist "-u" doch ein muß

[jens]

Also hier nochmal der Code, der anscheinent überall klappt:
test2.py

Code: Alles auswählen

#!/usr/bin/python
# -*- coding: UTF-8 -*-

import sys, pickle, subprocess

proc = subprocess.Popen(["python","-u", "test1.py"],0,None,
                        subprocess.PIPE,subprocess.PIPE,None)

data = ["Counter1:",0,"Counter2:",0]
for i in range(10):

    data[3] = data[3] + 1 # verändern des Counter2

    print "Writing data to subprocess:", data
    pickle.dump(data,proc.stdin, pickle.HIGHEST_PROTOCOL)
    data = pickle.load(proc.stdout)
    print "Got data from subprocess:", data

proc.stdin.close()
proc.stdout.close()

print "Closed stdin and out for subprocess. Terminating."

test1.py

Code: Alles auswählen

#!/usr/bin/python
# -*- coding: UTF-8 -*-

import sys, pickle

# Daten lesen und gleich wieder dumpen.
sys.stderr.write("Subprocess started!\n")
while True:
    try:
        data = pickle.load(sys.stdin)
        sys.stderr.write("Received in subprocess: %r\n" % data)
    except EOFError:
        sys.stderr.write("Parent has closed connection. Terminating.\n")
        break

    data[1] = data[1] + 1 # verändern des Counter1

    pickle.dump(data,sys.stdout, pickle.HIGHEST_PROTOCOL)
    sys.stdout.flush()

Die Frage ist dennoch, ob sowas Sinnvoll ist... Denn es wird wahrscheinlich schweg mit dem debuggen, denke ich...

[modelnine]

Dass keine Fehler kommen liegt schlicht und ergreifend daran dass Du keine Daten transferierst die möglicherweise durch die Uminterpretierung von \n -> \r\n -> \n kaputt gehen. Nimm -u.

--- Heiko.

[modelnine]

Noch eine Möglichkeit, direkt aus dem MoinMoin Quellcode geklaut (nach dezentem Hinweis im IRC) und in test1.py einzubauen:

Code: Alles auswählen

# force input/output to binary
if sys.platform == "win32":
    import msvcrt
    msvcrt.setmode(sys.stdin.fileno(), os.O_BINARY)
    msvcrt.setmode(sys.stdout.fileno(), os.O_BINARY)

Ist in gewisser Weise noch "portabler", weil ich ehrlich gesagt nicht weiß inwiefern unter Unix python das "-u" Flag mag. Es sagt zwar nix wenn man es angibt, aber ich weiß nicht inwiefern das andere Dinge verändert.

--- Heiko.

[gerold]

Hi Heiko!

Das wars. Es funktioniert perfekt, wenn man den Code (msvcrt) in "test1.py" einbaut.

lg
Gerold