How to manage subprocess in Python when the output is huge? -

I want to control the long-running simulation (hours, days, weeks) using a script that runs on all the parameters. I

If only one simulation runs simultaneously, the output is piped to "T", otherwise the output is piped in the output file ">" All outputs are huge: some log files ~ 2 GB and it can also be bigger.

The script is working, but there is a hell to maintain. When we add a new parameter, it takes some time to customize the script and all of its sed-foo so I've put it on Python. It's working great.

The only problem is that I have now been stopped from using it in production that I can not find the exact way to call Popen () to launch the program. If I pipe everything to the file and not display any output, then by running "silence", gigabytes of python before simulation takes place.

Here is the code snippet:

  FH = open (logfile, "w") pid = subprances POPEAN (shake. Split (command), stdout = fh) pids.append (pid)  

I have read a lot of things about popen production, but what i do though it is a file If piping is required, will the buffer flush?

Maybe subprocene 'Popen () is not the best for this?

You can use to clear the dragon file buffer.

R ") f.read () # 'I like ham!' # Now open spam.txt in another program and add a new line F.read () # 'I like eggs too!'