• 15
name

A PHP Error was encountered

Severity: Notice

Message: Undefined index: userid

Filename: views/question.php

Line Number: 191

Backtrace:

File: /home/prodcxja/public_html/questions/application/views/question.php
Line: 191
Function: _error_handler

File: /home/prodcxja/public_html/questions/application/controllers/Questions.php
Line: 433
Function: view

File: /home/prodcxja/public_html/questions/index.php
Line: 315
Function: require_once

Question: Is there a way to use flush=True for the print() function without getting the BrokenPipeError?

I have a script pipe.py:

for i in range(4000):
    print(i)

I call it like this from a Unix command line:

python3 pipe.py | head -n3000

And it returns:

0
1
2

So does this script:

import sys
for i in range(4000):
    print(i)
    sys.stdout.flush()

However, when I run this script and pipe it to head -n3000:

for i in range(4000):
    print(i, flush=True)

Then I get this error:

    print(i, flush=True)
BrokenPipeError: [Errno 32] Broken pipe
Exception BrokenPipeError: BrokenPipeError(32, 'Broken pipe') in <_io.TextIOWrapper name='<stdout>' mode='w' encoding='UTF-8'> ignored

I have also tried the solution below, but I still get the BrokenPipeError:

import sys
for i in range(4000):
    try:
        print(i, flush=True)
    except BrokenPipeError:
        sys.exit()
    • I am unable to reproduce the exception by running your script in Python 3.4.1. Which Python version are you using?
      • 1
    • I have tried with version 3.4.1 on Mac OS X 10.9.4 and version 3.3.2 on Ubuntu 12.04.2 LTS. I am currently unable to reproduce the error myself. I had to change range(4) to range(4000) and head -n3 to head -n3000 to reproduce the error.
      • 2
    • I just tried on OS X 10.9.4 and I was not able to reproduce it. I got the error on Ubuntu 12.04.2 LTS. I will try on Linux Mint Qiana.

The BrokenPipeError is normal as said phantom because the reading process (head) terminates and closes its end of the pipe while the writing process (python) still tries to write.

Is is an abnormal condition, and the python scripts receives a BrokenPipeError - more exactly, the Python interpreter receives a system SIGPIPE signal that it catches and raises the BrokenPipeError to allow the script to process the error.

And you effectively can process the error, because in your last example, you only see a message saying that the exception was ignored - ok it is not true, but seems related to this open issue in Python : Python developpers think important to warn user of the abnormal condition.

What really happens is that AFAIK the python interpreter always signals this on stderr, even if you catch the exception. But you just have to close stderr before exiting to get rid of the message.

I slightly changed your script to :

  • catch the error as you did in your last example
  • catch either IOError (that I get in Python34 on Windows64) or BrokenPipeError (in Python 33 on FreeBSD 9.0) - and display a message for that
  • display a custom Done message on stderr (stdout is closed due to the broken pipe)
  • close stderr before exiting to get rid of the message

Here is the script I used :

import sys

try:
    for i in range(4000):
            print(i, flush=True)
except (BrokenPipeError, IOError):
    print ('BrokenPipeError caught', file = sys.stderr)

print ('Done', file=sys.stderr)
sys.stderr.close()

and here the result of python3.3 pipe.py | head -10 :

0
1
2
3
4
5
6
7
8
9
BrokenPipeError caught
Done

If you do not want the extraneous messages just use :

import sys

try:
    for i in range(4000):
            print(i, flush=True)
except (BrokenPipeError, IOError):
    pass

sys.stderr.close()
  • 36
Reply Report

A note on SIGPIPE was added in Python 3.7 documentation, and it recommends to catch BrokenPipeError this way:

import os
import sys

def main():
    try:
        # simulate large output (your code replaces this loop)
        for x in range(10000):
            print("y")
        # flush output here to force SIGPIPE to be triggered
        # while inside this try block.
        sys.stdout.flush()
    except BrokenPipeError:
        # Python flushes standard streams on exit; redirect remaining output
        # to devnull to avoid another BrokenPipeError at shutdown
        devnull = os.open(os.devnull, os.O_WRONLY)
        os.dup2(devnull, sys.stdout.fileno())
        sys.exit(1)  # Python exits with error code 1 on EPIPE

if __name__ == '__main__':
    main()

Importantly, it says:

Do not set SIGPIPE’s disposition to SIG_DFL in order to avoid BrokenPipeError. Doing that would cause your program to exit unexpectedly also whenever any socket connection is interrupted while your program is still writing to it.

  • 11
Reply Report
      • 1
    • This seems to be most up-to-date answer and it's also the approach recommended in the official documentation. Tested on Mac OS 10.14 / Python 3.7.5

According to the Python documentation, this is thrown when:

trying to write on a pipe while the other end has been closed

This is due to the fact that the head utility reads from stdout, then promptly closes it.

As you can see, it can be worked around by merely adding a sys.stdout.flush() after every print(). Note that this sometimes does not work in Python 3.

You can alternatively pipe it to awk like this to get the same result as head -3:

python3 0to3.py | awk 'NR >= 4 {exit} 1'

Hope this helped, good luck!

  • 4
Reply Report
      • 2
    • @tommy.carstensen It wouldn't be, because it's different depending on the program you pipe the output into. The reason this error occurs is because when the program you pipe output into closes before your python script, it closes the output pipe, e.g. stdout. You are better off either manually piping with subprocess or using python's regex capabilities instead of grep.
    • @tommy.carstensen I am focusing on it. I'm telling you it's not possible. I quote The reason this error occurs is because when the program you pipe output into closes before your python script, it closes the output pipe, e.g. stdout.
    • Thanks for this. Unfortunately the awk workaround is not an option. I don't know how many lines the real output produces. I'll leave the question unanswered for a few days, if you don't mind. Thanks again.
      • 1
    • @tommy.carstensen Does sys.stdout.flush() not work? Also, with head don't you also have to know how many lines it will be anyway?
      • 1
    • @tommy.carstensen It is impossible to do what you are asking. There is no workaround to enable head to work with your program in this manner.

Ignore SIGPPIE temporarily

I'm not sure how bad an idea this is, but it works:

#!/usr/bin/env python3

import signal
import sys

sigpipe_old = signal.getsignal(signal.SIGPIPE)
signal.signal(signal.SIGPIPE, signal.SIG_DFL)
for i in range(4000):
    print(i, flush=True)
signal.signal(signal.SIGPIPE, sigpipe_old)
  • 2
Reply Report

As you can see in the output that you had posted the last exception is raised in the destructor phase : that is why you have ignored at the end

Exception BrokenPipeError: BrokenPipeError(32, 'Broken pipe') in <_io.TextIOWrapper name='<stdout>' mode='w' encoding='UTF-8'> ignored

A simple example to understand what's up in that context is the follow:

>> class A():
...     def __del__(self):
...         raise Exception("It will be ignored!!!")
... 
>>> a = A()
>>> del a
Exception Exception: Exception('It will be ignored!!!',) in <bound method A.__del__ of <__builtin__.A instance at 0x7ff1d5c06d88>> ignored
>>> a = A()
>>> import sys
>>> sys.stderr.close()
>>> del a

Every exception that is triggered while the object is destroyed will cause a standard error output that explain the exception occurred and ignored (that is because python will inform you that something could not be correctly handle in destroy phase). Anyway, that kind of exceptions cannot be cached and so you can just remove the calls that can generate it or close stderr.

Come back to the question. That exception is not a real problem (as say it is ignored) but if you don't want print it you must override the the function that can be called when the object will be destroyed or close stderr as @SergeBallesta correctly suggested : in you case you can shutdown write and flush function and no exception will be triggered in destroy context

That is an example of how you can do it:

import sys
def _void_f(*args,**kwargs):
    pass

for i in range(4000):
    try:
        print(i,flush=True)
    except (BrokenPipeError, IOError):
        sys.stdout.write = _void_f
        sys.stdout.flush = _void_f
        sys.exit()
  • 1
Reply Report
      • 1
    • This also works. Thank you very much! I chose the answer from @serge-ballesta, because he replied slightly faster and avoided definition of a new function. Thank you both. I have much to learn about exceptions, which I have never sat down and read about.
    • @tommy.carstensen Anyway I understand your choice because the Sarge Balestra answer is well written and that give a simple way to take a solution.

While others have covered the underlying issue in great detail there is a straightforward workaround:

python whatever.py | tail -n +1 | head -n3000

Explanation: tail buffers until it's STDIN is closed (python quits and closes its STDOUT). So only tail gets the SIGPIPE when head quits. The -n +1 is effectively a no-op, making tail output the "tail" starting at line 1, which is the entire buffer.

  • 1
Reply Report

I've often wished there were a command-line option to suppress these signal handlers.

import signal

# Don't turn these signal into exceptions, just die.
signal.signal(signal.SIGINT, signal.SIG_DFL)
signal.signal(signal.SIGPIPE, signal.SIG_DFL)

Instead, the best we can do is uninstall the handlers as soon as possible as the Python script starts running.

  • 1
Reply Report

Trending Tags