Run a subprocess in a pseudo terminal

Is it possible?

I'd like to spawn a process that reads stdin.

cat requirements.txt | ./script_that_spawns.py safety --check --stdin

When doing so and trying to read safety output, it blocks, and I have to interrupt it with control-c:

Traceback (most recent call last):
  File "/home/pawamoy/.cache/pypoetry/virtualenvs/mkdocstrings-ytlBmpdO-py3.8/bin/failprint", line 8, in <module>
    sys.exit(main())
  File "/home/pawamoy/.cache/pypoetry/virtualenvs/mkdocstrings-ytlBmpdO-py3.8/lib/python3.8/site-packages/failprint/cli.py", line 125, in main
    return run(
  File "/home/pawamoy/.cache/pypoetry/virtualenvs/mkdocstrings-ytlBmpdO-py3.8/lib/python3.8/site-packages/failprint/cli.py", line 54, in run
    output.append(process.read())
  File "/home/pawamoy/.cache/pypoetry/virtualenvs/mkdocstrings-ytlBmpdO-py3.8/lib/python3.8/site-packages/ptyprocess/ptyprocess.py", line 818, in read
    b = super(PtyProcessUnicode, self).read(size)
  File "/home/pawamoy/.cache/pypoetry/virtualenvs/mkdocstrings-ytlBmpdO-py3.8/lib/python3.8/site-packages/ptyprocess/ptyprocess.py", line 516, in read
    s = self.fileobj.read1(size)
KeyboardInterrupt

Here is the actual Python code I'm using:

process = PtyProcessUnicode.spawn(cmd)

output = []

while True:
    try:
        output.append(process.read())
    except EOFError:
        break

process.close()

• Adding more robust code to handle the case where the exec call within fails. Now, spawn will raise an exception if this happens.
• Adding a test to ensure this exception is raised when an invalid binary is run

Calls to self.fileobj.readline() from the PtyProcess readline() method read data one byte at a time (most likely since fileobj is opened with 'buffering=0'.) Thus, this program:

from ptyprocess import PtyProcess, PtyProcessUnicode
p = PtyProcess.spawn(['perl',  '-e', '''use 5.010; foreach my $letter ('a'..'z'){ say $letter x 1000; }'''])
while(1):
    try:
        print p.readline()
    except EOFError:
        break
p.close()

Has pretty poor performance (output from strace):

% time     seconds  usecs/call     calls    errors syscall
------ ----------- ----------- --------- --------- ----------------
 93.48    0.465020          18     26214         1 read
  2.28    0.011353          23       489       381 open
  0.84    0.004197        4197         1           clone
  0.61    0.003037          19       160       113 stat

Is there a compelling reason to specify that the fileobj should have unbuffered IO?

The PtyProcess read() method does not experience this behavior because it uses a default buffer size of 1024.

@jquast flit is my packaging tool to build wheels without involving setuptools. With this branch I can build a wheel by running flit wheel.

We can use this:

1. Standalone, by getting rid of setup.py and MANIFEST.in. This means that future releases would only have wheels on PyPI, not sdist tarballs. I'm already doing this for a number of my other projects - pip has been able to install wheels for about 2½ years - but it may surprise some people.
2. In parallel, using flit to build wheels and setup.py for sdists. There's a risk of duplicated information getting out of date, but the main thing we update is the version number, and flit takes that from __version__, which we need to update anyway.
3. I have a shim called flituptools so setup.py can use the flit information. But that would limit sdists to use with Python 3.

Hi :)

I would like to subprocess any given command and log its stdout and stderr separately. This would seem like an easy thing to do, but im having no luck because:

1. Processes which detect that their stdout/stderr are not ttys will modify their output
2. Processes which detect that their stdout/stderr are not the same file path will assume theres stdout redirection and modify their output.

So in jumps ptys and ptyprocess to the resuce. Tie the stdout and stderr to a pty, you get 1 file path like /dev/tty0021, and isatty() returns true for both. Problem is - now we can't distinguish stdout from stderr. Ok, no problem, just make two ptys - one for stdout and one for stderr. But now although both pass isatty(), their file paths will now look like /dev/tty0021 and /dev/tty0022 (for example). The subprocess reacts as if you weren't using a pty in the first place, and you log nothing.

I have been trying for four months to figure a way around this, and because you are the expert in ptys I thought i might ask you directly - could you think of a way to log stdout and stderr separatly in your program, and still fool a script like this: https://bpaste.net/show/000d6f70ef41

THANK YOU :D

• termios.tcgetattr(fd)[6][VEOF] becomes value of 1(int), where we expect '\x04' (^D). seems to be a bug with the compilation of python2.6 that Sun provides with Solaris 11. This raises TypeError when ord(1) happens when trying to determine the byte for EOF.

  If _is_solaris is True, and the return value is an integer, just assume its a bork and manually set eof = 4(int)(^D).

• Fix several "ValueError: zero length field name in format" errors for Python 2.6, which is what ships with Solaris 11. This caused an exception at a critical path in pty.fork(), where the parent would block on os.read of 'exec_err_pipe_read' indefinitely, as the child raised an exception trying to construct the 'tosend' value (though the exception could not be seen). Test runner just blocks indefinitely without such fixes.

• In spawn(), allow errno.ENXIO when calling inst.setwindowsize(), in some cases, such as spawn(['/bin/true']), the child process may have exited so quickly as to no longer be able to accept any terminal driver calls. Reported by @reynir in https://github.com/pexpect/pexpect/issues/206

Calling wait() on a PtyProcess object whose process has already terminated will raise an exception. This behavior is unexpected[1]. I have made a small test to show case the behavior. Using p.isalive() && p.wait() will only make the race condition worse in my tests.

[1]: subprocess.Popen.wait() returns normally even when the process has already terminated, although it doesn't say so explicitly in the documentation. In java the waitFor() method "returns immediately if the subprocess has already terminated."

Running the tests on FreeBSD 11 I see:

% py.test  
============================= test session starts ==============================
platform freebsd11 -- Python 2.7.9 -- py-1.4.26 -- pytest-2.6.4
collected 5 items 

tests/test_invalid_binary.py .
tests/test_preexec_fn.py ..
tests/test_spawn.py FF


=================================== FAILURES ===================================
__________________________ PtyTestCase.test_spawn_sh ___________________________

self = <tests.test_spawn.PtyTestCase testMethod=test_spawn_sh>

    def test_spawn_sh(self):
        env = os.environ.copy()
        env['FOO'] = 'rebar'
        p = PtyProcess.spawn(['sh'], env=env)
        p.read()
        p.write(b'echo $FOO\n')
        time.sleep(0.1)
        response = p.read()
>       assert b'rebar' in response
E       AssertionError: assert 'rebar' in 'echo $FOO\r\n'

tests/test_spawn.py:15: AssertionError
______________________ PtyTestCase.test_spawn_unicode_sh _______________________

self = <tests.test_spawn.PtyTestCase testMethod=test_spawn_unicode_sh>

    def test_spawn_unicode_sh(self):
        env = os.environ.copy()
        env['FOO'] = 'rebar'
        p = PtyProcessUnicode.spawn(['sh'], env=env)
        p.read()
        p.write(u'echo $FOO\n')
        time.sleep(0.1)
        response = p.read()
>       assert u'rebar' in response
\n'     AssertionError: assert 'rebar' in 'echo $FOO

tests/test_spawn.py:31: AssertionError
====================== 2 failed, 3 passed in 0.80 seconds ======================

The expected output is produced, it is just not returned by the second p.read() in the test. Inserting a dummy p.read() before response = p.read() gets the tests passing.

I found myself wanting to set the size of the pty as seen by the subprocess (really for pexpect.spawn, but this is needed first). In some situations it can make parsing/interaction a lot easier.

Happy to change any stylistic things, eg.

1. The dimensions=(24, 80) default arg could also be dimensions=None with some if dimensions is not None logic in spawn().
2. It could be r=24, c=80 instead, but does it make sense to only have one dimension with a default? (Maybe, I don't know.)

I meant to add a test, but couldn't figure out a simple enough way to test this (eg. if there was a reliable command to check console size it could be like the other spawn tests, but I don't know of one).

Got a FreeBSD (digital ocean droplet, freebsd.pexpect.org) build agent prepared. It raises exception very early in critical codepath causing test runner to fork and eventually the build agent is killed by the kernel due to an OOM condition.

Error is in method pty_make_controlling_tty at:

        # Verify we now have a controlling tty.
        fd = os.open("/dev/tty", os.O_WRONLY)

[[email protected] ~]$ sudo -u teamcity -s
$ cd /opt/TeamCity/work/210ae16cc3f30c30/ptyprocess
$ . `which virtualenvwrapper.sh`
$ mkvirtualenv pexpect27 --python=`which python2.7`
$ pip install -e .
$ cd ../pexpect
$ python
Python 2.7.9 (default, Jan  8 2015, 21:47:19)
[GCC 4.2.1 Compatible FreeBSD Clang 3.3 (tags/RELEASE_33/final 183502)] on freebsd10
Type "help", "copyright", "credits" or "license" for more information.
>>> import pexpect
>>> bash = pexpect.spawn('/bin/bash')
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "pexpect/pty_spawn.py", line 189, in __init__
    self._spawn(command, args, preexec_fn)
  File "pexpect/pty_spawn.py", line 281, in _spawn
    cwd=self.cwd, **kwargs)
  File "/opt/TeamCity/work/210ae16cc3f30c30/ptyprocess/ptyprocess/ptyprocess.py", line 220, in spawn
    pid, fd = _fork_pty.fork_pty()
  File "/opt/TeamCity/work/210ae16cc3f30c30/ptyprocess/ptyprocess/_fork_pty.py", line 30, in fork_pty
    pty_make_controlling_tty(child_fd)
  File "/opt/TeamCity/work/210ae16cc3f30c30/ptyprocess/ptyprocess/_fork_pty.py", line 76, in pty_make_controlling_tty
    fd = os.open("/dev/tty", os.O_WRONLY)
OSError: [Errno 6] Device not configured: '/dev/tty'

/dev/tty may be opened under normal conditions.

There seems to be some kind of race condition in the testsuite on both python-3.4 and python-2.7 here on Fedora 21 (it seems to occour more often in python-3.4 though).

This is the log of the tests: self = <tests.test_spawn.PtyTestCase testMethod=test_spawn_sh>

    def test_spawn_sh(self):
        env = os.environ.copy()
        env['FOO'] = 'rebar'
        p = PtyProcess.spawn(['sh'], env=env)
        p.read()
        p.write(b'echo $FOO\n')
        time.sleep(0.1)
        response = p.read()
        assert b'rebar' in response

        p.sendeof()
        p.read()

        with self.assertRaises(EOFError):
>           p.read()
E           AssertionError: EOFError not raised

tests/test_spawn.py:21: AssertionError

What could cause these random tests? How can I help debugging it?

Python 3.11 was released on 2022-10-24 🚀

Also maybe time to drop EOL Python <= 3.6?

Here's the pip installs for ptyprocess from PyPI for October 2022:

| category | percent | downloads | |:---------|--------:|-----------:| | 3.7 | 29.33% | 7,840,362 | | 3.8 | 23.53% | 6,289,131 | | 3.9 | 18.12% | 4,843,521 | | 3.10 | 12.51% | 3,344,769 | | 3.6 | 7.83% | 2,092,917 | | null | 6.31% | 1,687,690 | | 2.7 | 1.73% | 462,046 | | 3.11 | 0.42% | 111,185 | | 3.5 | 0.17% | 45,885 | | 3.4 | 0.05% | 13,545 | | 3.12 | 0.00% | 143 | | 3.3 | 0.00% | 69 | | 3.2 | 0.00% | 6 | | Total | | 26,731,269 |

Source: pip install -U pypistats && pypistats python_minor ptyprocess --last-month

When I started testing ptyprocess with Python 3.11 I found out that test_spawn_sh fails with:

E       AssertionError: assert b'echo $ENV_KEY; exit 0' in b'\[email protected]:...process/ptyprocess-0.7.0/tests$ echo $ENV \x08_KEY; exit 0\r\nenv_value\r\n'                                                
E        +  where b'echo $ENV_KEY; exit 0' = <built-in method strip of bytes object at 0x1fffc6b2a9f8b0>()                                                                                                              
E        +    where <built-in method strip of bytes object at 0x1fffc6b2a9f8b0> = b'echo $ENV_KEY; exit 0\n'.strip

I did some testing and it seems that this is caused by some kind of a race condition / it's speed dependent. When I run the test suite for the second time, it passes. But then when I delete the __pycache__ directory, it fails again.

I also found out that when I add a sleep after the spawn, it fails every time in every Python (and not just this test, test_spawn_sh_unicode as well):

--- ptyprocess-0.7.0/tests/test_spawn.py
+++ ptyprocess-0.7.0/tests/test_spawn.py
@@ -21,6 +21,7 @@ class PtyTestCase(unittest.TestCase):
     def _spawn_sh(self, ptyp, cmd, outp, env_value):
         # given,
         p = ptyp.spawn(['sh'], env=self.env)
+        time.sleep(1)
         p.write(cmd)
 
         # exercise,

Recently I made a Web terminal with ptyprocess, websocket, and XTerm, but eventually found that the Web terminal size could not adapt after the resize event was triggered.

I even thought my computer was actually a big phone until I switched the positions of two parameters and the page appeared normal.

Before:

def _setwinsize(fd, rows, cols):
    # Some very old platforms have a bug that causes the value for
    # termios.TIOCSWINSZ to be truncated. There was a hack here to work
    # around this, but it caused problems with newer platforms so has been
    # removed. For details see https://github.com/pexpect/pexpect/issues/39
    TIOCSWINSZ = getattr(termios, 'TIOCSWINSZ', -2146929561)
    # Note, assume ws_xpixel and ws_ypixel are zero.
    s = struct.pack('HHHH', rows, cols, 0, 0)
    fcntl.ioctl(fd, TIOCSWINSZ, s)

After using the following modified code, the interface display is normal:

def _setwinsize(fd, rows, cols):
    # Some very old platforms have a bug that causes the value for
    # termios.TIOCSWINSZ to be truncated. There was a hack here to work
    # around this, but it caused problems with newer platforms so has been
    # removed. For details see https://github.com/pexpect/pexpect/issues/39
    width, height = cols, rows
    TIOCSWINSZ = getattr(termios, 'TIOCSWINSZ', -2146929561)
    # Note, assume ws_xpixel and ws_ypixel are zero.
    s = struct.pack('HHHH', width, height, 0, 0)
    fcntl.ioctl(fd, TIOCSWINSZ, s)

The PtyProcessUnicode class accept keywords argument like encoding and codec_errors. However, it seems not easy to set these argument just using PtyProcess.spawn.

I think we could add a **kwargs to spawn and create the class instance using cls(pid, fd, **kwargs). It will be neat and improve extensibility.

This is a very severe bug. When calling Ptyprocess.read() the value returned is different almost every time:

ptyprocess.PtyProcess.spawn(['openssl', "ec", '-noout', '-text', '-in', '/opt/key/s128r1.key']).read()

The output:

And again with the same params:

And again:

I don't know what is causing this but this is very weird.

Currently, the preexec_fn is executed after the file descriptors are closed. This has some unwanted effects:

• if preexec_fn opens a file descriptor, it will be inherit by the child process
• if preexec_fn relays on having some file descriptor open, it will crash (see https://github.com/pexpect/pexpect/issues/368)

The proposal is to move the "close the fds" section below the "execute the preexec_fn" code: https://github.com/pexpect/ptyprocess/blob/master/ptyprocess/ptyprocess.py#L266-L285

For reference, this is what it does subprocess.Popen: https://github.com/python/cpython/blob/master/Modules/_posixsubprocess.c#L528-L549

If it is okay, I can do a PR with the fix but I would like to hear your opinions about this.