Running Processes in Python with the Subprocess Module

To execute processes in Python, you can use the os.system and subprocess modules. In this post, we will become familiar with the subprocess module.

Take a look at this example

1
2
3
4

from subprocess import Popen

process = Popen(['ls', '-l'])
process.communicate()

The above code executes ls -l command in the operating system.

Popen initiates the execution of the command, but it does not wait for the process to finish. This is where the communicate method comes in, which waits for the process to complete.

The communicate method takes a timeout parameter. If the process takes longer than this value, a TimeoutExpired error is raised.

To capture the output and potential errors of the command, you should use stdout=PIPE and stderr=PIPE.

1
2
3
4

from subprocess import PIPE

process = Popen(['ls', '-l'], stdout=PIPE, stderr=PIPE)
stdout, stderr = process.communicate()

stdout contains the process output, and stderr contains the process error when there is an error.

For example, if you run ls, stdout will contain the list of files and directories in the current path, and stderr will be empty. However, if you run ls -XXXX, where XXXX is an invalid option, stdout will be empty, and stderr will contain the following

1
2

ls: invalid option -- 'XXXX'
Try 'ls --help' for more information.

To merge stderr into stdout, use stderr=STDOUT.

1
2
3
4

from subprocess import STDOUT

process = Popen(['ls', '-l'], stdout=PIPE, stderr=STDOUT)
stdout = process.communicate()[0]

To ensure the process completion, you can use the wait method, which returns only the exit code.

1

exitcode = process.wait()

To check the exit code of a command in Linux, you can use:

1

echo $?

With Popen, you can change the current directory for the command to be executed

1

process = Popen(['myapp'], cwd='/home/myapp')

You can also set environment variables

1

process = Popen(['myapp'], cwd='/home/myapp', env={'LD_LIBRARY_PATH': './libs/'})

To terminate a process, you can use the terminate and kill methods. terminate sends SIGKILL signal to the process and kill sends SIGKILL signal. If you run into a timeout, you can catch TimeoutExpired and use kill to ensure termination.

1
2
3
4
5
6
7

process = subprocess.Popen(...)

try:
    stdout, stderr = process.communicate(timeout=15)
except TimeoutExpired:
    proc.kill()
    stdout, stderr = proc.communicate()

By default, Popen takes the program name and your desired switches in the form of a list. You can use shlex to convert your command into a list for better readability.

1
2
3

import shlex

process = Popen(shlex.split('ls -l'))

If you want to execute a command like ps aux | grep x, you might think of something like

1

process = Popen(['ps','aux','|','grep','x'])

But that doesn’t work because the inputs after the first value (ps in this case) should be flags or switches not a pipe | operator

there are two solutions

First

1
2

ps = Popen(['ps', 'aux'], stdout=PIPE)
grep = Popen(['grep', 'x'], stdout=PIPE, stdin=ps.stdout)

In this way the input of the first command is set as stdin of the second command

Second

1

process = Popen('ps aux | grep x', shell=True)

However, using shell=True is not secure and may lead to issues with process termination. It is recommended to use the first solution for better security.

Capture output like a stream

To capture the process output as a stream, you can use a context manager

1
2
3

with Popen(['strings', '/dev/urandom'], stdout=PIPE, stderr=STDOUT) as process:
    for line in process.stdout:
        print(line)

Make sure to read the documentation for the subprocess module for more information.