Subprocess in Python

Introduction

In this tutorial, we delve deep into one of Python’s most powerful modules: subprocess. Designed to spawn new processes, connect to their input/output/error pipes, and obtain their return codes, the subprocess module is instrumental for professionals seeking advanced Python functionality. By mastering subprocess in Python, developers gain a formidable toolset to execute and communicate with system commands directly through Python scripts.

Overview

Understanding subprocess in Python is pivotal for professionals eager to execute shell commands and interact with other software components directly from Python scripts. This module bridges the gap between Python and external commands, offering the capability to run these commands, communicate with them, and retrieve their outputs seamlessly. This tutorial will explore key functionalities, contrasting methods, and best practices, ensuring an enriched comprehension.

How Do We Start a Process in Python?

In Python, a process refers to a separate and independent instance of a running program. Each process has its memory space, resources, and execution environment. Processes are fundamental for multitasking, allowing your Python program to run multiple tasks concurrently.

To start a new process in Python, you can use the multiprocessing module, which provides a high-level interface for creating and managing processes. Here's an overview of how to start a process:

1. Import the multiprocessing module: You can use the syntax below to import the multiprocessing module.

Syntax:

import multiprocessing

2. Define a target function: Create a Python function that represents the task you want to run in the new process.

Syntax:

def my_function(arg1, arg2):
    # Your task logic here
    pass

3. Create a Process object: Instantiate a Process object and provide the target function and its arguments as arguments to the constructor.

Syntax:

my_process = multiprocessing.Process(target=my_function, args=(arg1_value, arg2_value))

4. Start the process: Use the start() method to launch the new process.

Syntax:

my_process.start()

Optionally, you can wait for the process to finish. We can use the join() method to wait for the new process to complete its execution.

Syntax:

my_process.join()

Here's a basic example of starting a simple process:

import multiprocessing

def print_numbers():
    for i in range(1, 6):
        print(f"Number {i}")

if __name__ == "__main__":
    # Create a process that prints numbers
    number_process = multiprocessing.Process(target=print_numbers)
    
    # Start the process
    number_process.start()
    
    # Wait for the process to finish
    number_process.join()

    print("Main program finished")

In this example, we define a target function print_numbers that prints numbers from 1 to 5. We create a new process, number_process, and start it. The main program waits for the process to finish using join().

What is subprocess.call()?

The subprocess.call() function in Python is a part of the subprocess module, and it is used to run external shell commands or applications from within a Python script. It allows you to interact with the operating system's shell, execute commands, and capture their output or return codes. subprocess.call() is a convenient way to execute external programs and manage their execution.

The subprocess.call() function is a versatile tool for running external programs and shell commands from your Python scripts. It's commonly used for tasks like running system utilities, interacting with other applications, or automating tasks that involve executing command-line tools. However, we have to be cautious when using subprocess.call() with user-generated input to avoid security vulnerabilities such as shell injection attacks.

Here's the basic syntax of subprocess.call():

subprocess.call(args, *, stdin=None, stdout=None, stderr=None, shell=False)

In the above syntax,

• args: This is a required argument and should be a list or a string representing the command and its arguments. If it's a string, you should set shell=True.
• stdin, stdout, stderr: These are optional arguments that allow you to specify the input, output, and error streams for the command. You can use subprocess.PIPE to capture these streams.
• shell: If set to True, the command is executed through the system shell (e.g., Bash on Unix-based systems). If set to False, the command is executed directly.

Using Subprocess to Run the ls Command

Here's a simple example of using subprocess.call() to run the ls command on a Unix-based system:

import subprocess

# Run the "ls" command
subprocess.call(["ls", "-l"])

In this example, we provide the command and its arguments as a list. subprocess.call() executes the command, and the output is displayed in the console.

You can also capture the command's output by redirecting stdout like this:

import subprocess

# Run the "ls" command and capture its output
output = subprocess.check_output(["ls", "-l"], universal_newlines=True)
print(output)

In this case, the universal_newlines=True argument makes sure that the output is returned as a string.

What is Subprocess stdout in Python?

Saving process output (stdout) in the subprocess module in Python allows you to capture and manipulate the standard output (stdout) generated by an external command or process. You can then process this output within your Python script, making it a powerful tool for automation and data processing tasks.

Here's an example:

import subprocess

# Define the command to list all files in the current directory
command = ["ls", "-l"]

# Run the command and capture stdout as a byte stream
process = subprocess.Popen(command, stdout=subprocess.PIPE, stderr=subprocess.PIPE, text=True)

# Read the stdout and stderr byte streams
stdout, stderr = process.communicate()

# Check the return code of the command
return_code = process.returncode

# Process the stdout
if return_code == 0:
    # Split the stdout into lines
    lines = stdout.splitlines()

 # Print the first few lines
    print("First few lines of directory listing:")
    for line in lines[:5]:  # Display the first 5 lines
        print(line)
    
    # Count the number of files and directories
    num_files = len(lines) - 1  # Subtract 1 for the header line
    print(f"Total number of files and directories: {num_files}")

else:
    # Handle errors
    print(f"Error executing command: {stderr}")

# Optionally, you can also process the stderr if needed
if stderr:
    print(f"Error output: {stderr}")

In the above example, we import the subprocess module to work with external commands. After that, we define the command variable as a list representing the command to list all files in the current directory, ["ls", "-l"].

We use subprocess.Popen to run the command and then specify stdout=subprocess.PIPE to capture stdout as a byte stream and stderr=subprocess.PIPE to capture stderr if there are any errors. The text=True argument indicates that we want to work with text data instead of byte data. Then, we use process.communicate() to read the stdout and stderr byte streams. This function waits for the command to complete and returns both stdout and stderr as strings.

We check the return code of the command using process.returncode. A return code of 0 typically indicates success. If the return code is 0, we split the stdout into lines and print the first few lines of the directory listing. We also count the number of files and directories by subtracting one for the header line.

If there's an error, we handle it by printing the error message from stderr. Optionally, we can process the stderr if needed.

Replaceable Functions Through Subprocess Module in Python

In Python, the subprocess module is recommended over older methods like os.system(), os.spawn*(), os.popen*(), and commands.*() for running external commands and managing processes. These older methods are now considered deprecated or less efficient compared to subprocess.

Here's a brief overview of these older methods and their replacement Python subprocess examples:

os.system() (Deprecated)

os.system() is used to run a shell command as if it were run in a terminal. It returns the exit status of the command.

Example:

import os

# Deprecated: Run a shell command using os.system()
exit_status = os.system("ls -l")

Replacement with subprocess:

import subprocess
# Recommended: Use subprocess.run() to run a command
result = subprocess.run(["ls", "-l"], stdout=subprocess.PIPE, text=True)

os.spawn*() (Deprecated)

The os.spawn*() functions are used to spawn a new process. They are lower-level and less portable than subprocess.

Example:

import os
# Deprecated: Use os.spawn*() to create a new process
pid = os.spawnlp(os.P_WAIT, "ls", "ls", "-l")


Replacement with subprocess:

import subprocess
# Recommended: Use subprocess.Popen() to start a new process
process = subprocess.Popen(["ls", "-l"], stdout=subprocess.PIPE, text=True)

os.popen*() (Deprecated)

The os.popen*() functions are used to open a pipe to a command's stdout or stderr. They are less flexible than subprocess.

Example:

import os

# Deprecated: Use os.popen*() to open a pipe to a command
pipe = os.popen("ls -l")

Replacement with subprocess:

import subprocess

# Recommended: Use subprocess.Popen() with stdout or stderr pipes
process = subprocess.Popen(["ls", "-l"], stdout=subprocess.PIPE, text=True)

commands.*() (Deprecated):

The commands module was used to run shell commands and capture their output.

It is deprecated and should not be used in modern Python.

Example:

import commands

# Run a command and capture its output
output = commands.getoutput("ls -l")

Replacement with subprocess:

import subprocess

# Run a command and capture its output
output = subprocess.check_output(["ls", "-l"], text=True)

Conclusion

Harnessing the capabilities of the subprocess module, developers are equipped to integrate Python with myriad external processes, expanding its utility and scope. As professionals in the tech realm, continuous learning is paramount. While this tutorial offers a primer into subprocess, there’s always more to explore. upGrad provides diverse courses that cater to the evolving needs of tech professionals, serving as a beacon for those looking to upskill and elevate their career trajectories.

FAQs

1. Why might one opt for subprocess.run over other methods?

The subprocess.run Python method encapsulates simplicity and efficiency, making it suitable for modern Python applications that require straightforward command execution.

2. How do Python subprocess Popen vs. run differ?

subprocess.Popen provides a granular approach, affording more control over the spawned process, while subprocess.run offers a direct method for command execution.

3. When should I use Python subprocess shell=true?

Use shell=true for direct shell command execution, but ensure untrusted input is stringently validated to prevent potential security risks.

4. What is the role of subprocess.poll?

The subprocess.poll method offers a non-blocking way to check the termination status of a spawned process.

5. How can I make my subprocess call wait until the command completes?

Use the subprocess Popen wait method to ensure sequential execution, waiting for a command to conclude before proceeding.

Take our Free Quiz on Python

Answer quick questions and assess your Python knowledge

Start Quiz