Author: Adrian Tam
Working on a machine learning project means we need to experiment. Having a way to configure your script easily will help you move faster. In Python, we have a way to adapt the code from command line. In this tutorial, we are going to see how we can leverage the command line arguments to a Python script to help you work better in your machine learning project.
After finishing this tutorial, you will learn
- Why we would like to control a Python script in command line
- How we can work in a command line efficiently
Let’s get started.
![](https://machinelearningmastery.com/wp-content/uploads/2022/02/insung-yoon-Uaf6XwW4n10-unsplash-scaled.jpg)
Command line arguments for your Python script. Photo by insung yoon. Some rights reserved
Overview
This tutorial is in three parts, they are
- Running a Python script in command line
- Working on the command line
- Alternative to command line arguments
Running a Python script in command line
There are many ways to run a Python script. Someone may run it as part of a Jupyter notebook. Someone may run it in an IDE. But in all platforms, it is always possible to run a Python script in command line. In Windows, you have the command prompt or PowerShell (or even better, the Windows Terminal). In macOS or Linux, you have the Terminal or xterm. Running a Python script in command line is powerful because you can pass in additional parameter to the script.
The following script allows us to pass in values from the command line into Python:
import sys n = int(sys.argv[1]) print(n+1)
When we save these few lines into a file, and run it in command line with an argument:
$ python commandline.py 15 16
you will see it takes our argument, convert it into integer, add one into it and print. The list sys.argv
contains the name of our script and all the arguments (all strings), which in the above case, it is ["commandline.py", "15"]
.
When you run a command line with more complicated set of arguments, it takes some effort to process the list sys.argv
. Therefore Python provided the library argparse
to help. This is to assume GNU-style, which can be explained using the following example:
rsync -a -v --exclude="*.pyc" -B 1024 --ignore-existing 192.168.0.3:/tmp/ ./
The optional arguments are introduced by “-
” or “--
“, which a single hyphen is to carry a single character “short option” (such as -a
, -B
and -v
above), and two hyphens are for multiple characters “long options” (such as --exclude
and --ignore-existing
above). The optional arguments may have additional parameters, such as in -B 1024
or --exclude="*.pyc"
, the 1024
and "*.pyc"
are parameters to -B
and --exclude
respectively. Additionally, we may also have compulsory arguments, which we just put them into the command line. The part 192.168.0.3:/tmp/
and ./
above are examples. The order of compulsory arguments are important. For example, the rsync
command above will copy files from 192.168.0.3:/tmp/
to ./
instead of the other way round.
The following is to replicate the above example in Python using argparse:
import argparse parser = argparse.ArgumentParser(description="Just an example", formatter_class=argparse.ArgumentDefaultsHelpFormatter) parser.add_argument("-a", "--archive", action="store_true", help="archive mode") parser.add_argument("-v", "--verbose", action="store_true", help="increase verbosity") parser.add_argument("-B", "--block-size", help="checksum blocksize") parser.add_argument("--ignore-existing", action="store_true", help="skip files that exist") parser.add_argument("--exclude", help="files to exclude") parser.add_argument("src", help="Source location") parser.add_argument("dest", help="Destination location") args = parser.parse_args() config = vars(args) print(config)
If you run the above script, you will see:
$ python argparse_example.py usage: argparse_example.py [-h] [-a] [-v] [-B BLOCK_SIZE] [--ignore-existing] [--exclude EXCLUDE] src dest argparse_example.py: error: the following arguments are required: src, dest
This is to mean you didn’t provide the compulsory arguments for src
and dest
. Perhaps the best reason to use argparse is to get a help screen for free if you provded -h
or --help
as the argument, like the following:
$ python argparse_example.py --help usage: argparse_example.py [-h] [-a] [-v] [-B BLOCK_SIZE] [--ignore-existing] [--exclude EXCLUDE] src dest Just an example positional arguments: src Source location dest Destination location optional arguments: -h, --help show this help message and exit -a, --archive archive mode (default: False) -v, --verbose increase verbosity (default: False) -B BLOCK_SIZE, --block-size BLOCK_SIZE checksum blocksize (default: None) --ignore-existing skip files that exist (default: False) --exclude EXCLUDE files to exclude (default: None)
While the script did nothing real, if you provided the arguments as required, you will see this:
$ python argparse_example.py -a --ignore-existing 192.168.0.1:/tmp/ /home {'archive': True, 'verbose': False, 'block_size': None, 'ignore_existing': True, 'exclude': None, 'src': '192.168.0.1:/tmp/', 'dest': '/home'}
The parser object created by ArgumentParser()
has a parse_args()
method that reads sys.argv
and returns a namespace object. This is an object that carries attributes and we can read using args.ignore_existing
for example. But usually it is easier to handle if it is a Python dictionary. Hence we can convert it into one using vars(args)
.
Usually for all optional arguments, we provide the long option and sometimes also the short option. Then we can access the value provided from the command line using the long option as the key (with hyphen replaced with underscore, or the single-character short option as the key if we don’t have a long version). The “positional arguments” are not optional and their names are provided in the add_argument()
function.
There are multiple types of arguments. For the optional arguments, sometimes we use them as a boolean flags but sometimes we expect them to bring in some data. In the above, we use action="store_true"
to make that option set to False
by default and toggle to True
if it is specified. For the other option such as -B
above, by default it expects additional data to go following it.
We can further require an argument to be a specific type. For example the -B
option above, we can make it to expect integer data by adding type
like the following
parser.add_argument("-B", "--block-size", type=int, help="checksum blocksize")
and if we provided the wrong type, argparse will help terminate our program with an informative error message:
python argparse_example.py -a -B hello --ignore-existing 192.168.0.1:/tmp/ /home usage: argparse_example.py [-h] [-a] [-v] [-B BLOCK_SIZE] [--ignore-existing] [--exclude EXCLUDE] src dest argparse_example.py: error: argument -B/--block-size: invalid int value: 'hello'
Working on the command line
Empowering your Python script with command line arguments can bring it to a new level of reusability. First, let’s look at a simple example on fitting an ARIMA model to GDP time series. World Bank collected historical GDP data of many countries. We can make use of the pandas_datareader
package to read the data. If you haven’t installed it yet, you can use pip
(or conda
if you installed Anacronda) to install the package:
pip install pandas_datareader
The code for the GDP data that we use is NY.GDP.MKTP.CN
, we can get the data of a country in the form of a pandas DataFrame by
from pandas_datareader.wb import WorldBankReader gdp = WorldBankReader("NY.GDP.MKTP.CN", "SE", start=1960, end=2020).read()
and then we can tidy up the DataFrame a bit using the tools provided by pandas:
import pandas as pd # Drop country name from index gdp = gdp.droplevel(level=0, axis=0) # Sort data in choronological order and set data point at year-end gdp.index = pd.to_datetime(gdp.index) gdp = gdp.sort_index().resample("y").last() # Convert pandas DataFrame into pandas Series gdp = gdp["NY.GDP.MKTP.CN"]
then fitting an ARIMA model and use the model for prediction is not difficult. In the following, we fit using the first 40 data points and forecast for next 3. Then compare the forecast with the actual in terms of relative error:
import statsmodels.api as sm model = sm.tsa.ARIMA(endog=gdp[:40], order=(1,1,1)).fit() forecast = model.forecast(steps=3) compare = pd.DataFrame({"actual":gdp, "forecast":forecast}).dropna() compare["rel error"] = (compare["forecast"] - compare["actual"])/compare["actual"] print(compare)
Putting it all together, and a little polishing, the following is the complete code:
import warnings warnings.simplefilter("ignore") from pandas_datareader.wb import WorldBankReader import statsmodels.api as sm import pandas as pd series = "NY.GDP.MKTP.CN" country = "SE" # Sweden length = 40 start = 0 steps = 3 order = (1,1,1) # Read the GDP data from WorldBank database gdp = WorldBankReader(series, country, start=1960, end=2020).read() # Drop country name from index gdp = gdp.droplevel(level=0, axis=0) # Sort data in choronological order and set data point at year-end gdp.index = pd.to_datetime(gdp.index) gdp = gdp.sort_index().resample("y").last() # Convert pandas dataframe into pandas series gdp = gdp[series] # Fit arima model result = sm.tsa.ARIMA(endog=gdp[start:start+length], order=order).fit() # Forecast, and calculate the relative error forecast = result.forecast(steps=steps) df = pd.DataFrame({"Actual":gdp, "Forecast":forecast}).dropna() df["Rel Error"] = (df["Forecast"] - df["Actual"]) / df["Actual"] # Print result with pd.option_context('display.max_rows', None, 'display.max_columns', 3): print(df)
This script prints the following output
Actual Forecast Rel Error 2000-12-31 2408151000000 2.367152e+12 -0.017025 2001-12-31 2503731000000 2.449716e+12 -0.021574 2002-12-31 2598336000000 2.516118e+12 -0.031643
The above code is short but we made it flexible enough by holding some parameters in variables. We can change the above code to use argparse so we can change some parameters from the command line, as follows:
from argparse import ArgumentParser, ArgumentDefaultsHelpFormatter import warnings warnings.simplefilter("ignore") from pandas_datareader.wb import WorldBankReader import statsmodels.api as sm import pandas as pd # Parse command line arguments parser = ArgumentParser(formatter_class=ArgumentDefaultsHelpFormatter) parser.add_argument("-c", "--country", default="SE", help="Two-letter country code") parser.add_argument("-l", "--length", default=40, type=int, help="Length of time series to fit the ARIMA model") parser.add_argument("-s", "--start", default=0, type=int, help="Starting offset to fit the ARIMA model") args = vars(parser.parse_args()) # Set up parameters series = "NY.GDP.MKTP.CN" country = args["country"] length = args["length"] start = args["start"] steps = 3 order = (1,1,1) # Read the GDP data from WorldBank database gdp = WorldBankReader(series, country, start=1960, end=2020).read() # Drop country name from index gdp = gdp.droplevel(level=0, axis=0) # Sort data in choronological order and set data point at year-end gdp.index = pd.to_datetime(gdp.index) gdp = gdp.sort_index().resample("y").last() # Convert pandas dataframe into pandas series gdp = gdp[series] # Fit arima model result = sm.tsa.ARIMA(endog=gdp[start:start+length], order=order).fit() # Forecast, and calculate the relative error forecast = result.forecast(steps=steps) df = pd.DataFrame({"Actual":gdp, "Forecast":forecast}).dropna() df["Rel Error"] = (df["Forecast"] - df["Actual"]) / df["Actual"] # Print result with pd.option_context('display.max_rows', None, 'display.max_columns', 3): print(df)
If we run the code above in a command line, we can see it can now accept arguments:
$ python gdp_arima.py --help usage: gdp_arima.py [-h] [-c COUNTRY] [-l LENGTH] [-s START] optional arguments: -h, --help show this help message and exit -c COUNTRY, --country COUNTRY Two-letter country code (default: SE) -l LENGTH, --length LENGTH Length of time series to fit the ARIMA model (default: 40) -s START, --start START Starting offset to fit the ARIMA model (default: 0) $ python gdp_arima.py Actual Forecast Rel Error 2000-12-31 2408151000000 2.367152e+12 -0.017025 2001-12-31 2503731000000 2.449716e+12 -0.021574 2002-12-31 2598336000000 2.516118e+12 -0.031643 $ python gdp_arima.py -c NO Actual Forecast Rel Error 2000-12-31 1507283000000 1.337229e+12 -0.112821 2001-12-31 1564306000000 1.408769e+12 -0.099429 2002-12-31 1561026000000 1.480307e+12 -0.051709
In the last command above, we pass in -c NO
to apply the same model to the GDP data of Norway (NO) instead of Sweden (SE). Hence, without the risk of messing up the code, we reused our code to a different dataset.
The power of introducing command line argument is that we can test out our code with varying parameters easily. For example, we want to see if ARIMA(1,1,1) model is a good model for predicting GDP and we want to verify with different time window of the nordic countries:
- Denmark (DK)
- Finland (FI)
- Iceland (IS)
- Norway (NO)
- Sweden (SE)
We want to check for the window of 40 years but with different starting points (since 1960, 1965, 1970, 1975). Depends on the OS, you can build a for loop in Linux and mac using the bash shell syntax:
for C in DK FI IS NO SE; do for S in 0 5 10 15; do python gdp_arima.py -c $C -s $S done done
or, as the shell syntax permits, we can put everything in one line:
for C in DK FI IS NO SE; do for S in 0 5 10 15; do python gdp_arima.py -c $C -s $S ; done ; done
or even better, give some information at each iteration of the loop, and we get our script run multiple times:
$ for C in DK FI IS NO SE; do for S in 0 5 10 15; do echo $C $S; python gdp_arima.py -c $C -s $S ; done; done DK 0 Actual Forecast Rel Error 2000-12-31 1.326912e+12 1.290489e+12 -0.027449 2001-12-31 1.371526e+12 1.338878e+12 -0.023804 2002-12-31 1.410271e+12 1.386694e+12 -0.016718 DK 5 Actual Forecast Rel Error 2005-12-31 1.585984e+12 1.555961e+12 -0.018931 2006-12-31 1.682260e+12 1.605475e+12 -0.045644 2007-12-31 1.738845e+12 1.654548e+12 -0.048479 DK 10 Actual Forecast Rel Error 2010-12-31 1.810926e+12 1.762747e+12 -0.026605 2011-12-31 1.846854e+12 1.803335e+12 -0.023564 2012-12-31 1.895002e+12 1.843907e+12 -0.026963 ... SE 5 Actual Forecast Rel Error 2005-12-31 2931085000000 2.947563e+12 0.005622 2006-12-31 3121668000000 3.043831e+12 -0.024934 2007-12-31 3320278000000 3.122791e+12 -0.059479 SE 10 Actual Forecast Rel Error 2010-12-31 3573581000000 3.237310e+12 -0.094099 2011-12-31 3727905000000 3.163924e+12 -0.151286 2012-12-31 3743086000000 3.112069e+12 -0.168582 SE 15 Actual Forecast Rel Error 2015-12-31 4260470000000 4.086529e+12 -0.040827 2016-12-31 4415031000000 4.180213e+12 -0.053186 2017-12-31 4625094000000 4.273781e+12 -0.075958
If you’re using Windows, you can use the following syntax in command prompt:
for %C in (DK FI IS NO SE) do for %S in (0 5 10 15) do python gdp_arima.py -c $C -s $S
or the following in PowerShell:
foreach ($C in "DK","FI","IS","NO","SE") { foreach ($S in 0,5,10,15) { python gdp_arima.py -c $C -s $S } }
both should produce the same result.
While we can put similar loop inside our Python script, sometimes it is easier if we can do it at the command line. It could be more convenient when we are exploring different options. Moreover, by taking the loop outside of the Python code, we can be assured that every time we run the script is independent because we will not share any variables between iterations.
Alternative to command line arguments
Using command line arguments is not the only way to pass in data to your Python script. At least, there are several other ways too:
- using environment variables
- using config files
Environment variables are features from your OS to keep small amount of data in memory. We can read environment variables in Python using the following syntax:
import os print(os.environ["MYVALUE"])
For example, in Linux, the above two-line script will work with the shell as follows
$ export MYVALUE="hello" $ python show_env.py hello
and in Windows, the syntax inside command prompt is similar:
C:MLM> set MYVALUE=hello C:MLM> python show_env.py hello
You may also add or edit environment variables in Windows using the dialog in Control Panel:
So we may keep the parameters to the script in some environment variables and let the script adapt its behavior like setting up command line arguments.
In case we have a lot of options to set, it is better to save the options to a file rather than overwhelming the command line. Depends on the format we chose, we can use the configparser
or json
module from Python for reading Windows INI format or JSON format respectively. We may also use the third party library PyYAML to read YAML format.
For the above example on running ARIMA model on GDP data, we can modify the code to use YAML config file:
import warnings warnings.simplefilter("ignore") from pandas_datareader.wb import WorldBankReader import statsmodels.api as sm import pandas as pd import yaml # Load config from YAML file with open("config.yaml", "r") as fp: args = yaml.safe_load(fp) # Set up parameters series = "NY.GDP.MKTP.CN" country = args["country"] length = args["length"] start = args["start"] steps = 3 order = (1,1,1) # Read the GDP data from WorldBank database gdp = WorldBankReader(series, country, start=1960, end=2020).read() # Drop country name from index gdp = gdp.droplevel(level=0, axis=0) # Sort data in choronological order and set data point at year-end gdp.index = pd.to_datetime(gdp.index) gdp = gdp.sort_index().resample("y").last() # Convert pandas dataframe into pandas series gdp = gdp[series] # Fit arima model result = sm.tsa.ARIMA(endog=gdp[start:start+length], order=order).fit() # Forecast, and calculate the relative error forecast = result.forecast(steps=steps) df = pd.DataFrame({"Actual":gdp, "Forecast":forecast}).dropna() df["Rel Error"] = (df["Forecast"] - df["Actual"]) / df["Actual"] # Print result with pd.option_context('display.max_rows', None, 'display.max_columns', 3): print(df)
and the YAML config file is named as config.yaml
, which its content is as follows:
country: SE length: 40 start: 0
Then we can run the above code and obtaining the same result as before. The JSON counterpart is very similar, which we use the load()
function from json
module:
import json import warnings warnings.simplefilter("ignore") from pandas_datareader.wb import WorldBankReader import statsmodels.api as sm import pandas as pd # Load config from JSON file with open("config.json", "r") as fp: args = json.load(fp) # Set up parameters series = "NY.GDP.MKTP.CN" country = args["country"] length = args["length"] start = args["start"] steps = 3 order = (1,1,1) # Read the GDP data from WorldBank database gdp = WorldBankReader(series, country, start=1960, end=2020).read() # Drop country name from index gdp = gdp.droplevel(level=0, axis=0) # Sort data in choronological order and set data point at year-end gdp.index = pd.to_datetime(gdp.index) gdp = gdp.sort_index().resample("y").last() # Convert pandas dataframe into pandas series gdp = gdp[series] # Fit arima model result = sm.tsa.ARIMA(endog=gdp[start:start+length], order=order).fit() # Forecast, and calculate the relative error forecast = result.forecast(steps=steps) df = pd.DataFrame({"Actual":gdp, "Forecast":forecast}).dropna() df["Rel Error"] = (df["Forecast"] - df["Actual"]) / df["Actual"] # Print result with pd.option_context('display.max_rows', None, 'display.max_columns', 3): print(df)
and the JSON config file, config.json
would be
{ "country": "SE", "length": 40, "start": 0 }
You may learn more about the syntax of JSON and YAML for your project. But the idea here is that we can separate the data and algorithm for better reusability of our code.
Further reading
This section provides more resources on the topic if you are looking to go deeper.
Libraries
- argparse module, https://docs.python.org/3/library/argparse.html
- Pandas Data Reader, https://pandas-datareader.readthedocs.io/en/latest/
- ARIMA in statsmodels, https://www.statsmodels.org/devel/generated/statsmodels.tsa.arima.model.ARIMA.html
- configparser module, https://docs.python.org/3/library/configparser.html
- json module, https://docs.python.org/3/library/json.html
- PyYAML, https://pyyaml.org/wiki/PyYAMLDocumentation
Articles
- Working with JSON, https://developer.mozilla.org/en-US/docs/Learn/JavaScript/Objects/JSON
- YAML on Wikipedia, https://en.wikipedia.org/wiki/YAML
Books
- Python Cookbook, third edition, by David Beazley and Brian K. Jones, https://www.amazon.com/dp/1449340377/
Summary
In this tutorial, you’ve see how we can make use of the command line for more efficient control of our Python script. Specifically, you learned
- How we can pass in parameters to your Python script using the argparse module
- How we can efficiently control the argparse-enabled Python script in a terminal under different OS
- We can also use environment variables, or config files to pass in parameters to a Python script
The post Command line arguments for your Python script appeared first on Machine Learning Mastery.