MatplotCLI
Create matplotlib visualizations from the command-line
MatplotCLI is a simple utility to quickly create plots from the command-line, leveraging Matplotlib.
plt "scatter(x,y,5,alpha=0.05); axis('scaled')" < sample.json
plt "hist(x,30)" < sample.json
MatplotCLI accepts both JSON lines and arrays of JSON objects as input. Look at the recipes section to learn how to handle other formats like CSV.
MatplotCLI executes python code (passed as argument) where some handy imports are already done (e.g. from matplotlib.pyplot import *) and where the input JSON data is already parsed and available in variables, making plotting easy. Please refer to matplotlib.pyplot's reference and tutorial for comprehensive documentation about the plotting commands.
Data from the input JSON is made available in the following way. Given the input myfile.json:
{"a": 1, "b": 2}
{"a": 10, "b": 20}
{"a": 30, "c$d": 40}
The following variables are made available:
data = {
"a": [1, 10, 30],
"b": [2, 20, None],
"c_d": [None, None, 40]
}
a = [1, 10, 30]
b = [2, 20, None]
c_d = [None, None, 40]
col_names = ("a", "b", "c_d")
So, for a scatter plot a vs b, you could simply do:
plt "scatter(a,b); title('a vs b')" < myfile.json
Notice that the names of JSON properties are converted into valid Python identifiers whenever they are not (e.g. c$d was converted into c_d).
Execution flow
- Import matplotlib and other libs;
- Read JSON data from standard input;
- Execute user code;
- Show the plot.
All steps (except step 3) can be skipped through command-line options.
Installation
The easiest way to install MatplotCLI is from pip:
pip install matplotcli
Recipes and Examples
Plotting JSON data
MatplotCLI natively supports JSON lines:
echo '
{"a":0, "b":1}
{"a":1, "b":0}
{"a":3, "b":3}' |
plt "plot(a,b)"
and arrays of JSON objects:
echo '[
{"a":0, "b":1},
{"a":1, "b":0},
{"a":3, "b":3}]' |
plt "plot(a,b)"
Plotting from a csv
SPyQL is a data querying tool that allows running SQL queries with Python expressions on top of different data formats. Here, SPyQL is reading a CSV file, automatically detecting if there's an header row, the dialect and the data type of each column, and converting the output to JSON lines before handing over to MatplotCLI.
cat my.csv | spyql "SELECT * FROM csv TO json" | plt "plot(x,y)"
Plotting from a yaml/xml/toml
yq converts yaml, xml and toml files to json, allowing to easily plot any of these with MatplotCLI.
cat file.yaml | yq -c | plt "plot(x,y)"
cat file.xml | xq -c | plt "plot(x,y)"
cat file.toml | tomlq -c | plt "plot(x,y)"
Plotting from a parquet file
parquet-tools allows dumping a parquet file to JSON format. jq -c makes sure that the output has 1 JSON object per line before handing over to MatplotCLI.
parquet-tools cat --json my.parquet | jq -c | plt "plot(x,y)"
Plotting from a database
Databases CLIs typically have an option to output query results in CSV format (e.g. psql --csv -c query for PostgreSQL, sqlite3 -csv -header file.db query for SQLite).
Here we are visualizing how much space each namespace is taking in a PostgreSQL database. SPyQL converts CSV output from the psql client to JSON lines, and makes sure there are no more than 10 items, aggregating the smaller namespaces in an All others category. Finally, MatplotCLI makes a pie chart based on the space each namespace is taking.
psql -U myuser mydb --csv -c '
SELECT
N.nspname,
sum(pg_relation_size(C.oid))*1e-6 AS size_mb
FROM pg_class C
LEFT JOIN pg_namespace N ON (N.oid = C.relnamespace)
GROUP BY 1
ORDER BY 2 DESC' |
spyql "
SELECT
nspname if row_number < 10 else 'All others' as name,
sum_agg(size_mb) AS size_mb
FROM csv
GROUP BY 1
TO json" |
plt "
nice_labels = ['{0}\n{1:,.0f} MB'.format(n,s) for n,s in zip(name,size_mb)];
pie(size_mb, labels=nice_labels, autopct='%1.f%%', pctdistance=0.8, rotatelabels=True)"
Plotting a function
Disabling reading from stdin and generating the output using numpy.
plt --no-input "
x = np.linspace(-1,1,2000);
y = x*np.sin(1/x);
plot(x,y);
axis('scaled');
grid(True)"
Saving the plot to an image
Saving the output without showing the interactive window.
cat sample.json |
plt --no-show "
hist(x,30);
savefig('myimage.png', bbox_inches='tight')"
Plot of the global temperature
Here's a complete pipeline from getting the data to transforming and plotting it:
- Downloading a CSV file with
curl; - Skipping the first row with
sed; - Grabbing the year column and 12 columns with monthly temperatures to an array and converting to JSON lines format using SPyQL;
- Exploding the monthly array with SPyQL (resulting in 12 rows per year) while removing invalid monthly measurements;
- Plotting with MatplotCLI .
curl https://data.giss.nasa.gov/gistemp/tabledata_v4/GLB.Ts+dSST.csv |
sed 1d |
spyql "
SELECT Year, cols[1:13] AS temps
FROM csv
TO json" |
spyql "
SELECT
json->Year + ((row_number-1)%12)/12 AS year,
json->temps AS temp
FROM json
EXPLODE json->temps
WHERE json->temps is not Null
TO json" |
plt "
scatter(year, temp, 2, temp);
xlabel('Year');
ylabel('Temperature anomaly w.r.t. 1951-80 (ºC)');
title('Global surface temperature (land and ocean)')"
12 Oct 20, 2022
9 Sep 2, 2022
3 Dec 14, 2022
1 Jul 12, 2022
10 Jun 1, 2022
0 Jul 9, 2022
1 Jan 11, 2022
2 Jan 13, 2022
3 Jan 19, 2022
8 Nov 01, 2022
1 Nov 24, 2021
69 Dec 15, 2022
1 Jan 23, 2022
400 Dec 31, 2022
11 Oct 26, 2021
1.2k Jan 01, 2023
1.4k Dec 15, 2022
6k Jan 08, 2023
251 Dec 19, 2022
744 Jan 06, 2023
284 Jan 01, 2023
6 Sep 29, 2022
1 Feb 05, 2022
4k Jan 08, 2023
487 Jan 08, 2023
1.7k Jan 09, 2023
2.4k Jan 07, 2023
2.3k Jan 04, 2023
68 Dec 08, 2022