linux,

Convert Linux commands output from CSV to JSON

Sep 18, 2021 · 2 mins read · Post a comment
Convert Linux commands output from CSV to JSON

CSV to JSON conversion is commonly used in the Monitoring part of the DevOps cycle, usually when we need to gather some data, parse it, and ship it via some monitoring agent, like fluentd for example, to a centralized monitoring service. And, if we don’t have any plugin installed that handles and parses the data for us, this is the most “native” thing we can do. In today’s blog post, the focus gonna be put on converting linux commands output data from CSV to JSON.

Prerequisites

  • Access to Linux bash environment

Example solution

Let’s say we want to convert the df -h command output from csv to json.

Step 1. Run the df -h command:

df -h

Example output:

Filesystem      Size  Used Avail Use% Mounted on
udev            3.4G     0  3.4G   0% /dev
tmpfs           697M  1.2M  696M   1% /run
/dev/sda2        40G  8.9G   29G  24% /
tmpfs           3.5G     0  3.5G   0% /dev/shm
tmpfs           5.0M     0  5.0M   0% /run/lock
tmpfs           3.5G     0  3.5G   0% /sys/fs/cgroup
/dev/loop0       89M   89M     0 100% /snap/core/7270
tmpfs           697M     0  697M   0% /run/user/1012

Step 2. Let’s say we want to filter the column names and root block device only:

df -h | grep -E 'Filesystem|/dev/sda2'

Expected output:

Filesystem      Size  Used Avail Use% Mounted on
/dev/sda2        40G  8.9G   29G  24% /

Step 3. Let’s remove the repetitive spaces and replace them with commas:

df -h | grep -E 'Filesystem|/dev/sda2' | tr -s ' ' ','

Expected output:

Filesystem,Size,Used,Avail,Use%,Mounted,on
/dev/sda2,40G,8.9G,29G,24%,/

Step 4. Now, do the final transformation using the jq command-line tool:

df -h | grep -E 'Filesystem|/dev/sda2' | tr -s ' ' ',' | \
jq -cnR '( input | split(",") ) as $keys | ( inputs | split(",") ) as $vals | [ [$keys, $vals] | transpose[] | {key:.[0],value:.[1]} ] | from_entries'

Expected output:

{"Filesystem":"/dev/sda2","Size":"40G","Used":"8.9G","Avail":"29G","Use%":"24%","Mounted":"/","on":null}

Conclusion

If you can’t get things done with a simple jq command, manipulating command outputs in Linux, using grep, awk, sed, tr, cut, and many more command-line utilities and tools, will definitely do the job. As always, there’s more then one way to skin a cat.
Feel free to leave a comment below and if you find this tutorial useful, follow our official channel on telegram.