📜 ⬆️ ⬇️

JSON pipes in the shell

The more I write one-liners in the shell, the more I come to two important ideas:
  1. This is a very powerful tool for "direct programming", that is, instructions to the computer what to do.
  2. Most of the one-liner is dedicated to grep / awk / cut / tr, which somehow pick out and bring the output of previous utilities into a human form.

Given that the pipe model is amazing, completely dirty hacks capture the necessary fields in the output in the second paragraph (“and here we can select the needed comma by a characteristic comma using awk -F, '{print $ 2}' ... ) make the procedure controversial for pleasure, and certainly unreadable.

Another serious problem: despite the fact that the shell gives quite a few idioms from functional programming, there is no list filtering idiom based on the result of executing an external program. That is, the list can be “burnt”. But to leave in the list only those elements for which some program returned "success" - no.

At the same time there is a hostile and not very well written environment - powershell (Windows). In which they took a good idea (the pipes convey not the text, but objects), but spoiled it with two things:
  1. Unergonomic console Windows ( Shift-PgUp where, eh? Say, Ctrl-PdUp in new versions)
  2. a suggestion to go and learn .net in order to work properly with the methods.
  3. Absence under most operating systems

I want to have objects in the pipe in a warm Linux tube shell. With hand-candy (little typing), eye-candy (nice to watch) and the overall ergonomic use of the process. I also want to be able to combine the “new approach” with the old, that is, the usual text pipe.


It is necessary to write a set of tools that will allow to operate in a pipe-style with structured data. The obvious choice is XML JSON.
We need:
  1. Utilities that will accept standard input formats and convert them to json.
  2. Utilities that will allow in the pipe'e to manipulate json'om.
  3. Utilities that will lead json in "normal" format.

In this case, the person will not see json on the screen, but will be able to work with him.

For seed

(for understanding, I will write long names of utilities, in real life these will be short abbreviations, that is, not json-get-object, but something like jgo or jg)

Displays only files for which file has managed to determine the type:
ls -la | ls2json | json-filter 'filename' --exec 'file {} >/dev/null' | json-print

It downloads a token for authorization from a certain site, picks it out from json and puts environment variables into environment variables, then downloads the list and filter on the regexp the author field downloads all urls:
curl mysite/api.json | env `json-get-to-env X-AUTH-TOKEN`;curl -H X-AUTH-TOKEN $X-AUTH-TOKEN mysite/api/list.json | json-filter --field 'author' --rmatch 'R.{1,2}dal\d*' | json-get --field 'url' | xargs wget

Parsing output find -ls, sorts by size field, cuts elements from 10 to 20 from an array, prints them to csv.
find . -ls | ls2josn | json-sort --field 'size' | json-slice [10:20] | json2csv



The main task is to make a json candy from a messy output. It is important: to have the option to process incorrect input: a) ignore, b) stop the pipe with an error.



As suggested in private, mysql-json fits perfectly on this idea. Run binary output from sql'ya? Easily.

Read the file, output it in json'e.

native json transforms

The most delicious is native manipulation of json. Similarly, they should have “non-json'a processing options -“ ignore ”/“ stop ”.
(add to taste and needs)


Give json a human readable view:


In fact, the xargs extension on json:


Of course, such methods cannot solve the problem of processing an arbitrary json - it may turn out to be too “unstructured”. But firstly, input'y do the same predictable type of json'a, and secondly, the processing of json'a is still more predictable than the processing of "like here elements are separated by space" in the existing shell.


I would have written it myself, but I don’t know part of what I need, I don’t have enough time for something. Not a programmer, me. The secret idea of ​​the article is that “someone will write for me,” but if there is no such one, then there will be at least a program article with the motivation to complete my education (do) and do it myself.

If someone is ready to take on this, I will be extremely grateful. If not, I will uncover my fig python - and ideas and suggestions are welcome.

UPDATE: It seems that people are a little stir. Your commits will be welcome here: github.com/amarao/json4shell . When I can use it, I don’t know yet. Do I have enough gunpowder? I don't know either.

Source: https://habr.com/ru/post/102072/

All Articles