Issues with Piping Data streams in bash shell



Rather than static programs where every instruction completes and a new one can begin without any time constraints, my questions are about processing data streams with bash in real time.
I have a bit of a weakness in this section of applying bash commands.

I cannot use Sed/Awk in my applications so please dont post any sed/awk.

Here is a trivial streaming example. The following procedure streams the data from an arbitrary file through a procedure, where Grep is the only processing command on the stream and simply just pipes it through unaltered. outfile.txt is then just an unaltered version of infile.txt

    ProcStream () {
	grep  --no-filename . 
    } < infile.txt > outfile.txt



Obviously in this case you must have a one liner processing string else you break the flow of the data. Usual static constructs wont work.
My questions are therefore

  1. Is there a bash tutorial available purely about processing datastreams as it is way different in execution than static data programs. By that I mean, how to convert the well known static uses of if/then/else/while/do etc etc and all other conditional operators and others to specific ally datastreams.
  2. As an example, what would a conditional switch look like that replaces the pass-through "grep --no-filename . " that would process the stream in one way based on an if condition, and another way based on “else”. I give an example in a post below involving while read as wrapper, but I need a way more in depth general treatise using commands in a data stream. The wrapper cannot be the only construct to handle this.

To add: This basically means a) you have to use pipes exclusively and that the entire bash instruction set needs to be understood as piped commands or b) there need to be a pipe routine or procedure available that will execute standard bash as pipe commands.
So, regarding a), How many standard bash commands are pipeable and exactly what are they (books and online manuals are very very sparse about this). Regarding b) is there such a procedure which uses buffers to make any usual bash command pipeable ?

This sort of thing seems to be very sparsely covered in bash tutorials.


In addition as an example

    ProcStream () {
    grep  --no-filename .  | while read i; do  echo "$i$i$i"; done | 
    } < infile.txt > outfile.txt


Will process infile such that the outfile contains three times duplicate of the infle lines.
So While read works piped from a datastream.

Then what about all the other bash conditions such as if then else etc etc ?

The following works for if then else, where the infile just contain numbers 1-10 in 10 lines
It tripples each line and then replaces a tripple “555” with text at that position.

    ProcStream () {
    grep  --no-filename . | while read i; do  echo "$i$i$i"; done | while read i; do if [ $i = "555" ]; then echo "blablabla"; else echo $i; fi; done
    } < infile.txt > outfile.txt

These are simple examples involving always wrapping with “while read” but I need something way more in-depth.

Where do I get an in depth book or tutorial on bash data stream processing such as this and way more complicated and in depth?


Read bash man page? May, I ask what are you trying to solve here?


bash man page is trivial and first grade regarding this issue and does not do inline programming or data stream processing. I think I explained it very well and you should get what I ask.