“The problem with being self-taught is that you seldom know what it is you don’t know.” --a wise Autodidact.
Recently, the my boss introduced me to the power of perl to parse log files from the CLI, after I tried loading a million line csv in OpenOffice and crashed “ALL THE THINGS!”.
After a few hours of working with it, I turned simple parsing on its head and was doing things I hadn’t even conceived of being possible. For example, I needed to turn a list of errors into a MySQL query, so I came up with this:
But that really got me thinking. What other tools do I know have in my belt just because I simply hadn’t had a reason to know about them yet?
xargs is on my list to learn next. What recommendations would You make to a relative newbie who spends about 6 hours a day staring at $PS1 and desperately needs to get better with BASH and the CLI in general?
ETA: Thanks for all the replies! To clarify, I’ve been using Linux for 6 years now, but I’ve only been a developer for a little over a year. In that time, I truly just ignored the CLI except as a platform to launch gcc, python, and the Metasploit framework until I took a new job that wasn’t InfoSec but instead s DevOps. I am fluent in C, Python, PHP, MySQL, & JS and can write some serious RegExes. But the tools of BASH are myriad and I want to get a jump on mastering the best of them!
Umm… is it “most useful BASH commands”, or “most useful Linux tools”?
It all depends on what are you trying to do.
As far as text processing/parsing is considered I strongly recommend learning how to use below tools:
regex (!!!. Not a command but something YOU really really REALLY WANT to learn. Trust me)
man (yes, you must learn how to use it. And USE IT)
grep / egrep
sed
awk
tr
head / tail
column (to make output look as a table - usually much more human readable)
Once you have a good grip on them try learning some shell scripting to bring more logic to the filters (i.e. commands listed above): loops (while, for), variables, &1 and &2 descriptors and data stream redirection, conditionals (either if-else or more geeky && {} || {} ).
Also learn how to use ‘less’ and ‘vim’ (or ‘more’ ‘vi’ in older machines… like MUCH older).
Once you’re good with shell scripting you will be able to parse any output (or input) into anything you want.
As far as other commands are concerned (not for text parsing)… well, that depends entirely on what is it you want to do. Basically you can do anything Once I was so bored I made a ping-pong game using only bash Later - a shell-based chat application for my team (with channels, control statements, members lists, colors, etc… just like mirc). Just because I can…
In addition to those already mentioned, check out:
cut
join
sort
uniq
wc
top
ps
bg
jobs
&
kill
killall
sleep
tr
head
tail
less
cat
find
With *nix, the philosophy was to have many small tools that do one thing really well; then, they can quickly be glued together with a shell script to perform a more complex task. Once you have the logic flow down, if performance is important then you can transfer the logic into a Python script or a compiled language.
Answers will largely depends on types of work, but for data analysis of large tabular data sets I couldn’t imagine how I would be efficient without good knowledge of:
gawk
parallel
split
paste
But this is only after getting to know pros and cons of other, dare I say more basic tools, such as cut and egrep.
Also knowledge on using a terminal multiplexer (tmux) is also quite useful.
Anything I should add to my .vimrc? It currently looks like this:
syntax on
set nocompatible
set smartindent
set shiftwidth=4
set backspace=indent,eol,start
set ruler
set number
set showcmd
set incsearc
set hlsearch
set mouse=a
filetype off
I’ve tried using some different VIM addons but they seem to be slightly beyond my grasp of VIM and modify the default behavior more than I like.
I use all of these. In fact one of my most commonly invoked aliases is histg, which is history | grep. So, histg awk gives me a quick reminder of how I last parsed that log file!
Read this thread for the first time. Good info thanks.
I use most of what is mentioned, although I have never mastered them all.
My question:
All you guys listing the necessary bash commands/structures do any of you feel that you actually totally mastered them ?
Its been twenty years since I started using Unx shell programming, but I feel it is an endless pit…
Every day I get stumped,… learn…get stumped … learn … relearn fundamentals…get stumped.
Sheesh!
Once I was so bored I made a ping-pong game using only bash 