Most useful BASH commands to master

unix
linux
bash

#1

“The problem with being self-taught is that you seldom know what it is you don’t know.” --a wise Autodidact.

Recently, the my boss introduced me to the power of perl to parse log files from the CLI, after I tried loading a million line csv in OpenOffice and crashed “ALL THE THINGS!”.

After a few hours of working with it, I turned simple parsing on its head and was doing things I hadn’t even conceived of being possible. For example, I needed to turn a list of errors into a MySQL query, so I came up with this:

cat failures.csv | grep key | grep -iv task | perl -pe 's|(\s+\"key":\s+\[")(\d+)("\]\R)|$2, |' > val_keys.sql

But that really got me thinking. What other tools do I know have in my belt just because I simply hadn’t had a reason to know about them yet?

xargs is on my list to learn next. What recommendations would You make to a relative newbie who spends about 6 hours a day staring at $PS1 and desperately needs to get better with BASH and the CLI in general?

ETA: Thanks for all the replies! To clarify, I’ve been using Linux for 6 years now, but I’ve only been a developer for a little over a year. In that time, I truly just ignored the CLI except as a platform to launch gcc, python, and the Metasploit framework until I took a new job that wasn’t InfoSec but instead s DevOps. I am fluent in C, Python, PHP, MySQL, & JS and can write some serious RegExes. But the tools of BASH are myriad and I want to get a jump on mastering the best of them!


#2

Someone who is using Linux or Unix since 1996 full time, my time mostly spent with the following tools:

  1. awk
  2. sed
  3. grep
  4. Regex (almost all commands use regex so master regex)
  5. Perl scripting
  6. Python scripting
  7. find
  8. bash scripting

How to master them? Read man pages. Second avoid using GUI tools. Instead use these tools to build your own solutions.


#3

To help with learning and to help with keeping track of your work two of the following tools are useful

screen - multiple terminal windows available on your target machine

script - redirects all STDIN/STDOUT/STDERR into a file for review. Great for working on instructions and step-by-step


#4

top
bash scripting
tail -f
and a much necessary - :wq to exit VI


#5

The most useful man page you can read: https://www.gnu.org/software/bash/manual/


#6

Umm… is it “most useful BASH commands”, or “most useful Linux tools”?

It all depends on what are you trying to do.

As far as text processing/parsing is considered I strongly recommend learning how to use below tools:

regex (!!!. Not a command but something YOU really really REALLY WANT to learn. Trust me)
man (yes, you must learn how to use it. And USE IT)
grep / egrep
sed
awk
tr
head / tail
column (to make output look as a table - usually much more human readable)

Once you have a good grip on them try learning some shell scripting to bring more logic to the filters (i.e. commands listed above): loops (while, for), variables, &1 and &2 descriptors and data stream redirection, conditionals (either if-else or more geeky [] && {} || {} ).

Also learn how to use ‘less’ and ‘vim’ (or ‘more’ ‘vi’ in older machines… like MUCH older).

Once you’re good with shell scripting you will be able to parse any output (or input) into anything you want.

As far as other commands are concerned (not for text parsing)… well, that depends entirely on what is it you want to do. Basically you can do anything :slight_smile: Once I was so bored I made a ping-pong game using only bash :slight_smile: Later - a shell-based chat application for my team (with channels, control statements, members lists, colors, etc… just like mirc). Just because I can…


#7

In addition to those already mentioned, check out:

  1. cut
  2. join
  3. sort
  4. uniq
  5. wc
  6. top
  7. ps
  8. bg
  9. jobs
  10. &
  11. kill
  12. killall
  13. sleep
  14. tr
  15. head
  16. tail
  17. less
  18. cat
  19. find

With *nix, the philosophy was to have many small tools that do one thing really well; then, they can quickly be glued together with a shell script to perform a more complex task. Once you have the logic flow down, if performance is important then you can transfer the logic into a Python script or a compiled language.

Enjoy…


#8

Answers will largely depends on types of work, but for data analysis of large tabular data sets I couldn’t imagine how I would be efficient without good knowledge of:

gawk
parallel
split
paste

But this is only after getting to know pros and cons of other, dare I say more basic tools, such as cut and egrep.

Also knowledge on using a terminal multiplexer (tmux) is also quite useful.


#9

hi,

info coreutils

#10

I like:

  • git - Keep project files in repository.
  • sudo - Very useful and comes with auditing feature.
  • iptables - If you manage server like CentOS 7 or Ubuntu. Must have to protect server.
  • strings - just show the printable strings from a file
  • tail and head - tail -f is really cool to view log files in real time
  • sftp/rsync - Send/upload and get file securely over the Internet
  • history - Recall commands

#11

VIM is BAE!

Anything I should add to my .vimrc? It currently looks like this:

syntax on
set nocompatible
set smartindent
set shiftwidth=4
set backspace=indent,eol,start
set ruler
set number
set showcmd
set incsearc
set hlsearch
set mouse=a

filetype off

I’ve tried using some different VIM addons but they seem to be slightly beyond my grasp of VIM and modify the default behavior more than I like.


#12

I love parallel! I was wondering how this was done. Thanks for the tip!


#13

I use all of these. In fact one of my most commonly invoked aliases is histg, which is history | grep. So, histg awk gives me a quick reminder of how I last parsed that log file!


#14

http://mywiki.wooledge.org/BashGuide
This bash guide help me a lot to start learning Bash seriously :wink: