#### Useful Linux/Unix One Liners ####
1. Display Username and UID sorted by UID Using cut, sort and tr
Cut command is used to extract specific part of a file. The following example cuts the username and UID from /etc/passwd file, and sort the output using sort command using username as a key and ":" as a delimiter
As a part of formatting the output, you can use any other character to display username and UID. Using tr command you can convert to ":" to any other character
$ cut -d ':' -f 1,3 /etc/passwd | sort -t ':' -k2n - | tr ':' '\t'
2. Find List of Unique Words in a file Using tr, sed, uniq
The following example lists the words which has only alphabets. 'tr' command converts all the character other than alphabets to newline. So all the words will be listed out with number of newlines. Sed command removes the empty lines and finally uniquely sort the output to avoid the duplicates.
$ tr -c a-zA-Z '\n' < Readme1.txt | sed '/^$/d' | sort | uniq -i -c
Note: uniq with -i ignores the cases
Linux 'sed' command plays a vital role in test manipulation operations
3. Join Two Files (Where one file is not sorted ) Using sort and Join
Join Command joins two files based on a common field between two files.For join to work properly, both the files should be sorted.
In the example below, the file m1.txt Employee name and Employee Id and its not sorted. Second file m2.txt has employee name and Department name. To join these two files, sort the first file and give the sorted output as one of the input stream of join.
$ sort m1.txt | join - m2.txt
4. Find out which process is using up your memory using ps,awk,sort
The following command lists all the process sorted based on the used memory size.
$ ps aux | awk ' { if ($5 != 0) print $2,$5,$6,$11 } ' | sort -k2n
The above command lists the PID, Used virtaul memory size, Used resident set-size and process command.
Awk is an extremely useful language to manipulate structured data very quickly.
5. Find Out Top 10 Largest File or Directory Using du,sort and head
'du' command shows summarized disk usage for each file and directory of a given location(/var/log/*). The output of a sort command is reversely sorted based on the size
# du -sk /var/log/* | sort -r -n | head -10
6. Find out Top 10 most Used commands.
Bash maintains all the commands you execute in a hidden file called .bash_history under your home directory
Use the following one liner to identify which command you execute a lot from your command line.
$cat ~/.bash_history | tr "\|\;" "\n" | sed -e "s/^ //g" | cut -d " " -f 1 | sort | uniq -c | sort -n | tail -n 15
7. Display timestamp using HISTTIMEFORMAT
Typically when you type history form command line, it displays the command# and the command. For auditing purpose, it may be beneficial to display the timestamp along the command
# export HISTTIMEFORMAT='%F %T'
8. Search the histroy using Control+R
9. Repeat perious command quickly using 4 different methods
Sometime you may end up repeatig the previous commands for various reasons. following are the 4 different ways to repeat the last executed command.
- Use the up arrow
- Type !!
- Type !-1
- Press Control+P
10. Eliminate the continuous repeated entry form history using HISTCONTROL
export HISTCONTROL=ignoredups
11. Erase duplictes accross the whole history using HISTCONTROL
export HISTCONTROL=erasedups
12. Disable the usage of history using HISTSIZE
export HISTSIZE=0
13. Ignore specific commands from the history using HISTIGNORE
Sometimes you may not want to clutter your history with basic commands such as pwd and ls
$export HISTIGNORE="pwd:ls:ls -ltr:"
##### Find, Exec , Xargs ###
## Find
Display the pathnames of all files in the current directory and all subdirectories. The commands
#find . -print
#find -print
#find .
This will search any filename that begins with foo and ends with bar
#find . -name foo\*bar
Example using two search criteria:
#find / -type f -mtime -7 | xargs tar -rf weekly_incremental.tar
Note
Will find any regular files (i.e., not directories or other special files) with the criteria "-type f", and only those modified seven or fewer days ago ("-mtime -7"). Note the use of xargs, a handy utility that coverts a stream of input (in this case the output of find) into command line arguments for the supplied command (in this case tar, used to create a backup archive).
Another use of xargs is illustrated below. This command will efficiently remove all files named core from your system (provided you run the command as root of course):
#find / -name core | xargs /bin/rm -f
#find / -name core -exec /bin/rm -f '{}' \; # same thing
#find / -name core -delete # same if using Gnu find
(The last two forms run the rm command once per file, and are not as efficient as the first form. However the first form is safer if rewritten to use "-print0".)
The find criteria is used to locate files modified less than 10 minutes ago
#find / -mmin -10
Recently downloaded file want to locate
#find / -cmin -10
-cmin n = File's status was last changed n minutes ago.
-mmin n = File's data was last modified n minutes ago.
-mtime n = File's data was last modified n*24 hours ago.
Find files with various permissions set. "-perm /permissions"
will locate files that are writeable by "others"
#find . -perm -o=w
#find . -mtime 0 # find files modified between now and 1 day ago
# (i.e., within the past 24 hours)
#find . -mtime -1 # find files modified less than 1 day ago
# (i.e., within the past 24 hours, as before)
#find . -mtime 1 # find files modified between 24 and 48 hours ago
#find . -mtime +1 # find files modified more than 48 hours ago
#find . -mmin +5 -mmin -10 # find files modified between
# 6 and 9 minutes ago
This says to seach the whole system, skipping the directories /proc, /sys, /dev, and /windows-C-Drive (presumably a Windows partition on a dual-booted computer). The Gnu -noleaf option tells find not to assume all remaining mounted filesystems are Unix file systems (you might have a mounted CD for instance). The "-o" is the Boolean OR operator, and "!" is the Boolean NOT operator (applies to the following criteria).
So these criteria say to locate files that are world writable ("-perm -2", same as "-o=w") and NOT symlinks ("! -type l") and NOT sockets ("! -type s") and NOT directories with the sticky (or text) bit set ("! \( -type d -perm -1000 \)"). (Symlinks, sockets and directories with the sticky bit set are often world-writable and generally not suspicious.)
#find / -noleaf -wholename '/proc' -prune \
-o -wholename '/sys' -prune \
-o -wholename '/dev' -prune \
-o -wholename '/windows-C-Drive' -prune \
-o -perm -2 ! -type l ! -type s \
! \( -type d -perm -1000 \) -print
Using -exec Efficiently:
# find whatever... | xargs command
Two limitations
- Firstly not all commands accept the list of files at the end of the command. A good example is cp:
#find . -name \*.txt | xargs cp /tmp # This won't work! [ -t - we can handle the issue ]
- Secondly filenames may contain spaces or newlines, which would confuse the command used with xargs. (Again Gnu tools have options for that, "find ... -print0 |xargs -0 ...".)
An alternate form of -exec ends with a plus-sign, not a semi-colon. This form collects the filenames into groups or sets, and runs the command once per set.
(This is exactly what xargs does, to prevent argument lists from becoming too long for the system to handle.)
In this form the {} argument expands to the set of filenames. For example:
find / -name core -exec /bin/rm -f '{}' +
#find /opt -name '*.txxt' -type f -exec sh -c 'exec cp -f "$@" /tmp' find-copy {} \;
#find /path/to/files* -mtime +5 -exec rm {} \;
-exec, allows you to pass in a command such as rm. The {} \; at the end is required to end the command.
# dig -x IP @Dns_Server
1. Display Username and UID sorted by UID Using cut, sort and tr
Cut command is used to extract specific part of a file. The following example cuts the username and UID from /etc/passwd file, and sort the output using sort command using username as a key and ":" as a delimiter
As a part of formatting the output, you can use any other character to display username and UID. Using tr command you can convert to ":" to any other character
$ cut -d ':' -f 1,3 /etc/passwd | sort -t ':' -k2n - | tr ':' '\t'
2. Find List of Unique Words in a file Using tr, sed, uniq
The following example lists the words which has only alphabets. 'tr' command converts all the character other than alphabets to newline. So all the words will be listed out with number of newlines. Sed command removes the empty lines and finally uniquely sort the output to avoid the duplicates.
$ tr -c a-zA-Z '\n' < Readme1.txt | sed '/^$/d' | sort | uniq -i -c
Note: uniq with -i ignores the cases
Linux 'sed' command plays a vital role in test manipulation operations
3. Join Two Files (Where one file is not sorted ) Using sort and Join
Join Command joins two files based on a common field between two files.For join to work properly, both the files should be sorted.
In the example below, the file m1.txt Employee name and Employee Id and its not sorted. Second file m2.txt has employee name and Department name. To join these two files, sort the first file and give the sorted output as one of the input stream of join.
$ sort m1.txt | join - m2.txt
4. Find out which process is using up your memory using ps,awk,sort
The following command lists all the process sorted based on the used memory size.
$ ps aux | awk ' { if ($5 != 0) print $2,$5,$6,$11 } ' | sort -k2n
The above command lists the PID, Used virtaul memory size, Used resident set-size and process command.
Awk is an extremely useful language to manipulate structured data very quickly.
5. Find Out Top 10 Largest File or Directory Using du,sort and head
'du' command shows summarized disk usage for each file and directory of a given location(/var/log/*). The output of a sort command is reversely sorted based on the size
# du -sk /var/log/* | sort -r -n | head -10
6. Find out Top 10 most Used commands.
Bash maintains all the commands you execute in a hidden file called .bash_history under your home directory
Use the following one liner to identify which command you execute a lot from your command line.
$cat ~/.bash_history | tr "\|\;" "\n" | sed -e "s/^ //g" | cut -d " " -f 1 | sort | uniq -c | sort -n | tail -n 15
7. Display timestamp using HISTTIMEFORMAT
Typically when you type history form command line, it displays the command# and the command. For auditing purpose, it may be beneficial to display the timestamp along the command
# export HISTTIMEFORMAT='%F %T'
8. Search the histroy using Control+R
9. Repeat perious command quickly using 4 different methods
Sometime you may end up repeatig the previous commands for various reasons. following are the 4 different ways to repeat the last executed command.
- Use the up arrow
- Type !!
- Type !-1
- Press Control+P
10. Eliminate the continuous repeated entry form history using HISTCONTROL
export HISTCONTROL=ignoredups
11. Erase duplictes accross the whole history using HISTCONTROL
export HISTCONTROL=erasedups
12. Disable the usage of history using HISTSIZE
export HISTSIZE=0
13. Ignore specific commands from the history using HISTIGNORE
Sometimes you may not want to clutter your history with basic commands such as pwd and ls
$export HISTIGNORE="pwd:ls:ls -ltr:"
##### Find, Exec , Xargs ###
## Find
Display the pathnames of all files in the current directory and all subdirectories. The commands
#find . -print
#find -print
#find .
This will search any filename that begins with foo and ends with bar
#find . -name foo\*bar
Example using two search criteria:
#find / -type f -mtime -7 | xargs tar -rf weekly_incremental.tar
Note
Will find any regular files (i.e., not directories or other special files) with the criteria "-type f", and only those modified seven or fewer days ago ("-mtime -7"). Note the use of xargs, a handy utility that coverts a stream of input (in this case the output of find) into command line arguments for the supplied command (in this case tar, used to create a backup archive).
Another use of xargs is illustrated below. This command will efficiently remove all files named core from your system (provided you run the command as root of course):
#find / -name core | xargs /bin/rm -f
#find / -name core -exec /bin/rm -f '{}' \; # same thing
#find / -name core -delete # same if using Gnu find
(The last two forms run the rm command once per file, and are not as efficient as the first form. However the first form is safer if rewritten to use "-print0".)
The find criteria is used to locate files modified less than 10 minutes ago
#find / -mmin -10
Recently downloaded file want to locate
#find / -cmin -10
-cmin n = File's status was last changed n minutes ago.
-mmin n = File's data was last modified n minutes ago.
-mtime n = File's data was last modified n*24 hours ago.
Find files with various permissions set. "-perm /permissions"
will locate files that are writeable by "others"
#find . -perm -o=w
#find . -mtime 0 # find files modified between now and 1 day ago
# (i.e., within the past 24 hours)
#find . -mtime -1 # find files modified less than 1 day ago
# (i.e., within the past 24 hours, as before)
#find . -mtime 1 # find files modified between 24 and 48 hours ago
#find . -mtime +1 # find files modified more than 48 hours ago
#find . -mmin +5 -mmin -10 # find files modified between
# 6 and 9 minutes ago
This says to seach the whole system, skipping the directories /proc, /sys, /dev, and /windows-C-Drive (presumably a Windows partition on a dual-booted computer). The Gnu -noleaf option tells find not to assume all remaining mounted filesystems are Unix file systems (you might have a mounted CD for instance). The "-o" is the Boolean OR operator, and "!" is the Boolean NOT operator (applies to the following criteria).
So these criteria say to locate files that are world writable ("-perm -2", same as "-o=w") and NOT symlinks ("! -type l") and NOT sockets ("! -type s") and NOT directories with the sticky (or text) bit set ("! \( -type d -perm -1000 \)"). (Symlinks, sockets and directories with the sticky bit set are often world-writable and generally not suspicious.)
#find / -noleaf -wholename '/proc' -prune \
-o -wholename '/sys' -prune \
-o -wholename '/dev' -prune \
-o -wholename '/windows-C-Drive' -prune \
-o -perm -2 ! -type l ! -type s \
! \( -type d -perm -1000 \) -print
Using -exec Efficiently:
# find whatever... | xargs command
Two limitations
- Firstly not all commands accept the list of files at the end of the command. A good example is cp:
#find . -name \*.txt | xargs cp /tmp # This won't work! [ -t - we can handle the issue ]
- Secondly filenames may contain spaces or newlines, which would confuse the command used with xargs. (Again Gnu tools have options for that, "find ... -print0 |xargs -0 ...".)
An alternate form of -exec ends with a plus-sign, not a semi-colon. This form collects the filenames into groups or sets, and runs the command once per set.
(This is exactly what xargs does, to prevent argument lists from becoming too long for the system to handle.)
In this form the {} argument expands to the set of filenames. For example:
find / -name core -exec /bin/rm -f '{}' +
#find /opt -name '*.txxt' -type f -exec sh -c 'exec cp -f "$@" /tmp' find-copy {} \;
#find /path/to/files* -mtime +5 -exec rm {} \;
-exec, allows you to pass in a command such as rm. The {} \; at the end is required to end the command.
# dig -x IP @Dns_Server
No comments:
Post a Comment