Useful Linux Commands

sort

sort -T . (use currently directory instead system tmp)

sort -t ”     ” -k3,3nr -k4,4n  input.txt > output.txt (sort the third numerical column in descending order and the forth numerical column in ascending order)

awk

awk '{s+=$1} END {print s}' mydatafile (sum number, one per line)

du

du –si * (show total file usage in human readable way)

ln

ln -f -s target_directory symbolic_link_directory (This will update existing link without removing old one.)

Uniq command

After sorting a file you will often find that some duplicate data, or you may be given various lists that need deduping. sort and uniq will quickly and easily remove duplicates, lsit only the dupilcates or only the unique data.

sort myfile.txt | uniq

List only the unique lines: sort myfile.txt | uniq -u

List only the duplicate lines: sort myfile.txt | uniq -d

Get a count of the number of lines by adding the -c option.

sort myfile.txt | uniq -uc

sort myfile.txt | uniq -dc

Skip fields: uniq -f 3 mylogfile. this could be useful with log files to skip the time stamp data

Skip characters. uniq -s 30 myfile.txt. Skip the first 30 characters

Compare characters. uniq -w 30 myfile.txt. Compare the first 30 characters

Generate SSH Key

  • ssh-keygen -t rsa -b 4096

Read RPM

  • rpm -qlp rpm-name

Search directory

  • find / -name directory-name -type d

Change to lower case

  • $ tr '[:upper:]' '[:lower:]' < input.txt > output.txt

Screen command

  • Ctrl+a c (new window)
  • Ctrl+a n (select the window)
  • Ctrl+a d (detach the session)
  • screen -ls (list session)
  • screen -r session name (attach to a detach session)
  • screen -rD session name (attach to an active session)

Read RPM content

  • rpm -qlp rpm-name

Find a directory or a file

  • find / -name directory-name -type d
  • find / -name file-name -type f

How do I extract a gz file?

  • $ gunzip file.gz
  • $ gzip -d file.gz

How do I extract a tar.gz or .tgz file?

  • $ gunzip < file.tar.gz | tar xvf –
  • $ gunzip < file.tgz | tar xvf –
  • $ tar xvzf file.tar.gz
  • $ tar xvzf file.tgz

Processing the delimited files

  • cut -d: -f1,3  /etc/passwd
  • awk -F’:’   ‘{print $1,$3}’   /etc/passwd

Copy files from remote machine

  • rcp -rp username@hostname(or IP):/path localPath (r: recursive, p: preserve)
  • scp is similar and based on SSH

Check HDFS system

  • bin/hadoop dfsadmin -report
    Configured Capacity: 603258257408 (561.83 GB)
    Present Capacity: 455640743936 (424.35 GB)
    DFS Remaining: 455623647232 (424.33 GB)
    DFS Used: 17096704 (16.3 MB)
    DFS Used%: 0%
    Under replicated blocks: 0
    Blocks with corrupt replicas: 0
    Missing blocks: 0
    
    -------------------------------------------------
    Datanodes available: 2 (2 total, 0 dead)
    
    Name: 192.168.241.22:50010
    Decommission Status : Normal
    Configured Capacity: 452167245824 (421.11 GB)
    DFS Used: 8548352 (8.15 MB)
    Non DFS Used: 30800699392 (28.69 GB)
    DFS Remaining: 421357998080(392.42 GB)
    DFS Used%: 0%
    DFS Remaining%: 93.19%
    Last contact: Thu Apr 21 09:59:01 PDT 2011
    
    Name: 192.168.241.23:50010
    Decommission Status : Normal
    Configured Capacity: 151091011584 (140.71 GB)
    DFS Used: 8548352 (8.15 MB)
    Non DFS Used: 116816814080 (108.79 GB)
    DFS Remaining: 34265649152(31.91 GB)
    DFS Used%: 0.01%
    DFS Remaining%: 22.68%
    Last contact: Thu Apr 21 09:59:02 PDT 2011
    

Others

Advertisements

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s