Linux Command Line and Shell Scripting Bible,3rd，PartI

PartI TheLinuxCommandLine 

Chapter1 Starting with Linux Shells

略...
--------------------------------------

Chapter2 Getting to the Shell

略...
--------------------------------------

Chapter3 Basic bash shell Commands

--------------------------------------

>>> Listing Files and Directories

$ls -F

The -F parameter flags the directories with a forward slash (/), to help identify them inthe listing. Similarly, it flags executable files with

an asterisk (*), to help you more easily find files that can be run on the system .

$ ls -a

To display hidden files along with normal files and directories, use the-aparameter. 

$ls -F -R

The -R parameter shows files that are contained within subdirectories in the current directory.  

$ls -l (另外参考 stat 命令)

The -l parameter produces a long listing format, providing more information  .

drwxr-xr-x. 2 hadoop hadoop 4096 Oct 8 2016Desktop/

---------------------------

• The file type — such as directory (d), file (-), linked file (l), character device (c),or block device (b)

• The file permissions (see Chapter 6)

• The number of file hard links (See the section “Linking Files” in Chapter 7.)

• The file owner username

• The file primary group name

• The file byte size

• The last time the file was modified

• The filename or directory name 

---------------------------

   Filtering listing output :

0. the ls command also provides a way for you to define a filter .

• A question mark (?) to represent one character

• An asterisk (*) to represent any number of characters 

>>>Handling Files

Creating  Files:


$ touch test_one

use the touch command to easily create an empty file。
The touch command can also be used to change the modification time.
This is done with- out changing the file contents


$ touch -a test_one
To change only the access time, use the -a parameter with the touch command: 


$ ls -l --time=atime test_one
This is because the modification time is shown by default
To see a file’s access time, you need to add an additional parameter, --time=atime.


Copying  Files

$ cp -i test_one test_two
add the -i option to force the shell to ask whether you want to overwrite a file

$ cp -R Scripts/ Mod_Scripts
The -R parameter recursively copy the contents of an entire directory in one command.


Linking  Files

If you need to maintain two (or more) copies of the same file on the system, 
instead of having separate physical copies, 
you can use one physical copy and multiple virtual copies, called links.
A link is a placeholder in a directory that points to the real location of the file.


■ A symbolic link

A symbolic link is simply a physical file that points to another file somewhere 
in the virtual directory structure. 
The two symbolically linked together files do not share the same contents.


$ ln -s data_file sl_data_file
$ ls -i *data_file
The inode number of a file or directory is a unique identification number 
that the kernel assigns to each object in the filesystem.
To view a file or directory’s inode number, add the -i parameter to the ls command.
 
 
■ A hard link

A hard link creates a separate virtual file that contains information about the original file and where to locate it.
However, they are physically the same file.
 
$ ln code_file hl_code_file




Renaming  Files:

In the Linux world, renaming files is called moving files.


$ mv fall fzll

Deleting  Filse

$ rm -i fall

>>>Managing Directories

$ rmdir New_Dir
By default, the rmdir command works only for removing empty directories.

$ rm -ri My_Dir
Using the -r option allows the command to descend into the directory, 
remove the files, and then remove the directory itself.

$ tree Small_Dir

>>>Viewing File Contents

$ file my_file
The file command can peek inside of a file and 3 determine just what kind of file it is.

$ cat test1
$ cat -n test1
$ cat -b test1  ( just want to number the lines that have text in them)
$ cat -T test1 (replaces any tabs in the text with the ^I character combination)
The cat command is for displaying all the data inside a text file.

$ more  test1(displays a text file, but stops after it displays each page)

$ lesstest1
the less command is an advanced version of the more command.
can also display a file’s contents before it finishes reading the entire file. The cat and more commands cannot do this.

$ tail log_file
$ tail -n 2 log_file
The -f parameter allows you to peek inside a file as the file is being used by other processes. 

$ head log_file(By default, it displays the first 10 lines of text)
$ head -5 log_file

/*******************************************************/

Chapter4  More bash shell Commands

/*******************************************************/

Section : Monitoring Programs

$ ps
By default, the ps command shows only the processes that belong to the current user and that are running on the current terminal.
The GNU ps command that’s used in Linux systems supports three different types of com- mand line parameters:


■ Unix-style parameters, which are preceded by a dash
■ BSD-style parameters, which are not preceded by a dash
■ GNU long parameters, which are preceded by a double dash

$ps --forest

CMDbash

 \_ bash     \_ bash

$ top
The top command displays process information similarly to the ps command,
        but it does it in real-time mode


$ kill 3940
# kill -s HUP 3940
To send a process signal, you must either be the owner of the process or be logged in as the root user
tells the process to kindly stop running,When you need to get forceful, the -s parameter allows you to specify other signals (either using their name or signal number).


# killall http*
stop processes by using their names
kills all the processes that start with http

Section: Monitoring Disk Space

$ mount
$ mount -t type device directory
$ mount -t vfat /dev/sdb1 /media/disk
# umount /home/rich/mnt


By default, the mount command displays a list of media devices currently mounted on the system:




$ df
$ df -h 
see how much disk space is available on an individual device
<-h : shows the disk space in human- readable form>

$ du
The du command shows the disk usage for a specific directory (by default, the current directory)
By default, the du command displays all the files, directories, and subdirectories under the current directory

-c: Produces a grand total of all the files listed
-h: Prints sizes in human-readable form,K/M/G
-s: Summarizes each argument

Section : Working with Data Files

$ sort file2
By default, the sort command sorts the data lines in a text file using standard sorting rules for the language you specify as the default for the session.
$ sort -n file2
use the -n parameter, which tells the sort command to recognize numbers as numbers instead of characters and to sort them based on their numerical values
$ sort -M file3
that’s used is -M, the month sort.for log file start with a  "Sep 13 07:10:09 testbox smartd[2718]: Device: /dev/sda, opened"
$ sort -t ':' -k 3 -n /etc/passwd




$ du -sh * | sort -nr




grep [options] pattern [file]
The grep command searches either the input or the file you specify .
The output from grep is the lines that contain the matching pattern.

$ grep -v t file1
output lines that don’t match the pattern


$ grep -n t file1
find the line numbers where the matching patterns are found

$ grep -c t file1
count of how many lines contain the matching pattern

$ grep -e t -e f file1
specify more than one matching pattern, use the -e parameter

$ grep [tf] file1
outputs lines that contain either the string t or the string f.
By default, the grep command uses basic Unix-style regular expressions to match patterns.



The egrep commandis an offshoot of grep, which allows you to specify POSIX extended regular expressions
The fgrep command  allows you to place a list of strings in a file and then use that list in the fgrep command to search for the strings in a larger file。


Compressing data:
The gzip utility is the most popular compression tool used in Linux.


■ gzipfor compressing files
■ gzcatfor displaying the contents of compressed text files
■ gunzip for uncompressing files


Archiving data:
By far the most popular archiving tool used in Unix and Linux is the tar command.

tar function [options] object1 object2 ...
function : defines what the tar command should do
Each function uses options to define a specific behavior for the tar archive file


$tar -cvf test.tar test/ test2/
creates an archive file called test.tar containing the contents of both the test directory and the test2 directory.


$tar -tf test.tar
lists (but doesn’t extract) the contents of the tar file test.tar


$tar -xvf test.tar
extracts the contents of the tar file test.tar.


$tar -zxvf filename.tgz
These are gzipped tar  files,which can beextracted

************************************************

Chapter5  Understanding the Shell

************************************************

Section : Exploring Shell Types

-------------------------------
In the /etc/ passwd file, the user ID has its default shell program listed in field #7 of its record.

Section : Exploring Parent and Child Shell Relationships

--------------------------------------------------------
a parent shell: The default interactive shell started when a user logs into a virtual console terminal or starts a terminal emulator in the GUI
a child shell : When the /bin/bash command or the equivalent bash command is entered at the CLI prompt, a new shell program is created
  A child shell is also called a subshell.
However, entering into a sub- shell is an expensive method and can significantly slow down processing
It is not truly multi-processing, because the terminal gets tied up with the subshell’s I/O.

a command list:
----------------
$ pwd ; ls ; cd /etc ; pwd ; cd ; pwd ; ls
a list of commands to be run one after another,
by entering a command list using a semicolon (;) between commands

a process list:(is a command grouping type.)
-------------------------------------------
$ (pwd ; ls ; cd /etc ; pwd ; cd ; pwd ; ls)
the commands must be encased in parentheses
Adding parentheses and turning the command list into a process list created a subshell to execute the commands.
$ ( pwd ; echo $BASH_SUBSHELL) 
/home/Christine
1
$ ( pwd ; (echo $BASH_SUBSHELL)) 
/home/Christine
2

Investigating background mode:
-----------------------------
Running a command in background mode allows the command to be processed and frees up
your CLI for other use.


$ sleep 10
$
The command sleep 10 causes the session to pause for 10 seconds and then return a shell CLI prompt:

$ sleep 3000&
[1] 2396
$ ps -f
To put a command into background mode, the & character is tacked onto its end.
The first informational item is the background job’s number (1)
The second item is the background job’s process ID (2396).


$ jobs
$ jobs -l :The -l parameter displays the command’s PID 
[1]+ Running sleep 3000 & 
$
The ps command was used to display the various processes.
The jobs command displays any user’s processes (jobs) currently running in background mode:


Putting process lists into the background:
-----------------------------------------
a process list is a command or series of commands executed within a sub- shell.


$ (sleep 2 ; echo $BASH_SUBSHELL ; sleep 2)&
$ (tar -cf Rich.tar /home/rich ; tar -cf My.tar /home/christine)&


Looking at co-processing:
------------------------
Co-processing does two things at the same time:
It spawns a subshell in background mode 
and
executes a command within that subshell.

To perform co-processing, the coproc command is used along with the command to be executed in the subshell:


$ coproc sleep 10


$ coproc My_Job { sleep 10;}
You can set the name given to the process by the coproc command.
My_Job 空格 {空格 sleep 10; 空格}
The only time you need to name a co-process is when you have multiple co-processes running, and you need to communi- cate with them all

Section : Understanding Shell Built-In Commands

-----------------------------------------------


Looking at external commands:
----------------------------
  is a program that exists outside of the bash shell. They are not built into the shell program.
 
  An external command program is typically located in /bin, /usr/bin, /sbin, or /usr/sbin.
 
  Whenever an external command is executed, a child process is created. 
  This action is termed forking.
 
If you fork a child process or create a subshell, you can still communicate with it via signaling,

Looking at built-in commands:
-----------------------------
they do not need a child process to execute.

$ type cd
cd is a shell builtin
$ type -a ps 
ps is /bin/ps

$ type -a echo
echo is a shell builtin 
echo is /bin/echo
To see multiple flavors for commands, use the -a option on the type command:


$ which echo
/bin/echo
the which command shows only the external command file.




$ history
To see a recently used commands list,
$!!
To recall and reuse your last command, type !! and press the Enter key:
$ !20
Command number 20 was pulled from command history.
$ history -a
force the command history to be written to the .bash_history file before leaving a shell session.


$history -n
multiple terminal sessions,forcethe .bash_history reread and reupdated



Using command aliases:
----------------------
A command alias allows you to create an alias name for common commands (along with their parameters) to help keep your typing to a minimum.
$ alias -p
To see a list of the active aliases


$ alias li='ls -li'
create your own aliases using the alias command:
Be aware that because command aliases are built-in commands, 
an alias is valid only for the shell process in which it is defined:

**************************************************

Chapter 6 : Using Linux Environment  Variables

**************************************************

Section : Exploring Environment Variables

==========================================


Global environment variables:
visible from the shell session and from any spawned child subshells
Local variables:
available only in the shell that creates them.

Looking at global environment variables
---------------------------------------


$ printenv
To view global environment variables, use the env or the printenv

$ printenv HOME
display an individual environment variable’s value


$ echo $HOME
display a variable’s value,place a dollar sign ($) before the environment variable name


Looking at local environment variables
------------------------------------------


$ set
displays all variables defined for a specific process, 
including both local and global environment variables and user-defined variables:

Section : Setting User-De ned Variables

==========================================


not use spaces between 
the variable name, 
the equal sign,
and the value.

Setting local user-de ned variables
-----------------------------------


$ my_variable="Hello World"
$ my_variable=Hello
$ echo $my_variable
If you need to assign a string value that contains spaces, you need to use a single or double quotation mark 

It’s extremely important that you 
not use spaces between 
the variable name, 
the equal sign,
and the value.


if you set a local variable in a child process, after you leave the child process, 
the local variable is no longer available.

Setting global environment variables
------------------------------------
first create a local variable and then export it to the global environment。

$ my_variable="I am Global now"
$ export my_variable


1.Changing a global environment variable within a child shell does not affect the variable’s value in the parent shell。
2.A child shell cannot even use the export command to change the parent shell’s global environment variable’s value。

Section : Removing Environment Variables

==========================================
You can do this with the unset command. remember not to use the dollar sign:

$ echo $my_variable
$ unset my_variable

Section : Uncovering Default Shell Environment Variables

=====================================================

Section : Setting the PATH Environment Variable

=================================================
The PATH environment vari- able defines the directories it searches looking for commands and programs.
The directories in the PATH are separated by colons.


$ echo $PATH
$ PATH=$PATH:/home/christine/Scripts
$ echo $PATH
The changes are not persistent:
Changes to the PATH variable last only until you exit the system or the system reboots.

Section : Locating System Environment Variables

===============================================
how these environment variables are made persistent.


When you start a bash shell by logging in to the Linux system, by default bash checks several files for commands. 
These files are called startup files or environment files.
The startup files that bash processes depend on the method you use to start the bash shell


You can start a bash shell in three ways:
■ As a default login shell at login time
■ As an interactive shell that is started by spawning a subshell
■ As a non-interactive shell to run a script



Understanding the login shell process
---------------------------------------
When you log in to the Linux system, the bash shell starts as a login shell.
The login shell typically looks for five different startup files to process commands from:
■ /etc/profile
the main default startup file
All users on the system execute this startup file when they log in.

■ $HOME/.bash_profile 
■ $HOME/.bashrc
■ $HOME/.bash_login
■ $HOME/.profile
The other four startup files are specific for each user and can be customized for an indi- vidual user’s requirements.

Understanding the interactive shell process
-------------------------------------------
If you start a bash shell without logging into a system (if you just type bash at a CLI prompt, for example), you start what’s called an interactive shell.
it doesn’t process the /etc/profile file. Instead, it only checks for the .bashrc file in the user’s HOME directory.






Understanding the non-interactive shell process
------------------------------------------------
 This is the shell where the system can start to execute a shell script.
any variables set but not exported by the parent shell are local variables. 
Local variables are not inherited by a subshell.




Making environment variables persistent
-----------------------------------------
For global environment variables ，
create a file ending with .sh in the /etc/profile.d directory. 
In that file, place all your new or modified global environment variable settings.


store an individual user’s persistent bash shell variables is in the $HOME/.bashrc file.

Section : Learning about Variable Arrays

==========================================
A really cool feature of environment variables is that they can be used as arrays.


$ mytest=(one two three four five)
To set multiple values for an environment variable,
just list them in parentheses, with values separated by spaces.

$ echo $mytest (Only the first value in the array appears.)
one
$ 


$ echo ${mytest[2]} reference an individual array element
three
$


$ echo ${mytest[*]}To display an entire array variable
one two three four five 
$




$ mytest[2]=sevenchange the value of an individual index position:
$
$ echo ${mytest[*]}
one two seven four five 
$




$ unset mytest[2] the unset command to remove an individual value within the array,
$
$ echo ${mytest[*]} 
one two four five
$
$ echo ${mytest[2]}be careful, because this gets tricky. 


$ echo ${mytest[3]} 
four
$






$ unset mytest remove the entire array just by using the array name
$
$ echo ${mytest[*]}


$

**************************************************

Chapter 7  Understanding Linux File Permissions

**************************************************

Section :  Linux Security

==========================================
The core of the Linux security system is the user account.


How Linux handles user accounts:


The /etc/passwd  file
------------------------
A special file to match the login name to a corresponding UID.
The Linux system creates lots of user accounts for various functions that aren’t actual users.
These are called system accounts.
A system account is a special account that services running on the system use to gain access to resources on the system.
All services that run in background mode need to be logged in to the Linux system under a system user account.

The fields of the /etc/passwd file contain the following information:
■ The login username
■ The password for the user
■ The numerical UID of the user account
■ The numerical group ID (GID) of the user account
■ A text description of the user account (called the comment field)
■ The location of the HOME directory for the user
■ The default shell for the user


 The /etc/shadow  file
 ----------------------- 
 Now, most Linux systems hold user passwords in a separate file 
 (called the shadow file, located at /etc/shadow). 
 Only special programs (such as the login program) are allowed access to this file.
 
 


 Adding a new user
 ---------------------
 useradd
 This command provides an easy way to create a new user account and set up the user’s HOME directory structure all at once. 
 Uses a combination of system default values and command line parameters to define a user account. 
 The system defaults are set in the /etc/default/useradd file.
 
 # /usr/sbin/useradd -D
 To see the system default values used
 
 
 Removing a user
 ------------------------
 userdel
 
 Modifying a user
 ----------------------
 Linux provides a few different utilities for modifying the information for existing user accounts.
 
 usermod：Edits user account  elds,
 passwd :Changes the password for an existing user
 chpasswd : Reads a  le of login name and password pairs, and updates the passwords
 chage :Changes the password’s expiration date
 chfn :Changes the user account’s comment information
 chsh : Changes the user account’s default shell
 
 

Section : Using Linux Groups

=================================
groups:
allowing groups of users to share resources
Group permissions allow multiple users to share a common set of permissions for an object on the system

The /etc/group  file
---------------------
 Just like user accounts, group information is stored in a file on the system.




Creating new groups
-----------------------
 groupadd:
 to create new groups on your system:
 doesn’t provide an option for adding user accounts to the group.
 to add new users, use the usermod command:
 
# /usr/sbin/groupadd shared
# tail /etc/group
# /usr/sbin/usermod -G shared test




Modifying groups
----------------------
# /usr/sbin/groupmod -n sharing shared
change the GID (using the -g param- eter) 
or the group name (using the -n parameter) of an existing group

Section : Decoding File Permissions

======================================


Using  file permission symbols
-------------------------------


$ ls -l
total 48
drwxr-xr-x.  2 hadoop hadoop 4096 Oct  8  2016 Desktop
drwxrwxr-x. 12 hadoop hadoop 4096 Nov 25  2016 hadoop
-rw-rw-r--.  1 hadoop hadoop 1541 Oct  9  2016 my_swap_message.txt
-rw-rw-r--.  1 hadoop hadoop    2 Oct  9  2016 my_swap_message.txt~




The first field : describes the permissions for the files and directories.
The first character in the field defines the type of the object:

■ - for files
■ d for directories
■ l for links
■ c for character devices
■ b for block devices
■ n for network devices




■ r for read permission for the object
■ w for write permission for the object
■ x for execute permission for the object
If a permission is denied, a dash appears in the location.




three levels of security for the object:
■ The owner of the object
■ The group that owns the object
■ Everyone else on the system


permissions for : the file owner+ group members+  everyone else




Default  file permissions
-----------------------------
The umask command 
sets the default permissions for any file or directory you create.


$ touch newfile
$ ls -al newfile
-rw-r--r-- 1 rich rich 0 Sep 20 19:16 newfile 
$
touch命令有两个功能：
一是用于把已存在文件的时间标签更新为系统当前的时间（默认方式），它们的数据将原封不动地保留下来；
二是用来创建新的空文件。


$ umask:It masks out the permissions you don’t want to give to the security level.
The umask value is subtracted from the full permission set for an object.
The umask value also applies to making new directories.
0022
$


The first digit :represents a special secu- rity feature called the sticky bit.
Octal mode security settings take the three rwx permission values and convert them into a 3-bit binary value, represented by a single octal value.


--- 000 0No permissions
rwx 111 7Read, write, and execute permissions


The full permis- sion for a file is mode 666 (read/write permission for all), 
but for a directory it’s 777 (read/ write/execute permission for all).

Section : Changing Security Settings

=======================================


Changing permissions
-----------------------


$ chmod 760 newfile
chmod options mode file
change the security settings for files and directories.

specifying a permission in sym- bolic mode:
-------------------------------------------
[ugoa...][[+-=][rwxXstugo...]


The first group of characters defines to whom the new permissions apply:
■ u for the user
■ g for the group
■ o for others (everyone else)
■ a for all of the above


Next, a symbol is used to indicate whether you want to 
add the permission to the existing permissions (+), 
subtract the permission from the existing permission (−), or 
set the per- missions to the value (=).

Finally, the third symbol is the permission used for the setting.
You may notice that there are more than the normal rwx values here. These are the additional settings:
■ X assigns execute permissions only if the object is a directory or if it already had execute permissions.
■ s sets the UID or GID on execution.
■ t saves program text.
■ u sets the permissions to the owner’s permissions.
■ g sets the permissions to the group’s permissions.
■ o sets the permissions to the other’s permissions.


$ chmod o+r newfile
$ ls -lF newfile
-rwxrw-r-- 1 rich rich 0 Sep 20 19:16 newfile*
The o+r entry adds the read permission to whatever permissions the everyone security
level already had.


$ chmod u-x newfile
$ ls -lF newfile
-rw-rw-r-- 1 rich rich 0 Sep 20 19:16 newfile
The u-x entry removes the execute permission that the user already had.


The options parameters:
The -R parameter performs the file and directory changes recursively.



Changing ownership
---------------------


chown options owner[.group] file
The chown command makes it easy to change the owner of a file。


# chown dan newfile
specify either the login name or the numeric UID for the new owner of the file
# chown dan.shared newfile
change both the user and group of a file

$ chgrp shared newfile

The chgrp command allows you to change the default group of a file。
The user account must own the file, and be a member of the new group as well to be able
to change the group.

Section : Sharing Files

==============================
in a large environment if you want to create and share documents among several people,
There are three additional bits of information that Linux stores for each file and directory:
■ The set user id (SUID): When a file is executed by a user, the program runs under the permissions of the file owner.
■ The set group id (SGID): For a file, the program runs under the permissions of the file group. For a directory, new files created in the directory use the directory group as the default group.
■ The sticky bit: The file remains (sticks) in memory after the process ends.

**************************************

Chapter 8 Managing Filesystems

**************************************

Section : Exploring Linux Filesystems

======================================

Section : Working with Filesystems

=====================================


[fsck options filesystem]
The fsck command is used to check and repair most Linux filesystem types

Section : Managing Logical Volumes

====================================
Linux Logical Volume Manager (LVM)
dynamically add more space to an existing file- system by just adding a partition from another hard drive to the existing filesystem.


physical volumes (PV)
 In the logical volume management world, hard drives are called physical volumes (PV). 

************************************

Chapter 9 Installing Software

************************************

Section : The Debian-Based Systems

=====================================
The dpkg command is at the core of the Debian-based family of PMS tools.


These other tools are included in this PMS:
■ apt-get
■ apt-cache 
■ aptitude




[aptitude show package_name]
$ aptitude show mysql-client


One detail you cannot get with aptitude is a listing of all the files associated with a par- ticular software package.
To get this list, you must go to the dpkg tool itself:
[dpkg -L package_name]



You can also do the reverse — find what package a particular file belongs to:
[dpkg --search absolute_file_name]
$ dpkg --search /usr/bin/xxd vim-common: /usr/bin/xxd

Section : The Red Hat–Based Systems

======================================
■ yum: Used in Red Hat and Fedora
■ urpm: Used in Mandriva
■ zypper: Used in openSUSE


Listing installed packages:
-----------------------------
#yum list installed
# yum list xterm :find out detailed information for a particular software package


#yum provides file_name : find out what software package provides a particular file
# yum provides /etc/yum.conf




Installing software with yum
------------------------------


#yum install package_name
#yum localinstall package_name.rpm :manually download an rpm installation file and install it using yum


Updating software with yum
-----------------------------
#yum list updates :To see the list of all the available updates for your installed packages
#yum update package_name :a particular software package needs updating
#yum update : update all the packages listed in the update list




Uninstalling software with yum
---------------------------------
#yum remove package_name
To just remove the software package and keep any configuration and data files,

#yum erase package_name
To uninstall the software and all its files



Dealing with broken dependencies
-----------------------------------
broken dependency:


Sometimes, as multiple software packages get loaded, 
a software dependency for one pack- age can get overwritten by the installation of another package. 
This is called a broken dependency.


If this should happen on your system:
yum clean all
yum update

If that doesn’t solve the problem, try the following command:
#yum deplist package_name


displays all the package’s library dependencies and what software package provides them
After you know the libraries required for a package, you can then install them


If that doesn’t solve your problem, you have one last tool:
#yum update --skip-broken

yum repositories
-------------------




Section : Installing from Source Code
=======================================
This section walks you through the process of unpacking and installing a tarball software package.


# tar -zxvf sysstat-11.1.1.tar.gz
# ./configure
# make
# make install