UNIX Tools and Tricks

Last modified Monday, 30-Jan-2023 16:37:51 UTC.

Editing
- There are 3 ways for interactive editing text under UNIX:
  - The vi way
  - The emacs way
  - The wrong way
- vi (the visual editor) is the text editor written by Bill Joy for BSD UNIX (published about 1978)
  - Screen-oriented WYSIWYG editor (for plain text)
  - Available on just about any UNIX system
  - About 35% of all serious UNIX hackers still prefer vi (or a derivative)
  - vim is an improved version
    vimtutor will get you going
  - gvim is a windows based interface to vim
- emacs (editing macros) started in 1976 as a set of TECO macros on ITS
  - Currently popular emacs versions (GNU Emacs and XEmacs) go back to 1985 GNU Emacs by Stallman. Both basically are a LISP system with a large text editing library and an editor-like user interface
  - About 35% of all serious UNIX hackers use Emacs. Also widespread use on other operating systems
  - emacs has zillions of extensions
    ^H t wil get you going
- pico is a wrong way
  nano is a wrong way
  gedit is a wrong way
  kedit is a wrong way
  Anything other than vi or emacs is a wrong way!
Less typing
- History
  - Set the number of commands to store in your shell resource file, e.g., set history=100.
  - history lists recent commands. Gurus alias that to "h".
  - !<number> redoes that command
  - !<partial> redoes the most recent command that started with <partial>
- Globbing
  - Describe to the shell (not OS) sets of file names with patterns
  - ~ means the home directory
  - ? means any character
    * means any string (including empty)
  - [<letters>] means any one of the <letters> (except ...), which can also be a range
  - ^ negates some glob expressions
- File name completion
  - Many shells, including tcsh, provide file name completion
  - In tcsh, turn on with
```
set autoexpand
set autolist 
```
    in the .tcshrc
  - Use tab to expand partial file names
Foreground and background processes
- Ooooh, the Ticker
- Foreground processes are started by typing a command. While a foreground process is running the shell waits.
- Background processes are started by appending a & to the command.
  - While a background process is running the shell continues.
  - Output from the background process is mixed in ... the solution is to send the stdout to a file, leaving stderr to screen.
  - If a background process needs input, it gets suspended (read on).
- Each process you start from the shell is known as a "job", i.e., you have maximally one foreground job and can have multiple background jobs.
- Background processes can be controlled
  - jobs lists background processes, and their numbers
  - kill %N terminates a background process by job number
  - fg %N brings a background process into the foreground
- The foreground process can be suspended and moved to the background
  - ^Z suspends the foreground process
  - jobs also lists suspended processes
  - bg %N activates a suspended process in the background
  - fg %N also activates a suspended process in the foreground
  - kill %N also terminates a suspended process
- A background process will be suspended automatically if it needs to read data from stdin. Some programs, e.g., perl check stdin when they start, so as background processes they hang. The solution is to redirect stdin to come from /dev/null.
- nohup can be used to leave jobs running when you logout ("hangup").
  - disown is a lesser version, with different effect
Sequences and groups
- Simple commands are commands
- Pipelines of simple commands are commands
- Sequences of simple commands with & are commands, and will be executed concurrently.
- Sequences of simple commands with ; are commands, and will be executed sequentially.
- Commands in ()s are simple commands.
- Aggregate - Script that uses aggregate commands
Manipulating files
- find searches for files
  - -name to find a file by name
  - -empty to find empty files and directories
  - -type to find files of various types (bcdflps)
- more and less display a file one page at a time
  head and tail display the start and end of a file
  od can display non-text files
- diff compares text files
  wc counts characters, words, lines
Regular expressions
- Regular expressions describe sets of strings with patterns (not the same as globbing)
  - A normal character matches itself
  - . matches any normal character
  - A range [<letters>] matches any one of the <letters> (except ...), which can also be a range
    [^<letters>] matches any not one of the <letters>
  - ? after a pattern makes it optional
    + after a pattern matches one or more repetitions
    * after a pattern matches any number of repetitions
    {<N>} after a pattern matches <N> repetitions in regexps
  - ^ means the start of the line
    $ means the end of the line
  - ()s round a regexp makes it one thing to which repetition and placement options can be applied.
- regex101 is a great tool to play and learn regular expressions.
Finding file contents with grep
- grep finds lines in files that match limited regexps.
- grep -E finds lines in files that match a regexp
  - grep -E "^[a-zA-Z]+" looks for lines that start with some alphabetic characters.
  - grep -E "a*b+c{4}" looks for lines containing any number of as followed by some bs followed by 4 cs
  - grep -E '^(a*b+c{4})+$' looks for lines containing exactly repetitions of the abcs (note the use of ''s to avoid confusion with shell variables)
- Example: Extract the legal, Java style, int variable declarations from GrepExInts.
- Example: Extract the University of Miami lines from GrepExUMs.
- grep has some useful options
  - -c to count number of matches
  - -l to list files names that match
  - -v to list lines that don't match
Stream editing with sed
- sed is a stream editor, i.e., designed to take input from a file or stdin, edit the input as it flows through, and send the edited result to stdout.
- The editing commands can be given with -e or in a file specified with -f.
- Example:
```
sed -e "s/foo/bar/g" MyFile
```
  will replace all instances of foo with bar in MyFile
- Example:
```
grep -l int *.c | sed -e "/^Example/d" -e "s/\.c//"
```
  will grep for the names of all .c files containing int (we could do better there :-), delete lines that start with Example, and remove the .c from the remaining names.
- Example:
```
sed -f MySedScript MyFile 
```
  will apply the commands in MySedScript to MyFile. Here's an example for MySedScript
```
1,5d
/^#/d
/^START/,/END$/s/  */ /g
s/ *$// 
```
  Those commands will delete lines 1 to 5, delete lines that start with a #, in the lines from the first that starts with START to the first that ends with END replace sequences of spaces with a single space, then on all lines delete all trailing spaces.
- sed can do a lot. Read Bruce Barnett's friendly tutorial. Then read the man pages!

Exercises

Set your history to store the last 100 commands, and install an alias h to list your history.
Is globbing a cool word or what?
Turn on file name completition and listing in your shell. Find out how to use name completion in your editor.
Copy and compile the Ticker.c program. Then do the following sequence
1. Start a ticker with a 7 second delay, in the foreground
2. Suspend the ticker
3. List your jobs
4. Start a ticker with a 9 second delay in the background
5. List your jobs
6. Activate the 7 second ticker in the background
7. List your jobs
8. kill the 9 second ticker
9. kill the 7 second ticker
Use find to make a list of all the empty files in your account.

Provide a grep command to extract the legal (both syntactically and with correct identifier style) integer variable definitions from the following lines:

    int aDog;
int   aDog  ;
//  int aCommentAboutADog;
    double aBigDog;
    int BadDog;
    int dogWithNoTail
    int aDog,aCat;
    int   aSpaceDog,   aSpaceCat;
    int aDog aBadCat;
    internationalDog;
    int a#Dog;
int internetName; // fooo

Exam Style Questions

At a command prompt, what would the following commands do:
```
!17
!gli 
```
In globbing, what do the * amd ? characters designate?
Explain the difference between a background process and the foreground process.
Explain the difference between these two commands
```
ls & ps & date &
ls ; ps ; date 
```
What does the following command output?
```
find / -empty | wc -w 
```
Which of the following lines are recognised by the regexp ^a*b+c{2}?
```
bbbbcc
abbccc
  aaabbbccc
aaabbbcccddd
bccbccbccbccbcc 
```