Newer
Older
1001
1002
1003
1004
1005
1006
1007
1008
1009
1010
1011
1012
1013
1014
1015
1016
1017
1018
1019
1020
1021
1022
1023
1024
1025
1026
1027
1028
1029
1030
1031
1032
1033
1034
1035
1036
1037
1038
1039
1040
1041
1042
1043
1044
1045
1046
1047
1048
1049
1050
1051
1052
1053
<code>select</code> construct lets the user choose which value is used next.
This also means that a <code>select</code> construct can run indefinitely,
because the user can keep selecting new choices. To avoid being trapped in it,
we have to explicitly use <code>break</code>. <code>break</code> is a builtin
command that makes bash jump out of the current <code>do</code> block.
Execution will continue after the <code>done</code>. <code>break</code> also
works in <code>for</code> and <code>while</code> loops.
</p>
<p>
As you can see in the example above, we used an <code>if</code> command inside
a <code>select</code> command. All of these conditional constructs
(<code>if</code>, <code>for</code>, <code>while</code>, <code>case</code> and
<code>select</code>) can be nested indefinitely.
</p>
<h1 data-number="7" id="input-and-output">
<span class="header-section-number">7</span> Input and Output
</h1>
<p>
Input and output in bash is very flexible and, consequentially, complex. We
will only look at the most widely used components.
</p>
<h2 data-number="7.1" id="command-line-arguments">
<span class="header-section-number">7.1</span> Command-line Arguments
</h2>
<p>
For many bash scripts, the first input we care about are the arguments given
to it via the command line. As we saw in the chapter on Parameters, these
arguments are contained in some <em>special parameters</em>. These are called
<em>positional parameters</em>. The first parameter is referred to with
<code>$1</code>, the second with <code>$2</code>, and so on. After number 9,
you have to enclose the numbers in curly braces: <code>${10}</code>,
<code>${11}</code> and so on.
</p>
<p>
In addition to referring to them one at a time, you may also refer to the
entire set of positional parameters with the <code>"$@"</code> substitution.
The double quotes here are <strong>extremely important</strong>. If you don’t
use the double quotes, each one of the positional parameters will undergo word
splitting and globbing. You don’t want that. By using the quotes, you tell
Bash that you want to preserve each parameter as a separate word.
</p>
<p>
There are even more ways to deal with parameters. For example, it is very
common for commands to accept <em>options</em>, which are single letters
starting with a <code>-</code>. For example, <code>ls -l</code> calls the
<code>ls</code> program with the <code>-l</code> option, which makes it output
more information. Usually, multiple options can be combined, as in
<code>ls -la</code>, which is equivalent to <code>ls -l -a</code>.
</p>
<p>
You might want to create your own scripts that accept some options. Bash has
the so called <code>getopts</code> builtin command to parse passed options.
</p>
<pre><code>while getopts 'hlf:' opt
do
case "$opt" in
h|\?)
echo 'available options: -h -l -f [filename]'
;;
f)
file="$OPTARG"
;;
l)
list=true
;;
esac
done
shift "$(( OPTIND - 1 ))"</code></pre>
<p>
As you can see, we use <code>getopts</code> within a while loop.
<code>getopts</code> will return 0 as long as there are more options remaining
and something else if there are no more options. That makes it perfectly
suitable for a loop.
</p>
<p>
<code>getopts</code> takes two arguments, the <em>optstring</em> and the
<em>variable name</em>. The optstring contains all the letters that are valid
options. In our example, these are <code>h</code>, <code>l</code> and
<code>f</code>.
</p>
<p>
The <code>f</code> is followed by a colon. This indicates that the f option
requires an argument. The script could for example be called like so:
</p>
1087
1088
1089
1090
1091
1092
1093
1094
1095
1096
1097
1098
1099
1100
1101
1102
1103
1104
1105
1106
1107
1108
1109
1110
1111
1112
1113
1114
1115
1116
1117
1118
1119
1120
1121
1122
1123
1124
1125
1126
1127
1128
1129
1130
<p>
<code>getopts</code> will set the variable that you specified as its second
argument to the letter of the option it found first. If the option required an
argument, the variable <code>OPTARG</code> is set to whatever the argument
was. In the example above, a case statement is used to react to the different
options.
</p>
<p>
There’s also a special option <code>?</code>. Whenever
<code>getopts</code> finds an option that is not present in the optstring, it
sets the shell variable (<code>opt</code> in the example) to <code>?</code>.
In the case statement above, that triggers the help message.
</p>
<p>
After all options are parsed, the remaining arguments are “moved” such that
they are now in <code>$1</code>, <code>$2</code>… even though previously,
these positional parameters were occupied by the options.
</p>
<p>
Also note the line <code>shift "$(( OPTIND - 1 ))"</code> at the end. The
<code>shift</code> builtin can be used to discard command-line arguments. Its
argument is a number and designates how many arguments we want to discard.
</p>
<p>
This is needed because we don’t know how many options the user will pass to
our script. If there are more positional parameters after all the options, we
have no way of knowing at which number they start. Fortunately,
<code>getopts</code> also sets the shell variable <code>OPTIND</code>, in
which it stores the index of the option it’s going to parse next.
</p>
<p>
So after parsing all the option, we just discard the first
<code>OPTIND - 1</code> options, and the remaining arguments now start from
<code>$1</code> onwards.
</p>
<h2 data-number="7.2" id="file-descriptors">
<span class="header-section-number">7.2</span> File Descriptors
</h2>
<p>
<em>File descriptors</em> are the way programs refer to files, or other things
that work like files (such as pipes, devices, or terminals). You can think of
them as pointers that point to data locations. Through these pointers,
programs can write to or read from these locations.
</p>
<p>By default, every program has three file descriptors:</p>
<ul>
<li>Standard Input (stdin): File Descriptor 0</li>
<li>Standard Output (stdout): File Descriptor 1</li>
<li>Standard Error (stderr): File Descriptor 2</li>
<p>
When you run a script in the terminal, then stdin contains everything you type
in that terminal. stdout and stderr both point to the terminal, and everything
that is written to these two is displayed as text in the terminal. stdout is
where programs send their normal information, and stderr is where they send
their error messages.
</p>
<p>
Let’s make these definitions a little more concrete. Consider this example:
</p>
<pre><code>echo 'What is your name?'
read name
echo "Good day, $name. Would you like some tea?"</code></pre>
<p>
You already know <code>echo</code>. It simply prints its argument to
<em>stdout</em>. Since stdout is connected to your terminal, you will see that
message there.
</p>
<p>
<code>read</code> is a command that reads one line of text from
<em>stdin</em> and stores it in a variable, which we specified to be
<code>name</code>. Because stdin is connected to what you type in your
terminal, it will let you type a line of text, and as soon as you press enter,
that line will be stored in the variable.
</p>
<p>So what about stderr? Consider this example:</p>
<pre><code>ehco 'Hello!'</code></pre>
<p>
The command <code>ehco</code> does not exist. If you run this, bash will print
an error message to <em>stderr</em>. Because stderr is connected to your
terminal, you will see that message there.
</p>
<h2 data-number="7.3" id="redirection">
<span class="header-section-number">7.3</span> Redirection
</h2>
<p>
<em>Redirection</em> is the most basic form of input/output manipulation in
bash. It is used to change the source or destination of
<em>File descriptors</em>, i.e. connect them to something other than your
terminal. For example, you can send a command’s output to a file instead.
</p>
<pre><code>echo 'It was a dark and stormy night. Too dark to write.' > story</code></pre>
1179
1180
1181
1182
1183
1184
1185
1186
1187
1188
1189
1190
1191
1192
1193
1194
1195
1196
1197
1198
1199
1200
1201
<p>
The <code>></code> operator begins an <em>output redirection</em>. It
redirects the <em>stdout</em> file descriptor of the command to the left, and
connects it to a file called “story”. That means if you run this, you will not
see the output of <code>echo</code>—after all, stdout no longer points to your
terminal.
</p>
<p>
Note that <code>></code> will just open the file you specify without
checking whether it already exists first. If the file already exists, its
contents will be overwritten and you will lose whatever was stored in there
before. <strong>Be careful.</strong>
</p>
<p>
If you don’t want to overwrite the existing content of a file, but rather
append your output to the end of that file, you can use
<code>>></code> instead of <code>></code>.
</p>
<p>
Now, let’s look at <em>input redirection</em>. For that, we first introduce a
command named <code>cat</code>. <code>cat</code> is often used to display the
contents of a file, like so:
</p>
<p>
If <code>cat</code> is called without an argument, however, it will simply
read from <em>stdin</em> and print that directly to <em>stdout</em>.
</p>
<p>
Try the following: Run <code>cat</code>, without arguments, in your terminal.
Then type some characters and hit enter. Can you figure out what is happening?
</p>
<p>
<em>Input redirection</em> uses the <code><</code> operator. It works as
follows:
</p>
<p>
The <code><</code> operator will take a command’s <em>stdin</em> file
descriptor and point it to a file, “story” in this example. This means
<code>cat</code> now ignores your terminal and reads from “story” instead.
Note that this has the same effect as typing <code>cat story</code>.
</p>
<p>
If you want to redirect <em>stderr</em> instead of <em>stdout</em>, you can do
as follows:
</p>
<pre><code>ehco 'Hello!' 2> errors</code></pre>
<p>
If you run this, you won’t see any error message, even though the command
<code>ehco</code> doesn’t exist. That’s because <em>stderr</em> is no longer
connected to your terminal, but to a file called “errors” instead.
</p>
<p>
Now that you know about redirection, there is one subtlety that you have to be
aware of: You can’t have two file descriptors point to the same file.
</p>
<p>
If you wanted to log a command’s complete output–stdout
<em>and</em> stderr–you might be tempted to do something like this:
</p>
<pre><code>mycommand > logfile 2> logfile</code></pre>
<p>
However, this is a <strong>bad</strong> idea. The two file descriptors will
now both point to the same file <em>independently</em>, which causes them to
constantly overwrite each other’s text.
</p>
<p>
If you still want to point both stdout and stderr to the same file, you can do
it like this:
</p>
<pre><code>mycommand > logfile 2>&1</code></pre>
<p>
Here, we use the <code>>&</code> syntax to duplicate file descriptor 1.
In this scenario, we no longer have two file descriptors pointing to one file.
Instead, we have only one file descriptor that acts as both stdout and stderr
at the same time.
</p>
<p>
To help remember the syntax, you can think of <code>&1</code> as “where 1
is”, and of the <code>2></code> as “point 2 to”. The whole thing,
<code>2>&1</code>, then becomes “point 2 to wherever 1 is”. This also
makes it clear that <code>> logfile</code> has to come <em>before</em>
<code>2>&1</code>: First you point 1 to “logfile”, and only then you
can point 2 to where 1 is.
</p>
<p>
There’s also a quick way to completely get rid of a command’s output using
redirections. Consider this:
</p>
<pre><code>mycommand > /dev/null 2>&1</code></pre>
1270
1271
1272
1273
1274
1275
1276
1277
1278
1279
1280
1281
1282
1283
1284
1285
1286
1287
1288
1289
1290
<p>
<code>/dev/null</code> is a special file in your file system that you can
write to, but the things you write to it are not stored. So,
<code>mycommand</code>’s output is written somewhere where it’s not stored or
processed in any way. It is discarded completely.
</p>
<p>
You could also leave out the <code>2>&1</code>. Then, you’d still see
error messages, but discard the normal output.
</p>
<h2 data-number="7.4" id="pipes">
<span class="header-section-number">7.4</span> Pipes
</h2>
<p>
Now that you know how to manipulate <em>file descriptors</em> to direct output
to files, it’s time to learn another type of I/O redirection.
</p>
<p>
The <code>|</code> operator can be used to connect one command’s
<em>stdout</em> to another command’s <em>stdin</em>. Have a look at this:
</p>
<pre><code>echo 'This is a beautiful day!' | sed 's/beauti/wonder'</code></pre>
1292
1293
1294
1295
1296
1297
1298
1299
1300
1301
1302
1303
1304
1305
1306
1307
1308
1309
1310
1311
1312
1313
1314
1315
1316
1317
1318
1319
1320
1321
1322
1323
1324
1325
1326
1327
<p>
The <code>sed</code> command (“sed” is short for “stream editor”) is a utility
that can be used to manipulate text “on the fly”. It reads text from stdin,
edits it according to some commands, and then prints the result to stdout. It
is very powerful. Here, we use it to replace “beauti” with “wonder”.
</p>
<p>
First, the <code>echo</code> command writes some text to it’s stdout. The
<code>|</code> operator connected <code>echo</code>’s stout to
<code>sed</code>’s stdin, so everything <code>echo</code> sends there is
immediately picked up by <code>sed</code>. <code>sed</code> will then edit the
text and print the result to its own stdout. <code>sed</code>’s stdout is
still connected to your terminal, so this is what you see.
</p>
<h1 data-number="8" id="compound-commands">
<span class="header-section-number">8</span> Compound Commands
</h1>
<p>
<em>Compound commands</em> is a catch-all phrase covering several different
concepts. We’ve already seen <code>if</code>, <code>for</code>,
<code>while</code>, <code>case</code>, <code>select</code> and the
<code>[[</code> keyword, which all fall into this category. Now we’ll look at
a few more.
</p>
<h2 data-number="8.1" id="subshells">
<span class="header-section-number">8.1</span> Subshells
</h2>
<p>
<em>Subshells</em> can be used to encapsulate a command’s effect. If a command
has undesired side effects, you can execute it in a subshell. Once the
subshell command ends, all side effects will be gone.
</p>
<p>
To execute a command (or several commands) in a subshell, enclose them in
parenthesis:
</p>
<pre><code>(
cd /tmp
pwd
)
pwd</code></pre>
<p>
The <code>cd</code> and the first <code>pwd</code> commands are executed in a
subshell. All side effects in that subshell won’t affect the second
<code>pwd</code> command. Changing the current directory is such a side
effect–even though we use <code>cd</code> to go to the
<code>/tmp</code> folder, we jump back to our original folder as soon as the
subshell ends.
</p>
<h2 data-number="8.2" id="command-grouping">
<span class="header-section-number">8.2</span> Command Grouping
</h2>
<p>
You can group several commands together by enclosing them in curly braces.
This makes bash consider them as a unit with regard to pipes, redirections and
control flow:
</p>
<pre><code>{
echo 'Logfile of my backup'
rsync -av . /backup
echo "Backup finished with exit code $#"
} > backup.log 2>&1</code></pre>
<p>
This redirects stdout and stderr of <em>all three commands</em> to a file
called backup.log. Note that while this looks similar to subshells, it is not
the same. Side effects that happen within the curly braces will still be
present outside of them.
</p>
<h2 data-number="8.3" id="arithmetic-evaluation">
<span class="header-section-number">8.3</span> Arithmetic Evaluation
</h2>
<p>
So far, we’ve only been manipulating strings in bash. Sometimes, though, it is
also necessary to manipulate numbers. This is done through arithmetic
evaluation.
</p>
<p>
Say you want to add the numbers 5 and 4. You might do something like this:
</p>
<p>
However, this will result in the variable <code>a</code> containing the string
<code>5+4</code>, rather than the number <code>9</code>. Instead, you should
do this:
</p>
<p>
The double parenthesis indicate that something arithmetic is happening. In
fact, <code>((</code> is a bash keyword, much like <code>[[</code>.
</p>
<p><code>((</code> can also be used to do arithmetic comparison:</p>
<pre><code>if (( 5 > 9 ))
then
echo '5 is greater than 9'
else
echo '5 is not greater than 9'
fi</code></pre>
<p>
It is important not to confuse <code>((</code> and <code>[[</code>.
<code>[[</code> is for comparing strings (among other things), while
<code>((</code> is only for comparing numbers.
</p>
<p>
There’s also <em>arithmetic substitution</em>, which works similarly to
<em>command substitution</em>:
</p>
<pre><code>echo "There are $(( 60 * 60 * 24 )) seconds in a day."</code></pre>
<h1 data-number="9" id="functions">
<span class="header-section-number">9</span> Functions
</h1>
<p>
Inside a bash script, functions are very handy. They are lists of
commands–much like a normal script–except they don’t reside in their own file.
They do however take arguments, just like scripts.
</p>
<p>Functions can be defined like this:</p>
<pre><code>sum() {
echo "$1 + $2 = $(( $1 + $2 ))"
}</code></pre>
<p>
If you put this in a script file and run it, absolutely nothing will happen.
The function <code>sum</code> has been defined, but it is never used.
</p>
<p>
You can use your function like any other command, but you have to define it
<em>before</em> you use it:
</p>
<pre><code>sum() {
echo "$1 + $2 = $(( $1 + $2 ))"
}
sum 1 2
sum 3 9
sum 6283 3141</code></pre>
<p>
As you can see, you can use the function <code>sum</code> multiple times, but
you only need to define it once. This is useful in larger scripts, where a
certain task has to be performed multiple times. Whenever you catch yourself
writing the same or very similar code twice in the same script, you should
consider using a function.
</p>
<h1 data-number="10" id="useful-commands">
<span class="header-section-number">10</span> Useful Commands
</h1>
<p>
This chapter provides an overview of useful commands that you can use in your
scripts. It is nowhere near complete, and serves only to provide a brief
overview. If you want to know more about a specific command, you should read
its manpage.
</p>
<p>
<code>grep</code> can be used to search for a string within a file, or within
the output of a command.
</p>
<pre><code># searches logfile.txt for lines containing the word error
grep 'error' logfile.txt
# searches the directory 'folder' for files
# containing the word 'analysis'
grep 'analysis' folder/
# searches the output of 'xrandr' for lines that say 'connected'.
# only matches whole words, so 'disconnected' will not match.
xrandr | grep -w 'connected' </code></pre>
<p>
<code>grep</code> returns 0 if it finds something, and returns an error if it
doesn’t. This makes it useful for conditionals.
</p>
<p>
<code>sed</code> can be used to edit text “on the fly”. It uses its own
scripting language to describe modifications to be made to the text, which
makes it extremely powerful. Here, we provide examples for the most common
usages of sed:
</p>
<pre><code># replaces the first occurrence of 'find' in every line by 'replace'
sed 's/find/replace' inputfile
# replaces every occurrence of 'find' in every line by 'replace'
sed 's/find/replace/g' inputfile
# deletes the first occurrence of 'find' in every line
sed 's/find//' inputfile
# deletes every occurrence of 'find' in every line
sed 's/find//g' inputfile
# displays only the 12th line
sed '12q;d' inputfile </code></pre>
<p>
<code>sed</code> is often used in combination with pipes to format the output
or get rid of unwanted characters.
</p>
<p>
<code>curl</code> and <code>wget</code> are two commands that can be used to
access websites or other content from the web. The difference is that
<code>wget</code> will simply download the content to a file, while
<code>curl</code> will output it to the console.
</p>
<pre><code>curl http://www.thealternative.ch
wget http://files.project21.ch/LinuxDays-Public/16FS-install-guide.pdf</code></pre>
<p><strong>xrandr</strong></p>
<p>
<code>xrandr</code> can be used to manipulate your video outputs,
i.e. enabling and disabling monitors or setting their screen resolution and
orientation.
</p>
<pre><code># list all available outputs and their status info
xrandr
# enables output HDMI-1
xrandr --output HDMI-1 --auto
# puts output HDMI-1 to the left of output LVDS-1
xrandr --output HDMI-1 --left-of LVDS-1
# disables output LVDS-1
xrandr --output LVDS-1 --off </code></pre>
<p><strong>ImageMagick (convert)</strong></p>
<p>
The <code>convert</code> command makes it possible to do image processing from
the commandline.
</p>
<pre><code># scale fullHDWallpaper.jpg to specified resolution
convert fullHDWallpaper.jpg -scale 3200x1800 evenBiggerWallpaper.jpg
# "automagically" adjusts the gamma level of somePicture.jpg
convert somePicture.jpg -auto-gamma someOtherPicture.jpg
# transform image to black and white
convert colorfulPicture.jpg -monochrome blackWhitePicture.jpg </code></pre>
<p>
It is extremely powerful and has lots of options. A good resource is the
<a href="http://www.imagemagick.org">official website</a>. It also provides
examples for most options.
</p>
<p>
<code>notify-send</code> can be used to display a desktop notification with
some custom text:
</p>
<pre><code>notify-send 'Battery warning' 'Your battery level is below 10%'</code></pre>
<p>
The first argument is the notification’s title, the second is its description.
</p>
<p>
<code>notify-send</code> requires a <em>notification daemon</em> to be
running, else it won’t work. Most desktop environments come with a
notification daemon set up and running. If you can’t see your notifications,
it might be that you don’t have such a daemon.
</p>
<p><strong>find</strong></p>
<p><code>find</code> can be used to find files in a directory structure.</p>
<pre><code># finds all files in the current directory and all subdirectories that end
# in .png
find -name '*.png'
# finds all files ending in .tmp and removes them. {} is replaced by the
# file's name when executing the command.
# Note that we don't use globbing here, but instead pass the * to find.
# find will then interpret the * as a wildcard.
find -name '*.tmp' -exec rm '{}'
# finds all files in the directory 'files' and prints their size and path
find 'files/' -printf '%s %p\n' </code></pre>
<p>
<code>find</code> has many options that allow you to perform arbitrary actions
on the files it found or pretty-print the output.
</p>
<p>
<code>sort</code> sorts lines of text files, or lines it reads from
<em>stdin</em>.
</p>
<pre><code>sort listOfNames.txt # sorts all lines in listOfNames.txt alphabetically</code></pre>
<p><strong>head and tail</strong></p>
<p>
<code>head</code> and <code>tail</code> can be used to show the beginning or
the end of a long stream of text, respectively.
</p>
<pre><code># display the last few lines of the dmesg log
dmesg | tail
# display only the first few lines of a very long text file
head verylongtext.txt </code></pre>
<p><strong>jshon</strong></p>
<p>
<code>jshon</code> is a very simple command-line json parser. It can read data
in json format and return specific values.
</p>
<p>
Its most important options are <code>-e</code> and <code>-a</code>.
<code>-e</code> extracts the value of a given key from a json array or object:
</p>
<pre><code># Find the object with key "title" from a json object stored in the file "json"
jshon -e 'title' < 'json'</code></pre>
<p>
The <code>-a</code> option maps all remaining options across the currently
selected element. It has to be combined with other options, for example the
<code>-e</code> option.
</p>
<pre><code># Find the names of all elements stored in the json object in file "json"
jshon -a -e 'name' < 'json'</code></pre>
<p><strong>shuf</strong></p>
<p>
<code>shuf</code> randomly permutates the lines of its input, effectively
<em>shuffling</em> them.
</p>
<pre><code># Shuffle the lines of file "playlist"
shuf 'playlist'
# Get a random line from file "quotes"
shuf -n 1 'quotes'</code></pre>
<p><strong>tee</strong></p>
<p><code>tee</code> takes input from <em>stdin</em> and writes it to a file:</p>
<pre><code>sudo zypper up | tee 'updatelog'</code></pre>
<p>Note that this is equivalent to</p>
<pre><code>sudo zypper up > 'updatelog'</code></pre>
<p>
<code>tee</code> is useful when you want to write to a file that requires root
access. Then you can do the following:
</p>
<pre><code>echo 'some configuration' | sudo tee '/etc/systemconfig'</code></pre>
<p>
A normal redirection wouldn’t work in this case, as that would open the file
as a normal user instead of root.
<strong>Please be careful when modifying system files.</strong>
</p>
<p>
<code>sleep</code> pauses the execution of your script for a number of
seconds. It basically takes a number as an argument, then does nothing for
that number of seconds, and then returns. It can be useful if you want to make
your script do something in a loop, but want to throttle the speed a little.
</p>
<pre><code># prints "Get back to work" once per second.
# Note that 'true' is a command that always returns 0.
while true
do
echo 'Get back to work!'
sleep 1
done</code></pre>
<p><strong>mplayer or mpv</strong></p>
<p>
<code>mplayer</code> and <code>mpv</code> are media players that can be used
from the console. <code>mpv</code> is based on <code>mplayer</code> and a lot
more modern, but they can do essentially the same.
</p>
<pre><code># play file "music.mp3"
mplayer 'music.mp3'
mpv 'music.mp3'
# play file "alarm.mp3" in an infinite loop
mplayer -loop 0 'alarm.mp3'
mplayer --loop=inf 'alarm.mp3'
# play a youtube video
# this only works with mpv and requires youtube-dl to be installed
mpv 'https://www.youtube.com/watch?v=lAIGb1lfpBw'</code></pre>
<p><strong>xdotool</strong></p>
<p>
<code>xdotool</code> can simulate keypresses. With this command, you can write
macros that perform actions for you. It allows you to write scripts that
interact with GUI programs.
</p>
<pre><code># press ctrl+s (in order to save a document)
xdotool key 'Control+s'
# type "hello"
xdotool type 'hello'</code></pre>
<!-- end pandoc -->