diff --git a/linux_toolkit_2021/images/xkcd.png b/linux_toolkit_2021/images/xkcd.png new file mode 100644 index 0000000000000000000000000000000000000000..d22863e618dea21846acebeadae761d262c2a6eb --- /dev/null +++ b/linux_toolkit_2021/images/xkcd.png @@ -0,0 +1,3 @@ +version https://git-lfs.github.com/spec/v1 +oid sha256:d630973206a3d96c6c762a22a04649e33a5fa26dc2107a0832cd545be9e457d2 +size 73581 diff --git a/linux_toolkit_2021/index.html b/linux_toolkit_2021/index.html index 0f370f80bc266f88cbe683eedf66dc4efb551486..0e2c734d0c7c7c99b75aafe108003a9f03b91961 100644 --- a/linux_toolkit_2021/index.html +++ b/linux_toolkit_2021/index.html @@ -17,7 +17,7 @@ class: center, middle --- -# We will see +# We will see TODO ...a short **Bash** introduction, to get the necessary intuition for what most tools in this course do. @@ -34,14 +34,16 @@ Reduce unknown unknowns (`what's available?`, `what do I search for?`, etc). Give a rough intuition for what classic Linux tools can do. -Provide pointers for where to learn the tools. - Learn vocabulary. -Non-goal: teach you how to use the tools - I would love to, but we don't have time! +Provide pointers for where to learn the tools. + --- +# example: a download filer + +Displays the most recently downloaded file and asks what to do with it ```bash file=$(ls -tp1 ~/Downloads | grep -v "/$" | head -1) @@ -53,31 +55,30 @@ choice=$(echo $dirs | rofi -dmenu -sep ',' -p "move '$file' where? ") mv ~/Downloads/"$file" "$choice" ``` -safe the above in `/home/nils/filer.sh` and add a few lines of text in `~/.xbindkeysrc`: +Save the above in `/home/nils/filer.sh` and add a few lines of text in `~/.xbindkeysrc`: -``` +```bash "bash /home/nils/filer.sh" control+shift+f ``` --- -# How to use this talk +# How to use this talk TODO - pick and choose the tools that seem interesting to you and try to learn them - come back to the slides and use them as a glossary for unknown words -- ...or just be amused with what people come up with --- -# Some philosophy +# Some philosophy before Bash Many of the classical Linux tools adhere to the "UNIX principle": <blockquote>Do one thing and do it well.</blockquote> -In a wider, non-programming sense: It's useful to -learn things that are small, orthogonal, and compose. +This usually implies that they are built with composability in mind. +We like that because it means we learn more things by doing less. --- @@ -88,7 +89,7 @@ Related words: terminal, terminal emulator, shell, command line, ... <img src="images/bash.png" width="50%"> In a nutshell: -Bash is a tool that let's you interact with your computer via text. +Bash is a tool that let's you interact with your computer via text commands. **Try it out:** Open a terminal (just search for `Terminal` in your launcher) @@ -99,15 +100,18 @@ Bash is a tool that let's you interact with your computer via text. Bash let's you interact with your computer by typing commands. There are **many** commands. Some examples: -`ls`, `pwd`, `echo`, `cd`, `mv`, `sleep`, `alias`, ... +`ls`, `pwd`, `cd`, `echo`, `mv`, `cat`, `alias`, ... Some commands are built into Bash, others can be downloaded. Some commands are really just programs: try typing `firefox` and hitting Enter if you have it installed. +Most commands take options that modify their default behavior. +`df` versus `df -h` + --- -# Filesystem basics +# Basics: Taking a Walk List files and show where we are: @@ -116,10 +120,10 @@ ls pwd ``` -Enter or leave directories: +Enter and leave directories: ```bash -cd my_folder +cd linuxdaysDemo cd .. cd /home/nils/Desktop cd ~/Desktop @@ -129,14 +133,13 @@ cd ~/Desktop # Bash terminology -**terminal** is the window in which bash runs. - -bash is a kind of **shell**. there are others! - +The **terminal** is the window in which bash runs. -<img src="images/terminal_old.webp" alt="https://www.reddit.com/r/raspberry_pi/comments/6se1qy/old_terminal_connected_to_my_pi_emulating_a_modem/" width="60%"> +Bash is a kind of **shell**. There are others (I use Zsh). +<img src="images/terminal_old.webp" alt="https://www.reddit.com/r/raspberry_pi/comments/6se1qy/old_terminal_connected_to_my_pi_emulating_a_modem/" width="60%"> +An actual terminal --- @@ -145,7 +148,8 @@ bash is a kind of **shell**. there are others! Bash can even be used as a (terrible) programming language, using commands like `for`, `if`, `while`, `[[` etc. -It also makes use of special constructs that aren't commands (e.g. arithmetic expansion: `echo $((5+3))`) +It also makes use of special constructs that aren't commands +(e.g. arithmetic expansion: `echo $((5+3))`) We'll only bother with `|` and `>` ("piping" and "redirecting"). @@ -163,35 +167,64 @@ Both redirect command output: --- -# bash summary +# Shell scripts -bash is a different way of interacting with your system +Shell scripts are just textfiles. +We can put commands into them and then execute them all at once. -bash runs in your terminal. +Let's use the editor **nano** to create a file: -bash is a shell. there are others: Zsh, fish, Dash, csh +```bash +nano ~/betterCd.sh +``` + +(inside nano:) +```bash +#!/bin/sh +cd "$1" +ls +``` + +Exit nano by pressing Control-o, Enter, and Control+x. +Then execute: + +````bash +. ~/betterCd.sh ~/Documents +``` -resources: +If you want, add `alias c='. ~/betterCd.sh'` to the file `~/.bashrc`. + +--- + +# Bash Summary + +Bash is a different way of interacting with your system + +Bash runs in your terminal. + +Bash is a shell. there are others: Zsh, fish, Dash, csh + +Resources: - our bash guide (https://thealternative.ch/guides/bash.php) - Lhunath and GreyCat's bash guide (http://mywiki.wooledge.org/FullBashGuide) --- -# bash power +# Teaser Move all `.png` files from Downloads to Images: ```bash -mv ~/Downloads/*.png ~/Images` +mv ~/Downloads/*.png ~/Images ``` -go to our images, make a new directory, and move newest five images there: +Go to our images, make a new directory, and move newest five images there: ```bash cd ~/Images mkdir newImages ls -1t *.png | head -5 | xargs -I {} mv {} newImages ``` -rename all images in a folder to `*_with_family`: +Rename all images in the current folder to `*_with_family`: ```bash for file in *.png; do mv "$file" "$file_with_family" @@ -200,23 +233,38 @@ done --- +# Some final tips + +Use the Tab key for autocompletion (hit twice for all options) + +Control+c terminates the current command + +You can recall recently used commands by using the arrow keys. + +Sometimes you don't know how to use commands. `man command` can help with that. +You can search it with /, and quit it with Q. + +--- + class: center, middle # Tool Galore -We will now talk about multiple tools and how to combine them using bash (mostly). +We will now talk about multiple tools and how to combine them. --- # Scripting Languages -bash is not ideal as a programming language.for more complex jobs,a real programming language is needed. +Bash is not suited for programming. +For more complex jobs, a real programming language is needed. -two main choices: -- Python (recommended for beginners) -- Ruby (only if you already know programming; compact syntax) +Two common choices: +- **Python**: Recommended for beginners. Lots of built-in functionality. +- **Ruby**: If you already program. Compact syntax & metaprogramming. + +These fill niches that bash doesn't, and are still close to the system. -these fill niches that bash doesn't, but are still close to the system. <img src="images/polygon.png" width="50%"> @@ -224,13 +272,14 @@ these fill niches that bash doesn't, but are still close to the system. # Notifications -typically linux has a notification demon integrated. +Typically, linux has a notification deamon integrated. ```bash notify-send "Hello" ``` -this is often useful for things that you want to run in the background. +This is often useful for things that you want to run in the background. + Example: Timer ```bash @@ -241,20 +290,21 @@ sleep 300; notify-send "Time is up!" # Rofi -can be used to display a list of things +Can be used to display a list of things and then let's us choose one of them. ```bash -echo "one\ntwo\nthree" | rofi -dmenu +echo "one,two,three" | rofi -sep ',' -dmenu ``` -on its own is not useful, but can serve as a powerful launcher when combined with other things: +Rofi's dmenu mode on its own is not useful, but can serve as a powerful launcher +when combined with other commands: ```bash -ls -1 ~/Documents/books/*.pdf | rofi -dmenu | xargs -I {} evince {} +ls -1 ~/books/*.pdf | rofi -dmenu | xargs -I {} xdg-open {} ``` -https://github.com/davatorium/rofi +See: https://github.com/davatorium/rofi --- @@ -262,32 +312,39 @@ https://github.com/davatorium/rofi A **keydeamon**! Let's us configure keyboard shortcuts. -in file `~/.xbindkeysrc`: +In file `~/.xbindkeysrc`: ```bash "notify-send test" control+t -"ls -1 ~/Documents/books/*.pdf | rofi -dmenu | xargs -I {} evince {}" +"ls -1 ~/books/*.pdf | rofi -dmenu | xargs -I {} xdg-open {}" control+Mod1+b ``` -<img src="images/booklauncher.png"> - --- ```bash +# pick most recently downloaded file. Store in a variable. file=$(ls -tp1 ~/Downloads | grep -v "/$" | head -1) dirs='Images,Videos,Documents,Documents/books,DELETE,OPEN' -choice=$(echo $dirs | rofi -dmenu -sep ',' -p "move '$file' where? ") + +# Display rofi to choose from actions. Store in variable. +choice=$(echo $dirs | rofi -dmenu -sep ',' -p "move '$file' to: ") [[ "$choice" = "" ]] && exit + +# if the choice was DELETE, then delete and exit [[ "$choice" = DELETE ]] && rm ~/Downloads/"$file" && exit + +# if the choice was OPEN, then open and exit [[ "$choice" = OPEN ]] && xdg-open ~/Downloads/"$file" && exit + +# otherwise move the file mv ~/Downloads/"$file" "$choice" ``` -safe the above in `/home/nils/filer.sh` and add a few lines of text in `~/.xbindkeysrc`: +Save the above in `/home/nils/filer.sh` and add a few lines of text in `~/.xbindkeysrc`: ``` "bash /home/nils/filer.sh" @@ -299,34 +356,64 @@ safe the above in `/home/nils/filer.sh` and add a few lines of text in `~/.xbind # grep -a command to look for **regular expressions**. +A command to look for **regular expressions** (commonly just "regex"). + +Think of regexes as being text patterns on steroids: -think of regular expressions as being text patterns on steroids: +`grep -E "^[0-9]" data.csv` prints all the lines starting with a number -`grep "^[0-9]" data.csv` prints all the lines starting with a number +We can put multiple greps together to chain filters: -we can try a multiple greps together to chain filters: +TODO -<img src="images/"> -<img src="images/"> -<img src="images/"> +https://www.rexegg.com/ + +--- + +#grep + +Hot tip: Put `alias grep='grep -E'` into `~/.bashrc`. + +`-E` switches to "extended regular expressions". + +Unfortunately, there are lots of regex standards with minor differences. +Stick with one and don't worry about it (until you do). + +--- + +#grep + +Hot tip: Put `alias grep='grep -E'` into `~/.bashrc`. + +`-E` switches to "extended regular expressions". + +Unfortunately, there are lots of regex standards with minor differences. +Stick with one and don't worry about it (until you do). + +<img src="images/xkcd.png" width="50%"> +<p style="color: grey;">https://xkcd.com/1171/</p> --- # sed -**s**tream **ed**itor +The **s**tream **ed**itor. -this tool can search for regular expressions like we did with `grep`, -and perform various operations on each output line separately +This tool can search for regular expressions [and more] like we did with `grep`, +and perform various operations on the matched lines. + +Example: Delete all comments from a file +```bash +sed -E 's/^#.*//g' file.py +``` --- # find -a general purpose tool for finding various files according to search filters. +A general purpose tool for finding various files according to search filters. -especially nice if used with xargs-style option or regular expressions: +Especially nice if used with the `-exec` option or regular expressions: Remove all files (`-type f`) that end in .php (`*.php`): ```bash @@ -342,11 +429,11 @@ find ~/Documents -type f -exec grep -l "//.*TODO" {} \; # file conversion -sometimes you want to convert between similar file formats. there are a lot of tools for this. most are specific for some kind of file content.here are some common ones: +Sometimes you want to convert between similar file formats. there are a lot of tools for this. most are specific for some kind of file content.here are some common ones: - `ffmpeg` for audio and video - `pandoc` for many hierarchical text formats (mark down, html, latex) -- `pandoc` for image files +- `convert` for image files. Also known as ImageMagick. <hr> @@ -363,61 +450,92 @@ done # wget -mirror webpage +A general purpose download tool. +Mirror a webpage: ```bash wget --mirror --convert-links --page-requisites \ --no-parent -e robots=off https://thealternative.ch ``` +Download all pdf files: +```bash +site=https://people.inf.ethz.ch/suz/teaching/252-0210.html +wget --no-parent -r -l 1 -A .pdf $site +``` --- # git -Git is a version control system: Keep track of who changed what, when and why, and revert changes easily. +Git is a version control system: +Keep track of who changed what, when and why, and revert changes easily. +Clone our courses repository: ```bash -git init -git add * -git commit -m "Initial commit" -git remote add git@gitlab.ethz.com/thealternative/courses.git -git push -u origin --all +git clone https://gitlab.ethz.ch/thealternative/courses.git +``` + +When you just want a pull changes: +```bash +git stash +git pull ``` *Visit the git course next week!* +Also recommended: Git for ages 4 and up (video) + --- # ssh -`Euler` is an ETH supercomputer: +SSH lets us connect to servers and execute commands there. + +Connect to `Euler`, an ETH supercomputer: ```bash ssh nilsl@euler.ethz.ch ``` -Connect with SSH and execute commands in the console of the server. +We land in a bash shell and can execute commands. +The command **scp** let's you copy files between the server and your PC. +There's also **rsync** which is a more advanced way of copying files. --- # tmux -Execute long-running processes in the background while your terminal is closed. Especially useful in combination with SSH. +Execute long-running processes in the background while your terminal is closed. +Especially useful in combination with SSH. ```bash -tmux new -s new_session -tmux attach new_session +ssh nilsl@euler.ethz.ch +tmux new -s my_session +tmux attach my_session ``` +Do some stuff, close the window, then come back later. + +```bash +ssh nilsl@euler.ethz.ch +tmux attach my_session +``` + +An older but more widely available alternative to tmux is **screen** + --- -# xdotools +# xdotool + + +Automates key presses and mouse movements. +Useful for repetitive user interface stuff. -Automate key presses & mouse movements. +In a browser, copy the link of the current tab and then close it. ``` -xdotool key ctrl+c key alt+Tab key ctrl+v -xdotool mousemove 20 100 click 1 sleep 2 mosemove 100 20 +"xdotool key --delay 300 F6 ctrl+c ctrl+w" + F10 ``` --- @@ -431,7 +549,8 @@ Categories: - text processing (like vim, sed, bash) - file conversion (like ffmpeg, pandoc, convert) - searching (like sed, grep, find) -- glue (like xdotools, rofi, xbindkeys, notify-send, python, bash) +- scripting (like xdotools, rofi, xbindkeys, notify-send, python, bash) +- interfaces --- @@ -439,7 +558,7 @@ Categories: **stow** lets you manage config files easily. **ranger** is a console-based file manager. -**i3** is an alternative window-manager. +**i3** is an alternative window manager. **borg** is a backup tool. **nano** is an easy to use console text-editor. **rsync** to copy files between machines.