Extended README

README.md

 # mossutils
 
-## Python Package Tutorials
-
-* https://python-packaging.readthedocs.io/en/latest/minimal.html
-* https://dev.to/rf_schubert/how-to-create-a-pip-package-and-host-on-private-github-repo-58pa
+A collection for useful scripts for working with Moss in the context of ETH (eDoz) and Code Expert.
 
 ## Installation
 
 ```
-$ pip install git+https://gitlab.ethz.ch/scmalte/mossutils.git`
+$ pip install --upgrade git+https://gitlab.ethz.ch/scmalte/mossutils.git`
 ```
+
+Argument `--upgrade` (re)installs the package, even if the version number hasn't changed.
+
+## Usage
+
+**TODO:** Add instructions for how to obtain and prepare the involved data. See also `preprocessing/README.md`.
+
+1. Obtain and prepare data
+
+1. Run `mu-moss --help` for arguments that can/must be configured. Afterwards, run `mu-moss` as desired.
+
+   `mu-moss` connects to the Moss service, uploads submissions and downloads the generated report.
+
+1. Run `mu-revise`.
+
+   **NOTE:** `mu-revise` is not yet configurable and expects `mu-moss` to have used its default file and directory names!
+
+   `mu-revise` improves the previously downloaded HTML report, e.g. by making the table sortable, and generates a CSV file with the report data.
+
+1. Run `mu-cluster`.
+
+   **NOTE:** `mu-cluster` is not yet configurable and expects `mu-revise` to have used its default file and directory names!
+
+   `mu-cluster` computes clusters of plagiarism, creates as DOT and SVG files (by running GraphViz, which *must be in the path*), and creates a summarising CSV file.
+
+1. Run `mu-aggr`.
+
+   **NOTE:** `mu-aggr` is not yet configurable and expects `mu-cluster` to have used its default file and directory names!
+
+   `mu-aggr` combines the cluster data with exports from eDoz and Code Expert, and generates a cluster report as an HTML file.
+
+## Python Package Tutorials
+
+I've used the following tutorials to get started with developing a Python package:
+
+* https://python-packaging.readthedocs.io/en/latest/minimal.html
+
+* https://dev.to/rf_schubert/how-to-create-a-pip-package-and-host-on-private-github-repo-58pa

preprocessing/README.md

+# README
+
+## Prerequisites
+
+* Tested with Python 3.7
+* Script `run_-_moss.py`
+  * https://pypi.org/project/mosspy/
+  * `pip install mosspy`
+* Script `rename_to_legi.sh`:
+  * https://stedolan.github.io/jq/
+  * Download `jq` and add to path
+* Script `visualize.py`
+  * https://github.com/hjalti/mossum
+  * `pip3 install git+https://github.com/hjalti/mossum@master`
+  * Replace `<python3>/Lib/site-packages/mossum/mossum.py` with `MODIFIED-mossum.py`
+  * http://networkx.github.io/
+  * `pip install networkx`
+
+## Tidying up files and directories
+
+* `cx-dump_bonus-exercise-1_2020-04-17.zip` contains the latest submission from each user
+
+* Execute next commands on the level of the user directories, e.g. in `./bonus_exercise_1`, so that, e.g. `./bonus_exercise_1/<user>` are the individual user directories
+
+* Assumption: relevant files per submission are `<user>/details.json` and `<user>/files/main.cpp`, whereas all other files and directories can be deleted:
+
+* Delete irrelevant files, and directory `cx_data`:
+
+  ```plain
+  $ cd ./bonus_exercise_1
+
+  $ find -type f ! \( -iname details.json -or -iname main.cpp \) -delete
+  
+  $ find . -type d -iname cx_data -delete
+  ```
+
+* Move `<user>/files/main.cpp` to `<user>/main.cpp` and delete the (now empty) directory `<user>/files`:
+
+  ```plain
+  $ cd ./bonus_exercise_1
+
+  $ find . -type d -iname files -execdir mv ./files/main.cpp . \;
+
+  $ find . -type d -iname files -delete
+  ```
+
+## Renaming user directories
+
+* Rename directories from user names to Legi numbers before submitting data to Moss, e.g. rename `scmalte` to `01-234-567`. The file `<user>/details.json` provides the Legi number.
+
+* Execute `rename_to_legi.sh` from e.g. `./bonus_exercise_1/`:
+
+  ```plain
+  $ cd ./bonus_exercise_1
+
+  $ ../rename_to_legi.sh
+  ```
+  
+  The script prompts for confirmation before the first renaming is executed.
+
+## Moss
+
+### moss.py
+
+* Edit `moss.py` and check configuration
+
+* Execute `moss.py` from e.g. `./bonus_exercise_1/`:
+
+  ```plain
+  $ cd ./bonus_exercise_1/
+
+  $ python ../moss.py
+  ```
+
+* If not configured otherwise, open `./bonus_exercise_1/moss-report.html` in your browser
+
+### clusters.py
+
+* Edit `clusters.py` and check configuration
+
+* Execute `clusters.py` from e.g. `./bonus_exercise_1/`:
+
+  ```plain
+  $ cd ./bonus_exercise_1/
+
+  $ python ../clusters.py
+  ```
+
+* If not configured otherwise, open `./bonus_exercise_1/clusters.html` in your browser

preprocessing/rename_to_legi.sh

+#!/bin/bash
+
+function quit_or_continue() {
+  read -p "Press Q to abort, any other key to continue ..." -n 1 -s KEY
+  echo
+  if [[ ${KEY^^} == "Q" ]]; then exit; fi # https://stackoverflow.com/a/32210715
+}
+
+
+shopt -s globstar # Enable glob patterns such as **/details.json
+first="true"
+
+for i in **/details.json; do # Whitespace-safe and recursive
+  echo "$i"
+  legiNumber="$(jq -r .legi $i)" # https://stedolan.github.io/jq/
+  
+  command="mv ./$(dirname $i)/ ./$legiNumber/"
+
+  if [[ $first == "true" ]]; then
+    echo "Current directory: $(pwd)"
+    echo "First directory will be renamed as follows (all others analogously):"
+    echo $command
+    quit_or_continue
+    first="false"
+  fi
+
+  $command
+done