Commit 7042ada3 authored by sfux's avatar sfux
Browse files

Adding support for environment proposed and implemented by Gül Sena Altıntaş

parent 90ecc9eb
......@@ -100,6 +100,7 @@ Optional arguments:
-s | --softwarestack SOFTWARE_STACK Software stack to be used (old, new)
-v | --version Display version of the script and exit
-w | --workdir WORKING_DIR Working directory for the jupyter notebook
-e | --environment ENV Use Python virtual environment
Examlples:
......@@ -120,7 +121,7 @@ JNB_WAITING_INTERVAL=60 # Time interval to check if the job on the cluster a
JNB_SSH_KEY_PATH="" # Path to SSH key with non-standard name
JNB_SOFTWARE_STACK="new" # Software stack to be used (old, new)
JNB_WORKING_DIR="$HOME" # Working directory for the jupyter notebook
JNB_ENV="" # Path to virtual environment
```
### Reconnect to a Jupyter notebook
......@@ -182,6 +183,15 @@ Then follow the instructions provided on our wiki:
https://scicomp.ethz.ch/wiki/R#Extensions
```
### Running with a Python Virtual Environment
You can create your own [virtual environment](https://scicomp.ethz.ch/wiki/Python_virtual_environment) in the cluster and run your jupyter notebook with that environment. Please make sure that the Python version used to create your virtual environment is compatible with the one used in the jupyter script.
```
./start_jupyter_nb.sh -u sfux -n 4 -W 04:00 -m 2048 -w /cluster/scratch/sfux -e sample_env
```
## Main author
* Samuel Fux
......@@ -191,3 +201,4 @@ https://scicomp.ethz.ch/wiki/R#Extensions
* Steven Armstrong
* Swen Vermeul
* Jarunan Panyasantisuk
* Gül Sena Altıntaş
......@@ -7,3 +7,4 @@ JNB_WAITING_INTERVAL=60 # Time interval to check if the job on the cluster a
JNB_SSH_KEY_PATH="" # Path to SSH key with non-standard name
JNB_SOFTWARE_STACK="new" # Software stack to be used (old, new)
JNB_WORKING_DIR="" # Working directory for the jupyter notebook
JNB_ENV="" # Path to virtual environment
......@@ -6,12 +6,15 @@
# #
# Main author : Samuel Fux #
# Contributions : Jarunan Panyasantisuk, Andrei Plamada, Swen Vermeul, #
# Urban Borsnik, Steven Armstrong, Henry Lütcke #
# Urban Borsnik, Steven Armstrong, Henry Lütcke, #
# Gül Sena Altıntaş #
# Date : December 2018 #
# Location : ETH Zurich #
# Version : 1.0 #
# Change history : #
# #
# 02.11.2021 Added virtual environment support #
# #
# 26.10.2021 The script was rewritten: #
# * clean up of the code (naming scheme for variables) #
# * replaced multiline echo's with heredoc's #
......@@ -84,6 +87,9 @@ JNB_SOFTWARE_STACK="new"
# Workdir default : no default
JNB_WORKING_DIR=""
# Virtual env default : no default
JNB_ENV=""
###############################################################################
# Usage instructions #
###############################################################################
......@@ -104,6 +110,7 @@ Options:
Optional arguments:
-c | --config CONFIG_FILE Configuration file for specifying options
-e | --environment ENV Python virtual environment
-g | --numgpu NUM_GPU Number of GPUs to be used on the cluster
-h | --help Display help for this script and quit
-i | --interval INTERVAL Time interval for checking if the job on the cluster already started
......@@ -130,7 +137,8 @@ JNB_MEM_PER_CPU_CORE=1024 # Memory limit in MB per core
JNB_WAITING_INTERVAL=60 # Time interval to check if the job on the cluster already started
JNB_SSH_KEY_PATH="" # Path to SSH key with non-standard name
JNB_SOFTWARE_STACK="new" # Software stack to be used (old, new)
JNB_WORKING_DIR="$HOME" # Working directory for the jupyter notebook
JNB_WORKING_DIR="" # Working directory for the jupyter notebook
JNB_ENV="" # Path to virtual environment
EOF
exit 1
......@@ -175,6 +183,11 @@ do
shift
shift
;;
-e |--environment)
JNB_ENV=$2
shift
shift
;;
-g|--numgpu)
JNB_NUM_GPU=$2
shift
......@@ -333,6 +346,11 @@ else
echo -e "Using $JNB_WORKING_DIR as working directory"
fi
# check if JNB_ENV is empty
if [ "$JNB_ENV" != "" ]; then
echo "Using $JNB_ENV as python environment"
fi
# put together string for SSH options
JNB_SSH_OPT="$JNB_SKPATH $JNB_USERNAME@$JNB_HOSTNAME"
......@@ -370,6 +388,7 @@ echo -e "Connecting to $JNB_HOSTNAME to start jupyter notebook in a batch job"
# FIXME: save jobid in a variable, that the script can kill the batch job at the end
ssh $JNB_SSH_OPT bsub -n $JNB_NUM_CPU -W $JNB_RUN_TIME -R "rusage[mem=$JNB_MEM_PER_CPU_CORE]" $JNB_SNUM_GPU <<ENDBSUB
module load $JNB_MODULE_COMMAND
if [ "$JNB_ENV" != "" ]; then echo -e "Activating the $JNB_ENV"; source $JNB_ENV/bin/activate; fi
export XDG_RUNTIME_DIR=
JNB_IP_REMOTE="\$(hostname -i)"
echo "Remote IP:\$JNB_IP_REMOTE" >> /cluster/home/$JNB_USERNAME/jnbip
......
Supports Markdown
0% or .
You are about to add 0 people to the discussion. Proceed with caution.
Finish editing this message first!
Please register or to comment