Commit 49da9d0d authored by Florian Landis's avatar Florian Landis 👾
Browse files

Remark regarding bench_Nexuse.m

parent ffe20f11
......@@ -7,13 +7,13 @@ Before setting up, you need permissions to access the codes and the input databa
```Note:: |:warning:| Currently, the codes are only accessible with an ETH account.
```
- Xuejiao Han (xuhan@eeh.ee.ethz.ch):
- permission for the **DistIv** repository
- Xuejiao Han (xuhan@eeh.ee.ethz.ch):
- permission for the **DistIv** repository
- access to the **PSL server** for the database (server name: itet-psl-s02)
- Jared Garrison (garrison@fen.ethz.ch):
- Jared Garrison (garrison@fen.ethz.ch):
- permissions for the **eMark** repository
- permissions for the **Shared** repository,
- Blazhe Gjorgiev (gblazhe@ethz.ch):
- Blazhe Gjorgiev (gblazhe@ethz.ch):
- permission for the **Cascades** repository
- permission for the **Run_Nexuse** repository
- Elena Raycheva (elena.raycheva@esc.ethz.ch):
......@@ -29,7 +29,7 @@ Even though the current Nexus-e platform is **not** suitable to run locally beca
Most the instructions below have been tested both on **Windows** and on **Mac**.
##### 1. Get the codes
Make sure you have installed [git](https://git-scm.com/).
Make sure you have installed [git](https://git-scm.com/).
The instructions here use **command lines**. If you have a GUI tool for git, you could use the corresponding operations in the GUI tool instead. An example using a GUI tool "GitKraken" can be found in section [Nexus-e repository instructions](nexuse_repo_instruction.html).
......@@ -62,20 +62,20 @@ DATABASE_PASSWORD
- Install MySQL and MySQL Workbench
- Connect to ETH VPN
- Connect to the database with MySQL Workbench
- Add a new MySQL connection
- Add a new MySQL connection
![](images/mysql-workbench-1.png)
- Input connection information
- Input connection information
![](images/mysql-workbench-2.png)
- Click the button `Test Connection`. You will be asked to input your database password. Then you will see a seccess message:
![](images/mysql-workbench-3.png)
- Click the button `Test Connection`. You will be asked to input your database password. Then you will see a seccess message:
![](images/mysql-workbench-3.png)
```eval_rst
|:warning:| By default, everyone has write permission to the database. Therefore, to be safe, always make a local copy of the database before playing with it - i.e., first “Dump” it to a local folder, then import it to your personal MySQL account. These can be done in this window:
```
|:warning:| By default, everyone has write permission to the database. Therefore, to be safe, always make a local copy of the database before playing with it - i.e., first “Dump” it to a local folder, then import it to your personal MySQL account. These can be done in this window:
```
![](images/mysql-workbench-dump.png)
- (Optional) Work with a local copy of the database
- Create a new text file to substitute `UserDBInfo.txt`.
- (Optional) Work with a local copy of the database
- Create a new text file to substitute `UserDBInfo.txt`.
For example, if you have dumped the common database to your local hosted database with IP address `127.0.0.1`, you could create a new text file (e.g., named as `UserDBInfo_local.txt`) as the following:
```
127.0.0.1
......@@ -84,13 +84,13 @@ DATABASE_PASSWORD
YOUR_LOCAL_PASSWORD
```
- Place the new file in the same folder as `UserDBInfo.txt`.
- Point to the new file in the script.
- Make a copy of `user_example_conf.m` in the folder `Run_Nexuse`. Name the copy as `user_conf.m`. (Note: You should only work with the local copy, because the original `xxx_example_conf.m` will be synchronized to git and it shouldn't be changed.)
- Point to the new file in the script.
- Make a copy of `user_example_conf.m` in the folder `Run_Nexuse`. Name the copy as `user_conf.m`. (Note: You should only work with the local copy, because the original `xxx_example_conf.m` will be synchronized to git and it shouldn't be changed.)
- Uncomment and adapt the variable `DB_INFO_FILE` in `user_conf.m` to be the new file name (e.g., `DB_INFO_FILE = 'UserDBInfo_local.txt';`).
You might encounter the following problems when you run Nexus-e with a local database, follow the link for possible solutions:
(1) [Error: "Access denied for user 'your_username'@'your_host_address' ..."](faq.html#access-denied-to-access-database)
(2) [Error: "The server time zone value 'xxx' is unrecognized ..."](faq.html#database-server-timezone-issue)
You might encounter the following problems when you run Nexus-e with a local database, follow the link for possible solutions:
(1) [Error: "Access denied for user 'your_username'@'your_host_address' ..."](faq.html#access-denied-to-access-database)
(2) [Error: "The server time zone value 'xxx' is unrecognized ..."](faq.html#database-server-timezone-issue)
(3) [Error: "The user specified as a definer ('xxx'@'%') does not exist ..."](faq.html#database-definer-missing)
##### 3. Prepare software & license
......@@ -100,33 +100,33 @@ You could skip this section if you don't plan to run the whole Nexus-e platform
- Set up database connector
- Download the **MySQL connector** from [here](https://dev.mysql.com/downloads/connector/j/).
- Download the **MySQL connector** from [here](https://dev.mysql.com/downloads/connector/j/).
Or, if you have access to the Nexus-e polybox, use the `mysql-connector-java-8.0.18.zip` provided in the folder `02_Model/02_Model_Setup_Instruction` (**Unzip** the file before the next step).
- Copy the MySQL connector folder (e.g. mysql-connector-java-8.0.18) into a folder at your preference.
- Copy the MySQL connector folder (e.g. mysql-connector-java-8.0.18) into a folder at your preference.
We recommend to put it into the **Matlab preferences folder**, which you can find by typing in Matlab Command Window ```prefdir```.
- Create a **javaclasspath.txt** file in the **Matlab preferences folder**.
- In the **javaclasspath.txt** file, write the path to the connector .jar file that you just copied.
- In the **javaclasspath.txt** file, write the path to the connector .jar file that you just copied.
E.g., On Blazhe's Windows computer the path is `C:\Users\gblazhe\AppData\Roaming\MathWorks\MATLAB\R2018a\mysql-connector-java-8.0.18\mysql-connector-java-8.0.18.jar`.
- Reload Matlab
- Test whether a database connector is set up successfully. Write the following commands in Matlab (substitue `YOUR_USERNAME` and `YOUR_PASSWORD` with your credentials for the database). If the second command returns `1`, it means success.
- Test whether a database connector is set up successfully. Write the following commands in Matlab (substitue `YOUR_USERNAME` and `YOUR_PASSWORD` with your credentials for the database). If the second command returns `1`, it means success.
```
conn = database("sys", YOUR_USERNAME, YOUR_PASSWORD, 'Vendor', 'MySQL', 'Server', "YOUR_HOST");
isopen(conn)
```
- **Python** (3.x)
- **Python** (3.x)
```eval_rst
|:warning:| Note: we are now in the phase of transferring from python 2.x to python 3.x. Therefore, it is recommended to use python 3.x locally and identify potential issues for the transfer.
```
```
- Python packages: install the required packages listed in `requirements.txt`.
- Gurobi (9.x): Download gurobi and set up a free Gurobi academic license <https://www.gurobi.com/academia/academic-program-and-licenses>
- Python path: If you use Mac or if you want to run Nexus-e in a python virtual environment, you need to configure Python path in Matlab. Otherwise, you could skip this step.
- Python path: If you use Mac or if you want to run Nexus-e in a python virtual environment, you need to configure Python path in Matlab. Otherwise, you could skip this step.
- Make a copy of `user_example_conf.m` in the folder `Run_Nexuse`. Name the copy as `user_conf.m`. (Note: You should only work with the local copy, because the original `xxx_example_conf.m` will be synchronized to git and it shouldn't be changed.)
- Find out your python path (it varies largely depending on your PC's operating system and how you installed python). Uncomment and adapt the variable `PATH_PYTHON` in `user_conf.m` accordingly.
- If you have a python virtual environment, you could configure it following the instructions in `user_conf.m`.
......@@ -145,6 +145,8 @@ You could skip this section if you don't plan to run the whole Nexus-e platform
- Connect to ETH VPN (in order to connect with the database)
- Open Matlab
- Before running `run_Nexuse.m` for the first time, you need to run `bench_Nexuse.m`.
It is recommended to repeat this step after major changes to the modules eMark and CentIv.
- Run the script `run_Nexuse.m` in the `Run_Nexuse` folder
- You could change the variables `scen`, `tpRes`, and `limDifference` as instructed in the script
- Consult Blazhe Gjorgiev (gblazhe@ethz.ch) if you want to run other scripts in the `Run_Nexuse` folder.
......@@ -159,24 +161,24 @@ To check whether you are added to the user group, use command line ssh to Euler
##### 2. Get the codes
- Create a folder where you want to put the codes
- cd to that folder
- Clone the overarching repository with this command:
```git clone --recursive https://gitlab.ethz.ch/xyan/nexus-e.git```
- Create a folder where you want to put the codes
- cd to that folder
- Clone the overarching repository with this command:
```git clone --recursive https://gitlab.ethz.ch/xyan/nexus-e.git```
You will be asked for your username and password for the repository and its submodules.
In the end, the folder will look like the screenshot below. Note: tThe `Results` folder will be automatically generated when you run Nexus-e.
![](images/euler-nexus-e-folder.png)
##### 3. Connect with the input database
- Download the **MySQL connector** from [here](https://dev.mysql.com/downloads/connector/j/).
- Download the **MySQL connector** from [here](https://dev.mysql.com/downloads/connector/j/).
Or, if you have access to the Nexus-e polybox, use the `mysql-connector-java-8.0.18.zip` provided in the folder `02_Model/02_Model_Setup_Instruction` (**Unzip** the file before the next step).
- Copy the MySQL folder in the hidden .matlab folder (e.g., .matlab/2019b)
- Copy the MySQL folder in the hidden .matlab folder (e.g., .matlab/2019b)
- Create a **javaclasspath.txt** file in the same folder as above.
- In the **javaclasspath.txt** file, write the path to the connector .jar file that you just copied.
- In the **javaclasspath.txt** file, write the path to the connector .jar file that you just copied.
E.g., On Blazhe's Euler account the path is `/cluster/home/gblazhe/.matlab/R2018a/mysql-connector-java-8.0.18/mysql-connector-java-8.0.18.jar`.
- Logoff and re-login to Euler
......@@ -184,13 +186,13 @@ E.g., On Blazhe's Euler account the path is `/cluster/home/gblazhe/.matlab/R2018
##### 4. Prepare dependencies
Unlike setting up Nexue-s locally, we don't need to download any software or license on Euler. Instead, they are all prepared for all users in the user group **MAVT-esc-nexus-e**. You should already be a member of the group if you have followed the [first step](#join-the-nexus-e-user-group).
But still, we need to explicitely load all the dependencies, including modules (an Euler term for "software") and python packages.
But still, we need to explicitely load all the dependencies, including modules (an Euler term for "software") and python packages.
- Load modules
**Option 1**: Manually load modules every time before running Nexus-e
**Option 1**: Manually load modules every time before running Nexus-e
Copy the following commands into Euler:
Copy the following commands into Euler:
```
module load new
module load gams/28.2
......@@ -202,22 +204,22 @@ But still, we need to explicitely load all the dependencies, including modules (
```eval_rst
|:warning:| Note: We used to load python/2.7.14. But we are now in the phase of transferring from python 2.x to python 3.x. Therefore, it is recommended to use python 3.x and identify potential issues for the transfer.
```
Your Euler window should look similar to this:
Your Euler window should look similar to this:
```eval_rst
.. image:: images/euler-module-load.png
:width: 300px
```
```
**Option 2**: Edit `.bash_profile` so that the modules are automatically loaded upon log in to Euler
**Option 2**: Edit `.bash_profile` so that the modules are automatically loaded upon log in to Euler
Write the same commands at the end of `.bash_profile`. This is a hidden file in your Euler's home directory. After adding these commands, your `.bash_profile` should look similar to this:
Write the same commands at the end of `.bash_profile`. This is a hidden file in your Euler's home directory. After adding these commands, your `.bash_profile` should look similar to this:
![](images/euler-bash-profile.png)
**Test**: To test whether the modules have been successfully loaded, type `module list` in the commaand line. It should then list all loaded modules.
- Install python packages
This you only need to do it **once**, i.e. you don't need to do it everytime when you run Nexus-e.
This you only need to do it **once**, i.e. you don't need to do it everytime when you run Nexus-e.
In command line under your home directory, type
```
......@@ -230,11 +232,9 @@ But still, we need to explicitely load all the dependencies, including modules (
```
bsub -n 36 -R "model==XeonGold_6150" -R "rusage[mem=5180]" -W "10:00" matlab -r run_Nexuse
```
The parameters in this command can be costomized. Details can be found [here](https://scicomp.ethz.ch/wiki/Getting_started_with_clusters).
The parameters in this command can be costomized. Details can be found [here](https://scicomp.ethz.ch/wiki/Getting_started_with_clusters).
Here we give a short explanation to the example command above:
![](images/euler-job-submission.png)
After submission, Euler will respond by telling you what the jobID is.
After submission, Euler will respond by telling you what the jobID is.
Supports Markdown
0% or .
You are about to add 0 people to the discussion. Proceed with caution.
Finish editing this message first!
Please register or to comment