To receive notifications about scheduled maintenance, please subscribe to the mailing-list gitlab-operations@sympa.ethz.ch. You can subscribe to the mailing-list at https://sympa.ethz.ch

Commit c40cefbb authored by Xuqian Yan's avatar Xuqian Yan
Browse files

Updated setup based on Pranjal's feedback; updated instruction on conf_local

parent c97e23f1
......@@ -75,7 +75,9 @@ Apart from MySQL introduced above, we also need the following software to run Ne
- ImageMagick
###### Matlab
- Download Matlab (R2018a or higher; R2020b is recomended). Available at [ETH IT shop](https://itshop.ethz.ch/EndUser/Items/Home).
- Download Matlab (R2020a or higher). Available at [ETH IT shop](https://itshop.ethz.ch/EndUser/Items/Home).
- During installation, tick the box to install "Database Toolbox" as well (This option is available if you downloaded Matlab through the ETH IT shop). If this is not an option during installation, you can download it seperately [here](https://ch.mathworks.com/products/database.html).
- Set up database connector
......@@ -107,18 +109,18 @@ Apart from MySQL introduced above, we also need the following software to run Ne
- Connect Gurobi with Python following [this instruction](https://support.gurobi.com/hc/en-us/articles/360044290292-How-do-I-install-Gurobi-for-Python-).
- Save Gurobi path for Matlab
- Find out the path of `gurobi_setup.m`. (E.g., for mac, it is similar to `/Library/gurobi903/mac64/matlab`.)
- Open Matlab; under the tab "Home", click **Set Path**; select **Add Folder...**; browse to the folder containing `gurobi_setup.m` (from the step above); click **Open**.
- Open Matlab; under the tab "Home", click **Set Path**; select **Add Folder...**; browse to the folder containing `gurobi_setup.m`; click **Open**.
- A new entry of the selected path will appear on the right side of the **Set Path** window; click **Save**.
![](images/matlab-setpath.png)
###### GAMS
- Download [GAMS](https://www.gams.com/download/). Version 32.2 is recomended. Our current license doesn't support versions newer than 32.
- Download [GAMS](https://www.gams.com/32/). Version 32.2 is recomended. Our current license doesn't support versions newer than 32.
- Set GAMS License
- Use the `gamslice.txt` provided in polybox `02_Model/02_Model_Setup_Instruction`.
- Install the license: [here](https://support.gams.com/installation:how_do_i_install_a_license_file) or [here](https://www.youtube.com/watch?v=vSe3YGkUVoc) or (for MacOS) [here](https://www.gams.com/latest/docs/UG_MAC_INSTALL.html).
- Save GAMS path for Matlab
- Find out your GAMS path. It varies largely depending on your PC's operating system and the version of GAMS. For example, GAMS 32 on Mac has the path: `/Library/Frameworks/GAMS.framework/Versions/32/Resources`.
- Similar to how you configured Gurobi for Matlab: Open Matlab; under the tab "Home", click **Set Path**; select **Add Folder...**; browse to the folder containing `gurobi_setup.m` (from the step above); click **Open**.
- Similar to how you configured Gurobi for Matlab: Open Matlab; under the tab "Home", click **Set Path**; select **Add Folder...**; browse to the GAMS path; click **Open**.
- A new entry of the selected path will appear on the right side of the **Set Path** window; click **Save**.
- Connect GAMS with Python
- Follow the instruction [here](https://github.com/NREL/gdx-pandas).
......@@ -128,45 +130,53 @@ Apart from MySQL introduced above, we also need the following software to run Ne
###### ImageMagick
- Instructions for Windows:
- Download ImageMagick from [here](https://imagemagick.org/script/download.php#windows). Install ImageMagick from the .exe file. Tick all boxes when asked which packages to be installed (by default only the first three are selected).
- Additionally, download dependable software "Ghostscript" from [here](https://www.ghostscript.com/download/gsdnld.html). Select Public License Ghostscript. Install the Ghostscript.
- Download ImageMagick from [here](https://imagemagick.org/script/download.php#windows). Install ImageMagick from the .exe file. Tick **all** boxes when asked which packages to be installed (by default only the first three are selected).
- Download dependable software "Ghostscript" from [here](https://www.ghostscript.com/download/gsdnld.html). Select Public License Ghostscript. Install the Ghostscript.
- Optional (if ImageMagick does not work), download and install Visual studio from [here](https://support.microsoft.com/en-us/help/2977003/the-latest-supported-visual-c-downloads).
- Instructions for Mac:
- Install ImageMagick from [here](https://imagemagick.org/script/download.php#macosx)
- Test if ImageMagick works:
- Open Windows command prompt and brows to folder that contains a pdf file, e.g., testch.pdf
- Use the following command to convert file: convert testch.pdf testch.jpg
- Open Windows command prompt and browse to folder that contains a pdf file, e.g., testch.pdf
- Use the following command to convert file: `convert testch.pdf testch.jpg`
##### 4. Edit conf_local.m
- Make a copy of `example_conf_local.m` in the folder `Run_Nexuse/conf_local`. Name the copy as `conf_local.m`. (Note: You should only work with the local copy, because the original `example_conf_local.m` will be synchronized to git and it shouldn't be changed.)
##### 4. Edit user_conf.m
- Make a copy of `user_example_conf.m` in the folder `Run_Nexuse`. Name the copy as `user_conf.m`. (Note: You should only work with the local copy, because the original `xxx_example_conf.m` will be synchronized to git and it shouldn't be changed.)
- In `conf_local.m` uncomment and modify the following variables based on your local computer's settings. More instructions can be found in the script `conf_local.m` itself.
- In `user_conf.m` uncomment and modify the following variables based on your local computer's settings. More instructions can be found in `user_conf.m`.
- PATH_MYSQL
- PATH_PYTHON
- PATH_GUROBI_FOR_PYTHON (i.e., the path of `gurobi.sh`.)
- PATH_GUROBI_FOR_PYTHON
- PATH_LATEX
- PATH_CONVERT
- PATH_GAMS
```eval_rst
.. note::
You don't necessarily need all software paths here. For example, if you only want to run the postprocess scripts, :code:`PATH_GUROBI_FOR_PYTHON` is not needed; if you only want to run DBcreation scripts, only :code:`DB_INFO_FILE` and :code:`PATH_MYSQL` are needed.
```
- (Optional) If you want to run Nexus-e with a [local copy of the database](#optional-make-a-local-copy-of-the-database)
- Create a new text file similar to `UserDBInfo.txt`. For example, if you have dumped the remote database to your local hosted database with IP address `127.0.0.1`, you could create a new text file as the following:
- (Optional) If you want to work with a [local copy of the database](#optional-make-a-local-copy-of-the-database)
- Make a copy of `example_UserDBInfo_local.txt` in the folder `Run_Nexuse/conf_local`. Name the copy as `UserDBInfo_local.txt`. (Note: You should only work with the local copy, because the original `example_UserDBInfo_local.txt` will be synchronized to git and it shouldn't be changed.)
- Edit `example_UserDBInfo_local.txt` based on your local settings:
```
127.0.0.1
127.0.0.1 (or "localhost")
3306
YOUR_LOCAL_USERNAME (e.g. "root")
YOUR_LOCAL_PASSWORD
```
- Name the new file as `UserDBInfo_xxx.txt` (e.g., `UserDBInfo_local.txt`). Note: An underscore "_" after "UserDBInfo" is needed to not synchronize it on git (see the file `.gitignore`).
- Place the new file in the same folder as `UserDBInfo.txt`.
- Uncomment and adapt the variable `DB_INFO_FILE` in `user_conf.m` to be the new file name (e.g., `DB_INFO_FILE = 'UserDBInfo_local.txt';`).
- Uncomment the variable `DB_INFO_FILE` in `conf_local.m`.
- Test `conf_local.m`: in Matlab, run the script `conf_local/conf_test.m`.
- If your `conf_local.m` is set up correctly, you should see outputs like "xxx works."
- If you see "xxx FAILED", there will be information in the output showing why xxx failed. However, this is not necessarily a problem. Because as mentioned above, you don't always need all software. For example, if you only want to run postprocess scripts, `PATH_GUROBI_FOR_PYTHON` is not needed; if you only want to run DBcreation scripts, only `DB_INFO_FILE` and `PATH_MYSQL` are needed.
##### 5. Run Nexus-e
- Connect to ETH VPN (in order to connect with the database)
- Open Matlab
- Double check the `user_conf.m` file, e.g., whether the paths of the software are correct.
- (Optional) Run `bench_Nexuse.m`. This is required before you run `run_Nexuse.m` for the first time, or after major changes have been made to eMark and/or CentIv. Running `bench_Nexuse.m` will create a file `/Gemel/results/benchmark_2015.mat` which is necessary to run `run_Nexuse.m`.
- Run the script `run_Nexuse.m` in the `Run_Nexuse` folder. There are a few parameters that you can customize in the beginning of this run script:
- You could change the variables `scen`, `scenShortName`, `tpRes`, `limDifference`, and `dbSuffix` as instructed in the script
......@@ -261,27 +271,14 @@ To check whether you are added to the user group, use command line ssh to Euler
In the end, the folder will look like the screenshot below. Note: tThe `Results` folder will be automatically generated when you run Nexus-e.
![](images/euler-nexus-e-folder.png)
##### 3. Connect with the input database
- Download the **MySQL connector** from [here](https://dev.mysql.com/downloads/connector/j/).
Or, if you have access to the Nexus-e polybox, use the `mysql-connector-java-8.0.18.zip` provided in the folder `02_Model/02_Model_Setup_Instruction` (**Unzip** the file before the next step).
- Copy the MySQL folder in the hidden .matlab folder (e.g., .matlab/2019b)
- Create a **javaclasspath.txt** file in the same folder as above.
- In the **javaclasspath.txt** file, write the path to the connector .jar file that you just copied.
E.g., On Blazhe's Euler account the path is `/cluster/home/gblazhe/.matlab/R2018a/mysql-connector-java-8.0.18/mysql-connector-java-8.0.18.jar`.
- Logoff and re-login to Euler
##### 4. Prepare dependencies
##### 3. Prepare dependencies
Unlike setting up Nexue-s locally, we don't need to download any software or license on Euler. Instead, they are all prepared for all users in the user group **MAVT-esc-nexus-e**. You should already be a member of the group if you have followed the [first step](#join-the-nexus-e-user-group).
But still, we need to explicitely load all the dependencies, including modules (an Euler term for "software") and python packages.
- Load modules
**Option 1**: Manually load modules every time before running Nexus-e
- Option 1: Manually load modules every time before running Nexus-e
Copy the following commands into Euler:
```
......@@ -293,11 +290,8 @@ But still, we need to explicitely load all the dependencies, including modules (
module load python/3.7.1
module load texlive
```
```eval_rst
|:warning:| Note: We used to load python/2.7.14 and gurobi/8.1.1. But we are now in the phase of testing the newer versions of the software. Therefore, it is recommended to use python 3 and gurobi 9 to identify potential issues.
```
**Option 2**: Edit `.bash_profile` so that the modules are automatically loaded upon log in to Euler
- Option 2: Edit `.bash_profile` so that the modules are automatically loaded upon log in to Euler
Write the same commands at the end of `.bash_profile`. This is a hidden file in your Euler's home directory. After adding these commands, your `.bash_profile` should look similar to this:
![](images/euler-bash-profile.png)
......@@ -308,29 +302,45 @@ But still, we need to explicitely load all the dependencies, including modules (
This you only need to do it **once**, i.e. you don't need to do it everytime when you run Nexus-e.
In command line under your home directory, type
```
pip install --user -r nexus-e/requirements.txt
```
In command line under your home directory,
- `cd` to the folder where you have stored the Nexus-e codes
- type `pip install --user -r requirements.txt`
You might encounter software compatibility problems when running Nexus-e on Euler, follow the link for possible solutions:
[Matlab & Python compatibility error: "ImportError: ...pyexpat...: undefined symbol: XML_SetHashSalt"](faq.html#matlab-python)
##### 4. Connect with the input database
- Download the **MySQL connector** from [here](https://dev.mysql.com/downloads/connector/j/). **Unzip** the file.
- Copy the MySQL connector folder into the hidden .matlab folder (e.g., .matlab/2020a).
- Tip: To see all folders including the hidden ones, use the command `ls -a`.
- If you don't see the .matlab folder, run Matlab once: type in `matlab` -> you will see Matlab is started in the terminal -> type in `exit` -> Matlab will stop -> you should then see the .matlab folder.
- Create a **javaclasspath.txt** file in the same folder as above.
- In the **javaclasspath.txt** file, write the path to the connector .jar file that you just copied.
E.g., On Blazhe's Euler account the path is `/cluster/home/gblazhe/.matlab/R2018a/mysql-connector-java-8.0.18/mysql-connector-java-8.0.18.jar`.
- Logoff and re-login to Euler
##### 5. Run Nexus-e
- cd to the `Run_Nexuse` folder
- `cd` to the `Run_Nexuse` folder
- (Optional) Run `bench_Nexuse.m`. This is required before you run `run_Nexuse.m` for the first time, or after major changes have been made to eMark and/or CentIv. Running `bench_Nexuse.m` will create a file `/Gemel/results/benchmark_2015.mat` which is necessary to run `run_Nexuse.m`.
- Submit the job (the standard run script is `run_Nexuse.m`) with a command such as
- Customize the run script `run_Nexuse.m`.
- You could change the variables `scen`, `scenShortName`, `tpRes`, `limDifference`, and `dbSuffix` as instructed in the script. (Recommendation: For quick test, use `tpRes = 8` and `limDifference = 0.1`.)
- If you don't want to run all 4 years, adjust `endY` for the case you are running. E.g., if endY=staY, it will only run one year.
- Submit the job (the standard run script is `run_Nexuse.m`) with a command such as (Note: Don't type `.m` after the script name.)
```
bsub -n 36 -R "model==XeonGold_6150" -R "rusage[mem=5180]" -W "10:00" matlab -r run_Nexuse
```
The parameters in this command can be costomized. Details can be found [here](euler_instruction.html#batch-system-how-to-run-nexus-e) and [here](https://scicomp.ethz.ch/wiki/Getting_started_with_clusters).
The parameters in this command can be costomized. Experience with resource usage to run Nexus-e with different time resolution and convergence criteria can be found [here](euler_instruction.html#experience-with-resource-usage). General instructions on bash and Euler commands can be found [here](euler_instruction.html).
Here we give a short explanation to the example command above:
![](images/euler-job-submission.png)
After submission, Euler will respond by telling you what the jobID is.
- Run script customization
- You could change the variables `scen`, `scenShortName`, `tpRes`, `limDifference`, and `dbSuffix` as instructed in the script
- If you don't want to run all 4 years, adjust `endY` for the case you are running. E.g., if endY=staY, it will only run one year.
- Consult [Nexus-e team](nexus-e@ethz.ch) if you want to run other scripts in the `Run_Nexuse` folder.
- Consult [Nexus-e team](nexus-e@ethz.ch) if you want to run other scripts in the `Run_Nexuse` folder.
Markdown is supported
0% or .
You are about to add 0 people to the discussion. Proceed with caution.
Finish editing this message first!
Please register or to comment