To receive notifications about scheduled maintenance, please subscribe to the mailing-list gitlab-operations@sympa.ethz.ch. You can subscribe to the mailing-list at https://sympa.ethz.ch

Commit 894c525a authored by Xuqian Yan's avatar Xuqian Yan
Browse files

Added more info for Gurobi and for the compatibility of Matlab & Python

parent 112a5d64
......@@ -27,6 +27,13 @@
##### Database definer missing
Add a dummy user (same name as the missing definer - you can see it from the error message) to the database and [grant all previleges](#grant-full-privileges-to-a-user-in-mysql-workbench) .
#### Software compatibility
##### Matlab & python
On Euler, sometimes you get a similar error `ImportError: ...pyexpat...: undefined symbol: XML_SetHashSalt`. This might be because of conflice between Matlab's and Python's `libexpat.so.1` library. If you encounter this, it is better to consult IT. Or, you could try different combinations of Matlab and python by yourself.
Here is an example to compare the versions of the `libexpat.so.1` library of Python 3.7.1 and Matlab R2020a:
- Input command `ldd /cluster/apps/python/3.7.1/x86_64/lib64/python3.7/lib-dynload/pyexpat.cpython-37m-x86_64-linux-gnu.so`. It gives `libexpat.so.1 => /lib64/libexpat.so.1 (0x00002b00348bd000)`. This means the Python 3.7.1's library `pyexpat.cpython-37m-x86_64-linux-gnu.so` is linked against the operating system's `libexpat.so.1` library and not the one from Matlab (`/cluster/apps/matlab/R2020a/bin/glnxa64/libexpat.so.1`).
- The screenshot below shows that the operating system's library links to the version 1.6.0; while Matlab R2020a's library links to the version 1.6.9. Experiences show that when if Matlab has a newer version than Python (namely, newer than the operating system), it should work.
![](images/euler-ldd-command.png)
......@@ -120,15 +120,24 @@ You could skip this section if you don't plan to run the whole Nexus-e platform
isopen(conn)
```
- **Python** (3.8.6 is recomended)
- **Gurobi** (9.0 is recomended)
- Download Gurobi and set up a free Gurobi academic license <https://www.gurobi.com/academia/academic-program-and-licenses>.
- Connect Gurobi with Python following [this instruction](https://support.gurobi.com/hc/en-us/articles/360044290292-How-do-I-install-Gurobi-for-Python-).
- Gurobi paths
- Make a copy of `user_example_conf.m` in the folder `Run_Nexuse`. Name the copy as `user_conf.m`. (Note: You should only work with the local copy, because the original `xxx_example_conf.m` will be synchronized to git and it shouldn't be changed.)
- Find out the path of `gurobi.sh`. Uncomment and adapt the variable `PATH_GUROBI_FOR_PYTHON` in `user_conf.m` accordingly.
- Find out the path of `gurobi_setup.m`. Uncomment and adapt the variable `PATH_GUROBI_FOR_MATLAB` in `user_conf.m` accordingly.
- **Python** (3.8.6 is recomended. Python 3.9 is not compatible with Gurobi 9.0.x yet.)
```eval_rst
|:warning:| Note: we are now in the phase of transferring from python 2.x to python 3.x. Therefore, it is recommended to use python 3.x locally and identify potential issues for the transfer.
```
- Python packages: install the required packages listed in `requirements.txt`.
- Python path: If you use Mac or if you want to run Nexus-e in a python virtual environment, you need to configure Python path in Matlab. Otherwise, you could skip this step.
- Make a copy of `user_example_conf.m` in the folder `Run_Nexuse`. Name the copy as `user_conf.m`. (Note: You should only work with the local copy, because the original `xxx_example_conf.m` will be synchronized to git and it shouldn't be changed.)
- Find out your python path (it varies largely depending on your PC's operating system and how you installed python). Uncomment and adapt the variable `PATH_PYTHON` in `user_conf.m` accordingly.
- Python path: If you have more than one Python instances locally (e.g. if you use Mac or if have a python virtual environment), you need to configure Python path in Matlab. Otherwise, you could skip this step.
- If you haven't done so, make a copy of `user_example_conf.m` in the folder `Run_Nexuse`. Name the copy as `user_conf.m`. (Note: You should only work with the local copy, because the original `xxx_example_conf.m` will be synchronized to git and it shouldn't be changed.)
- Find out your python path with which you connected Gurobi. Uncomment and adapt the variable `PATH_PYTHON` in `user_conf.m` accordingly.
- If you have a python virtual environment, you could configure it following the instructions in `user_conf.m`.
- **GAMS** (32.2 is recomended)
......@@ -139,10 +148,6 @@ You could skip this section if you don't plan to run the whole Nexus-e platform
- GAMS path
- If you haven't done so, make a copy of `user_example_conf.m` in the folder `Run_Nexuse`. Name the copy as `user_conf.m`. (Note: You should only work with the local copy, because the original `xxx_example_conf.m` will be synchronized to git and it shouldn't be changed.)
- Find out your GAMS path (it varies largely depending on your PC's operating system and the version of GAMS). Uncomment and adapt the variable `PATH_GAMS` in `user_conf.m` accordingly.
- **Gurobi**(9.0 is recomended)
- Download Gurobi and set up a free Gurobi academic license <https://www.gurobi.com/academia/academic-program-and-licenses>.
- Connect Gurobi with Matlab following [this instruction](https://www.gurobi.com/documentation/9.0/quickstart_mac/matlab_setting_up_grb_for_.html).
##### 4. Run Nexus-e
......@@ -268,19 +273,14 @@ But still, we need to explicitely load all the dependencies, including modules (
```
module load new
module load gams/28.2
module load gurobi/8.1.1
module load matlab/R2018a
module load gurobi/9.0.0
module load matlab/R2020a
module load gcc/4.8.2
module load python/3.7.1
```
```eval_rst
|:warning:| Note: We used to load python/2.7.14. But we are now in the phase of transferring from python 2.x to python 3.x. Therefore, it is recommended to use python 3.x and identify potential issues for the transfer.
```
Your Euler window should look similar to this:
```eval_rst
.. image:: images/euler-module-load.png
:width: 300px
```
**Option 2**: Edit `.bash_profile` so that the modules are automatically loaded upon log in to Euler
......@@ -296,13 +296,13 @@ But still, we need to explicitely load all the dependencies, including modules (
In command line under your home directory, type
```
pip install --user -r nexus-e/requirements.txt
```
```
##### 5. Run Nexus-e
- cd to the `Run_Nexuse` folder
- Before running `run_Nexuse.m` for the first time, you need to run `bench_Nexuse.m`.
It is recommended to repeat this step after major changes to the modules eMark and CentIv.
- Submit the job with a command such as
- Submit the job (the standard run script is `run_Nexuse.m`) with a command such as
```
bsub -n 36 -R "model==XeonGold_6150" -R "rusage[mem=5180]" -W "10:00" matlab -r run_Nexuse
```
......@@ -311,4 +311,9 @@ The parameters in this command can be costomized. Details can be found [here](ht
Here we give a short explanation to the example command above:
![](images/euler-job-submission.png)
After submission, Euler will respond by telling you what the jobID is.
After submission, Euler will respond by telling you what the jobID is.
- Run script customization
- You could change the variables `scen`, `tpRes`, and `limDifference` as instructed in the script
- If you don't want to run all 4 years, adjust `endY` for the case you are running. E.g., if endY=staY, it will only run one year.
- Consult Blazhe Gjorgiev (gblazhe@ethz.ch) if you want to run other scripts in the `Run_Nexuse` folder.
Markdown is supported
0% or .
You are about to add 0 people to the discussion. Proceed with caution.
Finish editing this message first!
Please register or to comment