To receive notifications about scheduled maintenance, please subscribe to the mailing-list gitlab-operations@sympa.ethz.ch. You can subscribe to the mailing-list at https://sympa.ethz.ch

setup.md 20.8 KB
Newer Older
1
2
3
4
### **Setup**

#### Get permissions

5
6
Before setting up, you need permissions to access the codes and the input database. Email the following people to ask for the corresponding permissions.

Xuqian Yan's avatar
Xuqian Yan committed
7
```Note:: |:warning:| Currently, the codes are only accessible with an ETH account.
8
9
```

10
11
- Xuejiao Han (xuhan@eeh.ee.ethz.ch):
  - permission for the **DistIv** repository
12
  - access to the **PSL server** for the database (server name: itet-psl-s02)
13
- Jared Garrison (garrison@fen.ethz.ch):
14
15
  - permissions for the **eMark** repository
  - permissions for the **Shared** repository,
16
- Blazhe Gjorgiev (gblazhe@ethz.ch):
17
18
19
20
21
22
23
  - permission for the **Cascades** repository
  - permission for the **Run_Nexuse** repository
- Elena Raycheva (elena.raycheva@esc.ethz.ch):
  - permission for the **CentIv** repository
- Florian Landis  (landisf@ethz.ch):
  - permission for the **GemEl** repository
- Xuqian Yan (xuqian.yan@esc.ethz.ch):
Xuqian Yan's avatar
Xuqian Yan committed
24
  - permission for the overarching **Nexus-e** repository (<https://gitlab.ethz.ch/xyan/nexus-e.git>)
25

26
#### Local setup
27
Even though the current Nexus-e platform is **not** suitable to run locally because of high consumption of resources such as memory, we do recommend to at least get the codes and connect to the database locally in order to view, understand, and edit them easily.
28

Xuqian Yan's avatar
Xuqian Yan committed
29
30
31
Most the instructions below have been tested both on **Windows** and on **Mac**.

#####  1. Get the codes
32
Make sure you have installed [git](https://git-scm.com/).
Xuqian Yan's avatar
Xuqian Yan committed
33

34
The instructions here use **command lines**. If you have a GUI tool for git, you could use the corresponding operations in the GUI tool instead. An example using a GUI tool "GitKraken" can be found in section [Nexus-e repository instructions](nexuse_repo_instruction.html).
Xuqian Yan's avatar
Xuqian Yan committed
35
36
37
38
39
40
41
42
43

- Create a folder where you want to put the codes

- cd to that folder

- Clone the overarching repository with this command:

  ```git clone --recursive https://gitlab.ethz.ch/xyan/nexus-e.git```

Xuqian Yan's avatar
Xuqian Yan committed
44
  You will be asked for your username and password for the repository and its submodules.
Xuqian Yan's avatar
Xuqian Yan committed
45

Xuqian Yan's avatar
Xuqian Yan committed
46
In the end, the folder will look like the screenshot below. Note: You might not see the hidden files whose names start with a dot, and the `Results` folder will be automatically generated when you run Nexus-e.
Xuqian Yan's avatar
Xuqian Yan committed
47
48
49
50
51
52

![](images/nexus-e-folder.png)


##### 2. Connect with the input database

Xuqian Yan's avatar
Xuqian Yan committed
53
- Find information to connect with the database in `UserDBInfo.txt` in the repository. The file contains four lines (for confidentiality the username and password is not written here, but can be found in the file):
54
55
56
```
itet-psl-s02
3306
Xuqian Yan's avatar
Xuqian Yan committed
57
58
DATABASE_USERNAME
DATABASE_PASSWORD
59
60
61
```

- View the database
62
  - Install MySQL (Version 8.0 is recommended)  and MySQL Workbench
63
64
  - Connect to ETH VPN
  - Connect to the database with MySQL Workbench
65
    - Add a new MySQL connection
66
    ![](images/mysql-workbench-1.png)
67
    - Input connection information
68
    ![](images/mysql-workbench-2.png)
69
70
    - Click the button `Test Connection`. You will be asked to input your database password. Then you will see a seccess message:
    ![](images/mysql-workbench-3.png)
71
    ```eval_rst
72
73
    |:warning:| By default, everyone has write permission to the database. Therefore, to be safe, always make a local copy of the database before playing with it - i.e., first “Dump” it to a local folder, then import it to your personal MySQL account. These can be done in this window:
    ```
Xuqian Yan's avatar
Xuqian Yan committed
74
    ![](images/mysql-workbench-dump.png)
75

Xuqian Yan's avatar
Xuqian Yan committed
76
##### 3. Prepare software & license
77
You could skip this section if you don't plan to run the whole Nexus-e platform locally.
Xuqian Yan's avatar
Xuqian Yan committed
78

79
- **Matlab** (R2018a or higher, Matlab R2020b is recomended)
Xuqian Yan's avatar
Xuqian Yan committed
80
81
82

  - Set up database connector

83
    - Download the **MySQL connector** from [here](https://dev.mysql.com/downloads/connector/j/).
84
    Or, if you have access to the Nexus-e polybox, use the `mysql-connector-java-8.0.18.zip` provided in the folder `02_Model/02_Model_Setup_Instruction` (**Unzip** the file before the next step).
Xuqian Yan's avatar
Xuqian Yan committed
85

86
    - Copy the MySQL connector folder (e.g. mysql-connector-java-8.0.18) into a folder at your preference.
Xuqian Yan's avatar
Xuqian Yan committed
87
88
89
90
    We recommend to put it into the **Matlab preferences folder**, which you can find by typing in Matlab Command Window ```prefdir```.

    - Create a **javaclasspath.txt** file in the **Matlab preferences folder**.

91
    - In the **javaclasspath.txt** file, write the path to the connector .jar file that you just copied.
Xuqian Yan's avatar
Xuqian Yan committed
92
93
94
    E.g., On Blazhe's Windows computer the path is `C:\Users\gblazhe\AppData\Roaming\MathWorks\MATLAB\R2018a\mysql-connector-java-8.0.18\mysql-connector-java-8.0.18.jar`.

    - Reload Matlab
95
96

    - Test whether a database connector is set up successfully. Write the following commands in Matlab (substitue `YOUR_USERNAME` and `YOUR_PASSWORD` with your credentials for the database). If the second command returns `1`, it means success.
Xuqian Yan's avatar
Xuqian Yan committed
97
    ```
Xuqian Yan's avatar
Xuqian Yan committed
98
    conn = database("sys", YOUR_USERNAME, YOUR_PASSWORD, 'Vendor', 'MySQL', 'Server', 'itet-psl-s02');
Xuqian Yan's avatar
Xuqian Yan committed
99
100
    isopen(conn)
    ```
101

102
103
104
- **Gurobi** (9.0 is recomended)  
    - Download Gurobi and set up a free Gurobi academic license <https://www.gurobi.com/academia/academic-program-and-licenses>. 
    - Connect Gurobi with Python following [this instruction](https://support.gurobi.com/hc/en-us/articles/360044290292-How-do-I-install-Gurobi-for-Python-).
105
106
107
108
109
110
    - Save Gurobi path for Matlab
        - Find out the path of `gurobi_setup.m`. (E.g., for mac, it is similar to `/Library/gurobi903/mac64/matlab`.)
        - Open Matlab; under the tab "Home", click **Set Path**; select **Add Folder...**; browse to the folder containing `gurobi_setup.m` (from the step above); click **Open**.
        - A new entry of the selected path will appear on the right side of the **Set Path** window; click **Save**.
        ![](images/matlab-setpath.png)
    - Add Gurobi path for python in  `user_conf.m`
111
112
113
114
115
        - Make a copy of `user_example_conf.m` in the folder `Run_Nexuse`. Name the copy as `user_conf.m`. (Note: You should only work with the local copy, because the original `xxx_example_conf.m` will be synchronized to git and it shouldn't be changed.)
        - Find out the path of `gurobi.sh`. Uncomment and adapt the variable `PATH_GUROBI_FOR_PYTHON` in `user_conf.m` accordingly.


- **Python** (3.8.6 is recomended. Python 3.9 is not compatible with Gurobi 9.0.x yet.)
Xuqian Yan's avatar
Xuqian Yan committed
116
117
118
  ```eval_rst
  |:warning:| Note: we are now in the phase of transferring from python 2.x to python 3.x. Therefore, it is recommended to use python 3.x locally and identify potential issues for the transfer.

119
  ```
Xuqian Yan's avatar
Xuqian Yan committed
120
  - Python packages: install the required packages listed in `requirements.txt`.
121
  - Add Python path in `user_conf.m`: If you have multiple Python instances locally (e.g. if you use Mac, if you have python virtual environments, or if you have multiple python versions), you need to configure Python path as instructed here. Otherwise, it might also work if you skip this. But to be safe, it is recommended to follow this step.
122
123
    - If you haven't done so, make a copy of `user_example_conf.m` in the folder `Run_Nexuse`. Name the copy as `user_conf.m`. (Note: You should only work with the local copy, because the original `xxx_example_conf.m` will be synchronized to git and it shouldn't be changed.)
    - Find out your python path with which you connected Gurobi. Uncomment and adapt the variable `PATH_PYTHON` in `user_conf.m` accordingly.
Xuqian Yan's avatar
Xuqian Yan committed
124

125
- **GAMS** (32.2 is recomended)
Xuqian Yan's avatar
Xuqian Yan committed
126
127
128
129
  - Download from [here](https://www.gams.com/download/)
  - Set GAMS License
    - Use the `gamslice.txt` provided in polybox `02_Model/02_Model_Setup_Instruction`.
    - Install the license: [here](https://support.gams.com/installation:how_do_i_install_a_license_file) or [here](https://www.youtube.com/watch?v=vSe3YGkUVoc) or (for MacOS) [here](https://www.gams.com/latest/docs/UG_MAC_INSTALL.html).
130
131
132
133
  - Save GAMS path for Matlab
    - Find out your GAMS path. It varies largely depending on your PC's operating system and the version of GAMS. For example, GAMS 32 on Mac has the path: `/Library/Frameworks/GAMS.framework/Versions/32/Resources`.
    - Similar to how you configured Gurobi for Matlab: Open Matlab; under the tab "Home", click **Set Path**; select **Add Folder...**; browse to the folder containing `gurobi_setup.m` (from the step above); click **Open**.
    - A new entry of the selected path will appear on the right side of the **Set Path** window; click **Save**.
Xuqian Yan's avatar
Xuqian Yan committed
134
135
 - Connect GAMS with Python
    - Follow the instruction [here](https://github.com/NREL/gdx-pandas).
Xuqian Yan's avatar
Xuqian Yan committed
136

137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
- **MySQL**  
So far you should have already downloaded MySQL from the section [Connect with the input database](#connect-with-the-input-database). Here we edit the MySQL-related info in the `user_conf.m` file.  
    - If you haven't done so, make a copy of `user_example_conf.m` in the folder `Run_Nexuse`. Name the copy as `user_conf.m`. (Note: You should only work with the local copy, because the original `xxx_example_conf.m` will be synchronized to git and it shouldn't be changed.)
    - Edit the variable `DB_SUFFIX` in to be your name initials. 
    - Uncomment and adate the variable `PATH_MYSQL` to be the path of your MySQL installation.
    - (Optional) Run Nexus-e with a local copy of the database
        - Create a new text file to substitute `UserDBInfo.txt`.  
        For example, if you have dumped the remote database to your local hosted database with IP address `127.0.0.1`, you could create a new text file as the following:
        ```
        127.0.0.1
        3306
        YOUR_LOCAL_USERNAME (e.g. "root")
        YOUR_LOCAL_PASSWORD
        ```  
        - Name the new file as  `UserDBInfo_xxx.txt`  (e.g., `UserDBInfo_local.txt`). An underscore "_" after "UserDBInfo" is needed to not synchronize it on git (see the file `.gitignore`).
        - Place the new file in the same folder as `UserDBInfo.txt`.
        - Uncomment and adapt the variable `DB_INFO_FILE` in `user_conf.m` to be the new file name (e.g., `DB_INFO_FILE = 'UserDBInfo_local.txt';`).

        You might encounter the following problems when you run Nexus-e with a local database, follow the link for possible solutions:  
        (1) [Error: "Access denied for user 'your_username'@'your_host_address' ..."](faq.html#access-denied-to-access-database)  
        (2) [Error: "The server time zone value 'xxx' is unrecognized ..."](faq.html#database-server-timezone-issue)  
        (3) [Error: "The user specified as a definer ('xxx'@'%') does not exist ..."](faq.html#database-definer-missing)  
159
160

- **ImageMagick**  
gblazhe's avatar
gblazhe committed
161
  - Instructions for Windows:
162
163
164
    - Download ImageMagick from [here](https://imagemagick.org/script/download.php#windows). Install ImageMagick from the .exe file. Tick all boxes when asked which packages to be installed (by default only the first three are selected).
    - Additionally, download dependable software "Ghostscript" from [here](https://www.ghostscript.com/download/gsdnld.html). Select Public License Ghostscript. Install the Ghostscript.
    - Optional (if ImageMagick does not work), download and install Visual studio from [here](https://support.microsoft.com/en-us/help/2977003/the-latest-supported-visual-c-downloads).
gblazhe's avatar
gblazhe committed
165
  
166
167
168
169
  - Test if ImageMagick works:
    - Open Windows command prompt and brows to folder that contains a pdf file, e.g., testch.pdf
    - Use the following command to identify file: magic identify testch.pdf
    - Use the following command to convert file: convert testch.pdf testch.jpg
gblazhe's avatar
gblazhe committed
170
171
    
  - Instructions for Mac:
gblazhe's avatar
gblazhe committed
172
 
Xuqian Yan's avatar
Xuqian Yan committed
173
##### 4. Run Nexus-e
174

Xuqian Yan's avatar
Xuqian Yan committed
175
176
- Connect to ETH VPN (in order to connect with the database)
- Open Matlab
177
- Double check the `user_conf.m` file: whether you have edited `DB_SUFFIX` to be your name initials; whether the paths of the software are correct.
178
179
- Before running `run_Nexuse.m` for the first time, you need to run `bench_Nexuse.m`.
  It is recommended to repeat this step after major changes to the modules eMark and CentIv.
180
- Run the script `run_Nexuse.m` in the `Run_Nexuse` folder. Note, make sure you are into the `Run_Nexuse` folder before runing the `run_Nexuse.m` script.
Xuqian Yan's avatar
Xuqian Yan committed
181
182
183
  - You could change the variables `scen`, `tpRes`, and `limDifference` as instructed in the script
  - Consult Blazhe Gjorgiev (gblazhe@ethz.ch) if you want to run other scripts in the `Run_Nexuse` folder.

184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
230
231
232
233
234
235
236
237
238
239
240
241
242
243
244
245
246
247
248
249
250
251
252
#### Software for interacting with Euler from local machine ####

In our experience, it is usefull to have two tools available for
- transferring files between Euler and your local machine and
- interacting with the command prompt on Euler.

**GUIs for file transfer**

For transferring files between Euler and your personal computer, a graphical user interface (GUI) is useful.
We suggest [FileZilla](https://filezilla-project.org/) since it is available for "all" platforms.
The scicomp wiki also has [information](https://scicomp.ethz.ch/wiki/Getting_started_with_clusters#Graphical_tools) on this.

**Command prompts for secure shell connections**

Unix systems (Linux/Mac) come with shells (command prompts) from which useful interactions with Euler are possible.
E.g., type `ssh <username>@euler.ethz.ch` to make a "secure shell" connection to Euler and issue commands to the Euler command prompt.
Windows users are less fortunate and have to install additional software to achieve this functionality.
Examples are
- putty.exe and
- [MobaXterm](https://mobaxterm.mobatek.net/).

We can highly recommend MobaXTerm.

<!-- ##### Install: putty.exe (not needed for Mac) ##### -->

<!-- Putty is command promt like interface from which you run your scripts on the cluster. -->
<!-- Use the commands below to run everything through putty. -->

<!-- ##### Install: WinSCP.exe/or Filezilla (for Mac) ##### -->

<!-- WinSCP is your GUI with your folder on the cluster, there you put your scripts and receive your results. -->
<!-- Results must be saved with a command in your script: -->
<!-- save (‘results’,’Var1’,’Var2’…). Var1, Var2, … denote the variables you want to save, while results denote the name of the file crated by Matlab (this is wat you see in WinSCP). -->

<!-- Remember to type in port 22 (default might be different) -->

**Login**

- Log-in with `ssh` on Unix-based operating systems:

  `ssh <ethz-username>@euler.ethz.ch`

  Lacking SSH keys, ssh will ask you for your ETH (LDAP) password and after that your following commands are issued to Euler.
  See [scicomp wiki](https://scicomp.ethz.ch/wiki/Getting_started_with_clusters#SSH) for instructions for setting up SSH keys if desired.
- File transfers between local and Euler in FileZilla can be made through `sftp` (secure file transfer protocol).
  For this, enter `sftp://euler.ethz.ch` in the 'server' field and your ETH-user name and password in the respective fields, then hit 'connect'.
- Access to Euler: since May 2020, first time access to Euler is only granted upon request. ETH account holders just write an email to	cluster-support@id.ethz.ch;
   If you used the Euler cluster before the 15. May 2020, then you
   would need to change
   your LDAP password to get again access to the Euler cluster
   (wait for 1 day until the change takes effect).
   (see https://scicomp.ethz.ch/wiki/Getting_started_with_clusters
   for more info).

<!-- #### Starting steps #### -->

<!--  - login on putty (Hostname: euler.ethz.ch; than ethz username and pasword) -->
<!--  - login on the WinSCP (Hostname: euler.ethz.ch; than ethz username and	password) -->
<!--  - -->

<!-- NOTE: You can use windows Command Prompt to run your jobs on Euler instead of Putty. In this case, you can connect by typing: ssh username@euler.ethz.ch -->

<!-- #### Alternative to Putty #### -->

<!-- MobaXterm https://mobaxterm.mobatek.net/ -->

<!-- It offers more options and better interface looks then Putty.  In the same time, it can replace WinSCP. Strongly recommended. However for copying/transferring files from/to Euler -->
<!-- folders is slow. For this purpose WinSCP is better. [FileZilla](https://filezilla-project.org/) does the job well, too. -->

253
#### Euler setup
254
255
256
In order to run the full Nexus-e platform efficiently, we use [Euler](https://scicomp.ethz.ch/wiki/Euler), an ETH cluster for High Performance Computing. To set up the Nexus-e platform on Euler, you first need to access Euler. Instructions on accessing Euler and commands for Euler can be found [here](https://scicomp.ethz.ch/wiki/Getting_started_with_clusters#Euler).

##### 1. Join the Nexus-e user group
257
We have a user group for access to Nexus-e specific licenses (e.g. GAMS) and the Euler pre-paid share. The user group is called: **MAVT-esc-nexus-e**, and is set and maintained by D-MAVT IT. To join the user group, ask Blazhe Gjorgiev (gblazhe@ethz.ch) to send an email to `servicedesk@mavt.ethz.ch`.
258
259

To check whether you are added to the user group, use command line ssh to Euler (type `ssh username@euler.ethz.ch`), then type `groups` to show a list of groups that your user account is linked to, which should contain “MAVT-esc-nexus-e”.
Xuqian Yan's avatar
Xuqian Yan committed
260
261

##### 2. Get the codes
262

263
264
265
266
- Create a folder where you want to put the codes
- cd to that folder
- Clone the overarching repository with this command:
  ```git clone --recursive https://gitlab.ethz.ch/xyan/nexus-e.git```
Xuqian Yan's avatar
Xuqian Yan committed
267
  You will be asked for your username and password for the repository and its submodules.
268

Xuqian Yan's avatar
Xuqian Yan committed
269
270
In the end, the folder will look like the screenshot below. Note: tThe `Results` folder will be automatically generated when you run Nexus-e.
![](images/euler-nexus-e-folder.png)
271
272

##### 3. Connect with the input database
273
- Download the **MySQL connector** from [here](https://dev.mysql.com/downloads/connector/j/).
274
275
Or, if you have access to the Nexus-e polybox, use the `mysql-connector-java-8.0.18.zip` provided in the folder `02_Model/02_Model_Setup_Instruction` (**Unzip** the file before the next step).

276
- Copy the MySQL folder in the hidden .matlab folder (e.g., .matlab/2019b)
277
278
279

- Create a **javaclasspath.txt** file in the same folder as above.

280
- In the **javaclasspath.txt** file, write the path to the connector .jar file that you just copied.
281
282
283
284
285
286
287
E.g., On Blazhe's Euler account the path is `/cluster/home/gblazhe/.matlab/R2018a/mysql-connector-java-8.0.18/mysql-connector-java-8.0.18.jar`.

- Logoff and re-login to Euler

##### 4. Prepare dependencies
Unlike setting up Nexue-s locally, we don't need to download any software or license on Euler. Instead, they are all prepared for all users in the user group **MAVT-esc-nexus-e**. You should already be a member of the group if you have followed the [first step](#join-the-nexus-e-user-group).

288
But still, we need to explicitely load all the dependencies, including modules (an Euler term for "software") and python packages.
289
290
291

- Load modules

292
  **Option 1**: Manually load modules every time before running Nexus-e
293

294
  Copy the following commands into Euler:
295
296
297
  ```
  module load new
  module load gams/28.2
298
299
  module load gurobi/9.0.0
  module load matlab/R2020a
300
301
302
303
  module load gcc/4.8.2
  module load python/3.7.1
  ```
    ```eval_rst
Xuqian Yan's avatar
Xuqian Yan committed
304
    |:warning:| Note: We used to load python/2.7.14 and gurobi/8.1.1. But we are now in the phase of testing the newer versions of the software. Therefore, it is recommended to use python 3 and gurobi 9 to identify potential issues.
305
306
    ```

307
  **Option 2**: Edit `.bash_profile` so that the modules are automatically loaded upon log in to Euler
308

309
  Write the same commands at the end of `.bash_profile`. This is a hidden file in your Euler's home directory. After adding these commands, your `.bash_profile` should look similar to this:
310
311
312
  ![](images/euler-bash-profile.png)

  **Test**: To test whether the modules have been successfully loaded, type `module list` in the commaand line. It should then list all loaded modules.
313

314
- Install python packages
315
316

  This you only need to do it **once**, i.e. you don't need to do it everytime when you run Nexus-e.
317
318
319
320

  In command line under your home directory, type
  ```
  pip install --user -r nexus-e/requirements.txt
321
  ```  
322
323
324
  You might encounter software compatibility problems when running Nexus-e on Euler, follow the link for possible solutions:
  [Matlab & Python compatibility error: "ImportError: ...pyexpat...: undefined symbol: XML_SetHashSalt"](faq.html#matlab-python)
  
Xuqian Yan's avatar
Xuqian Yan committed
325
##### 5. Run Nexus-e
326
- cd to the `Run_Nexuse` folder
327
328
- Before running `run_Nexuse.m` for the first time, you need to run `bench_Nexuse.m`.
  It is recommended to repeat this step after major changes to the modules eMark and CentIv.
329
- Submit the job (the standard run script is `run_Nexuse.m`) with a command such as
330
331
332
```
bsub -n 36 -R "model==XeonGold_6150" -R "rusage[mem=5180]" -W "10:00" matlab -r run_Nexuse
```
333
The parameters in this command can be costomized. Details can be found [here](https://scicomp.ethz.ch/wiki/Getting_started_with_clusters).
334
335
336
337

Here we give a short explanation to the example command above:
![](images/euler-job-submission.png)

338
339
340
341
342
343
After submission, Euler will respond by telling you what the jobID is.  

- Run script customization  
    - You could change the variables `scen`, `tpRes`, and `limDifference` as instructed in the script
    - If you don't want to run all 4 years,  adjust `endY` for the case you are running. E.g., if endY=staY, it will only run one year.
    - Consult Blazhe Gjorgiev (gblazhe@ethz.ch) if you want to run other scripts in the `Run_Nexuse` folder.