setup.md 21 KB
Newer Older
1
2
3
4
### **Setup**

#### Get permissions

5
Before setting up, ask for permission from <nexus-e@ethz.ch> to access the codes and the input database.
6

7
```Note:: |:warning:| Currently, the codes and databases are only accessible with an ETH account.
8
9
```

10
#### Local setup
Xuqian Yan's avatar
Xuqian Yan committed
11
Even though the current Nexus-e platform is **not** yet suitable to run locally because of high consumption of resources such as memory, we do recommend to at least get the codes and connect to the database locally in order to view, understand, and edit them easily.
12

Xuqian Yan's avatar
Xuqian Yan committed
13
14
15
Most the instructions below have been tested both on **Windows** and on **Mac**.

#####  1. Get the codes
16
Make sure you have installed [git](https://git-scm.com/).
Xuqian Yan's avatar
Xuqian Yan committed
17

18
The instructions here use **command lines**. If you have a GUI tool for git, you could use the corresponding operations in the GUI tool instead. An example using a GUI tool "GitKraken" can be found in section [Nexus-e repository instructions](nexuse_repo_instruction.html).
Xuqian Yan's avatar
Xuqian Yan committed
19
20
21
22
23
24
25

- Create a folder where you want to put the codes

- cd to that folder

- Clone the overarching repository with this command:

26
  ```git clone --recursive https://gitlab.ethz.ch/nexus-e/nexus-e-framework.git```
Xuqian Yan's avatar
Xuqian Yan committed
27

Xuqian Yan's avatar
Xuqian Yan committed
28
  You will be asked for your username and password for the repository and its submodules.
Xuqian Yan's avatar
Xuqian Yan committed
29

Xuqian Yan's avatar
Xuqian Yan committed
30
In the end, the folder will look like the screenshot below. Note: You might not see the hidden files whose names start with a dot, and the `Results` folder will be automatically generated when you run Nexus-e.
Xuqian Yan's avatar
Xuqian Yan committed
31
32
33
34
35
36

![](images/nexus-e-folder.png)


##### 2. Connect with the input database

Xuqian Yan's avatar
Xuqian Yan committed
37
- Find information to connect with the database in `UserDBInfo.txt` in the repository. The file contains four lines (for confidentiality the username and password is not written here, but can be found in the file):
38
39
40
```
itet-psl-s02
3306
Xuqian Yan's avatar
Xuqian Yan committed
41
42
DATABASE_USERNAME
DATABASE_PASSWORD
43
44
45
```

- View the database
46
  - Install [MySQL](https://dev.mysql.com/downloads/mysql/) (Version 8.0 is recommended) and [MySQL Workbench](https://dev.mysql.com/downloads/workbench/). For Windows, follwing [this instruction](https://www.youtube.com/watch?v=OM4aZJW_Ojs). For MacOS, follow [this instruction](https://www.youtube.com/watch?v=zbK8cNS_cg0).
47
48
  - Connect to ETH VPN
  - Connect to the database with MySQL Workbench
49
    - Add a new MySQL connection
50
    ![](images/mysql-workbench-1.png)
51
    - Input connection information
52
    ![](images/mysql-workbench-2.png)
53
54
    - Click the button `Test Connection`. You will be asked to input your database password. Then you will see a seccess message:
    ![](images/mysql-workbench-3.png)
55
    
56
###### 2.1. (Optional) Make a local copy of the database
57
    
58
59
60
61
```eval_rst
|:warning:| By default, everyone has write permission to the database. Therefore, to be safe, always make a local copy of the database before playing with it - i.e., first “Dump” it to a local folder, then import it to your personal MySQL account. These can be done in this window:
```
![](images/mysql-workbench-dump.png)
62

Xuqian Yan's avatar
Xuqian Yan committed
63
64
65
66
You might encounter the following problems when you run Nexus-e with a local database, follow the link for possible solutions:  
(1) [Error: "Access denied for user 'your_username'@'your_host_address' ..."](faq.html#access-denied-to-access-database)  
(2) [Error: "The server time zone value 'xxx' is unrecognized ..."](faq.html#database-server-timezone-issue)  
(3) [Error: "The user specified as a definer ('xxx'@'%') does not exist ..."](faq.html#database-definer-missing)  
Xuqian Yan's avatar
Xuqian Yan committed
67

Xuqian Yan's avatar
Xuqian Yan committed
68
69
70
71
72
73
74
75
##### 3. Prepare software & license
Apart from MySQL introduced above, we also need the following software to run Nexus-e locally. This will be introduced in this section. You could skip this section if you don't plan to run the whole Nexus-e platform locally.  
- Matlab
- Python
- Gurobi
- GAMS
- Latex
- ImageMagick
Xuqian Yan's avatar
Xuqian Yan committed
76

Xuqian Yan's avatar
Xuqian Yan committed
77
###### Matlab 
78
79
80
- Download Matlab (R2020a or higher). Available at [ETH IT shop](https://itshop.ethz.ch/EndUser/Items/Home).

- During installation, tick the box to install "Database Toolbox" as well (This option is available if you downloaded Matlab through the ETH IT shop). If this is not an option during installation, you can download it seperately [here](https://ch.mathworks.com/products/database.html).
Xuqian Yan's avatar
Xuqian Yan committed
81

Xuqian Yan's avatar
Xuqian Yan committed
82
- Set up database connector
Xuqian Yan's avatar
Xuqian Yan committed
83

Xuqian Yan's avatar
Xuqian Yan committed
84
  - Download the **MySQL connector** from [here](https://dev.mysql.com/downloads/connector/j/). **Unzip** the file.
85

Xuqian Yan's avatar
Xuqian Yan committed
86
87
  - Copy the MySQL connector folder (e.g. mysql-connector-java-8.0.18) into a folder at your preference.
  We recommend to put it into the **Matlab preferences folder**, which you can find by typing in Matlab Command Window ```prefdir```.
88

Xuqian Yan's avatar
Xuqian Yan committed
89
  - Create a **javaclasspath.txt** file in the **Matlab preferences folder**.
90

Xuqian Yan's avatar
Xuqian Yan committed
91
92
  - In the **javaclasspath.txt** file, write the path to the connector .jar file that you just copied.
  E.g., On a Windows computer the path is similar to `C:\Users\user\AppData\Roaming\MathWorks\MATLAB\R2018a\mysql-connector-java-8.0.18\mysql-connector-java-8.0.18.jar`.
93

Xuqian Yan's avatar
Xuqian Yan committed
94
  - Reload Matlab
Xuqian Yan's avatar
Xuqian Yan committed
95

Xuqian Yan's avatar
Xuqian Yan committed
96
97
98
99
  - Test whether a database connector is set up successfully. Write the following commands in Matlab (substitue `YOUR_USERNAME` and `YOUR_PASSWORD` with your credentials for the database). If the second command returns `1`, it means success.
  ```
  conn = database("sys", YOUR_USERNAME, YOUR_PASSWORD, 'Vendor', 'MySQL', 'Server', 'itet-psl-s02');
  isopen(conn)
100
  ```
101

Xuqian Yan's avatar
Xuqian Yan committed
102
103
104
105
106
107
108
109
110
111
###### Python 
- Download [Python](https://www.python.org/downloads/). Version 3.8.6 is recomended. Python 3.9 is not compatible with Gurobi 9.0.x yet.
- (Optional) If you (will) use Python for multiple projects, it is recommended to create a virtual environment for Nexus-e (for example, using [virtualenv](https://uoa-eresearch.github.io/eresearch-cookbook/recipe/2014/11/26/python-virtual-env/)). 
- Install the required python packages listed in `requirements.txt` (e.g., with command `pip install -r requirements.txt`). 

###### Gurobi 
- Download Gurobi (9.0 is recomended) and set up a free Gurobi academic license from [here](https://www.gurobi.com/academia/academic-program-and-licenses). 
- Connect Gurobi with Python following [this instruction](https://support.gurobi.com/hc/en-us/articles/360044290292-How-do-I-install-Gurobi-for-Python-).
- Save Gurobi path for Matlab
    - Find out the path of `gurobi_setup.m`. (E.g., for mac, it is similar to `/Library/gurobi903/mac64/matlab`.)
112
    - Open Matlab; under the tab "Home", click **Set Path**; select **Add Folder...**; browse to the folder containing `gurobi_setup.m`; click **Open**.
Xuqian Yan's avatar
Xuqian Yan committed
113
114
115
116
    - A new entry of the selected path will appear on the right side of the **Set Path** window; click **Save**.
    ![](images/matlab-setpath.png)

###### GAMS
117
- Download [GAMS](https://www.gams.com/32/). Version 32.2 is recomended. Our current license doesn't support versions newer than 32.
Xuqian Yan's avatar
Xuqian Yan committed
118
119
120
121
122
- Set GAMS License
  - Use the `gamslice.txt` provided in polybox `02_Model/02_Model_Setup_Instruction`.
  - Install the license: [here](https://support.gams.com/installation:how_do_i_install_a_license_file) or [here](https://www.youtube.com/watch?v=vSe3YGkUVoc) or (for MacOS) [here](https://www.gams.com/latest/docs/UG_MAC_INSTALL.html).
- Save GAMS path for Matlab
  - Find out your GAMS path. It varies largely depending on your PC's operating system and the version of GAMS. For example, GAMS 32 on Mac has the path: `/Library/Frameworks/GAMS.framework/Versions/32/Resources`.
123
  - Similar to how you configured Gurobi for Matlab: Open Matlab; under the tab "Home", click **Set Path**; select **Add Folder...**; browse to the GAMS path; click **Open**.
Xuqian Yan's avatar
Xuqian Yan committed
124
125
126
127
128
129
130
131
132
  - A new entry of the selected path will appear on the right side of the **Set Path** window; click **Save**.
- Connect GAMS with Python
  - Follow the instruction [here](https://github.com/NREL/gdx-pandas).       

###### Latex
- Download [Latex](https://www.latex-project.org/).

###### ImageMagick  
- Instructions for Windows:
133
134
  - Download ImageMagick from [here](https://imagemagick.org/script/download.php#windows). Install ImageMagick from the .exe file. Tick **all** boxes when asked which packages to be installed (by default only the first three are selected).
  - Download dependable software "Ghostscript" from [here](https://www.ghostscript.com/download/gsdnld.html). Select Public License Ghostscript. Install the Ghostscript.
Xuqian Yan's avatar
Xuqian Yan committed
135
136
137
138
139
140
  - Optional (if ImageMagick does not work), download and install Visual studio from [here](https://support.microsoft.com/en-us/help/2977003/the-latest-supported-visual-c-downloads).
  
- Instructions for Mac:
  - Install ImageMagick from [here](https://imagemagick.org/script/download.php#macosx)

- Test if ImageMagick works:
141
142
143
144
145
  - Open Windows command prompt and browse to folder that contains a pdf file, e.g., testch.pdf
  - Use the following command to convert file: `convert testch.pdf testch.jpg`

##### 4. Edit conf_local.m
- Make a copy of `example_conf_local.m` in the folder `Run_Nexuse/conf_local`. Name the copy as `conf_local.m`. (Note: You should only work with the local copy, because the original `example_conf_local.m` will be synchronized to git and it shouldn't be changed.)
Xuqian Yan's avatar
Xuqian Yan committed
146

147
- In `conf_local.m` uncomment and modify the following variables based on your local computer's settings. More instructions can be found in the script `conf_local.m` itself. 
Xuqian Yan's avatar
Xuqian Yan committed
148
149
150

  - PATH_MYSQL
  - PATH_PYTHON
151
  - PATH_GUROBI_FOR_PYTHON
Xuqian Yan's avatar
Xuqian Yan committed
152
153
154
  - PATH_LATEX
  - PATH_CONVERT
  - PATH_GAMS
155
156
157
158
  ```eval_rst
  .. note::
     You don't necessarily need all software paths here. For example, if you only want to run the postprocess scripts, :code:`PATH_GUROBI_FOR_PYTHON` is not needed; if you only want to run DBcreation scripts, only :code:`DB_INFO_FILE` and :code:`PATH_MYSQL` are needed.
  ```
Xuqian Yan's avatar
Xuqian Yan committed
159

160
161
162
- (Optional) If you want to work with a [local copy of the database](#optional-make-a-local-copy-of-the-database)
  - Make a copy of `example_UserDBInfo_local.txt` in the folder `Run_Nexuse/conf_local`. Name the copy as `UserDBInfo_local.txt`. (Note: You should only work with the local copy, because the original `example_UserDBInfo_local.txt` will be synchronized to git and it shouldn't be changed.)
  - Edit `example_UserDBInfo_local.txt` based on your local settings:  
Xuqian Yan's avatar
Xuqian Yan committed
163
  ```
164
  127.0.0.1 (or "localhost")
Xuqian Yan's avatar
Xuqian Yan committed
165
166
167
168
  3306
  YOUR_LOCAL_USERNAME (e.g. "root")
  YOUR_LOCAL_PASSWORD
  ```  
169
170
171
172
173
174
  - Uncomment the variable `DB_INFO_FILE` in `conf_local.m`.

- Test `conf_local.m`: in Matlab, run the script `conf_local/conf_test.m`. 
  - If your `conf_local.m` is set up correctly, you should see outputs like "xxx works." 
  - If you see "xxx FAILED", there will be information in the output showing why xxx failed. However, this is not necessarily a problem. Because as mentioned above, you don't always need all software. For example, if you only want to run postprocess scripts, `PATH_GUROBI_FOR_PYTHON` is not needed; if you only want to run DBcreation scripts, only `DB_INFO_FILE` and `PATH_MYSQL` are needed.
  
gblazhe's avatar
gblazhe committed
175
 
Xuqian Yan's avatar
Xuqian Yan committed
176
##### 5. Run Nexus-e
177

Xuqian Yan's avatar
Xuqian Yan committed
178
179
- Connect to ETH VPN (in order to connect with the database)
- Open Matlab
180
181
182
183
184
- (Optional) Run `bench_Nexuse.m`. This is required before you run `run_Nexuse.m` for the first time, or after major changes have been made to eMark and/or CentIv. Running `bench_Nexuse.m` will create a file `/Gemel/results/benchmark_2015.mat` which is necessary to run `run_Nexuse.m`.
- Run the script `run_Nexuse.m` in the `Run_Nexuse` folder. There are a few parameters that you can customize in the beginning of this run script:
  - You could change the variables `scen`, `scenShortName`, `tpRes`, `limDifference`, and `dbSuffix` as instructed in the script
  - If you don't want to run all 4 years,  adjust `endY` for the case you are running. E.g., if endY=staY, it will only run one year.
- Consult the [Nexus-e team](nexus-e@ethz.ch) if you want to run other scripts in the `Run_Nexuse` folder.
Xuqian Yan's avatar
Xuqian Yan committed
185

186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
230
231
232
233
234
235
236
237
238
239
240
241
242
243
244
245
246
247
248
249
250
251
252
253
254
#### Software for interacting with Euler from local machine ####

In our experience, it is usefull to have two tools available for
- transferring files between Euler and your local machine and
- interacting with the command prompt on Euler.

**GUIs for file transfer**

For transferring files between Euler and your personal computer, a graphical user interface (GUI) is useful.
We suggest [FileZilla](https://filezilla-project.org/) since it is available for "all" platforms.
The scicomp wiki also has [information](https://scicomp.ethz.ch/wiki/Getting_started_with_clusters#Graphical_tools) on this.

**Command prompts for secure shell connections**

Unix systems (Linux/Mac) come with shells (command prompts) from which useful interactions with Euler are possible.
E.g., type `ssh <username>@euler.ethz.ch` to make a "secure shell" connection to Euler and issue commands to the Euler command prompt.
Windows users are less fortunate and have to install additional software to achieve this functionality.
Examples are
- putty.exe and
- [MobaXterm](https://mobaxterm.mobatek.net/).

We can highly recommend MobaXTerm.

<!-- ##### Install: putty.exe (not needed for Mac) ##### -->

<!-- Putty is command promt like interface from which you run your scripts on the cluster. -->
<!-- Use the commands below to run everything through putty. -->

<!-- ##### Install: WinSCP.exe/or Filezilla (for Mac) ##### -->

<!-- WinSCP is your GUI with your folder on the cluster, there you put your scripts and receive your results. -->
<!-- Results must be saved with a command in your script: -->
<!-- save (‘results’,’Var1’,’Var2’…). Var1, Var2, … denote the variables you want to save, while results denote the name of the file crated by Matlab (this is wat you see in WinSCP). -->

<!-- Remember to type in port 22 (default might be different) -->

**Login**

- Log-in with `ssh` on Unix-based operating systems:

  `ssh <ethz-username>@euler.ethz.ch`

  Lacking SSH keys, ssh will ask you for your ETH (LDAP) password and after that your following commands are issued to Euler.
  See [scicomp wiki](https://scicomp.ethz.ch/wiki/Getting_started_with_clusters#SSH) for instructions for setting up SSH keys if desired.
- File transfers between local and Euler in FileZilla can be made through `sftp` (secure file transfer protocol).
  For this, enter `sftp://euler.ethz.ch` in the 'server' field and your ETH-user name and password in the respective fields, then hit 'connect'.
- Access to Euler: since May 2020, first time access to Euler is only granted upon request. ETH account holders just write an email to	cluster-support@id.ethz.ch;
   If you used the Euler cluster before the 15. May 2020, then you
   would need to change
   your LDAP password to get again access to the Euler cluster
   (wait for 1 day until the change takes effect).
   (see https://scicomp.ethz.ch/wiki/Getting_started_with_clusters
   for more info).

<!-- #### Starting steps #### -->

<!--  - login on putty (Hostname: euler.ethz.ch; than ethz username and pasword) -->
<!--  - login on the WinSCP (Hostname: euler.ethz.ch; than ethz username and	password) -->
<!--  - -->

<!-- NOTE: You can use windows Command Prompt to run your jobs on Euler instead of Putty. In this case, you can connect by typing: ssh username@euler.ethz.ch -->

<!-- #### Alternative to Putty #### -->

<!-- MobaXterm https://mobaxterm.mobatek.net/ -->

<!-- It offers more options and better interface looks then Putty.  In the same time, it can replace WinSCP. Strongly recommended. However for copying/transferring files from/to Euler -->
<!-- folders is slow. For this purpose WinSCP is better. [FileZilla](https://filezilla-project.org/) does the job well, too. -->

255
#### Euler setup
256
257
258
In order to run the full Nexus-e platform efficiently, we use [Euler](https://scicomp.ethz.ch/wiki/Euler), an ETH cluster for High Performance Computing. To set up the Nexus-e platform on Euler, you first need to access Euler. Instructions on accessing Euler and commands for Euler can be found [here](https://scicomp.ethz.ch/wiki/Getting_started_with_clusters#Euler).

##### 1. Join the Nexus-e user group
259
We have a user group for access to Nexus-e specific licenses (e.g. GAMS) and the Euler pre-paid share. The user group is called: **MAVT-esc-nexus-e**, and is set and maintained by D-MAVT IT. To join the user group, ask Blazhe Gjorgiev <gblazhe@ethz.ch> to send an email to `servicedesk@mavt.ethz.ch`.
260
261

To check whether you are added to the user group, use command line ssh to Euler (type `ssh username@euler.ethz.ch`), then type `groups` to show a list of groups that your user account is linked to, which should contain “MAVT-esc-nexus-e”.
Xuqian Yan's avatar
Xuqian Yan committed
262
263

##### 2. Get the codes
264

265
266
267
- Create a folder where you want to put the codes
- cd to that folder
- Clone the overarching repository with this command:
268
  ```git clone --recursive https://gitlab.ethz.ch/nexus-e/nexus-e-framework.git```
Xuqian Yan's avatar
Xuqian Yan committed
269
  You will be asked for your username and password for the repository and its submodules.
270

Xuqian Yan's avatar
Xuqian Yan committed
271
272
In the end, the folder will look like the screenshot below. Note: tThe `Results` folder will be automatically generated when you run Nexus-e.
![](images/euler-nexus-e-folder.png)
273

274
##### 3. Prepare dependencies
275
276
Unlike setting up Nexue-s locally, we don't need to download any software or license on Euler. Instead, they are all prepared for all users in the user group **MAVT-esc-nexus-e**. You should already be a member of the group if you have followed the [first step](#join-the-nexus-e-user-group).

277
But still, we need to explicitely load all the dependencies, including modules (an Euler term for "software") and python packages.
278
279
280

- Load modules

281
  - Option 1: Manually load modules every time before running Nexus-e
282

283
  Copy the following commands into Euler:
284
285
286
  ```
  module load new
  module load gams/28.2
287
288
  module load gurobi/9.0.0
  module load matlab/R2020a
289
290
  module load gcc/4.8.2
  module load python/3.7.1
Xuqian Yan's avatar
Xuqian Yan committed
291
  module load texlive
292
293
  ```

294
  - Option 2: Edit `.bash_profile` so that the modules are automatically loaded upon log in to Euler
295

296
  Write the same commands at the end of `.bash_profile`. This is a hidden file in your Euler's home directory. After adding these commands, your `.bash_profile` should look similar to this:
297
298
299
  ![](images/euler-bash-profile.png)

  **Test**: To test whether the modules have been successfully loaded, type `module list` in the commaand line. It should then list all loaded modules.
300

301
- Install python packages
302
303

  This you only need to do it **once**, i.e. you don't need to do it everytime when you run Nexus-e.
304

305
306
307
308
  In command line under your home directory, 
  - `cd` to the folder where you have stored the Nexus-e codes
  - type `pip install --user -r requirements.txt`

309
310
  You might encounter software compatibility problems when running Nexus-e on Euler, follow the link for possible solutions:
  [Matlab & Python compatibility error: "ImportError: ...pyexpat...: undefined symbol: XML_SetHashSalt"](faq.html#matlab-python)
311

312
313
314
315
316
317
318
319
320
321
322
323
324
325
326
327


##### 4. Connect with the input database
- Download the **MySQL connector** from [here](https://dev.mysql.com/downloads/connector/j/). **Unzip** the file.

- Copy the MySQL connector folder into the hidden .matlab folder (e.g., .matlab/2020a). 
  - Tip: To see all folders including the hidden ones, use the command `ls -a`. 
  - If you don't see the .matlab folder, run Matlab once: type in `matlab` -> you will see Matlab is started in the terminal -> type in `exit` -> Matlab will stop -> you should then see the .matlab folder.

- Create a **javaclasspath.txt** file in the same folder as above.

- In the **javaclasspath.txt** file, write the path to the connector .jar file that you just copied.
E.g., On Blazhe's Euler account the path is `/cluster/home/gblazhe/.matlab/R2018a/mysql-connector-java-8.0.18/mysql-connector-java-8.0.18.jar`.

- Logoff and re-login to Euler

328
  
329
##### 5. Run Nexus-e
330
- `cd` to the `Run_Nexuse` folder
Xuqian Yan's avatar
Xuqian Yan committed
331
- (Optional) Run `bench_Nexuse.m`. This is required before you run `run_Nexuse.m` for the first time, or after major changes have been made to eMark and/or CentIv. Running `bench_Nexuse.m` will create a file `/Gemel/results/benchmark_2015.mat` which is necessary to run `run_Nexuse.m`.
332
333
334
335
- Customize the run script `run_Nexuse.m`.
  - You could change the variables `scen`, `scenShortName`, `tpRes`, `limDifference`, and `dbSuffix` as instructed in the script. (Recommendation: For quick test, use `tpRes = 8` and `limDifference = 0.1`.)
  - If you don't want to run all 4 years,  adjust `endY` for the case you are running. E.g., if endY=staY, it will only run one year.
- Submit the job (the standard run script is `run_Nexuse.m`) with a command such as (Note: Don't type `.m` after the script name.)
336
337
338
```
bsub -n 36 -R "model==XeonGold_6150" -R "rusage[mem=5180]" -W "10:00" matlab -r run_Nexuse
```
339
The parameters in this command can be costomized. Experience with resource usage to run Nexus-e with different time resolution and convergence criteria can be found [here](euler_instruction.html#experience-with-resource-usage). General instructions on bash and Euler commands can be found [here](euler_instruction.html).
340
341
342
343

Here we give a short explanation to the example command above:
![](images/euler-job-submission.png)

344
After submission, Euler will respond by telling you what the jobID is.
345

346
- Consult [Nexus-e team](nexus-e@ethz.ch) if you want to run other scripts in the `Run_Nexuse` folder.
347
348
349
350

NOTE: For large memory submission, it is recommended to select EPYC_7742, for submission with smaller memory requirements either select EPYC_7742 or XeonGold_6150.

:warning: We have experienced issues with the database connector when computing node is not selected on Euler.