To receive notifications about scheduled maintenance, please subscribe to the mailing-list gitlab-operations@sympa.ethz.ch. You can subscribe to the mailing-list at https://sympa.ethz.ch

Due to server maintenance gitlab.ethz.ch-users might experience availability-problems on 6/21, between 9:00am and 9.30am.

Commit f145afb9 authored by stehess's avatar stehess

Update README.md

parent 1e581a2c
......@@ -9,7 +9,6 @@ Two examples for visualizing cGOM: syringe and bottle as used in data-sets (left
### User Guide
1. Install the tool. For this clone the public git repo to your target directory from
https://gitlab.ethz.ch/pdz/cgom.git.
Note that you should have pre-installed python and pip.
To be able to have a look into the source code, a source code editor such as atom (https://atom.io/) is very helpful.
Then install all requirements:
......@@ -17,54 +16,69 @@ Then install all requirements:
$ pip install −r requirements.txt
```
2. The following folders should exist in the git directory:
- data_sets
Datasets for training and validation can be added to this folder. The sets must be presented as the examples.
- gaze
Contains the text file containing information about the fixations. Note that it must have the same name as the corresponding video file.
- images
Is an empty folder. Serves as target directory for the images extracted later.
- labels
Contains a json file with the labels. This is automatically generated.
- misc
Contains functions to extract frames from a video, create masked videos, or manually label images.
- mrcnn
Contains everything responsible for mask R-CNN to work properly.
- toolbox
Contains all functions described in this work.
- videos
Contains a video from the eye tracking camera. Note that it must have the same name as the corresponding gaze file.
- weights
Contains the COCO weights, and two other files containing the weights of the partially trained agent and the fully trained agent.
3. Navigate to the misc-folder and run the function extract_images_from_video.py with the corresponding parameters to extract frames from a video. Note that a gaze file can be included in order to solely generate frames from fixations.
### 2. The following folders should exist in the git directory
data_sets - Datasets for training and validation can be added to this folder. The sets must be presented as the examples.
gaze - Contains the text file containing information about the fixations. Note that it must have the same name as the corresponding video file.
images - Is an empty folder. Serves as target directory for the images extracted later.
labels - Contains a json file with the labels. This is automatically generated.
misc - Contains functions to extract frames from a video, create masked videos, or manually label images.
mrcnn - Contains everything responsible for mask R-CNN to work properly.
toolbox - Contains all functions described in this work.
videos - Contains a video from the eye tracking camera. Note that it must have the same name as the corresponding gaze file.
weights - Contains the COCO weights, and two other files containing the weights of the partially trained agent and the fully trained agent.
### 3. Extracting images from video
Navigate to the misc-folder and run the function extract_images_from_video.py with the corresponding parameters to extract frames from a video. Note that a gaze file can be included in order to solely generate frames from fixations.
```ssh
$ python extract_images_from_video.py --video_path ../videos/Name-of-the-video.avi --output_dir ../images --num_images 60 --gaze_path ../gaze/Name-of-the-gaze-file.txt
```
4. The folder images should contain around 100 frames. Navigate into this folder and create a new folder in it called Object1_Object2. Drag and drop all images into that folder. Note that object 1 and object 2 are placeholder and should be replaced by names of your objects of interest. You may include more objects of interest by following the same logic Obj1_Obj2_Obj3. This procedure is necessary to use the included labelling tool for the training images in step 5.
### 4. Preparing training images
The folder images should contain around 100 frames. Navigate into this folder and create a new folder in it called Object1_Object2. Drag and drop all images into that folder. Note that object 1 and object 2 are placeholder and should be replaced by names of your objects of interest. You may include more objects of interest by following the same logic Obj1_Obj2_Obj3. This procedure is necessary to use the included labelling tool for the training images in step 5.
5. Navigate to the folder toolbox and inspect all the default configuration files. Make changes if required. Then call the function make_masks.py:
### 5. Labelling training images
Navigate to the folder toolbox and inspect all the default configuration files. Make changes if required. Then call the function make_masks.py:
```ssh
$ python make_masks.py
```
Note that this tool currently only works on Linus or iOS. If you are using Windows, please continue with step 8.
6. A window with an image and a bunch of sliders will appear. The sliders can be set to 0, 1, and 2. Now adjust the sliders that all masks corresponding to object1 are set to 1 and all corresponding to object2 to 2.
### 6. Using a simplified mask-labelling tool
A window with an image and a bunch of sliders will appear. The sliders can be set to 0, 1, and 2. Now adjust the sliders that all masks corresponding to object1 are set to 1 and all corresponding to object2 to 2.
Setting one mask to 0 disables the mask, and setting all masks to 0 dumps the image to a different unmasks folder. Note that all masks for an image must exist, otherwise it should be discharged to the unmasks folder since it can cause difficulties during the learning process. You should be able to label ca. half of all images with this method.
7. Now navigate to the data_sets folder. There should be a new subfolder called Object1_Object2. Extract the folder unmasks and place it at a desired destination.
8. Open via.html from misc. In it load ca. 90% of the images of unmasks, this will correspond to the training set.
9. On the first frame draw a polygon around the first object. Then open Region Attributes and replace [Add New] with label. Then label the newly created polygon with either cup or pen.
10. Repeat the above process for all images. Finally click on Save as JSON under Annotation.
11. Repeat steps 9 and 10 and create a validation set with the remaining images.
### 7. Intermediate step
Now navigate to the data_sets folder. There should be a new subfolder called Object1_Object2. Extract the folder unmasks and place it at a desired destination.
### 8. Conventional labelling tool
Open via.html from misc. In it load ca. 90% of the images of unmasks, this will correspond to the training set.
### 9. Creating the training set
On the first frame draw a polygon around the first object. Then open Region Attributes and replace [Add New] with label. Then label the newly created polygon with either cup or pen.
Repeat the above process for all images. Finally click on Save as JSON under Annotation.
### 10. Creating the validation set
Repeat step 9 and create a validation set with the remaining images.
12. Navigate back to data_sets in it create a new folder Object1_Object2_1. In Object1_Object2_1 place the two folders train and val containing the previously labeled images with the corresponding via_region_data.json file. All in all the folder structure with all its content should look similar to Object1_Object2. Also check the content of the via_region_data.json file.
13. Now call the training function from the toolbox:
### 11. Preparing the training of the neural network
Navigate back to data_sets in it create a new folder Object1_Object2_1. In Object1_Object2_1 place the two folders train and val containing the previously labeled images with the corresponding via_region_data.json file. All in all the folder structure with all its content should look similar to Object1_Object2. Also check the content of the via_region_data.json file.
### 13. Calling the training function
Now call the training function from the toolbox:
```ssh
$ python make_train.py
```
14. Once training is over extract the weights from the newly created log folder, rename it according to your preferences and place them in the weights folder.
### 14. Extracting the weights
Once training is over extract the weights from the newly created log folder, rename it according to your preferences and place them in the weights folder.
15. Now call the inference function:
### 15. Mapping objects and gaze data
Now call the inference function:
```ssh
$ python make_gaze.py
```
......
Markdown is supported
0% or
You are about to add 0 people to the discussion. Proceed with caution.
Finish editing this message first!
Please register or to comment