VO-scraper merge requestshttps://gitlab.ethz.ch/tgeorg/vo-scraper/-/merge_requests2021-09-27T11:01:59Zhttps://gitlab.ethz.ch/tgeorg/vo-scraper/-/merge_requests/4DBI Update Pack2021-09-27T11:01:59ZbieridoDBI Update PackNew features
- parallel downloads
- range inputs
- resuming / checking downloads
- more TUI
Also updated readme.mdNew features
- parallel downloads
- range inputs
- resuming / checking downloads
- more TUI
Also updated readme.mdhttps://gitlab.ethz.ch/tgeorg/vo-scraper/-/merge_requests/6DBI Update Pack2021-09-27T11:02:06ZbieridoDBI Update PackSee !4See !4https://gitlab.ethz.ch/tgeorg/vo-scraper/-/merge_requests/14Merge features from dev to master2022-11-27T17:35:30ZGeorg TeufelbergerMerge features from dev to master**Fixes:**
+ Catch an error caused by audio-only recordings (#7)**Fixes:**
+ Catch an error caused by audio-only recordings (#7)https://gitlab.ethz.ch/tgeorg/vo-scraper/-/merge_requests/13Merge features from dev to master2022-04-19T19:40:14ZGeorg TeufelbergerMerge features from dev to master**Features:**
+ Nicer progress-bar if `tqdm` Python package is installed. If the package is not installed the scraper will just fall back to the old progress-bar**Features:**
+ Nicer progress-bar if `tqdm` Python package is installed. If the package is not installed the scraper will just fall back to the old progress-barhttps://gitlab.ethz.ch/tgeorg/vo-scraper/-/merge_requests/12Merge features from dev to master2022-02-21T14:40:24ZGeorg TeufelbergerMerge features from dev to master**Features:**
+ Added more keywords to specify resolutions. E.g. you can use `--quality low` to download a recording in `360p`. (part of #5)
+ Print selected resolution for downloading at beginning of script (part of #5)
+ Allow comments...**Features:**
+ Added more keywords to specify resolutions. E.g. you can use `--quality low` to download a recording in `360p`. (part of #5)
+ Print selected resolution for downloading at beginning of script (part of #5)
+ Allow comments in parameter file
**Breakage:**
- Changed mapping for `HD` from `1080p` to `720p` and added a mapping for `FullHD` to `1080p` (part of #5)https://gitlab.ethz.ch/tgeorg/vo-scraper/-/merge_requests/11Merge features from dev to master2021-09-28T18:33:38ZGeorg TeufelbergerMerge features from dev to master**Features:**
+ Skipping lectures if metadata cannot be parsed. This mostly applies when trying to download recordings from a lecture that hasn't uploaded any yet.
+ Remove `www.` prefix from domain name. `www.video.ethz.ch` redirects to...**Features:**
+ Skipping lectures if metadata cannot be parsed. This mostly applies when trying to download recordings from a lecture that hasn't uploaded any yet.
+ Remove `www.` prefix from domain name. `www.video.ethz.ch` redirects to `video.ethz.ch` and can be used to download recordings but authentication with an ETH account would fail when using the former domain.https://gitlab.ethz.ch/tgeorg/vo-scraper/-/merge_requests/10Merge features from dev to master2021-09-21T15:39:03ZGeorg TeufelbergerMerge features from dev to master**Features:**
+ Allow downloading specific resolutions (e.g. `HD`, `4K`, or numeric values like `720p`) instead of the previously vaguely defined `high`, `medium`, `low`.
Default resolution is now `HD` if nothing is specified.
This a...**Features:**
+ Allow downloading specific resolutions (e.g. `HD`, `4K`, or numeric values like `720p`) instead of the previously vaguely defined `high`, `medium`, `low`.
Default resolution is now `HD` if nothing is specified.
This also fixes issues caused when a lecture supplied more then 3 different resolutions.
+ Added [`requirements.txt`](https://pip.pypa.io/en/stable/user_guide/#requirements-files)
+ Fixed the way the date is added to the filename to be consistent between older and newer lectures. ETH changed the timestamp formatting to include seconds halfway through SS21.
+ Fixed a few typos
**Changes:**
* Renamed a few variables
* Used some more [f-strings](https://docs.python.org/3/tutorial/inputoutput.html) for print formatting
**Breakage:**
- For the video quality `high`, `medium`, and `low` will no longer work. Instead use `highest` for highest, `lowest` for lowest, or a specific numeric value for the desired resolution (e.g. `720p` or keywords like `HD`, `4K`).
- As the quality of the video is also part of the filename this means what previously downloaded videos might get redownloaded when using the `--all` parameter. To prevent this generally it is recommended to use the `--history` parameter.https://gitlab.ethz.ch/tgeorg/vo-scraper/-/merge_requests/9Merge features from dev to master2020-10-05T14:01:25ZGeorg TeufelbergerMerge features from dev to master**Features:**
+ Added option for a parameter file. By default the scraper looks for a file called `parameters.txt` in the directory it is being run. If it finds it, it will read and apply any parameters described there. Check the [Readme...**Features:**
+ Added option for a parameter file. By default the scraper looks for a file called `parameters.txt` in the directory it is being run. If it finds it, it will read and apply any parameters described there. Check the [Readme](https://gitlab.ethz.ch/tgeorg/vo-scraper/-/blob/master/README.md) for more info.
+ Added usage hints to highlight some features after running the scraper. If you don't want to see them, use the parameter `--disable-hints`. You can also put it in `parameters.txt` ;)
+ If you dislike progress bars you can now hide it using `--no-progress-bar`
**Changes:**
* Size of progress bar now depends on your terminal window size.Georg TeufelbergerGeorg Teufelbergerhttps://gitlab.ethz.ch/tgeorg/vo-scraper/-/merge_requests/8Merge features from dev to master2020-09-18T12:33:03ZGeorg TeufelbergerMerge features from dev to master**Features:**
+ Added pseudo-hash to filename. Closes #3.
+ Updated documentation in README. Closes #4.
**Changes:**
* Changed `--print-src` to `--print-source`
**Breakage:**
- dd12de305b55c8985bd892a70ab818ec8c1b92a4 changes the filen...**Features:**
+ Added pseudo-hash to filename. Closes #3.
+ Updated documentation in README. Closes #4.
**Changes:**
* Changed `--print-src` to `--print-source`
**Breakage:**
- dd12de305b55c8985bd892a70ab818ec8c1b92a4 changes the filename of newly downloaded videos and subsequently re-downloads are only prevented if the video on disk already has the new filename.
Therefore, this will cause a re-download of all recordings specified by you if you rely solely on existing filenames to prevent downloading old videos again.
Users making use of the history file via `--history` will not be affected.
- 213e77a7b82f6c43fde69f8fea8cc93ab72f9dc6 will break any automated scripts making use of `--print-src`. Make sure to update them to use `--print-source` if you make use of this flag!Georg TeufelbergerGeorg Teufelbergerhttps://gitlab.ethz.ch/tgeorg/vo-scraper/-/merge_requests/7Merge features from dev to master2020-09-05T15:19:50ZGeorg TeufelbergerMerge features from dev to masterFeatures:
+ Added the possibility to select multiple videos by specifying a range. Documentation for this has been added to the [README file](https://gitlab.ethz.ch/tgeorg/vo-scraper/-/blob/master/README.md).
Fixes:
- Fixed no longer be...Features:
+ Added the possibility to select multiple videos by specifying a range. Documentation for this has been added to the [README file](https://gitlab.ethz.ch/tgeorg/vo-scraper/-/blob/master/README.md).
Fixes:
- Fixed no longer being able to authenticate for videos requiring an ETH login.
To be precise: Something in the backend of video.ethz.ch changed, so the "Referer" header is now required when sending a POST request to acquire a cookie for "ETH" protected videos.Georg TeufelbergerGeorg Teufelbergerhttps://gitlab.ethz.ch/tgeorg/vo-scraper/-/merge_requests/5Merge features from dev to master2020-03-23T09:08:49ZGeorg TeufelbergerMerge features from dev to masterFeatures:
+ `--destination` flag to specify the location to save videos to. Closes #1.
+ `--history` flag to specify a file where to print/read episodes URLs from/to so you can finally delete watched lectures without having to worry they...Features:
+ `--destination` flag to specify the location to save videos to. Closes #1.
+ `--history` flag to specify a file where to print/read episodes URLs from/to so you can finally delete watched lectures without having to worry they might re-download.
+ `--version` flag to print the program version and exit.
+ Ignore empty lines in link file.
+ Strip special characters such as `? < > : * | " ^` from filenames as they can cause issues on NTFS.
Changes:
* Moved date further to the front when printing available/selected lecture episodes.
* Collect all episode links first and then start the download so you no longer have to wait for a download to finish in order to select episodes of the next lecture if you pass multiple links.
* Moved the code that creates folders so no folders are created when you just want to print the source links of lecture episodes with `--print-src`.
* Replaced occurrences of `/` with `os.sep` to become more OS agnostic.
* Check if the scraper is being run directly to allow for use as library.
* and finally some general code clean up.https://gitlab.ethz.ch/tgeorg/vo-scraper/-/merge_requests/3Merge features from dev to master2020-03-14T12:23:38ZGeorg TeufelbergerMerge features from dev to masterA small update that includes mostly code clean-ups such as combining the print functionality for available and for selected lecture episodes.
As a new feature the scraper now prints a link to the changelog if a new version has been dete...A small update that includes mostly code clean-ups such as combining the print functionality for available and for selected lecture episodes.
As a new feature the scraper now prints a link to the changelog if a new version has been detected.
Finally an issue was fixed that caused the scraper to crash if a non-[video.ethz.ch](https://video.ethz.ch/) link passed.Georg TeufelbergerGeorg Teufelbergerhttps://gitlab.ethz.ch/tgeorg/vo-scraper/-/merge_requests/2Merge features from dev to master2020-03-13T19:30:17ZGeorg TeufelbergerMerge features from dev to masterThe new features include:
* Update detection mechanism
* Pretty printing of available and selected lecture episodes
* Comments in text files with lecture links
* Credentials in text files with lecture linksThe new features include:
* Update detection mechanism
* Pretty printing of available and selected lecture episodes
* Comments in text files with lecture links
* Credentials in text files with lecture linksGeorg TeufelbergerGeorg Teufelbergerhttps://gitlab.ethz.ch/tgeorg/vo-scraper/-/merge_requests/1Handle lectures that require custom credentials2020-02-20T16:59:23ZGeorg TeufelbergerHandle lectures that require custom credentials* Adds the required logic to handle lecture videos that use their own credentials to log in such as [Introduction to Machine Learning](https://video.ethz.ch/lectures/d-infk/2020/spring/252-0220-00L.html).
* Removes support for passing...* Adds the required logic to handle lecture videos that use their own credentials to log in such as [Introduction to Machine Learning](https://video.ethz.ch/lectures/d-infk/2020/spring/252-0220-00L.html).
* Removes support for passing a login-token as login with username and password is faster and simpler.
* Append the difference between lecture title and video title to the output filename of videos to handle cases such as where a lecture uploads two videos on the same day but one of them is titled `<Lecture name> - Tutorial`.
This should fix all issues that prevent correct downloading of videos from [Introduction to Machine Learning](https://video.ethz.ch/lectures/d-infk/2020/spring/252-0220-00L.html).