To receive notifications about scheduled maintenance, please subscribe to the mailing-list gitlab-operations@sympa.ethz.ch. You can subscribe to the mailing-list at https://sympa.ethz.ch

Commit 9789dffa authored by holukas's avatar holukas
Browse files

Executable adjustments

parent 4cfac557
# Folders not needed for repo (old)
# e.g.: /src
target/
# Byte-compiled / optimized / DLL files
......
......@@ -2,6 +2,17 @@
![DIIVE](images/logo_diive1_256px.png)
## v0.20.0 | 17 Jun 2021
### Executable
**CHANGES**:
- Adjusted code to facilitate the creation of stand-alone executables using the newest version of the fman build system (`fbs Pro`, https://build-system.fman.io/).
- Removed `-alpha` from version number since it resulted in compilation errors using pyinstaller.
- `Eddy Covariance > EC Quality Control`: Removed sankey plot and added bar plots instead.
**ADDITIONS**:
- Added splash screen, shown during script start
## v0.19.0-alpha | 14 Jun 2021
### More Outlier Detection Methods
......
https://pandas.pydata.org/pandas-docs/stable/development/contributing_docstring.html
## conda environments
### Create conda environment with Python 3.6
```
conda create -n myenv python=3.6
```
### Clone env
```
conda create --name myclone --clone myenv
```
### Remove conda environment
```python
conda remove --name myenv --all
```
### Create conda env from .yml file
```
conda env create -f environment.yml
```
### Create .yml file (conda)
```
conda env export > environment.yml
```
### Create requirements.txt file (pip)
```
conda list -e > requirements.txt
```
## fman build system (fbs)
For executable from PyQt5
- https://build-system.fman.io/manual/
- https://github.com/mherrmann
- https://github.com/mherrmann/fbs-tutorial
- https://build-system.fman.io/troubleshooting
- https://www.learnpyqt.com/courses/packaging-and-distribution/packaging-pyqt5-apps-fbs/
```
pip install fbs PyQt5==5.9.2 pyinstaller==3.4
```
https://docs.conda.io/projects/conda/en/latest/user-guide/tasks/manage-pkgs.html#preventing-packages-from-updating-pinning
## Code Publication
Packaging is tricky. The goal is to make an executable .exe for Windows (Mac OSX and Linux support later).
### Requirements
**Needs these versions in the conda env:**
- python 3.6
- pyinstaller 3.4
- pyqt5 5.9.2
**fbs also needs this installed:**
- **NSIS** (required) from https://nsis.sourceforge.io/Main_Page
–> install and add installation folder to environment variable PATH in Windows
atm no import pandas-profiler
### Important
Some notes about specific modules.
#### statsmodels
Generates an error when trying to use fbs. Code compiles OK, but then on trying to execute the script there is an error and app does not start.
##### In short:
- DOES NOT WORK: import statsmodels.api as sm
- **WORKS: import statsmodels.formula.api as smf**
##### More Info
- https://github.com/pyinstaller/pyinstaller/issues/3921
- https://stackoverflow.com/questions/56810739/error-when-using-statsmodels-with-pyinstaller
#### pandas-profiling
Convenient module that generates statistical output that can be exported to a file. Unfortunately including this module in the code generates this error when trying to run the .exe on Windows:
> confuse.ConfigReadError: file L:\Dropbox\luhk_work\Programming\ProjectRoom Repository\Amp_INDEV\Amp_dist_test\target\Amp\pandas_profiling\config_default.yaml could not be read: [Errno 2] No such file or directory: 'L:\\Dropbox\\luhk_work\\Programming\\ProjectRoom Repository\\Amp_INDEV\\Amp_dist_test\\target\\Amp\\pandas_profiling\\config_default.yaml'
##### GitHub Link
- https://github.com/pandas-profiling/pandas-profiling
##### Installation
```
pip install pandas-profiling
```
https://medium.com/@pypripackages/using-gitlab-pipelines-to-deploy-python-packages-in-production-and-staging-environments-8ab7dc979274
- publish code on gitlab or better github?
- create setup.py
- create pypi package
- create conda package
name: DIIVE2
name: DIIVE3
channels:
- anaconda
- conda-forge
......@@ -8,8 +8,8 @@ dependencies:
- bleach=3.3.0=pyhd3eb1b0_0
- boruta_py=0.3=py_0
- brotlipy=0.7.0=py38h2bbff1b_1003
- ca-certificates=2020.12.5=h5b45459_0
- certifi=2020.12.5=py38haa244fe_1
- ca-certificates=2021.5.25=haa95532_1
- certifi=2021.5.30=py38haa95532_0
- cffi=1.14.4=py38hcd4344a_0
- chardet=4.0.0=py38haa95532_1003
- cmarkgfm=0.4.2=py38he774522_0
......@@ -39,7 +39,7 @@ dependencies:
- numpy=1.19.1=py38h5510c5b_0
- numpy-base=1.19.1=py38ha3acd2a_0
- olefile=0.46=py_0
- openssl=1.1.1k=h8ffe710_0
- openssl=1.1.1k=h2bbff1b_0
- packaging=20.9=pyhd3eb1b0_0
- pandas=1.1.1=py38ha925a31_0
- patsy=0.5.1=py38_0
......@@ -88,9 +88,9 @@ dependencies:
- zlib=1.2.11=h62dcd97_4
- zstd=1.4.5=h04227a9_0
- pip:
- altgraph==0.17
- fbs==0.9.0
- fbs==0.9.8
- macholib==1.14
- pefile==2019.4.18
- pyinstaller==3.4
prefix: C:\Users\holukas\Anaconda3\envs\DIIVE2
- pyinstaller==4.3
- pyinstaller-hooks-contrib==2021.1
prefix: C:\Users\holukas\Anaconda3\envs\DIIVE3
bleach @ file:///tmp/build/80754af9/bleach_1612211392645/work
Boruta==0.3
brotlipy==0.7.0
certifi==2021.5.30
cffi @ file:///C:/ci/cffi_1606255208697/work
chardet @ file:///C:/ci/chardet_1607690654534/work
cmarkgfm==0.4.2
colorama @ file:///tmp/build/80754af9/colorama_1607707115595/work
cryptography @ file:///C:/ci/cryptography_1607637849569/work
cycler==0.10.0
docutils==0.16
fbs @ https://build-system.fman.io/pro/3201d706-2168-42fb-b8d7-876dd639194e/0.9.8
future==0.18.2
idna @ file:///home/linux1/recipes/ci/idna_1610986105248/work
joblib @ file:///tmp/build/80754af9/joblib_1594236160679/work
keyring @ file:///C:/ci/keyring_1611778732215/work
kiwisolver==1.2.0
macholib==1.14
matplotlib @ file:///C:/ci/matplotlib-base_1597876438601/work
mkl-fft==1.2.0
mkl-random==1.1.1
mkl-service==2.3.0
numpy @ file:///C:/ci/numpy_and_numpy_base_1596215850360/work
olefile==0.46
packaging @ file:///tmp/build/80754af9/packaging_1611952188834/work
pandas @ file:///C:/ci/pandas_1598371563714/work
patsy==0.5.1
pefile==2019.4.18
Pillow @ file:///C:/ci/pillow_1594298230227/work
pkginfo==1.7.0
plotly @ file:///tmp/build/80754af9/plotly_1599764639247/work
pycparser @ file:///tmp/build/80754af9/pycparser_1594388511720/work
Pygments @ file:///tmp/build/80754af9/pygments_1610565767015/work
pyinstaller==4.3
pyinstaller-hooks-contrib==2021.1
pyOpenSSL @ file:///tmp/build/80754af9/pyopenssl_1608057966937/work
pyparsing==2.4.7
PySocks @ file:///C:/ci/pysocks_1605287845585/work
python-dateutil==2.8.1
pytz==2020.1
pywin32-ctypes==0.2.0
readme-renderer==24.0
requests @ file:///tmp/build/80754af9/requests_1608241421344/work
requests-toolbelt==0.9.1
retrying==1.3.3
rfc3986 @ file:///tmp/build/80754af9/rfc3986_1594058972433/work
scikit-learn @ file:///C:/ci/scikit-learn_1598377018496/work
scipy==1.5.2
seaborn @ file:///tmp/build/80754af9/seaborn_1600553570093/work
sip==4.19.24
six==1.15.0
statsmodels==0.11.1
threadpoolctl @ file:///tmp/tmp9twdgx9k/threadpoolctl-2.1.0-py3-none-any.whl
tornado==6.0.4
tqdm @ file:///tmp/build/80754af9/tqdm_1611857934208/work
twine @ file:///C:/ci/twine_1610465923942/work
urllib3 @ file:///tmp/build/80754af9/urllib3_1611694770489/work
webencodings==0.5.1
win-inet-pton @ file:///C:/ci/win_inet_pton_1605306167264/work
wincertstore==0.2
bleach @ file:///tmp/build/80754af9/bleach_1612211392645/work
Boruta==0.3
brotlipy==0.7.0
certifi==2021.5.30
cffi @ file:///C:/ci/cffi_1606255208697/work
chardet @ file:///C:/ci/chardet_1607690654534/work
cmarkgfm==0.4.2
colorama @ file:///tmp/build/80754af9/colorama_1607707115595/work
cryptography @ file:///C:/ci/cryptography_1607637849569/work
cycler==0.10.0
docutils==0.16
fbs @ https://build-system.fman.io/pro/3201d706-2168-42fb-b8d7-876dd639194e/0.9.8
future==0.18.2
idna @ file:///home/linux1/recipes/ci/idna_1610986105248/work
joblib @ file:///tmp/build/80754af9/joblib_1594236160679/work
keyring @ file:///C:/ci/keyring_1611778732215/work
kiwisolver==1.2.0
macholib==1.14
matplotlib @ file:///C:/ci/matplotlib-base_1597876438601/work
mkl-fft==1.2.0
mkl-random==1.1.1
mkl-service==2.3.0
numpy @ file:///C:/ci/numpy_and_numpy_base_1596215850360/work
olefile==0.46
packaging @ file:///tmp/build/80754af9/packaging_1611952188834/work
pandas @ file:///C:/ci/pandas_1598371563714/work
patsy==0.5.1
pefile==2019.4.18
Pillow @ file:///C:/ci/pillow_1594298230227/work
pkginfo==1.7.0
plotly @ file:///tmp/build/80754af9/plotly_1599764639247/work
pycparser @ file:///tmp/build/80754af9/pycparser_1594388511720/work
Pygments @ file:///tmp/build/80754af9/pygments_1610565767015/work
pyinstaller==4.3
pyinstaller-hooks-contrib==2021.1
pyOpenSSL @ file:///tmp/build/80754af9/pyopenssl_1608057966937/work
pyparsing==2.4.7
PySocks @ file:///C:/ci/pysocks_1605287845585/work
python-dateutil==2.8.1
pytz==2020.1
pywin32-ctypes==0.2.0
readme-renderer==24.0
requests @ file:///tmp/build/80754af9/requests_1608241421344/work
requests-toolbelt==0.9.1
retrying==1.3.3
rfc3986 @ file:///tmp/build/80754af9/rfc3986_1594058972433/work
scikit-learn @ file:///C:/ci/scikit-learn_1598377018496/work
scipy==1.5.2
seaborn @ file:///tmp/build/80754af9/seaborn_1600553570093/work
sip==4.19.24
six==1.15.0
statsmodels==0.11.1
threadpoolctl @ file:///tmp/tmp9twdgx9k/threadpoolctl-2.1.0-py3-none-any.whl
tornado==6.0.4
tqdm @ file:///tmp/build/80754af9/tqdm_1611857934208/work
twine @ file:///C:/ci/twine_1610465923942/work
urllib3 @ file:///tmp/build/80754af9/urllib3_1611694770489/work
webencodings==0.5.1
win-inet-pton @ file:///C:/ci/win_inet_pton_1605306167264/work
wincertstore==0.2
[Paths]
Prefix = PyQt5/Qt
\ No newline at end of file
......@@ -2,7 +2,7 @@
"app_name": "DIIVE",
"author": "Grassland Sciences ETH Zurich",
"main_module": "src/main/python/main.py",
"version": "0.19.0-alpha",
"version": "0.20.0",
"extra_pyinstaller_args": [
"--additional-hooks-dir", "hooks"
],
......
import pandas as pd
def df_unique_values(df):
"""
Return numpy array of unique values across all columns of a dataframe
Parameters
----------
df: pandas DataFrame
Returns
-------
array
"""
return pd.unique(df.values.ravel())
def count_unique_values(df):
"""
Count number of occurrences of unique values across DataFrame
Parameters
----------
df: pandas DataFrame
Returns
-------
pandas DataFrame
"""
_unique_values = df_unique_values(df=df)
counts_df = pd.DataFrame(index=_unique_values)
for col in df.columns:
counts_df[col] = df[col].value_counts(dropna=False)
return counts_df.sort_index()
def flatten_multiindex_all_df_cols(df):
df.columns = ['_'.join(col).strip() for col in df.columns.values]
return df
def remove_duplicate_cols(df):
return df.loc[:, ~df.columns.duplicated()]
\ No newline at end of file
......@@ -24,8 +24,9 @@ class Ui_MainWindow(object):
Prepares the raw gui, i.e. the canvas that is filled with content later.
"""
def setupUi(self, MainWindow, ctx):
def setupUi(self, MainWindow, ctx, version_info):
self.ctx = ctx # Application context, resources
self.version_info = version_info
# # Load external CSS file
# dir_start = pathlib.Path(__file__) # Auto-detect where template_main.py is located.
......@@ -48,7 +49,8 @@ class Ui_MainWindow(object):
# ==================================================================
# Init MainWindow (level 0)
info = self.ctx.build_settings
info = self.version_info
# info = self.ctx.build_settings # deprecated
MainWindow.setWindowTitle(f"{info['app_name']} "
f"{info['version']}")
MainWindow.setWindowIcon(QIcon(self.ctx.icon_diive))
......
......@@ -3,7 +3,6 @@ import fnmatch
import logging
import os
import pathlib
import time
import zipfile as zf
from pathlib import Path
......@@ -883,15 +882,6 @@ def find_nans_in_df_col(df, col):
return gaps_df, gap_count
def remove_duplicate_cols(df):
return df.loc[:, ~df.columns.duplicated()]
def flatten_multiindex_all_df_cols(df):
df.columns = ['_'.join(col).strip() for col in df.columns.values]
return df
def verify_dir(dir):
""" Create dir if it does not exist. """
pathlib.Path(dir).mkdir(parents=True, exist_ok=True)
......
......@@ -13,6 +13,7 @@ import pandas as pd
from PyQt5 import QtCore as qtc
from PyQt5 import QtGui as qtg
from PyQt5 import QtWidgets as qw
from fbs_runtime import PUBLIC_SETTINGS as version_info
from fbs_runtime.application_context.PyQt5 import ApplicationContext, cached_property
from gui.DialogWindows.SelectFileTypeSettings import SelectFileTypeSettings
......@@ -31,7 +32,6 @@ from inout import DataFunctions as data_fn, \
ExampleFiles as example_files
from inout.VarGroups import *
from logger import log
from modboxes.Extensions import _FXN_ThresholdPlots as pl_FXN_ThresholdPlots
from modboxes.default.Analyses import GapFinder as an_GapFinder, Aggregator as an_Aggregator, \
ClassFinder as an_ClassFinder, ExtremeEventsFinder as an_ExtremeEventsFinder, \
FeatureSelection as an_FeatureSelection
......@@ -50,8 +50,8 @@ from modboxes.default.Plots import Scatter as pl_Scatter, Heatmap as pl_Heatmap,
Histogram as pl_Histogram, WindSectors as pl_WindRose, CorrelationMatrix as pl_CorrelationMatrix, \
Quantiles as pl_Quantiles, Cumulative as pl_Cumulative, \
Hexbins as pl_Hexbins, MultiPanel as pl_MultiPanel, DielCycles as pl_DielCycles
from modboxes.Extensions.EddyCovariance import ECQualityControl as ec_QualityControl, \
OffSeasonUptakeCorr as ec_OffSeasonUptakeCorr
from modboxes.Extensions.EddyCovariance.ECQualityControl import ECQualityControl as ec_QualityControl
from modboxes.Extensions.EddyCovariance import OffSeasonUptakeCorr as ec_OffSeasonUptakeCorr
from stats.StatsBoxes import TimeSeriesStats
......@@ -63,9 +63,10 @@ class DIIVE(qw.QMainWindow, Ui_MainWindow, gui_building_blocks.SelectFileTypeSet
twin_ax_active = False # By default no secondary y-axis
twin_ax = -9999
def __init__(self, parent, load_initial_data, ctx):
def __init__(self, parent, load_initial_data, ctx, version_info):
super(DIIVE, self).__init__(parent)
self.setupUi(self, ctx=ctx)
self.setupUi(self, ctx=ctx, version_info=version_info)
self.root_dir = os.path.dirname(os.path.abspath(__file__))
self.ctx = ctx # AppContext, needed to access resources, e.g. button icons
......@@ -83,7 +84,7 @@ class DIIVE(qw.QMainWindow, Ui_MainWindow, gui_building_blocks.SelectFileTypeSet
# self.txt_OnScreenOut.setFormatter(logging.Formatter('%(asctime)s - %(levelname)s - %(message)s'))
# Version info
log(name=DIIVE.__name__, dict={'Version': self.ctx.build_settings['version']}, highlight=False)
log(name=DIIVE.__name__, dict={'Version': self.version_info['version']}, highlight=False)
# Run ID
now_time_dt, now_time_str = get_current_time()
......@@ -168,7 +169,8 @@ class DIIVE(qw.QMainWindow, Ui_MainWindow, gui_building_blocks.SelectFileTypeSet
self.TopMenu.btn_modificationsConvertUnits.clicked.connect(lambda: self.make_tab(make_tab='MF_CONVERT_UNITS'))
self.TopMenu.btn_modificationsLimitDatasetTimeRange.clicked.connect(lambda: self.limit_dataset_timerange())
self.TopMenu.btn_modificationsRenameVar.clicked.connect(lambda: self.make_tab(make_tab='MF_RENAME_VAR'))
self.TopMenu.btn_modificationsRemoveTimeRange.clicked.connect(lambda: self.make_tab(make_tab='MF_REMOVE_TIME_RANGE'))
self.TopMenu.btn_modificationsRemoveTimeRange.clicked.connect(
lambda: self.make_tab(make_tab='MF_REMOVE_TIME_RANGE'))
self.TopMenu.btn_modificationsSubset.clicked.connect(lambda: self.make_tab(make_tab='MF_SUBSET'))
# Create Variable
......@@ -478,8 +480,6 @@ class DIIVE(qw.QMainWindow, Ui_MainWindow, gui_building_blocks.SelectFileTypeSet
'Error': e,
'Recommendation': 'Check if File Settings are correct for this file.'}, highlight=False)
return all_data_df
def select_source(self, action):
......@@ -966,9 +966,9 @@ class DIIVE(qw.QMainWindow, Ui_MainWindow, gui_building_blocks.SelectFileTypeSet
def make_tab_heatmap_plots(self):
"""Make new tab and store current data_df in instance"""
obj = pl_Heatmap.Run(app_obj=self,
title='Heatmap',
tab_id=buildTab.find_free_tab_id(self.TabWidget, tab_ids=['Heatmap']))
obj = pl_Heatmap.Run(app_obj=self, title='Heatmap',
tab_id=buildTab.find_free_tab_id(self.TabWidget, tab_ids=['Heatmap']),
version_info=version_info)
obj.lst_varlist_available.itemClicked.connect(lambda: obj.make_triplet())
# obj.lst_varlist_available.itemClicked.connect(lambda: obj.plot_heatmap_triplet())
obj.btn_save_triplet_png.clicked.connect(lambda: obj.save_heatmap_triplet())
......@@ -993,8 +993,8 @@ class DIIVE(qw.QMainWindow, Ui_MainWindow, gui_building_blocks.SelectFileTypeSet
def make_tab_remove_time_range(self):
"""Make new tab and store current data_df in instance"""
obj = mf_RemoveTimeRange.Run(app_obj=self,
title='Remove Time Range',
tab_id=buildTab.find_free_tab_id(self.TabWidget, tab_ids=['Remove Time Range']))
title='Remove Time Range',
tab_id=buildTab.find_free_tab_id(self.TabWidget, tab_ids=['Remove Time Range']))
obj.lst_varlist_available.itemClicked.connect(lambda: obj.update_target())
obj.btn_add_timerange.clicked.connect(lambda: obj.add_to_rangelist())
obj.lst_rangelist_selected.itemClicked.connect(lambda: obj.remove_from_rangelist())
......@@ -1304,8 +1304,8 @@ class DIIVE(qw.QMainWindow, Ui_MainWindow, gui_building_blocks.SelectFileTypeSet
def make_tab_hampel_od(self):
"""Make new tab and store current data_df in instance"""
obj = od_Hampel.Run(app_obj=self,
title='Hampel Filter',
tab_id=buildTab.find_free_tab_id(self.TabWidget, tab_ids=['Hampel Filter']))
title='Hampel Filter',
tab_id=buildTab.find_free_tab_id(self.TabWidget, tab_ids=['Hampel Filter']))
obj.lst_varlist_available.itemClicked.connect(lambda: obj.select_target())
obj.btn_calc.clicked.connect(lambda: obj.calc())
obj.btn_keep_marked.clicked.connect(lambda: obj.keep_marked())
......@@ -1456,21 +1456,6 @@ class DIIVE(qw.QMainWindow, Ui_MainWindow, gui_building_blocks.SelectFileTypeSet
limit_timerange_filter_dt=self.limit_timerange_filter_dt).get_ax()
canvas.draw_idle() # refresh plot
def make_tab_threshold_plots(self):
new_tab_id = buildTab.find_free_tab_id(self.TabWidget, tab_ids=['Ustar', 'Threshold', 'Plots', 'FXN'])
# Create tab instance
this_instance = pl_FXN_ThresholdPlots.MakeNewTab(TabWidget=self.TabWidget,
focus_name=self.focus_col,
col_list_pretty=self.col_list_pretty,
col_dict_tuples=self.col_dict_tuples,
data_df=self.data_df,
new_tab_id=new_tab_id)
# Object
obj = pl_FXN_ThresholdPlots.Run(tab_instance=this_instance)
obj.btn_show_in_plot.clicked.connect(lambda: obj.plot_threshold_plots())
class AppContext(ApplicationContext): # 1. Subclass ApplicationContext
......@@ -1478,12 +1463,22 @@ class AppContext(ApplicationContext): # 1. Subclass ApplicationContext
# https://build-system.fman.io/manual/#cached_property
def run(self): # 2. Implement run()
splash = self._splash_screen()
self.main_window.show()
splash.close()
return self.app.exec_() # 3. End run() with this line
def _splash_screen(self):
"""Show splash screen w/ logo"""
splash_pix = qtg.QPixmap(self.logo_diive_256px)
splash_pix.setDevicePixelRatio(2)
splash = qw.QSplashScreen(splash_pix, qtc.Qt.WindowStaysOnTopHint)
splash.show()
return splash
@cached_property
def main_window(self):
return DIIVE(parent=None, load_initial_data=True, ctx=self)
return DIIVE(parent=None, load_initial_data=True, ctx=self, version_info=version_info)
# Detect folder where script was started from
@cached_property
......@@ -1675,6 +1670,14 @@ class AppContext(ApplicationContext): # 1. Subclass ApplicationContext
def icon_diive(self):
return self.get_resource('icons/logo_diive2.png') # Returns absolute path
@cached_property
def logo_diive(self):
return self.get_resource('icons/logo_diive1.png') # Returns absolute path
@cached_property
def logo_diive_256px(self):
return self.get_resource('icons/logo_diive1_256px.png') # Returns absolute path
@cached_property