To receive notifications about scheduled maintenance, please subscribe to the mailing-list gitlab-operations@sympa.ethz.ch. You can subscribe to the mailing-list at https://sympa.ethz.ch

Commit 102551ac authored by holukas's avatar holukas
Browse files

A bit more LogBox output

parent defbc63f
This diff is collapsed.
# Time Resolution
- 1D … daily time resolution / frequency
- 1M … monthly time resolution / frequency
- 30T … half-hourly time resolution / frequency
# Acknowledgements
## 0.6.0
- PyQt5 to standalone: https://build-system.fman.io/
- https://towardsdatascience.com/exploring-your-data-with-just-1-line-of-python-4b35ce21a82d
- https://www.kaggle.com/nulldata/intro-to-pandas-profiling-simple-fast-eda
- https://stackoverflow.com/questions/23123131/pyqt-how-to-convert-from-python-to-qdate-style
- docstring formats: https://stackoverflow.com/questions/3898572/what-is-the-standard-python-docstring-format
- https://stackoverflow.com/questions/50110800/python-pathlib-make-directories-if-they-don-t-exist
- https://realpython.com/python-super/#an-overview-of-pythons-super-function
- https://stackoverflow.com/questions/16084623/python-is-it-okay-to-pass-self-to-an-external-function
## 0.5.1-alpha
- https://stackoverflow.com/questions/24635721/how-to-compare-frequencies-sampling-rates-in-pandas?answertab=votes#tab-top
- https://stackoverflow.com/questions/10057854/inverse-of-tan-in-python-tan-1
- https://stackoverflow.com/questions/44251040/transparent-navigationbar-in-matplotlib-or-can-the-navigation-buttonsbe-added
## 0.5.0-indev
- https://stackoverflow.com/questions/40995778/resize-column-width-to-fit-into-the-qtablewidget-pyqt
- https://stackoverflow.com/questions/48340463/how-to-understand-closed-and-label-arguments-in-pandas-resample-method
- https://scentellegher.github.io/programming/2017/05/24/pandas-bar-plot-with-formatted-dates.html
- http://stackoverflow.com/questions/12951065/get-bins-coordinates-with-hexbin-in-matplotlib
- https://stackoverflow.com/questions/28655198/best-way-to-display-logs-in-pyqt
## 0.4.0-indev
* https://www.materialui.co/icons
* https://pythonspot.com/matplotlib-legend/
* https://www.tutorialspoint.com/pyqt/pyqt_drag_and_drop.htm
* https://flothesof.github.io/pyqt-microphone-fft-application.html
* https://chrisalbon.com/python/data_wrangling/pandas_join_merge_dataframe
* https://matplotlib.org/3.1.0/tutorials/advanced/transforms_tutorial.html
* https://matplotlib.org/api/_as_gen/matplotlib.axes.Axes.tick_params.html
* https://stackoverflow.com/questions/51787248/python-pandas-select-data-by-month
* https://stackoverflow.com/questions/27844088/python-get-directory-two-levels-up
* https://stackoverflow.com/questions/20209711/avoid-duplicate-tabs-in-qtabwidget
* https://matplotlib.org/gallery/ticks_and_spines/multiple_yaxis_with_spines.html
* https://stackoverflow.com/questions/14088687/how-to-change-plot-background-color
* https://stackoverflow.com/questions/20110170/turn-pandas-multi-index-into-column
* https://stackoverflow.com/questions/38001976/pyqt5-qcombobox-get-value-of-combobox
* *https://stackoverflow.com/questions/4700614/how-to-put-the-legend-out-of-the-plot
* https://stackoverflow.com/questions/30840701/pyqt-qtabwidget-hide-and-close-certain-tab
* https://stackoverflow.com/questions/52723910/matplotlib-user-changes-amount-of-subplots
* https://stackoverflow.com/questions/36086361/embed-matplotlib-in-pyqt-with-multiple-plot
* https://stackoverflow.com/questions/40318759/change-qpushbutton-icon-on-hover-and-pressed
* https://stackoverflow.com/questions/45828478/how-to-set-current-tab-of-qtabwidget-by-name
* *https://towardsdatascience.com/simple-and-multiple-linear-regression-in-python-c928425168f9
* https://stackoverflow.com/questions/56147161/python-matplotlib-update-plot-in-the-background
* https://stackoverflow.com/questions/18262962/setting-dataframe-column-headers-to-a-multiindex
* https://stackoverflow.com/questions/25628496/getting-wider-output-in-pycharms-built-in-console
* https://stackoverflow.com/questions/22774168/how-to-drag-and-drop-from-listwidget-onto-combobox
* https://stackoverflow.com/questions/1541797/how-do-i-check-if-there-are-duplicates-in-a-flat-list
* https://stackoverflow.com/questions/28009370/get-weekday-day-of-week-for-datetime-column-of-dataframe
* https://stackoverflow.com/questions/36209575/how-to-detect-if-a-twin-axis-has-been-generated-for-a-matplotlib-axis
* *https://stackoverflow.com/questions/29370057/select-dataframe-rows-between-two-dates
* https://stackoverflow.com/questions/29007830/identifying-consecutive-nans-with-pandas
* https://stackoverflow.com/questions/29007830/identifying-consecutive-nans-with-pandas
* https://stackoverflow.com/questions/40118037/how-can-i-detect-gaps-and-consecutive-periods-in-a-time-series-in-pandas
# Kudos
SG
empty
\ No newline at end of file
......@@ -22,6 +22,23 @@ class QTextEditLogger(logging.Handler):
self.widget.append(msg)
# self.widget.appendPlainText(msg)
def log(name, dict, highlight):
""" Output to GUI text field (LogBox)
Parameters
----------
highlight
"""
whitespace = ' ' * 8
if name != '':
if highlight:
logging.info(f'<span style="background-color:#FFF9C4;"><b>{name}</b></span>')
else:
logging.info(f'<b>{name}</b>')
for key, value in dict.items():
logging.info(f'{whitespace}<font color="#0288D1">{key}</font>: {value}')
# logging.debug('damn, a bug')
# logging.info('something to remember')
......@@ -52,12 +69,4 @@ class QTextEditLogger(logging.Handler):
#
# return None
def log(name, dict):
""" Output to GUI text field (LogBox) """
whitespace = '&nbsp;' * 8
if name != '':
logging.info(f'<b>{name}</b>')
for key, value in dict.items():
logging.info(f'{whitespace}<font color="#0288D1">{key}</font>: {value}')
......@@ -3,6 +3,7 @@ import fnmatch
import logging
import pathlib
import sys
import time
import traceback
import numpy as np
......@@ -91,7 +92,7 @@ class Amp(QtWidgets.QMainWindow, Ui_MainWindow, GUI_BuildingBlocks.SettingsWindo
# self.txt_OnScreenOut.setFormatter(logging.Formatter('%(asctime)s - %(levelname)s - %(message)s'))
# Version info
log(name=Amp.__name__, dict={'Version': self.ctx.build_settings['version']})
log(name=Amp.__name__, dict={'Version': self.ctx.build_settings['version']}, highlight=False)
# Run id
now_time_dt, now_time_str = get_current_time()
......@@ -191,7 +192,7 @@ class Amp(QtWidgets.QMainWindow, Ui_MainWindow, GUI_BuildingBlocks.SettingsWindo
# Create folders, dir_out_main is automatically created if missing
pathlib.Path(dir_out_run).mkdir(parents=True, exist_ok=True)
log(name=self.setup_folder_structure.__name__, dict={'output dir': dir_out_run})
log(name=self.setup_folder_structure.__name__, dict={'output dir': dir_out_run}, highlight=False)
return dir_out_run
def filter_var_list(self):
......@@ -212,11 +213,11 @@ class Amp(QtWidgets.QMainWindow, Ui_MainWindow, GUI_BuildingBlocks.SettingsWindo
self.lst_Variables.setRowHidden(row, False)
else:
self.lst_Variables.setRowHidden(row, True)
log(name=self.filter_var_list.__name__, dict={'filtering for ': filter_txt})
log(name=self.filter_var_list.__name__, dict={'filtering for ': filter_txt}, highlight=False)
def view_stk_option(self, tab_idx):
self.stk_category_options.setCurrentIndex(tab_idx) # stackedWidget
log(name=self.view_stk_option.__name__, dict={'set tab index ': tab_idx})
log(name=self.view_stk_option.__name__, dict={'set tab index ': tab_idx}, highlight=False)
def view_stk_refinement(self, tab_idx):
self.stk_option_refinements.setCurrentIndex(tab_idx) # stackedWidget
......@@ -315,10 +316,14 @@ class Amp(QtWidgets.QMainWindow, Ui_MainWindow, GUI_BuildingBlocks.SettingsWindo
# Read incoming data to df
num_files = len(source_files_list)
# Log info
suffix = 's' if num_files > 1 else '' # Plural 's'
log(name=f">>> Starting Data Import From {num_files} File{suffix}", dict={}, highlight=True)
tic = time.time() # Timer
skipped_files_list = []
for ix, file in enumerate(source_files_list):
print("Working on file {} / {} {}".format(ix + 1, num_files, file))
try:
self.incoming_data_df = data_fn.read_selected_file(filepath=file,
settings_dict=self.settings_dict)
......@@ -349,8 +354,8 @@ class Amp(QtWidgets.QMainWindow, Ui_MainWindow, GUI_BuildingBlocks.SettingsWindo
# loaded data, i.e. add incoming data to all data.
# Add to already available data, outside method (self.data_df)
if (action == 'add_to_file'):
all_data_df = self.data_df
if (action == 'add_to_file') & (ix == 0):
all_data_df = self.data_df # Only needed for first file that is added
# Add to already available data, inside method (all_data_df already available)
elif (action == 'new_file'):
pass
......@@ -364,15 +369,29 @@ class Amp(QtWidgets.QMainWindow, Ui_MainWindow, GUI_BuildingBlocks.SettingsWindo
# Make sure that the merged df has the correct time resolution
if self.settings_dict['freq'] != '-none-':
all_data_df = all_data_df.asfreq(self.settings_dict['freq'])
except:
print("(!)ERROR while working on file {} (file was skipped)".format(file))
all_data_df = all_data_df.asfreq(self.first_file_freq)
toc = time.time() - tic
size_data_df = self.incoming_data_df.shape
log(name=f"+ Imported Data From File {ix + 1} / {num_files}",
dict={'Source': file,
'Action': action,
'Rows': size_data_df[0],
'Columns': size_data_df[1],
'Time Needed': f"{toc:.3f}s"},
highlight=False)
except ValueError as e:
print("".format(file))
traceback.print_exc()
skipped_files_list.append(file)
log(name=f"(!)ERROR while working on file {ix + 1} / {num_files} (file was skipped)",
dict={'Source': file,
'Action': action,
'Error': e,
'Recommendation': 'Check if File Settings are correct for this file.'}, highlight=False)
print("\n(!)SKIPPED FILES")
print(skipped_files_list)
print("\n")
......@@ -566,7 +585,7 @@ class Amp(QtWidgets.QMainWindow, Ui_MainWindow, GUI_BuildingBlocks.SettingsWindo
# self.dte_mf_ref_LimitTimeRange_end.setMaximumDate(max_data_day_qt)
if log_output:
log(name='Variables were updated.', dict={}) # Log info
log(name='Variables were updated.', dict={}, highlight=False) # Log info
def set_focus(self, action, marker_col):
"""
......@@ -1190,7 +1209,7 @@ class Amp(QtWidgets.QMainWindow, Ui_MainWindow, GUI_BuildingBlocks.SettingsWindo
self.btn_pl_ref_FXN_ThresholdPlots_showPlot.clicked.connect(call_threshold_plots)
def call_od_IQR(self):
log(name=self.call_od_IQR.__name__, dict={})
log(name=self.call_od_IQR.__name__, dict={}, highlight=False)
self.minimize_focus_df() # Reset current focus_df, shall only contain focus_col
self.focus_df, self.marker_col = \
od_InterQuartileRange.Call(**self.update_args(),
......@@ -1198,7 +1217,7 @@ class Amp(QtWidgets.QMainWindow, Ui_MainWindow, GUI_BuildingBlocks.SettingsWindo
self.active_buttons(add=False, keep=True, remove=True, addFocus=True)
def call_od_RunMed(self):
log(name=self.call_od_RunMed.__name__, dict={})
log(name=self.call_od_RunMed.__name__, dict={}, highlight=False)
self.minimize_focus_df() # Reset current focus_df, shall only contain focus_col
self.focus_df, timewindow, min_vals, multiplier_sd, self.marker_col = \
od_RunningMedian.Call(timewindow=self.lne_od_ref_RunningMedian_timewindowLen.text(),
......@@ -1446,30 +1465,30 @@ class Amp(QtWidgets.QMainWindow, Ui_MainWindow, GUI_BuildingBlocks.SettingsWindo
self.update_gui_lists()
def call_ex_VarsToFiles(self):
ex_VariablesToFiles.Execute(df=self.data_df,
dir_out=self.dir_out,
id=self.run_id,
button=self.btn_ex_ref_VarsToFiles,
filetype=self.drp_ex_ref_VarsToFiles_fileType.currentText(),
one_file_per=self.drp_ex_ref_VarsToFiles_oneFilePer.currentText(),
duration=int(self.lne_ex_ref_VarsToFiles_freqDuration.text()),
to_freq=self.drp_ex_ref_VarsToFiles_freq.currentText(),
min_vals=int(self.lne_ex_ref_VarsToFiles_minVals.text()),
agg_method=self.drp_ex_ref_VarsToFiles_agg.currentText(),
orig_freq=self.settings_dict['freq']).export()
ex_VariablesToFiles.Call(df=self.data_df,
dir_out=self.dir_out,
id=self.run_id,
button=self.btn_ex_ref_VarsToFiles,
filetype=self.drp_ex_ref_VarsToFiles_fileType.currentText(),
one_file_per=self.drp_ex_ref_VarsToFiles_oneFilePer.currentText(),
duration=int(self.lne_ex_ref_VarsToFiles_freqDuration.text()),
to_freq=self.drp_ex_ref_VarsToFiles_freq.currentText(),
min_vals=int(self.lne_ex_ref_VarsToFiles_minVals.text()),
agg_method=self.drp_ex_ref_VarsToFiles_agg.currentText(),
orig_freq=self.first_file_freq).export()
self.active_buttons(add=False, keep=False, remove=False, addFocus=False)
def call_ex_SaveFigure(self):
ex_SaveFigure.Execute(df=self.data_df,
dir_out=self.dir_out,
id=self.run_id,
button=self.btn_ex_ref_SaveFigure,
style=self.drp_ex_ref_SaveFigure_style.currentText(),
ax=self.ax_main,
fig=self.plt_Figure,
gs=self.gs,
focus_col=self.focus_col,
ctx=self.ctx).export(fig_out=self.plt_Figure)
ex_SaveFigure.Call(df=self.data_df,
dir_out=self.dir_out,
id=self.run_id,
button=self.btn_ex_ref_SaveFigure,
style=self.drp_ex_ref_SaveFigure_style.currentText(),
ax=self.ax_main,
fig=self.plt_Figure,
gs=self.gs,
focus_col=self.focus_col,
ctx=self.ctx).export(fig_out=self.plt_Figure)
self.active_buttons(add=False, keep=False, remove=False, addFocus=False)
def call_cr_StorageCorrection(self):
......@@ -1497,15 +1516,14 @@ def get_current_time():
now_time_dt = dt.datetime.now()
now_time_str = now_time_dt.strftime("%Y-%m-%d %H:%M:%S")
logging.info(f'{now_time_dt} / {now_time_str}')
log(name=get_current_time.__name__,
dict={'current time': [now_time_dt, now_time_str]})
log(name=get_current_time.__name__, dict={'current time': [now_time_dt, now_time_str]}, highlight=False)
return now_time_dt, now_time_str
def make_run_id(now_time_dt):
now_time_str = now_time_dt.strftime("%Y%m%d-%H%M%S")
run_id = 'AMP-{}'.format(now_time_str)
log(name=make_run_id.__name__, dict={'run id': run_id})
log(name=make_run_id.__name__, dict={'run id': run_id}, highlight=False)
return run_id
......
......@@ -53,7 +53,7 @@ class AddControls():
return self
class Execute():
class Call():
timestamp_col = ('TIMESTAMP', '[yyyy-mm-dd HH:MM:SS]')
def __init__(self, df, dir_out, id, button, style, ax, fig, gs, focus_col, ctx):
......
......@@ -117,7 +117,7 @@ class AddControls():
return self
class Execute():
class Call():
filename_suffix = ''
def __init__(self, df, dir_out, id, button, one_file_per, filetype, duration,
......@@ -226,10 +226,12 @@ class Execute():
GUI_Elements.btn_txt_live_update(btn=self.button, txt=self.button_orig_text, perc=-9999)
def export_file(self, df, filename_out_no_ext, ext):
""" Export variables to file
Variables are first saved as csv, regardless of which format was selected.
If a compressed format was selected, the csv file is first compressed, saved
in the compressed format and then deleted.
"""
Export variables to file
Variables are first saved as csv, regardless of which format was selected.
If a compressed format was selected, the csv file is first compressed, saved
in the compressed format and then deleted.
:param filename_out_no_ext: str
:param ext: str
......
......@@ -145,7 +145,7 @@ class Call:
f"predicted: {num_vals_predicted} values\n" \
f"predicted gaps: {num_vals_predicted_in_gaps} values\n"
logger.log(name=info_module, dict={'info': info_txt}) # Log info
logger.log(name=info_module, dict={'info': info_txt}, highlight=False) # Log info
return info_txt
def get_results(self):
......
......@@ -102,7 +102,8 @@ class Call:
data_df[standardized_qcflag_col] = pipeline_df[standardized_qcflag_col]
logger.log(name='> Performing Outlier Check', dict={'+ Added new flag column': standardized_qcflag_col})
logger.log(name='> Performing Outlier Check', dict={'+ Added new flag column': standardized_qcflag_col},
highlight=False)
return data_df, standardized_qcflag_col
......@@ -148,7 +149,8 @@ class Call:
logger.log(name='> Analyzing Absolute Limits Flag', dict={'Selected Variable': self.focus_col,
'Good Values, Upper Limit': self.upper_limit,
'Good Values, Lower Limit': self.lower_limit,
'# Bad Values': self.focus_df[rejectval_col].sum()})
'# Bad Values': self.focus_df[rejectval_col].sum()},
highlight=False)
self.show_in_plot(outlier_limit_upper_col=outlier_limit_upper_col,
outlier_limit_lower_col=outlier_limit_lower_col,
......
......@@ -96,7 +96,7 @@ class AddControls():
class Call:
def __init__(self, data_df, drp_selected_flux, collist_pretty, coldict_tuples, drp_ssitc, drp_scf, drp_abslim):
""" XXX """
logger.log(name='Pipelines / Amp QC Flag', dict={}) # Log info
logger.log(name='Pipelines / Amp QC Flag', dict={}, highlight=False) # Log info
self.data_df = data_df
self.coldict_tuples = coldict_tuples
......@@ -156,8 +156,8 @@ class Call:
detected_flux = '-none-'
logger.log(name='', dict={'selected flux column': self.flux_col,
'detected flux': detected_flux}) # Log info
logger.log(name='', dict=qc_settings_dict) # Log info
'detected flux': detected_flux}, highlight=False) # Log info
logger.log(name='', dict=qc_settings_dict, highlight=False) # Log info
return qc_settings_dict
def run_pipeline(self):
......@@ -192,7 +192,7 @@ class Call:
added_columns = [qcflag_col, qcflag_test1_col, qcflag_test2_col, qcflag_test3_col]
logger.log(name='+ Added new variables', dict={'qcflag(s)_*': added_columns}) # Log info
logger.log(name='+ Added new variables', dict={'qcflag(s)_*': added_columns}, highlight=False) # Log info
return data_df
......
......@@ -114,7 +114,8 @@ class Call:
"""
standardized_qcflag_col = Call.set_colnames(qcflag_col=qcflag_col, standardize_only=True)
data_df[standardized_qcflag_col] = data_df[qcflag_col]
logger.log(name='> Performing SSITC Test', dict={'+ Added new flag column': standardized_qcflag_col})
logger.log(name='> Performing SSITC Test', dict={'+ Added new flag column': standardized_qcflag_col},
highlight=False)
return data_df, standardized_qcflag_col
@staticmethod
......
......@@ -112,7 +112,7 @@ class Call:
data_df[standardized_qcflag_col] = pipeline_df[standardized_qcflag_col]
logger.log(name='> Performing Spectral Correction Factor Check',
dict={'+ Added new flag column': standardized_qcflag_col})
dict={'+ Added new flag column': standardized_qcflag_col}, highlight=False)
return data_df, standardized_qcflag_col
......
......@@ -134,7 +134,8 @@ class Call:
logger.log(name='> Analyzing EddyPro Raw Data Flags', dict={'Selected Flag': qcflag_col,
'Selected Variable': selected_flagvar,
'Bad Values': self.focus_df[rejectval_col].sum()})
'Bad Values': self.focus_df[rejectval_col].sum()},
highlight=False)
self.show_in_plot(rejectval_col=rejectval_col)
......
Markdown is supported
0% or .
You are about to add 0 people to the discussion. Proceed with caution.
Finish editing this message first!
Please register or to comment