To receive notifications about scheduled maintenance, please subscribe to the mailing-list gitlab-operations@sympa.ethz.ch. You can subscribe to the mailing-list at https://sympa.ethz.ch

Commit e3308f0f authored by luroth's avatar luroth
Browse files

Refactoring, documentation


Former-commit-id: 1ab7cfd9
parent 31a95870
......@@ -13,9 +13,13 @@ Roth et al. 2019 (unpublished)
Run ```ActiveLearningSegmentation/_Demo_.py```
**Hint**: If you get an error related to out of memory (e.g. BrokenPipeError: [Errno 32] Broken pipe) then reduce the number of CPUs for parallel processing:
```ActiveLearningSegmentation/Workflow.py```, Line 194, e.g.
```number_of_cpu = 2```
Prerequisit:
1. RAW images
2. Image masks generated (see [Image mask generation (Standalone Agisoft Script)](ImageProjectionAgisoft/README.md))
2. Image masks generated (see [Image mask generation (Standalone Agisoft Script)](../ImageProjectionAgisoft/README.md))
The campaign folder should therefore contain following subfolders:
......@@ -23,7 +27,7 @@ The campaign folder should therefore contain following subfolders:
- ```image_masks```
It will generate the following folders:
It will generate the following folder:
- ```segmentation```: Segmented images
......
......@@ -2,7 +2,7 @@
# Early growth trait extraction
Sample scripts to extract plant count, tiller count and begin of stem elongation values.
Sample scripts to extract plant count, tiller count and beginning of stem elongation values.
## Background
Based on:
......@@ -14,9 +14,9 @@ Roth et al. 2019 (unpublished)
Run ```EarlyGrowthTraitExtraction/_Demo_.py```
Prerequisit:
1. Image masks generated (see [Image mask generation (Standalone Agisoft Script)](ImageProjectionAgisoft/README.md))
2. Images segmented in plant and soin (see TODO)
3. Multi-view images gerated
1. Image masks generated (see [Image mask generation (Standalone Agisoft Script)](../ImageProjectionAgisoft/README.md))
2. Images segmented in plant and soin (see [Image segmentation with Random Forest](../ActiveLearningSegmentation/README.md))
3. Multi-view images gerated (see [Multi-view image generation](../MultiViewImage/README.md))
The campaign folder should therefore contain following subfolders:
- ```image_masks```
......
......@@ -34,11 +34,15 @@ def tiller_prediction(LA, GDD_BBCH30):
LA = (LA / 1.0) * (0.125 * 0.150)
#Asym: Model was fit to plot size of 125 * 50 * 3 (mm), 90% canopy coverage as theoretical maximum
Asym = ((125 * 50 * 3) * 0.9 ** 2)
Asym = ((125 * 50 * 3) * 0.9 )
# Prevent complex numbers if LA is larger than expected Asymp
if LA > Asym:
LA = Asym
# xmid and scal: Dependend on GDD_BBCH30, empirical estimation with 2018/2019 data
xmid = 3.040694822 + GDD_BBCH30 * -0.005602603
scal = 0.729949090 + GDD_BBCH30 * 0.002182503
xmid = 3.210957888 + GDD_BBCH30 * -0.004929346
scal = 0.806696378 + GDD_BBCH30 * 0.002618851
# Dependence leaf area to tiller count:
# LA ~ SSlogis(log(tiller_count+1), 125 * 50 * 3, xmid, scal)
......@@ -232,8 +236,7 @@ def process_plant_count(path_campaign, delta_to_BBCH30,
def process_tiller_count(path_campaign, path_BBCH30_estimation = None,
path_BBCH30_measured = None, path_GDD_BBCH30_estimation=None, design=None):
def process_tiller_count(path_campaign, path_BBCH30_estimation=None, design=None):
GDD_from = - 200
GDD_to = 0
......@@ -255,46 +258,16 @@ def process_tiller_count(path_campaign, path_BBCH30_estimation = None,
GDD = df_GDDs.loc[df_GDDs.campaign_date == campaign_date_long, 'GDD'].values[0]
# Read BBCH30 estimation
df_BBCH30_estimations = None
if path_BBCH30_estimation is not None:
df_BBCH30 = pd.read_csv(path_BBCH30_estimation)
# Join with GDD to get delta to BBCH30 in GDD
df_BBCH30['GDD_date'] = pd.to_datetime(df_BBCH30['trait_value.timestamp']).dt.strftime("%Y-%m-%d")
df_GGD_at_BBCH30 = pd.merge(df_BBCH30, df_GDDs, left_on="GDD_date", right_on="campaign_date")
df_GGD_at_BBCH30['GDD_BBCH30_estimation'] = GDD - df_GGD_at_BBCH30.GDD
df_GGD_at_BBCH30['GDD_BBCH30_estimation'] = df_GGD_at_BBCH30['GDD_BBCH30_estimation'].median()
df_BBCH30_estimations = df_GGD_at_BBCH30
elif path_GDD_BBCH30_estimation is not None:
df_BBCH30_estimations = pd.read_csv(path_GDD_BBCH30_estimation)
df_BBCH30_estimations['GDD_BBCH30_estimation'] = df_BBCH30_estimations['value']
df_BBCH30_estimations['plot.plot_label'] = df_BBCH30_estimations['plot_label']
# Read BBCH30 measurement
df_BBCH30_measurements = None
if path_BBCH30_measured is not None:
df_BBCH30 = pd.read_csv(path_BBCH30_measured)
# Join with GDD to get delta to BBCH30 in GDD
df_BBCH30['GDD_date'] = pd.to_datetime(df_BBCH30['trait_value.timestamp']).dt.strftime("%Y-%m-%d")
df_GGD_at_BBCH30 = pd.merge(df_BBCH30, df_GDDs, left_on="GDD_date", right_on="campaign_date")
df_GGD_at_BBCH30['GDD_BBCH30_measurement'] = GDD - df_GGD_at_BBCH30.GDD
df_GGD_at_BBCH30['GDD_BBCH30_measurement'] = df_GGD_at_BBCH30['GDD_BBCH30_measurement'].median()
df_BBCH30_measurements = df_GGD_at_BBCH30
df_BBCH30 = pd.read_csv(path_BBCH30_estimation)
df_BBCH30['delta_GDD_BBCH30'] = GDD - df_BBCH30.value
# Read LA
LA_data = {}
for LA_file in LA_files:
df_LA = pd.read_csv(LA_file)
if df_BBCH30_estimations is not None:
df_LA = pd.merge(df_LA, df_BBCH30_estimations[['plot.plot_label', 'GDD_BBCH30_estimation']], left_on="plot_label", right_on='plot.plot_label')
df_LA.drop(columns=['plot.plot_label'], inplace=True)
if df_BBCH30_measurements is not None:
df_LA = pd.merge(df_LA, df_BBCH30_measurements[['plot.plot_label', 'GDD_BBCH30_measurement']], left_on="plot_label", right_on='plot.plot_label')
df_LA.drop(columns=['plot.plot_label'], inplace=True)
df_LA = pd.merge(df_LA, df_BBCH30[['plot.UID', 'delta_GDD_BBCH30']], left_on="plot_label", right_on='plot.UID')
#df_LA.drop(columns=['plot.UID'], inplace=True)
parts = LA_file.name.split("_")
if (len(parts) == 3):
......@@ -320,77 +293,40 @@ def process_tiller_count(path_campaign, path_BBCH30_estimation = None,
for design_label, df_ in LA_data.items():
if design is not None and design_label != design:
print("Skip design", design_label)
path_trait_file = path_trait_csvs / (design_label + "_bbch30measurement-tillers.csv")
continue
print("-----------------------\nGenerate tiller trait csv for", design_label, "for date", campaign_date)
if path_BBCH30_estimation is not None or path_GDD_BBCH30_estimation is not None:
# Filter out too far GDD_BBCH30 values
df__ = df_.loc[(df_['GDD_BBCH30_estimation'] > GDD_from) & (df_['GDD_BBCH30_estimation'] < -GDD_to)].copy()
df__['tiller_estimation_bbch_estimation'] = [
tiller_prediction(row['LA'], row['GDD_BBCH30_estimation']) for index, row in df__.iterrows()]
df__['trait'] = "PltTilDen"
df__['trait_id'] = 34
df__['timestamp'] = datetime.strptime(campaign_date, "%Y%m%d")
df__['GDD_BBCH30'] = df__['GDD_BBCH30_estimation']
df__ = df__[['trait', 'trait_id', 'tiller_estimation_bbch_estimation', 'timestamp', 'GDD_BBCH30', 'plot_label', 'LA']].copy()
df__['value'] = df__['tiller_estimation_bbch_estimation']
df__['value_json'] = df__.apply(lambda row: row.iloc[4:5].to_json(), axis=1)
path_trait_file = path_trait_csvs / (design_label + "_bbch30estimation-tillers.csv")
if len(df__) > 0:
print("Calculated tillers, bbch30 estimation: median",
np.nanmedian(df__['tiller_estimation_bbch_estimation']))
print("Median GDD_BBCH30 estimated:", df__['GDD_BBCH30'].median())
print("Median LA:", df__['LA'].median())
df__.to_csv(path_trait_csvs / (design_label + "_bbch30estimation-tillers.csv"), index=False)
else:
# remove trait file if no measurements found
print("No estimated BBCH30 measurements left after filtering, delete potential trait file")
try:
path_trait_file.unlink()
except FileNotFoundError:
print("No file")
if path_BBCH30_measured is not None:
# Filter out too far GDD_BBCH30 values
df__ = df_.loc[(df_['GDD_BBCH30_measurement'] > GDD_from) & (df_['GDD_BBCH30_measurement'] < GDD_to)].copy()
df__['tiller_estimation_bbch_measurement'] = [
tiller_prediction(row['LA'], row['GDD_BBCH30_measurement']) for index, row in df__.iterrows()]
df__['trait'] = "PltTilDen"
df__['trait_id'] = 34
df__['timestamp'] = datetime.strptime(campaign_date, "%Y%m%d")
df__['GDD_BBCH30'] = df__['GDD_BBCH30_measurement']
df__ = df__[['trait', 'trait_id', 'tiller_estimation_bbch_measurement', 'timestamp', 'GDD_BBCH30', 'plot_label', 'LA']].copy()
df__['value'] = df__['tiller_estimation_bbch_measurement']
df__['value_json'] = df__.apply(lambda row: row.iloc[4:5].to_json(), axis=1)
path_trait_file = path_trait_csvs / (design_label + "_bbch30measurement-tillers.csv")
if len(df__) > 0:
print("Calcualted tillers, bbch30 measurement: median",
np.nanmedian(df__['tiller_estimation_bbch_measurement']))
print("Median GDD_BBCH30 measured:", df__['GDD_BBCH30'].median())
print("Median LA:", df__['LA'].median())
df__.to_csv(path_trait_file, index=False)
else:
print("No measured BBCH30 measurements left after filtering, delete potential trait file")
try:
path_trait_file.unlink()
except FileNotFoundError:
print("No file")
# Filter out too far GDD_BBCH30 values
df__ = df_.loc[(df_['delta_GDD_BBCH30'] > GDD_from) & (df_['delta_GDD_BBCH30'] < GDD_to)].copy()
df__['tiller_estimation'] = [
tiller_prediction(row['LA'], row['delta_GDD_BBCH30']) for index, row in df__.iterrows()]
df__['trait'] = "PltTilDen"
df__['trait_id'] = 34
df__['timestamp'] = datetime.strptime(campaign_date, "%Y%m%d")
df__ = df__[['trait', 'trait_id', 'tiller_estimation', 'timestamp', 'plot_label', 'delta_GDD_BBCH30', 'LA']].copy()
df__['value'] = df__['tiller_estimation']
df__['value_json'] = df__.apply(lambda row: row.iloc[4:6].to_json(), axis=1)
path_trait_file = path_trait_csvs / (design_label + "_bbch30estimation-tillers.csv")
if len(df__) > 0:
print("Calcualted tillers, bbch30 estimation: median",
np.nanmedian(df__['value']))
print("Median GDD_BBCH30 measured:", df__['delta_GDD_BBCH30'].median())
print("Median LA:", df__['LA'].median())
df__.to_csv(path_trait_file, index=False)
else:
print("No BBCH30 estimation left after filtering, delete potential trait file")
try:
path_trait_file.unlink()
except FileNotFoundError:
print("No file")
else:
print("-------------\nNo LA data found, skip campaign", str(path_campaign))
......@@ -549,6 +485,7 @@ def process_NadirCC_LCCC_LA_BBCH30(path_campaign, GSD):
df_LA['value'] = df_LA['LA']
df_LA['timestamp'] = datetime.strptime(campaign_date, "%Y%m%d")
df_LA['plot.UID'] = df_LA['plot_label']
df_LA.to_csv(path_trait_csvs / (design_label + "_LA.csv"), index=False)
......@@ -571,6 +508,7 @@ def process_NadirCC_LCCC_LA_BBCH30(path_campaign, GSD):
df_LA['value'] = value
df_LA['timestamp'] = datetime.strptime(campaign_date, "%Y%m%d")
df_LA['plot.UID'] = df_LA['plot_label']
df_LA.to_csv(path_trait_csvs / (design_label + "_BBCH30.csv"), index=False)
......
import os
from pathlib import Path
from EarlyGrowthTraitExtraction import MultiViewImageGenerator, TraitExtractor
from EarlyGrowthTraitExtraction import TraitExtractor
# Paths and settings for sample campaign
########################################
base_dir = Path(os.path.join(os.path.dirname(os.path.realpath(__file__))))
design_label = "FPWW025_lot2"
path_files = base_dir / "_TestData" / "sonyA9" / "20190402" / "28m_M600P"
path_files = base_dir / ".." / "_TestData" / "sonyA9" / "20190402" / "28m_M600P"
# Ground sampling distance
GSD = 0.003 # m
......@@ -28,7 +28,7 @@ TraitExtractor.process_plant_count(
delta_to_BBCH30=10,
GSD = GSD)
# Extract BBCH 30 estimations
# Extract CC and BBCH 30 estimations
#############################
TraitExtractor.process_NadirCC_LCCC_LA_BBCH30(
......@@ -40,5 +40,5 @@ TraitExtractor.process_NadirCC_LCCC_LA_BBCH30(
TraitExtractor.process_tiller_count(
path_files,
path_GDD_BBCH30_estimation= path_files / "trait_csvs" / "FPWW025_lot2_BBCH30.csv",
path_BBCH30_estimation= path_files / "trait_csvs" / "FPWW025_lot2_BBCH30.csv",
design=design_label)
\ No newline at end of file
......@@ -14,8 +14,8 @@ Roth et al. 2019 (unpublished)
Run ```MultiViewImage/_Demo_.py```
Prerequisit:
1. Image masks generated (see [Image mask generation (Standalone Agisoft Script)](ImageProjectionAgisoft/README.md))
2. Images segmented in plant and soin (see TODO)
1. Image masks generated (see [Image mask generation (Standalone Agisoft Script)](../ImageProjectionAgisoft/README.md))
2. Images segmented in plant and soin (see [Image segmentation with Random Forest](../ActiveLearningSegmentation/README.md))
The campaign folder should therefore contain following subfolders:
- ```image_masks```
......@@ -23,6 +23,7 @@ The campaign folder should therefore contain following subfolders:
It will generate the following folders:
- ```GC_AC```: Genereated segmented multi-view images
- ```GC_AC/*_canopy_coverages.csv```: Extracted canopy cover value per image and plot
import os
from pathlib import Path
from EarlyGrowthTraitExtraction import MultiViewImageGenerator, TraitExtractor
from MultiViewImage import MultiViewImageGenerator
# Paths and settings for sample campaign
########################################
base_dir = Path(os.path.join(os.path.dirname(os.path.realpath(__file__))))
design_label = "FPWW025_lot2"
path_files = base_dir / "_TestData" / "sonyA9" / "20190402" / "28m_M600P"
path_files = base_dir / ".." / "_TestData" / "sonyA9" / "20190402" / "28m_M600P"
# Ground sampling distance
GSD = 0.003 # m
......
......@@ -30,4 +30,4 @@ Additionally, usefull utils for other tasks can be found here:
[1]: Roth, Aasen, Walter, Liebisch 2018: Extracting leaf area index using viewing geometry effects—A new perspective on high-resolution unmanned aerial system photography, ISPRS Journal of Photogrammetry and Remote Sensing. https://doi.org/10.1016/j.isprsjprs.2018.04.012
[2]: Roth et al. 2019, unpublished
\ No newline at end of file
[2]: Roth et al. 2019, in prep.
\ No newline at end of file
Markdown is supported
0% or .
You are about to add 0 people to the discussion. Proceed with caution.
Finish editing this message first!
Please register or to comment