easyqcvbgc/qcv_ingester repository overview

⁠Repository overview

⁠General presentation

The qcv-ingester tool aggregates and harmonizes marine in-situ data following the needs of the UseCase 2.1-BCG - Fair-Ease. It converts files of individual or already aggregated data profiles into a concatenated single file with harmonized vocabulary needed for the project. Profiles are concatenated along the time dimension in the order given by the listing : For Argo data, if BGC data precede CORE profiles, all BGC data will precede CORE data. Use cycle_number variable to associate them again. Vocabulary translations are below. On the left, the writing of each variable in the output file, on the right, all possible translations found over data exploration.

Coordinates and variables :

"lon" : ["longitude", "LONGITUDE", "LON", "lon"],
"lat" : ["latitude", "LATITUDE", "LAT", "lat"],
"time" : ["date", "time", "TIME", "JULD"],
"time_qc" : ["date_qc", "time_qc", "TIME_QC", "JULD_QC"],
"depth" : ["DEPTH", "depth"],
"cycle_number" : ["CYCLE_NUMBER"],
"ref_time" : ["REFERENCE_DATE_TIME"],
"pos_qc" : ["POSITION_QC",]

--

"temperature" : ["TEMP", "TEMPERATURE"],
"salinity" : ["PSAL", "PRACTICAL_SALINITY"],
"oxygen" : ["DOXY"],
"pressure" : ["PRES"],
"chlorophylle" : ["CHLA"],
"nitrate" : ["NO3", "NITRATE", "n_an"],
"bbp700" : ["BBP700"],

All variables are tagged with a suffix to indicate the state of the data (_raw, _dmadjusted, _rtadjusted)

--

Variables extracted from meta files are :

LAUNCH_DATE
PLATFORM_TYPE
PI_NAME meta variables are stored as metadata

--

Units :

"degree_east" : ["degree_east"],
"degree_north" : ["degree_north"],
"degree_celsius" : ["degree_celsius", "degree_Celsius"], # temperature
"psu" : ["psu", "practical_salinity_unit", "PSU"], # salinity
"micromol/l" : ["micromole.l-1", "micromole_per_liter", "μmol.l-1", "μmol/l", "μmol/L", "μmol.L-1"], # concentration liters
"mg/m3" : ["mg/m3",],
"micromole/kg" : ["micromole/kg",],
"m-1" : ["m-1"],
"decibar" : ["decibar", "dbar"], # pressure

Arbitrarily,

dates are written following : "seconds since 1950-01-01T00:00:00 in julian calendar"
longitude is set between : -180° and 180°
latitude is set between : -90° and 90°

WARNING : This application works only platform by platform. For example, it is possible to aggregate and harmonize a whole argo trajectory but one at a time. If two argo trajectories are needed to process, this tool needs to be run 2 times. Version capabilities

⁠Version capabilities

This latest version 2.0 is available for x86 linux (arm64 available on demand) This version is capable of concatenate multiple argo datafiles (including xxx_meta.nc and synthetic profiles), harmonize the output and export it.

⁠How to use

How to use:

To use this tool,

1. install docker (https://docs.docker.com/engine/install/⁠⁠)

2. run the command as explained below with mandatory arguments.

Quick Demo

wget https://gitlab.com/pokapok-showcase/demo-launchers/-/raw/main/qcv-ingester-local.sh && chmod +x qcv-ingester-local.sh && ./qcv-ingester-local.sh

This example will create in your home a repo pokapok-example where you can find downloaded example data and ingested data in data-out.

Run example

if you decide to run it directly, the pull will be automatic.

It requires files located in the repository of your choice. the toll will aggregate all the data in the repository. The data need to be netCDF format. arguments are : VOLUMES :

• data-in : where your input data to harmonize/aggregate are
• data-out : where your output file will be ARGUMENTS
• FORCE_USER_ID : $(id -u)
• FORCE_GROUP_ID : $(id -g)
• APP_START_OPTION : GALAXY

To run the tool, fill this command with your personal paths and data :

docker run \
    -v <data-in>:/runtime/data-in \
    -v <data-out>:/runtime/data-out \
    -e APP_START_OPTION=GALAXY \
    -e FORCE_USER_ID=$(id -u) \
    -e FORCE_GROUP_ID=$(id -g) \
    easyqcvbgc/qcv_ingester:2.0

where (as reminder):

• data-in : where your input data to harmonize/aggregate are
• data-out : where your output file will be with the name .nc (uuid is automatically done by the tool)

WARNING : the command does not create any folder, all needs to exist before

⁠Acknowledgements

This tool was made in the framework of the Horizon Europe project fair-ease (grant agreement number: 101058785).