Pikobs: CMC Observational Data Processing Toolkit

Python Version Built with Sphinx

Welcome to the Pikobs documentation. Pikobs is a comprehensive Python utility designed for the extraction, processing, and visualization of CMC (Canadian Meteorological Centre) observational data.

This toolkit provides a robust suite of modules to handle various aspects of data analysis, statistical monitoring, and quality control for both conventional and satellite meteorological observations.

—

Overview

Pikobs simplifies observational data management by acting as a bridge between raw assimilation cycle files and data-driven decision-making. The core workflow allows researchers and meteorologists to:

  • Extract: Pull data directly from the CMC assimilation cycle outputs (monitoring/banco) used in the psmon system. This includes standard RDB files from directories such as evalalt, bgckalt, and postalt.

  • Filter: Isolate specific observations using precise bitmask criteria defined by assimilation flags and specific geographical regions.

  • Compute: Generate advanced statistical metrics including O-B (omp), O-A (oma), observation density, and bias corrections based on the extracted data.

  • Visualize: Render high-quality results tailored to each metric. You can explore the specific visualizations generated by each module on their dedicated pages, such as Scatter Plots, Geographical Maps, and Histograms.

—

Getting Started

Installation

It is highly recommended to install Pikobs within a virtual Python environment to maintain dependency integrity:

pip install pikobs

Note

Ensure all underlying system dependencies (such as specific libraries required by Cartopy or Matplotlib) are installed as outlined in your internal setup documentation.

Quick Start & HPC Usage

Important

Interactive Session Setup For resource-intensive computations (especially global datasets), you must start an interactive session on the compute nodes before running Pikobs to avoid overloading login nodes:

qsub -I -X -l select=4:ncpus=80:mpiprocs=80:ompthreads=1:mem=185gb -l place=scatter -l walltime=6:0:0

Once inside the session, modules can be executed directly from the command line. Below is an example of generating a geographical scatter plot:

python -c 'import pikobs; pikobs.scatter.arg_call()' \
    --path_experience_files /home/dlo001/sites8/Data_pikobs/monitoring0/ \
    --experience_name GIC5DA1E22_DDT2 \
    --pathwork yours_path \
    --datestart 2026020100 \
    --dateend 2026020200 \
    --region Monde \
    --family to_amsua_allsky \
    --flags_criteria assimilee \
    --fonction omp oma nobs obs dens bcorr \
    --boxsizex 2 \
    --boxsizey 2 \
    --projection cyl \
    --id_stn all \
    --channel all \
    --n_cpu 80
Command Breakdown:

—

Core Modules Overview

Each module in Pikobs is engineered with distinct characteristics and specialized parameters to handle specific facets of data analysis. Explore their individual pages for detailed usage and examples:

  • Scatter: Generate 2D geographical scatter plots to visualize the spatial distribution of metrics (e.g., O-B, O-A).

  • Timeserie: Analyze the temporal evolution of meteorological statistics over continuous periods.

  • Cardio: Monitor system health and data assimilation flow at a high level.

  • Vdedr: Process and visualize vertical profiles and specific observational metrics.

  • Zone: Perform statistical aggregations and evaluations across specific regional zones.

  • Flag: Deep dive into Quality Control (QC) by analyzing rejection and assimilation flags.

  • Mapobs: Create detailed geographical maps displaying precise observation locations and densities.

  • histogram: Calculate and plot statistical distributions and data frequencies.

  • Profile: is a diagnostic tool designed to evaluate the vertical distribution of meteorological metrics (such as Bias and Standard Deviation) across atmospheric pressure levels or satellite channels.

  • Obscountdb: A diagnostic tool designed to extract, compare, and visualize the volume of assimilated observations between two different assimilation cycle runs.

Documentation Index

🛠️ Core Modules

⚙️ Configuration

🧪 Development

—

Resources & Contact

For more information, feature requests, or bug reports: