{
 "cells": [
  {
   "cell_type": "markdown",
   "id": "bbfad06f",
   "metadata": {},
   "source": [
    "# Navigation QA: ODT2601\n",
    "\n",
    "Gulf of Mexico Exploration\n",
    "\n",
    "Re-runs the quality assessment behind this fileset's products and PDF report: it reads the primary GNSS feed (nav33), flags bad fixes, makes bestres, 1min and control, and computes the QA statistics and checks. Edit the paths or the config's `[thresholds]` and re-run to change the assessment.\n",
    "\n",
    "Generated with nav-qa-processing 0.1.0, numpy 2.5.3, pandas 3.0.6, pyproj 3.8.0, shapely 2.2.0."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "4c067bec",
   "metadata": {},
   "outputs": [],
   "source": [
    "from pathlib import Path\n",
    "\n",
    "# Inputs and outputs\n",
    "CONFIG = Path('/data/CruiseData/ODT2601/Products/NAV_QA/ODT2601_config.toml')\n",
    "FILESET = Path('/data/CruiseData/ODT2601/OpenRVDAS')\n",
    "THEME_FILE = None  # a --theme file; the config's [theme] applies on top\n",
    "OUTPUT_DIR = Path('/data/CruiseData/ODT2601/Products/NAV_QA')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "6dca123b",
   "metadata": {},
   "outputs": [],
   "source": [
    "%matplotlib inline\n",
    "import matplotlib.pyplot as plt\n",
    "import pandas as pd\n",
    "from IPython.display import display\n",
    "\n",
    "from nav_qa_processing import plots, reports, stats\n",
    "from nav_qa_processing import quantities as q\n",
    "from nav_qa_processing.config import load_config\n",
    "from nav_qa_processing.qa import run_qa\n",
    "from nav_qa_processing.qc import DESCRIPTIONS, Flag\n",
    "from nav_qa_processing.theme import resolve_theme\n",
    "\n",
    "pd.set_option(\"display.max_colwidth\", 120)\n",
    "\n",
    "config = load_config(CONFIG)\n",
    "result = run_qa(FILESET, config)\n",
    "theme = resolve_theme(THEME_FILE, result.config)\n",
    "print(result)\n",
    "print(result.stats.verdict[1])  # the checks in a line"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "f9d7e63c",
   "metadata": {},
   "source": [
    "## Checks\n",
    "\n",
    "Completeness, gaps, records out of sequence, bad quality indicators, unreasonable speeds and accelerations, and the expected area, against the config's `[thresholds]`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "28164a24",
   "metadata": {},
   "outputs": [],
   "source": [
    "display(pd.DataFrame([vars(c) for c in result.stats.checks]))"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "a8b3b52f",
   "metadata": {},
   "source": [
    "### Checks by file\n",
    "\n",
    "What each check counted, per nav file. A speed, a step out of sequence or a gap is counted in the file of the fix that ends it."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "8d157ebd",
   "metadata": {},
   "outputs": [],
   "source": [
    "display(stats.counts_by_file(result.nav, result.stats))\n",
    "# The sentences the reader couldn't use, and why\n",
    "display(stats.skipped_by_file(result.nav))"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "68c9fa11",
   "metadata": {},
   "source": [
    "### Gaps\n",
    "\n",
    "The steps of `max_gap` or longer between fixes, the longest first, with the files either side."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "66b17fe6",
   "metadata": {},
   "outputs": [],
   "source": [
    "gaps = stats.gaps(result.nav, result.qc, result.config.thresholds)\n",
    "# The length in seconds, and as hh:mm:ss\n",
    "gaps.insert(3, \"hh:mm:ss\", gaps[\"length\"].map(q.SAMPLING_INTERVAL.format_duration))\n",
    "display(gaps)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "39eefa97",
   "metadata": {},
   "source": [
    "## Statistics"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "a72c9a9b",
   "metadata": {},
   "outputs": [],
   "source": [
    "s = result.stats\n",
    "ranges = {\n",
    "    q.NSV: (\"nsv\", s.nsv, None),\n",
    "    q.HDOP: (\"hdop\", s.hdop, result.nav.decimals.get(\"hdop\")),\n",
    "    q.ANTENNA_HEIGHT: (\"altitude\", s.altitude, result.nav.decimals.get(\"alt\")),\n",
    "    q.SPEED_MADE_GOOD: (\"speed\", s.speed, None),\n",
    "    q.HORIZONTAL_ACCELERATION: (\"acceleration\", s.acceleration, None),\n",
    "}\n",
    "# Over every record, as R2R reports them, and over the fixes that pass QC\n",
    "display(\n",
    "    pd.DataFrame(\n",
    "        [\n",
    "            {\n",
    "                \"quantity\": quantity.label,\n",
    "                \"minimum\": quantity.format(r.minimum, places),\n",
    "                \"maximum\": quantity.format(r.maximum, places),\n",
    "                \"good minimum\": quantity.format(getattr(s.good, name).minimum, places),\n",
    "                \"good maximum\": quantity.format(getattr(s.good, name).maximum, places),\n",
    "            }\n",
    "            for quantity, (name, r, places) in ranges.items()\n",
    "        ],\n",
    "    )\n",
    ")\n",
    "print(q.FIXES.label, q.FIXES.format(s.records))\n",
    "print(q.FIXES.label, \"that pass QC\", q.FIXES.format(s.good.records))\n",
    "print(q.MODAL_INTERVAL.label, q.MODAL_INTERVAL.format(s.modal_interval))\n",
    "print(q.LONGEST_GAP.clock_label, q.LONGEST_GAP.format_duration(s.longest_gap))"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "678dd53b",
   "metadata": {},
   "source": [
    "## Quality control\n",
    "\n",
    "Records flagged by each flag (a record can have more than one), QC's notes, and the reader's and manifest's errors."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "252f0e79",
   "metadata": {},
   "outputs": [],
   "source": [
    "display(\n",
    "    pd.DataFrame(\n",
    "        [\n",
    "            {\"flag\": f.name, \"meaning\": DESCRIPTIONS[f], \"records\": result.qc.count(f)}\n",
    "            for f in Flag\n",
    "            if result.qc.count(f)\n",
    "        ],\n",
    "    )\n",
    ")\n",
    "for note in result.qc.notes:\n",
    "    print(note)\n",
    "for error in result.errors:\n",
    "    print(\"error:\", error)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "4103ab3e",
   "metadata": {},
   "source": [
    "## Figures"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "63adf2d1",
   "metadata": {},
   "outputs": [],
   "source": [
    "for make in plots.FIGURES.values():\n",
    "    figure = make(result, theme)\n",
    "    display(figure)\n",
    "    plt.close(figure)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "91da9c45",
   "metadata": {},
   "source": [
    "## Products\n",
    "\n",
    "bestres is every record, with its QC flags (0 for a good fix) and speed and course over ground; 1min is the good fix nearest each whole minute; control is Douglas–Peucker on 1min."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "fd56a757",
   "metadata": {},
   "outputs": [],
   "source": [
    "display(result.bestres.head())\n",
    "display(result.one_minute.head())\n",
    "display(result.control)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "4cf3188a",
   "metadata": {},
   "source": [
    "## Write the reports"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "721732c1",
   "metadata": {},
   "outputs": [],
   "source": [
    "written = reports.write_reports(\n",
    "    result, OUTPUT_DIR, theme_file=THEME_FILE, notebook=False\n",
    ")\n",
    "for path in written:\n",
    "    print(path)"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "name": "python"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}