Metabolomics drift correction

Metabolomics drift correction#

Within-batch correction of metabolomics data. This notebook shows how to correct for instrumental drift based on pooled QC data samples.

1. LOESS smoothing-based drift correction#

First, let’s load our packages, including acore.

%pip install acore

Hide code cell output

 
Requirement already satisfied: acore in /home/docs/checkouts/readthedocs.org/user_builds/analytics-core/envs/stable/lib/python3.11/site-packages (0.3.0)
Requirement already satisfied: numpy in /home/docs/checkouts/readthedocs.org/user_builds/analytics-core/envs/stable/lib/python3.11/site-packages (from acore) (2.4.6)
Requirement already satisfied: pandas in /home/docs/checkouts/readthedocs.org/user_builds/analytics-core/envs/stable/lib/python3.11/site-packages (from acore) (2.3.3)
Requirement already satisfied: dsp-pandas in /home/docs/checkouts/readthedocs.org/user_builds/analytics-core/envs/stable/lib/python3.11/site-packages (from acore) (0.0.5)
Requirement already satisfied: scipy in /home/docs/checkouts/readthedocs.org/user_builds/analytics-core/envs/stable/lib/python3.11/site-packages (from acore) (1.17.1)
Requirement already satisfied: scikit-learn>=1.5 in /home/docs/checkouts/readthedocs.org/user_builds/analytics-core/envs/stable/lib/python3.11/site-packages (from acore) (1.9.0)
Requirement already satisfied: gseapy!=1.1.5 in /home/docs/checkouts/readthedocs.org/user_builds/analytics-core/envs/stable/lib/python3.11/site-packages (from acore) (1.2.1)
Requirement already satisfied: pingouin<0.6.0 in /home/docs/checkouts/readthedocs.org/user_builds/analytics-core/envs/stable/lib/python3.11/site-packages (from acore) (0.5.5)
Requirement already satisfied: umap-learn in /home/docs/checkouts/readthedocs.org/user_builds/analytics-core/envs/stable/lib/python3.11/site-packages (from acore) (0.5.12)
Requirement already satisfied: statsmodels in /home/docs/checkouts/readthedocs.org/user_builds/analytics-core/envs/stable/lib/python3.11/site-packages (from acore) (0.14.6)
Requirement already satisfied: combat in /home/docs/checkouts/readthedocs.org/user_builds/analytics-core/envs/stable/lib/python3.11/site-packages (from acore) (0.3.3)
Requirement already satisfied: inmoose in /home/docs/checkouts/readthedocs.org/user_builds/analytics-core/envs/stable/lib/python3.11/site-packages (from acore) (0.9.1)
Requirement already satisfied: requests in /home/docs/checkouts/readthedocs.org/user_builds/analytics-core/envs/stable/lib/python3.11/site-packages (from acore) (2.34.2)
Requirement already satisfied: rarfile in /home/docs/checkouts/readthedocs.org/user_builds/analytics-core/envs/stable/lib/python3.11/site-packages (from acore) (4.2)
Requirement already satisfied: pandera in /home/docs/checkouts/readthedocs.org/user_builds/analytics-core/envs/stable/lib/python3.11/site-packages (from acore) (0.31.1)
Requirement already satisfied: matplotlib in /home/docs/checkouts/readthedocs.org/user_builds/analytics-core/envs/stable/lib/python3.11/site-packages (from pingouin<0.6.0->acore) (3.11.0)
Requirement already satisfied: pandas-flavor in /home/docs/checkouts/readthedocs.org/user_builds/analytics-core/envs/stable/lib/python3.11/site-packages (from pingouin<0.6.0->acore) (0.8.1)
Requirement already satisfied: seaborn in /home/docs/checkouts/readthedocs.org/user_builds/analytics-core/envs/stable/lib/python3.11/site-packages (from pingouin<0.6.0->acore) (0.13.2)
Requirement already satisfied: tabulate in /home/docs/checkouts/readthedocs.org/user_builds/analytics-core/envs/stable/lib/python3.11/site-packages (from pingouin<0.6.0->acore) (0.10.0)
Requirement already satisfied: contourpy>=1.0.1 in /home/docs/checkouts/readthedocs.org/user_builds/analytics-core/envs/stable/lib/python3.11/site-packages (from matplotlib->pingouin<0.6.0->acore) (1.3.3)
Requirement already satisfied: cycler>=0.10 in /home/docs/checkouts/readthedocs.org/user_builds/analytics-core/envs/stable/lib/python3.11/site-packages (from matplotlib->pingouin<0.6.0->acore) (0.12.1)
Requirement already satisfied: fonttools>=4.22.0 in /home/docs/checkouts/readthedocs.org/user_builds/analytics-core/envs/stable/lib/python3.11/site-packages (from matplotlib->pingouin<0.6.0->acore) (4.63.0)
Requirement already satisfied: kiwisolver>=1.3.1 in /home/docs/checkouts/readthedocs.org/user_builds/analytics-core/envs/stable/lib/python3.11/site-packages (from matplotlib->pingouin<0.6.0->acore) (1.5.0)
Requirement already satisfied: packaging>=20.0 in /home/docs/checkouts/readthedocs.org/user_builds/analytics-core/envs/stable/lib/python3.11/site-packages (from matplotlib->pingouin<0.6.0->acore) (26.2)
Requirement already satisfied: pillow>=9 in /home/docs/checkouts/readthedocs.org/user_builds/analytics-core/envs/stable/lib/python3.11/site-packages (from matplotlib->pingouin<0.6.0->acore) (12.2.0)
Requirement already satisfied: pyparsing>=3 in /home/docs/checkouts/readthedocs.org/user_builds/analytics-core/envs/stable/lib/python3.11/site-packages (from matplotlib->pingouin<0.6.0->acore) (3.3.2)
Requirement already satisfied: python-dateutil>=2.7 in /home/docs/checkouts/readthedocs.org/user_builds/analytics-core/envs/stable/lib/python3.11/site-packages (from matplotlib->pingouin<0.6.0->acore) (2.9.0.post0)
Requirement already satisfied: pytz>=2020.1 in /home/docs/checkouts/readthedocs.org/user_builds/analytics-core/envs/stable/lib/python3.11/site-packages (from pandas->acore) (2026.2)
Requirement already satisfied: tzdata>=2022.7 in /home/docs/checkouts/readthedocs.org/user_builds/analytics-core/envs/stable/lib/python3.11/site-packages (from pandas->acore) (2026.2)
Requirement already satisfied: six>=1.5 in /home/docs/checkouts/readthedocs.org/user_builds/analytics-core/envs/stable/lib/python3.11/site-packages (from python-dateutil>=2.7->matplotlib->pingouin<0.6.0->acore) (1.17.0)
Requirement already satisfied: joblib>=1.4.0 in /home/docs/checkouts/readthedocs.org/user_builds/analytics-core/envs/stable/lib/python3.11/site-packages (from scikit-learn>=1.5->acore) (1.5.3)
Requirement already satisfied: narwhals>=2.0.1 in /home/docs/checkouts/readthedocs.org/user_builds/analytics-core/envs/stable/lib/python3.11/site-packages (from scikit-learn>=1.5->acore) (2.22.1)
Requirement already satisfied: threadpoolctl>=3.5.0 in /home/docs/checkouts/readthedocs.org/user_builds/analytics-core/envs/stable/lib/python3.11/site-packages (from scikit-learn>=1.5->acore) (3.6.0)
Requirement already satisfied: mpmath>=1.1.0 in /home/docs/checkouts/readthedocs.org/user_builds/analytics-core/envs/stable/lib/python3.11/site-packages (from combat->acore) (1.4.1)
Requirement already satisfied: patsy>=0.5.1 in /home/docs/checkouts/readthedocs.org/user_builds/analytics-core/envs/stable/lib/python3.11/site-packages (from combat->acore) (1.0.2)
Requirement already satisfied: openpyxl in /home/docs/checkouts/readthedocs.org/user_builds/analytics-core/envs/stable/lib/python3.11/site-packages (from dsp-pandas->acore) (3.1.5)
Requirement already satisfied: anndata in /home/docs/checkouts/readthedocs.org/user_builds/analytics-core/envs/stable/lib/python3.11/site-packages (from inmoose->acore) (0.12.17)
Requirement already satisfied: fastcluster in /home/docs/checkouts/readthedocs.org/user_builds/analytics-core/envs/stable/lib/python3.11/site-packages (from inmoose->acore) (1.3.0)
Requirement already satisfied: jinja2 in /home/docs/checkouts/readthedocs.org/user_builds/analytics-core/envs/stable/lib/python3.11/site-packages (from inmoose->acore) (3.1.6)
Requirement already satisfied: array-api-compat>=1.7.1 in /home/docs/checkouts/readthedocs.org/user_builds/analytics-core/envs/stable/lib/python3.11/site-packages (from anndata->inmoose->acore) (1.15.0)
Requirement already satisfied: h5py>=3.8 in /home/docs/checkouts/readthedocs.org/user_builds/analytics-core/envs/stable/lib/python3.11/site-packages (from anndata->inmoose->acore) (3.16.0)
Requirement already satisfied: legacy-api-wrap in /home/docs/checkouts/readthedocs.org/user_builds/analytics-core/envs/stable/lib/python3.11/site-packages (from anndata->inmoose->acore) (1.5)
Requirement already satisfied: natsort in /home/docs/checkouts/readthedocs.org/user_builds/analytics-core/envs/stable/lib/python3.11/site-packages (from anndata->inmoose->acore) (8.4.0)
Requirement already satisfied: scverse-misc>=0.0.3 in /home/docs/checkouts/readthedocs.org/user_builds/analytics-core/envs/stable/lib/python3.11/site-packages (from anndata->inmoose->acore) (0.0.3)
Requirement already satisfied: zarr!=3.0.*,>=2.18.7 in /home/docs/checkouts/readthedocs.org/user_builds/analytics-core/envs/stable/lib/python3.11/site-packages (from anndata->inmoose->acore) (3.1.6)
Requirement already satisfied: session-info2 in /home/docs/checkouts/readthedocs.org/user_builds/analytics-core/envs/stable/lib/python3.11/site-packages (from scverse-misc>=0.0.3->anndata->inmoose->acore) (0.4.1)
Requirement already satisfied: typing-extensions in /home/docs/checkouts/readthedocs.org/user_builds/analytics-core/envs/stable/lib/python3.11/site-packages (from scverse-misc>=0.0.3->anndata->inmoose->acore) (4.15.0)
Requirement already satisfied: donfig>=0.8 in /home/docs/checkouts/readthedocs.org/user_builds/analytics-core/envs/stable/lib/python3.11/site-packages (from zarr!=3.0.*,>=2.18.7->anndata->inmoose->acore) (0.8.1.post1)
Requirement already satisfied: google-crc32c>=1.5 in /home/docs/checkouts/readthedocs.org/user_builds/analytics-core/envs/stable/lib/python3.11/site-packages (from zarr!=3.0.*,>=2.18.7->anndata->inmoose->acore) (1.8.0)
Requirement already satisfied: numcodecs>=0.14 in /home/docs/checkouts/readthedocs.org/user_builds/analytics-core/envs/stable/lib/python3.11/site-packages (from zarr!=3.0.*,>=2.18.7->anndata->inmoose->acore) (0.16.5)
Requirement already satisfied: pyyaml in /home/docs/checkouts/readthedocs.org/user_builds/analytics-core/envs/stable/lib/python3.11/site-packages (from donfig>=0.8->zarr!=3.0.*,>=2.18.7->anndata->inmoose->acore) (6.0.3)
Requirement already satisfied: MarkupSafe>=2.0 in /home/docs/checkouts/readthedocs.org/user_builds/analytics-core/envs/stable/lib/python3.11/site-packages (from jinja2->inmoose->acore) (3.0.3)
Requirement already satisfied: et-xmlfile in /home/docs/checkouts/readthedocs.org/user_builds/analytics-core/envs/stable/lib/python3.11/site-packages (from openpyxl->dsp-pandas->acore) (2.0.0)
Requirement already satisfied: xarray in /home/docs/checkouts/readthedocs.org/user_builds/analytics-core/envs/stable/lib/python3.11/site-packages (from pandas-flavor->pingouin<0.6.0->acore) (2026.4.0)
Requirement already satisfied: pydantic in /home/docs/checkouts/readthedocs.org/user_builds/analytics-core/envs/stable/lib/python3.11/site-packages (from pandera->acore) (2.13.4)
Requirement already satisfied: typeguard in /home/docs/checkouts/readthedocs.org/user_builds/analytics-core/envs/stable/lib/python3.11/site-packages (from pandera->acore) (4.5.2)
Requirement already satisfied: typing_inspect>=0.6.0 in /home/docs/checkouts/readthedocs.org/user_builds/analytics-core/envs/stable/lib/python3.11/site-packages (from pandera->acore) (0.9.0)
Requirement already satisfied: mypy-extensions>=0.3.0 in /home/docs/checkouts/readthedocs.org/user_builds/analytics-core/envs/stable/lib/python3.11/site-packages (from typing_inspect>=0.6.0->pandera->acore) (1.1.0)
Requirement already satisfied: annotated-types>=0.6.0 in /home/docs/checkouts/readthedocs.org/user_builds/analytics-core/envs/stable/lib/python3.11/site-packages (from pydantic->pandera->acore) (0.7.0)
Requirement already satisfied: pydantic-core==2.46.4 in /home/docs/checkouts/readthedocs.org/user_builds/analytics-core/envs/stable/lib/python3.11/site-packages (from pydantic->pandera->acore) (2.46.4)
Requirement already satisfied: typing-inspection>=0.4.2 in /home/docs/checkouts/readthedocs.org/user_builds/analytics-core/envs/stable/lib/python3.11/site-packages (from pydantic->pandera->acore) (0.4.2)
Requirement already satisfied: charset_normalizer<4,>=2 in /home/docs/checkouts/readthedocs.org/user_builds/analytics-core/envs/stable/lib/python3.11/site-packages (from requests->acore) (3.4.7)
Requirement already satisfied: idna<4,>=2.5 in /home/docs/checkouts/readthedocs.org/user_builds/analytics-core/envs/stable/lib/python3.11/site-packages (from requests->acore) (3.18)
Requirement already satisfied: urllib3<3,>=1.26 in /home/docs/checkouts/readthedocs.org/user_builds/analytics-core/envs/stable/lib/python3.11/site-packages (from requests->acore) (2.7.0)
Requirement already satisfied: certifi>=2023.5.7 in /home/docs/checkouts/readthedocs.org/user_builds/analytics-core/envs/stable/lib/python3.11/site-packages (from requests->acore) (2026.5.20)
Requirement already satisfied: numba>=0.51.2 in /home/docs/checkouts/readthedocs.org/user_builds/analytics-core/envs/stable/lib/python3.11/site-packages (from umap-learn->acore) (0.65.1)
Requirement already satisfied: pynndescent>=0.5 in /home/docs/checkouts/readthedocs.org/user_builds/analytics-core/envs/stable/lib/python3.11/site-packages (from umap-learn->acore) (0.6.0)
Requirement already satisfied: tqdm in /home/docs/checkouts/readthedocs.org/user_builds/analytics-core/envs/stable/lib/python3.11/site-packages (from umap-learn->acore) (4.68.2)
Requirement already satisfied: llvmlite<0.48,>=0.47.0dev0 in /home/docs/checkouts/readthedocs.org/user_builds/analytics-core/envs/stable/lib/python3.11/site-packages (from numba>=0.51.2->umap-learn->acore) (0.47.0)
Note: you may need to restart the kernel to use updated packages.

import matplotlib.pyplot as plt
import numpy as np
import pandas as pd
from sklearn.decomposition import PCA
from sklearn.preprocessing import StandardScaler

from acore import drift_correction as dc

Hide code cell source

 
def plot_loess_example_curve(
    df: pd.DataFrame,
    feature_idx: int,
    samples: list,
    qcs: list,
    sample_order: pd.DataFrame,
    show_corrected: bool = True,
    alpha: float = None,  # fixed smoothing span; if None, selected by LOOCV
):
    """
    Plot the raw intensities, LOESS drift curve, and optionally the corrected
    intensities for a single feature according to the loess drift correction function.
    Useful for inspecting drift behaviour before running full drift correction.

    The drift curve is estimated with the same method used by
    ldc.run_drift_correction: LOESS is fit to the QC points and then
    interpolated across all injection positions via a cubic spline
    (qc_rlsc_loess).

    Parameters
    ----------
    df : pd.DataFrame
        Feature matrix with samples as rows and features as columns.
        Metadata columns should have been removed already.
    feature_idx : int
        Column index of the feature to plot.
    samples : list of str
        Row index labels of the biological samples.
    qcs : list of str
        Row index labels of the pooled QC samples.
    sample_order : pd.DataFrame
        Injection-order table with columns `File Name` and `Sample ID`
        (integer run order).
    show_corrected : bool, optional
        If True (default), overlays drift-corrected sample intensities as
        diamond markers.
    alpha : float, optional
        LOESS smoothing span (0 < α ≤ 1). If None (default), the optimal span
        is selected automatically by leave-one-out cross-validation over
        α ∈ [0.40, 1.00]. The selected value is shown in the legend.
    """

    all_rows = samples + qcs
    feature_col = df.iloc[:, feature_idx]

    order_dict = sample_order.set_index("File Name")["Sample ID"].to_dict()
    x_all = np.array([order_dict.get(r, np.nan) for r in all_rows])
    y_all = feature_col.loc[all_rows].astype(float).values

    n_s = len(samples)
    x_sample, y_sample = x_all[:n_s], y_all[:n_s]
    x_qc_arr, y_qc_arr = x_all[n_s:], y_all[n_s:]

    valid_sample = ~np.isnan(x_sample) & ~np.isnan(y_sample)
    valid_qc = ~np.isnan(x_qc_arr) & ~np.isnan(y_qc_arr)
    x_s_v, y_s_v = x_sample[valid_sample], y_sample[valid_sample]
    x_qc_v, y_qc_v = x_qc_arr[valid_qc], y_qc_arr[valid_qc]

    feature_name = df.columns[feature_idx]

    fig, ax = plt.subplots(figsize=(11, 5))
    ax.scatter(x_s_v, y_s_v, label="Samples", color="steelblue", alpha=0.7, zorder=3)
    ax.scatter(
        x_qc_v, y_qc_v, label="QC", color="firebrick", edgecolor="k", s=60, zorder=4
    )

    if len(x_qc_v) >= 4:
        if alpha is not None:
            # Pass a single-element candidate list to skip LOOCV and use the given alpha
            # directly
            drift_curve, best_alpha = dc.qc_rlsc_loess(
                x_qc_v, y_qc_v, x_all, always_use_default=True, default=alpha
            )
        else:
            drift_curve, best_alpha = dc.qc_rlsc_loess(x_qc_v, y_qc_v, x_all)

        valid_curve = ~np.isnan(x_all) & ~np.isnan(drift_curve)
        sort_idx = np.argsort(x_all[valid_curve])
        ax.plot(
            x_all[valid_curve][sort_idx],
            drift_curve[valid_curve][sort_idx],
            label=f"LOESS drift curve (α={best_alpha:.2f})",
            color="black",
            lw=2,
            zorder=5,
        )

        if show_corrected:
            median_qc = np.median(y_qc_v)
            drift_at_samples = drift_curve[:n_s][valid_sample]
            corrected = (y_s_v / drift_at_samples) * median_qc
            ax.scatter(
                x_s_v,
                corrected,
                label="Corrected samples",
                color="lightsteelblue",
                marker="D",
                s=40,
                alpha=0.9,
                zorder=3,
            )
    else:
        print(f"Not enough valid QC points for LOESS ({len(x_qc_v)} found, need ≥4).")

    ax.set_xlabel("Injection Order")
    ax.set_ylabel("Intensity")
    ax.set_title(f"Drift correction example ({feature_name})")
    ax.legend()
    ax.grid(True, alpha=0.3)
    plt.tight_layout()
    plt.show()


def pca_for_cpca_drift(
    df: pd.DataFrame,
    samples,  # list of row index labels, OR dict {group_name: [row index labels]}
    qcs: list,
    log_transform: bool = True,
    title: str = "PCA",
):
    """
    PCA of samples and QC samples.

    Parameters
    ----------
    df      : feature matrix with samples as rows and features as columns.
              Metadata columns should have been removed already.
    samples : list of row index labels of the biological samples (all one group),
              or dict {group_name: [row index labels]} for multiple groups
    qcs     : list of row index labels of the pooled QC samples
    log_transform : apply log1p before scaling
    title   : plot title
    """
    # Normalise samples to a dict
    if isinstance(samples, list):
        sample_groups = {"Samples": samples}
    else:
        sample_groups = samples

    # Build ordered row + label arrays
    all_rows, labels = [], []
    for group, rows in sample_groups.items():
        for r in rows:
            if r in df.index:
                all_rows.append(r)
                labels.append(group)
    for r in qcs:
        if r in df.index:
            all_rows.append(r)
            labels.append("QC")

    # Coerce to numeric, drop features (columns) with any NaN
    X = df.loc[all_rows].apply(pd.to_numeric, errors="coerce").dropna(axis=1)

    if log_transform:
        X = np.log1p(X.clip(lower=0))

    X_scaled = StandardScaler().fit_transform(X.values.astype(float))

    pca = PCA(n_components=2)
    coords = pca.fit_transform(X_scaled)
    pc1_var = pca.explained_variance_ratio_[0] * 100
    pc2_var = pca.explained_variance_ratio_[1] * 100

    palette = plt.cm.tab10.colors
    color_map = {g: palette[i % len(palette)] for i, g in enumerate(sample_groups)}
    color_map["QC"] = "black"

    fig, ax = plt.subplots(figsize=(8, 6))
    for group in list(sample_groups) + ["QC"]:
        mask = [_label == group for _label in labels]
        pts = coords[mask]
        is_qc = group == "QC"
        ax.scatter(
            pts[:, 0],
            pts[:, 1],
            label=group,
            color=color_map[group],
            marker="D" if is_qc else "o",
            s=70 if is_qc else 50,
            edgecolors="k" if is_qc else "none",
            alpha=0.9 if is_qc else 0.75,
            zorder=5 if is_qc else 3,
        )

    ax.set_xlabel(f"PC1 ({pc1_var:.1f}%)")
    ax.set_ylabel(f"PC2 ({pc2_var:.1f}%)")
    ax.set_title(title)
    ax.axhline(0, color="grey", lw=0.5, ls="--")
    ax.axvline(0, color="grey", lw=0.5, ls="--")
    ax.legend(framealpha=0.8)
    ax.grid(True, alpha=0.2)
    plt.tight_layout()
    plt.show()

Load in data#

We will use some example data that can be found in this repo.

data_path = (
    "https://raw.githubusercontent.com/Multiomics-Analytics-Group/acore/"
    "refs/heads/main/"
)

df_original = pd.read_csv(
    "../../example_data/DidacMauricio_hilic/DM_FIS2018_Hilic_pos_results2023_filled_imputed.csv",
    index_col=0,
)

This is what the data frame, an intensity table, looks like.

df_original
Qidx SOIidx rtmed start end mass MaxInt formula anot AAA9485207 ... QC_35 QC_36 QC_37 QC_38 QC_39 QC_40 QC_41 QC_42 QC_43 QC_44
0 1 1 60.544 45.000 72.500 81.070 160,840.953 [C6H9]+ C6H8_M+H 279,488.531 ... 142,100.734 129,631.148 110,443.422 157,871.062 535,311.375 334,133.656 339,973.344 317,506.188 294,083.344 234,717.266
1 2 4 172.568 161.439 191.635 82.053 98,153.547 [C4H6N2]+ C4H5N2_M+H 73,170.344 ... 72,917.398 73,738.812 65,597.875 78,216.859 70,257.375 73,489.242 60,233.695 70,798.312 69,802.516 74,212.203
2 3 6 143.225 116.953 165.260 82.053 134,492.109 [C4H6N2]+ C4H5N2_M+H 106,222.969 ... 117,823.602 122,279.500 120,513.508 119,803.422 114,791.906 124,753.789 128,157.016 115,411.750 133,331.281 124,152.578
3 4 7 330.747 313.125 373.976 82.065 67,051.617 [C5H8N]+ C5H7N_M+H 40,187.371 ... 58,493.379 55,851.680 58,560.121 57,886.605 58,293.699 46,211.445 62,802.289 57,658.062 54,058.363 54,484.602
4 5 7 343.980 313.125 373.976 82.065 67,051.617 [C5H8N]+ C5H7N_M+H 16,231.437 ... 25,015.951 21,309.277 20,180.580 19,609.604 25,462.301 24,354.287 30,869.357 17,454.047 22,235.070 18,160.814
... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ...
2,539 2,540 2,363 290.896 290.000 312.500 892.654 3,486,662.000 [C48H94NO11S]+ C48H93NO11S_M+H 6,403,619.000 ... 4,217,977.000 3,872,732.000 3,356,819.000 3,444,360.000 3,641,683.000 4,531,348.000 4,375,519.000 3,864,418.000 3,730,628.000 5,249,600.000
2,540 2,541 2,363 299.617 290.000 312.500 892.654 3,486,662.000 [C48H94NO11S]+ C48H93NO11S_M+H 137,733.500 ... 66,690.800 53,758.060 100,864.900 88,069.460 94,612.190 87,438.410 78,174.880 61,854.690 92,978.760 69,441.320
2,541 2,542 2,364 54.789 50.000 67.500 892.739 179,925.600 [C57H98NO6]+ C57H94O6_M+NH4 130,200.000 ... 190,497.800 173,043.300 57,476.910 174,392.000 47,625.360 135,338.900 154,609.500 179,258.000 171,145.200 161,171.100
2,542 2,543 2,365 54.782 50.000 67.500 894.755 488,985.200 [C57H100NO6]+ C57H96O6_M+NH4 468,793.800 ... 523,346.000 453,992.800 133,935.100 509,662.500 114,995.100 381,900.100 421,614.900 503,656.400 439,513.500 434,035.700
2,544 2,545 2,367 54.821 50.000 67.500 896.770 843,227.000 [C57H102NO6]+ C57H98O6_M+NH4 821,122.200 ... 998,584.700 897,026.200 235,440.700 809,741.600 178,420.900 678,957.200 785,135.600 895,259.400 788,261.100 768,748.800

2287 rows × 486 columns

In order to run our further analysis, including the filtering functions, we have to transform the data and remove metadata such as mass and retention time.

df = df_original.T
df = df.drop(
    ["Qidx", "SOIidx", "rtmed", "start", "end", "mass", "MaxInt", "formula", "anot"]
)

Hide code cell source

 
df
0 1 2 3 4 5 6 7 8 9 ... 2,534 2,535 2,536 2,537 2,538 2,539 2,540 2,541 2,542 2,544
AAA9485207 279,488.531 73,170.344 106,222.969 40,187.371 16,231.437 211,807.094 319,754.562 112,320.398 46,083.371 48,803.125 ... 82,146.870 80,973.820 46,157.050 49,622.580 231,180.200 6,403,619.000 137,733.500 130,200.000 468,793.800 821,122.200
AAA9485216 247,458.016 86,581.648 132,690.734 82,426.359 24,345.967 12,622.342 389,471.938 84,265.992 73,903.742 43,815.148 ... 148,114.400 134,861.800 90,832.130 72,869.770 240,460.700 4,852,053.000 59,179.240 132,118.200 513,293.500 1,214,919.000
AAA9485239 99,304.359 93,201.195 152,236.844 74,535.336 35,357.852 7,571.239 417,576.844 199,175.516 68,742.586 44,511.543 ... 95,990.200 85,438.980 63,371.030 49,218.960 310,655.100 2,619,595.000 72,289.910 160,829.900 518,888.200 1,092,635.000
AAA9485258 119,563.797 72,692.320 113,827.773 51,309.215 20,640.715 259,447.391 340,227.594 271,096.281 41,593.598 61,431.602 ... 83,902.570 64,054.850 69,871.040 51,861.310 184,134.600 2,601,840.000 70,717.240 83,523.680 252,012.400 658,375.000
AAA9485261 191,762.188 64,645.020 115,821.445 60,884.336 18,506.797 235,303.953 320,328.281 174,622.797 49,389.219 41,346.922 ... 221,212.200 191,401.000 114,394.600 98,023.710 359,151.000 2,767,868.000 150,113.300 143,107.200 463,635.800 1,099,109.000
... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ...
QC_40 334,133.656 73,489.242 124,753.789 46,211.445 24,354.287 246,895.516 399,774.781 165,567.125 135,517.156 52,733.680 ... 81,523.700 77,415.330 52,939.590 40,989.100 141,548.000 4,531,348.000 87,438.410 135,338.900 381,900.100 678,957.200
QC_41 339,973.344 60,233.695 128,157.016 62,802.289 30,869.357 254,287.641 407,853.812 151,410.297 134,795.859 57,720.047 ... 62,895.500 70,159.380 89,829.210 46,564.210 172,408.800 4,375,519.000 78,174.880 154,609.500 421,614.900 785,135.600
QC_42 317,506.188 70,798.312 115,411.750 57,658.062 17,454.047 264,687.438 407,221.000 150,344.312 130,498.227 50,533.473 ... 73,050.480 85,322.640 49,600.440 44,505.460 161,372.500 3,864,418.000 61,854.690 179,258.000 503,656.400 895,259.400
QC_43 294,083.344 69,802.516 133,331.281 54,058.363 22,235.070 269,192.375 394,239.344 160,134.375 124,771.242 48,362.730 ... 71,588.570 78,372.000 54,991.750 53,880.830 145,470.600 3,730,628.000 92,978.760 171,145.200 439,513.500 788,261.100
QC_44 234,717.266 74,212.203 124,152.578 54,484.602 18,160.814 261,814.906 412,434.000 158,760.438 126,280.156 46,164.520 ... 78,520.450 82,817.930 55,926.430 50,953.160 160,843.600 5,249,600.000 69,441.320 161,171.100 434,035.700 768,748.800

477 rows × 2287 columns

We also have data that contains the order in which our samples were run. This information is crucial for the drift correction algorithm; it cannot be performed without it.

The data needs to contain the columns File Name and Sample ID, referring to the name of the sample and the index of the sample, meaning the order in which the samples were run.

Note: If you do not have metadata that contains the order in which samples and QCs were run, you can use the second method explained in this notebook, CPCA.

sample_order = pd.read_csv(
    "../../example_data/DidacMauricio_hilic/DidacMauricio_hilic_pos_injectionorder.csv"
)
sample_order["File Name"] = sample_order["SampleName"]
sample_order["Sample ID"] = sample_order["injectionOrder"]

sample_order
SampleName measureTime injectionOrder File Name Sample ID
0 Bi_1 2021-07-19 13:51:34 1 Bi_1 1
1 Bi_2 2021-07-19 14:05:49 2 Bi_2 2
2 QC_00 2021-07-19 14:20:04 3 QC_00 3
3 QC_01 2021-07-19 14:34:20 4 QC_01 4
4 QC_02 2021-07-19 14:48:36 5 QC_02 5
... ... ... ... ... ...
472 AAA9516105 2021-07-24 06:04:54 473 AAA9516105 473
473 QC_43 2021-07-24 06:19:11 474 QC_43 474
474 QC_44 2021-07-24 06:33:27 475 QC_44 475
475 Bf_1 2021-07-24 06:47:43 476 Bf_1 476
476 Bf_2 2021-07-24 07:01:57 477 Bf_2 477

477 rows × 5 columns

Run drift correction with LOESS smoothing#

We can now correct our data for experimental drift.

With the acore LOESS drift-correction function, a LOESS (locally estimated regression) smoother is applied separately to the features in the data to model slow temporal trends, and the resulting smooth trend is used to correct the data.

Before the function estimation and correction, the data can filtered, to remove features that have too many missing values in the QC samples.

First, we can create a dictionary for our sample names, ordering them into groups, to make the upcoming function call easier.

column_list = list(df_original.columns.values)
sample_cols = []
qc_cols = []
for col in column_list:
    if col.startswith("AAA"):
        sample_cols.append(col)
    elif col.startswith("QC"):
        qc_cols.append(col)

Now, are sample column names are summarised in the sample_cols variable:

sample_cols

Hide code cell output

 
['AAA9485207',
 'AAA9485216',
 'AAA9485239',
 'AAA9485258',
 'AAA9485261',
 'AAA9485274',
 'AAA9485282',
 'AAA9485295',
 'AAA9485915',
 'AAA9485917',
 'AAA9485919',
 'AAA9485921',
 'AAA9485950',
 'AAA9485953',
 'AAA9485954',
 'AAA9485967',
 'AAA9485968',
 'AAA9485969',
 'AAA9485972',
 'AAA9485973',
 'AAA9485977',
 'AAA9485981',
 'AAA9485983',
 'AAA9485989',
 'AAA9486247',
 'AAA9486248',
 'AAA9486256',
 'AAA9486260',
 'AAA9486273',
 'AAA9486279',
 'AAA9486282',
 'AAA9486283',
 'AAA9486294',
 'AAA9486295',
 'AAA9487423',
 'AAA9487435',
 'AAA9487460',
 'AAA9487467',
 'AAA9487477',
 'AAA9487603',
 'AAA9487607',
 'AAA9487621',
 'AAA9487624',
 'AAA9487626',
 'AAA9487642',
 'AAA9487667',
 'AAA9487683',
 'AAA9487684',
 'AAA9487690',
 'AAA9487695',
 'AAA9487713',
 'AAA9487717',
 'AAA9487719',
 'AAA9487746',
 'AAA9487751',
 'AAA9487902',
 'AAA9487951',
 'AAA9487963',
 'AAA9487964',
 'AAA9487986',
 'AAA9488111',
 'AAA9488132',
 'AAA9488146',
 'AAA9488149',
 'AAA9488163',
 'AAA9488169',
 'AAA9488179',
 'AAA9488193',
 'AAA9516105',
 'AAA9516121',
 'AAA9516126',
 'AAA9516138',
 'AAA9516143',
 'AAA9516151',
 'AAA9516174',
 'AAA9516184',
 'AAA9516187',
 'AAA9517319',
 'AAA9517325',
 'AAA9517349',
 'AAA9517367',
 'AAA9517384',
 'AAA9517396',
 'AAA9517534',
 'AAA9517553',
 'AAA9517555',
 'AAA9517556',
 'AAA9517582',
 'AAA9517589',
 'AAA9517615',
 'AAA9517622',
 'AAA9517643',
 'AAA9517645',
 'AAA9517646',
 'AAA9517661',
 'AAA9517676',
 'AAA9517677',
 'AAA9517691',
 'AAA9517694',
 'AAA9517901',
 'AAA9517902',
 'AAA9517907',
 'AAA9517921',
 'AAA9517926',
 'AAA9517934',
 'AAA9517973',
 'AAA9517977',
 'AAA9517985',
 'AAA9518304',
 'AAA9518311',
 'AAA9518330',
 'AAA9518335',
 'AAA9518343',
 'AAA9518349',
 'AAA9518353',
 'AAA9518356',
 'AAA9518358',
 'AAA9518362',
 'AAA9518363',
 'AAA9518375',
 'AAA9518380',
 'AAA9518382',
 'AAA9518611',
 'AAA9518631',
 'AAA9518639',
 'AAA9518644',
 'AAA9518668',
 'AAA9518675',
 'AAA9518681',
 'AAA9518690',
 'AAA9518691',
 'AAA9518703',
 'AAA9518710',
 'AAA9518728',
 'AAA9518751',
 'AAA9518752',
 'AAA9518758',
 'AAA9518762',
 'AAA9518767',
 'AAA9518769',
 'AAA9518775',
 'AAA9518781',
 'AAA9518787',
 'AAA9518794',
 'AAA9518907',
 'AAA9518914',
 'AAA9518916',
 'AAA9518921',
 'AAA9518943',
 'AAA9518952',
 'AAA9518955',
 'AAA9518981',
 'AAA9518988',
 'AAA9518991',
 'AAA9519101',
 'AAA9519114',
 'AAA9519118',
 'AAA9519120',
 'AAA9519125',
 'AAA9519136',
 'AAA9519150',
 'AAA9519178',
 'AAA9519311',
 'AAA9519317',
 'AAA9519337',
 'AAA9519344',
 'AAA9519358',
 'AAA9519386',
 'AAA9521917',
 'AAA9521918',
 'AAA9521921',
 'AAA9521922',
 'AAA9521923',
 'AAA9521931',
 'AAA9521934',
 'AAA9521951',
 'AAA9521956',
 'AAA9521960',
 'AAA9521965',
 'AAA9521972',
 'AAA9521977',
 'AAA9521985',
 'AAA9522108',
 'AAA9522113',
 'AAA9522125',
 'AAA9522150',
 'AAA9522156',
 'AAA9522168',
 'AAA9522173',
 'AAA9522182',
 'AAA9522188',
 'AAA9522307',
 'AAA9522309',
 'AAA9522312',
 'AAA9522324',
 'AAA9522327',
 'AAA9522349',
 'AAA9522351',
 'AAA9522361',
 'AAA9522389',
 'AAA9522508',
 'AAA9522509',
 'AAA9522517',
 'AAA9522526',
 'AAA9522537',
 'AAA9522543',
 'AAA9522546',
 'AAA9522550',
 'AAA9522553',
 'AAA9522568',
 'AAA9522590',
 'AAA9522820',
 'AAA9522830',
 'AAA9522834',
 'AAA9522850',
 'AAA9522851',
 'AAA9522861',
 'AAA9522896',
 'AAA9523122',
 'AAA9523129',
 'AAA9523152',
 'AAA9523162',
 'AAA9523164',
 'AAA9523168',
 'AAA9523182',
 'AAA9523184',
 'AAA9523185',
 'AAA9523189',
 'AAA9523191',
 'AAA9523333',
 'AAA9523370',
 'AAA9523375',
 'AAA9523377',
 'AAA9523389',
 'AAA9523504',
 'AAA9523509',
 'AAA9523580',
 'AAA9523701',
 'AAA9523703',
 'AAA9523712',
 'AAA9523724',
 'AAA9523732',
 'AAA9523741',
 'AAA9523752',
 'AAA9523756',
 'AAA9523787',
 'AAA9523794',
 'AAA9523796',
 'AAA9523914',
 'AAA9523922',
 'AAA9523940',
 'AAA9523957',
 'AAA9524119',
 'AAA9524152',
 'AAA9524154',
 'AAA9524174',
 'AAA9524182',
 'AAA9524188',
 'AAA9524194',
 'AAA9524309',
 'AAA9524339',
 'AAA9524365',
 'AAA9524380',
 'AAA9524519',
 'AAA9524537',
 'AAA9524563',
 'AAA9524578',
 'AAA9524579',
 'AAA9524583',
 'AAA9524585',
 'AAA9524588',
 'AAA9524705',
 'AAA9524706',
 'AAA9524746',
 'AAA9524751',
 'AAA9524759',
 'AAA9524762',
 'AAA9524767',
 'AAA9524768',
 'AAA9524775',
 'AAA9524785',
 'AAA9524917',
 'AAA9524931',
 'AAA9524967',
 'AAA9524991',
 'AAA9524993',
 'AAA9525105',
 'AAA9525119',
 'AAA9525120',
 'AAA9525136',
 'AAA9525305',
 'AAA9525315',
 'AAA9525325',
 'AAA9525338',
 'AAA9525373',
 'AAA9525383',
 'AAA9527202',
 'AAA9527218',
 'AAA9527230',
 'AAA9527249',
 'AAA9527250',
 'AAA9527274',
 'AAA9527283',
 'AAA9527284',
 'AAA9527307',
 'AAA9527313',
 'AAA9527321',
 'AAA9527337',
 'AAA9527345',
 'AAA9527346',
 'AAA9527358',
 'AAA9527379',
 'AAA9527396',
 'AAA9527501',
 'AAA9527535',
 'AAA9527543',
 'AAA9527556',
 'AAA9527557',
 'AAA9527575',
 'AAA9527585',
 'AAA9527594',
 'AAA9527720',
 'AAA9527724',
 'AAA9527751',
 'AAA9527760',
 'AAA9527773',
 'AAA9527788',
 'AAA9527793',
 'AAA9527910',
 'AAA9527921',
 'AAA9527937',
 'AAA9527938',
 'AAA9527941',
 'AAA9527956',
 'AAA9528203',
 'AAA9528213',
 'AAA9528219',
 'AAA9528221',
 'AAA9528227',
 'AAA9528251',
 'AAA9528268',
 'AAA9528270',
 'AAA9528272',
 'AAA9528279',
 'AAA9528285',
 'AAA9528291',
 'AAA9528313',
 'AAA9528321',
 'AAA9528328',
 'AAA9528346',
 'AAA9528357',
 'AAA9528371',
 'AAA9528389',
 'AAA9528396',
 'AAA9528503',
 'AAA9528509',
 'AAA9528548',
 'AAA9528559',
 'AAA9528568',
 'AAA9528723',
 'AAA9528728',
 'AAA9528730',
 'AAA9528741',
 'AAA9528750',
 'AAA9528753',
 'AAA9528771',
 'AAA9528917',
 'AAA9528919',
 'AAA9528972',
 'AAA9528973',
 'AAA9528989',
 'AAA9546613',
 'AAA9546627',
 'AAA9546638',
 'AAA9546652',
 'AAA9546655',
 'AAA9546656',
 'AAA9546686',
 'AAA9546709',
 'AAA9546715',
 'AAA9546737',
 'AAA9546740',
 'AAA9546761',
 'AAA9546762',
 'AAA9546770',
 'AAA9546791',
 'AAA9546793',
 'AAA9546930',
 'AAA9546931',
 'AAA9546932',
 'AAA9546943',
 'AAA9546950',
 'AAA9546955',
 'AAA9546974',
 'AAA9547107',
 'AAA9547115',
 'AAA9547118',
 'AAA9547119',
 'AAA9547122',
 'AAA9547163',
 'AAA9547182',
 'AAA9547185',
 'AAA9547187',
 'AAA9547502',
 'AAA9547540',
 'AAA9547541',
 'AAA9547555',
 'AAA9547571',
 'AAA9547589',
 'AAA9547591']

Our columns corresponding to QC data are saved in the qc_cols variable:

qc_cols

Hide code cell output

 
['QC_00',
 'QC_01',
 'QC_02',
 'QC_03',
 'QC_04',
 'QC_05',
 'QC_06',
 'QC_07',
 'QC_08',
 'QC_09',
 'QC_10',
 'QC_11',
 'QC_12',
 'QC_13',
 'QC_14',
 'QC_15',
 'QC_16',
 'QC_17',
 'QC_18',
 'QC_19',
 'QC_20',
 'QC_21',
 'QC_22',
 'QC_23',
 'QC_24',
 'QC_25',
 'QC_26',
 'QC_27',
 'QC_28',
 'QC_29',
 'QC_30',
 'QC_31',
 'QC_32',
 'QC_33',
 'QC_34',
 'QC_35',
 'QC_36',
 'QC_37',
 'QC_38',
 'QC_39',
 'QC_40',
 'QC_41',
 'QC_42',
 'QC_43',
 'QC_44']

Now we can run the drift correction, using the acore run_loess_drift_correction function.

Note: For this demonstration, we will use the parameter always_use_default, so that the leave-one-out cross validation, during which the best smoothing parameter for the curve is determined, is skipped. This is just for efficiency purposes; so that this does not take too long to load. However, it is recommended to not use this option, so that the best parameter can be determined for the curve of every feature.

Hide code cell source

 
help(dc.run_loess_drift_correction)

Help on function run_loess_drift_correction in module acore.drift_correction.loess_drift_correction:

run_loess_drift_correction(data, qc_rows, sample_rows, sample_order: pandas.core.frame.DataFrame, filter_percent: float = None, qc_min_threshold: int = 4, always_use_default=False, default=0.75)
    Perform QC-based drift correction across multiple features using
    LOESS regression and spline interpolation.
    
    For each feature:
    1. Extract QC intensities and corresponding injection order.
    2. Optionally filter features based on QC completeness.
    3. Compute QC relative standard deviation (RSD).
    4. If sufficient QC points exist, estimate a drift curve using
       `qc_rlsc_loess`, finding the best alpha smoothing span
       with leave-one-out cross validation (LOOCV).
    5. Normalize all intensities by dividing by the drift curve and
       scaling to the QC median.
    6. Record drift parameters and correction metadata.
    
    Parameters
    ----------
    data : pandas.DataFrame
        Input intensity matrix with samples as rows and features as columns.
    qc_rows : list of str
        Row indices corresponding to QC injections.
    sample_rows : list of str
        Row indices corresponding to biological samples.
    sample_order : pandas.DataFrame
        Table mapping file names to injection order. Must contain
        columns "File Name" and "Sample ID". Sample ID must be
        numeric.
    filter_percent : float, optional
        Minimum proportion of QC values that must be non-missing for
        a feature to be retained (e.g., 0.6 means at least 60% of QCs
        must be present).
    qc_min_threshold : int, optional
        Minimum number of QC values required to perform drift
        correction. Features with fewer QCs are returned uncorrected.
    always_use_default: bool, optional
        If True, LOOCV is skipped and `default` is used for the smoothing span.
        This option is less computationally heavy.
    default : float, optional
        Default alpha to use when `always_use_default=True` or when LOOCV fails.
        Defaults to 0.75.
    
    Returns
    -------
    corrected_df : pandas.DataFrame
        Full input DataFrame with corrected values applied to
        sample_rows + qc_rows. Rows outside those arguments are
        returned unchanged. Features that fail QC requirements are
        returned unchanged.
    correction_info : dict
        Dictionary keyed by feature name, containing:
        - 'alpha': selected LOESS alpha (or None if skipped)
        - 'drift_curve': the evaluated drift correction vector
        - 'y_qc': QC intensities used
        - 'x_qc': QC injection orders
        - 'rsd_qc': QC relative standard deviation
        - 'median': QC median intensity (used for scaling)
        - 'y_all': original intensities
        - 'status': "corrected", "skipped_due_to_few_qcs", or error note
    
    Notes
    -----
    - Features with insufficient QC points are not corrected.
    - High QC RSD (>20%) is flagged but does not prevent correction.
    - Drift correction rescales intensities so that QC medians remain
      unchanged.
    - Sample names in `data` and `sample_order` must match exactly.

corrected_df, correction_info = dc.run_loess_drift_correction(
    df,
    qc_cols,
    sample_cols,
    sample_order=sample_order,
    filter_percent=0.5,
    always_use_default=True,
)

Explanation of the parameters chosen:

  • filter_percent is the minimum percentage of values that must be present for this feature to be retained. If the percentage of non-missing is below this, the feature will be filtered out. If this parameter is set to “None”, no filtering will be done. In this case, the filter_percent parameter does not do anything, as the data is already imputed and there are no missing values in the QCs.

  • always_use_default disables the leave-one-out cross validation which calculates the ideal smoothing parameter for the LOESS curve for every feature. It will speed up the computation, but the curve which decides the value correction is less optimal.

  • default(not used here) changes the default smoothing value. If the parameter is
    not used, the default is 0.75. The default smoothing value is either used if always_use_default=True, or if no better smoothing parameter can be found during leave-one-out cross validation.

Now we can inspect our results. First, the corrected output dataframe.

Hide code cell source

 
corrected_df
0 1 2 3 4 5 6 7 8 9 ... 2,534 2,535 2,536 2,537 2,538 2,539 2,540 2,541 2,542 2,544
AAA9485207 170,751.900 74,733.789 102,226.405 40,643.193 17,407.179 207,850.329 301,741.759 122,012.671 46,842.939 49,591.009 ... 75,598.663 71,502.506 42,705.434 43,458.397 215,794.742 5,679,763.408 145,900.211 131,000.095 465,222.264 765,961.103
AAA9485216 254,972.445 84,599.084 132,040.201 81,751.376 23,178.037 13,570.560 475,861.616 77,889.100 74,396.454 43,084.318 ... 149,558.074 149,252.662 89,124.808 72,306.254 249,371.922 4,910,196.789 63,632.247 139,196.065 548,226.926 1,284,688.374
AAA9485239 105,650.309 88,336.415 149,311.684 73,415.302 32,831.615 8,150.034 507,872.985 181,674.039 72,859.812 43,520.085 ... 99,260.097 98,291.891 60,520.171 47,363.989 318,079.391 2,647,962.764 79,815.076 166,784.045 541,498.896 1,122,427.204
AAA9485258 141,472.575 63,427.966 106,948.841 49,569.608 17,890.650 281,878.555 404,315.506 237,912.047 52,412.441 58,980.814 ... 93,657.648 83,965.724 61,541.640 45,676.443 181,279.890 2,620,000.022 84,364.087 82,649.318 246,072.506 623,095.027
AAA9485261 168,001.076 70,318.948 116,960.462 61,511.168 19,872.293 251,266.323 340,996.938 182,178.869 46,781.756 41,506.735 ... 184,149.858 166,209.985 107,600.799 92,958.519 336,225.782 2,594,623.689 144,606.352 149,851.946 492,523.008 1,137,484.580
... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ...
QC_40 205,485.696 75,163.543 120,175.910 46,730.876 26,122.635 242,719.115 377,942.724 179,732.499 137,576.799 53,565.458 ... 74,795.591 68,295.325 48,961.170 35,936.651 132,050.733 4,022,401.965 92,399.577 136,253.218 379,418.707 634,454.149
QC_41 203,188.136 61,241.011 122,943.150 63,540.061 33,088.069 248,059.085 382,615.399 164,834.093 137,556.507 58,724.437 ... 58,505.657 62,169.680 83,238.874 40,640.780 161,296.257 3,870,973.303 83,490.771 155,264.598 416,902.656 728,309.185
QC_42 184,813.193 71,599.643 110,301.340 58,370.767 18,697.821 256,398.299 379,464.207 164,075.895 133,718.503 51,490.957 ... 68,901.645 75,972.002 46,061.444 38,701.025 151,468.529 3,409,242.520 66,731.838 179,643.882 496,034.602 825,262.125
QC_43 166,722.545 70,228.397 126,958.386 54,767.289 23,808.660 258,972.175 365,043.498 175,160.016 128,289.958 49,356.198 ... 68,540.946 70,166.545 51,196.168 46,699.476 137,077.894 3,282,893.947 101,360.384 171,183.195 431,245.208 722,368.417
QC_44 132,774.939 74,633.348 118,182.415 55,202.886 19,445.454 251,731.787 381,694.940 173,688.920 129,874.693 47,119.080 ... 75,275.158 74,182.938 52,077.927 44,150.561 151,617.540 4,618,642.405 75,767.913 161,181.787 425,742.116 704,153.537

477 rows × 2287 columns

We can also look further into the correction_info object, to see the parameters that were chosen for each feature.

For example, let’s check the parameters used for the 200th feature.

print(correction_info[200])

Hide code cell output

 
{'alpha': 0.75, 'drift_curve': [45535.72109399435, 95872.4434930491, 89953.26392811314, 72291.14995250461, 87694.9377725277, 109206.65835098362, 75798.0644986781, 88570.99616463114, 67836.0270363384, 98971.13501670034, 108219.7964385664, 113273.37982860759, 43845.87933440596, 80635.9432310476, 83680.60827632603, 85880.95606542386, 77706.78009381851, 40526.59355543977, 63425.86023823004, 78245.23273022538, 95881.30701904341, 104231.97107122617, 51879.75040291636, 109255.19471679186, 99591.04533801693, 74690.27594922273, 62240.87511687886, 38361.12141335771, 90050.61667193114, 56895.57046185177, 69866.6014022417, 54231.96356504821, 62537.34924693469, 112229.3235038123, 113557.85562700199, 78000.14912872188, 57192.274780472006, 74411.51620146564, 55414.03237144049, 64636.973949070925, 100522.03720143164, 63721.68056624164, 107002.77844470029, 86034.61795928585, 46958.20620612395, 76733.80927449469, 93718.25134900291, 63996.77343750127, 71973.2036218112, 45252.75494315645, 112408.02854503941, 113738.89390096688, 66235.80349743033, 84743.38378780411, 104857.46284391018, 89640.66081962943, 94806.577443715, 91212.89038155926, 91780.64226275028, 81475.21801695232, 48681.14287642709, 70729.76294120362, 105596.93545280874, 67252.09536159062, 37027.60204853086, 97422.41493838777, 98351.57660322594, 102853.10996390201, 36498.5212390837, 56006.1988095531, 70154.82672226225, 79047.72669236553, 52172.74890562638, 93696.56173286762, 88389.18115013688, 90925.8031754479, 95251.46798489643, 64316.92236331323, 71337.06633558593, 105153.46274953855, 49259.06371295138, 74514.48767162247, 93385.19723052456, 99281.03236396892, 101764.53159159663, 71018.87490337907, 89162.69096341936, 103004.10274147805, 103928.92688104839, 41899.5035260289, 43008.313097520775, 80372.7099029705, 39709.9861598459, 96492.80082434419, 77976.36537791807, 113475.3959736622, 59568.460720551804, 111947.32951966158, 69426.66940901759, 63129.85566844913, 67193.99114878377, 65197.533450400304, 77165.37593185387, 95077.09978802787, 77367.15103438363, 50419.95483278426, 92895.23298845005, 103621.26698629704, 107239.32739352513, 82200.5098649743, 66959.57253073803, 110223.34212497238, 64956.929101071095, 68127.407439809, 61944.27218840907, 97732.14889904257, 111409.31740465768, 89755.33032076659, 92450.42664568295, 105390.29687014401, 106626.3143031777, 55118.21262454557, 108600.16747404363, 58082.94660174531, 103307.9959683739, 40799.99415195745, 112806.6357178419, 76347.39235868001, 104388.74303804706, 113627.23516960874, 82971.4409283711, 84997.59444844803, 82458.42198466376, 103518.56709393949, 72441.62081397709, 78316.51738327215, 96408.84209711368, 111690.46385523224, 77050.52272902551, 64017.30279164507, 112576.80979128373, 85566.93855712532, 47817.58802372261, 83985.77187052759, 37825.864647044844, 95614.71156645918, 68999.03627206, 94940.75879616407, 68470.32862072276, 44126.17571342957, 104698.32185951907, 109904.70694438837, 103898.6267010647, 76417.0114380061, 111630.69161802504, 74132.06228809156, 74831.7801343425, 81682.08216753583, 75466.11918878114, 110535.73518872962, 47244.19054749177, 68708.9262778821, 92340.96817100488, 75148.99065547611, 108881.62922991722, 65786.09972242347, 69745.27869539055, 112978.05698156607, 84490.30515694866, 71016.45063414522, 44407.0168808168, 51587.08214687128, 90344.47411837307, 100832.66795762176, 97182.30028496738, 86301.13419665526, 41073.988584847626, 73879.65710639185, 53347.80213967161, 73571.1337215729, 79313.92181347558, 47530.65367214421, 99124.16901685379, 74968.32609209264, 72609.0147115489, 102364.08295701296, 55710.03210790249, 56302.51945201383, 48104.98604586831, 42175.84192868486, 83733.4311579242, 104847.84296335658, 88866.44798681163, 86508.66815905074, 45819.20494819395, 85511.18397317672, 73289.68994696865, 105765.61435154223, 90265.76263251563, 51294.75629966332, 39168.607589910345, 107316.66814710286, 67544.25160352817, 93991.03887942004, 80212.8474527307, 78780.89343785193, 53642.269443123194, 105200.28328435266, 68789.19489460082, 106072.80422240621, 96669.28774106555, 49548.666742553985, 85771.4488942993, 66555.29007509326, 86571.44063492645, 76893.51715479443, 94318.92238996012, 84309.80513933256, 86822.36162792462, 51002.78502346469, 100079.987320121, 68418.37859372767, 112617.92324691922, 106340.34094291406, 44688.397419567285, 97112.63070559905, 80528.58188308353, 106691.0572414641, 98171.046277093, 106071.7645072598, 104543.13761252136, 85252.81415213969, 57489.08094238845, 53053.60427202394, 107633.05537619289, 102110.83557842595, 50711.180480447605, 80844.22240526084, 104542.08848369299, 58379.97946932451, 70442.55110291277, 61350.73533702365, 103085.63439159287, 61053.82888042959, 113682.99872802648, 79581.10054899834, 90890.4514680262, 111840.95243952601, 101453.99140845504, 60162.72073795349, 84624.24688501608, 91514.73660405936, 70382.24394105125, 38629.67107812441, 81159.76809222848, 68151.37563850883, 66079.94670681315, 87408.20071593083, 52466.06549282912, 43566.133160746365, 108927.55991180523, 99838.6214956649, 59865.60045117259, 76073.12182929798, 83365.85464426437, 81160.45230950054, 67832.33542504185, 101335.68914576179, 71302.6025787479, 48392.840182222695, 66874.68565677885, 41348.57371653786, 111135.3104817121, 100323.9885196512, 113380.39208304542, 105023.72763056472, 113751.53439847086, 82420.98394731974, 113492.82167593666, 112040.39726838145, 64607.897491272066, 113702.21721420468, 113269.42422971071, 66666.69733118273, 77436.4569283295, 102612.05028560683, 86194.86611880807, 81790.57125239821, 60459.80797541178, 64902.84224381796, 92061.6481583617, 72926.79785270193, 93170.51901378376, 84938.5834086942, 98661.32564639233, 80898.53748906509, 73244.49932972156, 93073.71656980306, 88276.89051166535, 83050.99876072002, 101853.93575015136, 39981.58642181158, 108551.74822544416, 72724.91997204085, 105459.27686976007, 74197.11331355818, 110125.33648709307, 99599.27432069415, 107888.55497028596, 79265.0897698679, 69577.88674584076, 90578.15809257423, 49838.68810457203, 91497.77987390474, 102384.98759457587, 66373.483983137, 41623.74640945577, 69107.97498306003, 87125.02988769852, 87449.42061865715, 46103.20108875486, 96146.09952618393, 78948.98847106009, 98649.73760389003, 82736.04155272708, 113649.04600482754, 79844.47745387368, 96927.18946727247, 101077.59757025793, 87135.94596720251, 48969.88657212287, 95562.03120656297, 103312.96685452334, 44970.311912680634, 82105.82687153207, 91202.64391860881, 38898.83349570648, 111419.25046891143, 71873.25281097807, 97928.56133203517, 37559.15969788588, 65596.5537289856, 38093.185577599994, 113389.56938527705, 113763.23476358551, 91826.73068411516, 102694.75090291198, 70063.80336509609, 110675.54042114805, 89015.13675146927, 43286.94260945117, 58974.17734972154, 87984.68604270172, 109524.05448764132, 89459.17007966683, 87762.78502406136, 92762.12471881766, 92618.7729109457, 107560.80492160487, 113724.61042879919, 75522.2358060905, 56598.98101090622, 112411.81998322507, 89327.95214732723, 113025.4737220929, 95346.5601591983, 111176.53240020298, 54527.16519697096, 52759.68800235237, 97434.37320338319, 65276.788725610415, 109581.45076799054, 94629.90860848044, 64312.713053601496, 54822.58589159624, 101595.01107249479, 104069.65464718873, 78513.40210033761, 60756.848558064725, 106926.90348704872, 37293.071806316686, 72157.7278926263, 98041.86023738234, 99901.20158866327, 107952.26915949155, 63676.5262653384, 94535.2401170267, 71588.20717202207, 98411.23279251505, 83226.46701734418, 94007.80510871764, 82715.4124616368, 93444.80159721278, 65916.22501749334, 81421.64732483159, 99361.32906962003, 42452.7584798512, 62833.68071773772, 84237.91552736278, 59271.31515157388, 112885.65161692801, 110840.26501464719, 36762.75150072199, 77683.69342335407, 113579.28068726251, 113154.89508380475, 76100.12998677469, 78632.79741978986, 101143.34477854648, 110932.03920544006, 61110.94110972553, 61431.98858445569, 61752.93780972262, 62073.78771061668, 62394.5374954336, 62715.18624877619, 63035.73355731802, 63356.17994052788, 67513.20745740474, 71655.17576571088, 75783.16568801543, 79897.02004123642, 83995.25872987085, 88076.03862518787, 92138.62731851348, 96182.69987624028, 100211.52876572686, 104235.87028302664, 108274.63852451998, 112188.22634118651, 113759.17565068713, 112735.96464176611, 109831.03653888831, 105823.3660972169, 103715.72016798408, 100822.36394628878, 97683.16123175282, 94263.33479873007, 90636.34764530484, 86845.98030261825, 83480.45036103374, 80108.87787235614, 76620.86064755401, 73007.61376382793, 69288.69435604906, 65491.95725018029, 61647.55432236417, 57785.97592322261, 53936.994020206475, 50129.120242646866, 46387.7040986767, 42730.25004195554, 39438.9922845424, 36234.912339809554, 35971.92587909316], 'y_qc': [33303.6875, 100284.6015625, 45113.78515625, 70072.671875, 54953.80078125, 50177.8359375, 57586.01171875, 66693.546875, 73603.171875, 79239.7578125, 82793.421875, 80197.7109375, 74642.390625, 96679.125, 88123.2109375, 107308.46875, 93728.4296875, 112787.5625, 93568.828125, 124562.578125, 118008.984375, 125511.4140625, 124251.7734375, 136760.484375, 126815.7734375, 120638.0, 128232.84375, 147434.171875, 161396.0, 150885.796875, 29073.333984375, 34253.63671875, 25774.205078125, 30055.59375, 32262.681640625, 30580.208984375, 32329.052734375, 31555.744140625, 33427.6015625, 91476.8125, 58279.58203125, 60026.91015625, 57518.7890625, 61595.5078125, 58475.5390625], 'x_qc': [3, 4, 5, 6, 7, 8, 9, 10, 23, 36, 49, 62, 75, 88, 101, 114, 127, 140, 153, 166, 179, 192, 205, 218, 230, 242, 255, 268, 281, 294, 307, 320, 333, 346, 359, 372, 385, 398, 411, 424, 437, 450, 462, 474, 475], 'rsd_qc': np.float64(47.87089333301795), 'median': np.float64(74642.390625), 'y_all': [28007.08203125, 16296.9970703125, 27449.283203125, 30863.3125, 22390.869140625, 34741.3125, 20009.03515625, 42514.29296875, 26367.416015625, 18945.646484375, 16238.0244140625, 22514.484375, 19181.20703125, 29301.171875, 20158.927734375, 19499.875, 21820.6640625, 18712.6015625, 24751.26953125, 72414.1875, 18989.302734375, 17280.11328125, 22018.857421875, 41471.89453125, 15449.0400390625, 18165.814453125, 26751.875, 42703.9765625, 28701.765625, 15239.2890625, 15755.228515625, 17731.919921875, 18944.048828125, 29220.044921875, 33261.52734375, 29913.951171875, 34456.58203125, 35591.2265625, 21413.001953125, 17983.162109375, 39381.12109375, 16849.7578125, 18349.638671875, 28180.9375, 21231.080078125, 22663.888671875, 34689.109375, 24193.37890625, 17202.369140625, 19087.4609375, 16592.521484375, 29140.7421875, 19423.849609375, 24770.47265625, 22159.208984375, 36962.81640625, 18185.763671875, 22699.06640625, 59778.0, 39737.2265625, 38098.5234375, 34169.65234375, 15738.4677734375, 47369.52734375, 41119.65234375, 42405.4921875, 31871.068359375, 19640.150390625, 15765.35546875, 46184.63671875, 39038.78125, 20392.98046875, 30659.638671875, 36296.015625, 15862.4365234375, 51081.33984375, 31246.478515625, 50172.0703125, 18495.3203125, 22242.580078125, 40487.16796875, 33760.8671875, 31485.681640625, 40336.83984375, 16993.287109375, 16105.2314453125, 33578.77734375, 31758.203125, 47226.671875, 24906.5546875, 20036.96484375, 41745.58984375, 35900.4375, 32504.291015625, 36018.2734375, 22720.490234375, 28109.873046875, 17244.15625, 18469.888671875, 27294.333984375, 30607.658203125, 24869.69140625, 21287.017578125, 19110.83984375, 18349.287109375, 31375.330078125, 24428.908203125, 37566.9609375, 24602.37890625, 23767.322265625, 33600.33984375, 16589.080078125, 21204.38671875, 15133.697265625, 18716.96484375, 19532.26953125, 20160.064453125, 18366.5390625, 19239.16796875, 18224.25390625, 17973.392578125, 27917.689453125, 27150.111328125, 29046.041015625, 27901.99609375, 24420.419921875, 18045.51953125, 20020.859375, 25917.98828125, 19922.76171875, 25719.015625, 16613.234375, 45196.5703125, 25550.328125, 33327.59375, 62727.6484375, 32655.833984375, 33846.2890625, 21217.564453125, 22543.212890625, 42691.80078125, 27986.11328125, 16644.71484375, 20730.6875, 43440.265625, 20746.220703125, 29461.326171875, 31354.416015625, 20916.205078125, 17501.3515625, 20872.748046875, 19146.337890625, 34604.15625, 53078.93359375, 20308.943359375, 29038.236328125, 33373.8828125, 18547.8359375, 15391.5634765625, 18133.185546875, 17520.29296875, 26740.66796875, 24356.599609375, 38262.3984375, 28520.134765625, 15265.091796875, 34683.7578125, 30526.408203125, 20632.462890625, 19193.390625, 18386.248046875, 40312.72265625, 25310.20703125, 29359.693359375, 17524.29296875, 13251.7705078125, 18962.1015625, 17347.53125, 16611.701171875, 36482.57421875, 38963.625, 21529.796875, 19416.4609375, 33038.13671875, 22286.04296875, 33256.19921875, 25307.01953125, 17624.21484375, 17355.58984375, 33971.5, 38481.12890625, 42270.02734375, 22956.14453125, 52018.0703125, 20513.787109375, 22958.998046875, 31768.765625, 36052.3828125, 16357.2060546875, 18677.837890625, 16584.86328125, 35291.32421875, 17414.388671875, 47877.9296875, 24977.740234375, 21112.42578125, 22355.376953125, 34800.87109375, 25543.40625, 13251.63671875, 32517.869140625, 20390.94140625, 22164.478515625, 31417.2421875, 37506.56640625, 20006.212890625, 23420.126953125, 19507.4140625, 17493.791015625, 31124.34765625, 66139.6328125, 14401.2646484375, 29485.646484375, 23204.89453125, 19584.03125, 19400.310546875, 24927.044921875, 41946.0546875, 31946.05078125, 30732.416015625, 37127.171875, 20192.080078125, 36048.8515625, 41284.7578125, 61584.5703125, 18022.958984375, 21237.91796875, 46449.453125, 27093.458984375, 30143.337890625, 35807.25, 46669.17578125, 25792.248046875, 20210.8671875, 25258.662109375, 33864.3671875, 18226.63671875, 19854.0703125, 38269.34375, 19492.57421875, 36083.19140625, 18619.826171875, 17500.853515625, 34264.51171875, 43806.78515625, 18938.31640625, 41733.640625, 16110.7177734375, 17096.826171875, 16238.9462890625, 40564.84375, 15085.3134765625, 33319.74609375, 17958.025390625, 19144.33203125, 16793.658203125, 32987.8984375, 26783.904296875, 39660.48828125, 38873.26171875, 42018.05078125, 29185.416015625, 16213.185546875, 38960.55859375, 18461.880859375, 24575.00390625, 23181.736328125, 31648.58984375, 22811.7734375, 26976.439453125, 14631.1083984375, 26067.009765625, 35032.796875, 34084.296875, 19779.123046875, 26408.8671875, 17349.953125, 38357.84765625, 22538.869140625, 24908.14453125, 16464.8203125, 29196.181640625, 26376.541015625, 35384.2421875, 17600.66015625, 21313.7890625, 17225.513671875, 41900.17578125, 15712.1171875, 37593.6875, 21152.677734375, 17576.7109375, 16026.521484375, 43972.86328125, 37453.5859375, 35217.0703125, 30764.859375, 35626.8671875, 24299.05859375, 18487.822265625, 33290.19921875, 33146.96484375, 21655.67578125, 22729.56640625, 35923.3359375, 25141.533203125, 17520.666015625, 18397.5, 21394.041015625, 28283.970703125, 27539.46875, 23747.677734375, 15906.7275390625, 40573.3515625, 18811.84375, 34492.9765625, 18520.900390625, 18811.96484375, 18267.75, 24161.619140625, 34936.75, 14876.34375, 21880.310546875, 21161.021484375, 20896.033203125, 18300.037109375, 37567.30078125, 17947.0390625, 38150.16015625, 31080.8828125, 34456.75, 20526.537109375, 31069.482421875, 29605.9375, 28760.779296875, 19887.021484375, 14771.3564453125, 34691.40234375, 17295.3125, 30204.978515625, 15191.3828125, 35139.23828125, 29944.197265625, 27401.75, 34955.03125, 33403.8203125, 31093.333984375, 43611.640625, 18447.95703125, 36569.62890625, 34094.3828125, 19425.720703125, 44077.7109375, 33735.7265625, 64076.953125, 32269.16796875, 21638.95703125, 19548.19140625, 17674.80859375, 36888.3359375, 21579.8515625, 36391.8359375, 38444.1640625, 41213.77734375, 17595.072265625, 43798.5, 20532.353515625, 14854.26171875, 45456.37890625, 23542.044921875, 23363.033203125, 44178.953125, 29394.763671875, 27598.208984375, 16705.962890625, 51337.51953125, 23021.94921875, 18153.59765625, 21675.384765625, 16033.861328125, 21695.740234375, 24277.673828125, 28411.1953125, 26573.943359375, 69722.09375, 21446.71875, 29022.806640625, 23445.982421875, 20964.00390625, 24669.5546875, 30190.25390625, 44812.25390625, 34488.625, 18152.201171875, 20419.8046875, 36131.0078125, 36577.53515625, 44349.40625, 18524.1875, 22687.7734375, 33303.6875, 100284.6015625, 45113.78515625, 70072.671875, 54953.80078125, 50177.8359375, 57586.01171875, 66693.546875, 73603.171875, 79239.7578125, 82793.421875, 80197.7109375, 74642.390625, 96679.125, 88123.2109375, 107308.46875, 93728.4296875, 112787.5625, 93568.828125, 124562.578125, 118008.984375, 125511.4140625, 124251.7734375, 136760.484375, 126815.7734375, 120638.0, 128232.84375, 147434.171875, 161396.0, 150885.796875, 29073.333984375, 34253.63671875, 25774.205078125, 30055.59375, 32262.681640625, 30580.208984375, 32329.052734375, 31555.744140625, 33427.6015625, 91476.8125, 58279.58203125, 60026.91015625, 57518.7890625, 61595.5078125, 58475.5390625], 'new_values': [45909.35439295584, 12688.180013112324, 22777.16260369703, 31867.1293667342, 19058.203851663624, 23745.57245507768, 19703.962469756956, 35828.528528917486, 29013.00462298413, 14288.492753884797, 11199.845140909933, 14836.097766147735, 32653.721846150238, 27123.258304268234, 17981.59202621452, 16948.079685790446, 20960.15983011487, 34465.10532713224, 29128.401599552028, 69079.83888044629, 14782.93315415571, 12374.600157008615, 31679.800770733556, 28333.31046257881, 11578.885204619644, 18154.168011820522, 32082.195180441173, 83092.64125706822, 23790.713274218204, 19992.71573897131, 16832.190168787678, 24405.40239248917, 22610.953449026933, 19433.904963924633, 21863.039796482037, 28626.21230151048, 44969.73875319761, 35701.654414732584, 28843.19346272284, 20766.84796380326, 29242.354271429853, 19737.492693241617, 12800.236755361911, 24449.37392584397, 33747.85156029643, 22046.172962377623, 27628.322286549443, 28217.85446140543, 17840.333519360775, 31483.91113703378, 11017.94494682541, 19123.93014262733, 21889.106698018244, 21817.954550922514, 15773.949589214086, 30778.365036230407, 14317.87659053413, 18575.36335522891, 48615.619991059575, 36404.70881360645, 58416.148443273945, 36059.848523496716, 11124.914320159363, 52574.93829304799, 82891.38326007701, 32489.928673052684, 24188.048796754287, 14253.217796771434, 32241.410919966365, 61552.68109798298, 41535.958332881775, 19256.47805256171, 43864.062641342905, 28914.843045537193, 13395.419754319732, 41933.457710661365, 24485.836327360066, 58226.71752818106, 19352.280580277424, 15788.72732563945, 61350.313607947326, 33818.817188322624, 25166.371307354584, 30326.41869755522, 12464.25993990904, 16926.950452010773, 28110.414660212355, 23013.726056659114, 33918.48444704052, 44370.09099528817, 34774.83419113209, 38769.26170275934, 67481.6271327, 25143.823852658377, 34478.2681589476, 14945.193120624492, 35223.13820105316, 11497.77357203479, 19857.450411805148, 32271.804165222176, 34000.492315680734, 28472.445542448975, 20591.021065080356, 15003.389627673772, 17703.051460426024, 46448.46770384133, 19628.909363590657, 27060.929232430695, 17124.13180181368, 21581.97991198793, 37455.58098655167, 11233.996097640302, 24366.08593306178, 16580.92367450351, 22553.80444106661, 14917.663313127363, 13506.908048540135, 15273.994070692619, 15533.270563770657, 12907.27817756868, 12582.0441083454, 37806.79710538381, 18660.645396799286, 37327.065282409516, 20159.830535135814, 44676.4407917856, 11940.438692380834, 19573.750457595597, 18532.463838539843, 13087.377866054736, 23137.223956722923, 14589.254411376696, 40912.49838377149, 18423.145007141695, 34340.08299823731, 59784.855022928, 25283.049390755452, 22619.370017876776, 20554.43205221329, 26284.757856811393, 28306.167818280697, 24413.0552622256, 25982.0990258346, 18424.407370872974, 85721.3789001471, 16195.703405315042, 31870.93523713611, 24650.830662537282, 22801.63658162292, 29604.711912435243, 14880.771586984118, 13003.34145464222, 24860.16447062978, 51846.289467591596, 13579.671158883104, 29238.13680835846, 33289.41785283492, 16949.30856997097, 15223.56132389874, 12244.947903539643, 27680.790727555563, 29049.89922305605, 19688.279843166678, 38004.46107805638, 19551.608980410943, 17320.117010074846, 37119.05160333984, 20168.200324489913, 18227.610277617638, 20173.361913424593, 30904.87957176519, 58329.29226542029, 20911.233127007385, 21733.80656040687, 13459.808188988862, 11461.539178187662, 34459.195239227054, 17526.627148217598, 23242.514931928115, 37013.790845566335, 36668.69637660363, 33810.50720458475, 14620.965564972454, 32894.49871741503, 22910.151464903232, 24249.93358101191, 33907.29399953289, 23365.091680263296, 26929.90525783005, 60122.42689558698, 34303.18590544783, 30092.520776267294, 19281.759833585504, 44882.82164610019, 33418.260145441775, 20040.82297715602, 32355.11864180566, 25443.392517990153, 13526.069334152957, 27179.35657434068, 31605.25532238138, 24546.315623577015, 19244.444506862685, 38021.95584448582, 23243.137511141027, 20003.351871714338, 31107.16225146952, 24692.1408685629, 27716.866146302506, 9325.046618995595, 25108.403583034513, 30717.85203757324, 19288.582444276042, 35234.73582751594, 32338.375798823778, 19420.513103890855, 18534.290047278246, 17270.58932386794, 15039.655199616456, 45550.36974230506, 49328.74633960059, 15711.346037089834, 19542.88517549611, 16287.974880043907, 32710.927108221676, 14911.40284806085, 23105.016637283807, 29345.98156690852, 24289.540468069245, 21626.31131615883, 26508.300106379666, 17679.00735843169, 46804.93088082711, 58084.51775865719, 42708.251082084986, 13174.671787861691, 31260.34444606074, 42886.159595843426, 19344.558524001422, 38540.10950373204, 37942.106012065124, 56780.066769772624, 18675.68711367164, 24709.137348844277, 16584.422868219543, 31762.784209317186, 14968.346132811332, 13250.560187809158, 28155.77056647812, 24183.951810624756, 31826.997191497, 15186.934804869918, 18560.157665309507, 66207.7878714883, 40288.96639949086, 20742.078904954713, 47141.362258596346, 13757.776495162192, 24323.30241617292, 27822.385975231035, 27796.977324022846, 11278.239266024393, 41544.15030855656, 17620.28314111412, 17141.05512177553, 15444.945905921923, 36299.73206201434, 19728.633256880297, 41518.171169107445, 59959.14220270767, 46898.57946864578, 52685.47441867644, 10889.346721756729, 28987.177208901092, 12154.12028005985, 17465.929676708332, 15211.576947246172, 28661.71080118992, 15002.933917995051, 17971.963509808666, 16903.52032428269, 17112.277781583987, 23085.945097830994, 38161.983469551924, 19065.451690942093, 19210.423874062468, 15024.583676441329, 35005.51963938145, 27826.00757389903, 28645.948152179888, 13349.462821065466, 29883.0177513031, 21131.2344147546, 31094.99030164698, 13315.808824110656, 19665.524473999525, 17554.267309881543, 33602.71194902435, 13285.356811476871, 33787.46492253325, 15501.476919723375, 32814.298808910025, 11020.162241074575, 45132.25506746025, 26509.049510233403, 35428.4176504933, 20852.264558220733, 26699.738080339022, 16811.234740088792, 17409.62200841337, 35713.35908989141, 27315.29046297936, 32433.265649384713, 18542.408097445637, 26189.42226391431, 28273.702533617838, 31419.189995145436, 19870.838088658347, 18328.858751542593, 24141.76302956884, 44587.18126936894, 18436.353077465174, 15039.029549712042, 30699.442592071908, 16971.575666562476, 22654.28809923957, 17314.212901980445, 14486.853853995977, 13490.116149545958, 20697.32524285475, 53252.37045071066, 11619.73900344447, 15808.264311447803, 35123.377278940985, 18996.57956433564, 14977.181139496077, 72087.33238621253, 12023.145862384357, 39620.01224959409, 23690.24280867846, 68476.883238491, 23357.169149871137, 60879.66675104929, 19489.07614408202, 18870.536930652725, 16165.389042444378, 10736.375015841855, 36958.44474182831, 11664.397270562287, 25327.959799441644, 26195.45437189085, 44475.00360200278, 25403.357899897157, 18674.729828773725, 29165.563402254393, 28410.003208641923, 25019.702687173292, 35147.05510575449, 12802.057552129672, 24002.232371159294, 33697.1782263321, 25618.521870833763, 29267.969490607644, 28189.55567170312, 42316.628346775666, 25262.031861930864, 14528.09732929936, 26759.574492580377, 25005.64383962945, 28259.365663548946, 24676.025604269318, 24788.62631011323, 30324.07357675024, 47833.38661300171, 23956.15303011832, 32178.98902000467, 14726.520970589898, 14121.88971559231, 55845.1083498133, 16433.979249971755, 46761.30353620065, 45700.20111313392, 22379.16974456863, 20620.335519526627, 11551.151426066594, 60178.458394581285, 18177.489414609827, 18928.0885921734, 16440.222225790567, 14380.109878166237, 17226.462372990492, 21908.173572678646, 22694.462424778143, 30091.872830258195, 63916.95486236142, 16111.241400971465, 51029.232207173365, 27852.326308594565, 18575.99821693175, 31067.212408231484, 19962.437146443513, 30177.605226899967, 70025.04748096952, 17441.545720422007, 13419.551776942293, 23833.743973852575, 35876.87258493469, 42098.79101244066, 13670.594366966443, 15265.8299573594, 40677.93436799675, 121849.91200813386, 54530.2117379582, 84260.87627542391, 65741.06049816769, 59720.68098318252, 68189.2212349412, 78574.27298621029, 81375.436203338, 82543.16443253151, 81547.1204015891, 74923.30582464955, 66330.96394325083, 81933.30588178245, 71389.46308791472, 83276.52116351885, 69813.46505308426, 80766.18226439344, 64504.49629206647, 82875.43993598178, 77430.87674989729, 83101.00531022942, 84442.88336919258, 96463.85173010886, 91267.09512311108, 89312.61247768241, 97986.24341644208, 116745.70045781361, 132915.58618907523, 129683.33768429591, 25995.345528774815, 31916.229514058246, 25108.674938274533, 30728.59464234846, 34755.50676788455, 34852.82774015079, 39143.77154553275, 40760.68878263102, 46259.82850564934, 136209.2121047436, 93777.59498906498, 104856.67814944386, 108860.28452514039, 126884.14730438475, 121337.79112581126], 'status': 'corrected'}

Plot an example curve for one feature#

We can also plot an example feature, to see how the values have changed and what the LOESS curve would look like for the data of this feature.

Hide code cell source

 
plot_loess_example_curve(
    df=df,
    feature_idx=300,
    samples=sample_cols,
    qcs=qc_cols,
    sample_order=sample_order,
)
../_images/5719064136bc5df1c0f94bf9508b6871c3df1b3476b4eb81486f776ece3e81ba.png

In this plot, we can see all of the data points of this feature, ordered by the injection time. The red points are our QC samples, so we can see whether there was any instrumental drift over time. The LOESS curve is also calculated, with the smoothing value alpha chosen with leave-one-out cross validation, just like in the run_drift_correction function.

Alternative values for the smoothing parameter alpha can be tested by adding the argument “alpha” and choosing a value, just like in the example below.

Hide code cell source

 
plot_loess_example_curve(
    df=df,
    feature_idx=300,
    samples=sample_cols,
    qcs=qc_cols,
    sample_order=sample_order,
    alpha=0.6,
)
../_images/b8c86da902077f5cfeecd66a51299cbd01575e059582d38886544a2982f1a5c0.png

2. Common Principal Components Analysis-based drift correction#

We can also use another method for correcting drift which is based on Common Principal Components Analysis (CPCA). This method is based on common principal components in defined groups of the data. It assumes that when calculating common principal components of QC samples, the drift contribution can be identified as the direction capturing maximum variance that simultaneously diagonalizes the covariance matrices of a set of classes.

Therefore, the variability in the identified direction can be explained as caused by experimental drift and subtracted from all samples.

Let’s use different example data for demonstrating this method.

Load in data#

# Load data
df_original = pd.read_csv(
    "../../example_data/DidacMauricio_hilic/DM_FIS2018_Hilic_pos_results2023_filled_imputed.csv",
    index_col=0,
)

df = df_original.T
df = df.drop(
    ["Qidx", "SOIidx", "rtmed", "start", "end", "mass", "MaxInt", "formula", "anot"]
)

# Define sample columns and qc columns
collist = list(df_original.columns.values)
sample_cols = []
qc_cols = []
for col in collist:
    if col.startswith("AAA"):
        sample_cols.append(col)
    elif col.startswith("QC"):
        qc_cols.append(col)
df
0 1 2 3 4 5 6 7 8 9 ... 2,534 2,535 2,536 2,537 2,538 2,539 2,540 2,541 2,542 2,544
AAA9485207 279,488.531 73,170.344 106,222.969 40,187.371 16,231.437 211,807.094 319,754.562 112,320.398 46,083.371 48,803.125 ... 82,146.870 80,973.820 46,157.050 49,622.580 231,180.200 6,403,619.000 137,733.500 130,200.000 468,793.800 821,122.200
AAA9485216 247,458.016 86,581.648 132,690.734 82,426.359 24,345.967 12,622.342 389,471.938 84,265.992 73,903.742 43,815.148 ... 148,114.400 134,861.800 90,832.130 72,869.770 240,460.700 4,852,053.000 59,179.240 132,118.200 513,293.500 1,214,919.000
AAA9485239 99,304.359 93,201.195 152,236.844 74,535.336 35,357.852 7,571.239 417,576.844 199,175.516 68,742.586 44,511.543 ... 95,990.200 85,438.980 63,371.030 49,218.960 310,655.100 2,619,595.000 72,289.910 160,829.900 518,888.200 1,092,635.000
AAA9485258 119,563.797 72,692.320 113,827.773 51,309.215 20,640.715 259,447.391 340,227.594 271,096.281 41,593.598 61,431.602 ... 83,902.570 64,054.850 69,871.040 51,861.310 184,134.600 2,601,840.000 70,717.240 83,523.680 252,012.400 658,375.000
AAA9485261 191,762.188 64,645.020 115,821.445 60,884.336 18,506.797 235,303.953 320,328.281 174,622.797 49,389.219 41,346.922 ... 221,212.200 191,401.000 114,394.600 98,023.710 359,151.000 2,767,868.000 150,113.300 143,107.200 463,635.800 1,099,109.000
... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ...
QC_40 334,133.656 73,489.242 124,753.789 46,211.445 24,354.287 246,895.516 399,774.781 165,567.125 135,517.156 52,733.680 ... 81,523.700 77,415.330 52,939.590 40,989.100 141,548.000 4,531,348.000 87,438.410 135,338.900 381,900.100 678,957.200
QC_41 339,973.344 60,233.695 128,157.016 62,802.289 30,869.357 254,287.641 407,853.812 151,410.297 134,795.859 57,720.047 ... 62,895.500 70,159.380 89,829.210 46,564.210 172,408.800 4,375,519.000 78,174.880 154,609.500 421,614.900 785,135.600
QC_42 317,506.188 70,798.312 115,411.750 57,658.062 17,454.047 264,687.438 407,221.000 150,344.312 130,498.227 50,533.473 ... 73,050.480 85,322.640 49,600.440 44,505.460 161,372.500 3,864,418.000 61,854.690 179,258.000 503,656.400 895,259.400
QC_43 294,083.344 69,802.516 133,331.281 54,058.363 22,235.070 269,192.375 394,239.344 160,134.375 124,771.242 48,362.730 ... 71,588.570 78,372.000 54,991.750 53,880.830 145,470.600 3,730,628.000 92,978.760 171,145.200 439,513.500 788,261.100
QC_44 234,717.266 74,212.203 124,152.578 54,484.602 18,160.814 261,814.906 412,434.000 158,760.438 126,280.156 46,164.520 ... 78,520.450 82,817.930 55,926.430 50,953.160 160,843.600 5,249,600.000 69,441.320 161,171.100 434,035.700 768,748.800

477 rows × 2287 columns

Seeing as this method is based on calculating principal components, as with PCA, there must not be any missing data.

We will therefore first calculate missingness (NAs).

We need to check for missing values in both the sample columns and the QC columns.

Hide code cell source

 
if dc.check_missingness(df, sample_cols + qc_cols):
    print(
        "There are missing values. Consider imputing first, or use LOESS drift correction instead."
    )
else:
    print("\nThere is no missingness. We can proceed with the CPCA drift correction.")

There is no missingness. We can proceed with the CPCA drift correction.

Visualise non-corrected data#

Now that we know we can proceed, let’s visualise our data before drift correction with a PCA.

Hide code cell source

 
pca_for_cpca_drift(
    df,
    sample_cols,  # list of col names, OR dict {group_name: [col names]}
    qc_cols,
    log_transform=True,
    title="PCA",
)
../_images/b9e0bfb231fb332eedf2e6dbb7a05eab2637e4430cf0d63ac3cdb81e32278126.png

We see some clear indication of instrumental drift in the QC samples.

Run drift correction based on CPCA#

df_corrected = dc.run_cpca_drift_correction(df, sample_cols, qc_cols, n_comps=1)

Let’s plot the corrected data.

Hide code cell source

 
pca_for_cpca_drift(  # This function is defined in the beginning of this notebook.
    df_corrected,
    sample_cols,  # list of col names, OR dict {group_name: [col names]}
    qc_cols,
    log_transform=True,
    title="PCA",
)
../_images/1ae432c5c95f7675b2688ec5d39f19c0ddb60a12a4597e8fc6a56ef96ae59193.png

There is some change, but still a significant amount of drift is clearly visible from the PCA plot.

We can play around with the n_comps variable which decides the number of components.

df_corrected_2comps = dc.run_cpca_drift_correction(df, sample_cols, qc_cols, n_comps=2)
df_corrected_3comps = dc.run_cpca_drift_correction(df, sample_cols, qc_cols, n_comps=3)
df_corrected_4comps = dc.run_cpca_drift_correction(df, sample_cols, qc_cols, n_comps=4)

Hide code cell source

 
pca_for_cpca_drift(
    df_corrected_2comps,
    sample_cols,  # list of col names, OR dict {group_name: [col names]}
    qc_cols,
    log_transform=True,
    title="PCA with 2 components",
)

pca_for_cpca_drift(
    df_corrected_3comps,
    sample_cols,  # list of col names, OR dict {group_name: [col names]}
    qc_cols,
    log_transform=True,
    title="PCA with 3 components",
)

pca_for_cpca_drift(
    df_corrected_4comps,
    sample_cols,  # list of col names, OR dict {group_name: [col names]}
    qc_cols,
    log_transform=True,
    title="PCA with 4 components",
)
../_images/6b53acc628ae46f132bb2d285d40dba6eb36a14af042ae5dfdf8a1177f53f0fc.png ../_images/e5a99582c2a79a0b24ffedc168e7631a2c510e365f06b12eb001db396c7cf39b.png ../_images/ebd20ad6ae56e9ac2c6336481436cfec7de85a77653a8886a79e597b49b7a8c7.png

The correction methods with three and four components are already looking better. Let’s calculate the centroids of the QC principal components and the distance of the QC points to them, to objectively decide which number of n_comps is most favourable.

Hide code cell source

 
print(
    f"1 component: {dc.cpca_centroid(df_corrected, sample_cols, qc_cols, log_transform=True)}"
)
print(
    f"2 components: {dc.cpca_centroid(df_corrected_2comps, sample_cols, qc_cols, log_transform=True)}"
)
print(
    f"3 components: {dc.cpca_centroid(df_corrected_3comps, sample_cols, qc_cols, log_transform=True)}"
)
print(
    f"4 components: {dc.cpca_centroid(df_corrected_4comps, sample_cols, qc_cols, log_transform=True)}"
)

1 component: 6.659766536928962
2 components: 7.944509914596684
3 components: 1.8490307919985336
4 components: 2.251247334153261

According to this, the CPCA method with three principal components is most favourable in this case.

We can go ahead and continue our metabolomics data analysis with this data set.

df_corrected_3comps
0 1 2 3 4 5 6 7 8 9 ... 2,534 2,535 2,536 2,537 2,538 2,539 2,540 2,541 2,542 2,544
AAA9485207 264,911.666 76,093.970 107,731.115 41,789.199 17,777.028 208,108.097 306,494.837 121,963.335 44,889.022 49,012.511 ... 95,143.209 87,028.159 59,980.735 59,075.948 244,828.998 6,538,534.271 146,855.952 134,939.174 478,100.706 832,092.608
AAA9485216 244,329.882 86,310.281 132,094.291 82,321.150 24,351.410 10,278.931 382,619.571 87,171.869 78,372.927 43,883.784 ... 138,195.868 125,428.392 84,809.253 68,058.862 228,344.182 4,676,645.012 53,646.032 127,037.161 496,962.146 1,182,481.025
AAA9485239 121,545.312 87,838.557 148,081.789 72,700.333 33,388.147 10,176.755 428,086.054 186,668.522 76,799.608 43,731.288 ... 108,797.014 102,530.257 65,371.343 50,698.038 317,636.660 2,791,937.465 80,948.821 159,806.602 513,172.856 1,079,841.838
AAA9485258 162,138.286 63,197.573 106,017.203 48,916.587 17,646.600 264,734.544 359,824.836 247,385.677 54,000.715 59,581.508 ... 150,801.967 131,711.745 103,686.397 76,351.244 240,164.597 3,511,612.142 114,593.775 95,345.264 275,433.247 690,675.171
AAA9485261 161,038.603 68,937.047 118,463.154 61,758.290 20,026.160 226,478.584 292,481.564 194,665.402 54,494.756 42,554.703 ... 154,991.514 126,925.426 77,300.806 70,024.454 293,116.590 1,763,808.493 109,215.502 122,164.522 405,338.743 992,331.026
... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ...
QC_40 318,993.906 84,522.111 135,945.215 50,231.349 27,755.257 259,134.175 430,467.546 166,026.895 89,925.519 53,489.470 ... 137,906.342 121,833.542 97,626.237 76,460.554 223,249.699 5,587,889.118 117,029.993 175,457.154 513,082.085 939,820.835
QC_41 322,126.443 70,822.062 138,438.511 66,830.331 34,336.715 263,851.296 430,584.410 154,843.010 94,461.601 58,447.505 ... 116,465.481 111,050.115 133,386.967 80,658.278 248,372.252 5,344,841.278 107,022.501 191,280.421 539,560.564 1,017,289.233
QC_42 301,912.641 80,115.299 124,387.851 61,294.845 20,564.553 273,038.093 426,918.447 153,391.745 95,031.357 51,128.663 ... 124,406.926 124,794.934 90,877.844 76,611.083 232,267.111 4,772,819.371 90,008.901 212,727.546 610,192.128 1,103,739.770
QC_43 275,656.649 79,188.791 142,043.838 57,809.475 25,500.161 275,890.702 408,854.157 165,587.772 91,769.175 48,987.886 ... 120,654.097 114,891.805 95,548.660 85,149.221 212,780.727 4,580,556.612 120,231.160 202,758.257 539,008.118 981,474.638
QC_44 215,208.092 84,404.918 133,753.800 58,451.701 21,621.256 269,530.945 429,753.474 164,126.860 89,812.751 46,879.939 ... 128,429.509 119,914.776 97,422.963 83,167.893 231,041.389 6,134,909.597 96,677.520 194,838.649 541,233.603 978,352.995

477 rows × 2287 columns