Imputation data (MS-example)

Imputation data (MS-example)#

We will explore imputation of proteomics data using an Alzheimer dataset where the data was collected in four different sites.

  • k Nearest Neighbour imputation can also be used with other types of data

  • the replacement from the normal distribution on the sample level is typical to normally distributed samples from mass spectrometer data (in the log2 space)

Refers to the acore.imputation_analysis module.

Common shared parameters across: imputation_KNN, imputation_normal_distribution and imputation_mixed_norm_KNN function presented here:

  • data: pd.DataFrame with samples as rows and features as columns, which can can contain a group column.

  • drop_cols: optional iterable of column names excluded from imputation.

%pip install acore vuecore

Hide code cell output

 
Requirement already satisfied: acore in /home/docs/checkouts/readthedocs.org/user_builds/analytics-core/envs/stable/lib/python3.11/site-packages (0.3.0)
Requirement already satisfied: vuecore in /home/docs/checkouts/readthedocs.org/user_builds/analytics-core/envs/stable/lib/python3.11/site-packages (0.1.1)
Requirement already satisfied: numpy in /home/docs/checkouts/readthedocs.org/user_builds/analytics-core/envs/stable/lib/python3.11/site-packages (from acore) (2.4.6)
Requirement already satisfied: pandas in /home/docs/checkouts/readthedocs.org/user_builds/analytics-core/envs/stable/lib/python3.11/site-packages (from acore) (2.3.3)
Requirement already satisfied: dsp-pandas in /home/docs/checkouts/readthedocs.org/user_builds/analytics-core/envs/stable/lib/python3.11/site-packages (from acore) (0.0.5)
Requirement already satisfied: scipy in /home/docs/checkouts/readthedocs.org/user_builds/analytics-core/envs/stable/lib/python3.11/site-packages (from acore) (1.17.1)
Requirement already satisfied: scikit-learn>=1.5 in /home/docs/checkouts/readthedocs.org/user_builds/analytics-core/envs/stable/lib/python3.11/site-packages (from acore) (1.9.0)
Requirement already satisfied: gseapy!=1.1.5 in /home/docs/checkouts/readthedocs.org/user_builds/analytics-core/envs/stable/lib/python3.11/site-packages (from acore) (1.2.1)
Requirement already satisfied: pingouin<0.6.0 in /home/docs/checkouts/readthedocs.org/user_builds/analytics-core/envs/stable/lib/python3.11/site-packages (from acore) (0.5.5)
Requirement already satisfied: umap-learn in /home/docs/checkouts/readthedocs.org/user_builds/analytics-core/envs/stable/lib/python3.11/site-packages (from acore) (0.5.12)
Requirement already satisfied: statsmodels in /home/docs/checkouts/readthedocs.org/user_builds/analytics-core/envs/stable/lib/python3.11/site-packages (from acore) (0.14.6)
Requirement already satisfied: combat in /home/docs/checkouts/readthedocs.org/user_builds/analytics-core/envs/stable/lib/python3.11/site-packages (from acore) (0.3.3)
Requirement already satisfied: inmoose in /home/docs/checkouts/readthedocs.org/user_builds/analytics-core/envs/stable/lib/python3.11/site-packages (from acore) (0.9.1)
Requirement already satisfied: requests in /home/docs/checkouts/readthedocs.org/user_builds/analytics-core/envs/stable/lib/python3.11/site-packages (from acore) (2.34.2)
Requirement already satisfied: rarfile in /home/docs/checkouts/readthedocs.org/user_builds/analytics-core/envs/stable/lib/python3.11/site-packages (from acore) (4.2)
Requirement already satisfied: pandera in /home/docs/checkouts/readthedocs.org/user_builds/analytics-core/envs/stable/lib/python3.11/site-packages (from acore) (0.31.1)
Requirement already satisfied: matplotlib in /home/docs/checkouts/readthedocs.org/user_builds/analytics-core/envs/stable/lib/python3.11/site-packages (from pingouin<0.6.0->acore) (3.11.0)
Requirement already satisfied: pandas-flavor in /home/docs/checkouts/readthedocs.org/user_builds/analytics-core/envs/stable/lib/python3.11/site-packages (from pingouin<0.6.0->acore) (0.8.1)
Requirement already satisfied: seaborn in /home/docs/checkouts/readthedocs.org/user_builds/analytics-core/envs/stable/lib/python3.11/site-packages (from pingouin<0.6.0->acore) (0.13.2)
Requirement already satisfied: tabulate in /home/docs/checkouts/readthedocs.org/user_builds/analytics-core/envs/stable/lib/python3.11/site-packages (from pingouin<0.6.0->acore) (0.10.0)
Requirement already satisfied: plotly in /home/docs/checkouts/readthedocs.org/user_builds/analytics-core/envs/stable/lib/python3.11/site-packages (from vuecore) (6.8.0)
Requirement already satisfied: beautifulsoup4 in /home/docs/checkouts/readthedocs.org/user_builds/analytics-core/envs/stable/lib/python3.11/site-packages (from vuecore) (4.15.0)
Requirement already satisfied: dash in /home/docs/checkouts/readthedocs.org/user_builds/analytics-core/envs/stable/lib/python3.11/site-packages (from vuecore) (4.2.0)
Requirement already satisfied: networkx in /home/docs/checkouts/readthedocs.org/user_builds/analytics-core/envs/stable/lib/python3.11/site-packages (from vuecore) (3.6.1)
Requirement already satisfied: kaleido in /home/docs/checkouts/readthedocs.org/user_builds/analytics-core/envs/stable/lib/python3.11/site-packages (from vuecore) (1.3.0)
Requirement already satisfied: pyvis in /home/docs/checkouts/readthedocs.org/user_builds/analytics-core/envs/stable/lib/python3.11/site-packages (from vuecore) (0.3.2)
Requirement already satisfied: wordcloud in /home/docs/checkouts/readthedocs.org/user_builds/analytics-core/envs/stable/lib/python3.11/site-packages (from vuecore) (1.9.6)
Requirement already satisfied: cyjupyter in /home/docs/checkouts/readthedocs.org/user_builds/analytics-core/envs/stable/lib/python3.11/site-packages (from vuecore) (0.2.0)
Requirement already satisfied: nltk in /home/docs/checkouts/readthedocs.org/user_builds/analytics-core/envs/stable/lib/python3.11/site-packages (from vuecore) (3.9.4)
Requirement already satisfied: webweb in /home/docs/checkouts/readthedocs.org/user_builds/analytics-core/envs/stable/lib/python3.11/site-packages (from vuecore) (0.0.37)
Requirement already satisfied: dash-cytoscape in /home/docs/checkouts/readthedocs.org/user_builds/analytics-core/envs/stable/lib/python3.11/site-packages (from vuecore) (1.0.2)
Requirement already satisfied: pydantic in /home/docs/checkouts/readthedocs.org/user_builds/analytics-core/envs/stable/lib/python3.11/site-packages (from vuecore) (2.13.4)
Requirement already satisfied: nbformat>=4.2.0 in /home/docs/checkouts/readthedocs.org/user_builds/analytics-core/envs/stable/lib/python3.11/site-packages (from vuecore) (5.10.4)
Requirement already satisfied: contourpy>=1.0.1 in /home/docs/checkouts/readthedocs.org/user_builds/analytics-core/envs/stable/lib/python3.11/site-packages (from matplotlib->pingouin<0.6.0->acore) (1.3.3)
Requirement already satisfied: cycler>=0.10 in /home/docs/checkouts/readthedocs.org/user_builds/analytics-core/envs/stable/lib/python3.11/site-packages (from matplotlib->pingouin<0.6.0->acore) (0.12.1)
Requirement already satisfied: fonttools>=4.22.0 in /home/docs/checkouts/readthedocs.org/user_builds/analytics-core/envs/stable/lib/python3.11/site-packages (from matplotlib->pingouin<0.6.0->acore) (4.63.0)
Requirement already satisfied: kiwisolver>=1.3.1 in /home/docs/checkouts/readthedocs.org/user_builds/analytics-core/envs/stable/lib/python3.11/site-packages (from matplotlib->pingouin<0.6.0->acore) (1.5.0)
Requirement already satisfied: packaging>=20.0 in /home/docs/checkouts/readthedocs.org/user_builds/analytics-core/envs/stable/lib/python3.11/site-packages (from matplotlib->pingouin<0.6.0->acore) (26.2)
Requirement already satisfied: pillow>=9 in /home/docs/checkouts/readthedocs.org/user_builds/analytics-core/envs/stable/lib/python3.11/site-packages (from matplotlib->pingouin<0.6.0->acore) (12.2.0)
Requirement already satisfied: pyparsing>=3 in /home/docs/checkouts/readthedocs.org/user_builds/analytics-core/envs/stable/lib/python3.11/site-packages (from matplotlib->pingouin<0.6.0->acore) (3.3.2)
Requirement already satisfied: python-dateutil>=2.7 in /home/docs/checkouts/readthedocs.org/user_builds/analytics-core/envs/stable/lib/python3.11/site-packages (from matplotlib->pingouin<0.6.0->acore) (2.9.0.post0)
Requirement already satisfied: fastjsonschema>=2.15 in /home/docs/checkouts/readthedocs.org/user_builds/analytics-core/envs/stable/lib/python3.11/site-packages (from nbformat>=4.2.0->vuecore) (2.21.2)
Requirement already satisfied: jsonschema>=2.6 in /home/docs/checkouts/readthedocs.org/user_builds/analytics-core/envs/stable/lib/python3.11/site-packages (from nbformat>=4.2.0->vuecore) (4.26.0)
Requirement already satisfied: jupyter-core!=5.0.*,>=4.12 in /home/docs/checkouts/readthedocs.org/user_builds/analytics-core/envs/stable/lib/python3.11/site-packages (from nbformat>=4.2.0->vuecore) (5.9.1)
Requirement already satisfied: traitlets>=5.1 in /home/docs/checkouts/readthedocs.org/user_builds/analytics-core/envs/stable/lib/python3.11/site-packages (from nbformat>=4.2.0->vuecore) (5.15.1)
Requirement already satisfied: attrs>=22.2.0 in /home/docs/checkouts/readthedocs.org/user_builds/analytics-core/envs/stable/lib/python3.11/site-packages (from jsonschema>=2.6->nbformat>=4.2.0->vuecore) (26.1.0)
Requirement already satisfied: jsonschema-specifications>=2023.03.6 in /home/docs/checkouts/readthedocs.org/user_builds/analytics-core/envs/stable/lib/python3.11/site-packages (from jsonschema>=2.6->nbformat>=4.2.0->vuecore) (2025.9.1)
Requirement already satisfied: referencing>=0.28.4 in /home/docs/checkouts/readthedocs.org/user_builds/analytics-core/envs/stable/lib/python3.11/site-packages (from jsonschema>=2.6->nbformat>=4.2.0->vuecore) (0.37.0)
Requirement already satisfied: rpds-py>=0.25.0 in /home/docs/checkouts/readthedocs.org/user_builds/analytics-core/envs/stable/lib/python3.11/site-packages (from jsonschema>=2.6->nbformat>=4.2.0->vuecore) (2026.5.1)
Requirement already satisfied: platformdirs>=2.5 in /home/docs/checkouts/readthedocs.org/user_builds/analytics-core/envs/stable/lib/python3.11/site-packages (from jupyter-core!=5.0.*,>=4.12->nbformat>=4.2.0->vuecore) (4.10.0)
Requirement already satisfied: pytz>=2020.1 in /home/docs/checkouts/readthedocs.org/user_builds/analytics-core/envs/stable/lib/python3.11/site-packages (from pandas->acore) (2026.2)
Requirement already satisfied: tzdata>=2022.7 in /home/docs/checkouts/readthedocs.org/user_builds/analytics-core/envs/stable/lib/python3.11/site-packages (from pandas->acore) (2026.2)
Requirement already satisfied: six>=1.5 in /home/docs/checkouts/readthedocs.org/user_builds/analytics-core/envs/stable/lib/python3.11/site-packages (from python-dateutil>=2.7->matplotlib->pingouin<0.6.0->acore) (1.17.0)
Requirement already satisfied: typing-extensions>=4.4.0 in /home/docs/checkouts/readthedocs.org/user_builds/analytics-core/envs/stable/lib/python3.11/site-packages (from referencing>=0.28.4->jsonschema>=2.6->nbformat>=4.2.0->vuecore) (4.15.0)
Requirement already satisfied: joblib>=1.4.0 in /home/docs/checkouts/readthedocs.org/user_builds/analytics-core/envs/stable/lib/python3.11/site-packages (from scikit-learn>=1.5->acore) (1.5.3)
Requirement already satisfied: narwhals>=2.0.1 in /home/docs/checkouts/readthedocs.org/user_builds/analytics-core/envs/stable/lib/python3.11/site-packages (from scikit-learn>=1.5->acore) (2.22.1)
Requirement already satisfied: threadpoolctl>=3.5.0 in /home/docs/checkouts/readthedocs.org/user_builds/analytics-core/envs/stable/lib/python3.11/site-packages (from scikit-learn>=1.5->acore) (3.6.0)
Requirement already satisfied: soupsieve>=1.6.1 in /home/docs/checkouts/readthedocs.org/user_builds/analytics-core/envs/stable/lib/python3.11/site-packages (from beautifulsoup4->vuecore) (2.8.4)
Requirement already satisfied: mpmath>=1.1.0 in /home/docs/checkouts/readthedocs.org/user_builds/analytics-core/envs/stable/lib/python3.11/site-packages (from combat->acore) (1.4.1)
Requirement already satisfied: patsy>=0.5.1 in /home/docs/checkouts/readthedocs.org/user_builds/analytics-core/envs/stable/lib/python3.11/site-packages (from combat->acore) (1.0.2)
Requirement already satisfied: ipywidgets>=7.0.0 in /home/docs/checkouts/readthedocs.org/user_builds/analytics-core/envs/stable/lib/python3.11/site-packages (from cyjupyter->vuecore) (8.1.8)
Requirement already satisfied: comm>=0.1.3 in /home/docs/checkouts/readthedocs.org/user_builds/analytics-core/envs/stable/lib/python3.11/site-packages (from ipywidgets>=7.0.0->cyjupyter->vuecore) (0.2.3)
Requirement already satisfied: ipython>=6.1.0 in /home/docs/checkouts/readthedocs.org/user_builds/analytics-core/envs/stable/lib/python3.11/site-packages (from ipywidgets>=7.0.0->cyjupyter->vuecore) (9.14.1)
Requirement already satisfied: widgetsnbextension~=4.0.14 in /home/docs/checkouts/readthedocs.org/user_builds/analytics-core/envs/stable/lib/python3.11/site-packages (from ipywidgets>=7.0.0->cyjupyter->vuecore) (4.0.15)
Requirement already satisfied: jupyterlab_widgets~=3.0.15 in /home/docs/checkouts/readthedocs.org/user_builds/analytics-core/envs/stable/lib/python3.11/site-packages (from ipywidgets>=7.0.0->cyjupyter->vuecore) (3.0.16)
Requirement already satisfied: decorator>=5.1.0 in /home/docs/checkouts/readthedocs.org/user_builds/analytics-core/envs/stable/lib/python3.11/site-packages (from ipython>=6.1.0->ipywidgets>=7.0.0->cyjupyter->vuecore) (5.3.1)
Requirement already satisfied: ipython-pygments-lexers>=1.0.0 in /home/docs/checkouts/readthedocs.org/user_builds/analytics-core/envs/stable/lib/python3.11/site-packages (from ipython>=6.1.0->ipywidgets>=7.0.0->cyjupyter->vuecore) (1.1.1)
Requirement already satisfied: jedi>=0.18.2 in /home/docs/checkouts/readthedocs.org/user_builds/analytics-core/envs/stable/lib/python3.11/site-packages (from ipython>=6.1.0->ipywidgets>=7.0.0->cyjupyter->vuecore) (0.20.0)
Requirement already satisfied: matplotlib-inline>=0.1.6 in /home/docs/checkouts/readthedocs.org/user_builds/analytics-core/envs/stable/lib/python3.11/site-packages (from ipython>=6.1.0->ipywidgets>=7.0.0->cyjupyter->vuecore) (0.2.2)
Requirement already satisfied: pexpect>4.6 in /home/docs/checkouts/readthedocs.org/user_builds/analytics-core/envs/stable/lib/python3.11/site-packages (from ipython>=6.1.0->ipywidgets>=7.0.0->cyjupyter->vuecore) (4.9.0)
Requirement already satisfied: prompt_toolkit<3.1.0,>=3.0.41 in /home/docs/checkouts/readthedocs.org/user_builds/analytics-core/envs/stable/lib/python3.11/site-packages (from ipython>=6.1.0->ipywidgets>=7.0.0->cyjupyter->vuecore) (3.0.52)
Requirement already satisfied: psutil>=7 in /home/docs/checkouts/readthedocs.org/user_builds/analytics-core/envs/stable/lib/python3.11/site-packages (from ipython>=6.1.0->ipywidgets>=7.0.0->cyjupyter->vuecore) (7.2.2)
Requirement already satisfied: pygments>=2.14.0 in /home/docs/checkouts/readthedocs.org/user_builds/analytics-core/envs/stable/lib/python3.11/site-packages (from ipython>=6.1.0->ipywidgets>=7.0.0->cyjupyter->vuecore) (2.20.0)
Requirement already satisfied: stack_data>=0.6.0 in /home/docs/checkouts/readthedocs.org/user_builds/analytics-core/envs/stable/lib/python3.11/site-packages (from ipython>=6.1.0->ipywidgets>=7.0.0->cyjupyter->vuecore) (0.6.3)
Requirement already satisfied: wcwidth in /home/docs/checkouts/readthedocs.org/user_builds/analytics-core/envs/stable/lib/python3.11/site-packages (from prompt_toolkit<3.1.0,>=3.0.41->ipython>=6.1.0->ipywidgets>=7.0.0->cyjupyter->vuecore) (0.8.1)
Requirement already satisfied: parso<0.9.0,>=0.8.6 in /home/docs/checkouts/readthedocs.org/user_builds/analytics-core/envs/stable/lib/python3.11/site-packages (from jedi>=0.18.2->ipython>=6.1.0->ipywidgets>=7.0.0->cyjupyter->vuecore) (0.8.7)
Requirement already satisfied: ptyprocess>=0.5 in /home/docs/checkouts/readthedocs.org/user_builds/analytics-core/envs/stable/lib/python3.11/site-packages (from pexpect>4.6->ipython>=6.1.0->ipywidgets>=7.0.0->cyjupyter->vuecore) (0.7.0)
Requirement already satisfied: executing>=1.2.0 in /home/docs/checkouts/readthedocs.org/user_builds/analytics-core/envs/stable/lib/python3.11/site-packages (from stack_data>=0.6.0->ipython>=6.1.0->ipywidgets>=7.0.0->cyjupyter->vuecore) (2.2.1)
Requirement already satisfied: asttokens>=2.1.0 in /home/docs/checkouts/readthedocs.org/user_builds/analytics-core/envs/stable/lib/python3.11/site-packages (from stack_data>=0.6.0->ipython>=6.1.0->ipywidgets>=7.0.0->cyjupyter->vuecore) (3.0.1)
Requirement already satisfied: pure-eval in /home/docs/checkouts/readthedocs.org/user_builds/analytics-core/envs/stable/lib/python3.11/site-packages (from stack_data>=0.6.0->ipython>=6.1.0->ipywidgets>=7.0.0->cyjupyter->vuecore) (0.2.3)
Requirement already satisfied: Flask<3.2,>=1.0.4 in /home/docs/checkouts/readthedocs.org/user_builds/analytics-core/envs/stable/lib/python3.11/site-packages (from dash->vuecore) (3.1.3)
Requirement already satisfied: Werkzeug<3.2 in /home/docs/checkouts/readthedocs.org/user_builds/analytics-core/envs/stable/lib/python3.11/site-packages (from dash->vuecore) (3.1.8)
Requirement already satisfied: importlib-metadata in /home/docs/checkouts/readthedocs.org/user_builds/analytics-core/envs/stable/lib/python3.11/site-packages (from dash->vuecore) (9.0.0)
Requirement already satisfied: retrying in /home/docs/checkouts/readthedocs.org/user_builds/analytics-core/envs/stable/lib/python3.11/site-packages (from dash->vuecore) (1.4.2)
Requirement already satisfied: nest-asyncio in /home/docs/checkouts/readthedocs.org/user_builds/analytics-core/envs/stable/lib/python3.11/site-packages (from dash->vuecore) (1.6.0)
Requirement already satisfied: setuptools in /home/docs/checkouts/readthedocs.org/user_builds/analytics-core/envs/stable/lib/python3.11/site-packages (from dash->vuecore) (82.0.1)
Requirement already satisfied: janus>=1.0.0 in /home/docs/checkouts/readthedocs.org/user_builds/analytics-core/envs/stable/lib/python3.11/site-packages (from dash->vuecore) (2.0.0)
Requirement already satisfied: blinker>=1.9.0 in /home/docs/checkouts/readthedocs.org/user_builds/analytics-core/envs/stable/lib/python3.11/site-packages (from Flask<3.2,>=1.0.4->dash->vuecore) (1.9.0)
Requirement already satisfied: click>=8.1.3 in /home/docs/checkouts/readthedocs.org/user_builds/analytics-core/envs/stable/lib/python3.11/site-packages (from Flask<3.2,>=1.0.4->dash->vuecore) (8.4.1)
Requirement already satisfied: itsdangerous>=2.2.0 in /home/docs/checkouts/readthedocs.org/user_builds/analytics-core/envs/stable/lib/python3.11/site-packages (from Flask<3.2,>=1.0.4->dash->vuecore) (2.2.0)
Requirement already satisfied: jinja2>=3.1.2 in /home/docs/checkouts/readthedocs.org/user_builds/analytics-core/envs/stable/lib/python3.11/site-packages (from Flask<3.2,>=1.0.4->dash->vuecore) (3.1.6)
Requirement already satisfied: markupsafe>=2.1.1 in /home/docs/checkouts/readthedocs.org/user_builds/analytics-core/envs/stable/lib/python3.11/site-packages (from Flask<3.2,>=1.0.4->dash->vuecore) (3.0.3)
Requirement already satisfied: openpyxl in /home/docs/checkouts/readthedocs.org/user_builds/analytics-core/envs/stable/lib/python3.11/site-packages (from dsp-pandas->acore) (3.1.5)
Requirement already satisfied: zipp>=3.20 in /home/docs/checkouts/readthedocs.org/user_builds/analytics-core/envs/stable/lib/python3.11/site-packages (from importlib-metadata->dash->vuecore) (4.1.0)
Requirement already satisfied: anndata in /home/docs/checkouts/readthedocs.org/user_builds/analytics-core/envs/stable/lib/python3.11/site-packages (from inmoose->acore) (0.12.17)
Requirement already satisfied: fastcluster in /home/docs/checkouts/readthedocs.org/user_builds/analytics-core/envs/stable/lib/python3.11/site-packages (from inmoose->acore) (1.3.0)
Requirement already satisfied: array-api-compat>=1.7.1 in /home/docs/checkouts/readthedocs.org/user_builds/analytics-core/envs/stable/lib/python3.11/site-packages (from anndata->inmoose->acore) (1.15.0)
Requirement already satisfied: h5py>=3.8 in /home/docs/checkouts/readthedocs.org/user_builds/analytics-core/envs/stable/lib/python3.11/site-packages (from anndata->inmoose->acore) (3.16.0)
Requirement already satisfied: legacy-api-wrap in /home/docs/checkouts/readthedocs.org/user_builds/analytics-core/envs/stable/lib/python3.11/site-packages (from anndata->inmoose->acore) (1.5)
Requirement already satisfied: natsort in /home/docs/checkouts/readthedocs.org/user_builds/analytics-core/envs/stable/lib/python3.11/site-packages (from anndata->inmoose->acore) (8.4.0)
Requirement already satisfied: scverse-misc>=0.0.3 in /home/docs/checkouts/readthedocs.org/user_builds/analytics-core/envs/stable/lib/python3.11/site-packages (from anndata->inmoose->acore) (0.0.3)
Requirement already satisfied: zarr!=3.0.*,>=2.18.7 in /home/docs/checkouts/readthedocs.org/user_builds/analytics-core/envs/stable/lib/python3.11/site-packages (from anndata->inmoose->acore) (3.1.6)
Requirement already satisfied: session-info2 in /home/docs/checkouts/readthedocs.org/user_builds/analytics-core/envs/stable/lib/python3.11/site-packages (from scverse-misc>=0.0.3->anndata->inmoose->acore) (0.4.1)
Requirement already satisfied: donfig>=0.8 in /home/docs/checkouts/readthedocs.org/user_builds/analytics-core/envs/stable/lib/python3.11/site-packages (from zarr!=3.0.*,>=2.18.7->anndata->inmoose->acore) (0.8.1.post1)
Requirement already satisfied: google-crc32c>=1.5 in /home/docs/checkouts/readthedocs.org/user_builds/analytics-core/envs/stable/lib/python3.11/site-packages (from zarr!=3.0.*,>=2.18.7->anndata->inmoose->acore) (1.8.0)
Requirement already satisfied: numcodecs>=0.14 in /home/docs/checkouts/readthedocs.org/user_builds/analytics-core/envs/stable/lib/python3.11/site-packages (from zarr!=3.0.*,>=2.18.7->anndata->inmoose->acore) (0.16.5)
Requirement already satisfied: pyyaml in /home/docs/checkouts/readthedocs.org/user_builds/analytics-core/envs/stable/lib/python3.11/site-packages (from donfig>=0.8->zarr!=3.0.*,>=2.18.7->anndata->inmoose->acore) (6.0.3)
Requirement already satisfied: choreographer>=1.3.0 in /home/docs/checkouts/readthedocs.org/user_builds/analytics-core/envs/stable/lib/python3.11/site-packages (from kaleido->vuecore) (1.3.0)
Requirement already satisfied: logistro>=1.0.8 in /home/docs/checkouts/readthedocs.org/user_builds/analytics-core/envs/stable/lib/python3.11/site-packages (from kaleido->vuecore) (2.0.1)
Requirement already satisfied: orjson>=3.10.15 in /home/docs/checkouts/readthedocs.org/user_builds/analytics-core/envs/stable/lib/python3.11/site-packages (from kaleido->vuecore) (3.11.9)
Requirement already satisfied: simplejson>=3.19.3 in /home/docs/checkouts/readthedocs.org/user_builds/analytics-core/envs/stable/lib/python3.11/site-packages (from choreographer>=1.3.0->kaleido->vuecore) (4.1.1)
Requirement already satisfied: regex>=2021.8.3 in /home/docs/checkouts/readthedocs.org/user_builds/analytics-core/envs/stable/lib/python3.11/site-packages (from nltk->vuecore) (2026.5.9)
Requirement already satisfied: tqdm in /home/docs/checkouts/readthedocs.org/user_builds/analytics-core/envs/stable/lib/python3.11/site-packages (from nltk->vuecore) (4.68.2)
Requirement already satisfied: et-xmlfile in /home/docs/checkouts/readthedocs.org/user_builds/analytics-core/envs/stable/lib/python3.11/site-packages (from openpyxl->dsp-pandas->acore) (2.0.0)
Requirement already satisfied: xarray in /home/docs/checkouts/readthedocs.org/user_builds/analytics-core/envs/stable/lib/python3.11/site-packages (from pandas-flavor->pingouin<0.6.0->acore) (2026.4.0)
Requirement already satisfied: typeguard in /home/docs/checkouts/readthedocs.org/user_builds/analytics-core/envs/stable/lib/python3.11/site-packages (from pandera->acore) (4.5.2)
Requirement already satisfied: typing_inspect>=0.6.0 in /home/docs/checkouts/readthedocs.org/user_builds/analytics-core/envs/stable/lib/python3.11/site-packages (from pandera->acore) (0.9.0)
Requirement already satisfied: mypy-extensions>=0.3.0 in /home/docs/checkouts/readthedocs.org/user_builds/analytics-core/envs/stable/lib/python3.11/site-packages (from typing_inspect>=0.6.0->pandera->acore) (1.1.0)
Requirement already satisfied: annotated-types>=0.6.0 in /home/docs/checkouts/readthedocs.org/user_builds/analytics-core/envs/stable/lib/python3.11/site-packages (from pydantic->vuecore) (0.7.0)
Requirement already satisfied: pydantic-core==2.46.4 in /home/docs/checkouts/readthedocs.org/user_builds/analytics-core/envs/stable/lib/python3.11/site-packages (from pydantic->vuecore) (2.46.4)
Requirement already satisfied: typing-inspection>=0.4.2 in /home/docs/checkouts/readthedocs.org/user_builds/analytics-core/envs/stable/lib/python3.11/site-packages (from pydantic->vuecore) (0.4.2)
Requirement already satisfied: jsonpickle>=1.4.1 in /home/docs/checkouts/readthedocs.org/user_builds/analytics-core/envs/stable/lib/python3.11/site-packages (from pyvis->vuecore) (4.1.2)
Requirement already satisfied: charset_normalizer<4,>=2 in /home/docs/checkouts/readthedocs.org/user_builds/analytics-core/envs/stable/lib/python3.11/site-packages (from requests->acore) (3.4.7)
Requirement already satisfied: idna<4,>=2.5 in /home/docs/checkouts/readthedocs.org/user_builds/analytics-core/envs/stable/lib/python3.11/site-packages (from requests->acore) (3.18)
Requirement already satisfied: urllib3<3,>=1.26 in /home/docs/checkouts/readthedocs.org/user_builds/analytics-core/envs/stable/lib/python3.11/site-packages (from requests->acore) (2.7.0)
Requirement already satisfied: certifi>=2023.5.7 in /home/docs/checkouts/readthedocs.org/user_builds/analytics-core/envs/stable/lib/python3.11/site-packages (from requests->acore) (2026.5.20)
Requirement already satisfied: numba>=0.51.2 in /home/docs/checkouts/readthedocs.org/user_builds/analytics-core/envs/stable/lib/python3.11/site-packages (from umap-learn->acore) (0.65.1)
Requirement already satisfied: pynndescent>=0.5 in /home/docs/checkouts/readthedocs.org/user_builds/analytics-core/envs/stable/lib/python3.11/site-packages (from umap-learn->acore) (0.6.0)
Requirement already satisfied: llvmlite<0.48,>=0.47.0dev0 in /home/docs/checkouts/readthedocs.org/user_builds/analytics-core/envs/stable/lib/python3.11/site-packages (from numba>=0.51.2->umap-learn->acore) (0.47.0)
Note: you may need to restart the kernel to use updated packages.

Hide code cell source

 
from typing import Optional

import matplotlib.pyplot as plt
import numpy as np
import pandas as pd
import scipy
import sklearn
import sklearn.impute
import sklearn.preprocessing
import vuecore.decomposition

import acore.decomposition
from acore.imputation_analysis import (
    imputation_KNN,
    imputation_mixed_norm_KNN,
    imputation_normal_distribution,
)


def plot_umap(X_scaled, y, meta_column=None, random_state=42) -> plt.Axes:
    """Fit and plot UMAP embedding with two components with colors defined by meta_column."""
    embedding = acore.decomposition.umap.run_umap(
        X_scaled, y, random_state=random_state
    )
    if meta_column is None:
        meta_column = y.name
    ax = embedding.plot.scatter("UMAP 1", "UMAP 2", c=meta_column, cmap="Paired")
    return ax


def standard_normalize(X: pd.DataFrame) -> pd.DataFrame:
    """Standard normalize data and keep indices of DataFrame."""
    X_scaled = (
        sklearn.preprocessing.StandardScaler()
        .set_output(transform="pandas")
        .fit_transform(X)
    )
    return X_scaled


def median_impute(X: pd.DataFrame) -> pd.DataFrame:
    X_imputed = (
        sklearn.impute.SimpleImputer(strategy="median")
        .set_output(transform="pandas")
        .fit_transform(X)
    )
    return X_imputed


def run_and_plot_pca(
    X_scaled,
    y,
    meta_column: Optional[str] = None,
    n_components: int = 4,
) -> tuple[pd.DataFrame, plt.Figure]:
    PCs, _ = acore.decomposition.pca.run_pca(X_scaled, n_components=n_components)
    PCs.columns = [s.replace("principal component", "PC") for s in PCs.columns]
    fig = vuecore.decomposition.pca_grid(
        PCs=PCs, meta_column=y, n_components=n_components, meta_col_name=meta_column
    )
    return PCs, fig

Set some parameters#

BASE = (
    "https://raw.githubusercontent.com/Multiomics-Analytics-Group/acore/"
    "main/example_data/alzheimer_proteomics/"
)
# data is already preprocessed: log2, filtered
fname: str = "alzheimer_example_omics_and_clinic.csv"  # combined omics and meta data
covariates: list[str] = ["age", "male"]
group: str = "collection_site"
subject_col: str = "Sample ID"
drop_cols: list[str] = ["AD"]
factor_and_covars: list[str] = [group, *covariates]
group_label: Optional[str] = "site"  # optional: rename target variable

Data loading#

Use combined dataset for ANCOVA analysis.

Hide code cell source

 
omics_and_meta = pd.read_csv(f"{BASE}/{fname}", index_col=subject_col).convert_dtypes()
omics_and_meta
AD age male collection_site Q6UX72 O14773 A0A0A0MQU6 P36222 P51693-2 P17174 ... A0A075B6K4 O15041 J3KNA1 A0A0C4DH33 P16870 G3V533 Q9Y5I4 P55283 A1L4H1 Q7Z4T9
Sample ID
Sample_000 0 71 0 Sweden 16.047 18.412 16.381 20.948 18.658 20.232 ... 16.149 14.013 20.549 14.269 20.468 18.448 17.187 17.422 15.542 19.331
Sample_001 1 77 1 Sweden 14.457 17.869 16.196 21.083 18.446 19.776 ... 16.127 13.916 15.854 14.379 19.902 17.723 17.447 17.097 15.734 18.980
Sample_002 1 75 1 Sweden 15.631 17.662 16.071 21.206 18.967 20.066 ... 15.387 13.903 17.576 13.675 19.619 17.006 17.410 17.752 15.824 19.326
Sample_003 1 72 0 Sweden 16.204 18.437 16.356 20.729 18.798 20.195 ... 16.565 14.526 18.173 <NA> 20.170 17.212 17.545 17.483 15.515 18.953
Sample_004 1 63 0 Sweden 15.968 18.577 16.001 21.068 18.422 20.485 ... 16.418 14.933 15.440 <NA> 19.987 17.624 17.297 17.172 15.334 18.651
... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ...
Sample_205 1 69 0 Berlin 15.262 18.046 16.358 21.321 18.580 19.838 ... 15.350 13.572 13.482 <NA> 19.984 15.269 17.104 16.952 15.705 18.844
Sample_206 0 73 1 Berlin <NA> 16.573 16.099 20.663 19.191 18.388 ... 16.582 9.748 14.372 15.567 19.396 16.976 17.109 18.056 15.282 18.686
Sample_207 0 71 0 Berlin 15.463 17.991 16.062 20.770 19.050 19.361 ... 15.768 13.241 13.931 15.092 19.923 16.669 16.938 17.248 14.874 19.146
Sample_208 0 83 1 Berlin 15.786 17.216 15.929 20.938 18.216 19.183 ... 17.560 14.442 <NA> 14.267 19.831 16.258 17.155 16.353 15.471 16.853
Sample_209 0 63 0 Berlin 15.691 <NA> 15.914 20.366 19.308 19.534 ... 16.338 13.628 <NA> 13.051 19.427 14.848 16.776 16.597 14.699 18.087

197 rows × 104 columns

Separate omics and the grouping variable

Hide code cell source

 
omics = omics_and_meta.drop(columns=[*factor_and_covars, *drop_cols])
na_counts = omics.isna().sum().sort_values(ascending=False)

fig, axes = plt.subplots(1, 2, figsize=(8, 4), constrained_layout=True)

na_counts.plot(
    ax=axes[0],
    rot=45,
    style=".",
    alpha=0.5,
    ylabel=f"Number of missing values of {omics.shape[0]} samples",
    title="Missing values (count)",
)

ax1 = axes[1]
ax2 = ax1.twinx()

(na_counts / omics.shape[0]).plot(
    ax=ax1,
    rot=45,
    style=".",
    alpha=0.5,
    color="C0",
    ylabel="Ratio of missing values",
    title="Missing values & Completeness (ratios)",
)
not_nan_counts = omics.notna().sum().sort_values(ascending=False)
(not_nan_counts / omics.shape[0]).plot(
    ax=ax2,
    rot=45,
    style=".",
    alpha=0.0,
    color="C1",
    ylabel="Ratio of non-missing values\n(completeness)",
)
ax1.tick_params(axis="y", labelcolor="C0")
ax2.tick_params(axis="y", labelcolor="C1")
../_images/533cce71d42c98a26c202a20f990f017a4c9e9023642c800e7b3859f2c20ddc6.png

Show samples and features with at least 3 missing values

Hide code cell source

 
omics_and_y = omics_and_meta.drop(columns=[*covariates, *drop_cols])
assert omics_and_meta.notna().any(axis=None), "Nothing to impute"
omics_and_y.loc[omics_and_y.isna().sum(axis=1) >= 3].loc[
    :, omics_and_y.isna().sum(axis=0) >= 3
]
Q6UX72 O14773 A0A0A0MQU6 Q9BWS9 A0A0B4J2D9 Q13433 P01258 D6RJG0 P48745 Q10472 ... O60279 P69905 Q9BXJ3 A0A075B6K4 O15041 J3KNA1 A0A0C4DH33 G3V533 A1L4H1 Q7Z4T9
Sample ID
Sample_000 16.047 18.412 16.381 15.500 15.408 14.999 14.796 <NA> 17.516 16.143 ... 18.440 17.685 16.828 16.149 14.013 20.549 14.269 18.448 15.542 19.331
Sample_001 14.457 17.869 16.196 14.760 <NA> 14.374 15.063 <NA> <NA> 16.453 ... 18.305 17.978 16.793 16.127 13.916 15.854 14.379 17.723 15.734 18.980
Sample_002 15.631 17.662 16.071 <NA> 15.362 15.121 14.219 16.359 16.870 16.097 ... 18.484 21.023 17.229 15.387 13.903 17.576 13.675 17.006 15.824 19.326
Sample_003 16.204 18.437 16.356 15.300 <NA> 14.798 14.424 15.548 17.006 16.311 ... 18.381 16.445 16.886 16.565 14.526 18.173 <NA> 17.212 15.515 18.953
Sample_004 15.968 18.577 16.001 16.054 <NA> 15.097 14.190 <NA> 17.316 16.188 ... 18.472 23.001 16.946 16.418 14.933 15.440 <NA> 17.624 15.334 18.651
... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ...
Sample_199 15.913 17.399 15.685 15.444 <NA> 12.306 <NA> <NA> 17.841 16.002 ... 18.182 20.355 16.373 16.755 13.182 14.236 <NA> 16.024 14.467 18.661
Sample_200 13.295 <NA> <NA> 15.574 15.306 <NA> <NA> 14.257 16.625 16.677 ... 17.712 <NA> <NA> 18.582 <NA> <NA> 16.537 17.281 15.432 18.273
Sample_201 14.973 17.537 16.355 15.645 19.319 <NA> 13.310 16.158 16.649 16.946 ... 17.967 16.806 17.354 18.517 <NA> <NA> 16.568 17.980 15.268 19.282
Sample_206 <NA> 16.573 16.099 16.026 15.503 <NA> <NA> 16.283 17.283 16.446 ... 18.202 15.514 17.000 16.582 9.748 14.372 15.567 16.976 15.282 18.686
Sample_209 15.691 <NA> 15.914 15.653 13.784 13.923 13.667 16.084 16.659 16.680 ... 18.442 15.276 16.390 16.338 13.628 <NA> 13.051 14.848 14.699 18.087

132 rows × 58 columns

KNN imputation#

Can be generally applied

  • both by group and overall

  • returns the imputed data, per default only features that meet the criteria based on the selected cutoff for the fraction of non-missing values for a single feature (e.g. protein group).

  • setting alone=False will ensure that all features, imputed or not, are returned. This can be useful for downstream analysis where you want to keep all features, but only impute those that meet a minimal quality criteria.

overall#

cutoff = 0.60
omics_and_y_imputed = imputation_KNN(
    data=omics_and_y,
    drop_cols=[],
    group=None,
    cutoff=cutoff,
    alone=False,
)
assert omics_and_y_imputed.isna().sum().sum() == 0
omics_and_y_imputed
Q6UX72 O14773 A0A0A0MQU6 P36222 P51693-2 P17174 Q9BWS9 A0A0B4J2D9 P00734 Q13433 ... O15041 J3KNA1 A0A0C4DH33 P16870 G3V533 Q9Y5I4 P55283 A1L4H1 Q7Z4T9 collection_site
Sample ID
Sample_000 16.047 18.412 16.381 20.948 18.658 20.232 15.500 15.408 19.870 14.999 ... 14.013 20.549 14.269 20.468 18.448 17.187 17.422 15.542 19.331 Sweden
Sample_001 14.457 17.869 16.196 21.083 18.446 19.776 14.760 16.981 20.338 14.374 ... 13.916 15.854 14.379 19.902 17.723 17.447 17.097 15.734 18.980 Sweden
Sample_002 15.631 17.662 16.071 21.206 18.967 20.066 14.635 15.362 19.814 15.121 ... 13.903 17.576 13.675 19.619 17.006 17.410 17.752 15.824 19.326 Sweden
Sample_003 16.204 18.437 16.356 20.729 18.798 20.195 15.300 15.725 20.078 14.798 ... 14.526 18.173 14.462 20.170 17.212 17.545 17.483 15.515 18.953 Sweden
Sample_004 15.968 18.577 16.001 21.068 18.422 20.485 16.054 15.833 19.786 15.097 ... 14.933 15.440 14.163 19.987 17.624 17.297 17.172 15.334 18.651 Sweden
... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ...
Sample_205 15.262 18.046 16.358 21.321 18.580 19.838 14.942 14.204 20.530 14.518 ... 13.572 13.482 16.096 19.984 15.269 17.104 16.952 15.705 18.844 Berlin
Sample_206 15.217 16.573 16.099 20.663 19.191 18.388 16.026 15.503 21.106 13.421 ... 9.748 14.372 15.567 19.396 16.976 17.109 18.056 15.282 18.686 Berlin
Sample_207 15.463 17.991 16.062 20.770 19.050 19.361 15.551 16.418 20.477 13.842 ... 13.241 13.931 15.092 19.923 16.669 16.938 17.248 14.874 19.146 Berlin
Sample_208 15.786 17.216 15.929 20.938 18.216 19.183 15.176 14.104 20.483 13.929 ... 14.442 13.887 14.267 19.831 16.258 17.155 16.353 15.471 16.853 Berlin
Sample_209 15.691 17.468 15.914 20.366 19.308 19.534 15.653 13.784 21.183 13.923 ... 13.628 13.919 13.051 19.427 14.848 16.776 16.597 14.699 18.087 Berlin

197 rows × 101 columns

As we have increased the threshold cutoff for the fraction of non-missing values per feature, more features will not be imputed and therefore have missing values.

cutoff = 0.90
omics_and_y_imputed = imputation_KNN(
    data=omics_and_y,
    drop_cols=[],
    group=None,
    cutoff=cutoff,
    alone=False,
)
n_still_missing = omics_and_y_imputed.isna().sum().sum()
print(f"Still missing features with cutoff of {cutoff}: {n_still_missing}")

Still missing features with cutoff of 0.9: 892

Keep only imputed features#

Use the alone=True to only keep the imputed features. It is the default.

cutoff = 0.90
omics_and_y_imputed = imputation_KNN(
    data=omics_and_y,
    drop_cols=[],
    group=None,
    cutoff=cutoff,
    alone=True,
)
assert omics_and_y_imputed.isna().sum().sum() == 0
print("Shape of of input data: ", omics_and_y.shape)
print("Shape of imputed data: ", omics_and_y_imputed.shape)

Shape of of input data:  (197, 101)
Shape of imputed data:  (197, 77)

By group#

Do the imputation separately for each group (e.g. target vs control) and then combine the results.

Let’s see the ratio of missing (left y-axis) and of non-missing (right y-axis) values per feature (e.g. protein group) for which no missing values for each group:

Hide code cell source

 
frac_na_by_group = (
    omics_and_y.groupby(
        group,
    )
    .apply(lambda x: x.isna().sum() / x.shape[0], include_groups=False)
    .T
).sort_values(by="Berlin")
frac_non_na_by_group = (
    omics_and_y.groupby(group)
    .apply(lambda x: x.notna().sum() / x.shape[0], include_groups=False)
    .T
).sort_values(by="Berlin")
fig, ax = plt.subplots(1, 1, figsize=(6, 4), constrained_layout=True)
ax2 = ax.twinx()

for g in frac_na_by_group.columns:
    frac_na_by_group[g].plot(
        ax=ax,
        rot=45,
        style=".",
        alpha=0.5,
        label=g,
        ylabel="Ratio of missing values",
        title="Missing values & Completeness (ratios)",
    )
    frac_non_na_by_group[g].plot(
        ax=ax2,
        rot=45,
        style=".",
        alpha=0.0,
        ylabel="Ratio of non-missing values\n(completeness)",
    )
ax.legend(loc="upper left")
ax.set(xlabel="Protein Groups (sorted by fraction of missing values in Berlin)")
ax.tick_params(axis="y", labelcolor="C0")
ax2.tick_params(axis="y", labelcolor="C1")
../_images/52849aefcf992a6c6adbede2bb34c14d11702803b368c3ef39d444c22ae3f436.png

Clearly some protein groups only have missing values if combined from a certain collection site, and that the ratio can be different in each group. Therefore, imputation by KNN for a threshold of non-missing values per feature (e.g. protein group) per group can be a good option.

omics_and_y_imputed = imputation_KNN(
    data=omics_and_y,
    drop_cols=[],
    group=group,
    cutoff=0.65,  # selected to leave some missing values for demonstration
    alone=False,
)
omics_and_y_imputed.isna().sum().value_counts().sort_index()

0    87
9     1
22    1
26    2
27    2
29    1
33    1
37    2
38    2
39    1
41    1
Name: count, dtype: int64

If we look at the number of missing values still remaining by collection site, we see that Sweden has most of these missing values due to a higher fraction of missing values for these.

Hide code cell source

 
omics_and_y_imputed.groupby(group).apply(
    lambda x: x.isna().sum(), include_groups=False
).T.value_counts().sort_index()

Berlin  Kiel  Magdeburg  Sweden
0       0     0          0        86
                         22        1
                         26        2
                         27        2
                         29        1
                         33        1
                         37        2
                         38        2
                         39        1
        9     0          0         1
41      0     0          0         1
Name: count, dtype: int64

As we increase the threshold cutoff for the fraction of non-missing values per feature, more features will not be imputed and therefore have missing values.

omics_and_y_imputed = imputation_KNN(
    data=omics_and_y,
    drop_cols=[],
    group=group,
    cutoff=0.90,
    alone=False,
)
n_still_missing = omics_and_y_imputed.isna().sum().sum()
print(f"Still missing features with cutoff of {cutoff}: {n_still_missing}")

Still missing features with cutoff of 0.9: 927

Imputation from a shifted normal distribution per sample#

Specific to massspectrometry based data in log2 space: normal distributed data, with detection limit (if that applies it can be used)

  • based on mean and standard deviation missing values are replaced by drawing random values from a shifted normal distribution

  • assumption is that missing values are due to falling below the detection limit which can be revealed by the distribution of intensities

Below you find a generated example highlighting the idea

Hide code cell source

 
mu = 25.0
stddev = 1.0

x = np.linspace(mu - 3, mu + 3, num=101)

y_normal = scipy.stats.norm.pdf(x, loc=mu, scale=stddev)

mu_shifted = mu - (1.8 * stddev)
stddev_shifted = 0.3 * stddev
print(f"Downshifted: {mu_shifted = }, {stddev_shifted = }")
y_impute = scipy.stats.norm.pdf(x, loc=mu - (1.8 * stddev), scale=0.3 * stddev)

fig, ax = plt.subplots(1, 1, figsize=(5, 4))

for i, (y, label) in enumerate(zip([y_normal, y_impute], ["original", "down shifted"])):
    ax.plot(x, y, color=f"C{i}", label=label)
    ax.fill_between(x, y, color=f"C{i}", alpha=0.5)
    ax.set_label(label)
ax.set_xlabel("log2 intensity distribution in sample")
ax.set_ylabel("density")
ax.legend()
fig.tight_layout()

Downshifted: mu_shifted = 23.2, stddev_shifted = 0.3
../_images/b1b5d418f5ca15328a7118400eb078e7b8d1f018c3b609a45389580ec3bd10e7.png

This idea can be applied on a per sample basis using:

# does not account for groups as it is done on a per sample basis (along columns)
imputation_normal_distribution(
    data=omics_and_y,
    drop_cols=[group],
)
Q6UX72 O14773 A0A0A0MQU6 P36222 P51693-2 P17174 Q9BWS9 A0A0B4J2D9 P00734 Q13433 ... A0A075B6K4 O15041 J3KNA1 A0A0C4DH33 P16870 G3V533 Q9Y5I4 P55283 A1L4H1 Q7Z4T9
Sample ID
Sample_000 16.047 18.412 16.381 20.948 18.658 20.232 15.500 15.408 19.870 14.999 ... 16.149 14.013 20.549 14.269 20.468 18.448 17.187 17.422 15.542 19.331
Sample_001 14.457 17.869 16.196 21.083 18.446 19.776 14.760 11.019 20.338 14.374 ... 16.127 13.916 15.854 14.379 19.902 17.723 17.447 17.097 15.734 18.980
Sample_002 15.631 17.662 16.071 21.206 18.967 20.066 12.206 15.362 19.814 15.121 ... 15.387 13.903 17.576 13.675 19.619 17.006 17.410 17.752 15.824 19.326
Sample_003 16.204 18.437 16.356 20.729 18.798 20.195 15.300 12.219 20.078 14.798 ... 16.565 14.526 18.173 12.783 20.170 17.212 17.545 17.483 15.515 18.953
Sample_004 15.968 18.577 16.001 21.068 18.422 20.485 16.054 12.687 19.786 15.097 ... 16.418 14.933 15.440 12.722 19.987 17.624 17.297 17.172 15.334 18.651
... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ...
Sample_205 15.262 18.046 16.358 21.321 18.580 19.838 14.942 14.204 20.530 14.518 ... 15.350 13.572 13.482 12.086 19.984 15.269 17.104 16.952 15.705 18.844
Sample_206 14.085 16.573 16.099 20.663 19.191 18.388 16.026 15.503 21.106 11.743 ... 16.582 9.748 14.372 15.567 19.396 16.976 17.109 18.056 15.282 18.686
Sample_207 15.463 17.991 16.062 20.770 19.050 19.361 15.551 12.091 20.477 13.842 ... 15.768 13.241 13.931 15.092 19.923 16.669 16.938 17.248 14.874 19.146
Sample_208 15.786 17.216 15.929 20.938 18.216 19.183 15.176 14.104 20.483 13.929 ... 17.560 14.442 10.987 14.267 19.831 16.258 17.155 16.353 15.471 16.853
Sample_209 15.691 11.367 15.914 20.366 19.308 19.534 15.653 13.784 21.183 13.923 ... 16.338 13.628 9.967 13.051 19.427 14.848 16.776 16.597 14.699 18.087

197 rows × 100 columns

Note that using this type of imputation before differential regulation can lead to false positive and negative results. If values are not due to assumed missing mechanism (Missing not-at-random due to low abundance), but are due to technical noise, these values should not be replaced.

Therefore, in proteomics, many use a combined approach Santos et al., 2020:

Combining KNN based imputation and random imputation from a shifted random distribution#

  • For features (e.g. protein groups) that are present across groups in high enough frequency, use KNN-based imputation (which is deterministic)

  • for the remaining missing values, use - based on the distribution of observed values in a sample - a shifted normal distribution to draw replacements (random, but deterministic due to the set seed).

See the methods section of Santos et al., 2020 for more details.

imputation_mixed_norm_KNN(data=omics_and_y, drop_cols=[], group=group, cutoff=0.9)
Q6UX72 O14773 A0A0A0MQU6 P36222 P51693-2 P17174 Q9BWS9 A0A0B4J2D9 P00734 Q13433 ... A0A075B6K4 O15041 J3KNA1 A0A0C4DH33 P16870 G3V533 Q9Y5I4 P55283 A1L4H1 Q7Z4T9
Sample ID
Sample_000 16.047 18.412 16.381 20.948 18.658 20.232 15.500 15.408 19.870 14.999 ... 16.149 14.013 20.549 14.269 20.468 18.448 17.187 17.422 15.542 19.331
Sample_001 14.457 17.869 16.196 21.083 18.446 19.776 14.760 12.534 20.338 14.374 ... 16.127 13.916 15.854 14.379 19.902 17.723 17.447 17.097 15.734 18.980
Sample_002 15.631 17.662 16.071 21.206 18.967 20.066 11.854 15.362 19.814 15.121 ... 15.387 13.903 17.576 13.675 19.619 17.006 17.410 17.752 15.824 19.326
Sample_003 16.204 18.437 16.356 20.729 18.798 20.195 15.300 12.244 20.078 14.798 ... 16.565 14.526 18.173 11.338 20.170 17.212 17.545 17.483 15.515 18.953
Sample_004 15.968 18.577 16.001 21.068 18.422 20.485 16.054 12.632 19.786 15.097 ... 16.418 14.933 15.440 11.061 19.987 17.624 17.297 17.172 15.334 18.651
... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ...
Sample_205 15.262 18.046 16.358 21.321 18.580 19.838 14.942 14.204 20.530 14.518 ... 15.350 13.572 13.482 12.789 19.984 15.269 17.104 16.952 15.705 18.844
Sample_206 15.474 16.573 16.099 20.663 19.191 18.388 16.026 15.503 21.106 12.437 ... 16.582 9.748 14.372 15.567 19.396 16.976 17.109 18.056 15.282 18.686
Sample_207 15.463 17.991 16.062 20.770 19.050 19.361 15.551 15.776 20.477 13.842 ... 15.768 13.241 13.931 15.092 19.923 16.669 16.938 17.248 14.874 19.146
Sample_208 15.786 17.216 15.929 20.938 18.216 19.183 15.176 14.104 20.483 13.929 ... 17.560 14.442 12.303 14.267 19.831 16.258 17.155 16.353 15.471 16.853
Sample_209 15.691 12.638 15.914 20.366 19.308 19.534 15.653 13.784 21.183 13.923 ... 16.338 13.628 11.662 13.051 19.427 14.848 16.776 16.597 14.699 18.087

197 rows × 100 columns

done.