Normalization of samples

Normalization of samples#

We will explore an Alzheimer dataset where the data was collected in four different sites. We will see that the sites have a an effect where the data is in principal component space and in UMAP space. We will then normalize the data and see how the effect on these plots.

Refers to the acore.normalization module.

%pip install acore vuecore

Hide code cell output

 
Requirement already satisfied: acore in /home/docs/checkouts/readthedocs.org/user_builds/analytics-core/envs/stable/lib/python3.11/site-packages (0.3.0)
Requirement already satisfied: vuecore in /home/docs/checkouts/readthedocs.org/user_builds/analytics-core/envs/stable/lib/python3.11/site-packages (0.1.1)
Requirement already satisfied: numpy in /home/docs/checkouts/readthedocs.org/user_builds/analytics-core/envs/stable/lib/python3.11/site-packages (from acore) (2.4.6)
Requirement already satisfied: pandas in /home/docs/checkouts/readthedocs.org/user_builds/analytics-core/envs/stable/lib/python3.11/site-packages (from acore) (2.3.3)
Requirement already satisfied: dsp-pandas in /home/docs/checkouts/readthedocs.org/user_builds/analytics-core/envs/stable/lib/python3.11/site-packages (from acore) (0.0.5)
Requirement already satisfied: scipy in /home/docs/checkouts/readthedocs.org/user_builds/analytics-core/envs/stable/lib/python3.11/site-packages (from acore) (1.17.1)
Requirement already satisfied: scikit-learn>=1.5 in /home/docs/checkouts/readthedocs.org/user_builds/analytics-core/envs/stable/lib/python3.11/site-packages (from acore) (1.9.0)
Requirement already satisfied: gseapy!=1.1.5 in /home/docs/checkouts/readthedocs.org/user_builds/analytics-core/envs/stable/lib/python3.11/site-packages (from acore) (1.2.1)
Requirement already satisfied: pingouin<0.6.0 in /home/docs/checkouts/readthedocs.org/user_builds/analytics-core/envs/stable/lib/python3.11/site-packages (from acore) (0.5.5)
Requirement already satisfied: umap-learn in /home/docs/checkouts/readthedocs.org/user_builds/analytics-core/envs/stable/lib/python3.11/site-packages (from acore) (0.5.12)
Requirement already satisfied: statsmodels in /home/docs/checkouts/readthedocs.org/user_builds/analytics-core/envs/stable/lib/python3.11/site-packages (from acore) (0.14.6)
Requirement already satisfied: combat in /home/docs/checkouts/readthedocs.org/user_builds/analytics-core/envs/stable/lib/python3.11/site-packages (from acore) (0.3.3)
Requirement already satisfied: inmoose in /home/docs/checkouts/readthedocs.org/user_builds/analytics-core/envs/stable/lib/python3.11/site-packages (from acore) (0.9.1)
Requirement already satisfied: requests in /home/docs/checkouts/readthedocs.org/user_builds/analytics-core/envs/stable/lib/python3.11/site-packages (from acore) (2.34.2)
Requirement already satisfied: rarfile in /home/docs/checkouts/readthedocs.org/user_builds/analytics-core/envs/stable/lib/python3.11/site-packages (from acore) (4.2)
Requirement already satisfied: pandera in /home/docs/checkouts/readthedocs.org/user_builds/analytics-core/envs/stable/lib/python3.11/site-packages (from acore) (0.31.1)
Requirement already satisfied: matplotlib in /home/docs/checkouts/readthedocs.org/user_builds/analytics-core/envs/stable/lib/python3.11/site-packages (from pingouin<0.6.0->acore) (3.11.0)
Requirement already satisfied: pandas-flavor in /home/docs/checkouts/readthedocs.org/user_builds/analytics-core/envs/stable/lib/python3.11/site-packages (from pingouin<0.6.0->acore) (0.8.1)
Requirement already satisfied: seaborn in /home/docs/checkouts/readthedocs.org/user_builds/analytics-core/envs/stable/lib/python3.11/site-packages (from pingouin<0.6.0->acore) (0.13.2)
Requirement already satisfied: tabulate in /home/docs/checkouts/readthedocs.org/user_builds/analytics-core/envs/stable/lib/python3.11/site-packages (from pingouin<0.6.0->acore) (0.10.0)
Requirement already satisfied: plotly in /home/docs/checkouts/readthedocs.org/user_builds/analytics-core/envs/stable/lib/python3.11/site-packages (from vuecore) (6.8.0)
Requirement already satisfied: beautifulsoup4 in /home/docs/checkouts/readthedocs.org/user_builds/analytics-core/envs/stable/lib/python3.11/site-packages (from vuecore) (4.15.0)
Requirement already satisfied: dash in /home/docs/checkouts/readthedocs.org/user_builds/analytics-core/envs/stable/lib/python3.11/site-packages (from vuecore) (4.2.0)
Requirement already satisfied: networkx in /home/docs/checkouts/readthedocs.org/user_builds/analytics-core/envs/stable/lib/python3.11/site-packages (from vuecore) (3.6.1)
Requirement already satisfied: kaleido in /home/docs/checkouts/readthedocs.org/user_builds/analytics-core/envs/stable/lib/python3.11/site-packages (from vuecore) (1.3.0)
Requirement already satisfied: pyvis in /home/docs/checkouts/readthedocs.org/user_builds/analytics-core/envs/stable/lib/python3.11/site-packages (from vuecore) (0.3.2)
Requirement already satisfied: wordcloud in /home/docs/checkouts/readthedocs.org/user_builds/analytics-core/envs/stable/lib/python3.11/site-packages (from vuecore) (1.9.6)
Requirement already satisfied: cyjupyter in /home/docs/checkouts/readthedocs.org/user_builds/analytics-core/envs/stable/lib/python3.11/site-packages (from vuecore) (0.2.0)
Requirement already satisfied: nltk in /home/docs/checkouts/readthedocs.org/user_builds/analytics-core/envs/stable/lib/python3.11/site-packages (from vuecore) (3.9.4)
Requirement already satisfied: webweb in /home/docs/checkouts/readthedocs.org/user_builds/analytics-core/envs/stable/lib/python3.11/site-packages (from vuecore) (0.0.37)
Requirement already satisfied: dash-cytoscape in /home/docs/checkouts/readthedocs.org/user_builds/analytics-core/envs/stable/lib/python3.11/site-packages (from vuecore) (1.0.2)
Requirement already satisfied: pydantic in /home/docs/checkouts/readthedocs.org/user_builds/analytics-core/envs/stable/lib/python3.11/site-packages (from vuecore) (2.13.4)
Requirement already satisfied: nbformat>=4.2.0 in /home/docs/checkouts/readthedocs.org/user_builds/analytics-core/envs/stable/lib/python3.11/site-packages (from vuecore) (5.10.4)
Requirement already satisfied: contourpy>=1.0.1 in /home/docs/checkouts/readthedocs.org/user_builds/analytics-core/envs/stable/lib/python3.11/site-packages (from matplotlib->pingouin<0.6.0->acore) (1.3.3)
Requirement already satisfied: cycler>=0.10 in /home/docs/checkouts/readthedocs.org/user_builds/analytics-core/envs/stable/lib/python3.11/site-packages (from matplotlib->pingouin<0.6.0->acore) (0.12.1)
Requirement already satisfied: fonttools>=4.22.0 in /home/docs/checkouts/readthedocs.org/user_builds/analytics-core/envs/stable/lib/python3.11/site-packages (from matplotlib->pingouin<0.6.0->acore) (4.63.0)
Requirement already satisfied: kiwisolver>=1.3.1 in /home/docs/checkouts/readthedocs.org/user_builds/analytics-core/envs/stable/lib/python3.11/site-packages (from matplotlib->pingouin<0.6.0->acore) (1.5.0)
Requirement already satisfied: packaging>=20.0 in /home/docs/checkouts/readthedocs.org/user_builds/analytics-core/envs/stable/lib/python3.11/site-packages (from matplotlib->pingouin<0.6.0->acore) (26.2)
Requirement already satisfied: pillow>=9 in /home/docs/checkouts/readthedocs.org/user_builds/analytics-core/envs/stable/lib/python3.11/site-packages (from matplotlib->pingouin<0.6.0->acore) (12.2.0)
Requirement already satisfied: pyparsing>=3 in /home/docs/checkouts/readthedocs.org/user_builds/analytics-core/envs/stable/lib/python3.11/site-packages (from matplotlib->pingouin<0.6.0->acore) (3.3.2)
Requirement already satisfied: python-dateutil>=2.7 in /home/docs/checkouts/readthedocs.org/user_builds/analytics-core/envs/stable/lib/python3.11/site-packages (from matplotlib->pingouin<0.6.0->acore) (2.9.0.post0)
Requirement already satisfied: fastjsonschema>=2.15 in /home/docs/checkouts/readthedocs.org/user_builds/analytics-core/envs/stable/lib/python3.11/site-packages (from nbformat>=4.2.0->vuecore) (2.21.2)
Requirement already satisfied: jsonschema>=2.6 in /home/docs/checkouts/readthedocs.org/user_builds/analytics-core/envs/stable/lib/python3.11/site-packages (from nbformat>=4.2.0->vuecore) (4.26.0)
Requirement already satisfied: jupyter-core!=5.0.*,>=4.12 in /home/docs/checkouts/readthedocs.org/user_builds/analytics-core/envs/stable/lib/python3.11/site-packages (from nbformat>=4.2.0->vuecore) (5.9.1)
Requirement already satisfied: traitlets>=5.1 in /home/docs/checkouts/readthedocs.org/user_builds/analytics-core/envs/stable/lib/python3.11/site-packages (from nbformat>=4.2.0->vuecore) (5.15.1)
Requirement already satisfied: attrs>=22.2.0 in /home/docs/checkouts/readthedocs.org/user_builds/analytics-core/envs/stable/lib/python3.11/site-packages (from jsonschema>=2.6->nbformat>=4.2.0->vuecore) (26.1.0)
Requirement already satisfied: jsonschema-specifications>=2023.03.6 in /home/docs/checkouts/readthedocs.org/user_builds/analytics-core/envs/stable/lib/python3.11/site-packages (from jsonschema>=2.6->nbformat>=4.2.0->vuecore) (2025.9.1)
Requirement already satisfied: referencing>=0.28.4 in /home/docs/checkouts/readthedocs.org/user_builds/analytics-core/envs/stable/lib/python3.11/site-packages (from jsonschema>=2.6->nbformat>=4.2.0->vuecore) (0.37.0)
Requirement already satisfied: rpds-py>=0.25.0 in /home/docs/checkouts/readthedocs.org/user_builds/analytics-core/envs/stable/lib/python3.11/site-packages (from jsonschema>=2.6->nbformat>=4.2.0->vuecore) (2026.5.1)
Requirement already satisfied: platformdirs>=2.5 in /home/docs/checkouts/readthedocs.org/user_builds/analytics-core/envs/stable/lib/python3.11/site-packages (from jupyter-core!=5.0.*,>=4.12->nbformat>=4.2.0->vuecore) (4.10.0)
Requirement already satisfied: pytz>=2020.1 in /home/docs/checkouts/readthedocs.org/user_builds/analytics-core/envs/stable/lib/python3.11/site-packages (from pandas->acore) (2026.2)
Requirement already satisfied: tzdata>=2022.7 in /home/docs/checkouts/readthedocs.org/user_builds/analytics-core/envs/stable/lib/python3.11/site-packages (from pandas->acore) (2026.2)
Requirement already satisfied: six>=1.5 in /home/docs/checkouts/readthedocs.org/user_builds/analytics-core/envs/stable/lib/python3.11/site-packages (from python-dateutil>=2.7->matplotlib->pingouin<0.6.0->acore) (1.17.0)
Requirement already satisfied: typing-extensions>=4.4.0 in /home/docs/checkouts/readthedocs.org/user_builds/analytics-core/envs/stable/lib/python3.11/site-packages (from referencing>=0.28.4->jsonschema>=2.6->nbformat>=4.2.0->vuecore) (4.15.0)
Requirement already satisfied: joblib>=1.4.0 in /home/docs/checkouts/readthedocs.org/user_builds/analytics-core/envs/stable/lib/python3.11/site-packages (from scikit-learn>=1.5->acore) (1.5.3)
Requirement already satisfied: narwhals>=2.0.1 in /home/docs/checkouts/readthedocs.org/user_builds/analytics-core/envs/stable/lib/python3.11/site-packages (from scikit-learn>=1.5->acore) (2.22.1)
Requirement already satisfied: threadpoolctl>=3.5.0 in /home/docs/checkouts/readthedocs.org/user_builds/analytics-core/envs/stable/lib/python3.11/site-packages (from scikit-learn>=1.5->acore) (3.6.0)
Requirement already satisfied: soupsieve>=1.6.1 in /home/docs/checkouts/readthedocs.org/user_builds/analytics-core/envs/stable/lib/python3.11/site-packages (from beautifulsoup4->vuecore) (2.8.4)
Requirement already satisfied: mpmath>=1.1.0 in /home/docs/checkouts/readthedocs.org/user_builds/analytics-core/envs/stable/lib/python3.11/site-packages (from combat->acore) (1.4.1)
Requirement already satisfied: patsy>=0.5.1 in /home/docs/checkouts/readthedocs.org/user_builds/analytics-core/envs/stable/lib/python3.11/site-packages (from combat->acore) (1.0.2)
Requirement already satisfied: ipywidgets>=7.0.0 in /home/docs/checkouts/readthedocs.org/user_builds/analytics-core/envs/stable/lib/python3.11/site-packages (from cyjupyter->vuecore) (8.1.8)
Requirement already satisfied: comm>=0.1.3 in /home/docs/checkouts/readthedocs.org/user_builds/analytics-core/envs/stable/lib/python3.11/site-packages (from ipywidgets>=7.0.0->cyjupyter->vuecore) (0.2.3)
Requirement already satisfied: ipython>=6.1.0 in /home/docs/checkouts/readthedocs.org/user_builds/analytics-core/envs/stable/lib/python3.11/site-packages (from ipywidgets>=7.0.0->cyjupyter->vuecore) (9.14.1)
Requirement already satisfied: widgetsnbextension~=4.0.14 in /home/docs/checkouts/readthedocs.org/user_builds/analytics-core/envs/stable/lib/python3.11/site-packages (from ipywidgets>=7.0.0->cyjupyter->vuecore) (4.0.15)
Requirement already satisfied: jupyterlab_widgets~=3.0.15 in /home/docs/checkouts/readthedocs.org/user_builds/analytics-core/envs/stable/lib/python3.11/site-packages (from ipywidgets>=7.0.0->cyjupyter->vuecore) (3.0.16)
Requirement already satisfied: decorator>=5.1.0 in /home/docs/checkouts/readthedocs.org/user_builds/analytics-core/envs/stable/lib/python3.11/site-packages (from ipython>=6.1.0->ipywidgets>=7.0.0->cyjupyter->vuecore) (5.3.1)
Requirement already satisfied: ipython-pygments-lexers>=1.0.0 in /home/docs/checkouts/readthedocs.org/user_builds/analytics-core/envs/stable/lib/python3.11/site-packages (from ipython>=6.1.0->ipywidgets>=7.0.0->cyjupyter->vuecore) (1.1.1)
Requirement already satisfied: jedi>=0.18.2 in /home/docs/checkouts/readthedocs.org/user_builds/analytics-core/envs/stable/lib/python3.11/site-packages (from ipython>=6.1.0->ipywidgets>=7.0.0->cyjupyter->vuecore) (0.20.0)
Requirement already satisfied: matplotlib-inline>=0.1.6 in /home/docs/checkouts/readthedocs.org/user_builds/analytics-core/envs/stable/lib/python3.11/site-packages (from ipython>=6.1.0->ipywidgets>=7.0.0->cyjupyter->vuecore) (0.2.2)
Requirement already satisfied: pexpect>4.6 in /home/docs/checkouts/readthedocs.org/user_builds/analytics-core/envs/stable/lib/python3.11/site-packages (from ipython>=6.1.0->ipywidgets>=7.0.0->cyjupyter->vuecore) (4.9.0)
Requirement already satisfied: prompt_toolkit<3.1.0,>=3.0.41 in /home/docs/checkouts/readthedocs.org/user_builds/analytics-core/envs/stable/lib/python3.11/site-packages (from ipython>=6.1.0->ipywidgets>=7.0.0->cyjupyter->vuecore) (3.0.52)
Requirement already satisfied: psutil>=7 in /home/docs/checkouts/readthedocs.org/user_builds/analytics-core/envs/stable/lib/python3.11/site-packages (from ipython>=6.1.0->ipywidgets>=7.0.0->cyjupyter->vuecore) (7.2.2)
Requirement already satisfied: pygments>=2.14.0 in /home/docs/checkouts/readthedocs.org/user_builds/analytics-core/envs/stable/lib/python3.11/site-packages (from ipython>=6.1.0->ipywidgets>=7.0.0->cyjupyter->vuecore) (2.20.0)
Requirement already satisfied: stack_data>=0.6.0 in /home/docs/checkouts/readthedocs.org/user_builds/analytics-core/envs/stable/lib/python3.11/site-packages (from ipython>=6.1.0->ipywidgets>=7.0.0->cyjupyter->vuecore) (0.6.3)
Requirement already satisfied: wcwidth in /home/docs/checkouts/readthedocs.org/user_builds/analytics-core/envs/stable/lib/python3.11/site-packages (from prompt_toolkit<3.1.0,>=3.0.41->ipython>=6.1.0->ipywidgets>=7.0.0->cyjupyter->vuecore) (0.8.1)
Requirement already satisfied: parso<0.9.0,>=0.8.6 in /home/docs/checkouts/readthedocs.org/user_builds/analytics-core/envs/stable/lib/python3.11/site-packages (from jedi>=0.18.2->ipython>=6.1.0->ipywidgets>=7.0.0->cyjupyter->vuecore) (0.8.7)
Requirement already satisfied: ptyprocess>=0.5 in /home/docs/checkouts/readthedocs.org/user_builds/analytics-core/envs/stable/lib/python3.11/site-packages (from pexpect>4.6->ipython>=6.1.0->ipywidgets>=7.0.0->cyjupyter->vuecore) (0.7.0)
Requirement already satisfied: executing>=1.2.0 in /home/docs/checkouts/readthedocs.org/user_builds/analytics-core/envs/stable/lib/python3.11/site-packages (from stack_data>=0.6.0->ipython>=6.1.0->ipywidgets>=7.0.0->cyjupyter->vuecore) (2.2.1)
Requirement already satisfied: asttokens>=2.1.0 in /home/docs/checkouts/readthedocs.org/user_builds/analytics-core/envs/stable/lib/python3.11/site-packages (from stack_data>=0.6.0->ipython>=6.1.0->ipywidgets>=7.0.0->cyjupyter->vuecore) (3.0.1)
Requirement already satisfied: pure-eval in /home/docs/checkouts/readthedocs.org/user_builds/analytics-core/envs/stable/lib/python3.11/site-packages (from stack_data>=0.6.0->ipython>=6.1.0->ipywidgets>=7.0.0->cyjupyter->vuecore) (0.2.3)
Requirement already satisfied: Flask<3.2,>=1.0.4 in /home/docs/checkouts/readthedocs.org/user_builds/analytics-core/envs/stable/lib/python3.11/site-packages (from dash->vuecore) (3.1.3)
Requirement already satisfied: Werkzeug<3.2 in /home/docs/checkouts/readthedocs.org/user_builds/analytics-core/envs/stable/lib/python3.11/site-packages (from dash->vuecore) (3.1.8)
Requirement already satisfied: importlib-metadata in /home/docs/checkouts/readthedocs.org/user_builds/analytics-core/envs/stable/lib/python3.11/site-packages (from dash->vuecore) (9.0.0)
Requirement already satisfied: retrying in /home/docs/checkouts/readthedocs.org/user_builds/analytics-core/envs/stable/lib/python3.11/site-packages (from dash->vuecore) (1.4.2)
Requirement already satisfied: nest-asyncio in /home/docs/checkouts/readthedocs.org/user_builds/analytics-core/envs/stable/lib/python3.11/site-packages (from dash->vuecore) (1.6.0)
Requirement already satisfied: setuptools in /home/docs/checkouts/readthedocs.org/user_builds/analytics-core/envs/stable/lib/python3.11/site-packages (from dash->vuecore) (82.0.1)
Requirement already satisfied: janus>=1.0.0 in /home/docs/checkouts/readthedocs.org/user_builds/analytics-core/envs/stable/lib/python3.11/site-packages (from dash->vuecore) (2.0.0)
Requirement already satisfied: blinker>=1.9.0 in /home/docs/checkouts/readthedocs.org/user_builds/analytics-core/envs/stable/lib/python3.11/site-packages (from Flask<3.2,>=1.0.4->dash->vuecore) (1.9.0)
Requirement already satisfied: click>=8.1.3 in /home/docs/checkouts/readthedocs.org/user_builds/analytics-core/envs/stable/lib/python3.11/site-packages (from Flask<3.2,>=1.0.4->dash->vuecore) (8.4.1)
Requirement already satisfied: itsdangerous>=2.2.0 in /home/docs/checkouts/readthedocs.org/user_builds/analytics-core/envs/stable/lib/python3.11/site-packages (from Flask<3.2,>=1.0.4->dash->vuecore) (2.2.0)
Requirement already satisfied: jinja2>=3.1.2 in /home/docs/checkouts/readthedocs.org/user_builds/analytics-core/envs/stable/lib/python3.11/site-packages (from Flask<3.2,>=1.0.4->dash->vuecore) (3.1.6)
Requirement already satisfied: markupsafe>=2.1.1 in /home/docs/checkouts/readthedocs.org/user_builds/analytics-core/envs/stable/lib/python3.11/site-packages (from Flask<3.2,>=1.0.4->dash->vuecore) (3.0.3)
Requirement already satisfied: openpyxl in /home/docs/checkouts/readthedocs.org/user_builds/analytics-core/envs/stable/lib/python3.11/site-packages (from dsp-pandas->acore) (3.1.5)
Requirement already satisfied: zipp>=3.20 in /home/docs/checkouts/readthedocs.org/user_builds/analytics-core/envs/stable/lib/python3.11/site-packages (from importlib-metadata->dash->vuecore) (4.1.0)
Requirement already satisfied: anndata in /home/docs/checkouts/readthedocs.org/user_builds/analytics-core/envs/stable/lib/python3.11/site-packages (from inmoose->acore) (0.12.17)
Requirement already satisfied: fastcluster in /home/docs/checkouts/readthedocs.org/user_builds/analytics-core/envs/stable/lib/python3.11/site-packages (from inmoose->acore) (1.3.0)
Requirement already satisfied: array-api-compat>=1.7.1 in /home/docs/checkouts/readthedocs.org/user_builds/analytics-core/envs/stable/lib/python3.11/site-packages (from anndata->inmoose->acore) (1.15.0)
Requirement already satisfied: h5py>=3.8 in /home/docs/checkouts/readthedocs.org/user_builds/analytics-core/envs/stable/lib/python3.11/site-packages (from anndata->inmoose->acore) (3.16.0)
Requirement already satisfied: legacy-api-wrap in /home/docs/checkouts/readthedocs.org/user_builds/analytics-core/envs/stable/lib/python3.11/site-packages (from anndata->inmoose->acore) (1.5)
Requirement already satisfied: natsort in /home/docs/checkouts/readthedocs.org/user_builds/analytics-core/envs/stable/lib/python3.11/site-packages (from anndata->inmoose->acore) (8.4.0)
Requirement already satisfied: scverse-misc>=0.0.3 in /home/docs/checkouts/readthedocs.org/user_builds/analytics-core/envs/stable/lib/python3.11/site-packages (from anndata->inmoose->acore) (0.0.3)
Requirement already satisfied: zarr!=3.0.*,>=2.18.7 in /home/docs/checkouts/readthedocs.org/user_builds/analytics-core/envs/stable/lib/python3.11/site-packages (from anndata->inmoose->acore) (3.1.6)
Requirement already satisfied: session-info2 in /home/docs/checkouts/readthedocs.org/user_builds/analytics-core/envs/stable/lib/python3.11/site-packages (from scverse-misc>=0.0.3->anndata->inmoose->acore) (0.4.1)
Requirement already satisfied: donfig>=0.8 in /home/docs/checkouts/readthedocs.org/user_builds/analytics-core/envs/stable/lib/python3.11/site-packages (from zarr!=3.0.*,>=2.18.7->anndata->inmoose->acore) (0.8.1.post1)
Requirement already satisfied: google-crc32c>=1.5 in /home/docs/checkouts/readthedocs.org/user_builds/analytics-core/envs/stable/lib/python3.11/site-packages (from zarr!=3.0.*,>=2.18.7->anndata->inmoose->acore) (1.8.0)
Requirement already satisfied: numcodecs>=0.14 in /home/docs/checkouts/readthedocs.org/user_builds/analytics-core/envs/stable/lib/python3.11/site-packages (from zarr!=3.0.*,>=2.18.7->anndata->inmoose->acore) (0.16.5)
Requirement already satisfied: pyyaml in /home/docs/checkouts/readthedocs.org/user_builds/analytics-core/envs/stable/lib/python3.11/site-packages (from donfig>=0.8->zarr!=3.0.*,>=2.18.7->anndata->inmoose->acore) (6.0.3)
Requirement already satisfied: choreographer>=1.3.0 in /home/docs/checkouts/readthedocs.org/user_builds/analytics-core/envs/stable/lib/python3.11/site-packages (from kaleido->vuecore) (1.3.0)
Requirement already satisfied: logistro>=1.0.8 in /home/docs/checkouts/readthedocs.org/user_builds/analytics-core/envs/stable/lib/python3.11/site-packages (from kaleido->vuecore) (2.0.1)
Requirement already satisfied: orjson>=3.10.15 in /home/docs/checkouts/readthedocs.org/user_builds/analytics-core/envs/stable/lib/python3.11/site-packages (from kaleido->vuecore) (3.11.9)
Requirement already satisfied: simplejson>=3.19.3 in /home/docs/checkouts/readthedocs.org/user_builds/analytics-core/envs/stable/lib/python3.11/site-packages (from choreographer>=1.3.0->kaleido->vuecore) (4.1.1)
Requirement already satisfied: regex>=2021.8.3 in /home/docs/checkouts/readthedocs.org/user_builds/analytics-core/envs/stable/lib/python3.11/site-packages (from nltk->vuecore) (2026.5.9)
Requirement already satisfied: tqdm in /home/docs/checkouts/readthedocs.org/user_builds/analytics-core/envs/stable/lib/python3.11/site-packages (from nltk->vuecore) (4.68.2)
Requirement already satisfied: et-xmlfile in /home/docs/checkouts/readthedocs.org/user_builds/analytics-core/envs/stable/lib/python3.11/site-packages (from openpyxl->dsp-pandas->acore) (2.0.0)
Requirement already satisfied: xarray in /home/docs/checkouts/readthedocs.org/user_builds/analytics-core/envs/stable/lib/python3.11/site-packages (from pandas-flavor->pingouin<0.6.0->acore) (2026.4.0)
Requirement already satisfied: typeguard in /home/docs/checkouts/readthedocs.org/user_builds/analytics-core/envs/stable/lib/python3.11/site-packages (from pandera->acore) (4.5.2)
Requirement already satisfied: typing_inspect>=0.6.0 in /home/docs/checkouts/readthedocs.org/user_builds/analytics-core/envs/stable/lib/python3.11/site-packages (from pandera->acore) (0.9.0)
Requirement already satisfied: mypy-extensions>=0.3.0 in /home/docs/checkouts/readthedocs.org/user_builds/analytics-core/envs/stable/lib/python3.11/site-packages (from typing_inspect>=0.6.0->pandera->acore) (1.1.0)
Requirement already satisfied: annotated-types>=0.6.0 in /home/docs/checkouts/readthedocs.org/user_builds/analytics-core/envs/stable/lib/python3.11/site-packages (from pydantic->vuecore) (0.7.0)
Requirement already satisfied: pydantic-core==2.46.4 in /home/docs/checkouts/readthedocs.org/user_builds/analytics-core/envs/stable/lib/python3.11/site-packages (from pydantic->vuecore) (2.46.4)
Requirement already satisfied: typing-inspection>=0.4.2 in /home/docs/checkouts/readthedocs.org/user_builds/analytics-core/envs/stable/lib/python3.11/site-packages (from pydantic->vuecore) (0.4.2)
Requirement already satisfied: jsonpickle>=1.4.1 in /home/docs/checkouts/readthedocs.org/user_builds/analytics-core/envs/stable/lib/python3.11/site-packages (from pyvis->vuecore) (4.1.2)
Requirement already satisfied: charset_normalizer<4,>=2 in /home/docs/checkouts/readthedocs.org/user_builds/analytics-core/envs/stable/lib/python3.11/site-packages (from requests->acore) (3.4.7)
Requirement already satisfied: idna<4,>=2.5 in /home/docs/checkouts/readthedocs.org/user_builds/analytics-core/envs/stable/lib/python3.11/site-packages (from requests->acore) (3.18)
Requirement already satisfied: urllib3<3,>=1.26 in /home/docs/checkouts/readthedocs.org/user_builds/analytics-core/envs/stable/lib/python3.11/site-packages (from requests->acore) (2.7.0)
Requirement already satisfied: certifi>=2023.5.7 in /home/docs/checkouts/readthedocs.org/user_builds/analytics-core/envs/stable/lib/python3.11/site-packages (from requests->acore) (2026.5.20)
Requirement already satisfied: numba>=0.51.2 in /home/docs/checkouts/readthedocs.org/user_builds/analytics-core/envs/stable/lib/python3.11/site-packages (from umap-learn->acore) (0.65.1)
Requirement already satisfied: pynndescent>=0.5 in /home/docs/checkouts/readthedocs.org/user_builds/analytics-core/envs/stable/lib/python3.11/site-packages (from umap-learn->acore) (0.6.0)
Requirement already satisfied: llvmlite<0.48,>=0.47.0dev0 in /home/docs/checkouts/readthedocs.org/user_builds/analytics-core/envs/stable/lib/python3.11/site-packages (from numba>=0.51.2->umap-learn->acore) (0.47.0)
Note: you may need to restart the kernel to use updated packages.

Hide code cell source

 
from typing import Optional

import matplotlib.pyplot as plt
import numpy as np
import pandas as pd
import sklearn
import sklearn.impute
import sklearn.preprocessing
import vuecore.decomposition

import acore.decomposition
import acore.normalization


def plot_umap(X_scaled, y, meta_column=None, random_state=42) -> plt.Axes:
    """Fit and plot UMAP embedding with two components with colors defined by meta_column."""
    embedding = acore.decomposition.umap.run_umap(
        X_scaled, y, random_state=random_state
    )
    if meta_column is None:
        meta_column = y.name
    ax = embedding.plot.scatter("UMAP 1", "UMAP 2", c=meta_column, cmap="Paired")
    return ax


def standard_normalize(X: pd.DataFrame) -> pd.DataFrame:
    """Standard normalize data and keep indices of DataFrame."""
    X_scaled = (
        sklearn.preprocessing.StandardScaler()
        .set_output(transform="pandas")
        .fit_transform(X)
    )
    return X_scaled


def median_impute(X: pd.DataFrame) -> pd.DataFrame:
    X_imputed = (
        sklearn.impute.SimpleImputer(strategy="median")
        .set_output(transform="pandas")
        .fit_transform(X)
    )
    return X_imputed


def run_and_plot_pca(
    X_scaled,
    y,
    meta_column: Optional[str] = None,
    n_components: int = 4,
) -> tuple[pd.DataFrame, plt.Figure]:
    PCs, _ = acore.decomposition.pca.run_pca(X_scaled, n_components=n_components)
    PCs.columns = [s.replace("principal component", "PC") for s in PCs.columns]
    fig = vuecore.decomposition.pca_grid(
        PCs=PCs, meta_column=y, n_components=n_components, meta_col_name=meta_column
    )
    return PCs, fig

Set some parameters#

BASE = (
    "https://raw.githubusercontent.com/Multiomics-Analytics-Group/acore/"
    "main/example_data/alzheimer_proteomics/"
)
# data is already preprocessed: log2, filtered
fname: str = "alzheimer_example_omics_and_clinic.csv"  # combined omics and meta data
covariates: list[str] = ["age", "male"]
group: str = "collection_site"
subject_col: str = "Sample ID"
drop_cols: list[str] = ["AD"]
factor_and_covars: list[str] = [group, *covariates]
group_label: Optional[str] = "site"  # optional: rename target variable

Data loading#

Use combined dataset for ANCOVA analysis.

Hide code cell source

 
omics_and_meta = (
    pd.read_csv(f"{BASE}/{fname}", index_col=subject_col)
    .convert_dtypes()
    .dropna(subset=factor_and_covars)
)
omics_and_meta
AD age male collection_site Q6UX72 O14773 A0A0A0MQU6 P36222 P51693-2 P17174 ... A0A075B6K4 O15041 J3KNA1 A0A0C4DH33 P16870 G3V533 Q9Y5I4 P55283 A1L4H1 Q7Z4T9
Sample ID
Sample_000 0 71 0 Sweden 16.047 18.412 16.381 20.948 18.658 20.232 ... 16.149 14.013 20.549 14.269 20.468 18.448 17.187 17.422 15.542 19.331
Sample_001 1 77 1 Sweden 14.457 17.869 16.196 21.083 18.446 19.776 ... 16.127 13.916 15.854 14.379 19.902 17.723 17.447 17.097 15.734 18.980
Sample_002 1 75 1 Sweden 15.631 17.662 16.071 21.206 18.967 20.066 ... 15.387 13.903 17.576 13.675 19.619 17.006 17.410 17.752 15.824 19.326
Sample_003 1 72 0 Sweden 16.204 18.437 16.356 20.729 18.798 20.195 ... 16.565 14.526 18.173 <NA> 20.170 17.212 17.545 17.483 15.515 18.953
Sample_004 1 63 0 Sweden 15.968 18.577 16.001 21.068 18.422 20.485 ... 16.418 14.933 15.440 <NA> 19.987 17.624 17.297 17.172 15.334 18.651
... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ...
Sample_205 1 69 0 Berlin 15.262 18.046 16.358 21.321 18.580 19.838 ... 15.350 13.572 13.482 <NA> 19.984 15.269 17.104 16.952 15.705 18.844
Sample_206 0 73 1 Berlin <NA> 16.573 16.099 20.663 19.191 18.388 ... 16.582 9.748 14.372 15.567 19.396 16.976 17.109 18.056 15.282 18.686
Sample_207 0 71 0 Berlin 15.463 17.991 16.062 20.770 19.050 19.361 ... 15.768 13.241 13.931 15.092 19.923 16.669 16.938 17.248 14.874 19.146
Sample_208 0 83 1 Berlin 15.786 17.216 15.929 20.938 18.216 19.183 ... 17.560 14.442 <NA> 14.267 19.831 16.258 17.155 16.353 15.471 16.853
Sample_209 0 63 0 Berlin 15.691 <NA> 15.914 20.366 19.308 19.534 ... 16.338 13.628 <NA> 13.051 19.427 14.848 16.776 16.597 14.699 18.087

197 rows × 104 columns

Metadata here is of type integer. All floats are proteomics measurements.

Hide code cell source

 
omics_and_meta.dtypes.value_counts()

Float64          100
Int64              3
string[python]     1
Name: count, dtype: int64

Hide code cell source

 
omics_and_meta[factor_and_covars]
collection_site age male
Sample ID
Sample_000 Sweden 71 0
Sample_001 Sweden 77 1
Sample_002 Sweden 75 1
Sample_003 Sweden 72 0
Sample_004 Sweden 63 0
... ... ... ...
Sample_205 Berlin 69 0
Sample_206 Berlin 73 1
Sample_207 Berlin 71 0
Sample_208 Berlin 83 1
Sample_209 Berlin 63 0

197 rows × 3 columns

omics = omics_and_meta.drop(columns=[*factor_and_covars, *drop_cols])
y = omics_and_meta[group].astype("category").rename(group_label)

For simplicity we normalize here all samples together, but normally you would need to apply the normalization from you training data to the test data. So see these examples here as a way to do it for your training data.

Fill missing values for preliminary plots#

Impute using median to impute (before scaling, which can be changed).

Hide code cell source

 
omics_imputed = median_impute(omics)
assert omics_imputed.isna().sum().sum() == 0
omics_imputed.shape

(197, 100)

Explained variance by first four principal components in data.

Hide code cell source

 
PCs, pca = acore.decomposition.pca.run_pca(omics_imputed, n_components=4)
ax = vuecore.decomposition.plot_explained_variance(pca)
ax.locator_params(axis="x", integer=True)
../_images/4ebb0334e48806999a3267a014454509f3a2591566ebef62b751b0f3e62f2747.png

Normalization of samples in a dataset#

We will use the acore.normalization module to normalize the data.

We will do it for each of the data on the omics dataset which is log transformed, but not yet imputed and normalized. Then we will reapply standard normalization before replotting the PCA and UMAP plots. The execption is combat as it need complete data.

Hide code cell source

 
omics
Q6UX72 O14773 A0A0A0MQU6 P36222 P51693-2 P17174 Q9BWS9 A0A0B4J2D9 P00734 Q13433 ... A0A075B6K4 O15041 J3KNA1 A0A0C4DH33 P16870 G3V533 Q9Y5I4 P55283 A1L4H1 Q7Z4T9
Sample ID
Sample_000 16.047 18.412 16.381 20.948 18.658 20.232 15.500 15.408 19.870 14.999 ... 16.149 14.013 20.549 14.269 20.468 18.448 17.187 17.422 15.542 19.331
Sample_001 14.457 17.869 16.196 21.083 18.446 19.776 14.760 <NA> 20.338 14.374 ... 16.127 13.916 15.854 14.379 19.902 17.723 17.447 17.097 15.734 18.980
Sample_002 15.631 17.662 16.071 21.206 18.967 20.066 <NA> 15.362 19.814 15.121 ... 15.387 13.903 17.576 13.675 19.619 17.006 17.410 17.752 15.824 19.326
Sample_003 16.204 18.437 16.356 20.729 18.798 20.195 15.300 <NA> 20.078 14.798 ... 16.565 14.526 18.173 <NA> 20.170 17.212 17.545 17.483 15.515 18.953
Sample_004 15.968 18.577 16.001 21.068 18.422 20.485 16.054 <NA> 19.786 15.097 ... 16.418 14.933 15.440 <NA> 19.987 17.624 17.297 17.172 15.334 18.651
... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ...
Sample_205 15.262 18.046 16.358 21.321 18.580 19.838 14.942 14.204 20.530 14.518 ... 15.350 13.572 13.482 <NA> 19.984 15.269 17.104 16.952 15.705 18.844
Sample_206 <NA> 16.573 16.099 20.663 19.191 18.388 16.026 15.503 21.106 <NA> ... 16.582 9.748 14.372 15.567 19.396 16.976 17.109 18.056 15.282 18.686
Sample_207 15.463 17.991 16.062 20.770 19.050 19.361 15.551 <NA> 20.477 13.842 ... 15.768 13.241 13.931 15.092 19.923 16.669 16.938 17.248 14.874 19.146
Sample_208 15.786 17.216 15.929 20.938 18.216 19.183 15.176 14.104 20.483 13.929 ... 17.560 14.442 <NA> 14.267 19.831 16.258 17.155 16.353 15.471 16.853
Sample_209 15.691 <NA> 15.914 20.366 19.308 19.534 15.653 13.784 21.183 13.923 ... 16.338 13.628 <NA> 13.051 19.427 14.848 16.776 16.597 14.699 18.087

197 rows × 100 columns

Median normalization#

Substracts a constant from all features of a sample. All samples will have the same global median.

%%time
X = acore.normalization.normalize_data(omics, "median")
X

CPU times: user 8.65 ms, sys: 0 ns, total: 8.65 ms
Wall time: 8.36 ms
Q6UX72 O14773 A0A0A0MQU6 P36222 P51693-2 P17174 Q9BWS9 A0A0B4J2D9 P00734 Q13433 ... A0A075B6K4 O15041 J3KNA1 A0A0C4DH33 P16870 G3V533 Q9Y5I4 P55283 A1L4H1 Q7Z4T9
Sample ID
Sample_000 15.837 18.201 16.171 20.738 18.447 20.022 15.290 15.198 19.660 14.789 ... 15.938 13.802 20.338 14.059 20.257 18.238 16.977 17.212 15.332 19.121
Sample_001 14.325 17.737 16.064 20.952 18.315 19.644 14.628 <NA> 20.206 14.242 ... 15.996 13.784 15.723 14.248 19.771 17.591 17.315 16.965 15.602 18.848
Sample_002 15.442 17.474 15.882 21.017 18.778 19.877 <NA> 15.173 19.626 14.932 ... 15.198 13.715 17.387 13.486 19.430 16.817 17.222 17.563 15.635 19.138
Sample_003 16.053 18.286 16.205 20.578 18.647 20.044 15.149 <NA> 19.927 14.647 ... 16.414 14.375 18.022 <NA> 20.019 17.061 17.394 17.332 15.364 18.803
Sample_004 15.724 18.332 15.757 20.824 18.177 20.241 15.809 <NA> 19.542 14.853 ... 16.173 14.689 15.195 <NA> 19.742 17.380 17.052 16.928 15.090 18.406
... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ...
Sample_205 15.233 18.017 16.329 21.292 18.551 19.809 14.913 14.175 20.501 14.489 ... 15.321 13.543 13.453 <NA> 19.955 15.240 17.075 16.923 15.676 18.815
Sample_206 <NA> 16.483 16.009 20.573 19.102 18.298 15.936 15.413 21.016 <NA> ... 16.492 9.658 14.283 15.477 19.306 16.886 17.019 17.966 15.192 18.596
Sample_207 15.510 18.038 16.109 20.817 19.097 19.408 15.598 <NA> 20.525 13.889 ... 15.816 13.288 13.978 15.139 19.970 16.716 16.986 17.295 14.921 19.193
Sample_208 15.898 17.328 16.040 21.050 18.328 19.295 15.288 14.216 20.595 14.040 ... 17.672 14.554 <NA> 14.379 19.943 16.369 17.267 16.464 15.583 16.965
Sample_209 15.963 <NA> 16.185 20.638 19.579 19.806 15.924 14.055 21.454 14.194 ... 16.609 13.899 <NA> 13.322 19.698 15.119 17.047 16.868 14.970 18.359

197 rows × 100 columns

Hide code cell source

 
omics_imp = median_impute(X)
omics_imp_scaled = standard_normalize(omics_imp)
PCs, fig = run_and_plot_pca(omics_imp_scaled, y, y.name, n_components=4)
ax = plot_umap(omics_imp_scaled, y)
../_images/117a2c8222d3a8e634752c2b5a712505e100cc34ab6079c49b82e53c67986426.png ../_images/4f73b045a68276c02643125e69f9b89bf88cd07e87996d53ec51440914a2902a.png

See change by substracting median normalized data from original data.

Hide code cell source

 
omics - X
Q6UX72 O14773 A0A0A0MQU6 P36222 P51693-2 P17174 Q9BWS9 A0A0B4J2D9 P00734 Q13433 ... A0A075B6K4 O15041 J3KNA1 A0A0C4DH33 P16870 G3V533 Q9Y5I4 P55283 A1L4H1 Q7Z4T9
Sample ID
Sample_000 0.210 0.210 0.210 0.210 0.210 0.210 0.210 0.210 0.210 0.210 ... 0.210 0.210 0.210 0.210 0.210 0.210 0.210 0.210 0.210 0.210
Sample_001 0.132 0.132 0.132 0.132 0.132 0.132 0.132 <NA> 0.132 0.132 ... 0.132 0.132 0.132 0.132 0.132 0.132 0.132 0.132 0.132 0.132
Sample_002 0.189 0.189 0.189 0.189 0.189 0.189 <NA> 0.189 0.189 0.189 ... 0.189 0.189 0.189 0.189 0.189 0.189 0.189 0.189 0.189 0.189
Sample_003 0.151 0.151 0.151 0.151 0.151 0.151 0.151 <NA> 0.151 0.151 ... 0.151 0.151 0.151 <NA> 0.151 0.151 0.151 0.151 0.151 0.151
Sample_004 0.245 0.245 0.245 0.245 0.245 0.245 0.245 <NA> 0.245 0.245 ... 0.245 0.245 0.245 <NA> 0.245 0.245 0.245 0.245 0.245 0.245
... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ...
Sample_205 0.029 0.029 0.029 0.029 0.029 0.029 0.029 0.029 0.029 0.029 ... 0.029 0.029 0.029 <NA> 0.029 0.029 0.029 0.029 0.029 0.029
Sample_206 <NA> 0.090 0.090 0.090 0.090 0.090 0.090 0.090 0.090 <NA> ... 0.090 0.090 0.090 0.090 0.090 0.090 0.090 0.090 0.090 0.090
Sample_207 -0.047 -0.047 -0.047 -0.047 -0.047 -0.047 -0.047 <NA> -0.047 -0.047 ... -0.047 -0.047 -0.047 -0.047 -0.047 -0.047 -0.047 -0.047 -0.047 -0.047
Sample_208 -0.112 -0.112 -0.112 -0.112 -0.112 -0.112 -0.112 -0.112 -0.112 -0.112 ... -0.112 -0.112 <NA> -0.112 -0.112 -0.112 -0.112 -0.112 -0.112 -0.112
Sample_209 -0.271 <NA> -0.271 -0.271 -0.271 -0.271 -0.271 -0.271 -0.271 -0.271 ... -0.271 -0.271 <NA> -0.271 -0.271 -0.271 -0.271 -0.271 -0.271 -0.271

197 rows × 100 columns

Z-score normalization#

Normalize a sample by it’s mean and standard deviation.

%%time
X = acore.normalization.normalize_data(omics, "zscore")
X

CPU times: user 8.97 ms, sys: 109 μs, total: 9.08 ms
Wall time: 9.28 ms
Q6UX72 O14773 A0A0A0MQU6 P36222 P51693-2 P17174 Q9BWS9 A0A0B4J2D9 P00734 Q13433 ... A0A075B6K4 O15041 J3KNA1 A0A0C4DH33 P16870 G3V533 Q9Y5I4 P55283 A1L4H1 Q7Z4T9
Sample ID
Sample_000 -0.630 0.179 -0.516 1.048 0.263 0.803 -0.818 -0.849 0.678 -0.989 ... -0.596 -1.327 0.911 -1.239 0.883 0.192 -0.240 -0.160 -0.803 0.494
Sample_001 -1.077 0.045 -0.505 1.102 0.235 0.672 -0.978 <NA> 0.857 -1.105 ... -0.528 -1.255 -0.618 -1.103 0.713 -0.003 -0.094 -0.209 -0.658 0.410
Sample_002 -0.682 0.017 -0.531 1.236 0.466 0.843 <NA> -0.774 0.757 -0.857 ... -0.766 -1.276 -0.013 -1.355 0.690 -0.209 -0.070 0.048 -0.616 0.589
Sample_003 -0.526 0.248 -0.473 1.042 0.373 0.857 -0.839 <NA> 0.816 -1.013 ... -0.401 -1.107 0.156 <NA> 0.848 -0.177 -0.061 -0.083 -0.764 0.427
Sample_004 -0.609 0.253 -0.598 1.075 0.201 0.883 -0.580 <NA> 0.652 -0.896 ... -0.460 -0.950 -0.783 <NA> 0.718 -0.062 -0.170 -0.211 -0.818 0.277
... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ...
Sample_205 -0.797 0.199 -0.405 1.370 0.390 0.840 -0.911 -1.175 1.087 -1.062 ... -0.765 -1.401 -1.433 <NA> 0.892 -0.794 -0.138 -0.192 -0.638 0.484
Sample_206 <NA> -0.324 -0.483 1.056 0.560 0.289 -0.508 -0.685 1.206 <NA> ... -0.321 -2.626 -1.066 -0.663 0.629 -0.188 -0.143 0.177 -0.759 0.389
Sample_207 -0.738 0.165 -0.524 1.158 0.543 0.654 -0.707 <NA> 1.053 -1.317 ... -0.629 -1.532 -1.286 -0.871 0.855 -0.307 -0.211 -0.101 -0.949 0.578
Sample_208 -0.596 -0.099 -0.546 1.192 0.247 0.583 -0.807 -1.179 1.034 -1.240 ... 0.020 -1.062 <NA> -1.123 0.808 -0.432 -0.121 -0.399 -0.705 -0.225
Sample_209 -0.575 <NA> -0.500 0.994 0.639 0.715 -0.587 -1.215 1.268 -1.168 ... -0.358 -1.267 <NA> -1.460 0.679 -0.858 -0.211 -0.271 -0.908 0.229

197 rows × 100 columns

Hide code cell source

 
omics_imp = median_impute(X)
omics_imp_scaled = standard_normalize(omics_imp)
PCs, fig = run_and_plot_pca(omics_imp_scaled, y, n_components=4)
ax = plot_umap(omics_imp_scaled, y)
../_images/2d60e4d4d6b61faaba9a933be654d2077b65762f0702bcc07be3f3baf9b70ab8.png ../_images/8c144c2ae4cb18a63c4c95de215f627490f247645e50938a744b0412b89b3c79.png

See change by substracting z-score normalized data from original data.

Hide code cell source

 
omics_imp_scaled - X
Q6UX72 O14773 A0A0A0MQU6 P36222 P51693-2 P17174 Q9BWS9 A0A0B4J2D9 P00734 Q13433 ... A0A075B6K4 O15041 J3KNA1 A0A0C4DH33 P16870 G3V533 Q9Y5I4 P55283 A1L4H1 Q7Z4T9
Sample ID
Sample_000 1.063 0.672 0.542 -0.608 -0.273 0.485 0.720 0.409 -2.290 1.794 ... -0.454 2.106 1.602 -0.433 0.542 0.491 0.227 0.134 0.072 0.088
Sample_001 -0.403 0.122 0.601 -0.416 -0.356 -0.132 -0.230 <NA> -1.468 1.309 ... -0.287 2.238 0.603 -0.162 -0.514 0.044 1.074 -0.120 0.817 -0.289
Sample_002 0.894 0.009 0.460 0.060 0.313 0.678 <NA> 0.477 -1.927 2.348 ... -0.877 2.200 0.998 -0.663 -0.661 -0.426 1.215 1.195 1.031 0.516
Sample_003 1.406 0.952 0.781 -0.628 0.044 0.743 0.592 <NA> -1.654 1.694 ... 0.029 2.512 1.109 <NA> 0.326 -0.352 1.265 0.529 0.271 -0.214
Sample_004 1.134 0.972 0.087 -0.510 -0.452 0.863 2.128 <NA> -2.412 2.185 ... -0.118 2.802 0.495 <NA> -0.485 -0.089 0.634 -0.129 -0.001 -0.887
... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ...
Sample_205 0.518 0.751 1.162 0.537 0.093 0.659 0.165 0.116 -0.406 1.487 ... -0.874 1.970 0.070 <NA> 0.595 -1.764 0.818 -0.033 0.916 0.043
Sample_206 <NA> -1.380 0.724 -0.577 0.585 -1.938 2.556 0.558 0.143 <NA> ... 0.228 -0.293 0.310 0.714 -1.040 -0.377 0.792 1.857 0.299 -0.382
Sample_207 0.709 0.614 0.495 -0.216 0.538 -0.214 1.377 <NA> -0.560 0.416 ... -0.538 1.727 0.166 0.300 0.368 -0.651 0.395 0.436 -0.669 0.464
Sample_208 1.176 -0.465 0.374 -0.095 -0.319 -0.550 0.781 0.112 -0.649 0.741 ... 1.073 2.595 <NA> -0.201 0.074 -0.936 0.920 -1.092 0.575 -3.145
Sample_209 1.246 <NA> 0.632 -0.798 0.814 0.072 2.085 0.080 0.431 1.044 ... 0.136 2.217 <NA> -0.873 -0.729 -1.910 0.399 -0.436 -0.459 -1.101

197 rows × 100 columns

Median Polish Normalization#

  • normalize iteratively features and samples to have zero median.

%%time
X = acore.normalization.normalize_data(omics, "median_polish")
X

CPU times: user 6.68 s, sys: 14.6 ms, total: 6.7 s
Wall time: 6.69 s
Q6UX72 O14773 A0A0A0MQU6 P36222 P51693-2 P17174 Q9BWS9 A0A0B4J2D9 P00734 Q13433 ... A0A075B6K4 O15041 J3KNA1 A0A0C4DH33 P16870 G3V533 Q9Y5I4 P55283 A1L4H1 Q7Z4T9
Sample ID
Sample_000 15.826 17.988 16.362 20.793 18.761 19.615 15.493 15.944 20.774 14.461 ... 17.051 13.361 15.564 15.892 20.008 17.922 17.218 17.487 15.895 19.159
Sample_001 15.645 17.807 16.181 20.612 18.580 19.434 15.312 <NA> 20.593 14.280 ... 16.870 13.180 15.383 15.711 19.827 17.741 17.037 17.306 15.714 18.977
Sample_002 15.498 17.660 16.034 20.464 18.433 19.286 <NA> 15.616 20.446 14.132 ... 16.723 13.033 15.235 15.563 19.680 17.593 16.890 17.159 15.566 18.830
Sample_003 15.626 17.788 16.162 20.592 18.561 19.414 15.292 <NA> 20.574 14.260 ... 16.851 13.161 15.363 <NA> 19.808 17.721 17.018 17.287 15.694 18.958
Sample_004 15.643 17.805 16.179 20.610 18.578 19.431 15.310 <NA> 20.591 14.277 ... 16.868 13.178 15.380 <NA> 19.825 17.738 17.035 17.304 15.712 18.975
... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ...
Sample_205 15.639 17.801 16.175 20.606 18.574 19.428 15.306 15.757 20.587 14.274 ... 16.864 13.174 15.377 <NA> 19.821 17.735 17.031 17.300 15.708 18.971
Sample_206 <NA> 17.666 16.040 20.471 18.439 19.293 15.171 15.622 20.452 <NA> ... 16.729 13.039 15.242 15.570 19.686 17.599 16.896 17.165 15.573 18.836
Sample_207 15.581 17.743 16.117 20.548 18.517 19.370 15.248 <NA> 20.529 14.216 ... 16.807 13.116 15.319 15.647 19.763 17.677 16.974 17.243 15.650 18.914
Sample_208 15.545 17.707 16.082 20.512 18.481 19.334 15.212 15.663 20.493 14.180 ... 16.771 13.080 <NA> 15.611 19.727 17.641 16.938 17.207 15.614 18.878
Sample_209 15.431 <NA> 15.967 20.398 18.366 19.219 15.098 15.549 20.379 14.065 ... 16.656 12.966 <NA> 15.496 19.613 17.526 16.823 17.092 15.500 18.763

197 rows × 100 columns

Hide code cell source

 
omics_imp = median_impute(X)
omics_imp_scaled = standard_normalize(omics_imp)
PCs, fig = run_and_plot_pca(omics_imp_scaled, y, n_components=4)
ax = plot_umap(omics_imp_scaled, y)
../_images/b17e1fec4606e361df23360baf475e1f4f5c6aa3911a7a02b1369629fe30f8fc.png ../_images/7292cdb407141f4483cb42b0f65087278e3836bf12aeefab384eda56fd0fb109.png

See change by substracting median polish normalized data from original data.

Hide code cell source

 
omics_imp_scaled - X
Q6UX72 O14773 A0A0A0MQU6 P36222 P51693-2 P17174 Q9BWS9 A0A0B4J2D9 P00734 Q13433 ... A0A075B6K4 O15041 J3KNA1 A0A0C4DH33 P16870 G3V533 Q9Y5I4 P55283 A1L4H1 Q7Z4T9
Sample ID
Sample_000 -13.078 -15.220 -13.572 -18.046 -16.014 -16.867 -12.151 -12.955 -18.027 -11.452 ... -14.128 -10.113 -12.476 -12.771 -17.261 -15.144 -14.463 -14.740 -13.021 -16.357
Sample_001 -14.210 -16.379 -14.738 -19.175 -17.143 -17.996 -13.532 <NA> -19.156 -12.754 ... -15.312 -11.570 -13.799 -14.042 -18.390 -16.283 -15.597 -15.869 -14.224 -17.529
Sample_002 -15.132 -17.322 -15.688 -20.094 -18.062 -18.916 <NA> -15.150 -20.075 -13.813 ... -16.276 -12.756 -14.876 -15.078 -19.309 -17.210 -16.520 -16.788 -15.204 -18.483
Sample_003 -14.332 -16.503 -14.864 -19.296 -17.265 -18.118 -13.680 <NA> -19.277 -12.894 ... -15.439 -11.727 -13.941 <NA> -18.511 -16.406 -15.719 -15.991 -14.354 -17.655
Sample_004 -14.224 -16.393 -14.753 -19.189 -17.158 -18.011 -13.549 <NA> -19.170 -12.770 ... -15.327 -11.588 -13.816 <NA> -18.404 -16.297 -15.611 -15.884 -14.240 -17.544
... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ...
Sample_205 -14.247 -16.417 -14.777 -19.212 -17.181 -18.034 -13.577 -14.205 -19.193 -12.797 ... -15.351 -11.618 -13.843 <NA> -18.427 -16.321 -15.634 -15.907 -14.264 -17.568
Sample_206 <NA> -17.280 -15.646 -20.054 -18.022 -18.875 -14.607 -15.107 -20.035 <NA> ... -16.234 -12.704 -14.829 -15.032 -19.269 -17.170 -16.479 -16.748 -15.161 -18.441
Sample_207 -14.608 -16.786 -15.148 -19.572 -17.540 -18.393 -14.017 <NA> -19.553 -13.211 ... -15.728 -12.082 -14.264 -14.489 -18.787 -16.683 -15.995 -16.266 -14.647 -17.941
Sample_208 -14.833 -17.016 -15.380 -19.796 -17.764 -18.617 -14.291 -14.831 -19.777 -13.469 ... -15.963 -12.371 <NA> -14.742 -19.011 -16.909 -16.220 -16.490 -14.886 -18.173
Sample_209 -15.549 <NA> -16.118 -20.510 -18.479 -19.332 -15.165 -15.597 -20.492 -14.293 ... -16.713 -13.293 <NA> -15.547 -19.726 -17.630 -16.938 -17.205 -15.648 -18.915

197 rows × 100 columns

Quantile normalization#

quantile normalize each feature column.

%%time
X = acore.normalization.normalize_data(omics, "quantile")
X

CPU times: user 42.4 ms, sys: 1.03 ms, total: 43.5 ms
Wall time: 43.2 ms
Q6UX72 O14773 A0A0A0MQU6 P36222 P51693-2 P17174 Q9BWS9 A0A0B4J2D9 P00734 Q13433 ... A0A075B6K4 O15041 J3KNA1 A0A0C4DH33 P16870 G3V533 Q9Y5I4 P55283 A1L4H1 Q7Z4T9
Sample ID
Sample_000 15.713 18.119 16.157 21.368 18.508 19.616 15.131 15.049 19.505 14.744 ... 15.977 12.304 20.246 13.685 19.917 18.307 17.067 17.152 15.206 18.723
Sample_001 14.557 17.462 16.219 20.656 18.219 19.103 14.744 <NA> 19.616 14.151 ... 16.096 13.459 15.713 14.266 19.369 17.387 17.230 16.897 15.607 18.617
Sample_002 15.502 17.617 15.977 21.642 18.833 20.117 <NA> 15.275 19.917 14.979 ... 15.343 13.861 17.387 13.459 19.740 16.897 17.230 17.853 15.765 19.103
Sample_003 15.921 18.508 16.279 20.850 18.882 20.246 14.979 <NA> 19.917 14.822 ... 16.489 14.375 17.853 <NA> 20.117 17.067 17.310 17.152 15.206 18.983
Sample_004 15.812 18.307 15.871 20.304 17.935 19.917 15.921 <NA> 19.369 14.822 ... 16.219 14.744 15.343 <NA> 19.505 17.310 16.985 16.813 15.049 18.405
... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ...
Sample_205 15.275 18.405 16.734 23.081 19.229 20.489 14.822 13.861 21.368 14.464 ... 15.448 13.103 12.304 <NA> 20.850 15.343 17.387 17.152 15.871 19.740
Sample_206 <NA> 16.734 16.157 21.087 19.616 18.723 16.096 15.557 22.355 <NA> ... 16.813 12.304 14.006 15.653 19.917 16.985 17.152 18.219 15.206 18.983
Sample_207 15.343 18.307 16.411 22.355 19.740 20.246 15.502 <NA> 21.087 13.103 ... 15.977 12.304 13.685 15.206 20.850 16.813 17.310 17.538 14.822 20.117
Sample_208 15.812 17.538 15.977 22.823 18.833 20.117 15.275 13.861 21.642 13.103 ... 18.016 14.557 <NA> 14.266 20.656 16.650 17.387 16.813 15.557 17.230
Sample_209 15.713 <NA> 16.033 20.489 19.740 20.246 15.653 14.006 22.355 14.151 ... 16.411 13.685 <NA> 13.103 19.917 15.049 17.230 16.813 14.822 18.405

197 rows × 100 columns

Hide code cell source

 
omics_imp = median_impute(X)
omics_imp_scaled = standard_normalize(omics_imp)
PCs, fig = run_and_plot_pca(omics_imp_scaled, y, n_components=4)
ax = plot_umap(omics_imp_scaled, y)
../_images/206a1dda1202a18e9155f4417fd753cfcc313bc951bf555c9104a76e6468e0ba.png ../_images/bbadda2fbcd131ec96a3a6b76dbbdd7612591e43c3e3d3d97a6cd2eb919522c1.png 
omics - X
Q6UX72 O14773 A0A0A0MQU6 P36222 P51693-2 P17174 Q9BWS9 A0A0B4J2D9 P00734 Q13433 ... A0A075B6K4 O15041 J3KNA1 A0A0C4DH33 P16870 G3V533 Q9Y5I4 P55283 A1L4H1 Q7Z4T9
Sample ID
Sample_000 0.334 0.292 0.224 -0.420 0.150 0.617 0.369 0.359 0.365 0.256 ... 0.171 1.708 0.303 0.584 0.551 0.141 0.120 0.270 0.336 0.607
Sample_001 -0.100 0.407 -0.023 0.427 0.228 0.672 0.016 <NA> 0.723 0.223 ... 0.032 0.457 0.142 0.113 0.534 0.336 0.218 0.200 0.126 0.363
Sample_002 0.129 0.045 0.094 -0.437 0.135 -0.051 <NA> 0.087 -0.102 0.142 ... 0.043 0.042 0.189 0.216 -0.122 0.109 0.181 -0.101 0.059 0.223
Sample_003 0.283 -0.071 0.077 -0.121 -0.084 -0.050 0.321 <NA> 0.161 -0.024 ... 0.076 0.151 0.320 <NA> 0.053 0.144 0.235 0.331 0.310 -0.030
Sample_004 0.156 0.270 0.130 0.765 0.487 0.569 0.133 <NA> 0.418 0.275 ... 0.199 0.189 0.097 <NA> 0.481 0.314 0.312 0.360 0.285 0.246
... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ...
Sample_205 -0.013 -0.359 -0.376 -1.760 -0.649 -0.650 0.120 0.344 -0.838 0.054 ... -0.098 0.469 1.178 <NA> -0.866 -0.074 -0.284 -0.200 -0.166 -0.897
Sample_206 <NA> -0.161 -0.058 -0.424 -0.424 -0.335 -0.069 -0.054 -1.249 <NA> ... -0.231 -2.556 0.366 -0.086 -0.521 -0.009 -0.043 -0.162 0.076 -0.297
Sample_207 0.119 -0.316 -0.349 -1.585 -0.690 -0.885 0.049 <NA> -0.609 0.739 ... -0.209 0.937 0.246 -0.114 -0.928 -0.144 -0.372 -0.290 0.053 -0.971
Sample_208 -0.026 -0.322 -0.048 -1.885 -0.617 -0.933 -0.099 0.243 -1.159 0.826 ... -0.456 -0.116 <NA> 0.001 -0.826 -0.392 -0.232 -0.460 -0.085 -0.377
Sample_209 -0.021 <NA> -0.119 -0.122 -0.433 -0.711 -0.000 -0.223 -1.172 -0.228 ... -0.073 -0.057 <NA> -0.051 -0.490 -0.201 -0.453 -0.216 -0.123 -0.318

197 rows × 100 columns

Linear normalization#

%%time
X = acore.normalization.normalize_data(omics, "linear")
X

CPU times: user 9.5 ms, sys: 1.04 ms, total: 10.5 ms
Wall time: 9.55 ms
Q6UX72 O14773 A0A0A0MQU6 P36222 P51693-2 P17174 Q9BWS9 A0A0B4J2D9 P00734 Q13433 ... A0A075B6K4 O15041 J3KNA1 A0A0C4DH33 P16870 G3V533 Q9Y5I4 P55283 A1L4H1 Q7Z4T9
Sample ID
Sample_000 0.005 0.006 0.005 0.005 0.005 0.005 0.007 0.006 0.005 0.006 ... 0.005 0.007 0.009 0.006 0.005 0.005 0.005 0.005 0.005 0.005
Sample_001 0.005 0.005 0.005 0.005 0.005 0.005 0.007 0.000 0.005 0.006 ... 0.005 0.007 0.007 0.006 0.005 0.005 0.005 0.005 0.005 0.005
Sample_002 0.005 0.005 0.005 0.005 0.005 0.005 0.000 0.006 0.005 0.006 ... 0.005 0.007 0.008 0.006 0.005 0.005 0.005 0.005 0.005 0.005
Sample_003 0.005 0.006 0.005 0.005 0.005 0.005 0.007 0.000 0.005 0.006 ... 0.005 0.008 0.008 0.000 0.005 0.005 0.005 0.005 0.005 0.005
Sample_004 0.005 0.006 0.005 0.005 0.005 0.005 0.007 0.000 0.005 0.006 ... 0.005 0.008 0.007 0.000 0.005 0.005 0.005 0.005 0.005 0.005
... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ...
Sample_205 0.005 0.005 0.005 0.005 0.005 0.005 0.007 0.005 0.005 0.006 ... 0.005 0.007 0.006 0.000 0.005 0.004 0.005 0.005 0.005 0.005
Sample_206 0.000 0.005 0.005 0.005 0.005 0.005 0.007 0.006 0.005 0.000 ... 0.005 0.005 0.006 0.006 0.005 0.005 0.005 0.005 0.005 0.005
Sample_207 0.005 0.005 0.005 0.005 0.005 0.005 0.007 0.000 0.005 0.006 ... 0.005 0.007 0.006 0.006 0.005 0.005 0.005 0.005 0.005 0.005
Sample_208 0.005 0.005 0.005 0.005 0.005 0.005 0.007 0.005 0.005 0.006 ... 0.006 0.008 0.000 0.006 0.005 0.005 0.005 0.005 0.005 0.005
Sample_209 0.005 0.000 0.005 0.005 0.005 0.005 0.007 0.005 0.005 0.006 ... 0.005 0.007 0.000 0.005 0.005 0.004 0.005 0.005 0.005 0.005

197 rows × 100 columns

Hide code cell source

 
omics_imp = median_impute(X)
omics_imp_scaled = standard_normalize(omics_imp)
PCs, fig = run_and_plot_pca(omics_imp_scaled, y, n_components=4)
ax = plot_umap(omics_imp_scaled, y)
../_images/b16e5217067426735dc78a42f0ad4f098cfbdcd273960f677aa22a1561c48855.png ../_images/75c392dc1f6e8abd9e415d7d1a2da1c0e1a1316135b34d3510e6abf2ec20ff09.png

Hide code cell source

 
omics - X
Q6UX72 O14773 A0A0A0MQU6 P36222 P51693-2 P17174 Q9BWS9 A0A0B4J2D9 P00734 Q13433 ... A0A075B6K4 O15041 J3KNA1 A0A0C4DH33 P16870 G3V533 Q9Y5I4 P55283 A1L4H1 Q7Z4T9
Sample ID
Sample_000 16.042 18.406 16.376 20.943 18.652 20.227 15.493 15.403 19.865 14.993 ... 16.143 14.005 20.540 14.263 20.462 18.443 17.182 17.417 15.536 19.325
Sample_001 14.452 17.863 16.191 21.078 18.441 19.770 14.753 <NA> 20.333 14.368 ... 16.122 13.908 15.847 14.373 19.897 17.718 17.442 17.092 15.728 18.975
Sample_002 15.626 17.657 16.066 21.200 18.962 20.061 <NA> 15.356 19.810 15.115 ... 15.382 13.896 17.568 13.669 19.614 17.001 17.405 17.747 15.819 19.321
Sample_003 16.198 18.431 16.351 20.724 18.793 20.190 15.293 <NA> 20.073 14.792 ... 16.560 14.518 18.165 <NA> 20.165 17.206 17.540 17.478 15.510 18.948
Sample_004 15.963 18.572 15.996 21.063 18.417 20.480 16.047 <NA> 19.781 15.091 ... 16.412 14.925 15.433 <NA> 19.981 17.619 17.291 17.167 15.329 18.645
... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ...
Sample_205 15.257 18.040 16.352 21.316 18.575 19.833 14.935 14.199 20.525 14.512 ... 15.345 13.565 13.476 <NA> 19.979 15.265 17.098 16.947 15.700 18.838
Sample_206 <NA> 16.568 16.094 20.658 19.186 18.383 16.019 15.497 21.101 <NA> ... 16.576 9.743 14.366 15.560 19.391 16.971 17.104 18.051 15.277 18.681
Sample_207 15.458 17.986 16.056 20.765 19.045 19.355 15.544 <NA> 20.472 13.836 ... 15.763 13.234 13.925 15.086 19.918 16.664 16.933 17.242 14.869 19.141
Sample_208 15.781 17.211 15.923 20.933 18.211 19.178 15.169 14.099 20.478 13.923 ... 17.555 14.434 <NA> 14.261 19.826 16.253 17.150 16.348 15.466 16.848
Sample_209 15.686 <NA> 15.909 20.361 19.302 19.529 15.646 13.779 21.178 13.918 ... 16.333 13.621 <NA> 13.046 19.422 14.844 16.771 16.592 14.694 18.082

197 rows × 100 columns

Summmary#

Besides the median polish normalization, the structure of the data is not changed too much by the normalization using this Alzheimer example. This notebook can be opened on colab and might be a good starting point for investigating the effect of normalization on your data - or to disect some approaches further.