Batch correction of samples

Batch correction of samples#

We will explore an Alzheimer dataset where the data was collected in four different sites. We will see that the sites have a an effect where the data is in principal component space and in UMAP space. We will then batch correct the data and see how the effect on these plots.

Refers to the acore.batch_correction module.

%pip install acore vuecore

Hide code cell output

 
Requirement already satisfied: acore in /home/docs/checkouts/readthedocs.org/user_builds/analytics-core/envs/stable/lib/python3.11/site-packages (0.3.0)
Requirement already satisfied: vuecore in /home/docs/checkouts/readthedocs.org/user_builds/analytics-core/envs/stable/lib/python3.11/site-packages (0.1.1)
Requirement already satisfied: numpy in /home/docs/checkouts/readthedocs.org/user_builds/analytics-core/envs/stable/lib/python3.11/site-packages (from acore) (2.4.6)
Requirement already satisfied: pandas in /home/docs/checkouts/readthedocs.org/user_builds/analytics-core/envs/stable/lib/python3.11/site-packages (from acore) (2.3.3)
Requirement already satisfied: dsp-pandas in /home/docs/checkouts/readthedocs.org/user_builds/analytics-core/envs/stable/lib/python3.11/site-packages (from acore) (0.0.5)
Requirement already satisfied: scipy in /home/docs/checkouts/readthedocs.org/user_builds/analytics-core/envs/stable/lib/python3.11/site-packages (from acore) (1.17.1)
Requirement already satisfied: scikit-learn>=1.5 in /home/docs/checkouts/readthedocs.org/user_builds/analytics-core/envs/stable/lib/python3.11/site-packages (from acore) (1.9.0)
Requirement already satisfied: gseapy!=1.1.5 in /home/docs/checkouts/readthedocs.org/user_builds/analytics-core/envs/stable/lib/python3.11/site-packages (from acore) (1.2.1)
Requirement already satisfied: pingouin<0.6.0 in /home/docs/checkouts/readthedocs.org/user_builds/analytics-core/envs/stable/lib/python3.11/site-packages (from acore) (0.5.5)
Requirement already satisfied: umap-learn in /home/docs/checkouts/readthedocs.org/user_builds/analytics-core/envs/stable/lib/python3.11/site-packages (from acore) (0.5.12)
Requirement already satisfied: statsmodels in /home/docs/checkouts/readthedocs.org/user_builds/analytics-core/envs/stable/lib/python3.11/site-packages (from acore) (0.14.6)
Requirement already satisfied: combat in /home/docs/checkouts/readthedocs.org/user_builds/analytics-core/envs/stable/lib/python3.11/site-packages (from acore) (0.3.3)
Requirement already satisfied: inmoose in /home/docs/checkouts/readthedocs.org/user_builds/analytics-core/envs/stable/lib/python3.11/site-packages (from acore) (0.9.1)
Requirement already satisfied: requests in /home/docs/checkouts/readthedocs.org/user_builds/analytics-core/envs/stable/lib/python3.11/site-packages (from acore) (2.34.2)
Requirement already satisfied: rarfile in /home/docs/checkouts/readthedocs.org/user_builds/analytics-core/envs/stable/lib/python3.11/site-packages (from acore) (4.2)
Requirement already satisfied: pandera in /home/docs/checkouts/readthedocs.org/user_builds/analytics-core/envs/stable/lib/python3.11/site-packages (from acore) (0.31.1)
Requirement already satisfied: matplotlib in /home/docs/checkouts/readthedocs.org/user_builds/analytics-core/envs/stable/lib/python3.11/site-packages (from pingouin<0.6.0->acore) (3.11.0)
Requirement already satisfied: pandas-flavor in /home/docs/checkouts/readthedocs.org/user_builds/analytics-core/envs/stable/lib/python3.11/site-packages (from pingouin<0.6.0->acore) (0.8.1)
Requirement already satisfied: seaborn in /home/docs/checkouts/readthedocs.org/user_builds/analytics-core/envs/stable/lib/python3.11/site-packages (from pingouin<0.6.0->acore) (0.13.2)
Requirement already satisfied: tabulate in /home/docs/checkouts/readthedocs.org/user_builds/analytics-core/envs/stable/lib/python3.11/site-packages (from pingouin<0.6.0->acore) (0.10.0)
Requirement already satisfied: plotly in /home/docs/checkouts/readthedocs.org/user_builds/analytics-core/envs/stable/lib/python3.11/site-packages (from vuecore) (6.8.0)
Requirement already satisfied: beautifulsoup4 in /home/docs/checkouts/readthedocs.org/user_builds/analytics-core/envs/stable/lib/python3.11/site-packages (from vuecore) (4.15.0)
Requirement already satisfied: dash in /home/docs/checkouts/readthedocs.org/user_builds/analytics-core/envs/stable/lib/python3.11/site-packages (from vuecore) (4.2.0)
Requirement already satisfied: networkx in /home/docs/checkouts/readthedocs.org/user_builds/analytics-core/envs/stable/lib/python3.11/site-packages (from vuecore) (3.6.1)
Requirement already satisfied: kaleido in /home/docs/checkouts/readthedocs.org/user_builds/analytics-core/envs/stable/lib/python3.11/site-packages (from vuecore) (1.3.0)
Requirement already satisfied: pyvis in /home/docs/checkouts/readthedocs.org/user_builds/analytics-core/envs/stable/lib/python3.11/site-packages (from vuecore) (0.3.2)
Requirement already satisfied: wordcloud in /home/docs/checkouts/readthedocs.org/user_builds/analytics-core/envs/stable/lib/python3.11/site-packages (from vuecore) (1.9.6)
Requirement already satisfied: cyjupyter in /home/docs/checkouts/readthedocs.org/user_builds/analytics-core/envs/stable/lib/python3.11/site-packages (from vuecore) (0.2.0)
Requirement already satisfied: nltk in /home/docs/checkouts/readthedocs.org/user_builds/analytics-core/envs/stable/lib/python3.11/site-packages (from vuecore) (3.9.4)
Requirement already satisfied: webweb in /home/docs/checkouts/readthedocs.org/user_builds/analytics-core/envs/stable/lib/python3.11/site-packages (from vuecore) (0.0.37)
Requirement already satisfied: dash-cytoscape in /home/docs/checkouts/readthedocs.org/user_builds/analytics-core/envs/stable/lib/python3.11/site-packages (from vuecore) (1.0.2)
Requirement already satisfied: pydantic in /home/docs/checkouts/readthedocs.org/user_builds/analytics-core/envs/stable/lib/python3.11/site-packages (from vuecore) (2.13.4)
Requirement already satisfied: nbformat>=4.2.0 in /home/docs/checkouts/readthedocs.org/user_builds/analytics-core/envs/stable/lib/python3.11/site-packages (from vuecore) (5.10.4)
Requirement already satisfied: contourpy>=1.0.1 in /home/docs/checkouts/readthedocs.org/user_builds/analytics-core/envs/stable/lib/python3.11/site-packages (from matplotlib->pingouin<0.6.0->acore) (1.3.3)
Requirement already satisfied: cycler>=0.10 in /home/docs/checkouts/readthedocs.org/user_builds/analytics-core/envs/stable/lib/python3.11/site-packages (from matplotlib->pingouin<0.6.0->acore) (0.12.1)
Requirement already satisfied: fonttools>=4.22.0 in /home/docs/checkouts/readthedocs.org/user_builds/analytics-core/envs/stable/lib/python3.11/site-packages (from matplotlib->pingouin<0.6.0->acore) (4.63.0)
Requirement already satisfied: kiwisolver>=1.3.1 in /home/docs/checkouts/readthedocs.org/user_builds/analytics-core/envs/stable/lib/python3.11/site-packages (from matplotlib->pingouin<0.6.0->acore) (1.5.0)
Requirement already satisfied: packaging>=20.0 in /home/docs/checkouts/readthedocs.org/user_builds/analytics-core/envs/stable/lib/python3.11/site-packages (from matplotlib->pingouin<0.6.0->acore) (26.2)
Requirement already satisfied: pillow>=9 in /home/docs/checkouts/readthedocs.org/user_builds/analytics-core/envs/stable/lib/python3.11/site-packages (from matplotlib->pingouin<0.6.0->acore) (12.2.0)
Requirement already satisfied: pyparsing>=3 in /home/docs/checkouts/readthedocs.org/user_builds/analytics-core/envs/stable/lib/python3.11/site-packages (from matplotlib->pingouin<0.6.0->acore) (3.3.2)
Requirement already satisfied: python-dateutil>=2.7 in /home/docs/checkouts/readthedocs.org/user_builds/analytics-core/envs/stable/lib/python3.11/site-packages (from matplotlib->pingouin<0.6.0->acore) (2.9.0.post0)
Requirement already satisfied: fastjsonschema>=2.15 in /home/docs/checkouts/readthedocs.org/user_builds/analytics-core/envs/stable/lib/python3.11/site-packages (from nbformat>=4.2.0->vuecore) (2.21.2)
Requirement already satisfied: jsonschema>=2.6 in /home/docs/checkouts/readthedocs.org/user_builds/analytics-core/envs/stable/lib/python3.11/site-packages (from nbformat>=4.2.0->vuecore) (4.26.0)
Requirement already satisfied: jupyter-core!=5.0.*,>=4.12 in /home/docs/checkouts/readthedocs.org/user_builds/analytics-core/envs/stable/lib/python3.11/site-packages (from nbformat>=4.2.0->vuecore) (5.9.1)
Requirement already satisfied: traitlets>=5.1 in /home/docs/checkouts/readthedocs.org/user_builds/analytics-core/envs/stable/lib/python3.11/site-packages (from nbformat>=4.2.0->vuecore) (5.15.1)
Requirement already satisfied: attrs>=22.2.0 in /home/docs/checkouts/readthedocs.org/user_builds/analytics-core/envs/stable/lib/python3.11/site-packages (from jsonschema>=2.6->nbformat>=4.2.0->vuecore) (26.1.0)
Requirement already satisfied: jsonschema-specifications>=2023.03.6 in /home/docs/checkouts/readthedocs.org/user_builds/analytics-core/envs/stable/lib/python3.11/site-packages (from jsonschema>=2.6->nbformat>=4.2.0->vuecore) (2025.9.1)
Requirement already satisfied: referencing>=0.28.4 in /home/docs/checkouts/readthedocs.org/user_builds/analytics-core/envs/stable/lib/python3.11/site-packages (from jsonschema>=2.6->nbformat>=4.2.0->vuecore) (0.37.0)
Requirement already satisfied: rpds-py>=0.25.0 in /home/docs/checkouts/readthedocs.org/user_builds/analytics-core/envs/stable/lib/python3.11/site-packages (from jsonschema>=2.6->nbformat>=4.2.0->vuecore) (2026.5.1)
Requirement already satisfied: platformdirs>=2.5 in /home/docs/checkouts/readthedocs.org/user_builds/analytics-core/envs/stable/lib/python3.11/site-packages (from jupyter-core!=5.0.*,>=4.12->nbformat>=4.2.0->vuecore) (4.10.0)
Requirement already satisfied: pytz>=2020.1 in /home/docs/checkouts/readthedocs.org/user_builds/analytics-core/envs/stable/lib/python3.11/site-packages (from pandas->acore) (2026.2)
Requirement already satisfied: tzdata>=2022.7 in /home/docs/checkouts/readthedocs.org/user_builds/analytics-core/envs/stable/lib/python3.11/site-packages (from pandas->acore) (2026.2)
Requirement already satisfied: six>=1.5 in /home/docs/checkouts/readthedocs.org/user_builds/analytics-core/envs/stable/lib/python3.11/site-packages (from python-dateutil>=2.7->matplotlib->pingouin<0.6.0->acore) (1.17.0)
Requirement already satisfied: typing-extensions>=4.4.0 in /home/docs/checkouts/readthedocs.org/user_builds/analytics-core/envs/stable/lib/python3.11/site-packages (from referencing>=0.28.4->jsonschema>=2.6->nbformat>=4.2.0->vuecore) (4.15.0)
Requirement already satisfied: joblib>=1.4.0 in /home/docs/checkouts/readthedocs.org/user_builds/analytics-core/envs/stable/lib/python3.11/site-packages (from scikit-learn>=1.5->acore) (1.5.3)
Requirement already satisfied: narwhals>=2.0.1 in /home/docs/checkouts/readthedocs.org/user_builds/analytics-core/envs/stable/lib/python3.11/site-packages (from scikit-learn>=1.5->acore) (2.22.1)
Requirement already satisfied: threadpoolctl>=3.5.0 in /home/docs/checkouts/readthedocs.org/user_builds/analytics-core/envs/stable/lib/python3.11/site-packages (from scikit-learn>=1.5->acore) (3.6.0)
Requirement already satisfied: soupsieve>=1.6.1 in /home/docs/checkouts/readthedocs.org/user_builds/analytics-core/envs/stable/lib/python3.11/site-packages (from beautifulsoup4->vuecore) (2.8.4)
Requirement already satisfied: mpmath>=1.1.0 in /home/docs/checkouts/readthedocs.org/user_builds/analytics-core/envs/stable/lib/python3.11/site-packages (from combat->acore) (1.4.1)
Requirement already satisfied: patsy>=0.5.1 in /home/docs/checkouts/readthedocs.org/user_builds/analytics-core/envs/stable/lib/python3.11/site-packages (from combat->acore) (1.0.2)
Requirement already satisfied: ipywidgets>=7.0.0 in /home/docs/checkouts/readthedocs.org/user_builds/analytics-core/envs/stable/lib/python3.11/site-packages (from cyjupyter->vuecore) (8.1.8)
Requirement already satisfied: comm>=0.1.3 in /home/docs/checkouts/readthedocs.org/user_builds/analytics-core/envs/stable/lib/python3.11/site-packages (from ipywidgets>=7.0.0->cyjupyter->vuecore) (0.2.3)
Requirement already satisfied: ipython>=6.1.0 in /home/docs/checkouts/readthedocs.org/user_builds/analytics-core/envs/stable/lib/python3.11/site-packages (from ipywidgets>=7.0.0->cyjupyter->vuecore) (9.14.1)
Requirement already satisfied: widgetsnbextension~=4.0.14 in /home/docs/checkouts/readthedocs.org/user_builds/analytics-core/envs/stable/lib/python3.11/site-packages (from ipywidgets>=7.0.0->cyjupyter->vuecore) (4.0.15)
Requirement already satisfied: jupyterlab_widgets~=3.0.15 in /home/docs/checkouts/readthedocs.org/user_builds/analytics-core/envs/stable/lib/python3.11/site-packages (from ipywidgets>=7.0.0->cyjupyter->vuecore) (3.0.16)
Requirement already satisfied: decorator>=5.1.0 in /home/docs/checkouts/readthedocs.org/user_builds/analytics-core/envs/stable/lib/python3.11/site-packages (from ipython>=6.1.0->ipywidgets>=7.0.0->cyjupyter->vuecore) (5.3.1)
Requirement already satisfied: ipython-pygments-lexers>=1.0.0 in /home/docs/checkouts/readthedocs.org/user_builds/analytics-core/envs/stable/lib/python3.11/site-packages (from ipython>=6.1.0->ipywidgets>=7.0.0->cyjupyter->vuecore) (1.1.1)
Requirement already satisfied: jedi>=0.18.2 in /home/docs/checkouts/readthedocs.org/user_builds/analytics-core/envs/stable/lib/python3.11/site-packages (from ipython>=6.1.0->ipywidgets>=7.0.0->cyjupyter->vuecore) (0.20.0)
Requirement already satisfied: matplotlib-inline>=0.1.6 in /home/docs/checkouts/readthedocs.org/user_builds/analytics-core/envs/stable/lib/python3.11/site-packages (from ipython>=6.1.0->ipywidgets>=7.0.0->cyjupyter->vuecore) (0.2.2)
Requirement already satisfied: pexpect>4.6 in /home/docs/checkouts/readthedocs.org/user_builds/analytics-core/envs/stable/lib/python3.11/site-packages (from ipython>=6.1.0->ipywidgets>=7.0.0->cyjupyter->vuecore) (4.9.0)
Requirement already satisfied: prompt_toolkit<3.1.0,>=3.0.41 in /home/docs/checkouts/readthedocs.org/user_builds/analytics-core/envs/stable/lib/python3.11/site-packages (from ipython>=6.1.0->ipywidgets>=7.0.0->cyjupyter->vuecore) (3.0.52)
Requirement already satisfied: psutil>=7 in /home/docs/checkouts/readthedocs.org/user_builds/analytics-core/envs/stable/lib/python3.11/site-packages (from ipython>=6.1.0->ipywidgets>=7.0.0->cyjupyter->vuecore) (7.2.2)
Requirement already satisfied: pygments>=2.14.0 in /home/docs/checkouts/readthedocs.org/user_builds/analytics-core/envs/stable/lib/python3.11/site-packages (from ipython>=6.1.0->ipywidgets>=7.0.0->cyjupyter->vuecore) (2.20.0)
Requirement already satisfied: stack_data>=0.6.0 in /home/docs/checkouts/readthedocs.org/user_builds/analytics-core/envs/stable/lib/python3.11/site-packages (from ipython>=6.1.0->ipywidgets>=7.0.0->cyjupyter->vuecore) (0.6.3)
Requirement already satisfied: wcwidth in /home/docs/checkouts/readthedocs.org/user_builds/analytics-core/envs/stable/lib/python3.11/site-packages (from prompt_toolkit<3.1.0,>=3.0.41->ipython>=6.1.0->ipywidgets>=7.0.0->cyjupyter->vuecore) (0.8.1)
Requirement already satisfied: parso<0.9.0,>=0.8.6 in /home/docs/checkouts/readthedocs.org/user_builds/analytics-core/envs/stable/lib/python3.11/site-packages (from jedi>=0.18.2->ipython>=6.1.0->ipywidgets>=7.0.0->cyjupyter->vuecore) (0.8.7)
Requirement already satisfied: ptyprocess>=0.5 in /home/docs/checkouts/readthedocs.org/user_builds/analytics-core/envs/stable/lib/python3.11/site-packages (from pexpect>4.6->ipython>=6.1.0->ipywidgets>=7.0.0->cyjupyter->vuecore) (0.7.0)
Requirement already satisfied: executing>=1.2.0 in /home/docs/checkouts/readthedocs.org/user_builds/analytics-core/envs/stable/lib/python3.11/site-packages (from stack_data>=0.6.0->ipython>=6.1.0->ipywidgets>=7.0.0->cyjupyter->vuecore) (2.2.1)
Requirement already satisfied: asttokens>=2.1.0 in /home/docs/checkouts/readthedocs.org/user_builds/analytics-core/envs/stable/lib/python3.11/site-packages (from stack_data>=0.6.0->ipython>=6.1.0->ipywidgets>=7.0.0->cyjupyter->vuecore) (3.0.1)
Requirement already satisfied: pure-eval in /home/docs/checkouts/readthedocs.org/user_builds/analytics-core/envs/stable/lib/python3.11/site-packages (from stack_data>=0.6.0->ipython>=6.1.0->ipywidgets>=7.0.0->cyjupyter->vuecore) (0.2.3)
Requirement already satisfied: Flask<3.2,>=1.0.4 in /home/docs/checkouts/readthedocs.org/user_builds/analytics-core/envs/stable/lib/python3.11/site-packages (from dash->vuecore) (3.1.3)
Requirement already satisfied: Werkzeug<3.2 in /home/docs/checkouts/readthedocs.org/user_builds/analytics-core/envs/stable/lib/python3.11/site-packages (from dash->vuecore) (3.1.8)
Requirement already satisfied: importlib-metadata in /home/docs/checkouts/readthedocs.org/user_builds/analytics-core/envs/stable/lib/python3.11/site-packages (from dash->vuecore) (9.0.0)
Requirement already satisfied: retrying in /home/docs/checkouts/readthedocs.org/user_builds/analytics-core/envs/stable/lib/python3.11/site-packages (from dash->vuecore) (1.4.2)
Requirement already satisfied: nest-asyncio in /home/docs/checkouts/readthedocs.org/user_builds/analytics-core/envs/stable/lib/python3.11/site-packages (from dash->vuecore) (1.6.0)
Requirement already satisfied: setuptools in /home/docs/checkouts/readthedocs.org/user_builds/analytics-core/envs/stable/lib/python3.11/site-packages (from dash->vuecore) (82.0.1)
Requirement already satisfied: janus>=1.0.0 in /home/docs/checkouts/readthedocs.org/user_builds/analytics-core/envs/stable/lib/python3.11/site-packages (from dash->vuecore) (2.0.0)
Requirement already satisfied: blinker>=1.9.0 in /home/docs/checkouts/readthedocs.org/user_builds/analytics-core/envs/stable/lib/python3.11/site-packages (from Flask<3.2,>=1.0.4->dash->vuecore) (1.9.0)
Requirement already satisfied: click>=8.1.3 in /home/docs/checkouts/readthedocs.org/user_builds/analytics-core/envs/stable/lib/python3.11/site-packages (from Flask<3.2,>=1.0.4->dash->vuecore) (8.4.1)
Requirement already satisfied: itsdangerous>=2.2.0 in /home/docs/checkouts/readthedocs.org/user_builds/analytics-core/envs/stable/lib/python3.11/site-packages (from Flask<3.2,>=1.0.4->dash->vuecore) (2.2.0)
Requirement already satisfied: jinja2>=3.1.2 in /home/docs/checkouts/readthedocs.org/user_builds/analytics-core/envs/stable/lib/python3.11/site-packages (from Flask<3.2,>=1.0.4->dash->vuecore) (3.1.6)
Requirement already satisfied: markupsafe>=2.1.1 in /home/docs/checkouts/readthedocs.org/user_builds/analytics-core/envs/stable/lib/python3.11/site-packages (from Flask<3.2,>=1.0.4->dash->vuecore) (3.0.3)
Requirement already satisfied: openpyxl in /home/docs/checkouts/readthedocs.org/user_builds/analytics-core/envs/stable/lib/python3.11/site-packages (from dsp-pandas->acore) (3.1.5)
Requirement already satisfied: zipp>=3.20 in /home/docs/checkouts/readthedocs.org/user_builds/analytics-core/envs/stable/lib/python3.11/site-packages (from importlib-metadata->dash->vuecore) (4.1.0)
Requirement already satisfied: anndata in /home/docs/checkouts/readthedocs.org/user_builds/analytics-core/envs/stable/lib/python3.11/site-packages (from inmoose->acore) (0.12.17)
Requirement already satisfied: fastcluster in /home/docs/checkouts/readthedocs.org/user_builds/analytics-core/envs/stable/lib/python3.11/site-packages (from inmoose->acore) (1.3.0)
Requirement already satisfied: array-api-compat>=1.7.1 in /home/docs/checkouts/readthedocs.org/user_builds/analytics-core/envs/stable/lib/python3.11/site-packages (from anndata->inmoose->acore) (1.15.0)
Requirement already satisfied: h5py>=3.8 in /home/docs/checkouts/readthedocs.org/user_builds/analytics-core/envs/stable/lib/python3.11/site-packages (from anndata->inmoose->acore) (3.16.0)
Requirement already satisfied: legacy-api-wrap in /home/docs/checkouts/readthedocs.org/user_builds/analytics-core/envs/stable/lib/python3.11/site-packages (from anndata->inmoose->acore) (1.5)
Requirement already satisfied: natsort in /home/docs/checkouts/readthedocs.org/user_builds/analytics-core/envs/stable/lib/python3.11/site-packages (from anndata->inmoose->acore) (8.4.0)
Requirement already satisfied: scverse-misc>=0.0.3 in /home/docs/checkouts/readthedocs.org/user_builds/analytics-core/envs/stable/lib/python3.11/site-packages (from anndata->inmoose->acore) (0.0.3)
Requirement already satisfied: zarr!=3.0.*,>=2.18.7 in /home/docs/checkouts/readthedocs.org/user_builds/analytics-core/envs/stable/lib/python3.11/site-packages (from anndata->inmoose->acore) (3.1.6)
Requirement already satisfied: session-info2 in /home/docs/checkouts/readthedocs.org/user_builds/analytics-core/envs/stable/lib/python3.11/site-packages (from scverse-misc>=0.0.3->anndata->inmoose->acore) (0.4.1)
Requirement already satisfied: donfig>=0.8 in /home/docs/checkouts/readthedocs.org/user_builds/analytics-core/envs/stable/lib/python3.11/site-packages (from zarr!=3.0.*,>=2.18.7->anndata->inmoose->acore) (0.8.1.post1)
Requirement already satisfied: google-crc32c>=1.5 in /home/docs/checkouts/readthedocs.org/user_builds/analytics-core/envs/stable/lib/python3.11/site-packages (from zarr!=3.0.*,>=2.18.7->anndata->inmoose->acore) (1.8.0)
Requirement already satisfied: numcodecs>=0.14 in /home/docs/checkouts/readthedocs.org/user_builds/analytics-core/envs/stable/lib/python3.11/site-packages (from zarr!=3.0.*,>=2.18.7->anndata->inmoose->acore) (0.16.5)
Requirement already satisfied: pyyaml in /home/docs/checkouts/readthedocs.org/user_builds/analytics-core/envs/stable/lib/python3.11/site-packages (from donfig>=0.8->zarr!=3.0.*,>=2.18.7->anndata->inmoose->acore) (6.0.3)
Requirement already satisfied: choreographer>=1.3.0 in /home/docs/checkouts/readthedocs.org/user_builds/analytics-core/envs/stable/lib/python3.11/site-packages (from kaleido->vuecore) (1.3.0)
Requirement already satisfied: logistro>=1.0.8 in /home/docs/checkouts/readthedocs.org/user_builds/analytics-core/envs/stable/lib/python3.11/site-packages (from kaleido->vuecore) (2.0.1)
Requirement already satisfied: orjson>=3.10.15 in /home/docs/checkouts/readthedocs.org/user_builds/analytics-core/envs/stable/lib/python3.11/site-packages (from kaleido->vuecore) (3.11.9)
Requirement already satisfied: simplejson>=3.19.3 in /home/docs/checkouts/readthedocs.org/user_builds/analytics-core/envs/stable/lib/python3.11/site-packages (from choreographer>=1.3.0->kaleido->vuecore) (4.1.1)
Requirement already satisfied: regex>=2021.8.3 in /home/docs/checkouts/readthedocs.org/user_builds/analytics-core/envs/stable/lib/python3.11/site-packages (from nltk->vuecore) (2026.5.9)
Requirement already satisfied: tqdm in /home/docs/checkouts/readthedocs.org/user_builds/analytics-core/envs/stable/lib/python3.11/site-packages (from nltk->vuecore) (4.68.2)
Requirement already satisfied: et-xmlfile in /home/docs/checkouts/readthedocs.org/user_builds/analytics-core/envs/stable/lib/python3.11/site-packages (from openpyxl->dsp-pandas->acore) (2.0.0)
Requirement already satisfied: xarray in /home/docs/checkouts/readthedocs.org/user_builds/analytics-core/envs/stable/lib/python3.11/site-packages (from pandas-flavor->pingouin<0.6.0->acore) (2026.4.0)
Requirement already satisfied: typeguard in /home/docs/checkouts/readthedocs.org/user_builds/analytics-core/envs/stable/lib/python3.11/site-packages (from pandera->acore) (4.5.2)
Requirement already satisfied: typing_inspect>=0.6.0 in /home/docs/checkouts/readthedocs.org/user_builds/analytics-core/envs/stable/lib/python3.11/site-packages (from pandera->acore) (0.9.0)
Requirement already satisfied: mypy-extensions>=0.3.0 in /home/docs/checkouts/readthedocs.org/user_builds/analytics-core/envs/stable/lib/python3.11/site-packages (from typing_inspect>=0.6.0->pandera->acore) (1.1.0)
Requirement already satisfied: annotated-types>=0.6.0 in /home/docs/checkouts/readthedocs.org/user_builds/analytics-core/envs/stable/lib/python3.11/site-packages (from pydantic->vuecore) (0.7.0)
Requirement already satisfied: pydantic-core==2.46.4 in /home/docs/checkouts/readthedocs.org/user_builds/analytics-core/envs/stable/lib/python3.11/site-packages (from pydantic->vuecore) (2.46.4)
Requirement already satisfied: typing-inspection>=0.4.2 in /home/docs/checkouts/readthedocs.org/user_builds/analytics-core/envs/stable/lib/python3.11/site-packages (from pydantic->vuecore) (0.4.2)
Requirement already satisfied: jsonpickle>=1.4.1 in /home/docs/checkouts/readthedocs.org/user_builds/analytics-core/envs/stable/lib/python3.11/site-packages (from pyvis->vuecore) (4.1.2)
Requirement already satisfied: charset_normalizer<4,>=2 in /home/docs/checkouts/readthedocs.org/user_builds/analytics-core/envs/stable/lib/python3.11/site-packages (from requests->acore) (3.4.7)
Requirement already satisfied: idna<4,>=2.5 in /home/docs/checkouts/readthedocs.org/user_builds/analytics-core/envs/stable/lib/python3.11/site-packages (from requests->acore) (3.18)
Requirement already satisfied: urllib3<3,>=1.26 in /home/docs/checkouts/readthedocs.org/user_builds/analytics-core/envs/stable/lib/python3.11/site-packages (from requests->acore) (2.7.0)
Requirement already satisfied: certifi>=2023.5.7 in /home/docs/checkouts/readthedocs.org/user_builds/analytics-core/envs/stable/lib/python3.11/site-packages (from requests->acore) (2026.5.20)
Requirement already satisfied: numba>=0.51.2 in /home/docs/checkouts/readthedocs.org/user_builds/analytics-core/envs/stable/lib/python3.11/site-packages (from umap-learn->acore) (0.65.1)
Requirement already satisfied: pynndescent>=0.5 in /home/docs/checkouts/readthedocs.org/user_builds/analytics-core/envs/stable/lib/python3.11/site-packages (from umap-learn->acore) (0.6.0)
Requirement already satisfied: llvmlite<0.48,>=0.47.0dev0 in /home/docs/checkouts/readthedocs.org/user_builds/analytics-core/envs/stable/lib/python3.11/site-packages (from numba>=0.51.2->umap-learn->acore) (0.47.0)
Note: you may need to restart the kernel to use updated packages.

Hide code cell source

 
from typing import Optional

import dsp_pandas
import matplotlib.pyplot as plt
import pandas as pd
import sklearn
import sklearn.impute
import sklearn.preprocessing
import vuecore.decomposition

import acore.batch_correction
import acore.decomposition


def plot_umap(X_scaled, y, meta_column=None, random_state=42) -> plt.Axes:
    """Fit and plot UMAP embedding with two components with colors defined by meta_column."""
    embedding = acore.decomposition.umap.run_umap(
        X_scaled, y, random_state=random_state
    )
    if meta_column is None:
        meta_column = y.name
    ax = embedding.plot.scatter("UMAP 1", "UMAP 2", c=meta_column, cmap="Paired")
    return ax


def standard_normalize(X: pd.DataFrame) -> pd.DataFrame:
    """Standard normalize data and keep indices of DataFrame."""
    X_scaled = (
        sklearn.preprocessing.StandardScaler()
        .set_output(transform="pandas")
        .fit_transform(X)
    )
    return X_scaled


def median_impute(X: pd.DataFrame) -> pd.DataFrame:
    X_imputed = (
        sklearn.impute.SimpleImputer(strategy="median")
        .set_output(transform="pandas")
        .fit_transform(X)
    )
    return X_imputed


def run_and_plot_pca(
    X_scaled,
    y,
    meta_column: Optional[str] = None,
    n_components: int = 4,
) -> tuple[pd.DataFrame, plt.Figure]:
    PCs, _ = acore.decomposition.pca.run_pca(X_scaled, n_components=n_components)
    PCs.columns = [s.replace("principal component", "PC") for s in PCs.columns]
    fig = vuecore.decomposition.pca_grid(
        PCs=PCs, meta_column=y, n_components=n_components, meta_col_name=meta_column
    )
    return PCs, fig


dsp_pandas.format.set_pandas_options(
    max_columns=9,
    max_colwidth=20,
)

Set some parameters#

BASE = (
    "https://raw.githubusercontent.com/Multiomics-Analytics-Group/acore/"
    "main/example_data/alzheimer_proteomics/"
)
# data is already preprocessed: log2, filtered
fname: str = "alzheimer_example_omics_and_clinic.csv"  # combined omics and meta data
covariates: list[str] = ["age", "male"]
group: str = "collection_site"
subject_col: str = "Sample ID"
drop_cols: list[str] = ["AD"]
factor_and_covars: list[str] = [group, *covariates]
group_label: Optional[str] = "site"  # optional: rename target variable

Data loading#

Use combined dataset for ANCOVA analysis.

Hide code cell source

 
omics_and_meta = (
    pd.read_csv(f"{BASE}/{fname}", index_col=subject_col)
    .convert_dtypes()
    .dropna(subset=factor_and_covars)
)
omics_and_meta
AD age male collection_site ... Q9Y5I4 P55283 A1L4H1 Q7Z4T9
Sample ID
Sample_000 0 71 0 Sweden ... 17.187 17.422 15.542 19.331
Sample_001 1 77 1 Sweden ... 17.447 17.097 15.734 18.980
Sample_002 1 75 1 Sweden ... 17.410 17.752 15.824 19.326
Sample_003 1 72 0 Sweden ... 17.545 17.483 15.515 18.953
Sample_004 1 63 0 Sweden ... 17.297 17.172 15.334 18.651
... ... ... ... ... ... ... ... ... ...
Sample_205 1 69 0 Berlin ... 17.104 16.952 15.705 18.844
Sample_206 0 73 1 Berlin ... 17.109 18.056 15.282 18.686
Sample_207 0 71 0 Berlin ... 16.938 17.248 14.874 19.146
Sample_208 0 83 1 Berlin ... 17.155 16.353 15.471 16.853
Sample_209 0 63 0 Berlin ... 16.776 16.597 14.699 18.087

197 rows × 104 columns

Metadata here is of type integer. All floats are proteomics measurements.

Hide code cell source

 
omics_and_meta.dtypes.value_counts()

Float64          100
Int64              3
string[python]     1
Name: count, dtype: int64

Hide code cell source

 
omics_and_meta[factor_and_covars]
collection_site age male
Sample ID
Sample_000 Sweden 71 0
Sample_001 Sweden 77 1
Sample_002 Sweden 75 1
Sample_003 Sweden 72 0
Sample_004 Sweden 63 0
... ... ... ...
Sample_205 Berlin 69 0
Sample_206 Berlin 73 1
Sample_207 Berlin 71 0
Sample_208 Berlin 83 1
Sample_209 Berlin 63 0

197 rows × 3 columns

omics = omics_and_meta.drop(columns=[*factor_and_covars, *drop_cols])
y = omics_and_meta[group].astype("category").rename(group_label)

Before batch correction#

Explore data in PCA and UMAP space before batch correction

Hide code cell source

 
omics_imp = median_impute(omics)
omics_imp_scaled = standard_normalize(omics_imp)
PCs, fig = run_and_plot_pca(omics_imp, y, n_components=4)
ax = plot_umap(omics_imp, y)
../_images/b5d14773ea52bd8d304a964f3467c1731d62e98e400b3e85e4a25e35ec8897a4.png ../_images/1ded9e2d34faa7d5fcac241af05cd960b78429ab85c62d528034ad695c056554.png

Combat batch correction#

Correct for batch effects in the data using a robust regression approach removing mean and scale effects out for each provided co-variate by batch. Assumes normally distributed data.

⚠️ Combat needs imputed data

%%time
X = median_impute(omics)
X = acore.batch_correction.combat_batch_correction(
    X.join(y.astype("category")),
    batch_col=y.name,
)
X

CPU times: user 28.6 ms, sys: 1.19 ms, total: 29.8 ms
Wall time: 28.6 ms
Q6UX72 O14773 A0A0A0MQU6 P36222 ... Q9Y5I4 P55283 A1L4H1 Q7Z4T9
Sample ID
Sample_000 16.084 18.370 16.431 20.699 ... 16.946 17.013 15.281 19.146
Sample_001 14.542 17.857 16.256 20.849 ... 17.194 16.632 15.577 18.770
Sample_002 15.680 17.663 16.137 20.984 ... 17.159 17.400 15.716 19.141
Sample_003 16.236 18.394 16.408 20.457 ... 17.287 17.085 15.240 18.741
Sample_004 16.007 18.526 16.071 20.832 ... 17.051 16.721 14.961 18.417
... ... ... ... ... ... ... ... ... ...
Sample_205 15.222 18.208 16.436 21.341 ... 17.103 17.214 15.683 18.974
Sample_206 15.406 16.736 16.157 20.715 ... 17.109 18.135 15.344 18.833
Sample_207 15.407 18.153 16.117 20.817 ... 16.932 17.460 15.017 19.244
Sample_208 15.705 17.379 15.974 20.977 ... 17.157 16.713 15.496 17.194
Sample_209 15.618 17.783 15.958 20.434 ... 16.764 16.917 14.877 18.298

197 rows × 100 columns

Plot PCA and UMAP after batch correction on standard normalized data

Hide code cell source

 
PCs, fig = run_and_plot_pca(standard_normalize(X), y, n_components=4)
ax = plot_umap(X, y)
../_images/9feee163452ef95377bce51818ebee4e230a67cb7771da09a394e69e7ee29f26.png ../_images/37baea6e4c1d53bceb8f3d9201d9e148263c0642f13f45de3ac1954112945288.png

See change by substracting combat corrected data from original data.

  • NAs in original data will remain NA below (no imputation done here)

Hide code cell source

 
omics - X
Q6UX72 O14773 A0A0A0MQU6 P36222 ... Q9Y5I4 P55283 A1L4H1 Q7Z4T9
Sample ID
Sample_000 -0.037 0.042 -0.050 0.249 ... 0.241 0.409 0.261 0.185
Sample_001 -0.084 0.011 -0.060 0.234 ... 0.253 0.465 0.157 0.210
Sample_002 -0.049 -0.000 -0.066 0.222 ... 0.251 0.352 0.108 0.185
Sample_003 -0.032 0.043 -0.052 0.272 ... 0.258 0.398 0.275 0.212
Sample_004 -0.039 0.051 -0.070 0.236 ... 0.246 0.452 0.374 0.234
... ... ... ... ... ... ... ... ... ...
Sample_205 0.040 -0.162 -0.078 -0.020 ... 0.000 -0.261 0.022 -0.130
Sample_206 <NA> -0.163 -0.058 -0.053 ... -0.000 -0.079 -0.062 -0.147
Sample_207 0.056 -0.162 -0.055 -0.047 ... 0.006 -0.213 -0.143 -0.098
Sample_208 0.081 -0.163 -0.045 -0.039 ... -0.002 -0.360 -0.024 -0.341
Sample_209 0.074 <NA> -0.044 -0.067 ... 0.012 -0.320 -0.178 -0.210

197 rows × 100 columns

Done.