How do I validate & annotate arbitrary data structures?

This guide walks through the low-level API that lets you validate iterables.

You can then use the records create inferred during validation to annotate a dataset.

How do I validate based on a public ontology?

LaminDB makes it easy to validate categorical variables based on registries that inherit from CanCurate.

CanCurate methods validate against the registries in your LaminDB instance. In Manage biological registries, you’ll see how to extend standard validation to validation against public references using a PubliOntology object, e.g., via public_genes = bt.Gene.public(). By default, from_values() considers a match in a public reference a validated value for any bionty entity.

# pip install 'lamindb[bionty,zarr]'
!lamin init --storage ./test-curate-any --modules bionty

Show code cell output

Hide code cell output

→ initialized lamindb: testuser1/test-curate-any

Define a test dataset.

import lamindb as ln
import bionty as bt
import zarr
import numpy as np

data = zarr.open_group(store="data.zarr", mode="a")

data.create_dataset(name="temperature", shape=(3,), dtype="float32")
data.create_dataset(name="knockout_gene", shape=(3,), dtype=str)
data.create_dataset(name="disease", shape=(3,), dtype=str)

data["knockout_gene"][:] = np.array(
    ["ENSG00000139618", "ENSG00000141510", "ENSG00000133703"]
)
data["disease"][:] = np.random.default_rng().choice(
    ["MONDO:0004975", "MONDO:0004980"], 3
)

→ connected lamindb: testuser1/test-curate-any

Validate and standardize vectors

Read the disease array from the zarr group into memory.

disease = data["disease"][:]

validate() validates vectore-like values against reference values in a registry. It returns a boolean vector indicating where a value has an exact match in the reference values.

bt.Disease.validate(disease, field=bt.Disease.ontology_id)

Show code cell output

Hide code cell output

! Your Disease registry is empty, consider populating it first!
   → use `.import_source()` to import records from a source, e.g. a public ontology

array([False, False, False])

When validation fails, you can call inspect() to figure out what to do.

inspect() applies the same definition of validation as validate(), but returns a rich return value InspectResult. Most importantly, it logs recommended curation steps that would render the data validated.

Note: you can use standardize() to standardize synonyms.

bt.Disease.inspect(disease, field=bt.Disease.ontology_id)

Show code cell output

Hide code cell output

! received 1 unique term, 2 empty/duplicated terms are ignored

! 1 unique term (100.00%) is not validated for ontology_id: 'MONDO:0004980'

   detected 1 Disease term in public source for ontology_id: 'MONDO:0004980'

→  add records from public source to your Disease registry via .from_values()

<lamin_utils._inspect.InspectResult at 0x7faea25b8880>

Bulk creating records using from_values() only returns validated records.

diseases = bt.Disease.from_values(disease, field=bt.Disease.ontology_id).save()

Repeat the process for more labels:

projects = ln.ULabel.from_values(
    ["Project A", "Project B"],
    field=ln.ULabel.name,
    create=True,  # create non-validated labels
).save()
genes = bt.Gene.from_values(
    data["knockout_gene"][:], field=bt.Gene.ensembl_gene_id
).save()

Annotate the dataset

Register the dataset as an artifact:

artifact = ln.Artifact("data.zarr", key="my_dataset.zarr").save()

Show code cell output

Hide code cell output

! no run & transform got linked, call `ln.track()` & re-run

Annotate with features:

ln.Feature(name="project", dtype=ln.ULabel).save()
ln.Feature(name="disease", dtype=bt.Disease.ontology_id).save()
ln.Feature(name="knockout_gene", dtype=bt.Gene.ensembl_gene_id).save()
artifact.features.add_values(
    {"project": projects, "knockout_gene": genes, "disease": diseases}
)
artifact.describe()

Show code cell output

Hide code cell output

Artifact .zarr
├── General
│   ├── uid: BA6NyN3GPv1aoOpz0000          hash: PYt9KjDZ6hAtACPG6rsVSQ
│   ├── size: 1.2 KB                       n_files: 6
│   ├── space: all                         branch: main
│   ├── created_at: 2025-07-14 06:41:02    created_by: testuser1 (Test User1)
│   ├── key: my_dataset.zarr
│   └── storage location / path: 
│       /home/runner/work/lamindb/lamindb/docs/faq/test-curate-any/.lamindb/BA6NyN3GPv1aoOpz.zarr
├── Linked features
│   └── disease                         cat[bionty.Disease.ontology_id]    atopic eczema                           
│       knockout_gene                   cat[bionty.Gene.ensembl_gene_id]   BRCA2, KRAS, TP53                       
│       project                         cat[ULabel]                        Project A, Project B                    
└── Labels
    └── .genes                          bionty.Gene                        BRCA2, TP53, KRAS                       
        .diseases                       bionty.Disease                     atopic eczema                           
        .ulabels                        ULabel                             Project A, Project B                    

Show code cell content

Hide code cell content

# clean up test instance
!rm -r data.zarr
!rm -r ./test-curate-any
!lamin delete --force test-curate-any

• deleting instance testuser1/test-curate-any