ord_schema#
Generic helpers for ord_schema, including common message types.
Subpackages#
- ord_schema.artifacts
- ord_schema.artifacts.arrivals
- ord_schema.artifacts.base
- ord_schema.artifacts.history
- ord_schema.artifacts.occurrences
- ord_schema.artifacts.pivot
- ord_schema.artifacts.projection
- ord_schema.artifacts.structures
- ord_schema.macros
- ord_schema.orm
- ord_schema.orm.database
add_dataset()add_parquet_dataset()add_parquet_dataset_metadata()add_parquet_dataset_row()add_parquet_reactions()backfill_submission_times()delete_dataset()delete_orphaned_rdkit_structures()get_connection_string()get_dataset_md5()get_dataset_size()prepare_database()set_submitted_at()update_derived_data()update_derived_tables()update_rdkit_ids()update_rdkit_tables()
- ord_schema.orm.derived_mappers
- ord_schema.orm.loading
- ord_schema.orm.mappers
- ord_schema.orm.public_mappers
- ord_schema.orm.rdkit_mappers
- ord_schema.orm.sharding
- ord_schema.orm.database
- ord_schema.search
- ord_schema.search.check
- ord_schema.search.execute
- ord_schema.search.nl
- ord_schema.search.nl_eval
- ord_schema.search.nl_log
AskAsk.answer_msAsk.answer_textAsk.attemptsAsk.corpus_fingerprintAsk.declined_reasonAsk.effortAsk.errorAsk.modelAsk.outcomeAsk.prompt_fingerprintAsk.questionAsk.record_idAsk.resolvedAsk.row_countAsk.search_msAsk.session_idAsk.sourceAsk.structuresAsk.timestampAsk.translate_msAsk.turnsAsk.usage
AttemptJsonlSinkObjectStoreOutcomeS3SinkSinkStdoutSinkThumbUsageVerdictcompact()emit()event()label()new_identifier()read()thumb()write()
- ord_schema.search.placeholders
- ord_schema.search.query
- ord_schema.search.schema
- ord_schema.search.sql
ord_schema.atomic_io#
Atomic-write helpers.
Centralizes the write-to-tmp-then-rename pattern used throughout ord_schema so
partial files never appear at a destination path on crash, exception, or interrupt.
- ord_schema.atomic_io.atomic_path(path: str | PathLike[str]) Iterator[str]#
Yields a sibling temp path; renames it onto
pathon clean exit.- Usage:
- with atomic_path(dest) as tmp, open(tmp, “wb”) as f:
f.write(…)
On clean exit the temp path is renamed onto
pathviaos.replace, which is atomic on POSIX when source and destination live on the same filesystem (the temp is a sibling, so this holds). On any exception (includingKeyboardInterrupt) the temp is removed best-effort and the originalpathis left untouched.The temp is created via
tempfile.mkstempin the destination’s directory with a unique random suffix, so concurrent writers to the samepathdo not collide on a fixed sentinel name. The temp file is created (empty) immediately on entry; callers typically open it for writing and overwrite it.
ord_schema.datasets#
Whole-dataset file I/O, dispatching on filename suffix.
The entry point for reading and writing a complete Dataset, whatever it is
serialized as. .parquet reads back as a parquet.DatasetView, which
streams from the file instead of materializing it; every other suffix reads back
as a dataset_pb2.Dataset through message_helpers single-message I/O.
- ord_schema.datasets.load_dataset(filename: str | PathLike[str], *, as_dataset: Literal[False] = False) Dataset | DatasetView#
- ord_schema.datasets.load_dataset(filename: str | PathLike[str], *, as_dataset: Literal[True]) Dataset
- ord_schema.datasets.load_dataset(filename: str | PathLike[str], *, as_dataset: bool) Dataset | DatasetView
Loads a Dataset from disk, dispatching on filename suffix.
A Parquet dataset reads back as a
parquet.DatasetView, which takes its scalars and row count from the file footer and reads Reactions on demand. The view is a read-only stand-in for the message: iteration,len, emptiness, indexing, and slicing overreactionsbehave the same, so code that only reads needs no change. Materializing every Reaction is the exception rather than the default, which matters because published datasets can be large.Other suffixes have no streaming form and read back as a real message.
- Parameters:
filename – Path to a serialized Dataset (
.parquetor any suffix understood bymessage_helpers.load_message).as_dataset – If True, always return a
dataset_pb2.Dataset, materializing a Parquet dataset in full. Use this when the caller needs the protobuf surface – serialization, JSON conversion, mutation – and cannot know which format it was handed; a caller that knows it holds a view can callto_protoinstead.
- Returns:
A
parquet.DatasetViewfor.parquetunlessas_datasetis set, and adataset_pb2.Datasetotherwise.
- ord_schema.datasets.save_dataset(dataset: Dataset, filename: str | PathLike[str]) None#
Writes a Dataset to disk, dispatching on filename suffix.
.parquetroutes toparquet.save_dataset; other suffixes go throughmessage_helpers.save_message.- Parameters:
dataset – Dataset message to write.
filename – Output path; its suffix selects the serialization.
ord_schema.huggingface#
Fetches datasets and derived artifacts from Hugging Face.
Source datasets come from the ord-data mirror. The derived artifacts of
ord_schema.artifacts live in a repository of their own, ord-artifacts, so that
rebuilding them leaves the mirror’s history alone; they are published by
ord_schema.scripts.upload_artifacts.
Importing this module requires the optional huggingface extra (pip install 'ord-
schema[huggingface]'); huggingface_hub is not a default dependency because it
pulls in a heavy chain (typer, fsspec, the hf-xet native binary).
- class ord_schema.huggingface.FetchedArtifacts(revision: str, projections: Path, structures: Path, arrivals: Path, pivots: Path, occurrences: Path)#
Bases:
objectWhere
fetch_artifactsput each artifact, in the shapeCorpusreads.- Variables:
revision (str) – The commit the artifacts were fetched from.
projections (pathlib.Path) – Directory holding the projections.
structures (pathlib.Path) – Directory holding the structures artifacts.
arrivals (pathlib.Path) – Directory holding the arrivals artifacts.
pivots (pathlib.Path) – Directory holding one subdirectory per pivoted level.
occurrences (pathlib.Path) – Directory holding one subdirectory per indexed path.
- arrivals: Path#
- property arrivals_pattern: str#
A glob matching every arrivals artifact, for
Corpus.
- occurrences: Path#
- pivots: Path#
- property projection_pattern: str#
A glob matching every projection, for
Corpus.
- projections: Path#
- revision: str#
- structures: Path#
- property structures_pattern: str#
A glob matching every structures artifact, for
Corpus.
- ord_schema.huggingface.fetch_artifacts(local_dir: str | PathLike[str], *, revision: str = 'main', pivot_levels: Collection[str] | None = None) FetchedArtifacts#
Downloads the derived artifacts ord-artifacts holds, if this library reads them.
The repository holds one build at a time, laid out as a local tree. A
Corpusrefuses artifacts whose schema version or RDKit differs from its own, so the build’s stamps are read in place first and a mismatch is refused before anything is downloaded: a tree holding a build this library reads is not overwritten with one it cannot. An earlier build is fetched by passing the commit that holds it. Files are written underlocal_dirrather than linked into the Hugging Face cache, because aCorpusconfined to its own directories refuses a link leading out of them.Repeat calls download only what changed, and delete the Parquet files under what they fetched that the build no longer holds – a dataset removed upstream would otherwise stay in every local tree that once fetched it. Pivot levels already held under
local_dirare refreshed with the requested ones, since aCorpusrefuses a level left stale beside refreshed projections.- Parameters:
local_dir – Directory to download beneath; the repository’s layout is kept.
revision – Branch, tag, or commit SHA to download (default:
"main"). A branch is resolved to its commit first, so the build is checked and downloaded from the same one.pivot_levels – The pivoted levels to fetch. None fetches every level, which is what
Corpus(require_pivots=True)needs; a level left out is built in process when a query first ranges over it.
- Returns:
The commit fetched, and where each artifact landed.
- Raises:
LookupError – If the revision holds no artifacts of some kind, holds a build of another artifact schema version or RDKit, or lacks a requested pivot level. The message names what it does hold.
ValueError – If a file the build no longer holds sits in a directory reached through a link beneath
local_dir, which is not this function’s to delete.
- ord_schema.huggingface.fetch_dataset(dataset_id: str, *, revision: str = 'main', cache_dir: str | None = None, local_dir: str | None = None) str#
Downloads a dataset file from the ord-data Hugging Face mirror.
Prefers the Parquet serialization and falls back to the legacy
.pb.gzbinary format for datasets that have not been converted yet. The file is not parsed; dispatch on the returned suffix to choose a reader, e.g.parquet.load_dataset/parquet.DatasetViewfor.parquetandload_messagefor.pb.gz.Downloads are cached and content-verified by
huggingface_hub: repeat calls reuse the cached file (re-validating against the remote for moving revisions like"main") instead of re-downloading. By default the cache lives under~/.cache/huggingface/hub; set$HF_HOME(or passcache_dir) to relocate it, or set$HF_HUB_OFFLINE=1to reuse an already-warm cache without network access.- Parameters:
dataset_id – Dataset ID.
revision – Branch, tag, or commit SHA to download (default:
"main").cache_dir – Override the Hugging Face cache directory. Files are stored content-addressed and returned via a symlink into the cache.
local_dir – If set, materialize the file as a real (non-symlink) copy under this directory instead of returning a cache symlink; useful for staging files into a project tree.
- Returns:
Local filesystem path to the downloaded dataset file.
- Raises:
RuntimeError – If no file exists for the dataset.
ValueError – If the dataset ID is invalid.
- ord_schema.huggingface.parquet_files(directory: Path) list[Path]#
Returns the Parquet files under
directorythe way aCorpusfinds them.Through
glob, as the readers do, so a linked shard directory is descended and a dotted file is passed over.- Parameters:
directory – The directory to search beneath.
- Returns:
The paths, sorted. Empty if the directory does not exist.
- ord_schema.huggingface.parquet_pattern(directory: Path) str#
Returns a recursive glob for the Parquet files under
directory, escaped.
- ord_schema.huggingface.published_build(commit: str, published: Iterable[str]) Stamps | None#
Returns the stamps of the build ord-artifacts holds at
commit, read in place.Every file of a published build carries the same artifact schema version and RDKit version, which
upload_artifactschecks, so one footer speaks for the build. It is an occurrences file’s, whose few columns make for a small footer, and nothing else of the file is downloaded.- Parameters:
commit – The commit SHA
publishedwas listed at.published – Every repository path the commit holds.
- Returns:
The stamps, or None if the commit holds no occurrences artifacts.
- ord_schema.huggingface.resolve_revision(revision: str) str#
Returns the commit
revisionnames in ord-artifacts.- Parameters:
revision – Branch, tag, or commit SHA.
- Returns:
The commit SHA.
- Raises:
LookupError – If the Hub reports no commit for it.
ord_schema.logging#
Logging utilities.
- ord_schema.logging.get_logger(name: str, level: int = 20) Logger#
Creates a Logger.
- ord_schema.logging.silence_rdkit_logs(pattern: str = 'rdApp.*') None#
Disables noisy RDKit logs.
ord_schema.message_helpers#
Helper functions for constructing Protocol Buffer messages.
- class ord_schema.message_helpers.MessageFormat(value)#
Bases:
EnumInput/output types for protocol buffer messages.
BINARY/BINPB and PBTXT/TXTPB pairs use the same wire format; the second of each pair is the newer canonical suffix recommended by protobuf.dev.
- BINARY = '.pb'#
- BINPB = '.binpb'#
- JSON = '.json'#
- PBTXT = '.pbtxt'#
- TXTPB = '.txtpb'#
- ord_schema.message_helpers.build_compound(smiles: str | None = None, name: str | None = None, amount: str | None = None, role: str | None = None, is_limiting: bool | None = None, prep: str | None = None, prep_details: str | None = None, vendor: str | None = None) Compound#
Builds a Compound message with the most common fields.
- Parameters:
smiles – Text compound SMILES, recorded as CXSMILES if it has an extension block.
name – Text compound name.
amount – Text amount string, e.g. ‘1.25 g’.
role – Text reaction role. Must match a value in ReactionRoleType.
is_limiting – Boolean whether this compound is limiting for the reaction.
prep – Text compound preparation type. Must match a value in PreparationType.
prep_details – Text compound preparation details. If provided, prep is required.
vendor – Text compound vendor/supplier.
- Returns:
Compound message.
- Raises:
KeyError – if role or prep does not match a supported enum value.
TypeError – if amount units are not supported.
ValueError – if prep_details is provided and prep is None.
- ord_schema.message_helpers.build_data(filename: str, description: str) Data#
Reads raw data from a file and creates a Data message.
- Parameters:
filename – Text filename.
description – Text description of the data.
- Returns:
Data message.
- ord_schema.message_helpers.canonical_smiles(mol: Mol) str#
Returns a canonical SMILES, keeping the bonding plain SMILES cannot express.
Plain SMILES cannot express AND/OR stereo groups, so writing one would silently assert a more specific structure than the source recorded. Coordinate bonds are kept for the same reason: dropping the
|C:...|block leaves a dative bond written as a plain single bond, which is a different molecule and not recoverable by canonicalizing again. Presentation fields – atom labels, coordinates – are omitted.- Parameters:
mol – RDKit Mol.
- Returns:
Canonical SMILES, with a
|a:...|/|o...|/|C:...|block only for molecules that have enhanced stereochemistry or coordinate bonds. Others matchChem.MolToSmilesexactly; molecules with dative bonds do not, sinceMolToSmileswrites those inline as->instead.
- ord_schema.message_helpers.canonical_smiles_for_identifier(identifier_type: int, value: str) str | None#
Canonicalizes a structural identifier, or returns None if it will not parse.
- Parameters:
identifier_type – CompoundIdentifier type enum value.
value – The identifier value.
- Returns:
Canonical SMILES, or None if
valueis not a valid identifier of that type. Returns None for types no Mol can be built from.
- ord_schema.message_helpers.check_compound_identifiers(compound: Compound | ProductCompound) None#
Verifies that structural compound identifiers are consistent.
Compared through
canonical_smiles(), so identifiers that disagree about enhanced stereochemistry are inconsistent: they assert different things about the molecule even though they share a skeleton.- Parameters:
compound – reaction_pb2.Compound message.
- Raises:
ValueError – If structural identifiers are not consistent or are invalid.
- ord_schema.message_helpers.create_message(message_name: str) Message#
Converts a message name into an instantiation of that class.
The message belongs to the reaction_pb2 module.
- Parameters:
message_name – Text name of a message field. For example, “Reaction” or “TemperatureConditions.Measurement”.
- Returns:
Initialized message of the requested type.
- Raises:
ValueError if the name cannot be resolved. –
- ord_schema.message_helpers.derived_reaction_smiles(reaction: Reaction) str | None#
Returns the reaction SMILES that derived artifacts store, or None.
A recorded
REACTION_CXSMILESorREACTION_SMILESwins over generating one, because it carries atom mapping, which generation cannot reconstruct. It is normalized, not copied: agents removed and the result canonicalized, so agent placement and atom ordering stop deciding whether two reactions look alike. A recorded value RDKit cannot read, or that survives as a half reaction, falls through to generation.Either way the result carries no agents. An empty agent block is idiomatic, and excluding agents keeps a reagent, solvent, or catalyst recorded only by name – routine for ligands – from deciding whether the reaction gets a SMILES at all. Generation is otherwise strict: every reactant and product must be readable, since a SMILES silently missing one describes a different reaction and nothing marks it.
- Parameters:
reaction – Reaction message.
- Returns:
Canonical
reactants>>productsSMILES, or None if nothing recorded can be read and nothing complete can be generated.
- ord_schema.message_helpers.find_submessages(message: Message, submessage_type: type[MessageType]) list[MessageType]#
Recursively finds all submessages of a specified type.
- Parameters:
message – Protocol buffer.
submessage_type – Protocol buffer type.
- Returns:
List of messages.
- Raises:
TypeError – if submessage_type is not a protocol buffer type.
- ord_schema.message_helpers.generate_reaction_smiles(message: Reaction, *, allow_incomplete: bool = True, allow_unspecified_roles: bool = True, include_agents: bool = True) str#
Builds a reaction SMILES from a Reaction’s components, ignoring its identifiers.
- Parameters:
message – reaction_pb2.Reaction message.
allow_incomplete – Whether to omit components with no readable structure rather than raising. Only components the result would carry are considered, so an unreadable agent is irrelevant when
include_agentsis False. Tolerates missing components but not a missing half: RDKit calls a reaction with no reactants or no products an error however it got that way.allow_unspecified_roles – If True, components with the UNSPECIFIED reaction role are treated as reactants and products.
include_agents – Whether to emit the middle
>agents>block. Reagents, solvents, and catalysts are dropped when False, which leaves the SMILES describing the transformation alone.
- Returns:
Canonical reaction CXSMILES, so enhanced stereochemistry recorded on a component survives into the reaction.
- Raises:
ValueError – If a component the result would carry has no readable structure and
allow_incompleteis False, if there is no reactant or no product, or if RDKit reports errors in the assembled reaction.
- ord_schema.message_helpers.get_compound_identifier(compound: Compound | ProductCompound, identifier_type: <google.protobuf.internal.enum_type_wrapper.EnumTypeWrapper object at 0x7265551adb50>) str | None#
Returns the value of a compound identifier if it exists.
If multiple identifiers of that type exist, only the first is returned.
- Parameters:
compound – Compound message.
identifier_type – The CompoundIdentifier type to retrieve the value of.
- Returns:
Identifier value or None if the identifier is not defined.
- ord_schema.message_helpers.get_compound_molblock(compound: Compound | ProductCompound) str | None#
Returns the value of the compound’s MOLBLOCK identifier if it exists.
- Parameters:
compound – Compound message.
- Returns:
MOLBLOCK string or None if the compound has no MOLBLOCK identifier.
- ord_schema.message_helpers.get_compound_name(compound: Compound) str | None#
Returns the value of the compound’s NAME identifier if it exists.
- Parameters:
compound – Compound message.
- Returns:
NAME string or None if the compound has no NAME identifier.
- ord_schema.message_helpers.get_compound_smiles(compound: Compound | ProductCompound) str | None#
Returns the value of the compound’s SMILES identifier if it exists.
- Parameters:
compound – Compound message.
- Returns:
SMILES string or None if the compound has no SMILES identifier.
- ord_schema.message_helpers.get_product_yield(product: ProductCompound, as_measurement: bool = False) ProductMeasurement | float | None#
Returns the value of a product’s yield if it is defined.
If multiple measurements of type YIELD exist, only the first is returned.
A yield recorded as
float_value,amount, or text has no percentage to report: reading one as a percentage would assume a scale the source did not state. Passas_measurementto reach those.viewsapplies the same rule across every outcome and counts what it drops.- Parameters:
product – ProductCompound message.
as_measurement – Whether to return the full ProductMeasurement that corresponds to the yield measurement. Defaults to False.
- Returns:
The ProductMeasurement when
as_measurementis set. Otherwise the yield as a percentage, or None when the product records no yield or records one that is not a percentage.
- ord_schema.message_helpers.get_reaction_smiles(message: Reaction, generate_if_missing: bool = False, allow_incomplete: bool = True, allow_unspecified_roles: bool = True) str | None#
Fetches or generates a reaction SMILES.
A stored REACTION_CXSMILES identifier is returned whole, extension block and all. The block is chemistry the source recorded – enhanced stereochemistry, fragment grouping – and RDKit reads it, so there is nothing to gain by dropping it.
split_cxsmiles_extensionseparates the two parts for a caller that needs the bare SMILES, e.g. to look for atom maps.- Parameters:
message – reaction_pb2.Reaction message.
generate_if_missing – Whether to generate a reaction SMILES from the inputs and outputs if one is not defined explicitly.
allow_incomplete – Boolean whether to allow “incomplete” reaction SMILES that do not include all components (e.g. if a component does not have a structural identifier).
allow_unspecified_roles – If True, reactants and products with the UNSPECIFIED reaction role will be included when generating a reaction SMILES.
- Returns:
Text reaction SMILES, or None.
- Raises:
ValueError – If the reaction contains errors.
- ord_schema.message_helpers.has_transition_metal(mol: Mol) bool#
Determines if a molecule contains a transition metal.
- Parameters:
mol – The molecule in question. Should be of type rdkit.Chem.rdchem.Mol
- Returns:
Boolean for whether the molecule has a transition metal.
- ord_schema.message_helpers.id_filename(filename: str) str#
Converts a filename into a relative path for the repository.
- Parameters:
filename – Text basename including an ID.
- Returns:
Text filename relative to the root of the repository.
- ord_schema.message_helpers.identifier_parses_unsanitized(identifier_type: int, value: str) bool#
Tests whether an identifier parses at all once sanitization is skipped.
Separate from
canonical_smiles_for_identifier()so the extra parse happens only for the identifiers that failed, which are rare.- Parameters:
identifier_type – CompoundIdentifier type enum value.
value – The identifier value.
- Returns:
True if
valueparses withsanitize=False.
- ord_schema.message_helpers.is_transition_metal(atom: Atom) bool#
Determines if an atom is a transition metal.
- Parameters:
atom – The atom in question. Should be of type rdkit.Chem.rdchem.Atom
- Returns:
Boolean for whether the atom is a transition metal.
- ord_schema.message_helpers.load_message(filename: str | PathLike[str], message_type: type[MessageType]) MessageType#
Loads a protocol buffer message from a file.
- Parameters:
filename – Filename (
stror path-like) containing a serialized message.message_type – Message subclass.
- Returns:
Message object.
- Raises:
ValueError – if the message cannot be parsed, if
filenameis a Parquet dataset (usedatasets.load_dataset), or if the format is unsupported.
- ord_schema.message_helpers.message_to_row(message: Message, trace: tuple[str, ...] | None = None) dict[str, str | bytes | float | int | bool]#
Converts a proto into a flat dictionary mapping fields to values.
The keys indicate any nesting; for instance a proto that looks like this:
value: { subvalue: 5 }
will show up as
{'value.subvalue': 5}in the dict.- Parameters:
message – Proto to convert.
trace – Tuple of strings; the trace of nested field names.
- Returns:
Dict mapping string field names to scalar value types.
- ord_schema.message_helpers.messages_to_dataframe(messages: Iterable[Message], drop_constant_columns: bool = False) DataFrame#
Converts a list of protos to a pandas DataFrame.
- Parameters:
messages – List of protos.
drop_constant_columns – Whether to drop columns that have the same value for all rows.
- Returns:
DataFrame.
- ord_schema.message_helpers.mol_from_compound(compound: Compound | ProductCompound, return_identifier: bool = False) Mol | tuple[Mol, CompoundIdentifier]#
Creates an RDKit Mol from a Compound message.
Identifiers are tried in
structural_identifiers()order, so this andsmiles_from_compound()agree about which one describes the compound, and one malformed identifier does not mask a readable one recorded alongside it.- Parameters:
compound – reaction_pb2.Compound message.
return_identifier – If True, return the CompoundIdentifier used to create the Mol.
- Returns:
The RDKit Mol; with
return_identifier, a tuple of the Mol and the CompoundIdentifier used to create it.- Raises:
ValueError – If no structural identifier reads as a Mol. Unlike
smiles_from_compound(), which reports that as None, callers here want a Mol in hand and have nothing to do with its absence.
- ord_schema.message_helpers.molblock_from_compound(compound: Compound | ProductCompound) str#
Fetches or generates a MolBlock identifier for a compound.
A recorded MOLBLOCK is returned as deposited, keeping the atom coordinates that are the reason to record one, but only once it parses: an unreadable value is not a MolBlock, and returning it verbatim would hand the caller something no MolBlock reader accepts while a usable structure sat in another identifier.
That check skips sanitization, because the question is whether the value is a MolBlock rather than whether it describes a sanitizable molecule. ORD carries structures RDKit reads but will not sanitize – organometallics and charged-N rings, which validation warns about rather than rejecting – and regenerating those from another identifier would discard the coordinates for a value that was fine.
- Parameters:
compound – reaction_pb2.Compound or ProductCompound message.
- Returns:
MolBlock identifier, generated from the best structural identifier (see
structural_identifiers()) when none is recorded or the recorded one cannot be read.- Return type:
molblock
- Raises:
ValueError – if no structural identifier reads as a Mol.
- ord_schema.message_helpers.parse_doi(doi: str) str#
Parses a DOI from e.g. a URL.
- Parameters:
doi – DOI string.
- Returns:
The (possibly trimmed) DOI.
- Raises:
ValueError – if the DOI cannot be parsed.
- ord_schema.message_helpers.reaction_from_smiles(reaction_smiles: str) Reaction#
Builds a Reaction by splitting a reaction SMILES.
Components are written through
canonical_smiles(), so enhanced stereochemistry and coordinate bonds carried by the input reach the components that have them.- Parameters:
reaction_smiles – A reaction SMILES or CXSMILES.
- Returns:
A Reaction with one input holding the reactants and agents, and one outcome holding the products. Amounts are unmeasured; only structure is recovered.
- ord_schema.message_helpers.reaction_smiles_without_agents(reaction_smiles: str) str | None#
Returns a reaction SMILES with its agent block removed, or None if unreadable.
Splitting the string on
>would corrupt a CXSMILES extension, which indexes atoms positionally: every index in the block would point at the wrong atom once the agents are gone. Rebuilding through RDKit recomputes them.The surviving templates are added in canonical order because RDKit numbers the block against the order they were added and then writes them sorted; adding them in any other order leaves a stereo group marking whichever atom lands at that index.
Atom mapping survives, being part of the atoms themselves, and so does enhanced stereochemistry. Fragment grouping does not, and canonical atom ordering replaces whatever the source used.
- Parameters:
reaction_smiles – A reaction SMILES or CXSMILES, with or without agents.
- Returns:
Canonical
reactants>>productsCXSMILES. None if RDKit cannot read the input, reports errors in it, or it has no reactant or no product once agents are gone: a half reaction is not a restatement of the reaction the source recorded.
- ord_schema.message_helpers.safe_update(target: dict, update: Mapping) None#
Checks that update will not clobber any keys in target.
- ord_schema.message_helpers.save_message(message: Message, filename: str | PathLike[str]) None#
Writes a protocol buffer message to disk.
- Parameters:
message – Protocol buffer message.
filename – Output filename (
stror path-like).
- Raises:
ValueError – if filename does not have the expected suffix.
- ord_schema.message_helpers.set_compound_identifier(compound: Compound, identifier_type: <google.protobuf.internal.enum_type_wrapper.EnumTypeWrapper object at 0x7265551adb50>, value: str) CompoundIdentifier#
Sets the value of a compound identifier if it exists or creates one.
If multiple identifiers of that type exist, only the first is overwritten.
- Parameters:
compound – Compound message.
identifier_type – The CompoundIdentifier type to retrieve the value of.
value – The value to set.
- Returns:
The compound identifier that was modified or created.
- ord_schema.message_helpers.set_compound_molblock(compound: Compound, value: str) CompoundIdentifier#
Sets the value of the compound’s MOLBLOCK identifier if it exists or creates one.
- Parameters:
compound – Compound message.
value – The value to set.
- Returns:
The compound identifier that was modified or created.
- ord_schema.message_helpers.set_compound_name(compound: Compound, value: str) CompoundIdentifier#
Sets the value of the compound’s NAME identifier if it exists or creates one.
- Parameters:
compound – Compound message.
value – The value to set.
- Returns:
The compound identifier that was modified or created.
- ord_schema.message_helpers.set_compound_smiles(compound: Compound, value: str) CompoundIdentifier#
Sets the value of the compound’s SMILES identifier if it exists or creates one.
- Parameters:
compound – Compound message.
value – The value to set.
- Returns:
The compound identifier that was modified or created.
- ord_schema.message_helpers.set_dative_bonds(mol: Mol, from_atoms: tuple[str, ...] = ('N', 'P')) Mol#
Converts metal-ligand bonds to dative.
Replaces some single bonds between metals and atoms with atomic numbers in fromAtoms with dative bonds. For all atoms except carbon, the replacement is only done if the atom has “too many” bonds. To handle metal-carbene complexes, metal-carbon bonds are converted to dative if the sum of the explicit and implicit valence of the carbon atom does not equal its default valence, 4.
- Parameters:
mol – The molecule to be converted.
from_atoms – tuple of atomic symbols corresponding to atom types that should have atom-metal bonds converted to dative. Default is N and P
- Returns:
The modified molecule.
- ord_schema.message_helpers.set_solute_moles(solute: Compound, solvents: Sequence[Compound], concentration: str, overwrite: bool = False) list[Compound]#
Helps define components for stock solution inputs.
Handles a single solute and one or more solvent compounds.
- Parameters:
solute – Compound with identifiers, roles, etc.; this argument is modified in place to define an amount in moles.
solvents – list of Compounds each with defined volume.
concentration – string defining solute concentration.
overwrite – whether to overwrite an existing solute amount if defined. Defaults to False
- Raises:
ValueError – if any solvent does not have a defined volume.
ValueError – if the solute has an existing amount field and overwrite is set to False.
- Returns:
List of Compounds to assign to a repeated components field.
- ord_schema.message_helpers.smiles_from_compound(compound: Compound | ProductCompound) str | None#
Returns canonical SMILES for a compound, preferring its CXSMILES form.
Identifiers are tried in
structural_identifiers()order, so this andmol_from_compound()agree about which one describes the compound.A compound with nothing readable is an ordinary state in ORD – ligands and reagents are routinely recorded by name alone – so it reads as None rather than raising.
- Parameters:
compound – Compound or ProductCompound message.
- Returns:
Canonical SMILES, carrying an enhanced-stereochemistry block where the structure has one (see
canonical_smiles()), or None if the compound records no structure any loader can read.
- ord_schema.message_helpers.smiles_identifier_type(smiles: str) str#
Returns the CompoundIdentifier type a SMILES value belongs under.
Validation warns about an extension block recorded under SMILES.
- Parameters:
smiles – A SMILES or CXSMILES string.
- Returns:
"CXSMILES"if the value has an extension block, otherwise"SMILES".
- ord_schema.message_helpers.split_cxsmiles_extension(value: str) tuple[str, str | None]#
Splits a CXSMILES extension block off a value, if it has one.
The block follows whitespace and is introduced by
|. Whitespace alone does not identify one – records exist whose SMILES carry trailing non-breaking spaces – so anything else is returned intact rather than truncated at the space.- Parameters:
value – A SMILES or CXSMILES string.
- Returns:
(smiles, extension), where extension is None when there is no block.
- ord_schema.message_helpers.structural_identifiers(compound: Compound | ProductCompound) Iterator[CompoundIdentifier]#
Yields a compound’s structural identifiers, best first.
CXSMILES comes before SMILES because it is a superset and RDKit reads either, so preferring it keeps enhanced stereochemistry: a plain SMILES would assert one configuration where the source recorded a group. Everything else follows in message order.
Yielding rather than picking one lets a caller keep trying, so a malformed value does not hide a good identifier behind it. Nothing here parses the values, so a yielded identifier is a candidate rather than a promise.
- Parameters:
compound – Compound or ProductCompound message.
- Yields:
Each identifier with a non-empty value whose type a Mol can be built from.
- ord_schema.message_helpers.validate_reaction_smiles(reaction_smiles: str) None#
Validates reaction SMILES.
Prefer
_validate_reaction()where a parsed reaction is already at hand; this is for callers holding only the string, e.g. checking a recorded identifier.- Parameters:
reaction_smiles – Text reaction SMILES.
- Raises:
ValueError – If the reaction contains errors.
ord_schema.parquet#
Parquet-based serialization for Dataset messages.
Each Parquet file is a serialized Dataset:
Rows are
Reactionmessages with columnsreaction_id(string) andreaction(serialized wire-format bytes).The
Datasetscalar fields (dataset_id,name,description) are stored in the Parquet footer key-value metadata underord.*keys.Dataset.reaction_idsis not persisted — thereaction_idcolumn is the source of truth.
This layout supports random access (by row group) and streaming iteration without loading the full dataset into memory.
- class ord_schema.parquet.DatasetView(path: str | PathLike[str])#
Bases:
objectA read-only, streaming view of a Parquet-serialized Dataset.
Quacks like a
dataset_pb2.Datasetfor read-only access:name,description, anddataset_idcome from the Parquet footer, andreactionssupports iteration, length, emptiness, and indexing like a list while reading on demand: a fresh read per iteration, length from the footer, one row group per index.reactionsis a read-only property so accidental rebinding raises; the scalars are plain writable attributes, whichupdates.assign_dataset_idrelies on. The protobuf message methods (SerializeToString,CopyFrom) are absent and no write reaches the file;to_protomaterializes a real message.Two things differ from a real message and are worth knowing before substituting one for the other. Indexing returns a freshly deserialized Reaction rather than a live sub-message, so
reactions[0] is reactions[0]is False and mutating the result changes nothing; andreactionsdoes not compare equal to a list, so compare againstlist(view.reactions)orview.reactions[:].The backing file must not be replaced while a view is open: the row count is read once at construction, and
DatasetWriterpublishes by renaming onto its destination. Reads that can detect the mismatch raise and name the path; iteration reads whatever the path points at.reaction_idsis always empty by design: in the schema it names reactions stored outside the Dataset and is mutually exclusive withreactions, so serving it from thereaction_idcolumn would make every Parquet dataset fail validation. Useiter_reaction_idsfor the IDs of the reactions this dataset does contain, orget_reactionto look one up.- get_reaction(reaction_id: str) Reaction#
Reads a single Reaction by ID.
The first call reads the
reaction_idcolumn to build an ID-to-position index; later calls are a dict lookup plus one row-group read. Reading the whole column costs far less than deserializing the Reactions, so this beats a scan as soon as more than one lookup is made. Where a reaction ID appears more than once, the first occurrence wins.- Parameters:
reaction_id – ID of the Reaction to read.
- Returns:
A freshly deserialized Reaction.
- Raises:
KeyError – If
reaction_idis not present in the file.
- iter_reaction_ids() Iterator[str]#
Streams
reaction_idvalues without deserializing any Reaction.Reads only the
reaction_idcolumn, a row group at a time, so this stays cheap on very large files. Order matchesiter_reactions.- Yields:
Each reaction ID, in file order.
- iter_reactions(reaction_ids: Iterable[str] | None = None, *, row_group: int | None = None) Iterator[tuple[str, Reaction]]#
Streams
(reaction_id, Reaction)pairs from the backing file.Iterating
reactionsis the same read with the IDs dropped; use this when the ID is wanted alongside the message, or to restrict the read.- Parameters:
reaction_ids – Optional iterable of reaction IDs to restrict iteration to. IDs not present in the file are silently skipped. Filtering is applied row-wise; row groups are still read in full. Pass
None(the default) to iterate all reactions; an explicitly empty iterable raisesValueErrorsince it is almost always a bug.row_group – Optional row-group index. When set, only that row group is read; this is the unit of parallelism for callers that fan out across row groups (see
num_row_groups).
- Yields:
Each
(reaction_id, Reaction)pair, in file order.- Raises:
ValueError – If
reaction_idsis provided but empty.IndexError – If
row_groupis out of range.
- md5() str#
Returns a content hash, streamed a row group at a time.
The hash is fed in this order:
name=<value>\n,description=<value>\n,dataset_id=<value>\n(omitted when unset). These come from this view, so a scalar assigned throughupdates.assign_dataset_idchanges the hash before it reaches the file.For each Reaction in iteration order: the 8-byte big-endian length of the serialized Reaction blob, then the blob bytes. The reaction_id is not hashed separately since it is already inside the blob (field 1 of
Reaction); the length prefix disambiguates blob boundaries.
Different from
md5(Dataset.SerializeToString(deterministic=True))– re-ingesting an existing.pb.gzdataset as.parquetwill look like a content change once. That is correct: the on-disk bytes really did change, and the hash is only a cheap “did the content change since I last saw it?” indicator. The scheme is decoupled from the Parquet file layout (row group sizes, compression) so the same logical content rewritten with different writer settings still hashes the same.- Returns:
The MD5 hex digest.
- property num_row_groups: int#
Number of row groups, the unit of parallelism for
iter_reactions.
- property path: str#
Path to the backing Parquet file.
- property reactions: _ReactionStream#
Re-iterable stream of Reactions read from the backing file.
- to_proto() Dataset#
Materializes a real
Datasetmessage, deserializing every Reaction.Escape hatch for the operations a view cannot serve: serialization, JSON conversion, mutation, and helpers that require a protobuf message. Peak memory scales with the whole dataset, so prefer iterating
reactionswhen the message itself is not required.- Returns:
A Dataset whose Reactions are streamed from the backing file and whose scalars are taken from this view, so an assignment such as
updates.assign_dataset_idreaches the message.
- class ord_schema.parquet.DatasetWriter(path: str | PathLike[str], *, name: str, description: str, dataset_id: str | None = None, compression: str = 'zstd', row_group_size: int = 1000)#
Bases:
objectStreaming writer for a Parquet dataset.
Reactions are buffered and flushed to a new row group every
row_group_sizerows, so peak memory stays bounded regardless of the total number of reactions written.nameanddescriptionare required;dataset_idis optional because it is assigned during the submission process. Zero reactions is permitted: a streaming writer only learns the count atclose, where failing would be too late to help.save_datasethas the whole message and rejects an empty one up front.Example
- with DatasetWriter(path, name=”big dataset”, description=”…”) as writer:
- for reaction in source:
writer.write(reaction)
- close() None#
Flushes the final row group, closes the writer, and publishes the file.
On success the temp file is renamed onto the destination via
os.replace. On any exception during flush or close (includingKeyboardInterrupt) the temp is removed best-effort and the destination is left untouched; the original error propagates and any secondary close error is suppressed. Idempotent thereafter.
- write(reaction: Reaction) None#
Buffers a Reaction; flushes a row group when the buffer is full.
- Parameters:
reaction – Reaction to append. Serialized immediately, so later mutation of the message does not affect what is written.
- write_all(reactions: Iterable[Reaction]) None#
Writes an iterable of Reactions.
- Parameters:
reactions – Reactions to append, in order.
- ord_schema.parquet.load_dataset(path: str | PathLike[str]) Dataset#
Reads a full Dataset from a Parquet file.
Every Reaction is deserialized into memory. Prefer
DatasetView, which reads the same file without materializing it, whenever a real message is not required.- Parameters:
path – Path to the Parquet-serialized Dataset.
- Returns:
The Dataset message.
- ord_schema.parquet.save_dataset(dataset: Dataset, path: str | PathLike[str], *, compression: str = 'zstd', row_group_size: int = 1000) None#
Writes a Dataset as a Parquet file.
Dataset.reaction_idsis not persisted; thereaction_idcolumn derived fromDataset.reactionsis the source of truth.- Parameters:
dataset – Dataset message. Must have at least one Reaction.
path – Output filename.
compression – Parquet column compression codec.
row_group_size – Maximum number of rows per row group. Smaller groups give finer-grained random access; larger groups compress better.
- Raises:
ValueError – If
dataset.reactionsis empty or required scalar fields (name,description) are unset.
ord_schema.resolvers#
Name/string resolution to structured messages or identifiers.
- ord_schema.resolvers.canonicalize_smiles(smiles: str) str#
Canonicalizes a SMILES string, raising if it will not parse.
A thin wrapper over
message_helpers.canonical_smiles(), so a resolved structure is written the same way as one derived from a Compound. Callers holding a Mol should use that directly; this exists for the string-in, string-out case.- Parameters:
smiles – SMILES string, which may carry a CXSMILES extension block.
- Returns:
Canonical SMILES, carrying a block where the structure has enhanced stereochemistry or coordinate bonds.
- Raises:
ValueError – If the SMILES cannot be parsed by RDKit.
- ord_schema.resolvers.resolve_input(input_string: str) ReactionInput#
Resolves a text-based description of an input.
- Supported formats:
[AMOUNT] of [NAME]
[AMOUNT] of [CONCENTRATION] [SOLUTE] in [SOLVENT]
- Parameters:
input_string – String describing the input.
- Returns:
ReactionInput message.
- Raises:
ValueError – if the string cannot be parsed properly.
- ord_schema.resolvers.resolve_name(value_type: str, value: str) tuple[str, str]#
Resolves compound identifiers to SMILES via multiple APIs.
Resolvers are tried in order until one answers. Any of them failing falls through to the next, so only an exhausted chain is a failure.
- Parameters:
value_type – The kind of identifier being resolved, e.g. “name”.
value – The identifier to resolve.
- Returns:
A tuple of SMILES and the name of the resolver that produced it.
- Raises:
ValueError – If no resolver returns a structure.
- ord_schema.resolvers.resolve_names(message: Message) bool#
Attempts to resolve compound NAME identifiers to SMILES.
When a NAME identifier is resolved, a SMILES identifier is added to the list of identifiers for that compound. The first success ends work on that Compound; any remaining NAME identifiers on it are left unresolved.
A compound already carrying an identifier a Mol can be built from is skipped. Coordinates alone do not count, since nothing in this library reads them into a structure, so a compound recorded as XYZ plus a name is worth resolving.
- Parameters:
message – Protocol buffer tree containing Compound submessages (e.g. Reaction or ReactionInput).
- Returns:
Boolean whether message was modified.
ord_schema.templating#
Functions for creating Datasets by enumerating a template with a spreadsheet.
The templating code has specific expectations for how the reaction pbtxt and spreadsheet are defined, namely that placeholder values in the pbtxt begin and end with a “$” (dollar sign) and that these match a unique column header in the spreadsheet file.
- ord_schema.templating.generate_dataset(name: str, description: str, template_string: str, df: DataFrame, validate: bool = True) Dataset#
Generates a Dataset by enumerating a template reaction.
- Parameters:
name – Dataset name.
description – Dataset description.
template_string – The contents of a Reaction pbtxt where placeholder values to be replaced are defined between dollar signs. For example, a SMILES identifier value could be “$product_smiles$”. Placeholders may only use letters, numbers, and underscores.
df – Pandas Dataframe where each row corresponds to one reaction and column names match placeholders in the template_string.
validate – If True, validate each enumerated Reaction and raise on the first one with errors.
- Returns:
A Dataset message.
- Raises:
ValueError – If there is no match for a placeholder string in df.
ValueError – If validate is True and there are validation errors when validating an enumerated Reaction message.
- ord_schema.templating.load_spreadsheet(file_name_or_buffer: str | BinaryIO, suffix: str | None = None) DataFrame#
Reads a {csv, xls, xlsx} spreadsheet file.
- Parameters:
file_name_or_buffer – Filename, or an open binary buffer. A buffer requires
suffix, since there is no name to read it from.suffix – Filename suffix, used to determine the data encoding. Anything other than “.xls” or “.xlsx” is read as CSV.
- Returns:
DataFrame containing the reaction spreadsheet data.
ord_schema.units#
Helpers for translating strings with units.
- class ord_schema.units.UnitResolver(unit_synonyms: dict[type[Concentration | Current | FlowRate | Length | Mass | Moles | Pressure | Temperature | Time | Voltage | Volume | Wavelength], dict[int, list[str]]] | None = None, forbidden_units: dict[str, str] | None = None)#
Bases:
objectResolver class for translating value+unit strings into messages.
- convert(message: Concentration | Current | FlowRate | Length | Mass | Moles | Pressure | Temperature | Time | Voltage | Volume | Wavelength, new_units: str | int) Concentration | Current | FlowRate | Length | Mass | Moles | Pressure | Temperature | Time | Voltage | Volume | Wavelength#
Converts a united message into another of the same type with different units.
- Parameters:
message – a message with units, e.g., Mass, Length.
new_units – the desired units of the new message, expressed either as a string or an integer (ENUM value). Use of a string is recommended due to the ambiguity of using ENUM values; for example, Mass.GRAM == Time.MINUTE.
- Returns:
A new message with units, e.g., Mass, Length.
- resolve(string: str, allow_range: bool = False) Concentration | Current | FlowRate | Length | Mass | Moles | Pressure | Temperature | Time | Voltage | Volume | Wavelength#
Resolves a string into a message containing a value with units.
- Parameters:
string – The string to parse; must contain a numeric value and a string unit. For example: “1.25 h”.
allow_range – If True, ranges like “1-2 h” can be provided and the average value will be reported along with the standard deviation.
- Returns:
Message containing a numeric value with units listed in the schema.
- Raises:
ValueError – if string does not contain a value with units, or if the value is invalid.
- resolve_unit(string_unit: str) tuple[type[Concentration | Current | FlowRate | Length | Mass | Moles | Pressure | Temperature | Time | Voltage | Volume | Wavelength], int]#
Resolves a unit string into its message type and unit ENUM value.
- Parameters:
string_unit – The string unit to parse; for example: “gram”.
- Returns:
Tuple containing the message type and unit ENUM value.
- Raises:
KeyError – if string unit cannot be parsed.
- ord_schema.units.canonical_precision(message: Concentration | Current | FlowRate | Length | Mass | Moles | Pressure | Temperature | Time | Voltage | Volume | Wavelength, target: str) float | None#
Converts a united message’s precision to
targetunits, or returns None.Precision is recorded in the same units as the value, so it converts with the value and is null wherever the value is – a column named for
targethas nowhere to say its uncertainty is in different units, and an uncertainty published beside a null reads as a measurement nobody took but everybody bounded.- Parameters:
message – A united message, e.g. Temperature or Mass.
target – Unit to convert to, spelled as the resolver understands it, e.g. “K”.
- Returns:
The converted precision, or None when the message records no precision, no value to attach it to, no units, or units that cannot be converted to
target. Also None where the conversion declines to state one: an inverted interval reaching zero is unbounded, and no number describes it.
- ord_schema.units.canonical_value(message: Concentration | Current | FlowRate | Length | Mass | Moles | Pressure | Temperature | Time | Voltage | Volume | Wavelength, target: str) float | None#
Converts a united message to
targetunits, or returns None if it cannot.Null beats raising for a derived column: the column is named for
target, so an unconverted value has nowhere to say it is in different units, and a number there would be read astarget.- Parameters:
message – A united message, e.g. Temperature or Mass.
target – Unit to convert to, spelled as the resolver understands it, e.g. “K”.
- Returns:
The converted value, or None when the message records no value, records no units, or records units that cannot be converted to
target. A wavenumber of zero is in the last group: converting it to a wavelength divides by it.
- ord_schema.units.compute_solute_quantity(volume: Volume, concentration: Concentration) Amount#
Computes the quantity of a solute, given volume and concentration.
- ord_schema.units.format_message(message: Concentration | Current | FlowRate | Length | Mass | Moles | Pressure | Temperature | Time | Voltage | Volume | Wavelength) str | None#
Formats a united message into a string.
- Parameters:
message – a message with units, e.g., Mass, Length.
- Returns:
- A string describing the value, e.g., “5.0 (± 0.1) mL” using the
first unit synonym listed in _UNIT_SYNONYMS.
ord_schema.updates#
Automated updates for Reaction messages.
- ord_schema.updates.apply_cross_reference_substitutions(reaction: Reaction, id_substitutions: dict[str, str]) None#
Rewrites reaction_ids referenced in
reactionvia the substitution map.
- ord_schema.updates.apply_reaction_updates(reaction: Reaction, *, new_id: str | None) bool#
Applies per-reaction updates in place using a pre-computed reaction ID.
Splitting ID generation out of this function lets a streaming caller allocate IDs in a cheap pre-pass (e.g. from a Parquet
reaction_idcolumn) and inject them here without re-deriving them.- Parameters:
reaction – Reaction message to mutate.
new_id – Pre-computed reaction_id to assign, or
Noneto leave the existing ID untouched.
- Returns:
True if the reaction was modified.
- ord_schema.updates.assign_dataset_id(dataset: Dataset | DatasetView) str#
Assigns a canonical
dataset_idif missing or non-canonical.Mutates
dataset.dataset_idin place. Works for bothDatasetandDatasetView(which exposesdataset_idas a writable attribute).- Returns:
The (possibly newly-assigned) dataset_id.
- ord_schema.updates.assign_id_substitutions(old_ids: Iterable[str]) tuple[list[str | None], dict[str, str]]#
Pre-allocates canonical reaction IDs for a sequence of old IDs.
A reaction’s ID is replaced when the existing one is missing or does not match the canonical
ord-{32 hex}pattern. Cross-reference rewriting only applies to old IDs that were non-empty (i.e., user-supplied placeholders); reactions whose old ID was empty get a new ID but no substitution entry, since nothing else could have referenced them.NOTE(kearnes): This does not check for the case where a Dataset is edited and reaction_id values are changed inappropriately. This will need to be either (1) caught in review or (2) found by a complex check of the diff.
- Parameters:
old_ids – Reaction IDs in the order they appear in the dataset.
- Returns:
- List parallel to
old_ids; entry is the new reaction_id to assign, or
Noneif the old ID was already canonical.- id_substitutions: Map of
old_id -> new_idfor entries where the old ID was a non-empty placeholder. Used to rewrite cross-references.
- List parallel to
- Return type:
new_ids
- ord_schema.updates.update_dataset(dataset: Dataset) None#
Updates a Dataset message.
- Current updates:
Sets dataset_id if not already canonical.
Sets reaction_id on each Reaction if not already canonical, and appends a record_modified provenance event for any modified Reaction.
Rewrites reaction_id cross-references between Reactions in the dataset.
- Parameters:
dataset – dataset_pb2.Dataset message.
- Raises:
KeyError – if the dataset has not been validated and there exists a cross-referenced reaction_id in any Reaction that is not defined elsewhere in the Dataset.
- ord_schema.updates.update_parquet_dataset(input_path: str | PathLike[str], output_path: str | PathLike[str], *, dataset_id: str) None#
Applies
update_datasetto Parquetinput_path, writing tooutput_path.Two passes over
input_path:Pass 1 reads only the
reaction_idcolumn (no Reaction decode) to pre-allocate canonical reaction IDs and build the cross-reference map.Pass 2 streams full Reactions, applies per-reaction updates and cross-reference rewrites, and writes them via
DatasetWriter.
Peak memory is bounded by one row group plus the ID maps. The caller is responsible for choosing
output_pathbased on the resolveddataset_id(callassign_dataset_idon the input header first to learn it) and for any atomic-rename / validation dance — keeping the rename outside lets the caller validate the written file before publishing it.- Parameters:
input_path – Path to the input Parquet dataset.
output_path – Path to write the updated Parquet dataset to.
dataset_id – Resolved dataset_id to write into the output footer.
ord_schema.validations#
Helpers validating specific Message types.
- class ord_schema.validations.DatasetCrossRefState(defined_ids: set[str] = <factory>, referenced_ids: set[str] = <factory>, duplicate_count: int = 0, self_reference_count: int = 0)#
Bases:
objectAggregated cross-reference observations for a Dataset.
A worker validating a slice of reactions feeds each one into
observeand returns the resulting state. The master process merges the per-slice states withmergeand thenreportrecords a finding per duplicate occurrence, per self-reference, and one summary finding if any referenced reaction_ids are undefined. This keeps the streaming path behaviorally equivalent to the in-memory path.- defined_ids: set[str]#
- duplicate_count: int = 0#
- merge(other: DatasetCrossRefState) None#
Merges another state into this one, counting cross-slice duplicate IDs.
- observe(reaction: Reaction) None#
Records one reaction’s defined ID, referenced IDs, and self-references.
- referenced_ids: set[str]#
- report(context: ValidationContext) None#
Reports duplicate IDs, self-references, and undefined references.
- Parameters:
context – Where the findings are recorded.
- self_reference_count: int = 0#
- class ord_schema.validations.Severity(value)#
Bases:
IntEnumHow serious a validation finding is.
Ordered, so a caller can threshold on
>= Severity.ERRORrather than enumerating members.- ERROR = 2#
- INFO = 0#
- WARNING = 1#
- class ord_schema.validations.ValidationContext(options: ValidationOptions = <factory>, findings: list[tuple[str, ~ord_schema.validations.Severity]]=<factory>)#
Bases:
objectWhere a validator reports to, and the toggles it validates under.
Passed to every validator rather than held in module state, so what a call can see and affect is visible in its signature.
validate_messagereads back one message’s findings by slicing off whatever its validator appended.- error(message: str) None#
Records a finding that makes the message invalid.
- info(message: str) None#
Records a finding that needs no action, such as a suggested addition.
- options: ValidationOptions#
- warn(message: str) None#
Records a finding worth surfacing that does not fail validation.
- exception ord_schema.validations.ValidationError#
Bases:
ExceptionRaised when validation is asked to fail on invalid data.
A finding’s severity is
Severity; this is only the exception that carries an error out to the caller, viaraise_on_errororvalidate_datasets.
- class ord_schema.validations.ValidationOptions(validate_ids: bool = False, require_provenance: bool = True, allow_reaction_smiles_only: bool = True)#
Bases:
objectOptions for message validation.
- allow_reaction_smiles_only: bool = True#
- require_provenance: bool = True#
- validate_ids: bool = False#
- class ord_schema.validations.ValidationOutput(errors: list[str] = <factory>, warnings: list[str] = <factory>, info: list[str] = <factory>)#
Bases:
objectValidation output: errors, warnings, and info.
- errors: list[str]#
- extend(other: ValidationOutput) None#
Appends the findings from another output to this one.
- info: list[str]#
- warnings: list[str]#
- ord_schema.validations.get_referenced_reaction_ids(message: Reaction) set[str]#
Return the set of reaction IDs that are referenced in a Reaction.
- ord_schema.validations.has_atom_mapping(smiles: str) bool#
Returns whether a SMILES string contains atom-map numbers.
- ord_schema.validations.is_empty(message: Message) bool#
Returns whether the given message is empty.
- ord_schema.validations.is_url(value: str) bool#
Returns whether a string looks like an http(s) URL with a host.
- ord_schema.validations.is_valid_dataset_id(dataset_id: str) bool#
Returns whether a dataset ID matches the ord_dataset-<32 hex digits> format.
- ord_schema.validations.is_valid_orcid(orcid: str) bool#
Returns whether an ORCID is well-formed, including its checksum.
The final character is an ISO 7064 MOD 11-2 check digit over the preceding 15 digits; see https://support.orcid.org/hc/en-us/articles/360006897674.
- Parameters:
orcid – ORCID string, expected as 0000-0000-0000-0000.
- Returns:
True if
orcidis well-formed and the checksum is correct.
- ord_schema.validations.is_valid_reaction_id(reaction_id: str) bool#
Returns whether a reaction ID matches the ord-<32 hex digits> format.
- ord_schema.validations.parse_date_time(value: str) date | None#
Parses a DateTime value: a date, or a time with or without its UTC offset.
The one definition of the shapes a DateTime may take: validation accepts exactly the values this parses, and the artifact projection reads times through it.
Memoized: a DateTime string is parsed once by this module’s per-message walk and again by
_validate_reaction_provenance()to order the timestamps, and record timestamps repeat across the reactions of a dataset. Caching is safe here becausedateanddatetimeare immutable.- Parameters:
value – The DateTime value.
- Returns:
A
dateforYYYY-MM-DD; adatetimeforYYYY-MM-DDTHH:MM:SS[.ffffff], aware whenZor±HH:MMfollows and naive otherwise; or None for anything else, including a time whose UTC instant falls outside the yearsdatetimeholds.
- ord_schema.validations.reaction_has_internal_standard(message: Reaction) bool#
Whether any reaction component uses the internal standard role.
- ord_schema.validations.reaction_has_limiting_component(message: Reaction) bool#
Whether any reaction input compound is limiting.
- ord_schema.validations.reaction_needs_internal_standard(message: Reaction) bool#
Whether any analysis uses an internal standard.
- ord_schema.validations.validate_dataset_streaming(*, context: ValidationContext, name: str, description: str, dataset_id: str, reaction_ids: list[str], has_reactions: bool, state: DatasetCrossRefState) None#
Dataset-level validation for callers that have already streamed reactions.
Equivalent to
_validate_datasetfor a Dataset whose reactions have been iterated in slices (e.g., per Parquet row group) by upstream workers, with each worker contributing aDatasetCrossRefStatethat the caller has merged.has_reactionsshould reflect the source’s row count (e.g.,len(parquet.DatasetView(...).reactions)for parquet); inferring it fromstatewould misclassify reactions without reaction_ids or references as empty. Passreaction_ids=[]for the typical streaming case (parquet does not persist Dataset.reaction_ids).Validates under
context.options; there is no separate options argument, so this and_validate_datasetcannot disagree about which toggles are in force.
- ord_schema.validations.validate_datasets(datasets: Mapping[str, Dataset | DatasetView], write_errors: bool = False, options: ValidationOptions | None = None) None#
Runs validation for a set of datasets.
- Parameters:
datasets – Dict mapping text filenames to Dataset protos.
write_errors – If True, errors are written to disk.
options – ValidationOptions.
- Raises:
ValidationError – if any Dataset does not pass validation.
- ord_schema.validations.validate_message(message: Message, recurse: bool = True, raise_on_error: bool = True, options: ValidationOptions | None = None, trace: tuple[str, ...] | None = None, context: ValidationContext | None = None) ValidationOutput#
Template function for validating custom messages in the reaction_pb2.
Messages are not validated to check enum values, since these are enforced by the schema. Instead, we only check for validity of items that cannot be enforced in the schema (e.g., non-negativity of certain measurements, consistency of cross-referenced keys).
The message may be modified in place with any unambiguous changes needed to ensure validity.
- Parameters:
message – A message to validate.
recurse – If True, also validate submessages, meaning fields that are themselves messages.
raise_on_error – If True, raises a ValidationError exception when errors are encountered. If False, the user must manually check the return value to identify validation errors.
options – Toggles for the checks that are not always applied; see
ValidationOptions.trace – Tuple containing a string “stack trace” to track the position of the current message relative to the recursion root.
context – Where findings are reported, and the options they are validated under. Created for the root call and threaded through the recursion; callers do not normally pass one. Supplying both this and
optionsvalidates undercontext.optionsand ignoresoptions.
- Returns:
Errors, warnings, and info accumulated over the message and, when recursing, its submessages.
- Raises:
ValidationError – If any fields are invalid.