| Title: | Fast Single-Cell Format Conversion via SCX |
|---|---|
| Description: | R bindings to the SCX streaming interop engine for single-cell genomics. Reads AnnData (.h5ad and Zarr), MuData (.h5mu), H5Seurat, 10x HDF5, 10x MatrixMarket and BPCells files into SingleCellExperiment or Seurat objects, or wraps the matrix in a DelayedArray that reads cells on demand. Converts between formats through a memory-bounded streaming pipeline and inspects file metadata without loading the matrix. No Python dependency. |
| Authors: | Benjamin Carrillo [aut, cre], Atreya Choudhury [aut] |
| Maintainer: | Benjamin Carrillo <[email protected]> |
| License: | MIT + file LICENSE |
| Version: | 0.6.2 |
| Built: | 2026-11-21 14:56:27 UTC |
| Source: | https://github.com/btraven00/scx |
read_h5ad(as = "AnnData") returns one of these instead of a
SingleCellExperiment. It holds what the Rust reader produced and builds R
objects only when a component is actually touched.
## S3 method for class 'picklerick_anndata' x$name ## S3 method for class 'picklerick_anndata' dim(x) ## S3 method for class 'picklerick_anndata' dimnames(x) ## S3 method for class 'picklerick_anndata' print(x, ...)## S3 method for class 'picklerick_anndata' x$name ## S3 method for class 'picklerick_anndata' dim(x) ## S3 method for class 'picklerick_anndata' dimnames(x) ## S3 method for class 'picklerick_anndata' print(x, ...)
x |
A light AnnData handle, from |
name |
The field to get; see Details. |
... |
Unused. |
The reason this exists is measured, not aesthetic. Building a
SingleCellExperiment costs a constant ~3.3 seconds on this machine
regardless of dataset size — 0.05s vs 3.19s for a thousand cells, 3.72s vs
7.09s for a hundred thousand. That cost is S4 method-table and Bioconductor
namespace warm-up, not work proportional to the data, so it cannot be
optimised away inside the conversion; it can only be deferred until someone
asks for an SingleCellExperiment.
Fields are reached with $, and each is built on first access and cached:
$X, $obs, $var, $obsm, $layers, $obs_names, $var_names,
$n_obs, $n_vars. Convert with as_sce() or as_seurat().
dim() and dimnames() answer from what the reader already returned,
genes by cells as in a SingleCellExperiment, without building anything.
print() lists the components and which of them have been built so far.
$ returns the field; dim() the number of genes and cells;
dimnames() a list of gene and cell names; print() returns x
invisibly.
## Not run: a <- read_h5ad("pbmc.h5ad", as = "AnnData") dim(a) # cheap: shape comes from the reader, nothing is built a$obs$cell_type # builds and caches obs only sce <- as_sce(a) # pays the S4 cost, once, when it is actually wanted ## End(Not run)## Not run: a <- read_h5ad("pbmc.h5ad", as = "AnnData") dim(a) # cheap: shape comes from the reader, nothing is built a$obs$cell_type # builds and caches obs only sce <- as_sce(a) # pays the S4 cost, once, when it is actually wanted ## End(Not run)
The conversion is where the S4 cost is paid, so it is a separate call.
as_sce(x, ...) ## S3 method for class 'picklerick_anndata' as_sce(x, parse_uns = FALSE, ...) as_seurat(x, ...) ## S3 method for class 'picklerick_anndata' as_seurat(x, parse_uns = FALSE, constructor = c("standard", "fast"), ...)as_sce(x, ...) ## S3 method for class 'picklerick_anndata' as_sce(x, parse_uns = FALSE, ...) as_seurat(x, ...) ## S3 method for class 'picklerick_anndata' as_seurat(x, parse_uns = FALSE, constructor = c("standard", "fast"), ...)
x |
An object from |
... |
Unused. |
parse_uns |
Parse |
constructor |
Seurat constructor, |
A SingleCellExperiment or a Seurat object.
Streams the input file through the SCX engine and writes the result to
output. No R objects are constructed; memory usage is bounded by
chunk_size regardless of dataset size.
convert( input, output, chunk_size = 5000L, dtype = "f32", assay = NULL, layer = "counts", where = NULL )convert( input, output, chunk_size = 5000L, dtype = "f32", assay = NULL, layer = "counts", where = NULL )
input |
Path to the input file or directory. |
output |
Path to the output: |
chunk_size |
Number of cells per streaming chunk. Default |
dtype |
Output numeric type: |
assay |
Seurat assay to convert. Default |
layer |
Seurat layer to convert (H5Seurat inputs only). Default |
where |
Keep only some cells: a named list mapping |
The input format is detected from the content: AnnData .h5ad, AnnData
Zarr stores, H5Seurat, 10x HDF5, 10x MatrixMarket directories, BPCells
directories. The output format is chosen by the extension of output:
.h5ad, .h5seurat (SeuratDisk dgCMatrix layout) or .zarr (AnnData
Zarr v3 store, a directory).
output path, invisibly.
## Not run: convert("pbmc3k.h5seurat", "pbmc3k.h5ad") convert("pbmc3k.h5ad", "pbmc3k.zarr") convert("pbmc3k.h5seurat", "pbmc3k_f64.h5ad", dtype = "f64", chunk_size = 2000L) "atlas.h5ad" |> convert("tcells.h5ad", where = list(cell_type = "T cell")) ## End(Not run)## Not run: convert("pbmc3k.h5seurat", "pbmc3k.h5ad") convert("pbmc3k.h5ad", "pbmc3k.zarr") convert("pbmc3k.h5seurat", "pbmc3k_f64.h5ad", dtype = "f64", chunk_size = 2000L) "atlas.h5ad" |> convert("tcells.h5ad", where = list(cell_type = "T cell")) ## End(Not run)
Opens path once and reads its genes a block at a time, as
read_genes() reads them. For an .rds, opening inflates the file once;
each block is then read in parallel from the index that pass built, so a
block costs a fraction of a read_genes() call. Memory holds one block,
plus 4 bytes per nonzero of the whole matrix while it is read.
gene_blocks(path, block_size = 2000L, genes = NULL, chunk_size = 1000L)gene_blocks(path, block_size = 2000L, genes = NULL, chunk_size = 1000L)
path |
Path to the dataset. |
block_size |
Genes per block. |
genes |
Genes to read: 1-based integer positions or gene names, in
any order. |
chunk_size |
Genes per internal read. |
Forked workers (e.g. parallel::mclapply) can share one iterator and
read different blocks with $read(k).
A picklerick_gene_blocks list:
nNumber of blocks.
genesList of each block's gene positions.
read(k)Block k as a cells x genes dgCMatrix.
next_block()The next block, or NULL after the last.
## Not run: it <- gene_blocks("spe.rds", block_size = 2000) while (!is.null(m <- it$next_block())) fit(m) fits <- parallel::mclapply(seq_len(it$n), function(k) fit(it$read(k))) ## End(Not run)## Not run: it <- gene_blocks("spe.rds", block_size = 2000) while (!is.null(m <- it$next_block())) fit(m) fits <- parallel::mclapply(seq_len(it$n), function(k) fit(it$read(k))) ## End(Not run)
Reads only the metadata (shape, column names, embedding keys, etc.) from a supported file. No count data is loaded. Memory usage is minimal regardless of dataset size.
inspect(path, chunk_size = 5000L)inspect(path, chunk_size = 5000L)
path |
Path to a file or directory in any supported format
(see |
chunk_size |
Cells per internal streaming chunk. Default |
A named list with:
Detected file format string.
Number of observations (cells).
Number of variables (genes/features).
Character vector of cell metadata column names.
Character vector of feature metadata column names.
Character vector of low-dimensional embedding keys.
Named list with parallel vectors name, nnz, nnz_q1,
nnz_med, nnz_q3, nnz_max — one entry per layer.
Character vector of unstructured metadata keys.
Named list with parallel vectors name, nnz, nnz_q1,
nnz_med, nnz_q3, nnz_max — one entry per obsp matrix.
Character vector of feature embedding keys.
## Not run: info <- inspect("pbmc3k.h5seurat") info$n_obs # 2700 info$obs_cols # c("orig.ident", "nCount_RNA", ...) info$obsm_keys # c("X_pca", "X_umap") ## End(Not run)## Not run: info <- inspect("pbmc3k.h5seurat") info$n_obs # 2700 info$obs_cols # c("orig.ident", "nCount_RNA", ...) info$obsm_keys # c("X_pca", "X_umap") ## End(Not run)
PicklerickMatrix() returns a DelayedMatrix (features x cells) whose
blocks are read on demand through scx. Column (cell) selections read only
the selected cells from disk on h5ad and Zarr stores; other formats read
the whole matrix per block and keep the selection.
PicklerickMatrix(path, layer = NULL, chunk_size = 5000L) PicklerickMatrixSeed(path, layer = NULL, chunk_size = 5000L) ## S4 method for signature 'PicklerickMatrixSeed' dim(x) ## S4 method for signature 'PicklerickMatrixSeed' dimnames(x) ## S4 method for signature 'PicklerickMatrixSeed' show(object) ## S4 method for signature 'PicklerickMatrixSeed' is_sparse(x) ## S4 method for signature 'PicklerickMatrixSeed' chunkdim(x) ## S4 method for signature 'PicklerickMatrixSeed' extract_sparse_array(x, index) ## S4 method for signature 'PicklerickMatrixSeed' extract_array(x, index)PicklerickMatrix(path, layer = NULL, chunk_size = 5000L) PicklerickMatrixSeed(path, layer = NULL, chunk_size = 5000L) ## S4 method for signature 'PicklerickMatrixSeed' dim(x) ## S4 method for signature 'PicklerickMatrixSeed' dimnames(x) ## S4 method for signature 'PicklerickMatrixSeed' show(object) ## S4 method for signature 'PicklerickMatrixSeed' is_sparse(x) ## S4 method for signature 'PicklerickMatrixSeed' chunkdim(x) ## S4 method for signature 'PicklerickMatrixSeed' extract_sparse_array(x, index) ## S4 method for signature 'PicklerickMatrixSeed' extract_array(x, index)
path |
Path to an |
layer |
Name of a layer to expose instead of |
chunk_size |
Cells per internal read chunk. For formats other than
h5ad and Zarr, also the block width in cells (the seed's |
x |
A |
object |
A |
index |
A list of two subscripts (features, cells), each an integer
vector or |
Block-wise operations (rowSums(), colSums(), blockApply(), ...) take
every feature for a range of cells per block. On h5ad and Zarr the number
of cells per block follows DelayedArray's block size, so
DelayedArray::setAutoBlockSize() sets the peak memory, as for HDF5Array
seeds. Other formats read the whole matrix per block, so their blocks are
chunk_size cells wide, to keep the number of passes down.
PicklerickMatrix() returns a DelayedMatrix;
PicklerickMatrixSeed() the underlying seed.
## Not run: m <- PicklerickMatrix("pbmc.zarr") dim(m) m[1:10, 1:5] # reads 5 cells DelayedArray::colSums(m) # block-wise DelayedArray::setAutoBlockSize(50e6) # smaller blocks, lower peak memory ## End(Not run)## Not run: m <- PicklerickMatrix("pbmc.zarr") dim(m) m[1:10, 1:5] # reads 5 cells DelayedArray::colSums(m) # block-wise DelayedArray::setAutoBlockSize(50e6) # smaller blocks, lower peak memory ## End(Not run)
Reads only the genes in genes from path, for per-gene work (model
fitting, tests) on a gene subset.
read_genes(path, genes = NULL, chunk_size = 1000L)read_genes(path, genes = NULL, chunk_size = 1000L)
path |
Path to the dataset. |
genes |
Genes to read: 1-based integer positions or gene names, in
any order. |
chunk_size |
Genes per internal read. |
Gene-major .h5ad / Zarr stores (written with
convert(..., gene_major = TRUE)) read just those genes. A Seurat,
SingleCellExperiment or SpatialExperiment .rds is inflated once to
index it, and the selected genes are then read in parallel; memory is
about 12 bytes per kept nonzero plus 4 bytes per nonzero of the whole
matrix. Cell-major .h5ad and other formats are refused; convert them
with gene_major = TRUE first. To read many blocks of one file, use
gene_blocks(), which opens it once.
A dgCMatrix, cells x genes: one column per element of genes,
in that order, so each gene's counts are contiguous. Use t() for genes
x cells.
Reads any format picklerick supports (AnnData .h5ad files and Zarr
stores, H5Seurat, 10x HDF5 and MatrixMarket, BPCells; detected from the
content, not the extension) and returns a SingleCellExperiment
(default), a Seurat object, or the raw named list of pieces
(as = "list") for callers that want to assemble something else.
read_zarr() reads an AnnData Zarr store (a .zarr directory,
Zarr v2 or v3, including the sharded v3 arrays anndata >= 0.13 writes by
default). It is read_h5ad() under another name: the format is detected
from the content, so read_h5ad() reads Zarr stores too. With
lazy = TRUE the matrix is a PicklerickMatrix(). For Zarr stores
uns is only available through parse_uns = TRUE (the on-demand
uns() accessor reads HDF5).
read_h5ad( path, as = c("SingleCellExperiment", "Seurat", "AnnData", "list"), chunk_size = 5000L, lazy = FALSE, parse_uns = FALSE, constructor = c("standard", "fast") ) read_zarr(path, ...)read_h5ad( path, as = c("SingleCellExperiment", "Seurat", "AnnData", "list"), chunk_size = 5000L, lazy = FALSE, parse_uns = FALSE, constructor = c("standard", "fast") ) read_zarr(path, ...)
path |
Path to the file. |
as |
Target class: |
chunk_size |
Cells per internal streaming chunk. Default |
lazy |
When |
parse_uns |
When |
constructor |
How to build the Seurat object when |
... |
Passed on to |
In a SingleCellExperiment, X becomes the counts assay and each layer
an assay of its own name. When a layer is itself called counts, X is
named X instead, so the two never share a name.
Memory: this v1 materializes the X matrix in both Rust and R (~2× peak).
For very large datasets prefer convert() (streaming, file-to-file).
Integer X is cast to double to match Seurat conventions and avoid
integer-overflow in downstream normalization.
Layers whose shape doesn't match (n_obs, n_vars) are dropped with a
warning. obsp is currently only attached to SingleCellExperiment
(via colPair); it is dropped with a warning for Seurat output.
A SingleCellExperiment, Seurat, or named list.
## Not run: sce <- read_h5ad("pbmc3k.h5ad") obj <- read_h5ad("pbmc3k.h5ad", as = "Seurat") raw <- read_h5ad("pbmc3k.h5ad", as = "list") # advanced # Lazy mode — X stays on disk, no full materialization. sce <- read_h5ad("huge.h5ad", lazy = TRUE) counts(sce) # DelayedMatrix counts(sce)[1:10, ] # only these rows hit disk ## End(Not run)## Not run: sce <- read_h5ad("pbmc3k.h5ad") obj <- read_h5ad("pbmc3k.h5ad", as = "Seurat") raw <- read_h5ad("pbmc3k.h5ad", as = "list") # advanced # Lazy mode — X stays on disk, no full materialization. sce <- read_h5ad("huge.h5ad", lazy = TRUE) counts(sce) # DelayedMatrix counts(sce)[1:10, ] # only these rows hit disk ## End(Not run)
When read_h5ad() is called with the default parse_uns = FALSE, the
uns slot is not eagerly materialised — the source file path is
stashed in metadata(sce)$.uns_path (for SCE) or obj@misc$.uns_path
(for Seurat). Use uns() to read keys on demand via rhdf5, which is
both faster and ~10–20× cheaper in memory than the eager JSON path
(HDF5 stores uns as native typed arrays; the eager parse goes through
JSON which inflates every integer into REALSXP).
uns(x, key = NULL, sub_key = NULL)uns(x, key = NULL, sub_key = NULL)
x |
A |
key |
Top-level uns key. If |
sub_key |
Optional sub-key for nested uns (e.g. one condition under a per-condition dict). If supplied, only that leaf is read. |
If key is NULL: character vector of top-level uns keys.
Otherwise the value at that path (a vector, list, or scalar
depending on the on-disk layout).
## Not run: sce <- read_h5ad("norman.h5ad") # parse_uns = FALSE uns(sce) # list top-level keys x <- uns(sce, "top_non_zero_de_20") # read one whole key x <- uns(sce, "top_non_zero_de_20", "A549_AHR+FEV_1+1") # one leaf ## End(Not run)## Not run: sce <- read_h5ad("norman.h5ad") # parse_uns = FALSE uns(sce) # list top-level keys x <- uns(sce, "top_non_zero_de_20") # read one whole key x <- uns(sce, "top_non_zero_de_20", "A549_AHR+FEV_1+1") # one leaf ## End(Not run)
Materializes the counts matrix as a dgCMatrix and hands it to the SCX
Rust engine, which writes the AnnData layout directly. v1 writes only:
X (counts assay), obs (colData), var (rowData), and uns (metadata
coerced to JSON via jsonlite). obsm, obsp, varm, and additional
layers are not yet supported and will be silently dropped.
write_h5ad(sce, path, assay = "counts", slot = "X", dtype = "f32")write_h5ad(sce, path, assay = "counts", slot = "X", dtype = "f32")
sce |
A |
path |
Output |
assay |
Name of the assay to use as |
slot |
Where the matrix goes in the file: |
dtype |
Output X dtype: |
path, invisibly.