Package 'picklerick'

Title: Fast Single-Cell Format Conversion via SCX
Description: R bindings to the SCX streaming interop engine for single-cell genomics. Reads AnnData (.h5ad and Zarr), MuData (.h5mu), H5Seurat, 10x HDF5, 10x MatrixMarket and BPCells files into SingleCellExperiment or Seurat objects, or wraps the matrix in a DelayedArray that reads cells on demand. Converts between formats through a memory-bounded streaming pipeline and inspects file metadata without loading the matrix. No Python dependency.
Authors: Benjamin Carrillo [aut, cre], Atreya Choudhury [aut]
Maintainer: Benjamin Carrillo <[email protected]>
License: MIT + file LICENSE
Version: 0.6.2
Built: 2026-11-21 14:56:27 UTC
Source: https://github.com/btraven00/scx

Help Index


A light AnnData handle

Description

read_h5ad(as = "AnnData") returns one of these instead of a SingleCellExperiment. It holds what the Rust reader produced and builds R objects only when a component is actually touched.

Usage

## S3 method for class 'picklerick_anndata'
x$name

## S3 method for class 'picklerick_anndata'
dim(x)

## S3 method for class 'picklerick_anndata'
dimnames(x)

## S3 method for class 'picklerick_anndata'
print(x, ...)

Arguments

x

A light AnnData handle, from read_h5ad(as = "AnnData").

name

The field to get; see Details.

...

Unused.

Details

The reason this exists is measured, not aesthetic. Building a SingleCellExperiment costs a constant ~3.3 seconds on this machine regardless of dataset size — 0.05s vs 3.19s for a thousand cells, 3.72s vs 7.09s for a hundred thousand. That cost is S4 method-table and Bioconductor namespace warm-up, not work proportional to the data, so it cannot be optimised away inside the conversion; it can only be deferred until someone asks for an SingleCellExperiment.

Fields are reached with $, and each is built on first access and cached: ⁠$X⁠, ⁠$obs⁠, ⁠$var⁠, ⁠$obsm⁠, ⁠$layers⁠, ⁠$obs_names⁠, ⁠$var_names⁠, ⁠$n_obs⁠, ⁠$n_vars⁠. Convert with as_sce() or as_seurat().

dim() and dimnames() answer from what the reader already returned, genes by cells as in a SingleCellExperiment, without building anything. print() lists the components and which of them have been built so far.

Value

$ returns the field; dim() the number of genes and cells; dimnames() a list of gene and cell names; print() returns x invisibly.

Examples

## Not run: 
a <- read_h5ad("pbmc.h5ad", as = "AnnData")
dim(a)          # cheap: shape comes from the reader, nothing is built
a$obs$cell_type # builds and caches obs only
sce <- as_sce(a) # pays the S4 cost, once, when it is actually wanted

## End(Not run)

Convert a light AnnData handle to a Bioconductor or Seurat object

Description

The conversion is where the S4 cost is paid, so it is a separate call.

Usage

as_sce(x, ...)

## S3 method for class 'picklerick_anndata'
as_sce(x, parse_uns = FALSE, ...)

as_seurat(x, ...)

## S3 method for class 'picklerick_anndata'
as_seurat(x, parse_uns = FALSE, constructor = c("standard", "fast"), ...)

Arguments

x

An object from read_h5ad().

...

Unused.

parse_uns

Parse uns into metadata() (see read_h5ad()).

constructor

Seurat constructor, "standard" or the unofficial "fast" (see read_h5ad()). Ignored by as_sce().

Value

A SingleCellExperiment or a Seurat object.


Convert a single-cell file to another format

Description

Streams the input file through the SCX engine and writes the result to output. No R objects are constructed; memory usage is bounded by chunk_size regardless of dataset size.

Usage

convert(
  input,
  output,
  chunk_size = 5000L,
  dtype = "f32",
  assay = NULL,
  layer = "counts",
  where = NULL
)

Arguments

input

Path to the input file or directory.

output

Path to the output: .h5ad, .h5seurat or .zarr.

chunk_size

Number of cells per streaming chunk. Default 5000L.

dtype

Output numeric type: "f32", "f64", "i32", or "u32". Default "f32".

assay

Seurat assay to convert. Default NULL: "RNA" for H5Seurat input, the object's active assay for a Seurat .rds; for MuData .h5mu input, the modality (the only one, else "rna"). Also the assay name of H5Seurat output ("RNA" when NULL).

layer

Seurat layer to convert (H5Seurat inputs only). Default "counts".

where

Keep only some cells: a named list mapping obs columns to the values they may take, e.g. list(cell_type = c("B", "T"), donor = "d1"). A cell is kept when every column holds one of its values. Values are compared as text (TRUE/FALSE match Python's True/False); NA never matches. X, layers, raw counts, obs and obsm keep the matching cells, and obsp graphs those cells on both axes; genes and uns are unchanged. The matrix still streams: h5ad, Zarr, dgCMatrix H5Seurat and 10x HDF5 inputs read only the matching rows, others read everything and drop the rest. No match writes a file with 0 cells; an unknown column is an error. Default NULL: every cell.

Details

The input format is detected from the content: AnnData .h5ad, AnnData Zarr stores, H5Seurat, 10x HDF5, 10x MatrixMarket directories, BPCells directories. The output format is chosen by the extension of output: .h5ad, .h5seurat (SeuratDisk dgCMatrix layout) or .zarr (AnnData Zarr v3 store, a directory).

Value

output path, invisibly.

Examples

## Not run: 
convert("pbmc3k.h5seurat", "pbmc3k.h5ad")
convert("pbmc3k.h5ad", "pbmc3k.zarr")
convert("pbmc3k.h5seurat", "pbmc3k_f64.h5ad", dtype = "f64", chunk_size = 2000L)
"atlas.h5ad" |> convert("tcells.h5ad", where = list(cell_type = "T cell"))

## End(Not run)

Iterate over a dataset's genes in blocks

Description

Opens path once and reads its genes a block at a time, as read_genes() reads them. For an .rds, opening inflates the file once; each block is then read in parallel from the index that pass built, so a block costs a fraction of a read_genes() call. Memory holds one block, plus 4 bytes per nonzero of the whole matrix while it is read.

Usage

gene_blocks(path, block_size = 2000L, genes = NULL, chunk_size = 1000L)

Arguments

path

Path to the dataset.

block_size

Genes per block.

genes

Genes to read: 1-based integer positions or gene names, in any order. NULL reads all genes.

chunk_size

Genes per internal read.

Details

Forked workers (e.g. parallel::mclapply) can share one iterator and read different blocks with ⁠$read(k)⁠.

Value

A picklerick_gene_blocks list:

n

Number of blocks.

genes

List of each block's gene positions.

read(k)

Block k as a cells x genes dgCMatrix.

next_block()

The next block, or NULL after the last.

See Also

read_genes()

Examples

## Not run: 
it <- gene_blocks("spe.rds", block_size = 2000)
while (!is.null(m <- it$next_block())) fit(m)
fits <- parallel::mclapply(seq_len(it$n), function(k) fit(it$read(k)))

## End(Not run)

Inspect a single-cell file without loading data into memory

Description

Reads only the metadata (shape, column names, embedding keys, etc.) from a supported file. No count data is loaded. Memory usage is minimal regardless of dataset size.

Usage

inspect(path, chunk_size = 5000L)

Arguments

path

Path to a file or directory in any supported format (see read_h5ad()).

chunk_size

Cells per internal streaming chunk. Default 5000L. Only affects obs/var metadata reads; does not load the count matrix.

Value

A named list with:

format

Detected file format string.

n_obs

Number of observations (cells).

n_vars

Number of variables (genes/features).

obs_cols

Character vector of cell metadata column names.

var_cols

Character vector of feature metadata column names.

obsm_keys

Character vector of low-dimensional embedding keys.

layers

Named list with parallel vectors name, nnz, nnz_q1, nnz_med, nnz_q3, nnz_max — one entry per layer.

uns_keys

Character vector of unstructured metadata keys.

obsp

Named list with parallel vectors name, nnz, nnz_q1, nnz_med, nnz_q3, nnz_max — one entry per obsp matrix.

varm_keys

Character vector of feature embedding keys.

Examples

## Not run: 
info <- inspect("pbmc3k.h5seurat")
info$n_obs     # 2700
info$obs_cols  # c("orig.ident", "nCount_RNA", ...)
info$obsm_keys # c("X_pca", "X_umap")

## End(Not run)

DelayedArray seed backed by the scx engine

Description

PicklerickMatrix() returns a DelayedMatrix (features x cells) whose blocks are read on demand through scx. Column (cell) selections read only the selected cells from disk on h5ad and Zarr stores; other formats read the whole matrix per block and keep the selection.

Usage

PicklerickMatrix(path, layer = NULL, chunk_size = 5000L)

PicklerickMatrixSeed(path, layer = NULL, chunk_size = 5000L)

## S4 method for signature 'PicklerickMatrixSeed'
dim(x)

## S4 method for signature 'PicklerickMatrixSeed'
dimnames(x)

## S4 method for signature 'PicklerickMatrixSeed'
show(object)

## S4 method for signature 'PicklerickMatrixSeed'
is_sparse(x)

## S4 method for signature 'PicklerickMatrixSeed'
chunkdim(x)

## S4 method for signature 'PicklerickMatrixSeed'
extract_sparse_array(x, index)

## S4 method for signature 'PicklerickMatrixSeed'
extract_array(x, index)

Arguments

path

Path to an .h5ad file, .zarr directory, or any other format scx reads.

layer

Name of a layer to expose instead of X (h5ad and Zarr). NULL (default) exposes X.

chunk_size

Cells per internal read chunk. For formats other than h5ad and Zarr, also the block width in cells (the seed's chunkdim()).

x

A PicklerickMatrixSeed.

object

A PicklerickMatrixSeed.

index

A list of two subscripts (features, cells), each an integer vector or NULL for all, as S4Arrays::extract_array() passes it.

Details

Block-wise operations (rowSums(), colSums(), blockApply(), ...) take every feature for a range of cells per block. On h5ad and Zarr the number of cells per block follows DelayedArray's block size, so DelayedArray::setAutoBlockSize() sets the peak memory, as for HDF5Array seeds. Other formats read the whole matrix per block, so their blocks are chunk_size cells wide, to keep the number of passes down.

Value

PicklerickMatrix() returns a DelayedMatrix; PicklerickMatrixSeed() the underlying seed.

Examples

## Not run: 
m <- PicklerickMatrix("pbmc.zarr")
dim(m)
m[1:10, 1:5]                       # reads 5 cells
DelayedArray::colSums(m)           # block-wise
DelayedArray::setAutoBlockSize(50e6) # smaller blocks, lower peak memory

## End(Not run)

Read selected genes of a dataset's count matrix

Description

Reads only the genes in genes from path, for per-gene work (model fitting, tests) on a gene subset.

Usage

read_genes(path, genes = NULL, chunk_size = 1000L)

Arguments

path

Path to the dataset.

genes

Genes to read: 1-based integer positions or gene names, in any order. NULL reads all genes.

chunk_size

Genes per internal read.

Details

Gene-major .h5ad / Zarr stores (written with convert(..., gene_major = TRUE)) read just those genes. A Seurat, SingleCellExperiment or SpatialExperiment .rds is inflated once to index it, and the selected genes are then read in parallel; memory is about 12 bytes per kept nonzero plus 4 bytes per nonzero of the whole matrix. Cell-major .h5ad and other formats are refused; convert them with gene_major = TRUE first. To read many blocks of one file, use gene_blocks(), which opens it once.

Value

A dgCMatrix, cells x genes: one column per element of genes, in that order, so each gene's counts are contiguous. Use t() for genes x cells.

See Also

gene_blocks()


Read a single-cell file into an in-memory R object

Description

Reads any format picklerick supports (AnnData .h5ad files and Zarr stores, H5Seurat, 10x HDF5 and MatrixMarket, BPCells; detected from the content, not the extension) and returns a SingleCellExperiment (default), a Seurat object, or the raw named list of pieces (as = "list") for callers that want to assemble something else.

read_zarr() reads an AnnData Zarr store (a .zarr directory, Zarr v2 or v3, including the sharded v3 arrays anndata >= 0.13 writes by default). It is read_h5ad() under another name: the format is detected from the content, so read_h5ad() reads Zarr stores too. With lazy = TRUE the matrix is a PicklerickMatrix(). For Zarr stores uns is only available through parse_uns = TRUE (the on-demand uns() accessor reads HDF5).

Usage

read_h5ad(
  path,
  as = c("SingleCellExperiment", "Seurat", "AnnData", "list"),
  chunk_size = 5000L,
  lazy = FALSE,
  parse_uns = FALSE,
  constructor = c("standard", "fast")
)

read_zarr(path, ...)

Arguments

path

Path to the file.

as

Target class: "SingleCellExperiment" (default), "AnnData" for a light handle that defers building R objects until a component is touched (see AnnDataLight; building a SingleCellExperiment costs a constant ~3.3s of S4 warm-up whatever the dataset size), "Seurat", or "list".

chunk_size

Cells per internal streaming chunk. Default 5000L.

lazy

When TRUE, leave the X matrix on disk and wrap it in a DelayedMatrix. The Rust side only reads metadata (obs / var / obsm / varm / uns), so peak memory is bounded by metadata size rather than the count matrix. An .h5ad is wrapped with HDF5Array::H5SparseMatrix when that package is installed; everything else (AnnData Zarr, 10x, H5Seurat, ...) with PicklerickMatrix(), a seed that reads selected cells through scx. Only when as = "SingleCellExperiment" (Seurat does not consume DelayedArray natively). Layers are wrapped the same way, so the assays match the eager read; obsp is not attached in lazy mode.

parse_uns

When TRUE, read all of uns into metadata() (SingleCellExperiment) or ⁠@misc⁠ (Seurat). The default FALSE stores only the source path there, and uns() reads keys on demand.

constructor

How to build the Seurat object when as = "Seurat". "standard" (default) calls SeuratObject::CreateSeuratObject(). "fast" is an unofficial, experimental constructor: it assembles the Seurat v5 Assay5 directly from its slots, so the counts layer is the matrix just read rather than the two subset copies CreateSeuratObject() makes, and nCount/nFeature come from the sparse structure. On 10x 10k cells that is +0 MB instead of +535 MB over the matrix, and 0.14 s instead of 1.0 s, for an object identical in every checked respect. It relies on SeuratObject internals (the Assay5 slot layout, LogMap, the Seurat.object.assay.calcn option) and may drift with a SeuratObject release. It is only used on SeuratObject 5.x and on input that needs none of CreateSeuratObject()'s sanitising (no ⁠_⁠, empty or duplicate names; no orig.ident/nCount_RNA/nFeature_RNA columns in obs); otherwise it falls back to "standard" with a message.

...

Passed on to read_h5ad().

Details

In a SingleCellExperiment, X becomes the counts assay and each layer an assay of its own name. When a layer is itself called counts, X is named X instead, so the two never share a name.

Memory: this v1 materializes the X matrix in both Rust and R (~2× peak). For very large datasets prefer convert() (streaming, file-to-file).

Integer X is cast to double to match Seurat conventions and avoid integer-overflow in downstream normalization.

Layers whose shape doesn't match ⁠(n_obs, n_vars)⁠ are dropped with a warning. obsp is currently only attached to SingleCellExperiment (via colPair); it is dropped with a warning for Seurat output.

Value

A SingleCellExperiment, Seurat, or named list.

Examples

## Not run: 
sce <- read_h5ad("pbmc3k.h5ad")
obj <- read_h5ad("pbmc3k.h5ad", as = "Seurat")
raw <- read_h5ad("pbmc3k.h5ad", as = "list")  # advanced

# Lazy mode — X stays on disk, no full materialization.
sce <- read_h5ad("huge.h5ad", lazy = TRUE)
counts(sce)            # DelayedMatrix
counts(sce)[1:10, ]    # only these rows hit disk

## End(Not run)

Access uns metadata from a read_h5ad() result on demand

Description

When read_h5ad() is called with the default parse_uns = FALSE, the uns slot is not eagerly materialised — the source file path is stashed in metadata(sce)$.uns_path (for SCE) or obj@misc$.uns_path (for Seurat). Use uns() to read keys on demand via rhdf5, which is both faster and ~10–20× cheaper in memory than the eager JSON path (HDF5 stores uns as native typed arrays; the eager parse goes through JSON which inflates every integer into REALSXP).

Usage

uns(x, key = NULL, sub_key = NULL)

Arguments

x

A SingleCellExperiment, Seurat, or named list returned by read_h5ad(), OR a path string to an .h5ad file directly.

key

Top-level uns key. If NULL, returns the list of available keys (cheap; uses HDF5 group listing).

sub_key

Optional sub-key for nested uns (e.g. one condition under a per-condition dict). If supplied, only that leaf is read.

Value

If key is NULL: character vector of top-level uns keys. Otherwise the value at that path (a vector, list, or scalar depending on the on-disk layout).

Examples

## Not run: 
sce <- read_h5ad("norman.h5ad")           # parse_uns = FALSE
uns(sce)                                  # list top-level keys
x <- uns(sce, "top_non_zero_de_20")       # read one whole key
x <- uns(sce, "top_non_zero_de_20", "A549_AHR+FEV_1+1")  # one leaf

## End(Not run)

Write a SingleCellExperiment to an H5AD file

Description

Materializes the counts matrix as a dgCMatrix and hands it to the SCX Rust engine, which writes the AnnData layout directly. v1 writes only: X (counts assay), obs (colData), var (rowData), and uns (metadata coerced to JSON via jsonlite). obsm, obsp, varm, and additional layers are not yet supported and will be silently dropped.

Usage

write_h5ad(sce, path, assay = "counts", slot = "X", dtype = "f32")

Arguments

sce

A SingleCellExperiment (or any object with counts(), colData(), rowData(), metadata()).

path

Output .h5ad path.

assay

Name of the assay to use as X. Default "counts".

slot

Where the matrix goes in the file: "X" (default) or "layers/<name>", e.g. "layers/counts", which writes it there and leaves X absent.

dtype

Output X dtype: "f32", "f64", "i32", "u32". Default "f32" (matches scx convert default).

Value

path, invisibly.