PR #505 - usemoss/moss

Pull Request Checklist

I have read the CONTRIBUTING guide.
My code follows the style guidelines of this project.
I have performed a self-review of my own code.
I have added tests that prove my feature works.
Unit tests pass locally with my changes.

Description

Chunking isn't in any Moss SDK today, so every app that needs it writes its own. The two Python ones in this repo are a good example — they both already wrap their chunks in DocumentInfo, so the container is shared, but underneath they agree on almost nothing:

	`moss-pikachu`	`moss-llamaindex`
id	`{path}#chunk-0001`	`{filename}-p{page}-c{idx}`
metadata	`path`, `filename`, `chunk`, `extension`, `modified_at`	`source`, `page`
split	1800 chars / 300 overlap	400 words / 2-sentence overlap

Not a single metadata key in common. So anything downstream that wants to treat chunks uniformly - dedupe them, re-ingest them, point back to their source - can't, because there's no shared shape to rely on.

This package fixes that without freezing the part that should stay flexible. As came up in the contributors channel, how you split text is genuinely contested - code, prose, and transcripts all want different strategies - so that stays pluggable. What isn't contested is what a chunk should look like once it's out, so that's what this pins down.

The contract (chunk.py) - every chunk, whatever produced it, comes out with:

a stable ID, {source}#chunk-{index:04d}, zero-padded so IDs sort in cut order and stay identical across runs (re-chunking an unchanged doc reproduces them, so chunks replace instead of piling up as duplicates)
a metadata schema with a declared locator unit — char, line, or page — rather than a fixed set of fields, because a character offset means nothing for a PDF and a page number means nothing for a source file
native DocumentInfo output, rendered once by the contract (strategies never build it themselves), with every value stringified since Moss types metadata as Dict[str, str]

The strategies (strategies.py) — a small ChunkingStrategy Protocol (split(text) -> Iterable[Chunk]) and four implementations: CharSplitter and SentenceSplitter match pikachu and llamaindex exactly (the two duplications this replaces), and RecursiveSplitter and ParagraphSplitter round out the set named in the roadmap. Semantic splitting is deliberately left out for now — it needs embedding calls during ingestion, which is a different shape of dependency, and the README says so.

There's also an optional enrich step that carries over pikachu's trick of prepending the filename and path into the chunk text so BM25 can match on them — kept as a separate compose-able step, so pikachu can adopt it without having to switch splitters.

Addresses ROADMAP.md:112 (built-in text splitters).

Scope: this is the package half of what was discussed. The in-SDK zero-config default splitter is a natural follow-up against sdks/python/ — happy to do that next if the interface here looks right. Python only for now; vscode and moss-md-indexer are TypeScript and out of scope for this.

Notes for reviewers

ingest.py mirrors the connector template but drops its auto_id option on purpose — random UUIDs would defeat the stable IDs above.
publish-moss-chunking.yml is modelled on the zeroentropy connector (#490), which shipped its publish workflow in the same PR.

Type of Change

New feature (non-breaking change which adds functionality)

48 tests passing, ruff clean under CI's pinned 0.15.22.

Summary by CodeRabbit

New Features
- Added a pluggable text-chunking package with character, sentence, paragraph, and recursive splitting strategies.
- Added stable chunk identifiers, source-location tracking, metadata handling, and document conversion.
- Added enrichment helpers for searchable source and metadata context.
- Added asynchronous ingestion into a new Moss index.
Documentation
- Added installation guidance, usage examples, and documentation for chunking contracts and customization.
Chores
- Added automated build, validation, and package publishing workflow.