feat(dequant): packed dequant for Q4_0/Q4_1/Q4_K/Q3_K/Q2_K (#125)

Merged
RM
9d ago
+603-30

User description

Summary

  • dequantize_q4_0, dequantize_q4_1, dequantize_q2_k, dequantize_q3_k, dequantize_q4_k, wired into dequantize_packed and packed_row_size (Q4_0 32/18 B, Q4_1 32/20 B, Q2_K 256/84 B, Q3_K 256/110 B, Q4_K 256/144 B). Width-divisibility and row-length checks still fail closed.
  • Bit-exact golden fixtures (tests/golden_dequant.rs), generated by extending generate_golden.py; existing fixtures regenerate byte-identical.
  • Hand-derived unit tests for nibble/bit order, scale packing, and Q3_K hmask; dispatch/row-size/short/long/non-divisible rejection tests.
  • Opt-in real_gguf_token_rows_dequant_finite (ENGRAM_GGUF, #[ignore]).
  • README / CHANGELOG / crate docs updated.

Caveats

  • Golden expected values come from a Python port of the ggml scalar loops (same method as the existing Q8_0/Q5_K/Q6_K vectors), not from running llama.cpp.
  • The real-file OLMoE Q4_K_M smoke run (acceptance item 2) has not been run yet; no local GGUF was available.
  • Not in this PR: the cortex-tensor token-row allowlist follow-up (after next release).

Verification

cargo fmt --check, clippy -D warnings (default, all-features, and 1.97.1), cargo test --all-features --locked, and cargo doc with -D warnings all pass locally.

Closes #125

🤖 Generated with Claude Code

Summary by cubic

Adds packed dequantization for Q4_0, Q4_1, Q2_K, Q3_K, and Q4_K, wiring the new decoders into dequantize_packed and packed_row_size (Q4_0 32/18 B, Q4_1 32/20 B, Q2_K 256/84 B, Q3_K 256/110 B, Q4_K 256/144 B). Width-divisibility and row-length checks still fail closed, and the helpers are re-exported from the gguf module and crate root. README, CHANGELOG, and crate docs now list the new formats.

Testing

  • Bit-exact golden vectors generated by generate_golden.py (a Python port of the ggml scalar loops, not llama.cpp); existing fixtures regenerate byte-identical, and the new Q4/Q2/Q3 vectors are documented as not yet compared against a llama.cpp build.
  • Hand-derived unit tests cover nibble/bit order, scale packing, and Q3_K hmask.
  • Opt-in real_gguf_token_rows_dequant_finite smoke test decodes token rows from a local GGUF and fails if no rows decode.

The cortex-tensor token-row allowlist follow-up is intentionally out of scope.

Summary by CodeRabbit

  • New Features
    • Added packed dequantization support for Q4_0, Q4_1, Q2_K, Q3_K, and Q4_K, with public access to the corresponding helpers.
    • Packed decoding validates format-specific width alignment and expected row lengths.
  • Documentation
    • Updated the README and changelog to list the supported formats.
  • Tests
    • Added golden-vector coverage for the new formats and an opt-in check that decodes rows from local GGUF files and verifies the decoded values are finite. Golden vectors for these formats have not yet been compared with llama.cpp output.

CodeAnt-AI Description

Decode additional GGUF quantized tensor formats

What Changed

  • Decode Q4_0, Q4_1, Q2_K, Q3_K, and Q4_K packed weights into f32 values
  • Use the new formats through the shared packed decoder and row-size lookup
  • Reject rows with unsupported widths or incorrect packed lengths

Impact

✅ Decode five additional GGUF quantization formats
✅ Read packed tensor rows as f32
✅ Catch malformed packed rows

💡 Usage Guide

Checking Your Pull Request

Every time you make a pull request, our system automatically looks through it. We check for security issues, mistakes in how you're setting up your infrastructure, and common code problems. We do this to make sure your changes are solid and won't cause any trouble later.

Talking to CodeAnt AI

Got a question or need a hand with something in your pull request? You can easily get in touch with CodeAnt AI right here. Just type the following in a comment on your pull request, and replace "Your question here" with whatever you want to ask:

@codeant-ai ask: Your question here

This lets you have a chat with CodeAnt AI about your pull request, making it easier to understand and improve your code.

Example

@codeant-ai ask: Can you suggest a safer alternative to storing this secret?

Preserve Org Learnings with CodeAnt

You can record team preferences so CodeAnt AI applies them in future reviews. Reply directly to the specific CodeAnt AI suggestion (in the same thread) and replace "Your feedback here" with your input:

@codeant-ai: Your feedback here

This helps CodeAnt AI learn and adapt to your team's coding style and standards.

Example

@codeant-ai: Do not flag unused imports.

Retrigger review

Ask CodeAnt AI to review the PR again, by typing:

@codeant-ai: review

Check Your Repository Health

To analyze the health of your code repository, visit our dashboard at https://app.codeant.ai. This tool helps you identify potential issues and areas for improvement in your codebase, ensuring your repository maintains high standards of code health.