Samyak

Why Samyak?

Before documents enter a RAG or other AI pipeline, the corpus itself can contain empty files, tiny stubs, duplicate documents based on normalized content, oversized dumps, noisy text, extraction problems, unsupported files, and discovery or load failures.

Samyak inspects a local document directory, collects evidence, and surfaces findings. It identifies characteristics that may affect downstream processing. It does not guarantee retrieval quality, answer quality, or production readiness.

Corpus Intelligence

Analyze your document corpus before it becomes a RAG problem.

Many AI and RAG issues start in the source corpus, before embeddings, retrieval, or generation are involved. Corpus Intelligence inspects a local document directory and reports evidence-based findings — not a single quality score.

Typical corpus problems include:

• empty documents
• tiny or stub documents
• duplicate documents based on normalized content
• oversized documents
• noisy or repetitive content
• extraction problems
• unsupported files
• discovery and access failures
• document loading failures

Supported formats in this release are TXT, Markdown, PDF (text layer; no OCR), and HTML. PDF and HTML files also receive format-specific structural analysis. Other files are discovered and reported as unsupported; analysis continues.

How it works

1. 1Document directory
2. 2Samyak discovery
3. 3Document analysis
4. 4Findings
5. 5Text / JSON / HTML report

Analysis is local. Corpus Intelligence in this release makes no network or API calls and requires no API key or account.

Installation

Install the package from PyPI. Python 3.12 or later is required. GitHub is the source repository, not an alternate install path.

pip install samyak

samyak --version

Expected output for the current release:

samyak 0.1.0

CLI usage

Point Samyak at a document directory. Choose the output that matches how you want to read the findings.

samyak corpus ./documents

samyak corpus ./documents --format json

samyak corpus ./documents --format html --output report.html

Python API

The public API is intentionally small. Analyze a corpus and iterate over findings:

from samyak import analyze_corpus

report = analyze_corpus("./documents")

for finding in report.findings:
    print(finding)

Engineering characteristics

Local-first

Corpus analysis runs on your machine. Corpus Intelligence in this release makes no network or API calls and requires no API key or account.

Python API

A small public API (`analyze_corpus`) returns the same report the CLI produces.

CLI

Install with pip and run `samyak corpus` against a directory.

Deterministic findings

The same corpus and configuration produce the same findings. There is no LLM judge and no readiness score.

JSON output

Machine-readable reports for scripts, CI, and further processing.

HTML reports

Self-contained HTML you can open in a browser without a server.

Incremental corpus processing

Documents are processed incrementally rather than holding the entire corpus in memory.

Release history

1. 0.1.0 — Corpus Intelligence

   • Initial public release
   • Corpus discovery
   • TXT, Markdown, PDF (text layer; no OCR), and HTML analysis
   • PDF and HTML structural analysis
   • Normalized-content duplicate detection
   • Document-size analysis
   • Text-quality analysis
   • Discovery and load error reporting
   • Text, JSON, and HTML reports
   • Python API
   • Incremental corpus processing
   • Configurable analysis thresholds
   • Deterministic findings

Open source

Samyak is an open-source Python project developed by DataAIHub. It is a separate Python package, not a hosted DataAIHub service. Install the package from PyPI. Source and issue tracking are on GitHub. The repository slug (dataaihub-ai-engineering-toolkit) is a GitHub URL, not the product name.

• GitHub repository
• PyPI package
• Issues