Skip to content

Commit b3a0016

Browse files
srousseysroussey
authored
refactor(ai): simplify and consolidate RAG tasks (#427)
* chore: update package name and version, enhance build types and logging utilities - Changed package name from "workglow" to "@workglow-dev/libs" and updated version to 0.2.15 in package.json. - Modified build-types script in packages/util/package.json to include a check for the existence of a specific TypeScript declaration file. - Simplified TypeScript configuration by consolidating included files in packages/util/tsconfig.json. - Refactored logging and telemetry utilities to utilize a new runtime environment reader for improved environment variable handling.
1 parent 6801a77 commit b3a0016

28 files changed

Lines changed: 575 additions & 2117 deletions

‎.claude/CLAUDE.md‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -162,7 +162,7 @@ Model system: `ModelRepository`, `ModelRegistry`, `AiProviderRegistry`.
162162

163163
Task categories: text generation/embedding/summary/translation/rewriting/classification, image classification/embedding/segmentation, RAG (chunking, vector search, retrieval, reranking), vision/pose detection.
164164

165-
RAG tasks: `ChunkToVectorTask` (input: `vector` + `chunks` → output: `vectors`), `ChunkVectorUpsertTask` (input: `knowledgeBase` + `vectors`), `ChunkRetrievalTask` (input: `knowledgeBase` + `query` + `model`), `ChunkVectorSearchTask`, `ChunkVectorHybridSearchTask`, `HierarchyJoinTask`.
165+
RAG tasks: `ChunkVectorUpsertTask` (input: `knowledgeBase` + `chunks` + `vector`, optional `doc_title`), `ChunkRetrievalTask` (input: `knowledgeBase` + `query` + `model`, with `method: "similarity" | "hybrid"`), `HierarchyJoinTask`, `RerankerTask`, `QueryExpanderTask`, `TextChunkerTask`, `HierarchicalChunkerTask`.
166166

167167
### `@workglow/ai-provider` — provider implementations
168168

‎docs/developers/03_extending.md‎

Lines changed: 21 additions & 26 deletions
Original file line numberDiff line numberDiff line change
@@ -278,26 +278,23 @@ The `workglow` package provides a comprehensive set of tasks for building RAG (R
278278

279279
### Vector and Embedding Tasks
280280

281-
| Task | Description |
282-
| ----------------------- | ---------------------------------------------- |
283-
| `TextEmbeddingTask` | Generates embeddings using configurable models |
284-
| `ChunkToVectorTask` | Transforms chunks to vector store format |
285-
| `ChunkVectorUpsertTask` | Stores vectors in a repository |
286-
| `ChunkVectorSearchTask` | Searches vectors by similarity |
287-
| `VectorQuantizeTask` | Quantizes vectors for storage efficiency |
281+
| Task | Description |
282+
| ----------------------- | --------------------------------------------------------------------------- |
283+
| `TextEmbeddingTask` | Generates embeddings using configurable models |
284+
| `ChunkVectorUpsertTask` | Stores chunks + embeddings in a KnowledgeBase (1:1 aligned) |
285+
| `VectorQuantizeTask` | Quantizes vectors for storage efficiency |
288286

289287
### Retrieval and Generation Tasks
290288

291-
| Task | Description |
292-
| ----------------------------- | --------------------------------------------- |
293-
| `QueryExpanderTask` | Expands queries for better retrieval coverage |
294-
| `ChunkVectorHybridSearchTask` | Combines vector and full-text search |
295-
| `RerankerTask` | Reranks search results for relevance |
296-
| `HierarchyJoinTask` | Enriches results with parent context |
297-
| `ContextBuilderTask` | Builds context for LLM prompts |
298-
| `ChunkRetrievalTask` | Orchestrates end-to-end retrieval |
299-
| `TextQuestionAnswerTask` | Generates answers from context |
300-
| `TextGenerationTask` | General text generation |
289+
| Task | Description |
290+
| ------------------------ | ------------------------------------------------------------------------ |
291+
| `ChunkRetrievalTask` | End-to-end retrieval: embeds query, runs similarity or hybrid search |
292+
| `QueryExpanderTask` | Rule-based query expansion (multi-query / synonyms) |
293+
| `RerankerTask` | Reranks results (simple heuristic or reciprocal-rank-fusion) |
294+
| `HierarchyJoinTask` | Enriches retrieval metadata with parent context |
295+
| `ContextBuilderTask` | Builds formatted context for LLM prompts |
296+
| `TextQuestionAnswerTask` | Generates answers from context |
297+
| `TextGenerationTask` | General text generation |
301298

302299
### Chainable RAG Pipeline Example
303300

@@ -332,7 +329,6 @@ await new Workflow()
332329
.textEmbedding({
333330
model: "Xenova/all-MiniLM-L6-v2",
334331
})
335-
.chunkToVector()
336332
.chunkVectorUpsert({
337333
knowledgeBase: "my-kb",
338334
})
@@ -393,13 +389,12 @@ interface BaseNode {
393389

394390
Each task passes through what the next task needs:
395391

396-
| Task | Passes Through | Adds |
397-
| --------------------- | ------------------------ | ------------------------------------- |
398-
| `structuralParser` | - | `doc_id`, `documentTree`, `nodeCount` |
399-
| `documentEnricher` | `doc_id`, `documentTree` | `summaryCount`, `entityCount` |
400-
| `hierarchicalChunker` | `doc_id` | `chunks`, `text[]`, `count` |
401-
| `textEmbedding` | (implicit) | `vector[]` |
402-
| `chunkToVector` | - | `ids[]`, `vectors[]`, `metadata[]` |
403-
| `chunkVectorUpsert` | - | `count`, `ids` |
392+
| Task | Passes Through | Adds |
393+
| --------------------- | ------------------------ | ----------------------------------------- |
394+
| `structuralParser` | - | `doc_id`, `documentTree`, `nodeCount` |
395+
| `documentEnricher` | `doc_id`, `documentTree` | `summaryCount`, `entityCount` |
396+
| `hierarchicalChunker` | `doc_id` | `chunks: ChunkRecord[]`, `text[]`, `count` |
397+
| `textEmbedding` | (implicit) | `vector[]` |
398+
| `chunkVectorUpsert` | - | `count`, `doc_id`, `chunk_ids` |
404399

405400
This design eliminates the need for external loops - the entire pipeline chains together naturally.

‎docs/technical/08-knowledge-base-and-rag.md‎

Lines changed: 26 additions & 29 deletions
Original file line numberDiff line numberDiff line change
@@ -429,7 +429,7 @@ StructuralParser -----> DocumentRootNode tree
429429
Chunking Task --------> ChunkRecord[]
430430
|
431431
v
432-
ChunkToVectorTask ----> Float32Array[] (embeddings)
432+
TextEmbeddingTask ----> Float32Array[] (embeddings)
433433
|
434434
v
435435
ChunkVectorUpsertTask -> Stored in KnowledgeBase vector storage
@@ -443,31 +443,36 @@ AI Generation Task ---> Answer with context
443443

444444
### Key RAG Tasks
445445

446-
**ChunkToVectorTask** -- Generates embeddings for an array of chunks using a specified model:
446+
**TextEmbeddingTask** -- Generates embeddings for text or an array of chunks using a specified model:
447447

448448
```typescript
449-
// Input: { vector: TypedArray, chunks: ChunkRecord[] }
450-
// Output: { vectors: Float32Array[] }
449+
// Input: { text: string | string[], model: "..." }
450+
// Output: { vector: TypedArray | TypedArray[] }
451451
```
452452

453-
**ChunkVectorUpsertTask** -- Stores chunk vectors in a knowledge base:
453+
**ChunkVectorUpsertTask** -- Stores chunks + their embeddings in a knowledge base (1:1 aligned):
454454

455455
```typescript
456-
// Input: { knowledgeBase: "my-kb", vectors: ChunkVectorEntity[] }
456+
// Input: { knowledgeBase: "my-kb", chunks: ChunkRecord[], vector: TypedArray[] }
457+
// Output: { count: number, doc_id: string, chunk_ids: string[] }
457458
```
458459

459-
**ChunkRetrievalTask** -- Retrieves relevant chunks from a knowledge base given a query:
460+
**ChunkRetrievalTask** -- End-to-end retrieval. Embeds the query (if a string),
461+
then runs similarity or hybrid search:
460462

461463
```typescript
462-
// Input: { knowledgeBase: "my-kb", query: "What is...", model: "..." }
463-
// Output: { chunks: ChunkSearchResult[] }
464+
// Input: { knowledgeBase, query, model?, method?: "similarity" | "hybrid",
465+
// topK?, filter?, scoreThreshold?, vectorWeight?, returnVectors? }
466+
// Output: { chunks, chunk_ids, metadata, scores, count, query, vectors? }
464467
```
465468

466-
**ChunkVectorSearchTask** -- Direct vector similarity search against a knowledge base.
469+
**HierarchyJoinTask** -- Given retrieved metadata, walks the document tree to
470+
reconstruct section context and enrich the metadata with ancestor information:
467471

468-
**ChunkVectorHybridSearchTask** -- Combined vector + full-text search.
469-
470-
**HierarchyJoinTask** -- Given chunk search results, walks the document tree to reconstruct section context and enrich the results with ancestor information.
472+
```typescript
473+
// Input: { knowledgeBase, metadata, includeParentSummaries?, includeEntities? }
474+
// Output: { metadata: ChunkRecord[], count }
475+
```
471476

472477
### Example Workflow
473478

@@ -481,22 +486,14 @@ const kb = await createKnowledgeBase({
481486
vectorDimensions: 1024,
482487
});
483488

484-
// Build an ingestion pipeline
485-
const workflow = new Workflow("ingest");
486-
const parseTask = workflow.addTask("StructuralParseTask", {
487-
text: documentText,
488-
title: "My Paper",
489-
});
490-
const chunkTask = workflow.addTask("ChunkingTask", {});
491-
const embedTask = workflow.addTask("ChunkToVectorTask", {
492-
model: "text-embedding-3-small",
493-
});
494-
const upsertTask = workflow.addTask("ChunkVectorUpsertTask", {
495-
knowledgeBase: "research-papers",
496-
});
497-
498-
workflow.pipe(parseTask, chunkTask, embedTask, upsertTask);
499-
await workflow.run();
489+
// Build an ingestion pipeline — five chained tasks, no transform step needed
490+
await new Workflow()
491+
.structuralParser({ text: documentText, title: "My Paper" })
492+
.documentEnricher({ generateSummaries: true, extractEntities: true })
493+
.hierarchicalChunker({ maxTokens: 512 })
494+
.textEmbedding({ model: "text-embedding-3-small" })
495+
.chunkVectorUpsert({ knowledgeBase: "research-papers" })
496+
.run();
500497
```
501498

502499
## Global Registry

‎package.json‎

Lines changed: 2 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -1,7 +1,7 @@
11
{
2-
"name": "workglow",
2+
"name": "@workglow-dev/libs",
33
"type": "module",
4-
"version": "0.0.10",
4+
"version": "0.2.15",
55
"repository": {
66
"type": "git",
77
"url": "https://github.com/workglow-dev/workglow.git"

‎packages/ai/README.md‎

Lines changed: 11 additions & 15 deletions
Original file line numberDiff line numberDiff line change
@@ -516,23 +516,20 @@ The AI package provides a comprehensive set of tasks for building RAG pipelines.
516516

517517
### Vector and Storage Tasks
518518

519-
| Task | Description |
520-
| ----------------------- | ---------------------------------------------------------------------------------------- |
521-
| `ChunkToVectorTask` | Transforms chunks to vector store format (input: `vector` + `chunks`, output: `vectors`) |
522-
| `ChunkVectorUpsertTask` | Stores vectors in a KnowledgeBase (input: `knowledgeBase` + `vectors`) |
523-
| `ChunkVectorSearchTask` | Searches vectors by similarity |
524-
| `VectorQuantizeTask` | Quantizes vectors for storage efficiency |
519+
| Task | Description |
520+
| ----------------------- | --------------------------------------------------------------------------------- |
521+
| `ChunkVectorUpsertTask` | Stores chunks + their embeddings in a KnowledgeBase (input: `chunks` + `vector`) |
522+
| `VectorQuantizeTask` | Quantizes vectors for storage efficiency |
525523

526524
### Retrieval and Generation Tasks
527525

528-
| Task | Description |
529-
| ----------------------------- | --------------------------------------------- |
530-
| `QueryExpanderTask` | Expands queries for better retrieval coverage |
531-
| `ChunkVectorHybridSearchTask` | Combines vector and full-text search |
532-
| `RerankerTask` | Reranks search results for relevance |
533-
| `HierarchyJoinTask` | Enriches results with parent context |
534-
| `ContextBuilderTask` | Builds context for LLM prompts |
535-
| `ChunkRetrievalTask` | Orchestrates end-to-end retrieval |
526+
| Task | Description |
527+
| -------------------- | ------------------------------------------------------------------------ |
528+
| `ChunkRetrievalTask` | End-to-end retrieval: embeds the query, runs similarity or hybrid search |
529+
| `QueryExpanderTask` | Expands queries (multi-query / synonyms) for better retrieval coverage |
530+
| `RerankerTask` | Reranks search results (simple heuristic or reciprocal-rank-fusion) |
531+
| `HierarchyJoinTask` | Enriches retrieved metadata with parent summaries, section titles, entities |
532+
| `ContextBuilderTask` | Builds formatted context for LLM prompts |
536533

537534
### Complete RAG Workflow Example
538535

@@ -600,7 +597,6 @@ await new Workflow()
600597
.textEmbedding({
601598
model: "onnx:Xenova/all-MiniLM-L6-v2:q8",
602599
})
603-
.chunkToVector()
604600
.chunkVectorUpsert({
605601
knowledgeBase: "my-kb",
606602
})

0 commit comments

Comments
 (0)