diff --git a/packages/core-storage/README.md b/packages/core-storage/README.md index f24f858..f14ceac 100644 --- a/packages/core-storage/README.md +++ b/packages/core-storage/README.md @@ -2,54 +2,171 @@ [← Back to Root](../../README.md) -This package provides an **Offline-First Storage Engine** using PouchDB, tailored for Enterprise React applications. It is built to seamlessly sync with remote CouchDB instances, providing full fault tolerance and offline capabilities. +`@repo/core-storage` is the **Enterprise-grade, multi-tool storage engine** for the Eigen Monorepo. -## High-Level Overview +It provides a unified set of strictly-typed, secure, and fault-tolerant storage mechanisms tailored for React applications. It enforces strict **Inversion of Control (IoC)**β€”the core engine knows absolutely nothing about your application's business domains; instead, the consuming apps inject their own configurations and types. -Our storage architecture enforces strict **Inversion of Control (IoC)**. The core engine (`@repo/core-storage`) is a pure factoryβ€”it knows absolutely nothing about your business domains, data models, or specific databases. Consuming applications (like `apps/web`) dictate the rules by injecting their specific configurations and generic types into the storage engine. - -```mermaid -graph TD - subgraph UI ["Consuming App (apps/*)"] - COMP["React Components / Forms"] - end - - subgraph CoreStorage ["@repo/core-storage Engine"] - MGR["PouchDatabaseManager Factory"] - L_SALES[("Local PouchDB: Sales")] - L_INV[("Local PouchDB: Inventory")] - end - - subgraph RemoteServer ["CouchDB Cluster (Cloud/On-Prem)"] - R_SALES[("Remote CouchDB: sales_db")] - R_INV[("Remote CouchDB: inventory_db")] - end - - COMP -->|Read / Write| L_SALES - COMP -->|Read / Write| L_INV - MGR -->|Instantiates Multi-DB| L_SALES - MGR -->|Instantiates Multi-DB| L_INV - - L_SALES <-->|Native Sync Live and Retry| R_SALES - L_INV <-->|Native Sync Live and Retry| R_INV - - %% Styling Nodes - style MGR fill:#339af0,stroke:#1864ab,color:#fff - style L_SALES fill:#845ef7,stroke:#5f3dc4,color:#fff - style L_INV fill:#845ef7,stroke:#5f3dc4,color:#fff - style R_SALES fill:#fab005,stroke:#e67700,color:#fff - style R_INV fill:#fab005,stroke:#e67700,color:#fff -``` +This package provides three primary storage solutions: +1. **Secure Local Storage** (Strict Key-Gatekeeping & AES encryption) +2. **Secure IndexedDB** (For larger key-value payloads) +3. **Offline-First PouchDB** (For document-oriented, bi-directional sync data) --- -## Core Concepts & Usage +## πŸ”’ Secure Key-Value Storage (LocalStorage & IndexedDB) -### 1. Initialization & Registration (`PouchDatabaseManager`) +Browser storage is notoriously vulnerable to XSS attacks and pollution. The `LocalStorageService` and `IndexedDBService` implement a strict **Gatekeeper** pattern to solve this. -The `PouchDatabaseManager` acts as the IoC Factory. Apps use it to register and initialize multiple discrete PouchDB databases using a `PouchConfig`. +By forcing developers to register every key explicitly into either `plainTextKeys` or `encryptedKeys`, the engine guarantees: +1. No unapproved or rogue keys can ever be written or read (throws a `Security Exception`). +2. Highly sensitive tokens (e.g., JWTs) are automatically routed through the `@repo/utils` AES Encryption pipeline before touching the disk. -**Why we use this pattern:** Instead of scattering raw database instantiations across the codebase, the manager centralizes connections. If a database is requested twice, the manager efficiently returns the exact same instance. +### Architecture + +```mermaid +graph TD + %% ─── Styling Definitions (Dark-Mode Friendly Enterprise Palette) ─── + classDef appEntity fill:#dbeafe,stroke:#3b82f6,stroke-width:2px,color:#1e3a8a + classDef coreEntity fill:#e2e8f0,stroke:#64748b,stroke-width:2px,color:#0f172a + classDef gatekeeper fill:#10b981,stroke:#047857,stroke-width:2px,color:#ffffff + classDef encrypt fill:#f59e0b,stroke:#b45309,stroke-width:2px,color:#ffffff + classDef error fill:#f43f5e,stroke:#be123c,stroke-width:2px,color:#ffffff + classDef database fill:#6366f1,stroke:#4338ca,stroke-width:2px,color:#ffffff + + %% ─── Subgraphs ─── + subgraph Apps ["apps/* (App Autonomy)"] + REG[[AppStorageKey Config]] + UI([React Components / API]) + INST{{Storage Instances}} + end + + subgraph Core ["@repo/core-storage"] + FAC[Factory: createStorage] + API[IStorageService API] + VAL{Runtime Gatekeeper} + ERR>Throws Security Exception] + ENC{{AES Encryption Pipeline}} + LOCAL[(LocalStorage Adapter)] + IDB[(IndexedDB Adapter)] + end + + %% ─── Flow & Relationships ─── + %% 1. Initialization Flow + REG -.->|Injects Keys & Config| FAC + FAC -.->|Returns| INST + + %% 2. Runtime Execution Flow + UI ===>|getItem / setItem| INST + INST ---> API + API ---> VAL + + %% 3. Gatekeeper Decision Tree + VAL -.->|Invalid Key| ERR + VAL ===>|Sensitive Key| ENC + + VAL --->|Plain-text Key| LOCAL + VAL --->|Plain-text Key| IDB + + %% 4. Post-Encryption Storage + ENC ===>|Encrypted Data| LOCAL + ENC ===>|Encrypted Data| IDB + + %% ─── Apply Styles ─── + class REG,UI,INST appEntity; + class FAC,API coreEntity; + class VAL gatekeeper; + class ENC encrypt; + class ERR error; + class LOCAL,IDB database; + + %% ─── Subgraph Backgrounds (Transparent for Native GitHub Support) ─── + style Apps fill:transparent,stroke:#818cf8,stroke-width:2px,stroke-dasharray: 5 5 + style Core fill:transparent,stroke:#94a3b8,stroke-width:2px,stroke-dasharray: 5 5 +``` + +### Usage & Implementation + +```typescript +import { createLocalStorage, createIndexedDB } from '@repo/core-storage'; + +// 1. Define allowed keys (Strict Type Safety) +export type AppStorageKey = 'THEME' | 'ACCESS_TOKEN' | 'OFFLINE_CACHE'; + +// 2. Instantiate Local Storage +export const appStorage = createLocalStorage({ + plainTextKeys: new Set(['THEME']), + encryptedKeys: new Set(['ACCESS_TOKEN']), // Auto AES encrypted +}); + +// 3. Usage +await appStorage.setItem('ACCESS_TOKEN', 'ey...'); // Encrypted on disk +const theme = await appStorage.getItem('THEME'); // Plaintext on disk +``` + +### βœ… Do's and ❌ Don'ts + +* **βœ… DO use TypeScript Literal Types** for your storage keys (`type Keys = 'A' | 'B'`) to get full IntelliSense. +* **βœ… DO place Session/Auth tokens** exclusively inside the `encryptedKeys` Set. +* **❌ DON'T use native `window.localStorage` directly** anywhere in your React components. It bypasses our encryption and gatekeeper logic. +* **❌ DON'T mix domain data.** Keep UI preferences (Theme, Sidebar state) in LocalStorage, and large datasets (Offline Caches) in IndexedDB. + +--- + +## πŸ”„ Offline-First Document Storage (PouchDB & CouchDB) + +For complex, document-oriented data that requires fault-tolerance, offline support, and bi-directional cloud synchronization, we use the `PouchDatabaseManager`. + +### Architecture + +```mermaid +graph TD + %% ─── Styling Definitions (Dark-Mode Friendly Enterprise Palette) ─── + classDef appEntity fill:#dbeafe,stroke:#3b82f6,stroke-width:2px,color:#1e3a8a + classDef coreEntity fill:#e2e8f0,stroke:#64748b,stroke-width:2px,color:#0f172a + classDef localDb fill:#6366f1,stroke:#4338ca,stroke-width:2px,color:#ffffff + classDef remoteDb fill:#f59e0b,stroke:#b45309,stroke-width:2px,color:#ffffff + + %% ─── Subgraphs ─── + subgraph UI ["Consuming App (apps/*)"] + COMP([React Components / Forms]) + end + + subgraph CoreStorage ["@repo/core-storage Engine"] + MGR[PouchDatabaseManager Factory] + L_SALES[(Local PouchDB: Sales)] + L_INV[(Local PouchDB: Inventory)] + end + + subgraph RemoteServer ["CouchDB Cluster"] + R_SALES[(Remote CouchDB: sales_db)] + R_INV[(Remote CouchDB: inventory_db)] + end + + %% ─── Flow & Relationships ─── + COMP ===>|Read / Write| L_SALES + COMP ===>|Read / Write| L_INV + + MGR -.->|Instantiates Multi-DB| L_SALES + MGR -.->|Instantiates Multi-DB| L_INV + + L_SALES <===>|Native Sync Live and Retry| R_SALES + L_INV <===>|Native Sync Live and Retry| R_INV + + %% ─── Apply Styles ─── + class COMP appEntity; + class MGR coreEntity; + class L_SALES,L_INV localDb; + class R_SALES,R_INV remoteDb; + + %% ─── Subgraph Backgrounds (Transparent for Native GitHub Support) ─── + style UI fill:transparent,stroke:#3b82f6,stroke-width:2px,stroke-dasharray: 5 5 + style CoreStorage fill:transparent,stroke:#94a3b8,stroke-width:2px,stroke-dasharray: 5 5 + style RemoteServer fill:transparent,stroke:#f59e0b,stroke-width:2px,stroke-dasharray: 5 5 +``` + +### 1. Initialization (IoC Factory) + +The `PouchDatabaseManager` acts as a central singleton. It registers and manages all database instances. If a remote URL is provided, it automatically handles background synchronization. ```typescript import { PouchDatabaseManager } from '@repo/core-storage'; @@ -57,45 +174,34 @@ import type { Item } from './types'; export const dbManager = new PouchDatabaseManager(); -// Register a strictly-typed database with bi-directional sync export const itemDB = dbManager.register({ localName: 'items_db', remoteUrl: 'http://admin:password@localhost:5984/items_db' }); ``` -### 2. CRUD & Queries (`PouchDatabaseWrapper`) +### 2. CRUD & MongoDB-style Queries -When you register a database, you receive a strictly typed `PouchDatabaseWrapper`. This wrapper abstracts away the raw PouchDB API, giving developers clean, Promise-based helper methods without ever needing to pass `dbName` or complex identifiers repeatedly. +The registered database returns a `PouchDatabaseWrapper`. This wrapper fully abstracts the raw PouchDB API into clean, Promise-based helpers, auto-handling `_rev` conflicts. | Method | Description | |---|---| -| `create(data)` | Inserts a new document. PouchDB will auto-generate an `_id` if omitted. | -| `update(id, data)` | Automatically fetches the latest `_rev` to merge the payload, preventing conflict errors. | -| `delete(id)` | Automatically fetches the latest `_rev` to safely remove the document. | -| `getOne(id)` | Retrieves a single document by its `_id`. | -| `getAll()` | Retrieves all documents, automatically filtering out internal `_design/` docs. | +| `create(data)` | Inserts a new document. Auto-generates `_id` if omitted. | +| `update(id, data)` | Auto-fetches the latest `_rev` to merge payloads cleanly. | +| `delete(id)` | Auto-fetches the latest `_rev` to safely remove the document. | +| `getAll()` | Retrieves all documents (filters out internal `_design/` docs). | | `find(options)` | Queries using MongoDB-style selectors (via `pouchdb-find`). | -**Example of `find()` with Selectors:** -Instead of pulling all documents into memory and filtering them with JavaScript, we leverage native MongoDB-style selectors for performance: - ```typescript +// Example: Querying data using selectors const expensiveItems = await itemDB.find({ - selector: { - price: { $gt: 100 }, - category: 'electronics' - } + selector: { price: { $gt: 100 }, category: 'electronics' } }); ``` -### 3. Real-Time Reactivity (The `onChange` Pub/Sub Pattern) +### 3. Real-Time Reactivity (`onChange` Pub/Sub) -**CRITICAL CONCEPT:** We do **not** expose the raw `db.changes()` feed directly to React components. Instead, the `PouchDatabaseWrapper` utilizes a clean Pub/Sub abstraction via the `.onChange(callback)` method. - -**Why we use this pattern:** -1. **Memory Safety:** Direct bindings to PouchDB's raw changes feed often lead to zombie listeners and memory leaks. The `.onChange()` returns an unsubscribe function natively tailored for React's `useEffect` cleanup block. -2. **Connection Efficiency:** It maintains a *single* WebSocket/Polling connection to the database under the hood. Multiple React components can subscribe to the same wrapper without opening dozens of parallel database connections. +We implemented a **Publisher-Subscriber (Pub/Sub)** pattern inside the wrapper to handle real-time data changes efficiently. The wrapper maintains a *single* background connection to the changes feed and broadcasts events to all React subscribers. ```tsx import { useEffect, useCallback, useState } from 'react'; @@ -110,32 +216,34 @@ export function InventoryList() { }, []); useEffect(() => { - // 1. Initial Load loadData(); - // 2. Subscribe to local mutations AND remote CouchDB syncs + // Subscribe to background sync mutations const unsubscribe = itemDB.onChange(() => { - console.log('Database updated locally or remotely. Refreshing...'); loadData(); }); - // 3. Prevent memory leaks! - return () => { - unsubscribe(); - }; + // CRITICAL: Prevent memory leaks + return () => unsubscribe(); }, [loadData]); - - // UI rendering... } ``` -### 4. CouchDB Sync & CORS Troubleshooting +### βœ… Do's and ❌ Don'ts for PouchDB -By providing a `remoteUrl` to the manager, the engine automatically handles bi-directional synchronization in the background (`live: true, retry: true`). If the server goes down, the local app will continue working seamlessly and sync automatically when the connection is restored. +* **βœ… DO use `.onChange()`** to make your UI reactive to background cloud syncs. +* **βœ… DO return the `unsubscribe` function** in your `useEffect` cleanup block to prevent severe memory leaks. +* **❌ DON'T use `db.raw.changes()`** inside your React components. It creates zombie WebSocket connections and tightly couples your UI to PouchDB's specific API. +* **❌ DON'T pass the `_rev` property** manually when updating or deleting. The wrapper's `update()` and `delete()` methods handle revision fetching automatically. -> [!WARNING] -> **CORS Infinite Retries & Preflight Failures** -> If your browser blocks the synchronization with a CORS error, you will see PouchDB enter an infinite retry loop in the network tab. -> -> **Do NOT try to fix this in the frontend code!** -> This is exclusively a CouchDB server configuration issue. You must enable CORS directly on the CouchDB instance by editing its `local.ini` or using its dashboard configuration to allow origins, credentials, and headers. \ No newline at end of file +--- + +## ⚠️ Troubleshooting + +### CouchDB CORS Infinite Retries +By providing a `remoteUrl`, the engine runs bi-directional sync in the background (`live: true, retry: true`). Fault tolerance is guaranteed: if CouchDB crashes, local reads/writes continue uninterrupted. + +However, if your browser blocks CouchDB sync with a **CORS error**, PouchDB will misinterpret this as a network failure and enter an infinite retry loop, flooding your Network tab. + +> **DO NOT try to fix this in the frontend Vite config or proxy!** +> This is strictly a CouchDB server policy issue. You must enable CORS directly on the CouchDB cluster (editing its `local.ini` or via its dashboard) to allow `origins`, `credentials`, and `headers`. \ No newline at end of file