eigen/trackgo-fe
5
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases 12 Wiki Activity

feat: enhance README with detailed overview of core storage engine and PouchDB integration

Browse Source
This commit is contained in:
Firman Ramdhani
2026-05-29 16:01:55 +07:00
parent 6712558eaf
commit 9f1158653a
1 changed files with 186 additions and 78 deletions
Download Patch File Download Diff File Expand all files Collapse all files
packages/core-storage/README.md
+186 -78
View File
@@ -2,54 +2,171 @@
[← Back to Root](../../README.md)
This package provides an **Offline-First Storage Engine** using PouchDB, tailored for Enterprise React applications. It is built to seamlessly sync with remote CouchDB instances, providing full fault tolerance and offline capabilities.
`@repo/core-storage` is the **Enterprise-grade, multi-tool storage engine** for the Eigen Monorepo.
## High-Level Overview
It provides a unified set of strictly-typed, secure, and fault-tolerant storage mechanisms tailored for React applications. It enforces strict **Inversion of Control (IoC)**—the core engine knows absolutely nothing about your application's business domains; instead, the consuming apps inject their own configurations and types.
Our storage architecture enforces strict **Inversion of Control (IoC)**. The core engine (`@repo/core-storage`) is a pure factory—it knows absolutely nothing about your business domains, data models, or specific databases. Consuming applications (like `apps/web`) dictate the rules by injecting their specific configurations and generic types into the storage engine.
```mermaid
graph TD
subgraph UI ["Consuming App (apps/*)"]
COMP["React Components / Forms"]
end
subgraph CoreStorage ["@repo/core-storage Engine"]
MGR["PouchDatabaseManager Factory"]
L_SALES[("Local PouchDB: Sales")]
L_INV[("Local PouchDB: Inventory")]
end
subgraph RemoteServer ["CouchDB Cluster (Cloud/On-Prem)"]
R_SALES[("Remote CouchDB: sales_db")]
R_INV[("Remote CouchDB: inventory_db")]
end
COMP -->|Read / Write| L_SALES
COMP -->|Read / Write| L_INV
MGR -->|Instantiates Multi-DB| L_SALES
MGR -->|Instantiates Multi-DB| L_INV
L_SALES <-->|Native Sync Live and Retry| R_SALES
L_INV <-->|Native Sync Live and Retry| R_INV
%% Styling Nodes
style MGR fill:#339af0,stroke:#1864ab,color:#fff
style L_SALES fill:#845ef7,stroke:#5f3dc4,color:#fff
style L_INV fill:#845ef7,stroke:#5f3dc4,color:#fff
style R_SALES fill:#fab005,stroke:#e67700,color:#fff
style R_INV fill:#fab005,stroke:#e67700,color:#fff
```
This package provides three primary storage solutions:
1. **Secure Local Storage** (Strict Key-Gatekeeping & AES encryption)
2. **Secure IndexedDB** (For larger key-value payloads)
3. **Offline-First PouchDB** (For document-oriented, bi-directional sync data)
---
## Core Concepts & Usage
## 🔒 Secure Key-Value Storage (LocalStorage & IndexedDB)
### 1. Initialization & Registration (`PouchDatabaseManager`)
Browser storage is notoriously vulnerable to XSS attacks and pollution. The `LocalStorageService` and `IndexedDBService` implement a strict **Gatekeeper** pattern to solve this.
The `PouchDatabaseManager` acts as the IoC Factory. Apps use it to register and initialize multiple discrete PouchDB databases using a `PouchConfig`.
By forcing developers to register every key explicitly into either `plainTextKeys` or `encryptedKeys`, the engine guarantees:
1. No unapproved or rogue keys can ever be written or read (throws a `Security Exception`).
2. Highly sensitive tokens (e.g., JWTs) are automatically routed through the `@repo/utils` AES Encryption pipeline before touching the disk.
**Why we use this pattern:** Instead of scattering raw database instantiations across the codebase, the manager centralizes connections. If a database is requested twice, the manager efficiently returns the exact same instance.
### Architecture
```mermaid
graph TD
%% ─── Styling Definitions (Dark-Mode Friendly Enterprise Palette) ───
classDef appEntity fill:#dbeafe,stroke:#3b82f6,stroke-width:2px,color:#1e3a8a
classDef coreEntity fill:#e2e8f0,stroke:#64748b,stroke-width:2px,color:#0f172a
classDef gatekeeper fill:#10b981,stroke:#047857,stroke-width:2px,color:#ffffff
classDef encrypt fill:#f59e0b,stroke:#b45309,stroke-width:2px,color:#ffffff
classDef error fill:#f43f5e,stroke:#be123c,stroke-width:2px,color:#ffffff
classDef database fill:#6366f1,stroke:#4338ca,stroke-width:2px,color:#ffffff
%% ─── Subgraphs ───
subgraph Apps ["apps/* (App Autonomy)"]
REG[[AppStorageKey Config]]
UI([React Components / API])
INST{{Storage Instances}}
end
subgraph Core ["@repo/core-storage"]
FAC[Factory: createStorage]
API[IStorageService API]
VAL{Runtime Gatekeeper}
ERR>Throws Security Exception]
ENC{{AES Encryption Pipeline}}
LOCAL[(LocalStorage Adapter)]
IDB[(IndexedDB Adapter)]
end
%% ─── Flow & Relationships ───
%% 1. Initialization Flow
REG -.->|Injects Keys & Config| FAC
FAC -.->|Returns| INST
%% 2. Runtime Execution Flow
UI ===>|getItem / setItem| INST
INST ---> API
API ---> VAL
%% 3. Gatekeeper Decision Tree
VAL -.->|Invalid Key| ERR
VAL ===>|Sensitive Key| ENC
VAL --->|Plain-text Key| LOCAL
VAL --->|Plain-text Key| IDB
%% 4. Post-Encryption Storage
ENC ===>|Encrypted Data| LOCAL
ENC ===>|Encrypted Data| IDB
%% ─── Apply Styles ───
class REG,UI,INST appEntity;
class FAC,API coreEntity;
class VAL gatekeeper;
class ENC encrypt;
class ERR error;
class LOCAL,IDB database;
%% ─── Subgraph Backgrounds (Transparent for Native GitHub Support) ───
style Apps fill:transparent,stroke:#818cf8,stroke-width:2px,stroke-dasharray: 5 5
style Core fill:transparent,stroke:#94a3b8,stroke-width:2px,stroke-dasharray: 5 5
```
### Usage & Implementation
```typescript
import { createLocalStorage, createIndexedDB } from '@repo/core-storage';
// 1. Define allowed keys (Strict Type Safety)
export type AppStorageKey = 'THEME' | 'ACCESS_TOKEN' | 'OFFLINE_CACHE';
// 2. Instantiate Local Storage
export const appStorage = createLocalStorage<AppStorageKey>({
plainTextKeys: new Set(['THEME']),
encryptedKeys: new Set(['ACCESS_TOKEN']), // Auto AES encrypted
});
// 3. Usage
await appStorage.setItem('ACCESS_TOKEN', 'ey...'); // Encrypted on disk
const theme = await appStorage.getItem('THEME'); // Plaintext on disk
```
### ✅ Do's and ❌ Don'ts
* **✅ DO use TypeScript Literal Types** for your storage keys (`type Keys = 'A' | 'B'`) to get full IntelliSense.
* **✅ DO place Session/Auth tokens** exclusively inside the `encryptedKeys` Set.
* **❌ DON'T use native `window.localStorage` directly** anywhere in your React components. It bypasses our encryption and gatekeeper logic.
* **❌ DON'T mix domain data.** Keep UI preferences (Theme, Sidebar state) in LocalStorage, and large datasets (Offline Caches) in IndexedDB.
---
## 🔄 Offline-First Document Storage (PouchDB & CouchDB)
For complex, document-oriented data that requires fault-tolerance, offline support, and bi-directional cloud synchronization, we use the `PouchDatabaseManager`.
### Architecture
```mermaid
graph TD
%% ─── Styling Definitions (Dark-Mode Friendly Enterprise Palette) ───
classDef appEntity fill:#dbeafe,stroke:#3b82f6,stroke-width:2px,color:#1e3a8a
classDef coreEntity fill:#e2e8f0,stroke:#64748b,stroke-width:2px,color:#0f172a
classDef localDb fill:#6366f1,stroke:#4338ca,stroke-width:2px,color:#ffffff
classDef remoteDb fill:#f59e0b,stroke:#b45309,stroke-width:2px,color:#ffffff
%% ─── Subgraphs ───
subgraph UI ["Consuming App (apps/*)"]
COMP([React Components / Forms])
end
subgraph CoreStorage ["@repo/core-storage Engine"]
MGR[PouchDatabaseManager Factory]
L_SALES[(Local PouchDB: Sales)]
L_INV[(Local PouchDB: Inventory)]
end
subgraph RemoteServer ["CouchDB Cluster"]
R_SALES[(Remote CouchDB: sales_db)]
R_INV[(Remote CouchDB: inventory_db)]
end
%% ─── Flow & Relationships ───
COMP ===>|Read / Write| L_SALES
COMP ===>|Read / Write| L_INV
MGR -.->|Instantiates Multi-DB| L_SALES
MGR -.->|Instantiates Multi-DB| L_INV
L_SALES <===>|Native Sync Live and Retry| R_SALES
L_INV <===>|Native Sync Live and Retry| R_INV
%% ─── Apply Styles ───
class COMP appEntity;
class MGR coreEntity;
class L_SALES,L_INV localDb;
class R_SALES,R_INV remoteDb;
%% ─── Subgraph Backgrounds (Transparent for Native GitHub Support) ───
style UI fill:transparent,stroke:#3b82f6,stroke-width:2px,stroke-dasharray: 5 5
style CoreStorage fill:transparent,stroke:#94a3b8,stroke-width:2px,stroke-dasharray: 5 5
style RemoteServer fill:transparent,stroke:#f59e0b,stroke-width:2px,stroke-dasharray: 5 5
```
### 1. Initialization (IoC Factory)
The `PouchDatabaseManager` acts as a central singleton. It registers and manages all database instances. If a remote URL is provided, it automatically handles background synchronization.
```typescript
import { PouchDatabaseManager } from '@repo/core-storage';
@@ -57,45 +174,34 @@ import type { Item } from './types';
export const dbManager = new PouchDatabaseManager();
// Register a strictly-typed database with bi-directional sync
export const itemDB = dbManager.register<Item>({
localName: 'items_db',
remoteUrl: 'http://admin:password@localhost:5984/items_db'
});
```
### 2. CRUD & Queries (`PouchDatabaseWrapper`)
### 2. CRUD & MongoDB-style Queries
When you register a database, you receive a strictly typed `PouchDatabaseWrapper`. This wrapper abstracts away the raw PouchDB API, giving developers clean, Promise-based helper methods without ever needing to pass `dbName` or complex identifiers repeatedly.
The registered database returns a `PouchDatabaseWrapper`. This wrapper fully abstracts the raw PouchDB API into clean, Promise-based helpers, auto-handling `_rev` conflicts.
| Method | Description |
|---|---|
| `create(data)` | Inserts a new document. PouchDB will auto-generate an `_id` if omitted. |
| `update(id, data)` | Automatically fetches the latest `_rev` to merge the payload, preventing conflict errors. |
| `delete(id)` | Automatically fetches the latest `_rev` to safely remove the document. |
| `getOne(id)` | Retrieves a single document by its `_id`. |
| `getAll()` | Retrieves all documents, automatically filtering out internal `_design/` docs. |
| `create(data)` | Inserts a new document. Auto-generates `_id` if omitted. |
| `update(id, data)` | Auto-fetches the latest `_rev` to merge payloads cleanly. |
| `delete(id)` | Auto-fetches the latest `_rev` to safely remove the document. |
| `getAll()` | Retrieves all documents (filters out internal `_design/` docs). |
| `find(options)` | Queries using MongoDB-style selectors (via `pouchdb-find`). |
**Example of `find()` with Selectors:**
Instead of pulling all documents into memory and filtering them with JavaScript, we leverage native MongoDB-style selectors for performance:
```typescript
// Example: Querying data using selectors
const expensiveItems = await itemDB.find({
selector: {
price: { $gt: 100 },
category: 'electronics'
}
selector: { price: { $gt: 100 }, category: 'electronics' }
});
```
### 3. Real-Time Reactivity (The `onChange` Pub/Sub Pattern)
### 3. Real-Time Reactivity (`onChange` Pub/Sub)
**CRITICAL CONCEPT:** We do **not** expose the raw `db.changes()` feed directly to React components. Instead, the `PouchDatabaseWrapper` utilizes a clean Pub/Sub abstraction via the `.onChange(callback)` method.
**Why we use this pattern:**
1. **Memory Safety:** Direct bindings to PouchDB's raw changes feed often lead to zombie listeners and memory leaks. The `.onChange()` returns an unsubscribe function natively tailored for React's `useEffect` cleanup block.
2. **Connection Efficiency:** It maintains a *single* WebSocket/Polling connection to the database under the hood. Multiple React components can subscribe to the same wrapper without opening dozens of parallel database connections.
We implemented a **Publisher-Subscriber (Pub/Sub)** pattern inside the wrapper to handle real-time data changes efficiently. The wrapper maintains a *single* background connection to the changes feed and broadcasts events to all React subscribers.
```tsx
import { useEffect, useCallback, useState } from 'react';
@@ -110,32 +216,34 @@ export function InventoryList() {
}, []);
useEffect(() => {
// 1. Initial Load
loadData();
// 2. Subscribe to local mutations AND remote CouchDB syncs
// Subscribe to background sync mutations
const unsubscribe = itemDB.onChange(() => {
console.log('Database updated locally or remotely. Refreshing...');
loadData();
});
// 3. Prevent memory leaks!
return () => {
unsubscribe();
};
// CRITICAL: Prevent memory leaks
return () => unsubscribe();
}, [loadData]);
// UI rendering...
}
```
### 4. CouchDB Sync & CORS Troubleshooting
### ✅ Do's and ❌ Don'ts for PouchDB
By providing a `remoteUrl` to the manager, the engine automatically handles bi-directional synchronization in the background (`live: true, retry: true`). If the server goes down, the local app will continue working seamlessly and sync automatically when the connection is restored.
* **✅ DO use `.onChange()`** to make your UI reactive to background cloud syncs.
* **✅ DO return the `unsubscribe` function** in your `useEffect` cleanup block to prevent severe memory leaks.
* **❌ DON'T use `db.raw.changes()`** inside your React components. It creates zombie WebSocket connections and tightly couples your UI to PouchDB's specific API.
* **❌ DON'T pass the `_rev` property** manually when updating or deleting. The wrapper's `update()` and `delete()` methods handle revision fetching automatically.
> [!WARNING]
> **CORS Infinite Retries & Preflight Failures**
> If your browser blocks the synchronization with a CORS error, you will see PouchDB enter an infinite retry loop in the network tab.
>
> **Do NOT try to fix this in the frontend code!**
> This is exclusively a CouchDB server configuration issue. You must enable CORS directly on the CouchDB instance by editing its `local.ini` or using its dashboard configuration to allow origins, credentials, and headers.
---
## ⚠️ Troubleshooting
### CouchDB CORS Infinite Retries
By providing a `remoteUrl`, the engine runs bi-directional sync in the background (`live: true, retry: true`). Fault tolerance is guaranteed: if CouchDB crashes, local reads/writes continue uninterrupted.
However, if your browser blocks CouchDB sync with a **CORS error**, PouchDB will misinterpret this as a network failure and enter an infinite retry loop, flooding your Network tab.
> **DO NOT try to fix this in the frontend Vite config or proxy!**
> This is strictly a CouchDB server policy issue. You must enable CORS directly on the CouchDB cluster (editing its `local.ini` or via its dashboard) to allow `origins`, `credentials`, and `headers`.