chore: update .gitignore and improve coding standards documentation
- Added .cursor/sessions/* to .gitignore to prevent session files from being tracked. - Enhanced coding standards in SKILL.md by adding semicolons to TypeScript examples for consistency. - Improved formatting in continuous learning, detail layout, and other SKILL.md files for better readability. These changes aim to streamline development processes and maintain code quality across the project.
This commit is contained in:
@@ -5,6 +5,7 @@ A formal evaluation framework for Claude Code sessions, implementing eval-driven
|
||||
## Philosophy
|
||||
|
||||
Eval-Driven Development treats evals as the "unit tests of AI development":
|
||||
|
||||
- Define expected behavior BEFORE implementation
|
||||
- Run evals continuously during development
|
||||
- Track regressions with each change
|
||||
@@ -13,33 +14,41 @@ Eval-Driven Development treats evals as the "unit tests of AI development":
|
||||
## Eval Types
|
||||
|
||||
### Capability Evals
|
||||
|
||||
Test if Claude can do something it couldn't before:
|
||||
|
||||
```markdown
|
||||
[CAPABILITY EVAL: feature-name]
|
||||
Task: Description of what Claude should accomplish
|
||||
Success Criteria:
|
||||
- [ ] Criterion 1
|
||||
- [ ] Criterion 2
|
||||
- [ ] Criterion 3
|
||||
Expected Output: Description of expected result
|
||||
|
||||
- [ ] Criterion 1
|
||||
- [ ] Criterion 2
|
||||
- [ ] Criterion 3
|
||||
Expected Output: Description of expected result
|
||||
```
|
||||
|
||||
### Regression Evals
|
||||
|
||||
Ensure changes don't break existing functionality:
|
||||
|
||||
```markdown
|
||||
[REGRESSION EVAL: feature-name]
|
||||
Baseline: SHA or checkpoint name
|
||||
Tests:
|
||||
- existing-test-1: PASS/FAIL
|
||||
- existing-test-2: PASS/FAIL
|
||||
- existing-test-3: PASS/FAIL
|
||||
Result: X/Y passed (previously Y/Y)
|
||||
|
||||
- existing-test-1: PASS/FAIL
|
||||
- existing-test-2: PASS/FAIL
|
||||
- existing-test-3: PASS/FAIL
|
||||
Result: X/Y passed (previously Y/Y)
|
||||
```
|
||||
|
||||
## Grader Types
|
||||
|
||||
### 1. Code-Based Grader
|
||||
|
||||
Deterministic checks using code:
|
||||
|
||||
```bash
|
||||
# Check if file contains expected pattern
|
||||
grep -q "export function handleAuth" src/auth.ts && echo "PASS" || echo "FAIL"
|
||||
@@ -52,10 +61,13 @@ npm run build && echo "PASS" || echo "FAIL"
|
||||
```
|
||||
|
||||
### 2. Model-Based Grader
|
||||
|
||||
Use Claude to evaluate open-ended outputs:
|
||||
|
||||
```markdown
|
||||
[MODEL GRADER PROMPT]
|
||||
Evaluate the following code change:
|
||||
|
||||
1. Does it solve the stated problem?
|
||||
2. Is it well-structured?
|
||||
3. Are edge cases handled?
|
||||
@@ -66,7 +78,9 @@ Reasoning: [explanation]
|
||||
```
|
||||
|
||||
### 3. Human Grader
|
||||
|
||||
Flag for manual review:
|
||||
|
||||
```markdown
|
||||
[HUMAN REVIEW REQUIRED]
|
||||
Change: Description of what changed
|
||||
@@ -77,13 +91,17 @@ Risk Level: LOW/MEDIUM/HIGH
|
||||
## Metrics
|
||||
|
||||
### pass@k
|
||||
|
||||
"At least one success in k attempts"
|
||||
|
||||
- pass@1: First attempt success rate
|
||||
- pass@3: Success within 3 attempts
|
||||
- Typical target: pass@3 > 90%
|
||||
|
||||
### pass^k
|
||||
|
||||
"All k trials succeed"
|
||||
|
||||
- Higher bar for reliability
|
||||
- pass^3: 3 consecutive successes
|
||||
- Use for critical paths
|
||||
@@ -91,28 +109,34 @@ Risk Level: LOW/MEDIUM/HIGH
|
||||
## Eval Workflow
|
||||
|
||||
### 1. Define (Before Coding)
|
||||
|
||||
```markdown
|
||||
## EVAL DEFINITION: feature-xyz
|
||||
|
||||
### Capability Evals
|
||||
|
||||
1. Can create new user account
|
||||
2. Can validate email format
|
||||
3. Can hash password securely
|
||||
|
||||
### Regression Evals
|
||||
|
||||
1. Existing login still works
|
||||
2. Session management unchanged
|
||||
3. Logout flow intact
|
||||
|
||||
### Success Metrics
|
||||
|
||||
- pass@3 > 90% for capability evals
|
||||
- pass^3 = 100% for regression evals
|
||||
```
|
||||
|
||||
### 2. Implement
|
||||
|
||||
Write code to pass the defined evals.
|
||||
|
||||
### 3. Evaluate
|
||||
|
||||
```bash
|
||||
# Run capability evals
|
||||
[Run each capability eval, record PASS/FAIL]
|
||||
@@ -124,25 +148,25 @@ npm test -- --testPathPattern="existing"
|
||||
```
|
||||
|
||||
### 4. Report
|
||||
|
||||
```markdown
|
||||
EVAL REPORT: feature-xyz
|
||||
========================
|
||||
# EVAL REPORT: feature-xyz
|
||||
|
||||
Capability Evals:
|
||||
create-user: PASS (pass@1)
|
||||
validate-email: PASS (pass@2)
|
||||
hash-password: PASS (pass@1)
|
||||
Overall: 3/3 passed
|
||||
create-user: PASS (pass@1)
|
||||
validate-email: PASS (pass@2)
|
||||
hash-password: PASS (pass@1)
|
||||
Overall: 3/3 passed
|
||||
|
||||
Regression Evals:
|
||||
login-flow: PASS
|
||||
session-mgmt: PASS
|
||||
logout-flow: PASS
|
||||
Overall: 3/3 passed
|
||||
login-flow: PASS
|
||||
session-mgmt: PASS
|
||||
logout-flow: PASS
|
||||
Overall: 3/3 passed
|
||||
|
||||
Metrics:
|
||||
pass@1: 67% (2/3)
|
||||
pass@3: 100% (3/3)
|
||||
pass@1: 67% (2/3)
|
||||
pass@3: 100% (3/3)
|
||||
|
||||
Status: READY FOR REVIEW
|
||||
```
|
||||
@@ -150,26 +174,33 @@ Status: READY FOR REVIEW
|
||||
## Integration Patterns
|
||||
|
||||
### Pre-Implementation
|
||||
|
||||
```
|
||||
/eval define feature-name
|
||||
```
|
||||
|
||||
Creates eval definition file at `.cursor/evals/feature-name.md`
|
||||
|
||||
### During Implementation
|
||||
|
||||
```
|
||||
/eval check feature-name
|
||||
```
|
||||
|
||||
Runs current evals and reports status
|
||||
|
||||
### Post-Implementation
|
||||
|
||||
```
|
||||
/eval report feature-name
|
||||
```
|
||||
|
||||
Generates full eval report
|
||||
|
||||
## Eval Storage
|
||||
|
||||
Store evals in project:
|
||||
|
||||
```
|
||||
.cursor/
|
||||
evals/
|
||||
@@ -194,7 +225,9 @@ Store evals in project:
|
||||
## EVAL: add-authentication
|
||||
|
||||
### Phase 1: Define (10 min)
|
||||
|
||||
Capability Evals:
|
||||
|
||||
- [ ] User can register with email/password
|
||||
- [ ] User can login with valid credentials
|
||||
- [ ] Invalid credentials rejected with proper error
|
||||
@@ -202,19 +235,23 @@ Capability Evals:
|
||||
- [ ] Logout clears session
|
||||
|
||||
Regression Evals:
|
||||
|
||||
- [ ] Public routes still accessible
|
||||
- [ ] API responses unchanged
|
||||
- [ ] Database schema compatible
|
||||
|
||||
### Phase 2: Implement (varies)
|
||||
|
||||
[Write code]
|
||||
|
||||
### Phase 3: Evaluate
|
||||
|
||||
Run: /eval check add-authentication
|
||||
|
||||
### Phase 4: Report
|
||||
EVAL REPORT: add-authentication
|
||||
==============================
|
||||
|
||||
# EVAL REPORT: add-authentication
|
||||
|
||||
Capability: 5/5 passed (pass@3: 100%)
|
||||
Regression: 3/3 passed (pass^3: 100%)
|
||||
Status: SHIP IT
|
||||
|
||||
Reference in New Issue
Block a user