feat(bots): Implement comprehensive bot system with autonomous posting, content management, and API endpoints

- Add bot management system with creation, suspension, and reinstatement functionality - Implement autonomous bot posting with scheduling, rate limiting, and content generation - Add content fetching system supporting RSS feeds and multiple content sources - Implement LLM-based content generation with customizable bot personalities - Add mention handling and automated response system for bot interactions - Implement API key management with encryption using AUTH_SECRET for simplified deployment - Add comprehensive bot logging system for activity tracking and error monitoring - Create bot administration pages and settings UI for managing bot configurations - Add database migrations for bot system schema including users, sources, and content items - Implement cron job system for automated bot operations and scheduled tasks - Add extensive test coverage with unit and property-based tests for core bot modules - Simplify encryption by deriving keys from AUTH_SECRET instead of separate environment variable - Implement automatic content fetching on post trigger with retry logic - Add Reddit-specific link preview handling using oEmbed API for reliable metadata extraction - Create utility scripts for bot inspection and cleanup operations - Add comprehensive bot system documentation and improvement tracking
2026-01-25 16:22:41 +01:00
parent cfb558fff1
commit 8ad3b97b7e
102 changed files with 41692 additions and 164 deletions
@@ -0,0 +1,119 @@
+# Bot System Improvements
+
+## Recent Changes
+
+### 1. Simplified Encryption (✅ Completed)
+
+**Problem**: Required a separate `BOT_ENCRYPTION_KEY` environment variable, adding complexity to deployment.
+
+**Solution**: Now uses `AUTH_SECRET` (which is already required) for encryption. A 32-byte key is derived using SHA-256.
+
+**Benefits**:
+- One less environment variable to manage
+- Simpler deployment process
+- Same security level (AES-256-GCM)
+
+**Migration**: Remove `BOT_ENCRYPTION_KEY` from your `.env` file. Existing encrypted keys will need to be re-encrypted (or you can keep the old key temporarily for migration).
+
+### 2. Automatic Content Fetching (✅ Completed)
+
+**Problem**: Users had to manually set fetch intervals, and content fetching was a separate scheduled process.
+
+**Solution**: Content is now fetched automatically when a post is triggered. The system:
+1. Checks all active content sources
+2. Fetches fresh content (with retry logic)
+3. Selects the best content item
+4. Generates and posts
+
+**Benefits**:
+- No manual fetch interval configuration needed
+- Always uses fresh content
+- Simpler user experience
+- Removed `fetchIntervalMinutes` column from database
+
+**Migration**: Run the migration: `drizzle/0001_remove_fetch_interval.sql`
+
+### 3. Fixed Reddit Link Previews (✅ Completed)
+
+**Problem**: Reddit blocks regular HTTP scraping with aggressive bot detection, returning login/blocked pages instead of actual content with Open Graph tags. This caused bot posts with Reddit URLs to have NULL metadata for title, description, and image.
+
+**Solution**: Implemented Reddit-specific handling using their oEmbed API (`https://www.reddit.com/oembed`), which is designed for embedding and doesn't require authentication.
+
+**Benefits**:
+- Reddit link previews now work reliably
+- Extracts title, description (subreddit/author), and thumbnail when available
+- Falls back gracefully if oEmbed fails
+- Other sites continue to use standard OG tag scraping
+
+**Files Modified**:
+- `src/lib/bots/posting.ts` - Added `isRedditUrl()` and `fetchRedditPreview()` functions
+- `src/app/api/media/preview/route.ts` - Same Reddit-specific handling for the preview API
+
+## Implementation Details
+
+### Encryption Changes
+
+**File**: `src/lib/bots/encryption.ts`
+
+```typescript
+// Before
+function getEncryptionKey(): Buffer {
+  const keyEnv = process.env.BOT_ENCRYPTION_KEY;
+  // ... complex base64/hex decoding logic
+}
+
+// After
+function getEncryptionKey(): Buffer {
+  const keyEnv = process.env.AUTH_SECRET;
+  if (!keyEnv) {
+    throw new Error('AUTH_SECRET environment variable is not set');
+  }
+  // Create a 32-byte key from AUTH_SECRET using SHA-256
+  return crypto.createHash('sha256').update(keyEnv).digest();
+}
+```
+
+### Content Fetching Changes
+
+**File**: `src/lib/bots/posting.ts`
+
+The `triggerPost` function now automatically fetches content before posting:
+
+```typescript
+// Auto-fetch content from sources before posting
+const activeSources = await getActiveSourcesByBot(botId);
+if (activeSources.length > 0) {
+  const fetchResults = await Promise.allSettled(
+    activeSources.map(source => 
+      fetchContentWithRetry(source.id, 2, { maxItems: 10, timeout: 15000 })
+    )
+  );
+}
+```
+
+### Database Schema Changes
+
+**File**: `src/db/schema.ts`
+
+Removed:
+```typescript
+fetchIntervalMinutes: integer('fetch_interval_minutes').default(30).notNull(),
+```
+
+## Testing
+
+After these changes, test:
+
+1. ✅ Bot creation with API key encryption
+2. ✅ Manual post triggering (should auto-fetch content)
+3. ✅ Reddit content sources
+4. ✅ RSS content sources
+5. ✅ News API content sources
+
+## Next Steps
+
+Consider:
+- Add UI feedback showing when content is being fetched
+- Add content preview before posting
+- Add ability to regenerate post content if user doesn't like it
+- Add scheduling for automatic posts (cron-based)