s-m-r-t

@happyvertical/smrt-messages

Email persistence with AI integration, templating, and delivery tracking.

v0.19.0EmailMessaging

Overview

smrt-messages provides TypeScript-first abstractions for managing email across multiple providers. It persists email data with rich metadata (threading, attachments, flags, labels) and synchronizes emails from remote servers with configurable options.

Installation

bash 
npm install @happyvertical/smrt-messages
# or
pnpm add @happyvertical/smrt-messages

Depends on @happyvertical/smrt-core, @happyvertical/email, and @happyvertical/sql.

Quick Start (5 Minutes)

1. Set Up Email Account

typescript 
import { EmailAccount, EmailAccountCollection } from '@happyvertical/smrt-messages';

const accounts = new EmailAccountCollection({ db: {...} });

// Create Gmail account
const account = new EmailAccount({
  name: 'Personal Gmail',
  email: 'user@gmail.com',
  providerType: 'gmail'
});

account.setSettings({
  host: 'imap.gmail.com',
  port: 993,
  secure: true,
  auth: {
    user: 'user@gmail.com',
    pass: 'app-specific-password'  // NOT your Gmail password
  }
});

await account.save();

2. Sync Emails

typescript 
// Perform initial sync
const result = await account.syncFrom({
  folders: ['INBOX', 'SENT'],
  fullSync: true,
  downloadAttachments: true,
  maxConcurrency: 5,
  onProgress: (stats) => {
    console.log(`Synced ${stats.processed}/${stats.total}`);
  }
});

console.log(`Downloaded: ${result.messagesDownloaded}`);
console.log(`Duration: ${result.duration}ms`);

3. Query Emails

typescript 
import { EmailCollection } from '@happyvertical/smrt-messages';

const emails = new EmailCollection({ db: {...} });

// Get unread emails
const unread = await emails.getUnread(account.id);

// Get recent emails
const recent = await emails.getRecent(20, account.id);

// Get emails with attachments
const withAttachments = await emails.getWithAttachments(account.id);

// Get thread conversation
const email = await emails.get(emailId);
const threadEmails = await email.getThreadEmails();

Core Concepts

Data Model

Four core entities form the email management system:

Email Model

typescript 
class Email extends SmrtObject {
  accountId: string           // Parent account
  folderId: string            // Current folder
  messageId: string           // RFC 822 Message-ID (unique)
  threadId: string | null     // Conversation grouping
  subject: string
  fromAddress: string
  fromName: string
  toAddresses: string         // JSON array
  ccAddresses: string         // JSON array
  bccAddresses: string        // JSON array
  replyTo: string | null
  date: Date | null
  textBody: string
  htmlBody: string
  isRead: boolean
  isFlagged: boolean
  isAnswered: boolean
  isDraft: boolean
  hasAttachments: boolean
  size: number                // Bytes
  labels: string              // JSON array (Gmail)
  flags: string               // JSON array (IMAP)
  headers: string             // JSON object
  inReplyTo: string | null    // Parent message-ID
  references: string | null   // Thread references
  rawMessage: string | null   // Full MIME message
}

EmailAccount Model

typescript 
class EmailAccount extends SmrtObject {
  name: string
  email: string
  providerType: 'smtp' | 'imap' | 'pop3' | 'gmail'
  settings: string            // JSON (encrypt in production)
  lastSyncAt: Date | null
  active: boolean

  // Methods
  async syncFrom(options): Promise<SyncResult>
  async createClient(): Promise<EmailClient>
  async getFolders(): Promise<EmailFolder[]>
  async getEmails(limit?): Promise<Email[]>
  async getUnreadCount(): Promise<number>
}

EmailFolder Model

typescript 
class EmailFolder extends SmrtObject {
  accountId: string
  name: string
  path: string
  delimiter: string
  specialUse: string | null   // 'Inbox', 'Sent', 'Drafts', etc.
  messageCount: number
  unreadCount: number
  subscribed: boolean

  // Helper methods
  isInbox(): boolean
  isSent(): boolean
  isDrafts(): boolean
  isSpam(): boolean
  isSystemFolder(): boolean
}

EmailAttachment Model

typescript 
class EmailAttachment extends SmrtObject {
  emailId: string
  filename: string
  contentType: string
  size: number
  contentDisposition: string  // 'attachment' | 'inline'
  contentId: string | null    // For inline images
  filePath: string | null     // External storage path

  // Helper methods
  isImage(): boolean
  isPdf(): boolean
  isInline(): boolean
  getExtension(): string
  getFormattedSize(): string
  async readContent(): Promise<Buffer | null>
}

Message Threading

Emails are grouped into conversations using threadId:

typescript 
// Get all emails in a thread
const email = await emails.get(emailId);
const thread = await email.getThreadEmails();

// Sort chronologically
const sorted = thread.sort((a, b) => {
  return (a.date?.getTime() || 0) - (b.date?.getTime() || 0);
});

console.log(`Thread has ${sorted.length} messages`);

Sync Engine

The syncFrom() method synchronizes emails from remote servers:

typescript 
const result = await account.syncFrom({
  folders: ['INBOX'],          // Specific folders or omit for all
  fullSync: false,             // true = re-download everything
  downloadAttachments: true,   // Save attachment files
  maxAttachmentSize: 10485760, // 10MB limit
  batchSize: 100,              // Messages per batch
  maxConcurrency: 5,           // Parallel connections
  since: new Date('2024-01-01'), // Incremental sync
  onProgress: (stats) => {
    // stats: { total, processed, current, folder }
  },
  onError: (error, message) => {
    // Handle sync errors
  }
});

// Result contains:
// - messagesDownloaded: number
// - attachmentsDownloaded: number
// - errors: Array<{ error, message }>
// - duration: number (ms)

API Reference

EmailCollection Methods

typescript 
// Queries
await emails.getByMessageId(accountId, messageId)
await emails.getByAccount(accountId)
await emails.getByFolder(folderId)
await emails.getByThread(threadId)
await emails.getUnread(accountId?)
await emails.getFlagged(accountId?)
await emails.getWithAttachments(accountId?)
await emails.getRecent(limit, accountId?)

// Counts
await emails.countByFolder(folderId)
await emails.countUnreadByFolder(folderId)
await emails.countByAccount(accountId)

// CRUD
await emails.list(options)
await emails.get(id)
await emails.create(data)
await emails.update(id, data)
await emails.delete(id)

Email Instance Methods

typescript 
// Status management
await email.markRead()
await email.markUnread()
await email.toggleFlagged()

// Relationships
await email.getAccount()
await email.getFolder()
await email.getAttachments()
await email.getThreadEmails()

// Address parsing
email.getToAddresses(): Array<{ name: string, address: string }>
email.getCcAddresses(): Array<{ name: string, address: string }>
email.getBccAddresses(): Array<{ name: string, address: string }>

// Metadata
email.getLabels(): string[]
email.setLabels(labels: string[]): void
email.getFlags(): string[]
email.setFlags(flags: string[]): void
email.getHeaders(): Record<string, string>
email.setHeaders(headers: Record): void

// Utilities
email.isUnread(): boolean
email.getPreview(maxLength: number): string

EmailAccountCollection Methods

typescript 
await accounts.getByEmail(email: string)
await accounts.getByProviderType(type: string)
await accounts.getActive()
await accounts.getInactive()
await accounts.getNeedingSync(maxAgeMinutes: number)
await accounts.search(query: string, filters?)

Real-World Examples

Example 1: Email Dashboard Widget

typescript 
// Show unread counts per folder
const account = await accounts.get(accountId);
const folders = await account.getFolders();

for (const folder of folders) {
  if (folder.isSystemFolder()) {
    console.log(`${folder.name}: ${folder.unreadCount} unread`);
  }
}

Example 2: Conversation Thread UI

typescript 
const email = await emails.get(emailId);
const thread = await email.getThreadEmails();

const sorted = thread.sort((a, b) => {
  return (a.date?.getTime() || 0) - (b.date?.getTime() || 0);
});

const conversation = await Promise.all(
  sorted.map(async (e) => ({
    from: e.fromName || e.fromAddress,
    subject: e.subject,
    date: e.date,
    preview: e.getPreview(100),
    isRead: e.isRead,
    attachments: await e.getAttachments()
  }))
);

console.log(`Thread with ${conversation.length} messages`);

Example 3: Attachment Processing

typescript 
// Process all PDF attachments
const withAttachments = await emails.getWithAttachments(accountId);

for (const email of withAttachments) {
  const attachments = await email.getAttachments();
  const pdfs = attachments.filter(a => a.isPdf());

  for (const pdf of pdfs) {
    console.log(`Processing: ${pdf.filename} (${pdf.getFormattedSize()})`);

    const content = await pdf.readContent();
    if (content) {
      // Process PDF content
      await processPdfData(content);
    }
  }
}

Example 4: Automated Email Archive

typescript 
// Archive emails older than 6 months
const sixMonthsAgo = new Date();
sixMonthsAgo.setMonth(sixMonthsAgo.getMonth() - 6);

const all = await emails.list({ where: { accountId } });
const older = all.filter(e => e.date && e.date < sixMonthsAgo);

for (const email of older) {
  // Mark as archived
  email.flags = JSON.stringify(['\\Archive']);
  await email.save();

  console.log(`Archived: ${email.subject}`);
}

Integration Patterns

With @happyvertical/email

EmailAccount creates provider clients for sync:

typescript 
const client = await account.createClient();
const folders = await client.listFolders();
const messages = await client.fetch({ folder: 'INBOX' });

With @happyvertical/ai

Email model inherits SmrtObject with AI capabilities:

typescript 
// AI-powered email summarization
const summary = await email.do('summarize this email thread');

// Sentiment analysis
const sentiment = await email.is('positive or negative');

With smrt-users

Link email accounts to user profiles:

typescript 
// User can have multiple email accounts
const userAccounts = await accounts.list({
  where: { userId: user.id, active: true }
});

// Permission-based access
if (user.can('view:emails')) {
  const emails = await emails.getByAccount(accountId);
}

With smrt-assets

Store attachments as assets:

typescript 
// Save attachment to asset system
const attachment = await attachments.get(attachmentId);
const content = await attachment.readContent();

const asset = await assets.create({
  filename: attachment.filename,
  contentType: attachment.contentType,
  size: attachment.size,
  content: content
});

Best Practices

DOs

  • Use app-specific passwords for Gmail/Outlook (not main password)
  • Encrypt settings JSON before storing in database
  • Implement incremental sync with since option
  • Handle sync errors gracefully with onError callback
  • Set reasonable batch sizes (100-500) to balance memory/speed
  • Cache folder lists in UI (they change infrequently)
  • Use thread IDs for conversation grouping
  • Parse address arrays with getToAddresses() helpers

DON'Ts

  • Don't store passwords in plaintext (always encrypt)
  • Don't assume all providers support same features
  • Don't download all attachments unconditionally
  • Don't skip threadId during sync
  • Don't modify JSON fields directly (use accessor methods)
  • Don't run concurrent syncs on same account
  • Don't ignore onError callbacks
  • Don't bulk-delete without backup

Common Issues and Troubleshooting

Issue: "Message already exists" after sync

Cause: Incremental sync skips existing messages

Solution: Use fullSync: true for complete re-sync

Issue: Attachment reads return null

Cause: File not found or file service not configured

Solution: Check attachment.filePath exists and file service is set up

Issue: ThreadId is empty after sync

Cause: Email provider didn't return threadId

Solution: Reconstruct threads using inReplyTo and message-ID headers

Issue: Sync takes too long

Cause: Large mailbox (10k+ emails) on first sync

Solution: Use smaller batchSize, higher maxConcurrency, or sync only recent emails with since

typescript 
await account.syncFrom({
  batchSize: 50,
  maxConcurrency: 10,
  since: new Date(Date.now() - 90 * 24 * 60 * 60 * 1000) // 90 days
});

Issue: Unread count stale after marking read

Cause: Folder counts not updated after email operations

Solution: Call folder.refreshCounts() after bulk operations

typescript 
await email.markRead();
const folder = await email.getFolder();
await folder?.refreshCounts();

Related Modules