Home Docs | Sign In
ChangeWatch / Technical Documentation
SYSTEM SPECIFICATION

Architecture & Features Guide

Complete reference explaining how ChangeWatch captures web pages, neutralizes noise, protects against SSRF attacks, executes diffing, and operates across local, containerized, and Vercel serverless cloud runtimes.

#overview #normalization #visual-regions #diff-engine #numeric-tracking #ssrf-defense #vercel-serverless #storage-db

1. Why ChangeWatch?

Standard website monitors merely check HTTP 200 OK status codes. Conventional web scrapers alert on every single request because modern web pages fluctuate incessantly due to:

  • Relative and absolute timestamps (e.g. "updated 4 minutes ago", live UTC clocks).
  • Real-time counters and vanity metrics (e.g. "1,429 views", "98 likes").
  • Session tokens, CSRF nonces, and dynamic advertisement scripts.
  • A/B testing wrappers and localized banners.

ChangeWatch solves this through deterministic noise suppression, targeted CSS subtrees, and semantic change verification. Only qualifying modifications trigger alert dispatches, preventing notification exhaustion.

2. The Noise Normalization Pipeline

Before computing a cryptographic SHA-256 fingerprint, raw HTML and extracted text pass through several deterministic filtering passes in app/services/normalizer.py:

Timestamp Neutralization

Matches ISO 8601 strings, UNIX epoch dates, and natural phrases (e.g., "5 mins ago", "just now") replacing them with invariant placeholders ([volatile-timestamp]).

Dynamic Counter Filtering

Strips volatile social engagement metrics and follower numbers (e.g., "12,304 reads", "45 comments") into [volatile-counter] tokens.

Whitespace Normalization

Collapses non-breaking spaces ( ), extraneous line breaks, and indentation into uniform single-line paragraph structures.

Sensitivity Thresholds

Users can tune thresholds (low, medium, high) measuring edit distance ratios so trivial punctuation or single-letter spelling fixes do not trigger alerts.

The Store-on-Change Invariant: When a scheduled check detects that the normalized content matches the existing baseline, ChangeWatch strictly records lightweight run telemetry (CheckRun status, duration, timestamp). Full screenshot files and DOM snapshots are saved only for the initial baseline and qualified change events, preserving disk and S3 bandwidth.

3. Visual Region Studio

ChangeWatch features a point-and-click visual region picker (/monitors/{id}/regions):

  • Sanitized Sandboxed Preview: Renders target HTML in an isolated <iframe sandbox> with scripts, external fetches, and event handlers neutralized.
  • Element Hover & Inspector: Injected JavaScript highlights DOM elements on hover, displaying floating selector tags.
  • Bidirectional postMessage: Clicking an element calculates optimal, resilient CSS selectors (IDs, distinct classes, and structural paths) and transmits them to the parent ChangeWatch studio.
  • Dual Mode Configuration:
    • Watch Regions: Narrow the capture scope exclusively to designated blocks (e.g., #pricing-table or .release-notes).
    • Ignore Regions: Excise distracting blocks like cookie banners, comment widgets, or real-time clocks prior to comparison.

4. Multi-Mode Version Comparison Studio

When a change is detected, ChangeWatch captures the new state as a distinct numbered version (Version). The Comparison Studio provides four specialized analysis lenses:

1. Line-by-Line Text Diff

Accessible additions (+ green) and deletions (- red) with line numbering and diff summaries.

2. Side-by-Side Dual Pane

Split viewport comparing Baseline vs Current screenshots with locked synchronized vertical scrolling.

3. Draggable Swipe Slider

Interactive before/after slider with drag handle, keyboard arrow navigation, and touch swipe gestures.

4. Computer Vision Pixel Mask

Calculated via Pillow image analysis, rendering a high-contrast red overlay revealing layout and styling shifts.

5. Numeric Tracking & Keyword Rules

Numeric Value Extraction & Sparklines

Target specific numeric elements (e.g. .stock-price, #inventory-count). ChangeWatch automatically parses currencies ($, €, £, ₹) and metrics into structured history records, plotting real-time SVG sparklines with High, Low, and Current badges.

Keyword-Only Alert Filtering

Specify trigger keywords (e.g. "out of stock", "restocked", "price drop"). If content changes, ChangeWatch archives the new version for historical accuracy, but suppresses notification dispatch unless one of your specified keywords appears in the added lines.

6. Two-Tier SSRF Defense Fortress

Allowing automated browser agents to visit user-supplied URLs poses significant Server-Side Request Forgery (SSRF) risks. ChangeWatch implements defense-in-depth:

Tier 1: Pre-Navigation DNS & IP Validation
Every target URL is checked before navigation. ChangeWatch resolves domain DNS and strictly blocks RFC 1918 private IPv4 subnets (10.0.0.0/8, 172.16.0.0/12, 192.168.0.0/16), loopbacks (127.0.0.1, ::1), broadcast addresses, and cloud instance metadata services (169.254.169.254, metadata.google.internal). Embedded credentials (user:pass@host) are rejected.
Tier 2: Playwright Subresource Route Interception
A malicious public website could embed an internal resource (e.g. <img src="http://169.254.169.254/latest/meta-data/">) or issue an HTTP 302 redirect to an internal IP. ChangeWatch attaches a network interceptor (page.route("**/*")) inside Chromium that intercepts every outbound subresource fetch, validating destinations and immediately aborting suspicious traffic.

7. Vercel Serverless & Remote Chromium Architecture

ChangeWatch's FastAPI web application is designed to run seamlessly on Vercel as a Python serverless function. Because serverless runtimes are short-lived, have read-only temporary filesystems, and cannot launch Chromium locally or run perpetual scheduler loops, the architecture bifurcates cleanly:

OPTION A (RECOMMENDED)

Managed Browserbase CDP

Vercel connects to Browserbase's managed browser cloud over Chrome DevTools Protocol (CDP). Configure BROWSERBASE_API_KEY and BROWSERBASE_PROJECT_ID in Vercel project settings. The web app acquires short-lived sessions, performs captures, applies SSRF filters, and releases the session.

OPTION B

Self-Hosted Browser Worker

Deploy the repository Docker container with python -m uvicorn app.worker_api:app --port 8001. Set BROWSER_WORKER_URL and BROWSER_WORKER_TOKEN in Vercel. The worker exposes authenticated /internal/capture to execute Playwright Chromium remotely.

Automated Background Scheduling on Serverless

Since Vercel cannot run background loop daemons:

  • Immediate First Check: When a user creates a new monitor on Vercel, the first baseline check executes synchronously inline so the monitor is never stuck in a pending queue.
  • GitHub Actions Cron Runner: The included workflow (.github/workflows/monitor-scheduler.yml) calls the protected endpoint /internal/worker/tick every 5 minutes using the shared secret CRON_SECRET. Each invocation schedules due monitors and executes queued checks.
  • High-Volume Daemon Mode: For enterprise scale or sub-minute intervals, run workers/scheduler_process.py and workers/check_worker.py as persistent container processes.

8. Database & Object Storage Configuration

PostgreSQL & Neon Pooling

ChangeWatch supports Neon pooled connection strings (DATABASE_URL) for fast serverless execution. Alembic migrations are applied via unpooled connections (NEON_DATABASE_URL) in GitHub Actions to prevent schema lock contention.

S3-Compatible Artifact Storage

Configure ARTIFACT_STORAGE_BACKEND=s3 with S3_BUCKET, S3_REGION, and optional S3_ENDPOINT_URL (supports AWS S3, Cloudflare R2, MinIO). If S3 credentials are not configured, text comparison continues seamlessly while screenshot storage is gracefully omitted.

Ready to start tracking?

Deploy an autonomous monitor in 30 seconds with noise suppression.

Contact Support Get Started →