Web Analytics Overview

KarmaFlow's Web Analytics module is a full GA4-class analytics platform plus an AI-assisted tag manager, built into the same backend as the CRM. It ships events from customer websites into BigQuery for analytical queries, into per-tenant MongoDB collections for sub-100 ms dashboard loads, and through a Gemini-backed AI loop that proposes tag wiring and named conversions automatically.

What the Module Does

• Collects events from customer websites via a hand-written IIFE tracker (/p/:siteKey/k.js, ~30 KB minified).
• Stores raw events in a single shared BigQuery dataset (karmaflow_analytics), partitioned by day and clustered by (tenant_id, site_id, event_name).
• Serves dashboards from Mongo rollups (hourly + daily) so reads stay under 100 ms p95. BigQuery is only hit for ad-hoc Explore queries.
• Runs a Tag Manager with Tags / Triggers / Variables stored per tenant in Mongo, compiled into a versioned container JSON file served alongside the tracker.
• Proposes AI suggestions on two paths: a background job that mines the last hour of events per site, and a synchronous element picker the admin can click on the customer site.

Architecture at a Glance

Customer site
  └── snippet ──► GET /p/:siteKey/k.js (cached 5 min, CORP: cross-origin)
       │         GET /p/:siteKey/container.json (latest published tag config)
       │
       └── batches POST /collect/:siteKey  (CORS, anonymous, rate-limited)
            │
            ├── express.json(text/plain) — dodges sendBeacon preflight
            ├── validateBatch (zod schema)
            ├── applySiteGuards (origin allow-list, bot UA filter, DNT/GPC)
            ├── toBigQueryRow → BigQuery streaming insert  ──► events
            ├── realtimeService.appendToBuffer            ──► Mongo capped buffer
            └── eventDefinitionMatcher.deriveBatch        ──► derived rows

Tick jobs (cron via x-tick-secret header):
  /api/analytics/tick           → Mongo rollup hourly / daily
  /api/analytics/identity-tick  → identity_map MERGE from identify events
  /api/analytics/ai-suggest-tick → Gemini live-stream suggestion sweep

Admin UI:
  /api/analytics/sites/...      → tenant-scoped CRUD + queries

Key Modules and Where They Live

Concern	Path
Tracker source	public/js/k.js
Snippet + container delivery	src/routes/cdn/snippetRoutes.js
Event ingest endpoint	src/routes/api/collectRoutes.js
Server-side event ingest	src/routes/api/serverEventRoutes.js
Tenant dashboard endpoints	src/routes/analytics/*Routes.js
Tag Manager CRUD	src/routes/analytics/tagManagerRoutes.js
Tick endpoints (cron)	src/routes/analytics/analyticsTickRoutes.js
BigQuery wrapper	src/services/analytics/bigQueryService.js
Collector logic	src/services/analytics/collectorService.js
Tag compiler	src/services/analytics/containerCompilerService.js
Workspace forking	src/services/analytics/workspaceService.js
AI suggestions	src/services/analytics/aiSuggestionsService.js
Identity stitching	src/services/analytics/identityStitchingService.js
Fleet status	src/services/analytics/fleetStatusService.js

What's Different from GA4

Pain point in GA4	What we do instead
12–48 h data freshness	BigQuery streaming insert + Mongo capped buffer ⇒ ~5 s end-to-end
14-month retention cap	Default 25 months (Tenant.analyticsRetentionDays = 760); null = unlimited
90-day lookback cap on funnels/cohorts	No lookback limit — limited only by retention
Sampling on large queries	Never sample; queries always read every row
Easy ad-blocker fingerprinting	First-party CNAME proxy supported (CustomDomain model)
Replaced bounce rate with "engagement rate"	sessions view exposes both is_bounce and is_engaged with clean rules
Undercounts ecom revenue	Server-side events path for canonical purchase data
No native funnel-drop-off or cohort retention	First-class endpoints + UI for both

What's Different from GTM

• Tags / Triggers / Variables are Mongo documents instead of JSON in a vendor console, so every change is auditable in your own DB.
• The container is compiled to a single static JSON file (/p/:siteKey/container.json) and served with HTTP caching — no opaque "publish" step from a black-box editor.
• Multi-draft Workspaces let multiple teammates edit in parallel without blocking each other. Only Main is what the tracker executes.
• Versions snapshot every publish; rollback is restoring Site.publishedContainerVersion.
• AI suggestions look at real event data and propose tag configurations ranked by confidence — eliminates most of the "wire this selector to that event" work.

Reading the Rest of These Docs

If you're setting up a site for the first time, start with Installing the Tracker. If you're trying to figure out what gets collected and what you can call from JavaScript, see Auto-Collected & Custom Events. For tag-manager work, the Tag Manager doc covers Tags / Triggers / Variables / Workspaces; the AI Element Picker doc is for the visual point-and-click flow. For the data side, Funnels & Cohorts and BigQuery Schema Reference are the deepest dives.