# Web Analytics Overview

Karmaflow.ai's Web Analytics module is a full GA4-class analytics platform plus an AI-assisted tag manager, built into the same backend as the CRM. It ships events from customer websites into BigQuery for analytical queries, into per-tenant MongoDB collections for sub-100 ms dashboard loads, and through a Gemini-backed AI loop that proposes tag wiring and named conversions automatically.

## What the Module Does

- **Collects events** from customer websites via a hand-written IIFE tracker (`/p/:siteKey/k.js`, ~30 KB minified).
- **Stores raw events** in a single shared BigQuery dataset (`karmaflow_analytics`), partitioned by day and clustered by `(tenant_id, site_id, event_name)`.
- **Serves dashboards** from Mongo rollups (hourly + daily) so reads stay under 100 ms p95. BigQuery is only hit for ad-hoc Explore queries.
- **Runs a Tag Manager** with Tags / Triggers / Variables stored per tenant in Mongo, compiled into a versioned container JSON file served alongside the tracker.
- **Proposes AI suggestions** on two paths: a background job that mines the last hour of events per site, and a synchronous element picker the admin can click on the customer site.

## Architecture at a Glance

```
Customer site
  └── snippet ──► GET /p/:siteKey/k.js (cached 5 min, CORP: cross-origin)
       │         GET /p/:siteKey/container.json (latest published tag config)
       │
       └── batches POST /collect/:siteKey  (CORS, anonymous, rate-limited)
            │
            ├── express.json(text/plain) — dodges sendBeacon preflight
            ├── validateBatch (zod schema)
            ├── applySiteGuards (origin allow-list, bot UA filter, DNT/GPC)
            ├── toBigQueryRow → BigQuery streaming insert  ──► events
            ├── realtimeService.appendToBuffer            ──► Mongo capped buffer
            └── eventDefinitionMatcher.deriveBatch        ──► derived rows

Tick jobs (cron via x-tick-secret header):
  /api/analytics/tick           → Mongo rollup hourly / daily
  /api/analytics/identity-tick  → identity_map MERGE from identify events
  /api/analytics/ai-suggest-tick → Gemini live-stream suggestion sweep

Admin UI:
  /api/analytics/sites/...      → tenant-scoped CRUD + queries
```

## Key Modules and Where They Live

| Concern | Path |
| --- | --- |
| Tracker source | `public/js/k.js` |
| Snippet + container delivery | `src/routes/cdn/snippetRoutes.js` |
| Event ingest endpoint | `src/routes/api/collectRoutes.js` |
| Server-side event ingest | `src/routes/api/serverEventRoutes.js` |
| Tenant dashboard endpoints | `src/routes/analytics/*Routes.js` |
| Tag Manager CRUD | `src/routes/analytics/tagManagerRoutes.js` |
| Tick endpoints (cron) | `src/routes/analytics/analyticsTickRoutes.js` |
| BigQuery wrapper | `src/services/analytics/bigQueryService.js` |
| Collector logic | `src/services/analytics/collectorService.js` |
| Tag compiler | `src/services/analytics/containerCompilerService.js` |
| Workspace forking | `src/services/analytics/workspaceService.js` |
| AI suggestions | `src/services/analytics/aiSuggestionsService.js` |
| Identity stitching | `src/services/analytics/identityStitchingService.js` |
| Fleet status | `src/services/analytics/fleetStatusService.js` |

## What's Different from GA4

| Pain point in GA4 | What we do instead |
| --- | --- |
| 12–48 h data freshness | BigQuery streaming insert + Mongo capped buffer ⇒ ~5 s end-to-end |
| 14-month retention cap | No cap: events are kept indefinitely until you set a window (account-wide or per site) and turn removal on |
| 90-day lookback cap on funnels/cohorts | No lookback limit — limited only by retention |
| Sampling on large queries | Never sample; queries always read every row |
| Easy ad-blocker fingerprinting | First-party CNAME proxy supported (`CustomDomain` model) |
| Replaced bounce rate with "engagement rate" | `sessions` view exposes both `is_bounce` and `is_engaged` with clean rules |
| Undercounts ecom revenue | Server-side events path for canonical purchase data |
| No native funnel-drop-off or cohort retention | First-class endpoints + UI for both |

## What's Different from GTM

- Tags / Triggers / Variables are Mongo documents instead of JSON in a vendor console, so every change is auditable in your own DB.
- The container is compiled to a single static JSON file (`/p/:siteKey/container.json`) and served with HTTP caching — no opaque "publish" step from a black-box editor.
- Multi-draft **Workspaces** let multiple teammates edit in parallel without blocking each other. Only `Main` is what the tracker executes.
- **Versions** snapshot every publish; rollback is restoring `Site.publishedContainerVersion`.
- **AI suggestions** look at real event data and propose tag configurations ranked by confidence — eliminates most of the "wire this selector to that event" work.

## Reading the Rest of These Docs

If you're setting up a site for the first time, start with **Installing the Tracker**. If you're trying to figure out what gets collected and what you can call from JavaScript, see **Auto-Collected & Custom Events**. For tag-manager work, the **Tag Manager** doc covers Tags / Triggers / Variables / Workspaces; the **AI Element Picker** doc is for the visual point-and-click flow. For the data side, **Funnels & Cohorts** and **BigQuery Schema Reference** are the deepest dives.