Intel — Data Flow Diagram ========================== Last updated: 2026-07-12 Three layers: COLLECTION (ingest) → STORAGE (BQ + PG) → PROCESSING (analysis) → PRESENTATION (API, UI, MCP, /cc agent). ┌─────────────────────────────────────────────────────────────────────────────┐ │ ① COLLECTION (Cloud Run Jobs) │ ├─────────────────────────────────────────────────────────────────────────────┤ │ │ │ ┌──────────────┐ ┌──────────────┐ ┌──────────────┐ ┌────────────────┐ │ │ │ XPOZ API │ │ScrapeCreators│ │ScrapeCreators│ │ official X API │ │ │ │ (TW + IG) │ │ (TikTok) │ │ (Facebook) │ │ (col:x-api / │ │ │ │ │ │ │ │ │ │ kashaviya) │ │ │ └──────┬───────┘ └──────┬───────┘ └──────┬───────┘ └────────┬───────┘ │ │ │ │ │ │ │ │ ▼ ▼ ▼ ▼ │ │ ┌──────────────┐ ┌──────────────┐ ┌──────────────┐ ┌────────────────┐ │ │ │ ingest-tw │ │ingest-tiktok │ │ ingest-fb │ │ ingest-x-api │ │ │ │ (hourly) │ │ (daily) │ │ (daily) + │ │ (daily 13:00, │ │ │ │+ ingest-ig │ │ │ │ ingest-fb- │ │ + Fri 05:00) │ │ │ │ (daily,seq) │ │ │ │ publishers │ │ │ │ │ └──────┬───────┘ └──────┬───────┘ └──────┬───────┘ └────────┬───────┘ │ │ │ │ │ │ │ │ ┌──────────────┐ ┌──────────────┐ ┌──────────────┐ ┌────────────────┐ │ │ │ Drive (TV) │ │ Online news │ │ Drive (KB) │ │ Telegram │ │ │ └──────┬───────┘ └──────┬───────┘ └──────┬───────┘ └────────┬───────┘ │ │ ▼ ▼ ▼ ▼ │ │ ┌──────────────┐ ┌──────────────┐ ┌──────────────┐ ┌────────────────┐ │ │ │ ingest-tv │ │ingest-online │ │ ingest-kb │ │ingest-telegram │ │ │ │ (daily) │ │ -papers (2h) │ │ (daily) │ │ │ │ │ └──────┬───────┘ └──────┬───────┘ └──────┬───────┘ └────────┬───────┘ │ │ │ │ │ │ │ │ ┌──────────────────────────────────────────────────────────────────────┐ │ │ │ ingest-gly (Galei Yisrael radio, daily) — runs in parallel │ │ │ └───────────────────────────────────┬──────────────────────────────────┘ │ │ │ │ │ │ │ │ │ └─────────┬───────┴──────────┴──────────┴───────────┬───────┘ │ │ │ │ │ │ Separate "tohar" collection (col:3x-tagged POIs, twice daily 08:00 & 20:00, │ │ via the `tohar-collection` workflow, NOT the above jobs): │ │ ingest-x-api-tohar (official X API) + ingest-fb-tohar (ScrapeCreators) │ │ ──► twitter_tweets, fb_pages/profiles/posts │ │ │ │ Pixel (disguised-venue detection, ScrapeCreators, daily 06:00): │ │ ingest-pixel ──► pixel_venues, pixel_posts, pixel_venue_snapshots, │ │ pixel_lean_daily, pixel_flip_state, pixel_coordination │ │ │ └───────────────────┼─────────────────────────────────────────┼────────────────┘ │ │ │ ▼ ▼ ▼ ┌─────────────────────────────────────────────────────────────────────────────┐ │ ② STORAGE — BigQuery `intel-487218.datawarehouse` │ ├─────────────────────────────────────────────────────────────────────────────┤ │ │ │ Raw tables (per platform): twitter_*, tiktok_*, fb_pages/profiles/posts, │ │ tv_*, online_paper_*, mainstream_articles, kb_*, telegram_*, gly_*, │ │ pixel_* (venues/posts/snapshots/lean_daily/flip_state/coordination — │ │ investigation targets, kept separate from fb_pages/fb_posts) │ │ Audit tables: _etl_runs, _etl_run_errors │ │ Fetch tracking: _posts_fetches (delta windows) │ │ │ │ ──── Reads ────► ③ Analysis ──── Writes ────► │ │ │ │ Analysis tables: narratives, mention_daily, narrative_sources, │ │ subjects_daily, subjects_sources, emerging_terms, │ │ dashboard_summary, brief_correlation, post_correlation │ │ PG analysis results: events, post_alignment, weekly_briefs, daily_brief │ │ │ └─────────────────────────────────────────────────────────────────────────────┘ │ ▼ ┌─────────────────────────────────────────────────────────────────────────────┐ │ ③ PROCESSING — Analysis pipelines (Cloud Run Jobs) │ ├─────────────────────────────────────────────────────────────────────────────┤ │ │ │ ┌──────────────────────┐ ┌──────────────────────┐ │ │ │ analysis │ │ analysis-subjects │ │ │ │ (narratives, │ │ (subject extraction │ │ │ │ meta-narratives, │ │ + alignment) │ │ │ │ embeddings) │ │ │ │ │ └──────────┬───────────┘ └──────────┬───────────┘ │ │ │ │ │ │ └──────────┬───────────────┘ │ │ ▼ │ │ ┌──────────────────────────────────────────────────────────┐ │ │ │ messaging-funnel-workflow │ │ │ │ collection → ingest-fb-publishers → alignment │ │ │ │ (Twitter + Facebook posts → Gemini judge → BQ) │ │ │ └────────────────────────┬─────────────────────────────────┘ │ │ ▼ │ │ ┌────────────────────┐ ┌───────────────────┐ ┌────────────────────┐ │ │ │ events-pipeline │ │ weekly-brief │ │ messaging-briefs │ │ │ │ (every 2h) │ │ (daily) │ │ (daily) │ │ │ └────────────────────┘ └───────────────────┘ └────────────────────┘ │ │ ┌───────────────────┐ │ │ │ daily-brief │ │ │ │ (daily) │ │ │ └───────────────────┘ │ │ │ └─────────────────────────────────────────────────────────────────────────────┘ │ ▼ ┌─────────────────────────────────────────────────────────────────────────────┐ │ ④ PRESENTATION — Backend API + Frontend │ ├─────────────────────────────────────────────────────────────────────────────┤ │ │ │ FastAPI backend (Render Web Service) │ │ ├─ /api/v1/sources/... ──► /sources page (overview, drill-down) │ │ ├─ /api/v1/bq-cache/... ──► /cc Twitter cache (read + writeback) │ │ ├─ /api/v1/fb-cache/... ──► /cc Facebook cache (read + writeback) │ │ ├─ /api/v1/cc/... ──► /cc Claude Code session orchestrator │ │ ├─ /api/v1/daily-brief/... ──► daily brief (per audience, PDF/Slack) │ │ ├─ /api/v1/events/, papers/, narratives/, subjects/, ... (read API) │ │ └─ /mcp ──► MCP Intel server (~14 tool modules: twitter, tiktok, │ │ facebook, instagram, tv, gly, kb, papers, narratives, │ │ subjects, events, briefs, sources, domain_knowledge, …) │ │ │ │ PostgreSQL (Render managed) │ │ ├─ poi (source-of-truth POIs to track) │ │ ├─ publisher_outlet (handles per platform — tw, tt, fb, ig) │ │ ├─ users / sessions / roles (auth) │ │ ├─ domain_knowledge, briefs metadata │ │ └─ post_alignment (alignment results) │ │ │ │ React UI (Render Static Site) │ │ ├─ /sources — platform tiles (real brand icons via │ │ │ react-icons + lucide), drill-down, timelines │ │ ├─ /cc — Claude Code chat (SSE streaming) │ │ ├─ /daily-brief, /weekly-brief, /events, /narratives, /subjects, … │ │ └─ /admin/... — admin tools (users, domain knowledge, …) │ │ │ └─────────────────────────────────────────────────────────────────────────────┘ ▲ │ ┌─────────────────────────────────────────────────────────────────────────────┐ │ ⑤ /cc agent (Claude Code Web UI) — interactive │ ├─────────────────────────────────────────────────────────────────────────────┤ │ │ │ User → /cc UI → Backend `cc_service` → ClaudeSDKClient (Anthropic API) │ │ │ │ │ ├──► MCP Intel server (tweets, fb, papers, …) │ │ ├──► XPOZ MCP (live Twitter retrieval) │ │ ├──► /api/v1/bq-cache (Twitter cache + writeback) │ │ └──► /api/v1/fb-cache (Facebook cache + writeback) │ │ │ └─────────────────────────────────────────────────────────────────────────────┘ ═══════════════════════════════════════════════════════════════════════════════ PIPELINE A: Publishers / Messaging-funnel ═══════════════════════════════════════════════════════════════════════════════ Goal: judge whether each publisher post aligns with the daily messaging briefs. ┌─────────────────────────────────────────────────────────────────────────────┐ │ PostgreSQL — publisher_outlet table │ │ (manually curated in /messaging/outlets admin UI) │ │ │ │ id │ publisher_id │ platform │ handle │ collection_cadence │ │ ── │ ──────────── │ ──────── │ ─────────────── │ ────────────────── │ │ .. │ │ twitter │ omer_nahmany │ daily │ │ .. │ │ tiktok │ kalpinoded │ daily │ │ .. │ │ facebook │ lior.schleien │ daily │ └────────────────┬───────────────────────────┬────────────────────────────────┘ │ │ │ reads handles by platform │ ▼ ▼ ┌──────────────────────────────────┐ ┌────────────────────────────────────┐ │ ingest-messaging-briefs │ │ Workflow: messaging-funnel │ │ (daily 06:00 IL) │ │ (daily 07:30 IL) │ │ │ │ │ │ Drive (docx) ──► Gemini parser │ │ Step 1: messaging-funnel- │ │ │ │ │ collection (parallel) │ │ ▼ │ │ ─ _run_twitter (xpoz, radius=0) │ │ PG: message_brief, │ │ ─ _run_tiktok (ScrapeCreators) │ │ message_narrative, │ │ ─ _run_telegram │ │ message_subtitle, │ │ → BQ: twitter_tweets, tiktok_*, │ │ message_item │ │ telegram_messages │ └──────────────┬───────────────────┘ │ │ │ │ Step 2: ingest-fb-publishers │ │ candidate units │ ─ FbEtlRunner --source publishers│ │ (briefs) │ (ScrapeCreators) │ │ │ → BQ: fb_pages, fb_profiles, │ │ │ fb_posts │ │ │ │ │ │ Step 3: messaging-funnel- │ │ │ alignment │ │ │ ─ Read posts from BQ for active │ │ │ outlets, per platform │ │ │ (twitter_tweets, fb_posts) │ │ │ ─ Embed (Gemini) │ │ │ ─ Retrieve top-K matching units │ │ │ ─ Judge alignment (Gemini judge) │ └──────────────────────► │ → PG: post_alignment │ └────────────────────┬───────────────┘ │ ▼ ┌─────────────────────────────────────┐ │ UI: │ │ /messaging/outlets — publisher │ │ list + per-outlet detail with │ │ last-crawled-at, 7d histogram │ │ /messaging-funnel — daily │ │ scorecard, per-outlet timeline │ │ /admin/messaging-alignment — │ │ QA queue + operator overrides │ │ /admin/messaging-briefs — │ │ brief list + parser status │ └─────────────────────────────────────┘ Notes: ─ Alignment runs for Twitter, Facebook, and TikTok. TikTok text is the caption (description) with transcript as fallback when caption is empty. ─ Telegram outlets are collected in Step 1 but not yet judged for alignment. ─ Instagram is collected via ingest-ig (XPOZ, runs sequentially after ingest-tw daily) but Instagram outlets in messaging-funnel are not yet judged for alignment. ─ TikTok and Facebook collection both use ScrapeCreators (migrated from Apify) — see backend/app/services/scrapecreators_{tiktok,fb}_client.py. ─ All collection (twitter, tiktok, facebook, telegram) happens inside messaging-funnel-workflow — there is no separate `ingest-publisher-outlets` job. publisher_outlet_etl.py is a shared library imported by messaging-funnel-collection. ═══════════════════════════════════════════════════════════════════════════════ PIPELINE B: Events (online media) ═══════════════════════════════════════════════════════════════════════════════ Goal: cluster mainstream news articles into events, score importance, and correlate to subjects/narratives. ┌─────────────────────────────────────────────────────────────────────────────┐ │ Online news sources — mainstream Israeli papers + wires (events' │ │ clustering input; distinct taxonomy from the Online papers ingest above) │ │ Outlet registry in backend/app/models/online_paper.py │ └────────────────────────────────────┬────────────────────────────────────────┘ │ scraped hourly ▼ ┌─────────────────────────────────────────────────────────────────────────────┐ │ Workflow: events-pipeline (every 2h, 06–22 Asia/Jerusalem) │ ├─────────────────────────────────────────────────────────────────────────────┤ │ │ │ Step 1: ingest-mainstream-papers │ │ backend/mainstream_papers_etl.py (separate from online_paper_etl.py — │ │ different taxonomy + downstream consumer) │ │ → BQ: mainstream_articles (article text, url, published_at, │ │ source_slug, ...) │ │ → BQ: mainstream_etl_runs / mainstream_etl_run_errors │ │ │ │ Step 2: events-pipeline job │ │ backend/events_pipeline.py / app/services/events_service.py │ │ │ │ A. Pull recent articles from mainstream_articles (3h hourly / │ │ 7d daily rescore window) │ │ B. Cluster (TF-IDF + source-weight aware) │ │ C. Gemini extraction per cluster │ │ dedup via cluster_fingerprint = sha1(sorted article_ids) │ │ identical clusters skip Gemini │ │ D. Amplification (counts from social + TV around each event's window; │ │ right-paper echo counted from online_paper_articles) │ │ E. Importance scoring │ │ F. Correlate to subjects + narratives via cached embedding centroids │ │ (narrative_centroids / subject_centroids) │ │ G. Upsert PG: events, event_sources, event_feedback │ └────────────────────────────────────┬────────────────────────────────────────┘ │ ▼ ┌─────────────────────────────────────────────────────────────────────────────┐ │ UI: │ │ /events — timeline + calendar + drawer detail │ │ (camp/category streams, year heatmap, month calendar) │ │ /admin/events — event list with inline source-health editing, │ │ feedback actions │ │ /admin/events-dashboard — volume + importance distribution + source │ │ coverage │ │ │ │ MCP: │ │ list_events / get_event / search_events / get_events_calendar / │ │ get_events_camp_stream / get_events_category_stream │ └─────────────────────────────────────────────────────────────────────────────┘ Key paths ========= POI handles in PG → daily ETL Cloud Run Jobs → BigQuery raw tables │ ▼ analysis & alignment jobs │ ▼ BigQuery analysis tables + PG `post_alignment` │ ▼ Backend API ─► Frontend UI / MCP / /cc Cadences ======== hourly : ingest-tw, mainstream-papers (within events-pipeline) every 2h: events-pipeline (mainstream-papers → events analysis) daily : collection-analysis (13:00 IL, extra Friday 05:00): ingest-tw (+ ingest-ig sequential), ingest-x-api (col:x-api/ kashaviya POIs, official X API), ingest-gly, ingest-tv, ingest-online-papers, ingest-kb, ingest-tiktok, ingest-fb, ingest-telegram → analysis → events-pipeline → weekly-brief + daily-brief (parallel) messaging-funnel-workflow (07:30 IL — collection + fb + alignment) messaging-briefs (06:00 IL) ingest-pixel (06:00 IL — disguised-venue collection) events-pipeline-daily (04:00 IL — 7d rescore window) ingest-fb (publishers — within messaging-funnel-workflow) twice daily (08:00 & 20:00 IL): tohar-collection workflow (ingest-x-api-tohar + ingest-fb-tohar, col:3x-tagged POIs — separate from the daily ingest-x-api / ingest-fb jobs above) Schedulers (Cloud Scheduler, all in us-west1, all Asia/Jerusalem) ================================================================ collection-analysis-trigger → Workflow collection-analysis (daily 13:00 + extra Friday 05:00; includes ingest-x-api) events-pipeline-hourly-trigger → Workflow events-pipeline events-pipeline-daily-trigger → Workflow events-pipeline-daily (daily 04:00) messaging-funnel-workflow-daily-trigger → Workflow messaging-funnel-workflow messaging-brief-etl-daily-trigger → Job ingest-messaging-briefs ingest-tw-trigger → Job ingest-tw (hourly) pixel-ingest-daily-trigger → Job ingest-pixel (daily 06:00) tohar-collection-trigger-morning → Workflow tohar-collection (08:00) tohar-collection-trigger-evening → Workflow tohar-collection (20:00)