Intel Data Platform Guide

Who this is for: analysts, researchers, and team members who want to use the data we collect — where it lives, how it gets there, when it refreshes, and how to query it. No code access required.

This file is the source of truth. The Markdown file docs/data/data_guide.md is the source; the online page at datadocs.hml.st is generated from it automatically by CI on every push to main. Edit here, open a PR — CI renders and publishes. See How this page is built.

Overview

Everything we collect — social media, TV, radio, online news — plus everything we derive from it — narratives, subjects, events, briefs — lives in two stores:

Rule of thumb: if you want to analyze content or trends, you want BigQuery. If you want the curated definitions and finished results the product shows, that's PostgreSQL — reachable via the app at hml.st or the API.

The data moves through four layers: Collection → Storage → Processing → Presentation. The full diagram is in docs/diagrams/data-flow.txt.

Architecture diagrams

Two diagrams live in docs/diagrams/: end-to-end data flow (Collection → Storage → Processing → Presentation, plus detailed Publishers/messaging-funnel and Events sub-pipelines) and deployment topology (Render vs GCP, schedulers, workflows, jobs, secrets, external APIs). Click either image to open the full-size SVG.

Refreshed 2026-08-13 to match the ScrapeCreators migration, the official X API / col:5x frequent-collection paths, and the Pixel pipeline. If this guide and the diagrams ever drift again, the tables below in this guide are the current source of truth — diagrams are refreshed on request, not on every change.

Ingestion (Collection)

Scheduled Cloud Run Jobs (region us-west1) pull from each source and write raw rows to BigQuery. POI handles to track come from Postgres (pois). No ingest job writes to Postgres — all raw data lands in BigQuery.

Each pipeline follows the same pattern: raw content tables, plus <platform>_etl_runs / _etl_run_errors (run audit) and <platform>_*_fetches (delta-window tracking). Raw tables upsert through a transient <table>_staging twin.

Live path, flag on: X's Activity API (XAA) pushes new posts as they land instead of waiting for ingest-x-api-5x. The x-activity-stream reader holds the single socket and publishes each NDJSON event to Pub/Sub (x-activity-events); it does not MERGE and does not transcribe. x-activity-worker instances MERGE into the shared twitter_tweets table (ingestion_source=x_activity) and then publish the mapped tweet to Pub/Sub topic twitter-live-tweets for other teams. Videos reuse the existing twitter-transcribe topic and service. Gated by Hamal ConfigCat xActivityStreamEnabled (default on; missing SDK / outage / missing flag => on) — dashboard Off is the kill-switch; do not set X_ACTIVITY_STREAM_ENABLED=true. Allowlist is POIs tagged camp:red or col:x-api (union) with a twitter_id. Hourly job x-activity-subscription-sync keeps XAA post.create subscriptions matched to that allowlist (ADR 0011). Paid polling via pois.x_api_since_id still runs and stays col:x-api only. Connection health is tracked in x_activity_stream_connections. See ADR 0012 and ADR 0008.

Processing (Analysis)

Analysis jobs read from BigQuery, load their catalogs (narratives, subjects, POIs, domain knowledge) from Postgres at startup, and write outputs back to both stores.

Most analysis uses Gemini for classification, extraction, embeddings, and Hebrew synthesis.

The D-Day Signs pipeline is a three-stage chain — candidate nomination (keyword match), judging (Gemini + grounding check), aggregation into scored events for the /dday/triage queue — documented in full in docs/dday_signs.md (content selection, the judge's grounding rule, event merge/corroboration, authority bands). This section and the tables below cover only where its data lives.

When the flows run

All schedules are Asia/Jerusalem; schedulers live in us-west1 (Cloud Scheduler → Workflows).

To check the last successful run of any pipeline, query its <platform>_etl_runs table in BigQuery, or use /admin/jobs in the app.

Getting notified when collection finishes (Pub/Sub)

Don't want to poll BigQuery or /admin/jobs? Both collection-analysis and collection publish a Pub/Sub event — with per-source success/failure counts — right after their collection phase finishes (Twitter video transcripts for that run included), before analysis and briefs even start.

• Topic: projects/intel-487218/topics/ingestion-completed
• Fires: once per workflow run — collection-analysis daily 12:00 (+ Fri 05:00), collection at 00:00, 08:00, 16:00 and 20:00, all Asia/Jerusalem (see When the flows run above)
• Attributes (filter subscriptions on these without decoding the message body): workflow (collection-analysis | collection), phase (always collection), status (success | partial_failure)

Message body (JSON, base64-encoded in the Pub/Sub data field — any client library decodes this automatically):

{
  "workflow": "collection-analysis",
  "status": "partial_failure",
  "timestamp": "2026-08-12T09:03:41Z",
  "summary": {
    "total_sources": 11,
    "sources_succeeded": 10,
    "sources_failed": 1,
    "items_succeeded": 4231,
    "items_failed": 7
  },
  "sources": {
    "tw": { "status": "success", "succeeded": 812, "failed": 0 },
    "kb": { "status": "failed", "succeeded": 0, "failed": 7 }
  },
  "failed_sources": ["kb"]
}

sources.<name>.succeeded/.failed are item-level counts (posts/documents processed vs. failed within that source's own run), not just whether the job process exited 0.

To subscribe, you need roles/pubsub.subscriber on the topic — ask a project developer to grant it to you or your service account (see For admins for the equivalent BigQuery grant pattern; same idea, roles/pubsub.subscriber instead). Once granted:

gcloud pubsub subscriptions create my-subscription \
  --project=intel-487218 \
  --topic=ingestion-completed \
  --topic-project=intel-487218

gcloud pubsub subscriptions pull my-subscription --project=intel-487218 --auto-ack --limit=5

If you're on a different GCP project (same org or not), you don't need access to this repo or project console — ask a project developer to grant your service account subscriber rights on the topic (or set up a push subscription to your endpoint on your behalf); you only need the topic name above and your own gcloud/client-library access to your own project.

For server-side filtering (e.g. only failures, or only one workflow), the full cross-project setup (both "you subscribe" and "we push to your endpoint" patterns), and delivery/idempotency notes, see Subscribing to ingestion-completed — everything above is enough to get started even without access to that file.

Where things live — BigQuery vs PostgreSQL

The same concept often has a definition in Postgres and time-series data in BigQuery. This map tells you which side to query:

BigQuery table catalog

Grouped by source. Key columns are a starting point — the full schema is visible in the BigQuery console.

🐦 Twitter / X

reply_settings is only populated for the official-X-API rows (col:x-api-tagged POIs — see the Ingestion table above). It's NULL for the bulk of twitter_tweets, which comes via XPOZ: verified live against XPOZ's API that it silently drops unsupported field names from a fields request rather than erroring, and replySettings/conversationControl are not among the fields it returns — this isn't a config gap, XPOZ just doesn't expose who-can-reply data.

🎵 TikTok

Videos are transcribed. Every collection pass pulls the audio track off each new video (yt-dlp → ffmpeg → Gemini, same path as podcasts and FB/X video) and writes three columns: transcript (the text), transcript_language (he-IL), and transcribed_at. This is spoken audio, which is a different thing from description — the caption the author typed. For "what did this person actually say", search transcript; for "how did they frame the post", read description.

Transcription is gated on engagement — share_count×3 + comment_count×2 + like_count, the same weighting the warehouse uses to rank TikTok posts for subjects and narratives, with a default threshold of 250. Low-reach videos are deliberately left untranscribed. This matches how Facebook, X and Instagram already work.

transcript IS NULL therefore means one of three things, and they are worth telling apart: the video was below the engagement gate, or we tried and got nothing back (no speech in the clip), or it hasn't been attempted yet. transcribed_at is the discriminator — it is stamped on every finished attempt, successful or not, so transcribed_at IS NOT NULL AND transcript IS NULL is precisely "we tried, there was nothing to transcribe".

Historical coverage — collected before the gate existed, when everything was transcribed — was ~79% of videos with any text but only ~61% with 100+ characters, median ~430 chars. The gap skews by account type: politicians' own accounts ran 90–96%, short-form news desks much lower (mako.israel 61%, n12news 71%), because so much of their output is captioned b-roll over music with no speech in it. Expect lower volume and a similar hit rate going forward — the gate cuts spend, it does not improve the ratio, because engagement barely predicts whether a clip contains speech.

Use video_duration_ms, not video_duration. The old video_duration column mixed units by collector — milliseconds from ScrapeCreators, seconds from XPOZ and Apify (measured on the same videos: 100.75 where ScrapeCreators said 100750) — so any query over it was wrong for part of the table. video_duration_ms is milliseconds for every row, backfilled from the collector recorded on each row rather than guessed from magnitude. Divide by 1000 for seconds.

⚠️ video_duration is left in place, unit-mixed and no longer written, purely as a rollback path. Rows collected from 2026-08-20 have it NULL. Do not query it.

The same convention holds for fb_posts and twitter_tweets: transcribed_at records the attempt, transcript holds text or NULL. Both carried [empty]/[failed] marker strings for a few days from 2026-08-13; if you are querying historical rows and see those values, treat them as NULL.

📘 Facebook

📸 Instagram

✈️ Telegram

📺 TV (Channel 14) & 📻 Radio (Galei Yisrael)

🎙️ Podcasts

📰 Online papers & 📚 Knowledge Base

🚨 Mashlat analytics (incident reports)

Mirrors Mashlat's read-only analytics.* Postgres views (a separate GCP project, mashlat-c2) for D-Day incident response data — signals from the field, triaged events, cases, and tasks, plus dimension lookups. signals/events/cases/tasks sync incrementally (hourly, via a last_modified watermark); the rest are small dimension/junction tables, fully re-extracted every run. reporter_name/reporter_phone (PII on mashlat_signals) are intentionally not mirrored.

location columns are GEOGRAPHY (point), decoded server-side from the source's PostGIS geometry.

As of 2026-08-11, a signal can link to more than one event (signals.event was retired for a many-to-many relation). mashlat_signals.event_id still exists but now only reflects the earliest-linked event, kept for backward compat — join mashlat_signals_events for the full picture.

📈 Analysis time-series

engagement vs raw_engagement (HML-155). Every per-item post query wraps its weighted-interaction formula in LN(1 + ...) — a log-compressed score meant only for ranking one post against another (it stops one viral post dominating a narrative's top-posts list). narrative_sources.engagement holds that ranking score; is_poi ordering and top-posts selection should keep using it. narrative_sources.raw_engagement holds the true, unlogged interaction count and is what mention_daily.engagement sums — summing the log-compressed score instead (as this table did before HML-155) makes mention_daily.engagement track mention_daily.volume almost 1:1, which is what the weekly-brief chart bug looked like. tv/podcast/online_paper have no native engagement signal, so their raw_engagement is always 0 (their engagement stays a constant ranking proxy, not a measurement).

🕵️ Pixel (disguised-venue detection)

Kept deliberately separate from fb_pages/fb_posts: these are investigation targets (suspected sleeper pages), not tracked POIs.

🚨 D-Day Signs (סימנים מעידים)

The Sign detection pipeline's audit trail — see docs/dday_signs.md for what a "Sign" is and how a candidate becomes a detection. Sign definitions (dday_signs, dday_sign_versions) and finished events (dday_events, dday_event_detections) are in Postgres — these BQ tables are the append-only trail behind them.

A candidate is nominated at roughly 50:1 versus detections that actually match — most of dday_sign_candidates is the judge saying no. A row's matched AND grounded is what makes it a real detection; see the file above for why an ungrounded quote is demoted rather than dropped.

Ignore operational tables ending in _etl_runs, _run_errors, _fetches, and _staging — they are internal ETL bookkeeping, not analysis data.

Full BigQuery table inventory

The sections above are the analyst-facing curated view. For completeness, this is every table any backend code defines or writes to in intel-487218.datawarehouse, grouped by the source file that owns its schema. 🗃️ = content/analysis table (covered above); ⚙️ = operational/internal (audit, delta-tracking, cache) — safe to ignore for analysis.

Query these through the app / API rather than direct SQL. Grouped by role (representative, not exhaustive):

Accessing the data

BigQuery (analysts)

Your identity is your @hamal-elections.co.il Google account. To query, you need these IAM roles on project intel-487218:

Ask one of the project developers to grant these — see For admins.

Three ways to query:

1. BigQuery Studio (no install): open https://console.cloud.google.com/bigquery?project=intel-487218, expand intel-487218 → datawarehouse, write SQL, Run.
2. bq CLI: bash gcloud auth login gcloud config set project intel-487218 bq query --use_legacy_sql=false \ 'SELECT username, followers_count FROM `intel-487218.datawarehouse.twitter_users` ORDER BY followers_count DESC LIMIT 10'
3. Python: ```python from google.cloud import bigquery

client = bigquery.Client(project="intel-487218") for r in client.query( "SELECT username, followers_count " "FROM intel-487218.datawarehouse.twitter_users " "ORDER BY followers_count DESC LIMIT 10" ).result(): print(r.username, r.followers_count) ```

PostgreSQL

The Postgres DB is the application's operational store. For almost everyone, the right way to read it is through the product:

• The app UI at hml.st (narratives, subjects, events, briefs, sources).
• The read API under /api/v1/... and the Intel MCP server at api.hml.st/mcp (for AI clients).

Direct SQL access to Postgres is reserved for engineers via the Render dashboard.

Example queries (BigQuery)

Top tweets by likes in the last 7 days (filter on the partition column to keep it cheap):

SELECT author_username, text, like_count, retweet_count
FROM `intel-487218.datawarehouse.twitter_tweets`
WHERE created_at_date >= DATE_SUB(CURRENT_DATE(), INTERVAL 7 DAY)
ORDER BY like_count DESC
LIMIT 10;

Daily mention volume of a narrative:

SELECT mention_date, SUM(volume) AS volume, SUM(engagement) AS engagement
FROM `intel-487218.datawarehouse.mention_daily`
WHERE narrative_id = 'NARRATIVE_ID_HERE'
GROUP BY mention_date
ORDER BY mention_date;

Search Telegram messages for a term:

SELECT channel_username, created_at_date, views, text
FROM `intel-487218.datawarehouse.telegram_messages`
WHERE text LIKE '%TERM%'
  AND created_at_date >= DATE_SUB(CURRENT_DATE(), INTERVAL 30 DAY)
ORDER BY views DESC
LIMIT 50;

Top subjects on TikTok yesterday:

SELECT subject_id, SUM(volume) AS volume
FROM `intel-487218.datawarehouse.subjects_daily`
WHERE mention_date = DATE_SUB(CURRENT_DATE(), INTERVAL 1 DAY)
  AND source_platform = 'tiktok'
GROUP BY subject_id
ORDER BY volume DESC
LIMIT 20;

Things worth knowing (gotchas)

• Filter by date to save money. twitter_tweets and the post tables are partitioned by created_at_date. Always add WHERE created_at_date >= ... — otherwise the query scans the whole table and costs more.
• Skip operational tables (_etl_runs, _run_errors, _fetches, _staging).
• camp / political_camp marks the political camp — useful for slicing subjects/narratives.
• Hebrew text: use LIKE '%...%' for exact matching, or REGEXP_CONTAINS for smarter matching.
• Data is not real-time. Most sources refresh daily (Twitter ~12:00 IL, papers every 2h). Check freshness via the _etl_runs tables or /admin/jobs.
• twitter_tweets.reply_settings is NULL on most rows. It's only populated for the small official-X-API subset; XPOZ (the source of the bulk of twitter_tweets) doesn't expose who-can-reply data at all — confirmed by requesting it directly against XPOZ's API and observing it's silently omitted from the response.
• mashlat_signals excludes PII. Reporter name/phone from the source system are intentionally not mirrored into BigQuery.
• The catalog can drift. This page is regenerated from the codebase by the /update-datadocs maintenance command — if a table here looks wrong, re-run it (see below).

For admins: granting BigQuery access

Give a new analyst read-only access with:

gcloud projects add-iam-policy-binding intel-487218 \
  --member="user:NEW_ANALYST@hamal-elections.co.il" \
  --role="roles/bigquery.dataViewer" --condition=None
gcloud projects add-iam-policy-binding intel-487218 \
  --member="user:NEW_ANALYST@hamal-elections.co.il" \
  --role="roles/bigquery.jobUser" --condition=None

# optional — BigQuery Studio / notebooks
gcloud projects add-iam-policy-binding intel-487218 \
  --member="user:NEW_ANALYST@hamal-elections.co.il" \
  --role="roles/bigquery.studioUser" --condition=None
gcloud projects add-iam-policy-binding intel-487218 \
  --member="user:NEW_ANALYST@hamal-elections.co.il" \
  --role="roles/bigquery.readSessionUser" --condition=None

Developers get a broader custom role (intelDev, includes write) via make gcloud-grant-dev-access. Do not give that to analysts — read-only is enough.

How this page is built and published

• docs/data/render.py renders this Markdown into a styled, self-contained docs/data/site/index.html (table of contents, dark/light, copy-to-clipboard on code).
• .github/workflows/publish-datadocs.yml runs the renderer and deploys to Cloudflare Pages on every push to main touching docs/data/**.
• Authentication is enforced by Cloudflare Access — only @hamal-elections.co.il Google logins reach datadocs.hml.st.

Preview locally:

uv run --with markdown python docs/data/render.py
open docs/data/site/index.html

Keep it current: run the /update-datadocs command — it re-scans the codebase for schema, source, and schedule changes and updates this file. One-time CI/Access setup is documented in docs/data/README.md.