Intel Data Platform Guide

Who this is for: analysts, researchers, and team members who want to use the data we collect — where it lives, how it gets there, when it refreshes, and how to query it. No code access required.

This file is the source of truth. The Markdown file docs/data/data_guide.md is the source; the online page at datadocs.hml.st is generated from it automatically by CI on every push to main. Edit here, open a PR — CI renders and publishes. See How this page is built.

Overview

Everything we collect — social media, TV, radio, online news — plus everything we derive from it — narratives, subjects, events, briefs — lives in two stores:

Rule of thumb: if you want to analyze content or trends, you want BigQuery. If you want the curated definitions and finished results the product shows, that's PostgreSQL — reachable via the app at hml.st or the API.

The data moves through four layers: Collection → Storage → Processing → Presentation. The full diagram is in docs/diagrams/data-flow.txt.

Architecture diagrams

Two diagrams live in docs/diagrams/: end-to-end data flow (Collection → Storage → Processing → Presentation, plus detailed Publishers/messaging-funnel and Events sub-pipelines) and deployment topology (Render vs GCP, schedulers, workflows, jobs, secrets, external APIs). Click either image to open the full-size SVG.

Refreshed 2026-07-12 to match the ScrapeCreators migration, the official X API / col:3x tohar collection paths, and the Pixel pipeline. If this guide and the diagrams ever drift again, the tables below in this guide are the current source of truth — diagrams are refreshed on request, not on every change.

Ingestion (Collection)

Scheduled Cloud Run Jobs (region us-west1) pull from each source and write raw rows to BigQuery. POI handles to track come from Postgres (pois). No ingest job writes to Postgres — all raw data lands in BigQuery.

Each pipeline follows the same pattern: raw content tables, plus <platform>_etl_runs / _etl_run_errors (run audit) and <platform>_*_fetches (delta-window tracking). Raw tables upsert through a transient <table>_staging twin.

Processing (Analysis)

Analysis jobs read from BigQuery, load their catalogs (narratives, subjects, POIs, domain knowledge) from Postgres at startup, and write outputs back to both stores.

Most analysis uses Gemini for classification, extraction, embeddings, and Hebrew synthesis.

When the flows run

All schedules are Asia/Jerusalem; schedulers live in us-west1 (Cloud Scheduler → Workflows).

To check the last successful run of any pipeline, query its <platform>_etl_runs table in BigQuery, or use /admin/jobs in the app.

Where things live — BigQuery vs PostgreSQL

The same concept often has a definition in Postgres and time-series data in BigQuery. This map tells you which side to query:

BigQuery table catalog

Grouped by source. Key columns are a starting point — the full schema is visible in the BigQuery console.

🐦 Twitter / X

🎵 TikTok

📘 Facebook

📸 Instagram

✈️ Telegram

📺 TV (Channel 14) & 📻 Radio (Galei Yisrael)

📰 Online papers & 📚 Knowledge Base

📈 Analysis time-series

🕵️ Pixel (disguised-venue detection)

Kept deliberately separate from fb_pages/fb_posts: these are investigation targets (suspected sleeper pages), not tracked POIs.

Ignore operational tables ending in _etl_runs, _run_errors, _fetches, and _staging — they are internal ETL bookkeeping, not analysis data.

Full BigQuery table inventory

The sections above are the analyst-facing curated view. For completeness, this is every table any backend code defines or writes to in intel-487218.datawarehouse, grouped by the source file that owns its schema. 🗃️ = content/analysis table (covered above); ⚙️ = operational/internal (audit, delta-tracking, cache) — safe to ignore for analysis.

Query these through the app / API rather than direct SQL. Grouped by role (representative, not exhaustive):

Accessing the data

BigQuery (analysts)

Your identity is your @hamal-elections.co.il Google account. To query, you need these IAM roles on project intel-487218:

Ask one of the project developers to grant these — see For admins.

Three ways to query:

1. BigQuery Studio (no install): open https://console.cloud.google.com/bigquery?project=intel-487218, expand intel-487218 → datawarehouse, write SQL, Run.
2. bq CLI: bash gcloud auth login gcloud config set project intel-487218 bq query --use_legacy_sql=false \ 'SELECT username, followers_count FROM `intel-487218.datawarehouse.twitter_users` ORDER BY followers_count DESC LIMIT 10'
3. Python: ```python from google.cloud import bigquery

client = bigquery.Client(project="intel-487218") for r in client.query( "SELECT username, followers_count " "FROM intel-487218.datawarehouse.twitter_users " "ORDER BY followers_count DESC LIMIT 10" ).result(): print(r.username, r.followers_count) ```

PostgreSQL

The Postgres DB is the application's operational store. For almost everyone, the right way to read it is through the product:

• The app UI at hml.st (narratives, subjects, events, briefs, sources).
• The read API under /api/v1/... and the Intel MCP server at api.hml.st/mcp (for AI clients).

Direct SQL access to Postgres is reserved for engineers via the Render dashboard.

Example queries (BigQuery)

Top tweets by likes in the last 7 days (filter on the partition column to keep it cheap):

SELECT author_username, text, like_count, retweet_count
FROM `intel-487218.datawarehouse.twitter_tweets`
WHERE created_at_date >= DATE_SUB(CURRENT_DATE(), INTERVAL 7 DAY)
ORDER BY like_count DESC
LIMIT 10;

Daily mention volume of a narrative:

SELECT mention_date, SUM(volume) AS volume, SUM(engagement) AS engagement
FROM `intel-487218.datawarehouse.mention_daily`
WHERE narrative_id = 'NARRATIVE_ID_HERE'
GROUP BY mention_date
ORDER BY mention_date;

Search Telegram messages for a term:

SELECT channel_username, created_at_date, views, text
FROM `intel-487218.datawarehouse.telegram_messages`
WHERE text LIKE '%TERM%'
  AND created_at_date >= DATE_SUB(CURRENT_DATE(), INTERVAL 30 DAY)
ORDER BY views DESC
LIMIT 50;

Top subjects on TikTok yesterday:

SELECT subject_id, SUM(volume) AS volume
FROM `intel-487218.datawarehouse.subjects_daily`
WHERE mention_date = DATE_SUB(CURRENT_DATE(), INTERVAL 1 DAY)
  AND source_platform = 'tiktok'
GROUP BY subject_id
ORDER BY volume DESC
LIMIT 20;

Things worth knowing (gotchas)

• Filter by date to save money. twitter_tweets and the post tables are partitioned by created_at_date. Always add WHERE created_at_date >= ... — otherwise the query scans the whole table and costs more.
• Skip operational tables (_etl_runs, _run_errors, _fetches, _staging).
• camp / political_camp marks the political camp — useful for slicing subjects/narratives.
• Hebrew text: use LIKE '%...%' for exact matching, or REGEXP_CONTAINS for smarter matching.
• Data is not real-time. Most sources refresh daily (Twitter hourly, papers every 2h). Check freshness via the _etl_runs tables or /admin/jobs.
• The catalog can drift. This page is regenerated from the codebase by the /update-datadocs maintenance command — if a table here looks wrong, re-run it (see below).

For admins: granting BigQuery access

Give a new analyst read-only access with:

gcloud projects add-iam-policy-binding intel-487218 \
  --member="user:NEW_ANALYST@hamal-elections.co.il" \
  --role="roles/bigquery.dataViewer" --condition=None
gcloud projects add-iam-policy-binding intel-487218 \
  --member="user:NEW_ANALYST@hamal-elections.co.il" \
  --role="roles/bigquery.jobUser" --condition=None

# optional — BigQuery Studio / notebooks
gcloud projects add-iam-policy-binding intel-487218 \
  --member="user:NEW_ANALYST@hamal-elections.co.il" \
  --role="roles/bigquery.studioUser" --condition=None
gcloud projects add-iam-policy-binding intel-487218 \
  --member="user:NEW_ANALYST@hamal-elections.co.il" \
  --role="roles/bigquery.readSessionUser" --condition=None

Developers get a broader custom role (intelDev, includes write) via make gcloud-grant-dev-access. Do not give that to analysts — read-only is enough.

How this page is built and published

• docs/data/render.py renders this Markdown into a styled, self-contained docs/data/site/index.html (table of contents, dark/light, copy-to-clipboard on code).
• .github/workflows/publish-datadocs.yml runs the renderer and deploys to Cloudflare Pages on every push to main touching docs/data/**.
• Authentication is enforced by Cloudflare Access — only @hamal-elections.co.il Google logins reach datadocs.hml.st.

Preview locally:

uv run --with markdown python docs/data/render.py
open docs/data/site/index.html

Keep it current: run the /update-datadocs command — it re-scans the codebase for schema, source, and schedule changes and updates this file. One-time CI/Access setup is documented in docs/data/README.md.