How to Stream GitHub Activity for AI Agents at Scale

Ask an AI coding agent “what did the React team ship this week?” and it has a problem: GitHub’s activity is scattered across five different endpoints — commits, pull requests, issues, releases, user events — each with its own schema, its own pagination, and a shared rate limit that punishes exactly this kind of question. Unauthenticated, you get 60 requests per hour per IP. An agent that hits those endpoints directly reinvents merging and normalization every time and burns that budget in minutes. This guide covers how to get time-windowed, normalized dev activity as one clean event stream without an API key.

The job is a dev-grounding layer: one repo, org or user in, and a table of activity events out — each row carrying its type, repo, actor, title, state, timestamps, code-change metrics and a source URL. Structured enough to drop straight into a prompt, a digest, or a vector store.

What’s worth extracting

Every run streams one activity event per row to the dataset, normalized to a common schema across all types:

• type — commit, pull, issue, or release (plus GitHub event types like PushEvent in user mode). This single field lets you pivot the whole dataset.
• repo — the owner/repo slug the event belongs to.
• number — PR or issue number (null for commits).
• title — PR/issue title, release name, or a commit’s first-line message.
• actor — the author or actor login (null when GitHub anonymized the commit or the author has no account).
• state — open/closed for issues and PRs.
• createdAt, updatedAt — ISO 8601 timestamps.
• additions, deletions, commits — code-change metrics, populated on PR rows.
• merged — whether a PR was merged.
• labels — array of label names on issues and PRs.
• tagName, isPrerelease — release metadata (null for other types).
• url — the canonical GitHub URL for the event.
• scrapedAt — capture timestamp.

The nullable fields are the point: a commit row and a release row share one schema, with tagName null on the commit and additions null on the release. That predictability is what makes it agent-friendly — no per-type re-parsing downstream.

How the source actually works

This is a pure HTTP wrapper over the public GitHub REST API on a small Node 20 container — no headless browser. The behavior is driven by a mode:

• repo — one repository. Big repos yield thousands of events.
• org — lists an org’s public repos sorted by recent activity, then pulls activity from each. Monitor an entire ecosystem (kubernetes, vercel, facebook) in one run, capped by maxOrgRepos.
• user — a user’s public event feed (pushes, PRs, reviews). GitHub keeps only the last 90 days here.
• bulk — many repos in one run.

Each object type maps to its own endpoint. Commits come from /repos/{o}/{r}/commits?since=… (paginated), PRs from /repos/{o}/{r}/pulls?state=all&sort=created with a client-side since filter, issues from /repos/{o}/{r}/issues?state=all&since=… (with PRs filtered out, since they come from /pulls), and releases from /repos/{o}/{r}/releases. User mode targets /users/{u}/events/public.

A single-repo weekly pull:

{
  "mode": "repo",
  "repos": ["microsoft/vscode"],
  "eventTypes": ["commits", "pulls", "issues", "releases"],
  "since": "2026-06-25",
  "maxPerType": 200
}

An org-wide ecosystem scan, trimmed to the high-signal types:

{
  "mode": "org",
  "org": "kubernetes",
  "maxOrgRepos": 15,
  "eventTypes": ["pulls", "releases"],
  "since": "2026-06-01"
}

The since field (ISO date) is your delta window — it filters commits, issues and PRs to events after that date. eventTypes trims which types to fetch, maxPerType caps rows per type per repo (1–1000), and maxOrgRepos bounds an org scan (1–200).

▶ Run the GitHub Activity Stream on Apify — a repo, org or user in, normalized commits/PRs/issues/releases out. Keyless, proxy-rotated, thousands of events per run. Export to JSON, CSV or Excel, or call it via MCP.

The access reality

The friction here is entirely GitHub’s rate limit, and the design addresses it head-on. Unauthenticated, GitHub caps you at 60 requests per hour per IP. This scraper routes every request through Apify’s datacenter proxy, rotating the source IP per request, which spreads that limit and effectively removes it for monitoring a handful of repos. For heavy use — org-wide scans or frequent runs — you can supply an optional ghToken (a free Personal Access Token), sent as Authorization: Bearer, which raises the ceiling to 5,000/hr.

Two more realities worth knowing:

• Retry and backoff are built in. A 403 (rate-limit) or 5xx triggers exponential backoff, up to 6 attempts. A 404 (deleted or private repo) is treated as a graceful empty rather than a crash.
• Commits don’t carry per-commit stats. GitHub’s /commits list endpoint returns messages and authors but not additions/deletions — those only exist on the per-commit detail endpoint, and fetching every commit’s detail would multiply requests heavily. So PR rows carry the code-change metrics; commit rows don’t. For per-commit stats, post-process by fetching specific commit URLs.

And the obvious boundary: this targets public monitoring. It uses public endpoints without auth (or a token’s public scope). Private repos need a token with repo scope and aren’t the design target.

Example output

A pull-request row:

{
  "repo": "microsoft/vscode",
  "type": "pull",
  "number": 210543,
  "title": "Fix terminal rendering on macOS",
  "state": "closed",
  "actor": "user123",
  "createdAt": "2026-06-28T10:00:00Z",
  "updatedAt": "2026-07-01T15:00:00Z",
  "additions": 142,
  "deletions": 38,
  "commits": 3,
  "labels": ["bug", "terminal"],
  "merged": true,
  "tagName": null,
  "url": "https://github.com/microsoft/vscode/pull/210543",
  "scrapedAt": "2026-07-02T12:00:00.000Z"
}

A release row instead carries type: "release", tagName: "1.21.0", isPrerelease: false; a commit row carries type: "commit" with the first-line message as title.

Use cases

The README frames these around agents, and each is one Actor call or a scheduled run:

• Weekly dev summary agent — run repo mode with since set to seven days ago, then summarize the commits, PRs and releases into a digest.
• Release tracker — watch releases for a dependency org like kubernetes or vercel, alert on new versions, and draft upgrade notes from the release bodies.
• Security-commit detector — scan commit and PR title fields for security, cve, vuln, fix and flag them. Pairs naturally with a CVE advisory monitor.
• Dependency health scoring — pull issue metadata (open count, stale, label distribution) and PR merge velocity to score a repo’s maintenance health before you adopt it.
• Contributor monitoring — track a key maintainer’s public events in user mode to spot activity shifts.
• Org-wide radar — scan an entire org’s repos in org mode to map where work is actually happening across an ecosystem.
• RAG over dev activity — embed commit messages and PR titles into a vector store and answer “when did X get fixed?” with a citation to the exact commit.

Cost and effort math

Doing this by hand means writing five paginators, a since filter that GitHub only half-supports (PRs need client-side filtering), a normalizer that reconciles five different object shapes into one, IP rotation or token management to survive the 60/hr cap, and backoff logic for the 403s you will hit. It’s a small system, and it’s the kind that silently under-delivers when the rate limit throttles it mid-run.

The managed scraper is pay-per-result: one result event per saved activity row, charged per event and not per run, with empty results (deleted or empty repo) free. Proxy IP rotation is included, so the 60/hr limit is handled without you provisioning anything. A weekly since delta on a busy repo yields hundreds of events for a few cents; an org-wide scan yields thousands. If you’re running frequently, adding a free ghToken for the 5,000/hr ceiling is the one optimization worth making, and it’s a config field, not a rebuild.

Common pitfalls

• Always set since for monitoring. Without it you pull full history capped by maxPerType, which is slower and re-fetches what you already have. Set since to the last run time for clean deltas.
• Commits are the heaviest type. They’re the highest-volume and slowest. If you only need releases, set eventTypes: ["releases"] — don’t pay for commit pagination you won’t use.
• additions/deletions are PR-only. Don’t expect code-change metrics on commit rows; they aren’t in GitHub’s list endpoint.
• actor can be null. GitHub anonymizes noreply-email commits and authors without accounts. Handle null actors rather than assuming every event has one.
• User events cap at 90 days. The public event feed is a rolling window (10 pages max). For older user activity, there’s no backfill — accumulate by scheduling.
• Org mode is bounded by maxOrgRepos. A huge org has more repos than the default cap. Raise maxOrgRepos deliberately, and remember it multiplies the request count.

Wrapping up

GitHub activity is high-signal grounding for coding agents and tedious to collect correctly — five endpoints, one shared rate limit, five schemas to merge. You can build the merging-and-throttling layer yourself, but it’s the part that quietly breaks under the 60/hr cap. A keyless, proxy-rotated, pay-per-result stream that normalizes it all into one event schema is the layer worth not maintaining.

▶ Open the GitHub Activity Stream on Apify — commits, PRs, issues and releases for any repo/org/user, time-windowed and normalized. Keyless, IP-rotated, pay per event. Start on Apify’s free monthly credit.