A paper made by a pipeline.

How DevAtlas works

DevAtlas separates collection from reading. A scheduled GitHub Actions run gathers public items, normalizes and scores them, then creates a dated Markdown report and JSON snapshots. The site turns those records into a readable front page and browsable archive.

The application is a TypeScript monorepo: a discovery pipeline, a Hono API designed for Cloudflare Workers, and a Next.js static front end.

The daily pipeline

1. Discover. Query public GitHub, Hacker News, and job APIs through source adapters.
2. Normalize. Clean titles and descriptions, normalize timestamps, and canonicalize URLs.
3. Deduplicate and score. Use SHA-256 URL and content hashes, then calculate a transparent relevance score with freshness, popularity, developer value, and technology impact.
4. Enrich. Use the configured Gemini or Groq provider when available, with deterministic rules as a fallback.
5. Publish. Write the daily Markdown report and JSON snapshot. When MongoDB is configured, sync the catalog and export edge snapshots.

Public sources → discovery adapters → normalize & score
               → optional AI enrichment → report + snapshots
               → Worker API / static site

Reading the edition

The front page is arranged like a daily paper: one lead story, a short briefing column, a moving headline wire, and desks for work, tools, and open source. The archive keeps dated reports. The index supports keyword search and section filters.

Use the source link on each item to check current details. Job listings, repository activity, and tool information can change after publication.

Operations and CI

GitHub Actions validates the TypeScript workspaces, runs their existing tests, and builds the app. The scheduled discovery workflow publishes generated artifacts when the commit gate detects changes. Separate workflows cover deployment, security scanning, dependency updates, release artifacts, and weekly database maintenance.

The operations page reports the health response and the latest pipeline record available to the site. When the API is unreachable, the static catalog remains available; snapshot-only status is shown explicitly.

Useful endpoints include /api/v1/health, /api/v1/stats, /api/v1/items, /api/v1/jobs, /api/v1/tools, /api/v1/repositories, and /api/v1/reports.

Data and sources

Discovery adapters use public endpoints and retain source links. URL canonicalization removes common tracking parameters. The current job collection combines public board APIs with a curated adapter; its source attribution is shown with each listing.

Snapshots are checked into the repository so the static site can render without a live database connection. MongoDB is used by the batch pipeline when configured. The Worker’s current read path serves cached edge JSON snapshots.