We already had GrantRadar pulling from the Innovation Canada dataset every week. It was working well. But we kept finding programs on department websites that were not in the centralized dataset. New programs that had been announced but not yet added. Programs that had been restructured under new names. Regional programs that lived only on the funding pages of specific agencies.
The centralized dataset was a floor, not a ceiling. We needed to go to the source.
How the scraper works
Every Monday at 6 AM, the system starts with a curated list of seed URLs. These are the official funding pages for ISED, NRCan, DND, AAFC, ECCC, ESDC, PacifiCan, PrairiesCan, FedDev Ontario, ACOA, CED, CanNor, and the Canada Infrastructure Bank. Each seed URL is the department's main grants or funding page.
The system fetches each seed page and scans the HTML for every link that looks like a program. A filtering layer separates program links from navigation noise, social media links, PDFs, French-language duplicates, and boilerplate pages. Only links from allowed Canadian government domains pass through. Only links whose anchor text contains program-related terms (fund, grant, contribution, initiative, incentive, support, loan) survive.
All extracted program URLs are deduplicated across departments, then the system loops through each one individually.
What happens at each program page
For every program URL, the system fetches the full page, strips all HTML (scripts, styles, navigation, headers, footers) down to clean text, and sends that text to Claude with a structured extraction prompt.
Claude reads the full page and returns: program name, description, eligibility criteria, requirements, funding type (grant, contribution, loan, tax credit, mixed, or unknown), and current status (open, closed, ongoing, or unknown).
Each extracted program gets formatted into a clean row and upserted to a master Google Sheet. New programs appear automatically. Existing ones get refreshed with the latest data from the actual source page. A 2-second wait between requests keeps the system respectful of government servers.
Why this matters alongside GrantRadar
GrantRadar gives the strategic layer: AI-enriched scores, ideal client profiles, responsible ministers, sector tagging. It works from a curated dataset that is structured and reliable but not comprehensive.
GovPrograms gives the operational layer: what are the actual eligibility requirements, is the intake open or closed right now, what does the program page actually say. It works from the source pages themselves and catches programs that the centralized dataset misses.
Together they form a complete picture. GrantRadar scores the landscape. GovPrograms maps it from the ground up. The Federal Program Monitor watches for daily changes across both.
The result
Before this system, someone on the team had to manually browse department websites every week to spot new programs. It was slow, inconsistent, and the kind of work that always got deprioritized when client work was busy.
Now the team opens one sheet and sees everything. Pulled and parsed directly from the government's own pages. Every Monday. Automatically.
Need to scrape a public data source?
Government sites, regulatory databases, industry directories, competitor pages. If the data is public, we can extract it, structure it, and deliver it to your team automatically. Book a free 30-minute audit.
Book a Free AuditWe build these systems for government relations firms. See what yours could look like.
Built for public affairs →Get every teardown in your inbox.
One email per build. The full node-by-node breakdown, plus the JSON to fork it. No fluff, no spam.