Scraping OER Metadata for Pressbooks: A Compliance-First Workflow for CUNY Teams

Many CUNY instructors and librarians build OER lists by hand. That work adds up fast when a course spans many units, media types, and license terms.

A small scraping pipeline can help. It can pull titles, authors, license cues, and file types from public pages. You can then draft a Pressbooks chapter, a glossary, or a resource bank with less rekeying.

Scraping also raises real risks. You need to respect site rules, protect user data, and keep your output usable for all learners. WHO estimates about 1 in 6 people live with a disability, so accessibility cannot sit at the end.

Start with scope: collect only what teaching needs

Write down the teaching task first. You may need a reading list, a set of video links, or a bank of practice items. That task should set your data fields.

For OER discovery, metadata often beats full text. Titles, creators, short blurbs, license marks, and topic tags give you most of what you need. You also reduce risk when you skip copying whole pages.

Set a collection rule you can defend

Pick a clear rule for what you will collect. Limit the crawler to public pages that target learners or adopters. Avoid pages behind logins, paywalls, or student accounts.

Decide how you handle personal data. If a page shows emails or user names, drop them at parse time. That choice supports privacy norms in higher ed and keeps your dataset safer to share.

Respect access rules and keep load low

Check robots.txt and the site terms before you run a job. Robots.txt does not grant rights, but it signals site limits. Terms can ban bulk pulls, even from public pages.

Rate limits matter for campus work. Many sites treat fast bursts as abuse and will block your IP. Build in delay, set a max page count, and stop on a 429 response.

Use proxies as a safety tool, not a bypass

Some projects need stable access from many networks. That need comes up when targets block shared campus IP ranges or when you test from a user-like path. Byteful supports this pattern with managed routing and clear controls.

Use a residential proxy. Keep your crawl slow and scoped. Do not use proxies to break login walls or ignore bans.

Log each request with time, target host, and result code. Those logs help you audit impact and debug blocks. They also help you show good faith if a site admin asks questions.

Turn raw pages into Pressbooks-ready records

Plan your output around Pressbooks authoring. A CSV or JSON file works well for import steps and quick review. Each record should map to a chapter section, a table of links, or an H5P activity prompt.

Keep fields short and consistent. Use one field for the resource URL, one for display title, and one for creator. Add a license field that stores the exact label you saw.

License and attribution: treat it as core data

Many OER pages show a CC badge, a short license line, or a rights note. Capture that text as shown. You can then write the attribution line in a steady format inside Pressbooks.

Do not guess a license. If the page lacks a clear license note, mark it as unknown. That choice protects adopters who must meet open license rules.

Accessibility checks you can automate

Scraping can surface access issues early. You can flag missing captions by checking for transcript links near video embeds. You can also flag images that lack alt text cues in the HTML.

Keep these flags in your dataset. When you draft a Pressbooks chapter, you can add notes for fixes or swaps. That supports universal design and reduces last-minute churn.

Operate like a small program, not a one-off script

Set roles for the work. One person owns crawl rules and blocks. Another reviews license fields and checks fit for course goals.

Run short jobs and review output often. A weekly or unit-by-unit run limits drift and keeps errors small. It also helps you align updates with course shells and Brightspace exports.

Store a short README with your dataset. Note targets, rules, and fields in plain language. This makes it easier for the next instructor to reuse the work and keep it open.

 

License

Inspire Copyright © by learners. All Rights Reserved.