Opinion corpus + argument mining
A scraping and normalization pipeline for opinion journalism, plus models that tag claims and premises.
- selenium
- drissionpage
- pandas
- longformer
- gemma
- pydantic-ai
A scraping and normalization pipeline for opinion journalism, plus models that tag claims and premises inside each article.
Problem
Studying argumentation in opinion journalism needed a large, labeled corpus that didn't exist yet.
Decisions
- A scraping and normalization pipeline built on Selenium and DrissionPage.
- Longformer and Gemma span classification to tag claims and premises.
- PydanticAI agents on top of the classifiers for structured extraction.
Results
- 100k+ articles