2.6 KiB

Original Anotar Histórico

Feedcake

„Gib mir ein Stück Kuchen und ich will den ganzen cake.“

The Problem

Most news platforms don’t give you the full article via rss/atom.
This wouldn’t be a big problem. But some of them do crazy 1984-ish stuff on their websites or they have built up paywalls for users with privacy addons.

Goal of this script

Getting a full-featured news feed (full articles with images) from various news pages

Benefits for the user

They don’t need to go on the website to read the articles
No ads
No tracking

Possible downsides for the user

articles don’t get updated once they are scraped
articles arrive with some delay
interactive/special elements in articles may not work

What it does

Fetching the news feed from the original website
scrape contents of new entries and save them into a directory structure
save a full featured RSS file

... and what it doesn’t

Managing when it scrapes (use crontab or sth else for that)
serving the feeds and assets via HTTPS (use your favorite web server for that)
Dealing with article comments
Archiving feeds (But content and assets - but without meta data)
Using some sort of database (the file structure is everything)
Cleaning up old assets
Automaticly updating the basedir if it changed.

Ugly stuff?

the html files (feed content) get stored along the assets, even if they don’t need to be exploited via HTTPS.

How to use

git clone this project and enter directory
install python3, pip and virtualenv
Create virtualenv: virtualenv -p python3 ~/.virtualenvs/feedcake
Activate your new virtualenv: source ~/.virtualenvs/feedcake/bin/activate
switch into the projects directory: cd feedcake
Install requirements: pip3 install -r requirements.txt
copy the config-example: cp config-example.json config.json.
edit config.json
copy the cron-example: cp cron-example.sh cron.sh.
edit cron.sh
make cron.sh executable: chmod +x cron.sh
add cronjob for cron.sh: crontab -e
- */5 * * * * /absolute/path/to/cron.sh > /path/to/logfile 2>&1
setup your webserver:
- let your webserver somehow point to the feeds directory. You should protect the http path with a basic authentication.
- let the assets_url specified in the config point to the assets directory.
After running the script the first time, your desired feed is available at base_url/destination (e.g. https://yourdomain.tld/some-url/newspaper.xml)

TODOs

Decide what should happen with old news articles and assets which are not listed in the current feed anymore.

2.6 KiB Original Anotar Histórico