TextBin - Online Text and Code Sharing

New Paste Browse

Here's how to scrape an RSS feed for http://magazinelib.com and http://freemagazines.top

I'm using the site:

https://rss-bridge.lewd.tech/

It's 1 of the free & open instances of RSS-Bridge which you can host yourself if 
you prefer. i'm using the XPath Bridge to extract scrapes from the first 10 
pages of the site and the FeedMerge Bridge to return all 10 requests as one RSS 
feed which should more than cover a days worth of new postings.

RSS-Bridge caches request results for a time so don't expect different results 
if you try to hammer this tool too fast.

Now because RSS-Bridge puts all those parameters into querystring args you may 
end up with a very long request URL, too long for your RSS reader to save or call.

Instead I'll save the huge URL in a file & use a command line tool like CURL or 
WGET to call it saving the requested ATOM RSS feed returned to a local file that 
my RSS reader can then pull from or I can push it to a webserver etc.

Create this file wherever you want to keep your fetching script.

-------------- magazineliburl --------------
https://rss-bridge.lewd.tech/?action=display&bridge=FeedMergeBridge&feed_name=magazinelib.com+Pages+1+To+9&feed_1=https%3A%2F%2Frss-bridge.lewd.tech%2F%3Faction%3Ddisplay%26bridge%3DXPathBridge%26url%3Dhttps%253A%252F%252Fmagazinelib.com%252Fpage%252F1%252F%26item%3D%252F%252Farticle%26title%3D.%252Fdiv%252Fheader%252F%252Fh3%252Fa%26content%3D%26uri%3D.%252Ffigure%252Fa%252F%2540href%26author%3D.%252Ffigure%252Fa%252Fspan%252Fimg%252F%2540data-lazy-src%26timestamp%3D.%252F%252Ffooter%252Fp%252Ftime%255B%2540class%253D%2522entry-date%2522%255D%252F%2540datetime%26enclosures%3D.%252Ffigure%252Fa%252Fspan%252Fimg%252F%2540data-lazy-src%26categories%3D.%252F%252Fdiv%255B%2540class%253D%2522entry-before-title%2522%255D%252Fp%252F%252Fa%255B2%255D%252Ftext%2528%2529%26format%3DAtom&feed_2=https%3A%2F%2Frss-bridge.lewd.tech%2F%3Faction%3Ddisplay%26bridge%3DXPathBridge%26url%3Dhttps%253A%252F%252Fmagazinelib.com%252Fpage%252F2%252F%26item%3D%252F%252Farticle%26title%3D.%252Fdiv%252Fheader%252F%252Fh3%252Fa%26content%3D%26uri%3D.%252Ffigure%252Fa%252F%2540href%26author%3D.%252Ffigure%252Fa%252Fspan%252Fimg%252F%2540data-lazy-src%26timestamp%3D.%252F%252Ffooter%252Fp%252Ftime%255B%2540class%253D%2522entry-date%2522%255D%252F%2540datetime%26enclosures%3D.%252Ffigure%252Fa%252Fspan%252Fimg%252F%2540data-lazy-src%26categories%3D.%252F%252Fdiv%255B%2540class%253D%2522entry-before-title%2522%255D%252Fp%252F%252Fa%255B2%255D%252Ftext%2528%2529%26format%3DAtom&feed_3=https%3A%2F%2Frss-bridge.lewd.tech%2F%3Faction%3Ddisplay%26bridge%3DXPathBridge%26url%3Dhttps%253A%252F%252Fmagazinelib.com%252Fpage%252F3%252F%26item%3D%252F%252Farticle%26title%3D.%252Fdiv%252Fheader%252F%252Fh3%252Fa%26content%3D%26uri%3D.%252Ffigure%252Fa%252F%2540href%26author%3D.%252Ffigure%252Fa%252Fspan%252Fimg%252F%2540data-lazy-src%26timestamp%3D.%252F%252Ffooter%252Fp%252Ftime%255B%2540class%253D%2522entry-date%2522%255D%252F%2540datetime%26enclosures%3D.%252Ffigure%252Fa%252Fspan%252Fimg%252F%2540data-lazy-src%26categories%3D.%252F%252Fdiv%255B%2540class%253D%2522entry-before-title%2522%255D%252Fp%252F%252Fa%255B2%255D%252Ftext%2528%2529%26format%3DAtom&feed_4=https%3A%2F%2Frss-bridge.lewd.tech%2F%3Faction%3Ddisplay%26bridge%3DXPathBridge%26url%3Dhttps%253A%252F%252Fmagazinelib.com%252Fpage%252F4%252F%26item%3D%252F%252Farticle%26title%3D.%252Fdiv%252Fheader%252F%252Fh3%252Fa%26content%3D%26uri%3D.%252Ffigure%252Fa%252F%2540href%26author%3D.%252Ffigure%252Fa%252Fspan%252Fimg%252F%2540data-lazy-src%26timestamp%3D.%252F%252Ffooter%252Fp%252Ftime%255B%2540class%253D%2522entry-date%2522%255D%252F%2540datetime%26enclosures%3D.%252Ffigure%252Fa%252Fspan%252Fimg%252F%2540data-lazy-src%26categories%3D.%252F%252Fdiv%255B%2540class%253D%2522entry-before-title%2522%255D%252Fp%252F%252Fa%255B2%255D%252Ftext%2528%2529%26format%3DAtom&feed_5=https%3A%2F%2Frss-bridge.lewd.tech%2F%3Faction%3Ddisplay%26bridge%3DXPathBridge%26url%3Dhttps%253A%252F%252Fmagazinelib.com%252Fpage%252F5%252F%26item%3D%252F%252Farticle%26title%3D.%252Fdiv%252Fheader%252F%252Fh3%252Fa%26content%3D%26uri%3D.%252Ffigure%252Fa%252F%2540href%26author%3D.%252Ffigure%252Fa%252Fspan%252Fimg%252F%2540data-lazy-src%26timestamp%3D.%252F%252Ffooter%252Fp%252Ftime%255B%2540class%253D%2522entry-date%2522%255D%252F%2540datetime%26enclosures%3D.%252Ffigure%252Fa%252Fspan%252Fimg%252F%2540data-lazy-src%26categories%3D.%252F%252Fdiv%255B%2540class%253D%2522entry-before-title%2522%255D%252Fp%252F%252Fa%255B2%255D%252Ftext%2528%2529%26format%3DAtom&feed_6=https%3A%2F%2Frss-bridge.lewd.tech%2F%3Faction%3Ddisplay%26bridge%3DXPathBridge%26url%3Dhttps%253A%252F%252Fmagazinelib.com%252Fpage%252F6%252F%26item%3D%252F%252Farticle%26title%3D.%252Fdiv%252Fheader%252F%252Fh3%252Fa%26content%3D%26uri%3D.%252Ffigure%252Fa%252F%2540href%26author%3D.%252Ffigure%252Fa%252Fspan%252Fimg%252F%2540data-lazy-src%26timestamp%3D.%252F%252Ffooter%252Fp%252Ftime%255B%2540class%253D%2522entry-date%2522%255D%252F%2540datetime%26enclosures%3D.%252Ffigure%252Fa%252Fspan%252Fimg%252F%2540data-lazy-src%26categories%3D.%252F%252Fdiv%255B%2540class%253D%2522entry-before-title%2522%255D%252Fp%252F%252Fa%255B2%255D%252Ftext%2528%2529%26format%3DAtom&feed_7=https%3A%2F%2Frss-bridge.lewd.tech%2F%3Faction%3Ddisplay%26bridge%3DXPathBridge%26url%3Dhttps%253A%252F%252Fmagazinelib.com%252Fpage%252F7%252F%26item%3D%252F%252Farticle%26title%3D.%252Fdiv%252Fheader%252F%252Fh3%252Fa%26content%3D%26uri%3D.%252Ffigure%252Fa%252F%2540href%26author%3D.%252Ffigure%252Fa%252Fspan%252Fimg%252F%2540data-lazy-src%26timestamp%3D.%252F%252Ffooter%252Fp%252Ftime%255B%2540class%253D%2522entry-date%2522%255D%252F%2540datetime%26enclosures%3D.%252Ffigure%252Fa%252Fspan%252Fimg%252F%2540data-lazy-src%26categories%3D.%252F%252Fdiv%255B%2540class%253D%2522entry-before-title%2522%255D%252Fp%252F%252Fa%255B2%255D%252Ftext%2528%2529%26format%3DAtom&feed_8=https%3A%2F%2Frss-bridge.lewd.tech%2F%3Faction%3Ddisplay%26bridge%3DXPathBridge%26url%3Dhttps%253A%252F%252Fmagazinelib.com%252Fpage%252F8%252F%26item%3D%252F%252Farticle%26title%3D.%252Fdiv%252Fheader%252F%252Fh3%252Fa%26content%3D%26uri%3D.%252Ffigure%252Fa%252F%2540href%26author%3D.%252Ffigure%252Fa%252Fspan%252Fimg%252F%2540data-lazy-src%26timestamp%3D.%252F%252Ffooter%252Fp%252Ftime%255B%2540class%253D%2522entry-date%2522%255D%252F%2540datetime%26enclosures%3D.%252Ffigure%252Fa%252Fspan%252Fimg%252F%2540data-lazy-src%26categories%3D.%252F%252Fdiv%255B%2540class%253D%2522entry-before-title%2522%255D%252Fp%252F%252Fa%255B2%255D%252Ftext%2528%2529%26format%3DAtom&feed_9=https%3A%2F%2Frss-bridge.lewd.tech%2F%3Faction%3Ddisplay%26bridge%3DXPathBridge%26url%3Dhttps%253A%252F%252Fmagazinelib.com%252Fpage%252F9%252F%26item%3D%252F%252Farticle%26title%3D.%252Fdiv%252Fheader%252F%252Fh3%252Fa%26content%3D%26uri%3D.%252Ffigure%252Fa%252F%2540href%26author%3D.%252Ffigure%252Fa%252Fspan%252Fimg%252F%2540data-lazy-src%26timestamp%3D.%252F%252Ffooter%252Fp%252Ftime%255B%2540class%253D%2522entry-date%2522%255D%252F%2540datetime%26enclosures%3D.%252Ffigure%252Fa%252Fspan%252Fimg%252F%2540data-lazy-src%26categories%3D.%252F%252Fdiv%255B%2540class%253D%2522entry-before-title%2522%255D%252Fp%252F%252Fa%255B2%255D%252Ftext%2528%2529%26format%3DAtom&feed_10=&limit=270&format=Atom
--------------------------------------------

I'll use WGET here to request an rss feed from the bridge server.
Point the --input-file option to magazineliburl for the URL to fetch.
For example:  --input-file=".\scrapes\magazineliburl"

With all the querystring encoding it may try to double encode some args so we 
use the --no-iri option to tell WGET to not encode the args in the file.
in CURL I think the --get --data-urlencode options cover this.

I'll also tell WGET to save the output to an atom file, 
For example: -O ".\scrapes\magazinelib.atom"

So my basic command line without any more fun options is:

wget --no-iri --input-file=".\scrapes\magazineliburl" -O ".\scrapes\magazinelib.atom"

Go look at: magazinelib.atom
It should contain an RSS feed in atom format that any reader/aggregator can work.

use a file:// url in your local reader or put it on a webserver.
You can run the wget command in a script or as a cron job to regularly
refresh this file, once or twice a day is enough.

I applied the same technique to the site http://freemagazines.top

-------------- freemagazinestopurl --------------
https://rss-bridge.lewd.tech/?action=display&bridge=FeedMergeBridge&feed_name=freemagazines.top+Last+10+Pages&limit=999&format=Mrss&feed_1=https%3A%2F%2Ffreemagazines.top%2Ffeed%3Fpaged%3D1&feed_2=https%3A%2F%2Ffreemagazines.top%2Ffeed%3Fpaged%3D2&feed_3=https%3A%2F%2Ffreemagazines.top%2Ffeed%3Fpaged%3D3&feed_4=https%3A%2F%2Ffreemagazines.top%2Ffeed%3Fpaged%3D4&feed_5=https%3A%2F%2Ffreemagazines.top%2Ffeed%3Fpaged%3D5&feed_6=https%3A%2F%2Ffreemagazines.top%2Ffeed%3Fpaged%3D6&feed_7=https%3A%2F%2Ffreemagazines.top%2Ffeed%3Fpaged%3D7&feed_8=https%3A%2F%2Ffreemagazines.top%2Ffeed%3Fpaged%3D8&feed_9=https%3A%2F%2Ffreemagazines.top%2Ffeed%3Fpaged%3D9&feed_10=https%3A%2F%2Ffreemagazines.top%2Ffeed%3Fpaged%3D10
-------------------------------------------------

With the WGET command:

wget --no-iri --input-file=".\scrapes\freemagazinestopurl" -O ".\scrapes\freemagazinestop.rss"

To create the RSS file freemagazinestop.rss

Share and Enjoy.

Paste Settings

Paste Title : [Optional]

Paste Folder : [Optional]

Syntax Highlighting : [Optional]

Paste Expiration : [Optional]

Paste Status : [Optional]

Password : [Optional]

Description: [Optional]

Tags: [Optional]

Encrypt Paste (?)

You are currently not logged in, this means you can not edit or delete anything you paste. Sign Up or Login

Paste Settings

Site Languages