Sign up
Login
New paste
Home
English
English
Português
Sign up
Login
New Paste
Browse
Here's how to scrape an RSS feed for http://magazinelib.com and http://freemagazines.top I'm using the site: https://rss-bridge.lewd.tech/ It's 1 of the free & open instances of RSS-Bridge which you can host yourself if you prefer. i'm using the XPath Bridge to extract scrapes from the first 10 pages of the site and the FeedMerge Bridge to return all 10 requests as one RSS feed which should more than cover a days worth of new postings. RSS-Bridge caches request results for a time so don't expect different results if you try to hammer this tool too fast. Now because RSS-Bridge puts all those parameters into querystring args you may end up with a very long request URL, too long for your RSS reader to save or call. Instead I'll save the huge URL in a file & use a command line tool like CURL or WGET to call it saving the requested ATOM RSS feed returned to a local file that my RSS reader can then pull from or I can push it to a webserver etc. Create this file wherever you want to keep your fetching script. -------------- magazineliburl -------------- https://rss-bridge.lewd.tech/?action=display&bridge=FeedMergeBridge&feed_name=magazinelib.com+Pages+1+To+9&feed_1=https%3A%2F%2Frss-bridge.lewd.tech%2F%3Faction%3Ddisplay%26bridge%3DXPathBridge%26url%3Dhttps%253A%252F%252Fmagazinelib.com%252Fpage%252F1%252F%26item%3D%252F%252Farticle%26title%3D.%252Fdiv%252Fheader%252F%252Fh3%252Fa%26content%3D%26uri%3D.%252Ffigure%252Fa%252F%2540href%26author%3D.%252Ffigure%252Fa%252Fspan%252Fimg%252F%2540data-lazy-src%26timestamp%3D.%252F%252Ffooter%252Fp%252Ftime%255B%2540class%253D%2522entry-date%2522%255D%252F%2540datetime%26enclosures%3D.%252Ffigure%252Fa%252Fspan%252Fimg%252F%2540data-lazy-src%26categories%3D.%252F%252Fdiv%255B%2540class%253D%2522entry-before-title%2522%255D%252Fp%252F%252Fa%255B2%255D%252Ftext%2528%2529%26format%3DAtom&feed_2=https%3A%2F%2Frss-bridge.lewd.tech%2F%3Faction%3Ddisplay%26bridge%3DXPathBridge%26url%3Dhttps%253A%252F%252Fmagazinelib.com%252Fpage%252F2%252F%26item%3D%252F%252Farticle%26title%3D.%252Fdiv%252Fheader%252F%252Fh3%252Fa%26content%3D%26uri%3D.%252Ffigure%252Fa%252F%2540href%26author%3D.%252Ffigure%252Fa%252Fspan%252Fimg%252F%2540data-lazy-src%26timestamp%3D.%252F%252Ffooter%252Fp%252Ftime%255B%2540class%253D%2522entry-date%2522%255D%252F%2540datetime%26enclosures%3D.%252Ffigure%252Fa%252Fspan%252Fimg%252F%2540data-lazy-src%26categories%3D.%252F%252Fdiv%255B%2540class%253D%2522entry-before-title%2522%255D%252Fp%252F%252Fa%255B2%255D%252Ftext%2528%2529%26format%3DAtom&feed_3=https%3A%2F%2Frss-bridge.lewd.tech%2F%3Faction%3Ddisplay%26bridge%3DXPathBridge%26url%3Dhttps%253A%252F%252Fmagazinelib.com%252Fpage%252F3%252F%26item%3D%252F%252Farticle%26title%3D.%252Fdiv%252Fheader%252F%252Fh3%252Fa%26content%3D%26uri%3D.%252Ffigure%252Fa%252F%2540href%26author%3D.%252Ffigure%252Fa%252Fspan%252Fimg%252F%2540data-lazy-src%26timestamp%3D.%252F%252Ffooter%252Fp%252Ftime%255B%2540class%253D%2522entry-date%2522%255D%252F%2540datetime%26enclosures%3D.%252Ffigure%252Fa%252Fspan%252Fimg%252F%2540data-lazy-src%26categories%3D.%252F%252Fdiv%255B%2540class%253D%2522entry-before-title%2522%255D%252Fp%252F%252Fa%255B2%255D%252Ftext%2528%2529%26format%3DAtom&feed_4=https%3A%2F%2Frss-bridge.lewd.tech%2F%3Faction%3Ddisplay%26bridge%3DXPathBridge%26url%3Dhttps%253A%252F%252Fmagazinelib.com%252Fpage%252F4%252F%26item%3D%252F%252Farticle%26title%3D.%252Fdiv%252Fheader%252F%252Fh3%252Fa%26content%3D%26uri%3D.%252Ffigure%252Fa%252F%2540href%26author%3D.%252Ffigure%252Fa%252Fspan%252Fimg%252F%2540data-lazy-src%26timestamp%3D.%252F%252Ffooter%252Fp%252Ftime%255B%2540class%253D%2522entry-date%2522%255D%252F%2540datetime%26enclosures%3D.%252Ffigure%252Fa%252Fspan%252Fimg%252F%2540data-lazy-src%26categories%3D.%252F%252Fdiv%255B%2540class%253D%2522entry-before-title%2522%255D%252Fp%252F%252Fa%255B2%255D%252Ftext%2528%2529%26format%3DAtom&feed_5=https%3A%2F%2Frss-bridge.lewd.tech%2F%3Faction%3Ddisplay%26bridge%3DXPathBridge%26url%3Dhttps%253A%252F%252Fmagazinelib.com%252Fpage%252F5%252F%26item%3D%252F%252Farticle%26title%3D.%252Fdiv%252Fheader%252F%252Fh3%252Fa%26content%3D%26uri%3D.%252Ffigure%252Fa%252F%2540href%26author%3D.%252Ffigure%252Fa%252Fspan%252Fimg%252F%2540data-lazy-src%26timestamp%3D.%252F%252Ffooter%252Fp%252Ftime%255B%2540class%253D%2522entry-date%2522%255D%252F%2540datetime%26enclosures%3D.%252Ffigure%252Fa%252Fspan%252Fimg%252F%2540data-lazy-src%26categories%3D.%252F%252Fdiv%255B%2540class%253D%2522entry-before-title%2522%255D%252Fp%252F%252Fa%255B2%255D%252Ftext%2528%2529%26format%3DAtom&feed_6=https%3A%2F%2Frss-bridge.lewd.tech%2F%3Faction%3Ddisplay%26bridge%3DXPathBridge%26url%3Dhttps%253A%252F%252Fmagazinelib.com%252Fpage%252F6%252F%26item%3D%252F%252Farticle%26title%3D.%252Fdiv%252Fheader%252F%252Fh3%252Fa%26content%3D%26uri%3D.%252Ffigure%252Fa%252F%2540href%26author%3D.%252Ffigure%252Fa%252Fspan%252Fimg%252F%2540data-lazy-src%26timestamp%3D.%252F%252Ffooter%252Fp%252Ftime%255B%2540class%253D%2522entry-date%2522%255D%252F%2540datetime%26enclosures%3D.%252Ffigure%252Fa%252Fspan%252Fimg%252F%2540data-lazy-src%26categories%3D.%252F%252Fdiv%255B%2540class%253D%2522entry-before-title%2522%255D%252Fp%252F%252Fa%255B2%255D%252Ftext%2528%2529%26format%3DAtom&feed_7=https%3A%2F%2Frss-bridge.lewd.tech%2F%3Faction%3Ddisplay%26bridge%3DXPathBridge%26url%3Dhttps%253A%252F%252Fmagazinelib.com%252Fpage%252F7%252F%26item%3D%252F%252Farticle%26title%3D.%252Fdiv%252Fheader%252F%252Fh3%252Fa%26content%3D%26uri%3D.%252Ffigure%252Fa%252F%2540href%26author%3D.%252Ffigure%252Fa%252Fspan%252Fimg%252F%2540data-lazy-src%26timestamp%3D.%252F%252Ffooter%252Fp%252Ftime%255B%2540class%253D%2522entry-date%2522%255D%252F%2540datetime%26enclosures%3D.%252Ffigure%252Fa%252Fspan%252Fimg%252F%2540data-lazy-src%26categories%3D.%252F%252Fdiv%255B%2540class%253D%2522entry-before-title%2522%255D%252Fp%252F%252Fa%255B2%255D%252Ftext%2528%2529%26format%3DAtom&feed_8=https%3A%2F%2Frss-bridge.lewd.tech%2F%3Faction%3Ddisplay%26bridge%3DXPathBridge%26url%3Dhttps%253A%252F%252Fmagazinelib.com%252Fpage%252F8%252F%26item%3D%252F%252Farticle%26title%3D.%252Fdiv%252Fheader%252F%252Fh3%252Fa%26content%3D%26uri%3D.%252Ffigure%252Fa%252F%2540href%26author%3D.%252Ffigure%252Fa%252Fspan%252Fimg%252F%2540data-lazy-src%26timestamp%3D.%252F%252Ffooter%252Fp%252Ftime%255B%2540class%253D%2522entry-date%2522%255D%252F%2540datetime%26enclosures%3D.%252Ffigure%252Fa%252Fspan%252Fimg%252F%2540data-lazy-src%26categories%3D.%252F%252Fdiv%255B%2540class%253D%2522entry-before-title%2522%255D%252Fp%252F%252Fa%255B2%255D%252Ftext%2528%2529%26format%3DAtom&feed_9=https%3A%2F%2Frss-bridge.lewd.tech%2F%3Faction%3Ddisplay%26bridge%3DXPathBridge%26url%3Dhttps%253A%252F%252Fmagazinelib.com%252Fpage%252F9%252F%26item%3D%252F%252Farticle%26title%3D.%252Fdiv%252Fheader%252F%252Fh3%252Fa%26content%3D%26uri%3D.%252Ffigure%252Fa%252F%2540href%26author%3D.%252Ffigure%252Fa%252Fspan%252Fimg%252F%2540data-lazy-src%26timestamp%3D.%252F%252Ffooter%252Fp%252Ftime%255B%2540class%253D%2522entry-date%2522%255D%252F%2540datetime%26enclosures%3D.%252Ffigure%252Fa%252Fspan%252Fimg%252F%2540data-lazy-src%26categories%3D.%252F%252Fdiv%255B%2540class%253D%2522entry-before-title%2522%255D%252Fp%252F%252Fa%255B2%255D%252Ftext%2528%2529%26format%3DAtom&feed_10=&limit=270&format=Atom -------------------------------------------- I'll use WGET here to request an rss feed from the bridge server. Point the --input-file option to magazineliburl for the URL to fetch. For example: --input-file=".\scrapes\magazineliburl" With all the querystring encoding it may try to double encode some args so we use the --no-iri option to tell WGET to not encode the args in the file. in CURL I think the --get --data-urlencode options cover this. I'll also tell WGET to save the output to an atom file, For example: -O ".\scrapes\magazinelib.atom" So my basic command line without any more fun options is: wget --no-iri --input-file=".\scrapes\magazineliburl" -O ".\scrapes\magazinelib.atom" Go look at: magazinelib.atom It should contain an RSS feed in atom format that any reader/aggregator can work. use a file:// url in your local reader or put it on a webserver. You can run the wget command in a script or as a cron job to regularly refresh this file, once or twice a day is enough. I applied the same technique to the site http://freemagazines.top -------------- freemagazinestopurl -------------- https://rss-bridge.lewd.tech/?action=display&bridge=FeedMergeBridge&feed_name=freemagazines.top+Last+10+Pages&limit=999&format=Mrss&feed_1=https%3A%2F%2Ffreemagazines.top%2Ffeed%3Fpaged%3D1&feed_2=https%3A%2F%2Ffreemagazines.top%2Ffeed%3Fpaged%3D2&feed_3=https%3A%2F%2Ffreemagazines.top%2Ffeed%3Fpaged%3D3&feed_4=https%3A%2F%2Ffreemagazines.top%2Ffeed%3Fpaged%3D4&feed_5=https%3A%2F%2Ffreemagazines.top%2Ffeed%3Fpaged%3D5&feed_6=https%3A%2F%2Ffreemagazines.top%2Ffeed%3Fpaged%3D6&feed_7=https%3A%2F%2Ffreemagazines.top%2Ffeed%3Fpaged%3D7&feed_8=https%3A%2F%2Ffreemagazines.top%2Ffeed%3Fpaged%3D8&feed_9=https%3A%2F%2Ffreemagazines.top%2Ffeed%3Fpaged%3D9&feed_10=https%3A%2F%2Ffreemagazines.top%2Ffeed%3Fpaged%3D10 ------------------------------------------------- With the WGET command: wget --no-iri --input-file=".\scrapes\freemagazinestopurl" -O ".\scrapes\freemagazinestop.rss" To create the RSS file freemagazinestop.rss Share and Enjoy.
Paste Settings
Paste Title :
[Optional]
Paste Folder :
[Optional]
Select
Select
Syntax Highlighting :
[Optional]
Select
Markup
CSS
JavaScript
Bash
C
C#
C++
Java
JSON
Lua
Plaintext
C-like
ABAP
ActionScript
Ada
Apache Configuration
APL
AppleScript
Arduino
ARFF
AsciiDoc
6502 Assembly
ASP.NET (C#)
AutoHotKey
AutoIt
Basic
Batch
Bison
Brainfuck
Bro
CoffeeScript
Clojure
Crystal
Content-Security-Policy
CSS Extras
D
Dart
Diff
Django/Jinja2
Docker
Eiffel
Elixir
Elm
ERB
Erlang
F#
Flow
Fortran
GEDCOM
Gherkin
Git
GLSL
GameMaker Language
Go
GraphQL
Groovy
Haml
Handlebars
Haskell
Haxe
HTTP
HTTP Public-Key-Pins
HTTP Strict-Transport-Security
IchigoJam
Icon
Inform 7
INI
IO
J
Jolie
Julia
Keyman
Kotlin
LaTeX
Less
Liquid
Lisp
LiveScript
LOLCODE
Makefile
Markdown
Markup templating
MATLAB
MEL
Mizar
Monkey
N4JS
NASM
nginx
Nim
Nix
NSIS
Objective-C
OCaml
OpenCL
Oz
PARI/GP
Parser
Pascal
Perl
PHP
PHP Extras
PL/SQL
PowerShell
Processing
Prolog
.properties
Protocol Buffers
Pug
Puppet
Pure
Python
Q (kdb+ database)
Qore
R
React JSX
React TSX
Ren'py
Reason
reST (reStructuredText)
Rip
Roboconf
Ruby
Rust
SAS
Sass (Sass)
Sass (Scss)
Scala
Scheme
Smalltalk
Smarty
SQL
Soy (Closure Template)
Stylus
Swift
TAP
Tcl
Textile
Template Toolkit 2
Twig
TypeScript
VB.Net
Velocity
Verilog
VHDL
vim
Visual Basic
WebAssembly
Wiki markup
Xeora
Xojo (REALbasic)
XQuery
YAML
HTML
Plaintext
Paste Expiration :
[Optional]
Never
Self Destroy
10 Minutes
1 Hour
1 Day
1 Week
2 Weeks
1 Month
6 Months
1 Year
Paste Status :
[Optional]
Public
Unlisted
Private (members only)
Password :
[Optional]
Description:
[Optional]
Tags:
[Optional]
Encrypt Paste
(
?
)
Create New Paste
You are currently not logged in, this means you can not edit or delete anything you paste.
Sign Up
or
Login
Site Languages
×
English
Português
Do you like cookies?
🍪 We use cookies to ensure you get the best experience on our website.
Learn more
I agree