Extract Links | how to scrape website urls | Python + Scrapy Link Extractors

# What are Link Extractors? Link Extractors are the objects used for extracting links from web pages using scrapy.http.Response objects. "A link extractor is an object that extracts links from responses." Though Scrapy has built-in extractors like scrapy.linkextractors import LinkExtractor, you can customize your own link extractor based on your needs by implementing a simple interface. == chapters === 0:00 Intro 2:15 Scrapy Docs 4:39 Import Link Extractor 08:25 code to delete old output 12:38 convert link to str 17:03 output 22:05 looking at other people's code 25:30 urllib.parse Useful links: Code on Gitub: 🌏 https://github.com/RGGH/Scrapy18 Scrapy articles and code: _________________________________________________ 🌏 https://www.programcreek.com/python/example/106165/scrapy.linkextractors.LinkExtractor 🌏 https://github.com/scrapy/scrapy/blob/2.5/docs/topics/link-extractors.rst 🌏https://github.com/scrapy/scrapy/blob/master/scrapy/linkextractors/lxmlhtml.py 🌏https://w3lib.readthedocs.io/en/latest/_modules/w3lib/url.html 🌏https://docs.scrapy.org/en/latest/_modules/scrapy/linkextractors/lxmlhtml.html#LxmlLinkExtractor.extract_links 🌏https://docs.python.org/3/library/urllib.parse.html At the end of the video I also look at ProgramCreek which is a great place to see how others have used a given module that you may be interested im. 🌏 https://www.programcreek.com/python/example/106165/scrapy.linkextractors.LinkExtractor Visit redandgreen blog for more Tutorials ========================================= 🌏 http://redandgreen.co.uk/about/blog/ Subscribe to the YouTube Channel ================================= 🌏 https://www.youtube.com/c/DrPiCode Follow on Twitter - to get notified of new videos ================================================= 🌏 https://twitter.com/RngWeb Buy Dr Pi a coffee (or Tea) ☕ https://www.buymeacoffee.com/DrPi Proxies ================================================= If you need a good, easy to use proxy, I was recommended this one, and having used ScraperAPI for a while I can vouch for them. If you were going to sign up anyway, then maybe you would be kind enough to use the link and the coupon code below? You can also do a full working trial first as well, (unlike some other companies). The trial doesn't ask for any payment details either so all good! 👍 🌏 10% off ScraperAPI : https://www.scraperapi.com?fpr=ken49 ◼️ Coupon Code: DRPI10 (You can also get started with 1000 free API calls. No credit card required.) Thumbs up yeah? (cos Algos..) #webscraping #python #scrapy