+ 5

Web scraping

Hello friends! Last few days I was working on developement of some simple scraping codes in python. I was thinking if its possible to make some script challenge where lot of coders can join, post their codes and discuss about given solutions? Other thing, is there any programer who did some web scripts that is willing to share it with us here? I will be very pleased. I am asking this since data extraction and web scraping is very interesting branch of programming and I feel like we don't give enough attention to it!

27th Aug 2018, 9:18 AM
Luka Vasiljevic
Luka Vasiljevic - avatar
4 Answers
+ 11
I made a movie web scraper for kodi with python. It uses one scraper to pull the movie informations and 2 scrapers to scrape the embedded source links. There’s only two scrapers for the source links for now. But I could make 5. Make each run in a new thread and have one powerfull scraper.
27th Aug 2018, 9:36 AM
Toni Isotalo
Toni Isotalo - avatar
+ 9
A year ago, I created a profile list for a Sololearn contest, by web scraping the Sololearn profiles. https://code.sololearn.com/WrHot3955gBr/?ref=app
27th Aug 2018, 10:07 AM
Calviղ
Calviղ - avatar
+ 7
i watch youtube tuturial videos a lot, there are so many Professional developers making helpful stuffs for free, but my wifi is crazy and I dont want to load the videos every time to waste data. so its not much but a life saver shell script to download youtube videos for me step 1. $sudo apt-get install youtube-dl step 2: save the shell script written below in a folder where u want videos with file name dwnld. sh step 3: in same folder compile a list of links of all youtube videos you wanna download in a single go with file name links. txt step 4: run the script by typing . /dwnld.sh it basically saves me from doing $youtube-dl - f 22 video _ link (enter) hundred times and waiting till one file downloads to start for another I do this everytime when I see a tuturial series, if anyone has better idea please suggest, coz right now the only boring task is copy paste the links. https://code.sololearn.com/c0u56B1Z2wf5/?ref=app
27th Aug 2018, 11:16 AM
Morpheus
Morpheus - avatar
+ 4
Not mine but someone else's code. https://code.sololearn.com/c9YHb9DfYHUn/?ref=app When i was learning about scraping using nokogiri in ruby i used the below resource among others to learn. https://thaiwood.io/screen-scraping-with-a-saw-a-nokogiri-tutorial-with-examples/
27th Aug 2018, 12:34 PM
Lord Krishna
Lord Krishna - avatar