Answer 1

From my experience -- the best way is when it works! :)
Scrapy is great as it handles most of webscraping successfully, albeit can be a bit slow sometimes. But if real-time speed is not your concern, it is very efficient.