Would you rather build a web crawler with PHP or Python? And why?

Question

well the title is explanatory, so what are the reasons that will influence your choice?
python has scrappy,
PHP has simple html_dom parse which you can include to your bot script file!

Answer 1

It depends a bit one the nuances. PHP is a language mainly used to develop websites (to present them), whereas Python is used for both programming websites and programming other applications. Thus, it depends on how application-like the scraping will be or how much it will be included into the (presentational) website.

Python is well-suited for scraping tasks as it has a lot of third party libraries (Scrapy, BeautifulSoup, lxml, request, ...) and also built in libraries for requesting URLs. With PHP you will have a harder life requesting other websites. As said, it is mainly focused on building an own website.

Also it of course depends on your skill level in each language.

I'd split the application into two parts anyway:

presentation logic (your own website)
scraping logic (scraping other websites)

Both will be connected through a database. The scraping logic collects information and stores it into the database. The presentation logic takes information from the database and displays it to the user as HTML.

For the scraping part, you then want to prefer Python over PHP. And for the presentation part, it actually does not matter and is a matter of preference or knowledge.

Answer 2

Welcome bro!!
Good luck 💙

Answer 3

Meru Mario Egie Glad to help you..
recommend you to check that:
https://stackoverflow.com/questions/372885/how-do-i-connect-to-a-mysql-database-in-JUMP_LINK__&&__python__&&__JUMP_LINK

Answer 4

Here is a similar question on CRAWLER:
https://www.sololearn.com/Discuss/1172526/what-is-the-best-and-fastest-programming-language-for-web-scraping-

Answer 5

thanks man

Answer 6

Great @Baraa thanks. but is there a decent way to connect to MySQL database and query it with Python?

cause your suggestion is actually my plan but I know less of how flexible I can use python to scrap then send the info straight to a database! php can handle the website back end and other functionalities l!

Answer 7

thanks bill