Is it possible to write Rust code that is as compact and expressive as Python?

Question

For example, consider a simple web crawler using the spider library in Rust that extracts all links from a website with just a few lines of code:

Description:
Here’s an example Rust code snippet using the spider crate:

use spider::tokio;
use spider::website::Website;

#[tokio::main]
async fn main() {
let mut website: Website = Website::new("https://spider.cloud");

website.crawl().await;

let links = website.get_links();

for link in links {
println!("- {:?}", link.as_ref());
}
}

With just this code, you can crawl a website and extract all the links on the server. The question is whether Rust can achieve the same level of conciseness and expressiveness as Python for tasks like this.

Answer 1

# yes it's a good idea to learn Rust. it's an awesome language. but Rust is not about compact code. it's about safe and fast code.

# use Rust where it actually have the technical advantage. maybe for rapidly processing the 'huge' data you already gathered.

# but webcrawling is not the case where it's decisively better than other languages. the speed is limited by slow operations like network requests rather than the speed of the CPU processing the data.

# this Python code will get you the result faster than the amount of time you need to compile your Rust code:

import requests
from bs4 import BeautifulSoup

url = 'https://spider.cloud'
reqs = requests.get(url)
soup = BeautifulSoup(reqs.text, 'html.parser')

for link in soup.find_all('a'):
print(link.get('href'))

Answer 2

And with that code I don't think I can exceed the three minutes it takes to collect price images from all of Walmart, approximately 5,000 products, in three minutes, and I don't think that Python code does either asynchrony or parallelism.And at Walmart.com we have your code with 69 links collected, vs my rust code 893 links approx, it seems the code has no comparison in any sense but I appreciate your comment the idea if you buy is to do exactly the same

Answer 3

it's good.
i have nothing against using Rust in this case.
one thing to consider is crawling a site with multiple concurrent requests might increase the chance of getting blocked. it makes your bots very visible. being discreet and unnoticeable when collecting data on other people's site is part of the game plan.

Answer 4

Of course, excellent observation and very good, you always have to take it into account, rather I don't know if you already know the trick to know what can be done or how to do it and respect conditions and ethics, trick (https://domain.com/robots.txt) basically any page or domain, one manually adds robots.txt and there appears what is allowed or if you are blocked by a pseudo DDOS attack

Answer 5

thanks but my compilation codes on my pc do not last more than 1 minute, the most loaded with crates is used in a few 10 seconds and it is already copied, whether with burn or candle or Tokyo, etc., but great, your opinion is respected but it is noticeable that you do not know much about Rust since it has real concurrency and parallelism, which in the end gives the limit would be ram cpu bandwidth, and the logic of the program without extra layers like Python but great I'm just asking to add Rust, and demonstrate that Rust is all in one that manages to be more compact than Python in many cases that is real, just as it can be extremely bare metal it is also as it is intermediate as most would use it, and it has technical capabilities mathematical guarantees that no other language has but we are working to make backends in Rust for Python