https://blog.mojeek.com/2022/10/mojeek-now-six-billion-pages.html
Really appreciate your following of the alternatives Brad, cheers.
Some geek tech, I'd been working on recalculating our pagerank aka gravity, the calc was too big this time to do on one server, so I'd rejigged it to run on multiple servers. Interesting quirks is the template-esque links like WP templates and heavy interlinked sites. Our last run was on 6.08 billion documents and took a week.