Screaming frog - POS keeps crashing all the time not saving my data

[Rumbas]

I'm crawling a 100K+ pages site and the f-ing thing keeps crashing and not saving my data. I tried using some more memory allocation, but it will only allow 1gb other the crap wont start.

On a pretty new pc with lots of ram, solid state etc. but it SF just can't handle that many pages.

Any ideas?

[littleman]

Could Xenu do the job?  It runs great on my old hardware.

[Drastic]

Needs more available ram - sort that out and you should be good.

[Rumbas]

I tried Dras, Got 8G on the machine but when I try to edit the .ini file to go beyond 1024mb, it wont run. Setting it to 512 mb it runs fine for a while. Just increasing it byt 512 it runs twice as fast, but still keeps shutting down.
Well, thanks, I'll try again though.

[Rumbas]

Win! Figured out that I didn't have the 64bit version of Java installed! That helped and now the ram allocation is tops.
Thanks for pointing it out Dras!

>Xenu
Great tool, but does not give 100K titles and descriptions.

[JasonD]

64bit Java.

Nice and simple to fix. I am off to check mine now, thanks!

[Mackin USA]

I found this while surfing:

C:Program Files/Screaming Frog SEO Spider
- Locate the file called ScreamingFrogSEOSpider.l4j.ini
- Open this file with Notepad
- Locate the line -Xmx512M
- The '512' part denotes the memory allocation. Amend this to the RAM value you wish to allocate, for example –Xmx1024M for 1GB of memory allocation. If you input an allocation higher than what you have on your device, the SEO spider won't start.

[Drastic]

Awesome, glad you got it straightened out. You can throw me your first scrape after you've pillaged it. ;-)

[Rumbas]

>i found this

Right, Mackin that's the file.

>throw me
That depends on what you're going to do with 100K+ internal URL's from huge DK retailer

[Drastic]

Ah, I thought you were scraping for expireds.

[JasonD]

> Ah, I thought you were scraping for expireds

The most inefficient way I can possibly imagine to get them.

[Drastic]

>The most inefficient way I can possibly imagine to get them.

There are absolutely worse ways, Xenu for starters. I know of at least 2 guys who run multiple instances of SF across a number of dedis, and sell expireds for quite a profit. Nice enough for it to be a fat, full time yearly income.

[JasonD]

>Nice enough for it to be a fat, full time yearly income.

I'm biased of course, but I hope you're right as we move closer and closer to launch with TDN

[Drastic]

Doesn't matter to me, as far as SF, I do hope TDN does great! I just know it works well for a lot of people. I'm sure there are better solutions, just like so many so-called "web designers" today are nothing but wordpress monkeys. But hey, they end up with a nice product and the client is happy so who am I to judge.

[JasonD]

thank you and I agree about the other comments too.

I suppose when all is said and done, the technicalities don't matter - only the end outcome.