Warrior General Games

Now that forums are accessible without being logged in how about letting webcrawlers index it all. It will potentially drive more traffic to the site and allow us a forum search without having the stress on the servers.

I think the web crawlers were in here the other morning. The forums were down/very slow for a while - seems like a memory leak or something but anyway... while I was looking at that, I checked a few things and I don't see anything that's keeping the spiders out.

It'll take google a few days to get the pages it has crawled into the index, and then a month or so to make them really rank well. And based upon what happened yesterday and the night before, I would expect it to take a good long while to get a large number of forum pages - but it should happen without them doing anything. Since Google has been blocked improperly for several years - it might actually take a bit longer to figure everything out - but it should happen.

TIP to the DEVs: There are a lot of useless pages in the index - replays especially. There are only a limited number of pages that Google will index per site, so if you want to encourage the forum pages - block the replay pages in the robots.txt file.

Yeah I would assume part of making the forums public was to allow for easier searching.

Originally posted by Ali Khaman
TIP to the DEVs: There are a lot of useless pages in the index - replays especially. There are only a limited number of pages that Google will index per site, so if you want to encourage the forum pages - block the replay pages in the robots.txt file.

Seems to be a good tip. Just ran a search on something w/ google for a test (just using a keyword and site:goallineblitz.com). Replays gunked up my results and I only ended up with a post from 2008, lol.