Question Not a bug or feature request, just some feedback :)
- Status
neounix
Customer
Actually, this type of behavior is seen when the URL structure of a site changes.
When you used DBSEO for a while, you will see that the crawl stats will go back down; and then, if you disabled DBSEO and went back to the default forum URL format, you will see a similar spike again, especially if you submit a new site map of the new URLs.
When we dumped VBSEO and went back to the original forum links (because DBSEO was not available and we did not want to write our own mod for this), we saw the same "big increase" as GoogleBot rediscovered all the new URLs, even though the "new URLS" were just the original vB default (non-SEO'ed URLs).
When you used DBSEO for a while, you will see that the crawl stats will go back down; and then, if you disabled DBSEO and went back to the default forum URL format, you will see a similar spike again, especially if you submit a new site map of the new URLs.
When we dumped VBSEO and went back to the original forum links (because DBSEO was not available and we did not want to write our own mod for this), we saw the same "big increase" as GoogleBot rediscovered all the new URLs, even though the "new URLS" were just the original vB default (non-SEO'ed URLs).
neounix
Customer
Interesting.... so what is the exact reason you think your GoogleBot crawl stats "went up".... since the URL structure did not change, perhaps it is because all your HTML meta data (keywords and descriptions) changed and GoogleBot recognized the changes and increase their crawl rate?
We have a custom mod we wrote in place where we actually measure this (we update each thread and post and tag when Google or BingBot "hits it"), and we see that our site is completely reindexed (threads) every 3 to 4 days, three days if we make changes and submit a new site map; and typically around a week if nothing has changed.
Also after I wrote our own code / mod to extract keywords from threads and update the meta data, we saw increases in crawl rates as well.
When we install DBSEO on our main server, I will "zero out" all our counters and we will see how long it tags to reindex the site. We keep all these stats on a daily basis, so we can look at the data and see the indexing rate on a weekly, daily, or hourly basis.
We have a custom mod we wrote in place where we actually measure this (we update each thread and post and tag when Google or BingBot "hits it"), and we see that our site is completely reindexed (threads) every 3 to 4 days, three days if we make changes and submit a new site map; and typically around a week if nothing has changed.
Also after I wrote our own code / mod to extract keywords from threads and update the meta data, we saw increases in crawl rates as well.
When we install DBSEO on our main server, I will "zero out" all our counters and we will see how long it tags to reindex the site. We keep all these stats on a daily basis, so we can look at the data and see the indexing rate on a weekly, daily, or hourly basis.
neounix
Customer
yes, it's hard to tell....
When we removed VBSEO our avg crawl time went from 248ms a page to around 120 - 150 ms a page; but our page rank dropped from 5 to 4, so i don't think performance is that critical, LOL.. if it was our PR should have gone up, not down, LOL !!!
I think GoogleBot recrawls based on site changes more than speed of download; but I don't know either.. but I can say that removing VBSEO (nothing to do with DBSEO, just turning off VBSEO and doing the 301 redirects), our site became twice as fast, and PR dropped 25 %, hahaha
That does lead me to believe, however, that it is important to keep links back to your site (for established sites), working without 301s; and perhaps the keywords in the URL are important as well; our custom meta code is better than VBSEOs, and that code did not help increase our PR. So, my best guess is that the links back to the site are the most important in our case; and the 301s from the VBSEO URLs to the default URLS from those link backs were the main cause of the PR drop.
When we removed VBSEO our avg crawl time went from 248ms a page to around 120 - 150 ms a page; but our page rank dropped from 5 to 4, so i don't think performance is that critical, LOL.. if it was our PR should have gone up, not down, LOL !!!
I think GoogleBot recrawls based on site changes more than speed of download; but I don't know either.. but I can say that removing VBSEO (nothing to do with DBSEO, just turning off VBSEO and doing the 301 redirects), our site became twice as fast, and PR dropped 25 %, hahaha
That does lead me to believe, however, that it is important to keep links back to your site (for established sites), working without 301s; and perhaps the keywords in the URL are important as well; our custom meta code is better than VBSEOs, and that code did not help increase our PR. So, my best guess is that the links back to the site are the most important in our case; and the 301s from the VBSEO URLs to the default URLS from those link backs were the main cause of the PR drop.
- Status