On Oct 14, 2:29 am, a2avia...@gmail.com wrote:
> I used to use a program called Pagesucker to snarf sites. It was
> shareware.
SiteSucker and PageSucker grab the index and thats it. Even with
robots.txt ignored. The apple2clones site was a dynamic driven site,
most of the content on it was also from other sources. So those other
sources are mostly still out there too.
Doing Page/SiteSucker anymore seems to not work on many sites anymore.
They come back right away as done.
That is because they use Drupal or some other community board type
software. Apple2Clones.com was done in Drupal.