Azekeil (azekeil) wrote,
Azekeil
azekeil
  • Mood: shocked

Disturbing!

As a direct result of my last post, I've discovered LJ has about 10 'spiders' monitoring new posts for images and then grabbing them from the servers they're hosted on. That means there are at least 10 bots downloading every new post; the amount of traffic this generates must be considerable! I wonder if this is an LJ-provided function, or if they're individually going to grab their own copies of the pages. Now I think about it, it MUST be LJ-provided, surely, because I don't know of a function or way to grab all new pages.

UPDATE: Apparently LJ does support the use of bots, and even hosts XML showing the URLs of the latest X pictures. Mystery solved, although it's a little anti-social for people just to copy and host the PHP code to do this. Kills home web servers for a while...
Subscribe
  • Post a new comment

    Error

    switch
    default userpic

    Your reply will be screened

    Your IP address will be recorded 

    When you submit the form an invisible reCAPTCHA check will be performed.
    You must follow the Privacy Policy and Google Terms of use.
    Help
  • 10 comments