Channel Register

Comments on: Publishers punt new web crawler blocking standards

Or... 

By Anonymous Coward Posted Thursday 29th November 2007 12:40 GMT

Pirate

Just make a really horrible puke coloured website so nobody will bother reading the conent - like this: http://www.the-acap.org/

Regarding Copiepresse 

By Mark Posted Thursday 29th November 2007 13:31 GMT

They also sued Google when they stopped indexing their site.

Missed that bit out. Maybe Orlowski has to agree your copy before it can be printed...

Re: Regarding Copiepresse 

By Chris Williams Posted Thursday 29th November 2007 14:39 GMT

(Written by Reg staff.)

Erm, it's not in the copy because it's not relevant.

I mention the Copiepresse-Google News case as one of a few examples of publishers worried about how their content is used, which is why the APAC project was started - relevant.

The same publisher sues because the dominant search engine doesn't index them in tit for tat action - interesting, but irrelevant to this story.

See you at the tinfoil hat shop, anyway.

tinfoil not wanted 

By Mark Posted Thursday 29th November 2007 15:19 GMT

I *like* to see how people are trying to control me!

Solution looking for a problem? 

By Gareth Posted Thursday 29th November 2007 15:44 GMT

Thumb Down

Doesn't HTTP already support an "expires" header? Can't a page be unpublished from a site or moved into an authenticated area of the site when it should no longer be visible to search engines?

robots.txt is a small, elegant, simple solution for keeping crawlers away from non-content areas which still need to be publically accessible (ie. Javascript files, site templates, etc).

This seems like rather a lot of committee-designed overly complex, redundant cruft which is forcing the hand of the "old media" way of doing things.

Re: Solution looking for a problem 

By Mark Posted Thursday 29th November 2007 17:05 GMT

The problem was that the content producers didn't want to go to the effort of understanding and obeying the current standards for the internet (such as, if your webserver is asked for a page, your webserver handing it over means you're allowing the recipient to copy it).

They want all the web indexers to change THEIR stuff to make their life easier.

Yeah, yeah, been there before 

By DV Henkel-Wallace Posted Thursday 29th November 2007 17:06 GMT

Boffin

Err, ACAP was defined in RFC-2244 exactly 10 years ago in November 1997?

I predict today's ACAP will be just as popular and effective as ACAP was 10 years ago.