Robots.txt, mediawiki and Google Sitemap October 13, 2005
Posted by Andy Roberts in : learning, internet , add a commentI used to have my ukcider mediawiki excluded from most search engines through a robots.txt file which looked like this:
User-agent: * Disallow: /wiki/
but then I decided I’d like to have another go at allowing the Googlebot to index some of the really useful content which has been building up there recently, so I removed the robots.txt file for a few days and monitored carefully.
What appears to be happening is that the googlebot visits about once per day and spiders a little further down into the Wiki each day, but using up an ever increasing amount of bandwidth as it does so - not good. So the list of french cider producers can already be searched for, but the Asturian Campsites - not as yet.
My own webstats and research told me that Googlebot can get caught up in a wiki site, spidering all of the previous versions, page history, user contributions and so on, and if you are paying for the remote hosting then this needs to be avoided. So rather than disallow /wiki/ I’ve disallowed “oldid” and “contributions” for now, and maybe I’ll tweak it a bit later or go fishing for the definitive mediawiki (not pretty URLs) robots.txt configuration. Meanwhile in my travels, I came across a reference to Google sitemaps which should allow me to tame the over eager googlebot some more. I’ve included data to the effect that the site is updated weekly, which should help towards my goal of having deep-linked pages listed on search results without having all the bandwidth used up by spiders.
Googlebot is not the only search engine spider, there are many others ( such as the enigmatically named “inktomi slurp” it’s just that the Gb is probably the most important and also the most resource consuming.
Plumber July 22, 2005
Posted by Andy Roberts in : internet , add a comment
This is the bloglines plumber, so we are told, and he’s fixing the database. Plumbers are expensive, but cheaper than specialist sofware engineers I suppose. This is only to be expected, now that we have the “mass amateurisation of everything” but is it really surprising that he’s shrugging his shoulders like that?
All my blog are belong Jeeves July 19, 2005
Posted by Andy Roberts in : internet , 2commentsBloglines has decided to throw away all of my RSS subscriptions, all 200 plus many more of them, even the ones I never read.
I’ve been having trouble with net access too, not just in one place. It makes me wonder if perhaps one day we’ll all wake up and the internet won’t be there any more, or perhaps it will be electricity that goes first.
On my way to work thois morning I was thinking about how inept the terrorists are. The IRA were particularly bad it it, blowing themselves up without even meaning to. And this latest lot, well 4 suicides to take out 50+ civilian workers and disrupt the transport system for a few hours doesn’t create the most inspiring spectacle to show for all that planning does it? Not compared with flying an aeroplane full of fuel into a skyscraper. Sitting on a tube or bus with a backpack full of explosives shows real lack of imagination. If you wanted to create mahem and terror the best target would be the water supply. Hijack a few tankers full of the most toxic chemicals and dump them simultaneously into the reservoirs.
I was also wondering how many of the large proportion of London’s population who only gravitated here relatively recently are thinking about how and when they’ll be planning to move out somewhere else as soon as possible.
World map June 24, 2005
Posted by Andy Roberts in : internet , 2commentsHelping to test some new social software produces a world map.
The blue is where I’ve been and the orange is where I’d like to go according to what I’ve tagged so far. Some of the orange has overwritten blue though, because I’ve been to Queensland but I’d like to go to Tasmania.
update:
The site is now live, and open to anybody to help build - if interested just make your way over to 43places.com. Seeya.
Contacts June 23, 2005
Posted by Andy Roberts in : internet , add a comment
Flickr Ideas May 26, 2005
Posted by Andy Roberts in : internet , add a comment
(also posted to Flickr Ideas group)
Related Groups
I’d like to see a way of linking related groups together, so that when splinter groups are forked off from a main group, they can be kept track of.
Ideally, this would be done by having a Wiki page for each group, which the members can edit to add links to the subgroups and perhaps other information.
Discussions are great, but they are ephemeral. It would be nice to have somewhere linked to a group where persistent information and relationships can be accumulated.
The case I have in mind is the London group, which has now spawned several subgroups. The main group has 550 odd members and people have set up groups for London meetups, for Monthly topics, and now for several of the London Boroughs (districts)[1] ,[2] , [3]. I expect this kind of thing has already happened with large North American city groups, but I haven’t bothered to check.
Without some kind of mechanism for making the relationships between these groups explicit, and linking them together, I think there’s probably a danger that overlapping small and under-used groups could proliferate causing a dilution of the potential conversations and degradation of the flickr communities.
Googortal May 23, 2005
Posted by Andy Roberts in : internet , add a commentStephen Powell welcomes Google Personalised and supplies an interesting wishlist.
It looks like a Portal to me.
I don’t want to be locked in to anybody’s portal, not even Google’s. If I really needed to have ‘everything on one page’ then I’d build one myself, and if I didn’t know how to do that I could use a commercial blog or wiki which allowed me to add my own links and incorporate rendered RSS streams. All of this stuff is available now, or very soon, but the thing is, I just don’t get this drive to try and ‘contain’ the whole internet experience within the confines of one particular starting point. As far as I can tell it stems from an advertisment based business model which failed because people preferred to define their own homepages rather than have them imposed.
I’ve shifted quite a lot of my stuff onto web based services in the past year or so, so that I can synchronise between home and various workplaces easily, but I do often miss some of the functionality and efficiency of installed client programs.
I may switch back from Bloglines to a Client RSS reader, and I could really do with a program as powerful as Forte Agent for analysing my mailing list data. Gmail is great, but not as powerful as having the database accessible on your own disk with your own choice of tools.
Stephen’s list of three types of space presents the real challenge. the first and last are easy, but the middle one - “for me and my friends, I can control who has access” is not so well provided for on the net as yet. You could set up a ‘group’ within within a web service such as smartgroups, yahoogroups, googlegroups2 or maybe use something like jotspot but hen you are at the mercy of whichever provider you pick once again. I’m not sure how you can set up a completely independent but also private community without having to have somebody in charge, but then I suppose ‘me and my friends’ fits in with the benevolent dictator managment model so that’s OK.
Les Blogs Paris April 28, 2005
Posted by Andy Roberts in : internet , add a comment
Originally uploaded by Hans on Experience.
Blogs are really big in France, now.
According to government figures, half of all schoolchildren are bloggers, an estimated 3 million. Almost 2 million of them use Skyblog, a service operated by youth radio station Skyrock that is growing by around 600 new journals and 200,000 entries every day.
Many teens use mobile phones to post diary entries and pop-culture news flashes to their skyblogs on the move.
Link to: Les Blogs conference wiki
Google maps April 19, 2005
Posted by Andy Roberts in : internet , 3commentsI agree with Jonathan that Tom Smith is being excessively grumpy about Google maps . Coupled with Google local uk, this is a big improvement on multimap and streetmap.
My only complaint is that they have vanished my local railway station !!!
Exhibition and barn raising March 11, 2005
Posted by Andy Roberts in : ultraversity, learning, internet, Wiki , add a commentOnline Exhibition now open at
http://distributedresearch.net/blog/?page_id=75
Barn raising for Distributed Action Research - until March 15th at
http://distributedresearch.net/wiki/index.php/Barn_raising
(post backdated 12 months)
Thanks for reading Andy Roberts articles about internet
on Darnet
is an online professional who initiated DARnet 
