MJBot - The MJB Data Web Robot

Information about the MJB Data Search Robot, 'MJBot'

MJB Data have long been involved with Search Engine technology. We run our own special search indexes including the Cambridge Jobs Database, Property Database UK, and Cambridge's most popular property website, the Cambridge Property Database, and these, along with several other websites, use variants of the 'Crawler' software MJBot.

Robots.txt and crawling courtesy

MJBot is configured manually and does not currently read robots.txt files. As most websites that we crawl receive more traffic, and sometimes an increased Google PageRank, objections to MJBot are extremely rare. However, if you would prefer to block MJBot please email us.

Frequency

MJBot usually visits a predetermined set of pages on a website once per day, and the crawler activity is usually scheduled for the early hours of the morning in order to have a negligible impact on host systems.

[ 23-Oct-2011 ]