»Discuss It.(1)

The BLEXBot crawler

Sharer:adminCategory:Vertical search
Score: (7)PR:PR:PR:0
Alexa:Alexa Rank:871022(871022)Popularity:11
Clicks: 166 viewsTags:crawler BLEXBot Robot
Website:http://webmeup-crawler.com
Summary:The BLEXBot crawler is an automated robot that visits pages to examine and analyse the content, in this sense it is similar to the robots used by the major search engine companies.
SNS:
Share to facebook Share to twitter Share to linkedin
»Improve It.

Website Details

The BLEXBot crawler is an automated robot that visits pages to examine and analyse the content, in this sense it is similar to the robots used by the major search engine companies.
The BLEXBot crawler is identified by having a user-agent of the following form:
Mozilla/5.0 (compatible; BLEXBot/1.0; +http://webmeup-crawler.com/)

The BLEXBot crawler can be identified by the user-agent above. If you are suspicious about requests being spoofed you should first check the IP address of the request and make a reverse DNS lookup to see its domain name via appropriate tools - it should point to one of the sub-domains of *.webmeup.com.

We care about your site's performance and will never hurt it!
BLEXbot is a very site-friendly crawler. We made it as "gentle" as possible when crawling sites: it makes only 1 request per 3 seconds, or even less frequently, if another crawl delay is specified in your robots.txt file. BLEXbot respects rules you specify in your robots.txt file.

If any problems arise, they may be due to peculiarities of your particular site, or a bug on another site linking to you. Therefore, we would like to ask you, if you noticed any problem with BLEXbot, please report it to [email protected] We will quickly make unique settings for your particular site, so that the crawling will never affect your site's performance.

Why is it crawling my site
BLEXBot assists internet marketers to get information on the link structure of sites and their interlinking on the web, to avoid any technical and possible legal issues and improve overall online experience. To do this it is necessary to examine, or crawl, the page to collect and check all the links it has in its content.

If the BLEXBot Crawler has visited your site, this means that links have never been collected and tested on that page before or needed to be refreshed. For this reason you will not see recurring requests fro m the BLEXBot crawler to the same page.

The Crawler systems are engineered to be as friendly as possible, such as limiting request rates to any specific site (BLEXBot doesn't make more than one hit per 3 seconds), automatically backing away if a site is down or slow.

Blocking with robots.txt
Firstly note that BLEXBot is:

Collecting only publicly available information that can be accessed by any random visitor. In case you think the Crawler collects some sensitive information please remove it fro m public access.

Cannot overload your site and do any harm to it - BLEXBot is designed to be very polite and it can make only 1 hit per 3 seconds max. Besides, you can easily slow BLEXBot (and any other robot / crawler which takes directions fro m the robots.txt file that should be on your site).

Does not read, parse, collect or store any information fro m your site but the links fro m your pages. This refers to any texts, graphical or video materials or anything else on your pages.
With a robots.txt file you may block the BLEXBot Crawler fro m parts or all of your site or slow it, as shown in the following examples:

Block specific parts of your site:
User-agent: BLEXBot

Disallow: /private/

Disallow: /messages/

Block entire site:
User-agent: BLEXBot

Disallow: /

Slow the Crawler:
User-agent: BLEXBot

Crawl-delay: 10

Attention: As soon as you make changes to your robots.txt, please give the crawler up to 10 minutes to fully stop crawling your website. This is caused by the fact that some pages might be in the processing queue already, so we cannot guarantee the crawler will be able to stop immediately. However, it should fully stop crawling after 10 minutes max.

For a general introduction to the robots.txt protocol, please see http://www.robotstxt.org. Please also see the Wikipedia article for more details and examples of robots.txt rules.

Contact Us

All that said, we of course take any request to desist crawling any site, or parts of a site, or any other feedback on the Crawler operations seriously and will act on it in a prompt and appropriate manner.

If this is the case for you please don't hesitate to contact us at [email protected] and we will be happy to exclude your site, or otherwise investigate immediately.
Welcome to 0430 website library
Previous website: AMiner (aminer.org)
Current Recommended:: The BLEXBot crawler
Next website: MJ12Bot :Majestic crawler


Declared: This info is provided by user, please retain the source, source:http://en.0430.com/us/web238644/。
The BLEXBot crawler Thank you for your support and sharing!

»If you think'The BLEXBot crawler' is less,You Can Collaboration Update it.
Now, 1 Person Post Review About 'The BLEXBot crawler'
N a m e: * Optionally, blank is Anonymous.
Content:
Characters left: * Press Ctrl + Enter to post.

Tips: Comments submitted only once, please wait patiently in order to display.
1 floor The BLEXBot crawler [*.*.*.*] published in 2020/01/06/ 08:31:00
Welcome you to visit our website!
The BLEXBot crawler is an automated robot that visits pages to examine and analyse the content, in this sense it is similar to the robots used by the major search engine companies.

» What about The BLEXBot crawler?

The Recommend to Website Of Vertical search Website Directory

0430.com offered in: English 中文(简体) 中文(繁體) 日本語 Español
About Us | Contact Us | Submit a web | Disclaimer | Friendly Link | Feedback | Categories | Principles | Add to Favorite
Copyright 2004-2020 By en.0430.com
Worldwide Library - 24-four hours online free directory - make sites closer to our life!
Disclaimer: This webistes is taken from the network by user, please refer to identify themselves when using, 0430 does not assume any responsibility.