Home Business Forums - Online Business Forums

Home Business Forums for small business owners to discuss online home business
It is currently Mon Feb 19, 2018 2:49 am

A warm welcome to the forums!
We hope you will make this your home. Dave and Heidi, the founders of HomeBusinessOnline.com, are committed to helping you find all the information and resources you need to succeed in your home-based business. John is an excellent moderator of the forums and has an amazing amount of knowledge in small business.


-----> Hey, We Need Forum Moderators! <-----

Most Recent Forum Posts...

All times are UTC - 7 hours




Post new topic Reply to topic  [ 14 posts ] 
Author Message
 Post subject: Importance of robot.txt
PostPosted: Tue Sep 04, 2012 4:29 am 
Offline

Joined: Mon Jun 04, 2012 4:11 am
Posts: 60
Location: India
Importance of robot.txt ?

robots.txt in very important to business or company sites, you can use this file to restrict search engines to index your confidential part of you website.

_________________
engagedots.com
[SEO Signature Removed by Moderator. Please read forum signature rules and modify your signature to comply. Thanks.]


Top
 Profile  
 
PostPosted: Tue Sep 04, 2012 6:55 am 
Offline
User avatar

Joined: Thu Mar 15, 2007 10:29 am
Posts: 4804
Location: Melbourne AU
Provided that the individual search engines actually honour robots.txt files. Some don't — notoriously Alexa.com, which is a favourite of download thieves and other criminals.

If you have digital products online, don't rely on robots.txt or even obscure URLs to protect them... Alexa will post your download links to the world.

_________________
John Counsel — Global Moderator
The Marketing Doctor at The Profit Clinic
NEW to FORUM Marketing? Get better, faster results without breaking the rules.
FREE help at https://www.facebook.com/groups/ForumMarketingForBeginners/


Top
 Profile  
 
PostPosted: Tue Sep 04, 2012 7:38 am 
Offline
User avatar

Joined: Sun Sep 23, 2001 5:00 pm
Posts: 2288
Location: Utah, USA
There's also the other side of robots.txt to tell what you DO want to be indexed, i.e. your sitemaps for search engines.

_________________
Moderator of Home Business Forums
Follow Me on Twitter


Top
 Profile  
 
PostPosted: Fri Sep 07, 2012 10:55 pm 
Offline

Joined: Fri Sep 07, 2012 11:31 am
Posts: 14
By including robots.txt file to set your targeted pages that SE crawls for this increase your popularity.

_________________
Everyday problems that need inventions to solve from www.problemsolutions24.com
Dynamic Web Site USD 300 & Cheap E-commerce UpTo $850 www.shadmansoft.com


Top
 Profile  
 
PostPosted: Sat Sep 08, 2012 10:35 am 
Offline
User avatar

Joined: Thu Sep 06, 2012 11:47 am
Posts: 15
Location: Polgardi, Hungary
Here is a good robots.txt generator if you need one:

http://tools.seobook.com/robots-txt/generator/

_________________
OWN a Dot Com Business You Love: DIY 'scam-proof' Business Opportunity


Top
 Profile  
 
PostPosted: Mon Jan 06, 2014 2:12 am 
Offline
User avatar

Joined: Thu Mar 15, 2007 10:29 am
Posts: 4804
Location: Melbourne AU
The robots.txt is not mandatory — it will only be honoured by search engines that choose to do it.

Others, like Alexa.com, won't. Alexa will reveal your download pages and other supposedly hidden pages to the entire world.

_________________
John Counsel — Global Moderator
The Marketing Doctor at The Profit Clinic
NEW to FORUM Marketing? Get better, faster results without breaking the rules.
FREE help at https://www.facebook.com/groups/ForumMarketingForBeginners/


Top
 Profile  
 
PostPosted: Mon Jan 06, 2014 6:37 am 
Offline

Joined: Mon Jan 06, 2014 5:58 am
Posts: 6
Location: india
not a pro at this. but what i know this allow crawler to come on our website and it has several parameters which are decided by site administrator..


Top
 Profile  
 
PostPosted: Thu Feb 19, 2015 9:44 am 
Offline

Joined: Fri Jan 02, 2015 11:30 am
Posts: 11
When a search engine crawler comes to your site, it will look for a special file on your site. That file is called robots.txt and it tells the search engine spider, which Web pages of your site should be indexed and which Web pages should be ignored.

_________________
auto house review
auto house scam


Top
 Profile  
 
PostPosted: Thu Feb 19, 2015 12:15 pm 
Offline
User avatar

Joined: Thu Mar 15, 2007 10:29 am
Posts: 4804
Location: Melbourne AU
Reminder:

SOME search engine crawlers will respect your robots.txt instructions.

Other WON’T.

_________________
John Counsel — Global Moderator
The Marketing Doctor at The Profit Clinic
NEW to FORUM Marketing? Get better, faster results without breaking the rules.
FREE help at https://www.facebook.com/groups/ForumMarketingForBeginners/


Top
 Profile  
 
PostPosted: Tue Dec 13, 2016 12:39 am 
Offline

Joined: Fri Dec 02, 2016 11:20 pm
Posts: 8
U use Robot.txt for Invisible a Page on Search Engine

_________________
Asthijivak | Fat Cutter


Top
 Profile  
 
PostPosted: Mon Feb 20, 2017 10:54 pm 
Offline

Joined: Mon Feb 20, 2017 10:08 pm
Posts: 2
Using robot.txt has many benefits. A while back, within a year or so, Google and Yahoo both arose and said they would now look to the robots.txt file to find the link to your sitemap.xml file. The sitemap.xml file is important because, it lists all the pages to your site in a nice robot friendly way. Their request actually is serious. They use up a lot of bandwidth searching for things that they can't or won't index. Things such as Images, javascript, dead files and a huge array of different things people keep on their servers.

_________________
Buy Mangoes Online | Cake Decorating Classes


Top
 Profile  
 
PostPosted: Mon Feb 20, 2017 11:11 pm 
Offline

Joined: Mon Feb 20, 2017 10:08 pm
Posts: 2
profitclinic wrote:
Reminder:

SOME search engine crawlers will respect your robots.txt instructions.

Other WON’T.



and what you do with search engines which totally ignore the robots.txt file?? :)

_________________
Buy Mangoes Online | Cake Decorating Classes


Top
 Profile  
 
PostPosted: Tue Feb 21, 2017 2:11 pm 
Offline
User avatar

Joined: Thu Mar 15, 2007 10:29 am
Posts: 4804
Location: Melbourne AU
mickeyy wrote:
…and what do you do with search engines which totally ignore the robots.txt file? :)


You plan and execute accordingly.

For example, Alexa totally ignores robot.txt files — so it's the digital product thief's friend.

Alexa will reveal the location of your product download pages to anyone looking for them.

So you have to ensure that your digital product download pages are adequately protected from Alexa.com.

_________________
John Counsel — Global Moderator
The Marketing Doctor at The Profit Clinic
NEW to FORUM Marketing? Get better, faster results without breaking the rules.
FREE help at https://www.facebook.com/groups/ForumMarketingForBeginners/


Top
 Profile  
 
PostPosted: Tue Dec 19, 2017 4:06 am 
Offline

Joined: Thu Dec 14, 2017 6:24 am
Posts: 17
Hi

Robots.txt is a text file that you want to hide some web pages from Google crawlers. If you do not want to show some of your pages to Google, you can put those pages in a Robots.txt file.

Hope this helps you.


Top
 Profile  
 
Display posts from previous:  Sort by  
Post new topic Reply to topic  [ 14 posts ] 

All times are UTC - 7 hours


Who is online

Users browsing this forum: No registered users and 2 guests


You cannot post new topics in this forum
You cannot reply to topics in this forum
You cannot edit your posts in this forum
You cannot delete your posts in this forum

Search for:
Jump to:  
cron
Powered by phpBB® Forum Software © phpBB Group