You are using an out of date browser. It may not display this or other websites correctly.
You should upgrade or use an alternative browser.
You should upgrade or use an alternative browser.
robots.txt
- Thread starter Timon
- Start date
Dr.Eftekasy
New Member
yes it si correct but must not put the Allowed direction coz the search engens open it Automaticly so put the unAllowed directions only in this form
# Allow Archiver
User-agent: ia_archiver
Allow: /
# All Agents
User-agent: *
Disallow: /forum/clientscript/
Disallow: /forum/cpstyles/
Disallow: /forum/customavatars/
Disallow: /forum/customprofilepics/
Disallow: /forum/images/
Disallow: /forum/includes/
Disallow: /forum/info/
Disallow: /forum/install/
Disallow: /forum/signaturepics/
Disallow: /forum/*.php
# Sitemap
Sitemap: http://www.forum.com/forum/sitemap_index.xml.gz
# Allow Archiver
User-agent: ia_archiver
Allow: /
# All Agents
User-agent: *
Disallow: /forum/clientscript/
Disallow: /forum/cpstyles/
Disallow: /forum/customavatars/
Disallow: /forum/customprofilepics/
Disallow: /forum/images/
Disallow: /forum/includes/
Disallow: /forum/info/
Disallow: /forum/install/
Disallow: /forum/signaturepics/
Disallow: /forum/*.php
# Sitemap
Sitemap: http://www.forum.com/forum/sitemap_index.xml.gz
what about the below line
i blocks all .php file????then how google crawl links????
Disallow: /forum/*.php
i blocks all .php file????then how google crawl links????
blackknights
New Member
Here's what I use and as you can see I block a lot of area's due to I don't want it spider-ed. I have re-moved a lot from this just so you can see some area's you may want to consider. Also remember when you up-load to name it robots not robot
Code:
User-agent: *
Disallow: /
Disallow: /ajax.php
Disallow: /arcade.php
Disallow: /attachment.php
Disallow: /calendar.php
Disallow: /editpost.php
Disallow: /global.php
Disallow: /inlinemod.php
Disallow: /joinrequests.php
Disallow: /login.php
Disallow: /member.php
Disallow: /memberlist.php
Disallow: /misc.php
Disallow: /moderator.php
Disallow: /newattachment.php
Disallow: /newreply.php
Disallow: /newthread.php
Disallow: /poll.php
Disallow: /printthread.php
Disallow: /private.php
Disallow: /profile.php
Disallow: /register.php
Disallow: /report.php
Disallow: /reputation.php
Disallow: /search.php
Disallow: /sendmessage.php
Disallow: /showgroups.php
Disallow: /subscription.php
Disallow: /threadrate.php
Disallow: /usercp.php
Disallow: /usernote.php
Disallow: /admincp/
Disallow: /archive/
Disallow: /ads/
Disallow: /attachments/
Disallow: /cgi-bin/
Disallow: /clientscript/
Disallow: /cron.php
Disallow: /cpstyles/
Disallow: /customavatars/
Disallow: /customeprofilepics/
Disallow: /faq.php
Disallow: /gab_redirect.php
Disallow: /geek/
Disallow: /includes/
Disallow: /install/
Disallow: /ioncube/
Disallow: /image.php
Disallow: /modcp/
Disallow: /newthread.php
Disallow: /newreply.php
Disallow: /online.php
Disallow: /postings.php
Disallow: /printthread.php
Disallow: /payments.php
Disallow: /profile.php
Disallow: /private.php
Disallow: /signaturepics/
Disallow: /subscription.php
Disallow: /sendmessage.php
Disallow: /testvb/
Disallow: /tags/
Disallow: /threadrate.php
Disallow: /*goto=
Disallow: /*mode=
Disallow: /*&pp=
Disallow: /*postcount=
Disallow: /*daysprune=
Disallow: /*&sort=
Disallow: /Timon said:iam using this robots.txt..its correct for my forum???????? i upload it in root of the forum
see in the top you allow more .php but at down you bloked all .php
i think this mistake robots can't write the your allow .php coz in down you block them
i would disable all .php and all directories for sitemap when generate, generate your sitemap from forum archive url.
An example: www.myforum.com/forums/archive/
get these urls (should be html only) or some generator allow you to save as xml, upload to your forum root and link sitemap.xml to google webmaster.
An example: www.myforum.com/forums/archive/
get these urls (should be html only) or some generator allow you to save as xml, upload to your forum root and link sitemap.xml to google webmaster.
bluescorpion
New Member
robots.txt goes in the webroot, not the forum root.
Lose the Allows, search engines don't care about allow the will spider anything and every thing that is not specifically disallowed.
You should take a hard look at BlackKnights robot.txt and see how you can use it for your site. He is running in the root but you can easily modify it to use in /forum. He has also optimized to remove some to the duplicate content that yours will not.
HTH
Lose the Allows, search engines don't care about allow the will spider anything and every thing that is not specifically disallowed.
You should take a hard look at BlackKnights robot.txt and see how you can use it for your site. He is running in the root but you can easily modify it to use in /forum. He has also optimized to remove some to the duplicate content that yours will not.
HTH
blackknights
New Member
Thanks for the comment BlueScorpion also wanted to mention a common mistake I see some people making is blocking private files / folders with a robot text file, so as a side note for those who might consider this remember anyone can read your robot text file and see what is listed.
Disallow: /*goto=
Disallow: /*mode=
Disallow: /*&pp=
Disallow: /*postcount=
Disallow: /*daysprune=
Disallow: /*&sort=
Does that block the robot from all the link or just part of the link?
Disallow: /*goto=
Disallow: /*mode=
Disallow: /*&pp=
Disallow: /*postcount=
Disallow: /*daysprune=
Disallow: /*&sort=
Does that block the robot from all the link or just part of the link?
Disallow: /*goto=
Disallow: /*mode=
Disallow: /*&pp=
Disallow: /*postcount=
Disallow: /*daysprune=
Disallow: /*&sort=
Does that block the robot from all the link or just part of the link?