IPB, WordPress, IP.Board, php-Fusion, Invision Power Board

Robots популярных сайтов на WordPress

0

Индексирование сайтов поисковиками это очень важно. Можно сколько угодно заниматься раскруткой, но если на сайте проблемы, то всё зря.

И главным будет правильная настройка Robots.txt

Посмотрим на пример популярных сайтов на Wordpres

The Wall Street Journal

Открываем и видим такой код

User-agent: *
Allow: /ads/public/
Allow: /svc/news/v3/all/pshb.rss
Disallow: /ads/
Disallow: /adx/bin/
Disallow: /archives/
Disallow: /auth/
Disallow: /cnet/
Disallow: /college/
Disallow: /external/
Disallow: /financialtimes/
Disallow: /idg/
Disallow: /indexes/
Disallow: /library/
Disallow: /nytimes-partners/
Disallow: /packages/flash/multimedia/TEMPLATES/
Disallow: /pages/college/
Disallow: /paidcontent/
Disallow: /partners/
Disallow: /register
Disallow: /thestreet/
Disallow: /svc
Disallow: /video/embedded/*
Disallow: /web-services/
Disallow: /gst/travel/travsearch*
Disallow: /1996/06/17/nyregion/guest-at-diplomat-s-party-accused-of-rape.html
Disallow: /*.amp.html$

User-agent: googlebot
Allow: /*.amp.html$

User-agent: bingbot
Allow: /*.amp.html$

User-Agent: omgilibot
Disallow: /

User-Agent: omgili
Disallow: /

Sitemap: https://www.nytimes.com/sitemaps/www.nytimes.com/sitemap.xml.gz
Sitemap: https://www.nytimes.com/sitemaps/sitemap_news/sitemap.xml.gz
Sitemap: https://www.nytimes.com/sitemaps/sitemap_video/sitemap.xml.gz
Sitemap: https://www.nytimes.com/sitemaps/www.nytimes.com_realestate/sitemap.xml.gz
Sitemap: https://www.nytimes.com/sitemaps/www.nytimes.com/2016_election_sitemap.xml.gz

 

А теперь посмотрим на сайта The New York Times
Сразу бросилось в глаза, что карта идет вверху, а никак обычно внизу

User-agent: *
Sitemap: https://www.wsj.com/sitemaps/web/wsj/en/sitemap_wsj_en_index.xml
Sitemap: https://www.wsj.com/wsj_google_news.xml
Sitemap: https://www.wsj.com/live_news_sitemap.xml
Sitemap: https://www.wsj.com/sitemap.xml
Sitemap: https://www.wsj.com/wsj_graphics_sitemap.xml
Sitemap: https://www.wsj.com/sitemaps/web/video/en/sitemap_video_en_index.xml
Disallow: /article_email/*
Disallow: /user/*
Disallow: /pdf/documents/*

Disallow: /login/*
Disallow: /acct/*
Disallow: /msgcenter/*
Disallow: /setup/*
Disallow: /marketing/*
Disallow: /public/article/*
Disallow: /public/search/
Disallow: /public/search*
Disallow: /search*
Disallow: /public/page/wsj-x-marketing.html
Disallow: /public/page/news-media-marketing.html
Disallow: /public/page/0_0_WP_RT_MARKETING.html
Disallow: /news/articles/SB2*
Disallow: /news/articles/SB3*
Disallow: /news/articles/SB4*
Disallow: /articles/SB2*
Disallow: /articles/SB3*
Disallow: /articles/SB4*
Disallow: /article/AP*
Disallow: /article/BT-CO*
Disallow: /article/DN-CO*
Disallow: /article/PR-CO*
Disallow: /article/HUG*
Disallow: /video/search/*
Disallow: /articles/BT-CO*
Disallow: /articles/DN-CO*
Disallow: /articles/PR-CO*
Disallow: /news/articles/BT-CO*
Disallow: /news/articles/DN-CO*
Disallow: /news/articles/PR-CO*

User-agent: MSNPTC/1.0
Disallow: /article_email/*
Disallow: /login/*
Disallow: /acct/*
Disallow: /msgcenter/*
Disallow: /setup/*
Disallow: /user/*
Disallow: /marketing/*
Disallow: /public/article/*
Disallow: /public/search/
Disallow: /public/search*
Disallow: /search*
Disallow: /public/page/wsj-x-marketing.html
Disallow: /public/page/news-media-marketing.html
Disallow: /public/page/0_0_WP_RT_MARKETING.html
Disallow: /news/articles/SB2*
Disallow: /news/articles/SB3*
Disallow: /news/articles/SB4*
Disallow: /articles/SB2*
Disallow: /articles/SB3*
Disallow: /articles/SB4*
Disallow: /article/AP*
Disallow: /article/BT-CO*
Disallow: /article/DN-CO*
Disallow: /article/PR-CO*
Disallow: /article/HUG*
Disallow: /video/search/*
Disallow: /articles/BT-CO*
Disallow: /articles/DN-CO*
Disallow: /articles/PR-CO*
Disallow: /news/articles/BT-CO*
Disallow: /news/articles/DN-CO*
Disallow: /news/articles/PR-CO*

User-agent: Twitterbot
Disallow: /amp/*

А вот как выглядит Forbest

User-agent: *
Disallow: /includes/
Disallow: /lj-post/
Disallow: /misc/
Disallow: /modules/
Disallow: /*profiles
Disallow: /scripts/
Disallow: /themes/
Disallow: /search
Disallow: /nodesearch
Disallow: /node
Disallow: /sp_data/
Disallow: /explain/
Disallow: /user
Disallow: /users
Disallow: /sms
Disallow: /admin/
Disallow: /blog/
Disallow: /notify/
Disallow: /comment/
Disallow: /*.php
Disallow: /*.html
Disallow: /*.txt
Disallow: /*.doc
Allow: /misc/*.css
Allow: /misc/*.js
Allow: /misc/*.gif
Allow: /misc/*.jpg
Allow: /misc/*.jpeg
Allow: /misc/*.png
Allow: /modules/*.css
Allow: /modules/*.js
Allow: /modules/*.gif
Allow: /modules/*.jpg
Allow: /modules/*.jpeg
Allow: /modules/*.png
Allow: /profiles/*.css
Allow: /profiles/*.js
Allow: /profiles/*.gif
Allow: /profiles/*.jpg
Allow: /profiles/*.jpeg
Allow: /profiles/*.png
Allow: /themes/*.css
Allow: /themes/*.js
Allow: /themes/*.gif
Allow: /themes/*.jpg
Allow: /themes/*.jpeg
Allow: /themes/*.png
Allow: /themes/*.svg
Allow: /themes/*.woff

User-agent: Yandex
Disallow: /includes/
Disallow: /lj-post/
Disallow: /misc/
Disallow: /modules/
Disallow: /*profiles
Disallow: /scripts/
Disallow: /themes/
Disallow: /search
Disallow: /nodesearch
Disallow: /node
Disallow: /sp_data/
Disallow: /explain/
Disallow: /user
Disallow: /users
Disallow: /sms
Disallow: /admin/
Disallow: /blog/
Disallow: /notify/
Disallow: /comment/
Disallow: /amp/
Disallow: /*.php
Disallow: /*.html
Disallow: /*.txt
Disallow: /*.doc
Clean-param: from&from_alt_domain
Allow: /misc/*.css
Allow: /misc/*.js
Allow: /misc/*.gif
Allow: /misc/*.jpg
Allow: /misc/*.jpeg
Allow: /misc/*.png
Allow: /modules/*.css
Allow: /modules/*.js
Allow: /modules/*.gif
Allow: /modules/*.jpg
Allow: /modules/*.jpeg
Allow: /modules/*.png
Allow: /profiles/*.css
Allow: /profiles/*.js
Allow: /profiles/*.gif
Allow: /profiles/*.jpg
Allow: /profiles/*.jpeg
Allow: /profiles/*.png
Allow: /themes/*.css
Allow: /themes/*.js
Allow: /themes/*.gif
Allow: /themes/*.jpg
Allow: /themes/*.jpeg
Allow: /themes/*.png
Allow: /themes/*.svg
Allow: /themes/*.woff

User-Agent: YandexNews
Allow: /newrss.xml

User-agent: Googlebot
Disallow: /includes/
Disallow: /lj-post/
Disallow: /misc/
Disallow: /modules/
Disallow: /*profiles
Disallow: /scripts/
Disallow: /themes/
Disallow: /search
Disallow: /nodesearch
Disallow: /node
Disallow: /sp_data/
Disallow: /explain/
Disallow: /user
Disallow: /users
Disallow: /sms
Disallow: /admin/
Disallow: /blog/
Disallow: /notify/
Disallow: /comment/
Disallow: /*.php
Disallow: /*.html
Disallow: /*.txt
Disallow: /*.doc
Allow: /misc/*.css
Allow: /misc/*.js
Allow: /misc/*.gif
Allow: /misc/*.jpg
Allow: /misc/*.jpeg
Allow: /misc/*.png
Allow: /modules/*.css
Allow: /modules/*.js
Allow: /modules/*.gif
Allow: /modules/*.jpg
Allow: /modules/*.jpeg
Allow: /modules/*.png
Allow: /profiles/*.css
Allow: /profiles/*.js
Allow: /profiles/*.gif
Allow: /profiles/*.jpg
Allow: /profiles/*.jpeg
Allow: /profiles/*.png
Allow: /themes/*.css
Allow: /themes/*.js
Allow: /themes/*.gif
Allow: /themes/*.jpg
Allow: /themes/*.jpeg
Allow: /themes/*.png
Allow: /themes/*.svg
Allow: /themes/*.woff

Sitemap: http://pronets.ru/sitemap.xml

Leave A Reply