Индексирование сайтов поисковиками это очень важно. Можно сколько угодно заниматься раскруткой, но если на сайте проблемы, то всё зря.
И главным будет правильная настройка Robots.txt
Посмотрим на пример популярных сайтов на Wordpres
The Wall Street Journal
Открываем и видим такой код
User-agent: * Allow: /ads/public/ Allow: /svc/news/v3/all/pshb.rss Disallow: /ads/ Disallow: /adx/bin/ Disallow: /archives/ Disallow: /auth/ Disallow: /cnet/ Disallow: /college/ Disallow: /external/ Disallow: /financialtimes/ Disallow: /idg/ Disallow: /indexes/ Disallow: /library/ Disallow: /nytimes-partners/ Disallow: /packages/flash/multimedia/TEMPLATES/ Disallow: /pages/college/ Disallow: /paidcontent/ Disallow: /partners/ Disallow: /register Disallow: /thestreet/ Disallow: /svc Disallow: /video/embedded/* Disallow: /web-services/ Disallow: /gst/travel/travsearch* Disallow: /1996/06/17/nyregion/guest-at-diplomat-s-party-accused-of-rape.html Disallow: /*.amp.html$ User-agent: googlebot Allow: /*.amp.html$ User-agent: bingbot Allow: /*.amp.html$ User-Agent: omgilibot Disallow: / User-Agent: omgili Disallow: / Sitemap: https://www.nytimes.com/sitemaps/www.nytimes.com/sitemap.xml.gz Sitemap: https://www.nytimes.com/sitemaps/sitemap_news/sitemap.xml.gz Sitemap: https://www.nytimes.com/sitemaps/sitemap_video/sitemap.xml.gz Sitemap: https://www.nytimes.com/sitemaps/www.nytimes.com_realestate/sitemap.xml.gz Sitemap: https://www.nytimes.com/sitemaps/www.nytimes.com/2016_election_sitemap.xml.gz
А теперь посмотрим на сайта The New York Times
Сразу бросилось в глаза, что карта идет вверху, а никак обычно внизу
User-agent: * Sitemap: https://www.wsj.com/sitemaps/web/wsj/en/sitemap_wsj_en_index.xml Sitemap: https://www.wsj.com/wsj_google_news.xml Sitemap: https://www.wsj.com/live_news_sitemap.xml Sitemap: https://www.wsj.com/sitemap.xml Sitemap: https://www.wsj.com/wsj_graphics_sitemap.xml Sitemap: https://www.wsj.com/sitemaps/web/video/en/sitemap_video_en_index.xml Disallow: /article_email/* Disallow: /user/* Disallow: /pdf/documents/* Disallow: /login/* Disallow: /acct/* Disallow: /msgcenter/* Disallow: /setup/* Disallow: /marketing/* Disallow: /public/article/* Disallow: /public/search/ Disallow: /public/search* Disallow: /search* Disallow: /public/page/wsj-x-marketing.html Disallow: /public/page/news-media-marketing.html Disallow: /public/page/0_0_WP_RT_MARKETING.html Disallow: /news/articles/SB2* Disallow: /news/articles/SB3* Disallow: /news/articles/SB4* Disallow: /articles/SB2* Disallow: /articles/SB3* Disallow: /articles/SB4* Disallow: /article/AP* Disallow: /article/BT-CO* Disallow: /article/DN-CO* Disallow: /article/PR-CO* Disallow: /article/HUG* Disallow: /video/search/* Disallow: /articles/BT-CO* Disallow: /articles/DN-CO* Disallow: /articles/PR-CO* Disallow: /news/articles/BT-CO* Disallow: /news/articles/DN-CO* Disallow: /news/articles/PR-CO* User-agent: MSNPTC/1.0 Disallow: /article_email/* Disallow: /login/* Disallow: /acct/* Disallow: /msgcenter/* Disallow: /setup/* Disallow: /user/* Disallow: /marketing/* Disallow: /public/article/* Disallow: /public/search/ Disallow: /public/search* Disallow: /search* Disallow: /public/page/wsj-x-marketing.html Disallow: /public/page/news-media-marketing.html Disallow: /public/page/0_0_WP_RT_MARKETING.html Disallow: /news/articles/SB2* Disallow: /news/articles/SB3* Disallow: /news/articles/SB4* Disallow: /articles/SB2* Disallow: /articles/SB3* Disallow: /articles/SB4* Disallow: /article/AP* Disallow: /article/BT-CO* Disallow: /article/DN-CO* Disallow: /article/PR-CO* Disallow: /article/HUG* Disallow: /video/search/* Disallow: /articles/BT-CO* Disallow: /articles/DN-CO* Disallow: /articles/PR-CO* Disallow: /news/articles/BT-CO* Disallow: /news/articles/DN-CO* Disallow: /news/articles/PR-CO* User-agent: Twitterbot Disallow: /amp/*
А вот как выглядит Forbest
User-agent: * Disallow: /includes/ Disallow: /lj-post/ Disallow: /misc/ Disallow: /modules/ Disallow: /*profiles Disallow: /scripts/ Disallow: /themes/ Disallow: /search Disallow: /nodesearch Disallow: /node Disallow: /sp_data/ Disallow: /explain/ Disallow: /user Disallow: /users Disallow: /sms Disallow: /admin/ Disallow: /blog/ Disallow: /notify/ Disallow: /comment/ Disallow: /*.php Disallow: /*.html Disallow: /*.txt Disallow: /*.doc Allow: /misc/*.css Allow: /misc/*.js Allow: /misc/*.gif Allow: /misc/*.jpg Allow: /misc/*.jpeg Allow: /misc/*.png Allow: /modules/*.css Allow: /modules/*.js Allow: /modules/*.gif Allow: /modules/*.jpg Allow: /modules/*.jpeg Allow: /modules/*.png Allow: /profiles/*.css Allow: /profiles/*.js Allow: /profiles/*.gif Allow: /profiles/*.jpg Allow: /profiles/*.jpeg Allow: /profiles/*.png Allow: /themes/*.css Allow: /themes/*.js Allow: /themes/*.gif Allow: /themes/*.jpg Allow: /themes/*.jpeg Allow: /themes/*.png Allow: /themes/*.svg Allow: /themes/*.woff User-agent: Yandex Disallow: /includes/ Disallow: /lj-post/ Disallow: /misc/ Disallow: /modules/ Disallow: /*profiles Disallow: /scripts/ Disallow: /themes/ Disallow: /search Disallow: /nodesearch Disallow: /node Disallow: /sp_data/ Disallow: /explain/ Disallow: /user Disallow: /users Disallow: /sms Disallow: /admin/ Disallow: /blog/ Disallow: /notify/ Disallow: /comment/ Disallow: /amp/ Disallow: /*.php Disallow: /*.html Disallow: /*.txt Disallow: /*.doc Clean-param: from&from_alt_domain Allow: /misc/*.css Allow: /misc/*.js Allow: /misc/*.gif Allow: /misc/*.jpg Allow: /misc/*.jpeg Allow: /misc/*.png Allow: /modules/*.css Allow: /modules/*.js Allow: /modules/*.gif Allow: /modules/*.jpg Allow: /modules/*.jpeg Allow: /modules/*.png Allow: /profiles/*.css Allow: /profiles/*.js Allow: /profiles/*.gif Allow: /profiles/*.jpg Allow: /profiles/*.jpeg Allow: /profiles/*.png Allow: /themes/*.css Allow: /themes/*.js Allow: /themes/*.gif Allow: /themes/*.jpg Allow: /themes/*.jpeg Allow: /themes/*.png Allow: /themes/*.svg Allow: /themes/*.woff User-Agent: YandexNews Allow: /newrss.xml User-agent: Googlebot Disallow: /includes/ Disallow: /lj-post/ Disallow: /misc/ Disallow: /modules/ Disallow: /*profiles Disallow: /scripts/ Disallow: /themes/ Disallow: /search Disallow: /nodesearch Disallow: /node Disallow: /sp_data/ Disallow: /explain/ Disallow: /user Disallow: /users Disallow: /sms Disallow: /admin/ Disallow: /blog/ Disallow: /notify/ Disallow: /comment/ Disallow: /*.php Disallow: /*.html Disallow: /*.txt Disallow: /*.doc Allow: /misc/*.css Allow: /misc/*.js Allow: /misc/*.gif Allow: /misc/*.jpg Allow: /misc/*.jpeg Allow: /misc/*.png Allow: /modules/*.css Allow: /modules/*.js Allow: /modules/*.gif Allow: /modules/*.jpg Allow: /modules/*.jpeg Allow: /modules/*.png Allow: /profiles/*.css Allow: /profiles/*.js Allow: /profiles/*.gif Allow: /profiles/*.jpg Allow: /profiles/*.jpeg Allow: /profiles/*.png Allow: /themes/*.css Allow: /themes/*.js Allow: /themes/*.gif Allow: /themes/*.jpg Allow: /themes/*.jpeg Allow: /themes/*.png Allow: /themes/*.svg Allow: /themes/*.woff Sitemap: https://pronets.ru/sitemap.xml
