Follow us on Twitter X-Cart on Facebook Wiki
Shopping cart software Solutions for online shops and malls

Googlebot CSS and JS Block Warning
 
Reply
   X-Cart forums > Considering X-Cart > Considering using X-Cart for my project
 
Thread Tools Search this Thread
  #11  
Old 08-05-2015, 07:49 PM
 
georgewf georgewf is offline
 

Advanced Member
  
Join Date: Feb 2004
Posts: 69
 

Default Re: Googlebot CSS and JS Block Warning

Hi guys, Why do you block /skin in robots.txt? It will just alert bad bots to the presence of the /skin folder.

As far as ranking, I was listening to some podcasts where I heard that google has a sophisticated engine that ranks based on what the web surfer SEES in his browser. If you block google from accessing CSS and JS google might impose some kind of "doubt" factor on you since you are saying "don't look at my css". On the other hand if google indexes a page I wonder if it will check robots.txt before loading your css and js from /skin. Some log analysis over a few days or weeks trying different setups would be needed. Who has time for that!?

For reference see
http://googlewebmastercentral.blogspot.ca/2014/10/updating-our-technical-webmaster.html
http://www2.webmasterradio.fm/seo-rockstars/2015/how-google-handles-new-tlds-for-seo-yahoo-testing-search-...-google-search around 8:30
__________________
xcart 4.7.7
Reply With Quote
  #12  
Old 08-24-2015, 09:23 AM
  rocky's Avatar 
rocky rocky is offline
 

X-Cart team
  
Join Date: Jul 2005
Posts: 715
 

Default Re: Googlebot CSS and JS Block Warning

Hi guys,

Can you please shed some light on why you hide /skin folder from crawler?

As far as I understand you do it for security reasons? So that nobody knows that you've got this folder on the server? However, it makes no sense whether you've got this /skin folder or not it has correct file permissions and protected with .htaccess file.

Thank you for the explanation in advance.
__________________
Alexander Diatchkov
Head of Technical Support Department

Official X-Cart Support at:
http://www.x-cart.com/technical-support.html
Reply With Quote
  #13  
Old 08-24-2015, 08:16 PM
 
elmirage001 elmirage001 is offline
 

X-Wizard
  
Join Date: Apr 2007
Posts: 1,562
 

Default Re: Googlebot CSS and JS Block Warning

Quote:
Originally Posted by rocky
Hi guys,

Can you please shed some light on why you hide /skin folder from crawler?

As far as I understand you do it for security reasons? So that nobody knows that you've got this folder on the server? However, it makes no sense whether you've got this /skin folder or not it has correct file permissions and protected with .htaccess file.

Thank you for the explanation in advance.
Hi Rocky, I do it so that I don't have tpl files indexed by Google so that the following does not happen.

1. Google cart.tpl
2. On page 1 is a link to cart.tpl - Shazzmin e-Store
3. If you click on this link you open the following file (only showing part of file)

Code:
<?php echo $header; ?> <?php if ($attention) { ?> <div class="attention"><?php echo $attention; ?><img src="catalog/view/theme/fashionteddy/image/close.png" alt="" class="close" /></div> <?php } ?> <?php if ($success) { ?> <div class="success"><?php echo $success; ?><img src="catalog/view/theme/fashionteddy/image/close.png" alt="" class="close" /></div> <?php } ?> <?php if ($error_warning) { ?> <div class="warning"><?php echo $error_warning; ?><img src="catalog/view/theme/fashionteddy/image/close.png" alt="" class="close" /></div> <?php } ?> <?php echo $column_left; ?><?php echo $column_right; ?> <div id="content"><?php echo $content_top; ?> <div class="breadcrumb"> <?php foreach ($breadcrumbs as $breadcrumb) { ?> <?php echo $breadcrumb['separator']; ?><a href="<?php echo $breadcrumb['href']; ?>"><?php echo $breadcrumb['text']; ?></a> <?php } ?> </div> <h1><?php echo $heading_title; ?> <?php if ($weight) { ?> &nbsp;(<?php echo $weight; ?>) <?php } ?> </h1> <form action="<?php echo $action; ?>" method="post" enctype="multipart/form-data"> <div class="cart-info"> <table> <thead> <tr> <td class="image"><?php echo $column_image; ?></td> <td class="name"><?php echo $column_name; ?></td> <td class="model"><?php echo $column_model; ?></td> <td class="quantity"><?php echo $column_quantity; ?></td> <td class="price"><?php echo $column_price; ?></td> <td class="total"><?php echo $column_total; ?></td> </tr> </thead> <tbody> <?php foreach ($products as $product) { ?> <tr> <td class="image"><?php if ($product['thumb']) { ?> <a href="<?php echo $product['href']; ?>"><img src="<?php echo $product['thumb']; ?>" alt="<?php echo $product['name']; ?>" title="<?php echo $product['name']; ?>" /></a> <?php } ?></td> . . . . . . }, complete: function() { $('#button-quote').attr('disabled', false); $('.wait').remove(); }, success: function(json) { $('.success, .warning, .attention, .error').remove(); if (json['error']) { if (json['error']['warning']) { $('#notification').html('<div class="warning" style="display: none;">' + json['error']['warning'] + '<img src="catalog/view/theme/fashionteddy/image/close.png" alt="" class="close" /></div>'); $('.warning').fadeIn('slow'); $('html, body').animate({ scrollTop: 0 }, 'slow'); } if (json['error']['country']) { $('select[name=\'country_id\']').after('<span class="error">' + json['error']['country'] + '</span>'); } if (json['error']['zone']) { $('select[name=\'zone_id\']').after('<span class="error">' + json['error']['zone'] + '</span>'); } if (json['error']['postcode']) { $('input[name=\'postcode\']').after('<span class="error">' + json['error']['postcode'] + '</span>'); } } if (json['shipping_method']) { html = '<h2><?php echo $text_shipping_method; ?></h2>'; html += '<form action="<?php echo $action; ?>" method="post" enctype="multipart/form-data">'; html += ' <table class="radio">'; for (i in json['shipping_method']) { html += '<tr>'; html += ' <td colspan="3"><b>' + json['shipping_method'][i]['title'] + '</b></td>'; html += '</tr>'; if (!json['shipping_method'][i]['error']) { for (j in json['shipping_method'][i]['quote']) { html += '<tr class="highlight">'; if (json['shipping_method'][i]['quote'][j]['code'] == '<?php echo $shipping_method; ?>') { html += '<td><input type="radio" name="shipping_method" value="' + json['shipping_method'][i]['quote'][j]['code'] + '" id="' + json['shipping_method'][i]['quote'][j]['code'] + '" checked="checked" /></td>'; } else { html += '<td><input type="radio" name="shipping_method" value="' + json['shipping_method'][i]['quote'][j]['code'] + '" id="' + json['shipping_method'][i]['quote'][j]['code'] + '" /></td>'; } html += ' <td><label for="' + json['shipping_method'][i]['quote'][j]['code'] + '">' + json['shipping_method'][i]['quote'][j]['title'] + '</label></td>'; html += ' <td style="text-align: right;"><label for="' + json['shipping_method'][i]['quote'][j]['code'] + '">' + json['shipping_method'][i]['quote'][j]['text'] + '</label></td>'; html += '</tr>'; } } else { html += '<tr>'; html += ' <td colspan="3"><div class="error">' + json['shipping_method'][i]['error'] + '</div></td>'; html += '</tr>'; } } html += ' </table>'; html += ' <br />'; html += ' <input type="hidden" name="next" value="shipping" />'; <?php if ($shipping_method) { ?> html += ' <input type="submit" value="<?php echo $button_shipping; ?>" id="button-shipping" class="button" />'; <?php } else { ?> html += ' <input type="submit" value="<?php echo $button_shipping; ?>" id="button-shipping" class="button" disabled="disabled" />'; <?php } ?> html += '</form>'; $.colorbox({ overlayClose: true, opacity: 0.5, width: '600px', height: '400px', href: false, html: html }); $('input[name=\'shipping_method\']').bind('change', function() { $('#button-shipping').attr('disabled', false); }); } } }); }); //--></script> <script type="text/javascript"><!-- $('select[name=\'country_id\']').bind('change', function() { $.ajax({ url: 'index.php?route=checkout/cart/country&country_id=' + this.value, dataType: 'json', beforeSend: function() { $('select[name=\'country_id\']').after('<span class="wait">&nbsp;<img src="catalog/view/theme/fashionteddy/image/loading.gif" alt="" /></span>'); }, complete: function() { $('.wait').remove(); }, success: function(json) { if (json['postcode_required'] == '1') { $('#postcode-required').show(); } else { $('#postcode-required').hide(); } html = '<option value=""><?php echo $text_select; ?></option>'; if (json['zone'] != '') { for (i = 0; i < json['zone'].length; i++) { html += '<option value="' + json['zone'][i]['zone_id'] + '"'; if (json['zone'][i]['zone_id'] == '<?php echo $zone_id; ?>') { html += ' selected="selected"'; } html += '>' + json['zone'][i]['name'] + '</option>'; } } else { html += '<option value="0" selected="selected"><?php echo $text_none; ?></option>'; } $('select[name=\'zone_id\']').html(html); }, error: function(xhr, ajaxOptions, thrownError) { alert(thrownError + "\r\n" + xhr.statusText + "\r\n" + xhr.responseText); } }); }); $('select[name=\'country_id\']').trigger('change'); //--></script> <?php } ?> <?php echo $footer; ?>

More info than I want to people to see doing a Google search...

Hope that helps,

Paul
__________________
X-Cart GoldPlus v4.7.11 | PHP 7.2
  • XCARTMODS.CO.UK | reBOOT Template |
  • Smack Digital | CDSEO Pro 2.0 + Custom Replacements | Category SEO Map |
  • Total Server Solutions Hosting | Linux | PHP v7.2 | MySQL v5.6.40 |
Reply With Quote
  #14  
Old 08-24-2015, 09:49 PM
  rocky's Avatar 
rocky rocky is offline
 

X-Cart team
  
Join Date: Jul 2005
Posts: 715
 

Default Re: Googlebot CSS and JS Block Warning

Paul,

The source code you provided us with here is not an X-Cart one.

Your store is also not X-Cart based. Thus, I'm sot sure what are we talking about. Do you have an X-Cart based store?

Thank you.
__________________
Alexander Diatchkov
Head of Technical Support Department

Official X-Cart Support at:
http://www.x-cart.com/technical-support.html
Reply With Quote
  #15  
Old 08-24-2015, 11:24 PM
  PhilJ's Avatar 
PhilJ PhilJ is offline
 

X-Guru
  
Join Date: Nov 2002
Location: UK
Posts: 3,578
 

Default Re: Googlebot CSS and JS Block Warning

Rocky, it's recommended in the docs.
Reply With Quote

The following user thanks PhilJ for this useful post:
elmirage001 (08-25-2015)
  #16  
Old 08-25-2015, 04:51 AM
 
elmirage001 elmirage001 is offline
 

X-Wizard
  
Join Date: Apr 2007
Posts: 1,562
 

Default Re: Googlebot CSS and JS Block Warning

Quote:
Originally Posted by rocky
Paul,

The source code you provided us with here is not an X-Cart one.

Your store is also not X-Cart based. Thus, I'm sot sure what are we talking about. Do you have an X-Cart based store?

Thank you.

Yes I have an X-Cart store and it's configured properly to not show tpl files so I could not use mine as an example. Not all X-Cart user stores are configured properly and if not the above example could happen.

I'm simply telling Google that I don't want them trying to index parts of my store that should not be indexed. This practice also keeps errors from showing in my Google Search Console.

I'm not understanding why you think it's okay to tell Google to try to index files in the skin folder. I also feel a properly configured robots.txt file helps save Google resources by focusing the time spent on my site to just that areas I do want indexed.

Thanks!

Paul
__________________
X-Cart GoldPlus v4.7.11 | PHP 7.2
  • XCARTMODS.CO.UK | reBOOT Template |
  • Smack Digital | CDSEO Pro 2.0 + Custom Replacements | Category SEO Map |
  • Total Server Solutions Hosting | Linux | PHP v7.2 | MySQL v5.6.40 |
Reply With Quote

The following user thanks elmirage001 for this useful post:
Vacman (09-01-2015)
  #17  
Old 11-09-2015, 02:39 AM
 
zebu zebu is offline
 

eXpert
  
Join Date: Oct 2006
Posts: 310
 

Default Re: Googlebot CSS and JS Block Warning

I found after allowing google access to css and JS that I suddenly had more than 500 errors appear in Google webmaster tools over teh next month or so. All 404 errors with links like shown further below.

If i was to add the following code to robots.txt file - would this resolve or would it stop google accessing css or JS it seems it just has to have...

Disallow: /skin/common_files/modules/*.*


skin/common_files/modules/HTML_Editor/editors/ckeditor/plugins/link/dialogs/?DD
404
10/16/15

343
skin/common_files/modules/HTML_Editor/scripts/language/english/?SA
404
10/13/15

344
skin/common_files/modules/HTML_Editor/editors/ckeditor/plugins/showblocks/
404
10/11/15

345
skin/common_files/modules/alteredCart_On_Sale/sale-create-tpls/?MA
404
10/14/15

346
skin/common_files/modules/HTML_Editor/editors/tinymce/plugins/table/
404
10/10/15

347
skin/common_files/modules/HTML_Editor/editors/ckeditor/plugins/menubutton/
404
10/11/15

348
skin/common_files/modules/alteredCart_On_Sale/messages/
404
10/13/15

349
skin/common_files/modules/HTML_Editor/editors/ckeditor/plugins/uicolor/
404
10/11/15

350
skin/common_files/modules/HTML_Editor/editors/ckeditor/plugins/a11yhelp/
404
10/11/15

351
skin/common_files/modules/HTML_Editor/editors/ckeditor/plugins/enterkey/
404
10/11/15

352
skin/common_files/modules/HTML_Editor/editors/ckeditor/plugins/a11yhelp/lang/
404
10/11/15

353
skin/common_files/modules/HTML_Editor/editors/ckeditor/plugins/panel/
404
10/11/15

354
skin/common_files/modules/HTML_Editor/editors/ckeditor/plugins/pastefromword/
404
10/11/15

355
skin/common_files/modules/HTML_Editor/editors/ckeditor/plugins/maximize/
404
10/11/15

356
skin/common_files/modules/HTML_Editor/editors/ckeditor/plugins/undo/
404
10/11/15

357
skin/common_files/modules/Magnifier/skins/classic/thumbnails/
404
10/12/15

358
skin/common_files/modules/Magnifier/skins/classic/tools/
404
10/12/15

359
skin/common_files/modules/HTML_Editor/editors/ckeditor/plugins/stylescombo/styles/
404
10/12/15
__________________
Version 4.7.7
Reply With Quote
  #18  
Old 11-09-2015, 09:07 PM
 
elmirage001 elmirage001 is offline
 

X-Wizard
  
Join Date: Apr 2007
Posts: 1,562
 

Default Re: Googlebot CSS and JS Block Warning

Below is my robots.txt file. Every couple of months I'll get a soft 404 error but nothing like you've listed.

Code:
User-agent: Googlebot Disallow: /*printable=Y* Disallow: /*print_cat=* Disallow: /*mode=add_vote* Disallow: /private*/ User-agent: * Disallow: /admin/ Disallow: /catalog/ Disallow: /customer/ Disallow: /files/ Disallow: /include/ Disallow: /mail/ Disallow: /modules/ Disallow: /partner/ Disallow: /payment/ Disallow: /provider/ Disallow: /schemes/ Disallow: /shipping/ Disallow: /skin/ Disallow: /skin_backup/ Disallow: /sql/ Disallow: /upgrade/ Disallow: /var/ Disallow: /404.php Disallow: /CHANGELOG Disallow: /COPYRIGHT Disallow: /INSTALL Disallow: /README Disallow: /VERSION Disallow: /adaptive.php Disallow: /address_book.php Disallow: /adv_counter.php Disallow: /antibot_image.php Disallow: /auth.php Disallow: /banner.php Disallow: /bonuses.php Disallow: /cart.php Disallow: /change_password.php Disallow: /check_requirements.php Disallow: /choosing.php Disallow: /cleanup.php Disallow: /cmpi_popup.php Disallow: /comparison.php Disallow: /comparison_list.php Disallow: /config.php Disallow: /cron.php Disallow: /default_icon.gif Disallow: /default_image.gif Disallow: /default_logo.gif Disallow: /dispatcher.php Disallow: /download.php Disallow: /error_message.php Disallow: /featured_products.php Disallow: /get_block.php Disallow: /get_info.php Disallow: /giftcert.php Disallow: /giftreg_manage.php Disallow: /giftregs.php Disallow: /help.php Disallow: /https.php Disallow: /image.php Disallow: /index.php Disallow: /init.php Disallow: /install.php Disallow: /login.php Disallow: /magnifier_xml.php Disallow: /message.html Disallow: /minicart.php Disallow: /news.php Disallow: /offers.php Disallow: /order.php Disallow: /orders.php Disallow: /pages.php Disallow: /pconf.php Disallow: /permission_denied.html Disallow: /popup_address.php Disallow: /popup_ask.php Disallow: /popup_edit_label.php Disallow: /popup_estimate_shipping.php Disallow: /popup_fc_products.php Disallow: /popup_image.php Disallow: /popup_info.php Disallow: /popup_magnifier.php Disallow: /popup_poptions.php Disallow: /postauth.php Disallow: /preauth.php Disallow: /prepare.php Disallow: /probe.php Disallow: /preview_image.gif Disallow: /process_order.php Disallow: /products.php Disallow: /products_map.php Disallow: /recommends.php Disallow: /referer.php Disallow: /register.php Disallow: /returns.php Disallow: /search.php Disallow: /send_to_friend.php Disallow: /shop_closed.html Disallow: /shop_registration.php Disallow: /sitemap.php Disallow: /slabel.php Disallow: /smarty.php Disallow: /survey.php Disallow: /top.inc.php Disallow: /vote.php Disallow: /wysiwyg.php
__________________
X-Cart GoldPlus v4.7.11 | PHP 7.2
  • XCARTMODS.CO.UK | reBOOT Template |
  • Smack Digital | CDSEO Pro 2.0 + Custom Replacements | Category SEO Map |
  • Total Server Solutions Hosting | Linux | PHP v7.2 | MySQL v5.6.40 |
Reply With Quote
  #19  
Old 11-09-2015, 09:14 PM
 
zebu zebu is offline
 

eXpert
  
Join Date: Oct 2006
Posts: 310
 

Default Re: Googlebot CSS and JS Block Warning

Thanks for that - much appreciated. You have a lot more in your robots.txt so i will build mine up a little and see how it works. Appreciate the time to post a good reply!
__________________
Version 4.7.7
Reply With Quote

The following user thanks zebu for this useful post:
elmirage001 (11-11-2015)
Reply
   X-Cart forums > Considering X-Cart > Considering using X-Cart for my project


Thread Tools Search this Thread
Search this Thread:

Advanced Search

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

vB code is On
Smilies are On
[IMG] code is On
HTML code is Off
Forum Jump


All times are GMT -8. The time now is 01:37 AM.

   

 
X-Cart forums © 2001-2018