Tuesday, March 22, 2016

HTTPS All About It

Ever sine the cat came out of the bag HTTPS protocol has become more popular. There is a ranking benefit in case of using this one. The hyper text transfer protocol is used to transfer data to the browser. This is what makes it possible to download and see website and use the interactive mechanism.     

Like the search engines the users have also become aware of security accorded by this protocol as compared to simple version HTTP. The difference is accentuated by the security certificate associated by intervention of a reputed third party.  

Hence ii offers security through authorized transaction very useful in case of eCommerce stores and payment sites. Overall trust created by this environment attracts more and more converts in the present times.  This protocol is associated with a secured socket layer or SSL sometimes also referred as TLS.    

This technology allows encryption of data that is being exchanged. This also assures that data corruption cannot take place without being discovered.    

But for those sites being transferred to this secured technology it is important that transition takes place in the right manner and does not end up confusing the search engines.  

Earlier Article on HTTPs

AMP HTML Specification

This article has been picked up for readers...due credit is given to the source linked below.  

AMP HTML is a subset of HTML for authoring content pages such as news articles in a way that guarantees certain baseline performance characteristics.

Being a subset of HTML, it puts some restrictions on the full set of tags and functionality available through HTML but it does not require the development of new rendering engines: Existing user agents can render AMP HTML just like all other HTML.

Also, AMP HTML documents can be uploaded to a web server and served just like any other HTML document; no special configuration for the server is necessary. However, they are also designed to be optionally served through specialized AMP serving systems that proxy AMP documents. These documents serve them from their own origin and are allowed to apply transformations to the document that provide additional performance benefits. An incomplete list of optimizations such a serving system might do is:
  • Replace image references with images sized to the viewer’s viewport.
  • Inline images that are visible above the fold.
  • Inline CSS variables.
  • Preload extended components.
  • Minify HTML and CSS.
AMP HTML uses a set of contributed but centrally managed and hosted custom elements to implement advanced functionality such as image galleries that might be found in an AMP HTML document. While it does allow styling the document using custom CSS, it does not allow author written JavaScript beyond what is provided through the custom elements to reach its performance goals.
By using the AMP format, content producers are making the content in AMP files available to be crawled (subject to robots.txt restrictions), cached, and displayed by third parties.

Performance

Predictable performance is a key design goal for AMP HTML. Primarily we are aiming at reducing the time until the content of a page can be consumed / used by the user. In concrete terms this means that:
  • HTTP requests necessary to render and fully layout the document should be minimized.
  • Resources such as images or ads should only be downloaded if they are likely to be seen by the user.
  • Browsers should be able to calculate the space needed by every resource on the page without fetching that resource.

The AMP HTML format

Sample document



<!doctype html>
<html >
  <head>
    <meta charset="utf-8">
    <title>Sample document</title>
    <link rel="canonical" href="./regular-html-version.html">
    <meta name="viewport" content="width=device-width,minimum-scale=1,initial-scale=1">
    <style amp-custom>
      h1 {color: red}
    </style>
    <script type="application/ld+json">
    {
      "@context": "http://schema.org",
      "@type": "NewsArticle",
      "headline": "Article headline",
      "image": [
        "thumbnail1.jpg"
      ],
      "datePublished": "2015-02-05T08:00:00+08:00"
    }
    </script>
    <script async custom-element="amp-carousel" src="https://cdn.ampproject.org/v0/amp-carousel-0.1.js"></script>
    <style amp-boilerplate>body{-webkit-animation:-amp-start 8s steps(1,end) 0s 1 normal both;-moz-animation:-amp-start 8s steps(1,end) 0s 1 normal both;-ms-animation:-amp-start 8s steps(1,end) 0s 1 normal both;animation:-amp-start 8s steps(1,end) 0s 1 normal both}@-webkit-keyframes -amp-start{from{visibility:hidden}to{visibility:visible}}@-moz-keyframes -amp-start{from{visibility:hidden}to{visibility:visible}}@-ms-keyframes -amp-start{from{visibility:hidden}to{visibility:visible}}@-o-keyframes -amp-start{from{visibility:hidden}to{visibility:visible}}@keyframes -amp-start{from{visibility:hidden}to{visibility:visible}}</style><noscript><style amp-boilerplate>body{-webkit-animation:none;-moz-animation:none;-ms-animation:none;animation:none}</style></noscript>
    <script async src="https://cdn.ampproject.org/v0.js"></script>
  </head>
  <body>
    <h1>Sample document</h1>
    <p>
      Some text
      <amp-img src=sample.jpg width=300 height=300></amp-img>
    </p>
    <amp-ad width=300 height=250
        type="a9"
        data-aax_size="300x250"
        data-aax_pubname="test123"
        data-aax_src="302">
    </amp-ad>
  </body>
</html>

Required markup

AMP HTML documents MUST
  • start with the doctype <!doctype html>. 🔗
  • contain a top-level <html > tag (<html amp> is accepted as well). 🔗
  • contain <head> and <body> tags (They are optional in HTML). 🔗
  • contain a <link rel="canonical" href="$SOME_URL" /> tag inside their head that points to the regular HTML version of the AMP HTML document or to itself if no such HTML version exists. 🔗
  • contain a <meta charset="utf-8"> tag as the first child of their head tag. 🔗
  • contain a <meta name="viewport" content="width=device-width,minimum-scale=1"> tag inside their head tag. It’s also recommend to include initial-scale=1 (1). 🔗
  • contain a <script async src="https://cdn.ampproject.org/v0.js"></script> tag inside their head tag. 🔗
  • contain the AMP boilerplate code in their head tag. 🔗
(1) width=device-width,minimum-scale=1 is required to ensure GPU rasterization is enabled.

Metadata

It is encouraged that AMP HTML documents are annotated with standardized metadata: Open Graph Protocol, Twitter Cards, etc.
We also recommend that AMP HTML documents are marked up with schema.org/CreativeWork or any of its more specific types such as schema.org/NewsArticle or schema.org/BlogPosting.

HTML Tags

HTML tags can be used unchanged in AMP HTML. Certain tags have equivalent custom tags (such as <img> and <amp-img>) and other tags are outright prohibited:
Tag Status in AMP HTML
script Prohibited unless the type is application/ld+json. (Other non-executable values may be added as needed.) Exception is the mandatory script tag to load the AMP runtime and the script tags to load extended components.
base Prohibited
img Replaced with amp-img.
video Replaced with amp-video.
audio Replaced with amp-audio.
iframe Replaced with amp-iframe.
frame Prohibited.
frameset Prohibited.
object Prohibited.
param Prohibited.
applet Prohibited.
embed Prohibited.
form Prohibited. Support coming in the future.
input elements Prohibited. Includes input, textarea, select, option. Notably, button element is allowed.
button Allowed.
style Required style tags for adjusting opacity. One additional style tag is allowed in head tag for the purpose of custom styling. This style tag must have the attribute amp-custom. [🔗](https://github.com/ampproject/amphtml/blob/master/spec/#cust)
link rel values registered on microformats.org are allowed. If a rel value is missing from our whitelist, please submit an issue. stylesheet and other values like preconnect, prerender and prefectch that has side effects in the browser are disallowed. There is a special case for fetching stylesheets from whitelisted font providers.
meta The http-equiv attribute is banned. Otherwise allowed.
a The href attribute value must not begin with javascript:. If set, the target attribute value must be _blank. Otherwise allowed. [🔗](https://github.com/ampproject/amphtml/blob/master/spec/#ancr)
svg Most SVG elements are allowed.
Validator implementations should use a whitelist based on the HTML5 specification with the above tags removed. See AMP Tag Addendum.

Comments

Conditional HTML comments are not allowed.

HTML attributes

Attribute names starting with on (such as onclick or onmouseover) are disallowed in AMP HTML. The attribute with the literal name on (no suffix) is allowed.
The style attribute must not be used.
XML-related attributes, such as xmlns, xml:lang, xml:base, and xml:space are disallowed in AMP HTML.
The javascript: schema is disallowed.

Stylesheets

Major semantic tags and the AMP custom elements come with default styles to make authoring a responsive document reasonably easy. An option to opt out of default styles may be added in the future.

@-rules

The following @-rules are allowed in stylesheets:
@font-face, @keyframes, @media, @supports.
@import will not be allowed. Others may be added in the future.

Author stylesheets

Authors may add custom styles to a document using a single <style amp-custom> tag in the head of the document.

Selectors

The following restrictions apply to selectors in author style sheets:
Universal selector
The universal selector * may not be used in author stylesheets. This is because it can have negative performance implications and could be used to circumvent the rules set out in the following paragraph.
not selector
:not() may not be used in selectors because it can be used to simulate the universal selector.
Pseudo-selectors, pseudo-classes and pseudo-elements
Pseudo-selectors, pseudo-classes and pseudo-elements are only allowed in selectors that contain tag names and those tag names must not start with amp-.
Example OK: a:hover, div:last-of-type
Example not OK: amp-img:hover, amp-img:last-of-type
Class and tag names
Class names, in author stylesheets, may not start with the string -amp-. These are reserved for internal use by the AMP runtime. It follows, that the user’s stylesheet may not reference CSS selectors for -amp- classes and i-amp tags. These classes and elements are not meant to be customized by authors. Authors, however, can override styles of amp- classes and tags for any CSS properties not explicitly forbidden by these components’ spec.
To prevent usage of attribute selectors to circumvent class name limitations it is generally not allowed for CSS selectors to contain tokens and strings starting with -amp- and i-amp.

Important

Usage of the !important qualifier is not allowed. This is a necessary requirement to enable AMP to enforce its element sizing invariants.

Properties

These CSS properties are permanently banned:
  • behavior
  • -moz-binding
The following properties are currently blacklisted due to performance concerns:
  • filter
AMP only allows transitions and animations of properties that can be GPU accelerated in common browsers. We currently whitelist: opacity, transform (also -vendorPrefix-transform).
In the following examples <property> needs to be in the whitelist above.
  • transition <property> (Also -vendorPrefix-transition)
  • @keyframes name { from: {<property>: value} to {<property: value>} } (also @-vendorPrefix-keyframes)
overflow (and overflow-y, overflow-x) may not be styled as “auto” or “scroll”. No user defined element in an AMP document may have a scrollbar.
Maximum size
It is a validation error if the author stylesheet is larger than 50,000 bytes.

Custom fonts

Authors may include stylesheets for custom fonts. The 2 supported methods are link tags pointing to whitelisted font providers and @font-face inclusion.
Example:


<link rel="stylesheet" href="https://fonts.googleapis.com/css?family=Tangerine">
Font providers can be whitelisted if they support CSS-only integrations and serve over HTTPS. The following origins are currently allowed for font serving via link tags:
  • https://fast.fonts.net
  • https://fonts.googleapis.com
IMPLEMENTERS NOTE: Adding to this list requires a change to the Google AMP Cache CSP rule.
Authors are free to include all custom fonts via a @font-face CSS instruction via their custom CSS. Fonts included via @font-face must be fetched via the HTTP or HTTPS scheme.

AMP runtime

The AMP runtime is a piece of JavaScript that runs inside every AMP document. It provides implementations for AMP custom elements, manages resource loading and prioritization and optionally includes a runtime validator for AMP HTML for use during development.
The AMP runtime is loaded via the mandatory <script src="https://cdn.ampproject.org/v0.js"></script> tag in the AMP document <head>.
The AMP runtime can be placed into a development mode for any page. Development mode will trigger AMP validation on the embedded page, which will emit the validation status and any errors to the javascript developer console. Development mode may be triggered by appending #development=1 to the URL of the page.

Resources

Resources such as images, videos, audio files or ads must be included into an AMP HTML file through custom elements such as <amp-img>. We call them managed resources because whether and when they will be loaded and displayed to the user is decided by the AMP runtime.
There are no particular guarantees as to the loading behavior of the AMP runtime, but it should generally strive to load resources quickly enough, so that they are loaded by the time the user would like to see them if possible. The runtime should prioritize resources currently in viewport and attempt to predict changes to the viewport and preload resources accordingly.
The AMP runtime may at any time decide to unload resources that are not currently in viewport or reuse the resource containers such as iframes to reduce overall RAM consumption.

AMP Components

AMP HTML uses custom elements called, “AMP components” to substitute built-in resource-loading tags such as <img> and <video> and to implement features with complex interactions such as image lightboxes or carousels.
See the AMP component spec for details about supported components.
There are 2 types of supported AMP components:
  1. Built-in
  2. Extended
Built-in components are always available in an AMP document and have a dedicated custom element such as <amp-img>. Extended components must be explicitly included into the document.

Common attributes

layout, width, height, media, placeholder, fallback

These attributes define the layout of an element. The key goal here is to ensure that the element can be displayed and its space can be properly reserved before any of the JavaScript or remote resources have been downloaded.
See the AMP Layout System for details about the layout system.

on

The on attribute is used to install event handlers on elements. The events that are supported depend on the element.
The value for the syntax is a simple domain specific language of the form:


eventName:targetId[.methodName[(arg1=value, arg2=value)]]
Example: on="tap:fooId.showLightbox"
If methodName is omitted the default method is executed if defined for the element. Example: on="tap:fooId"
Some actions, if documented, may accept arguments. The arguments defined in the parenthesis in the key=value notation. The accepted values are:
  • simple unquoted strings: simple-value;
  • quoted strings: "string value" or 'string value';
  • boolean values: true or false;
  • numbers: 11 or 1.1.

Extended components

Extended components are components that do not necessarily ship with the AMP runtime. Instead they must be explicitly included into the document.
Extended components are loaded by including a <script> tag in the head of the document like this:


<script async custom-element="amp-carousel" src="https://cdn.ampproject.org/v0/amp-carousel-0.1.js"></script>
The <script> tag must have an async attribute and must have a custom-element attribute referencing the name of the element.
Runtime implementations may use the name to render placeholders for these elements.
The script URL must start with “https://cdn.ampproject.org” and must follow a very strict pattern of /v\d+/[a-z-]+-(latest|\d+|\d+.\d+).js.
URL
The URL for extended components is of the form:


https://cdn.ampproject.org/$RUNTIME_VERSION/$ELEMENT_NAME-$ELEMENT_VERSION.js

Versioning
Extended components are versioned via semver. The version is referenced explicitly (See $ELEMENT_VERSION above) when loading the component in the URL. It may have the value “latest”. Changes to the PATCH version component (x in 1.1.x) must strictly maintain backward compatibility or fix urgent security issues.
AMP documents may only reference versions with one or two components. Version “1” states “I accept every version of this component with major version 1”. Version “1.1” states “I accept every PATCH level of 1.1”. It is now allowed to explicitly reference the PATCH level in the version string.

Extended templates

Templates render HTML content based on the language-specific template and provided JSON data.
See the AMP template spec for details about supported templates.
Extended templates are not shipped with the AMP runtime and have to be downloaded just as with extended elements. Extended components are loaded by including a <script> tag in the head of the document like this:


<script async custom-template="amp-mustache" src="https://cdn.ampproject.org/v0/amp-mustache-0.1.js"></script>
The <script> tag must have an async attribute and must have a custom-template attribute referencing the type of the template. The script URL must start with “https://cdn.ampproject.org” and must follow a very strict pattern of /v\d+/[a-z-]+-(latest|\d+|\d+.\d+).js.
The templates are declared in the document as following:


<template type="amp-mustache" id="template1">
  Hello !
</template>
The type attribute is required and must reference a declared custom-element script.
The id attribute is optional. Individual AMP elements discover their own templates. Typical scenario would involve an AMP element looking for a <template> either among its children or referenced by ID.
The syntax within the template element depends on the specific template language. However, the template language could be restricted within AMP. For instance, in accordance with the “template” element, all productions have to be over a valid well-formed DOM. All of the template outputs are also subject to sanitizing to ensure AMP-valid output.
See documentation for a specific extended template on the syntax and restrictions.
URL
The URL for extended components is of the form:


https://cdn.ampproject.org/$RUNTIME_VERSION/$TEMPLATE_TYPE-$TEMPLATE_VERSION.js

Versioning
See versioning of custom elements for more details.

Security

AMP HTML documents must not trigger errors when served with a Content Security Policy that does not include the keywords unsafe-inline and unsafe-eval.
The AMP HTML format is designed so that is always the case.
All AMP template elements must go through AMP security review before they can be submitted into AMP repository.

SVG

Currently, the following SVG elements are allowed:
  • basics: “svg”, “g”, “path”, “glyph”, “glyphRef”, “marker”, “view”
  • shapes: “circle”, “line”, “polygon”, “polyline”, “rect”
  • text: “text”, “textPath”, “tref”, “tspan”
  • rendering: “clipPath”, “filter”, “linearGradient”, “radialGradient”, “mask”, “pattern”, “vkern”, “hkern”
  • special: “defs” (all children above are allowed here), “symbol”, “use”
  • aria: “desc”, “title”
As well as these attributes:
  • “xlink:href”: only URIs starting with “#” are allowed
  • “style”

AMP document discovery

If AMP documents are alternative representations of a canonical document, then the canonical document should point to the AMP document via a link tag with the relation “amphtml”.
Example:


<link rel="amphtml" href="https://www.example.com/url/to/amp/document.html">
The AMP document itself is expected to point back with its canonical relation to a document that has the “amphtml” relation.
Note, that AMP document may also be linked to directly. The mechanism described here provides a standardized way for software to discover whether an AMP version exists for a canonical document.

Source: AMP HTML Specification 

Friday, February 5, 2016

Writing Contents For Users

There has been excessive concern regarding search engines and how they index and rank contents. In earlier times this worked since the algorithms were is early stage of evolution but with constant addition and changes this is not the case anymore.   

In the contemporary times it has become mandatory to add meaningful content relevant to the topic or product in case of e-commerce sites. It is not necessary to add content just for the sake of it. But yes meaningful content is desirable that which is readable and useful. The placement of content is of importance as well. Too much content above the fold will hide or reduce visual element this is not good for user attraction. It is better to add a paragraph followed by an image or a video. 

In case of product description the read more element is very useful. Since adding too many lines will hide the product image away and detract visitor, it is better to add just two lines followed by read more for those interested in reading the description further. The format in case of product/item description may include reviews, ratings, specification, about and more. More angles to image are usually added so fit in that aspect as well. Creating product pages is a challenging job, hastily done it will detract the user in one go. In case of many similar products it is tempting repeat description which is a dead giveaway. Hire a content writer and unique descriptions.                

Remember when it comes to online success usability wins hands down. Better user experience is the right formula for success as for search engine they have become more sophisticated and can decipher your naturally written content to index and rank properly.   

Monday, January 18, 2016

Content Creation for Websites & Seo

With dynamic changes in search engine algorithms it is becoming more and more difficult for average webmasters to rank websites. This is because rules have been tightened and are being implemented more stringently day by day. Link building without any authority or editorial scrutiny hardly works. The out bounds are important but not the way they were earlier. Third grade content will not work at all.  

Recently the Panda filter was incorporated into the algorithms by Google bringing all sites under the scanner. Albeit the search engines have set in preference regarding which sites do better on SERPs there is plenty of scope to fetch in organic traffic. (One such example of this preference is the favor accorded to major brands.)  

The algorithms are constantly evolving bringing about perplexing changes. This evolution is always challenging webmasters as to how to perform better on SERPs. One issue that has been fixed by Panda is that there is no scope now for loosely written content for creating an opportunity to link. Scammers had a field day till the rot was stemmed.  Many blogs and websites where thrown out giving greater opportunity for those with relevant and properly written unique content. 


Website & Off Page Contents

Unique content means that which explains best the offerings of the site yet stays out of the glut. Hence if your are writing about  your site which is on hotels you should be more descriptive using important and LSI search terms in the write up. Use words like, rooms, suite, cottages, bathroom, accommodation, tariff, service, amenities, facilities, ambiance etc.    

When we talk about Internet Marketing  greater emphasis is accorded to search engine optimization. For off page promotion content creation in terms of articles, blogs, videos become important. So does what is there on your website or portal. 

Though unnecessary information is discouraged it is imperative that the interface contains relevant content. Sites with thin content will not rank well until unless they cater to some very unique and rare search term. Hence the first criteria before taking on a new project is to evaluate the site content along with the link architecture, usability, usage of tags, internal linking etc.     

There are hundreds of articles on how contents should be written according to the new guidelines. Usage of excessive keyword is not advisable at all. Add relevant LSI terms or simply put be descriptive without repeating important phrases that define the topic.   Remember search terms are important since the indexing and ranking process depends much on them...logical isn't it. 

When I say avoid glut I mean do not venture into a topic which is already saturated rather spend your efforts on something rare or less popular. Well for survival webmasters cannot be choosy and they have to take on projects as they come.    

For websites and content marketing well researched effort will always pay. This means doing extensive keyword research using various tools available online. You must write naturally in a flow as the writer do. Dig out aspects which have been ignored by others. Provide complete information about the products, services, topics or what not. On many instances location should be added - use structured mark up where ever you can. 

This way you will accord enough information to search engines as well as make the contents an interesting read. The latter is a prerequisite. In case if you are not able to naturally incorporate search terms in your article, do not worry. The SEs are becoming more and more smarter these days to dig out the crux.    

Another suggestion that I wish to give out to content writers is to work on numerous platforms which have editorial scrutiny for contents thereon. If you can score better in such platforms you are bound to become a better writer.  

Without over doing divide the write up into relevant paragraphs decorate in limit with tags, headings, italics, etc. Add relevant highly optimized light weight pictures with proper alt tags.  

As this articles states contents have become very important. If you cannot write yourself then hire a content writer for website or promotional efforts. Internet Marketing is heavily dependent on the written word be assured.     

Thursday, January 7, 2016

Website Accessibility: Usability: On Page Seo & Design Factors

Design & Content Guidelines 

Make a well defined link architecture using few H tags the H1 Tag being used only once. The font size, text to back ground contrast all should make readability easier.The font size used should be 12 preferable or more according to the design.

Whence writing contents use a unique language not copied from any source. Include words that people type to search relevant information in the Internet.

Use low weight but high resolution images labeled and added with the correct alt text. The height and width attributes should be in the place.

Write all important information in text especially headings since the search engines cannot read text embedded in the images. Label each block of content for visitors to know what they are going to read about. 

One of the important factors in on page optimization is website accessibility.  This is applicable to both the users and the search engines.

The configuration of robot.txt file is important this will help keep the less desired pages from search engine crawls while informing them of important pages.

Certain scripts may not gel well with browsers and different form of machines like Ipad etc.


Check whether server is well configured. Use the DNS Check Tool:

When you conduct a ping and trace route test the load time should be less then one minute. The less load time means your website is comparatively faster than many websites.

Mobile Friendly 

Use the view port meta  to make the site more friendly. Place reasonable distance between the links and use the font size as stated above. Above the use of flash and heavy images.

Use rich snippets to best describe your website to search engines. The structured data use should be limited and at present is restricted to  products, reviews, events, local business, videos and software applications. 

For better understanding read guidelines set up by search engines such as Google Guidelines 

Friday, December 25, 2015

SEO: HTTPS Over HTTP

In the webmaster blog Google announced its preference of HTTPS over HTTP protocol. This was bound to be since the search company is quite conscious of visitor privacy and security. Simply put this protocol encrypts message to be sent and on arrival decrypts messages to be received over HTTP protocol.   

HTTPS is a security protocol which carries a connection encrypted by TLS or Transfer Security Layer. The main objective is to secure visitor interest, prevent malicious packet injection Ad inserts, information espionage and privacy of course. Hence Data Exchange takes place in a secured environment for the benefit of the user.   

Earlier this protocol  was used for electronic payment systems over the WWW or Web. It was also preferred by corporate portals and e-mail providers in order to attain extreme secrecy.  

Browser software and certification authorities should be trustworthy in this type of data exchange. Some of the certification authorities are: Symantec, Comodo, GoDaddy and GlobalSign.

The encryption protection layer should also be trustworthy and provide security. However HTTPS can be called reasonably secure and do prevent tempering and malicious attacks but they are not hundred percent foolproof.       

Keeping in mind the good aspects of this protocol the search engine company will give preference to it over HTTP which is not as secured.    

In this case the port used is 443 and the URL begins with https://

For More Information visit:  HTTPS

Read at Google Webmaster Blog for more information on HTTPS 

robot.txt: What it is?

Search engines visit websites, blogs and other online portal to scan and then store or cache the information. This is then used to rank them according to the relevancy. In a large portal the whole website may be indexed according to criteria of search engines. Most of the search engine used indexed web pages collectively to rank the site. This means pages that are not important or hold information not meant for the engines are also indexed. 

In order to specify to the robots which pages to index and which to ignore a common protocol robot.txt is used. Hence a file is uploaded into the server along with the pages in the root directory. This is the robot.txt file which the robots first visit and index the pages accordingly. But remember this protocol may not be adhered to by some robots especially those with malicious intent. But all the popular search companies adhere to this standard which is public.       

The file contains the following command/commands:

User-agent: *
Disallow: /

More instructions are given below: 

To block the entire site, use a forward slash.

Disallow: /


To block a directory and everything in it, follow the directory name with a forward slash.

Disallow: /junk-directory/

To block a page, list the page.

Disallow: /private_file.html

To remove a specific image from Google Images, add the following:

User-agent: Googlebot-Image
Disallow: /images/dogs.jpg

To remove all images on your site from Google Images:

User-agent: Googlebot-Image
Disallow: /

To block files of a specific file type (for example, .gif), use the following:

User-agent: Googlebot
Disallow: /*.gif$

More information can be had from: Here Robot.TXT

Also visit Robottxt.org 

This information can also be given in the meta robot tag which should be present in every page. Another methodology is to insert x-robot header this is placed in the header which then works for all pages. Care should be taken that directives are proper and no important pages is barred.  This also applies to CMS portals.     

You have to regulary check the directives so that over time no misinformation/ wrong case is inserted. There are number of tools that can enable you to keep things in line.