A mini guide to HTTP referer

In HTTP header, there is a field named Referer which is to provide the referrer of the current accessed page. In this post, we will introduce the usage of HTTP referer field.

On the web, when a user visits a webpage, s/he must be from some place. This place is usually referred a s referer. This information is very important to some website operators and server owners as they want to know where they get the traffic from and this helps them provide better service for potential targeted users.

In the request header of a HTTP request, there is a Referer filed which can be set to the referrer URL indicating the source of the visit.

This is an optional field which means sometimes you would not see value set for this field in a HTTP request. One funny thing about this field is that the field name actually has a typo. The correct English word is Referrer instead of Referer, but when this was finalized in the RFC standard, it seems no one found that there is a missing r. Later after the standard was finalized, people have to follow this wrong convention.

As mentioned earlier, Referer is an optional field in HTTP request, hence there are cases where Referer will be sent and be omitted. Normally when a user types the URL in browser address bar or opens a link from Bookmark, there would be no Referer sent. 

By default, in below three cases, the Referer field will be sent.

1. When a user clicks a link on a webpage
2. When user sends a form(click submit)
3. When loading static resource like image, JavaScript, CSS file.

<img src="foo.jpg">
<script src="foo.js"></script>
<link href="foo.css" rel="stylesheet">

In above cases, browser will set the Referer field as the current webpage URL and put it in the HTTP request header. In addition, JavaScript also provides document.referrer to check the Referer.

The Referer filed is actually telling the target web server the original URL before visiting the current page, this can be used for user tracking. One typical usage of this field is for blocking loading static resource such as image by external website. This can be achieved by checking the Referer, if the Referer is from external website, then reporting error, otherwise, allow the resource to be loaded.

But in cases where we don't want to expose sensitive information, we may choose not to send Referer. One example is when accessing some webpages from intranet, we don't want external users know about the internal architecture or configuration of an intranet domain. With above concern, modern browsers all provide some ways for website owners to change the default Referer behavior. 

HTML provides an attribute named rel which can be set to noreferrer to block sending referrer information when user clicks a link. This attribute is applicable to a, area and form tag in HTML. 

<a href="..." rel="noreferrer" target="_blank">xxx</a>

The request to the target URL above will not have Referer set in the request header.

rel attribute can only be set on a single HTML element, and it can only control whether to send the Referer or not. Due to these limitations, W3C comes out a more powerful setting -- Referrer Policy.

 There are 8 allowed values for Referrer Policy.

enum ReferrerPolicy {
  "",
  "no-referrer",
  "no-referrer-when-downgrade",
  "same-origin",
  "origin",
  "strict-origin",
  "origin-when-cross-origin",
  "strict-origin-when-cross-origin",
  "unsafe-url"
};

The Referrer Policy can be set at a few different places.

1. HTTP header

When web server sends the webpage to browser, it can tell the browser the Referrer Policy in the header.

Referrer-Policy: origin

2.meta tag

At the head of a HTML page, can also set the meta tag.

<meta name="referrer" content="origin">

3. referrerpolicy attribute

In a HTML element, can also set the referrerpolicy attribute. This is applicable to a, area, img, iframe and link tags.

<a href="..." referrerpolicy="origin" target="_blank">xxx</a>

One last tip. There is also traditional way adopted by big companies like Google and Facebook to block sending HTTP Referer. i.e, set href to an internal webpage with the redirection URL. This can effectively avoids sending Referer when accessing other URL.

<a  href="/exit.php?url=http%3A%2F%2Fexample.com">Example.com</a>

Reference: http://www.ruanyifeng.com/blog/2019/06/http-referer.html