Last-Modified Heading, Symfony and Search Indexing Acceleration

When developing websites, many developers forget about the very useful Last-Modified header, thanks to which you can optimize the loading of web-pages and facilitate the work of search robots. Next, I will try to fill this annoying gap.

What is the Last-Modified header for?

The title function as you can guess from the name is informing the client about the date of the last modification of the web document. Based on the rfc 2616 specification, the client can “ask” the web server if the page has not changed from a certain number by sending the “If-Modified-Since” header to the server. If the page has not changed, the server only returns the “304 Not Modified” header; otherwise, the server returns the “200 OK” header and the body of the page. As you can see, the benefit is obvious both for the server and for the client: the browser will not load the page again and again, and the web server will give less data.


')

Which sites are indexed better? Dynamic or static?

A couple of years ago, there were controversies among SEO people about which sites are indexed better? Dynamic, written for example in php, or static, without the use of programming languages. Knowing the title of Last-Modified, you can answer this question. The thing is that the web server itself processes the “If-Modified-Since” header if the file is static. In the case of dynamic page generation, all responsibility for the answer lies with the programming language and developer. And since the developer is often not interested in this issue, the headers are not given at all.

How does the Last-Modified header speed up search indexing?

Everything is simple, as written in the help of Yandex , “the robot will not be able to get information about whether the site page has been updated since the last indexing. And since the number of pages received by the robot from the site in one go is limited, the changed pages will be reindexed less often. ”

Imagine. There is a site with 10 thousand pages. The site is written in php. Correct Last-Modified header is not given. The search robot cannot get information on whether the site page has been updated since the last indexing. What is he doing? Indexes all pages !!! And not just those that have changed.
Of course! Many sites use a sitemap . But Sitemap is a recommendation, help search engine optimizer. Replacing the Last-Modified header, it can not be!

Setup and processing of the Last-Modified header in php

In order for the web server to transmit the If-Modified-Since header to the php-backend, it needs to inform it from it!

For the nginx + php bundle,

location ~ \.php$
{
…
if_modified_since off;

fastcgi_pass fcgi;
fastcgi_index index.php;
fastcgi_param SCRIPT_FILENAME /< > /web$fastcgi_script_name;
…
fastcgi_pass_header Last-Modified;
include fastcgi_params;
}

* This source code was highlighted with Source Code Highlighter .

For a bunch of apache + php, so

# If-Modified-Since ( if php is not installed as cgi then comment lines below)
RewriteRule .* - [E=HTTP_IF_MODIFIED_SINCE:%{HTTP:If-Modified-Since}]
RewriteRule .* - [E=HTTP_IF_NONE_MATCH:%{HTTP:If-None-Match}]

* This source code was highlighted with Source Code Highlighter .

If php works as a module, then nothing needs to be set up!

A simple php-code processing request for If-Modified-Since,

$qtime = isset($_SERVER[ 'HTTP_IF_MODIFIED_SINCE' ])? $_SERVER[ 'HTTP_IF_MODIFIED_SINCE' ]: '' ;

$modified = substr(gmdate( 'r' , $timestamp), 0, -5). 'GMT' ;

if ($hdr == $modified)
{
header ( "HTTP/1.1 304 Not Modified " );
header ( "Last-Modified: $modified" );
exit();
}
header ( "Last-Modified: $modified" );
//render

* This source code was highlighted with Source Code Highlighter .

How to handle an If-Modified-Since request in symfony?

Symfony already has a header handling mechanism. All the developer needs is to pass a header to the sfWebResponse object. If it is specified, the framework will do everything by itself.

$datestamp = time();
$response->setHttpHeader( 'Last-Modified' , $response->getDate($datestamp));

* This source code was highlighted with Source Code Highlighter .

Since the page is usually located different content, I wrote a method that exposes the latest of the transferred titles!

static public function setLastModified($datestamp)
{
$response = sfContext::getInstance()->getResponse();
$request = sfContext::getInstance()->getRequest();

if (is_array($datestamp))
{
rsort($datestamp, SORT_NUMERIC);
$datestamp = $datestamp[0];
}

if (!$response->hasHttpHeader( 'Last-Modified' ))
{
$response->setHttpHeader( 'Last-Modified' , $response->getDate($datestamp));
}
else
{
$origLastModified = strtotime($response->getHttpHeader( 'Last-Modified' ));
if ($origLastModified < $datestamp)
$response->setHttpHeader( 'Last-Modified' , $response->getDate($datestamp));
}
}

* This source code was highlighted with Source Code Highlighter .

It is very convenient to use it if on the page, for example, there are 3 last videos, 3 last articles and there is something else. By loading each model from the database, we can call the method, and as a result we get the latest modification date in the response.
For those interested, the header processing code is in the sfCacheFilter.class.php class.

In conclusion, I want to say that the use of the Last-Modified header is not always justified. For example, if the site has 5,000 pages and each has the same block with frequently changing content, using the title will be useless! In this case, you can only give different headers for customers and search robots. But as for me, deception of robots does not lead to anything good. Well, or remove this block;).

Still,

Check the site for correct processing of the title here or so,

<?php
$ch = curl_init();

$url = 'http://site.ru/1.php ' ;

curl_setopt($ch, CURLOPT_URL, $url);
curl_setopt($ch, CURLOPT_HEADER, true );
curl_setopt($ch, CURLOPT_NOBODY, true );
curl_setopt($ch, CURLOPT_HTTPHEADER, array(
'If-Modified-Since: Sun, 28 Nov 2010 15:45:53 GMT'
));

ob_start();
curl_exec ($ch);
curl_close ($ch);
$data = ob_get_contents();
ob_end_clean();


echo nl2br($data);
?>

* This source code was highlighted with Source Code Highlighter .