Caching problems with mod_expires

Sometimes, when you’re dealing with large images, flash movies or large javascript files, it’s generally a good idea to force them in the client’s cache.

A very simple way to achieve this is by using Apache’s mod_expires. For instance, if you add the following example, taken from the manual, to your .htaccess file – assuming of course that mod_expires is properly installed and configured – it will tell the browser to cache all the files for a month.

ExpiresDefault "access plus 4 weeks"

So, every time the client returns in the following month to the site, the browser won’t download all the static content again, but load it locally from the cache, thus minimising the loading time. Of course, there are some issues that usually appear after updates. Particularly after updates of the cached files

For instance, you add a new Javascript functionality to the site or make some changes in the css or swf files and the user doesn’t hit at CTRL+F5 to fully refresh the page and clear its cache, then he will see the old version of the site. Of course, one can take the short road to LamerVille and post a message on the site, asking the user to refresh the page. But that’s a little too lame to be taken into consideration, especially when dealing with respectable sites.

But there’s another way, much more elegant. First of all, place all the static, cache-able files in a separate folder. The browser will cache all the files based on their URL. If you want the browser to reload all the static data on every new request, you need to change the URLs on every new request.

Let’s say, the site resides at www.example.com and that all the static information will be served from www.example.com/static/. Now, a good idea is to make the links look like this:

http://www.example.com/static/(release-number)/css/style.css

http://www.example.com/static/(release-number)/js/cool-ajax-app.js

Where release-number is a number that increments with every new release. This way, the URLs will be different after each release thus forcing the browsers to fetch the new files. You don’t need to go through lots of files and increment the release number by hand, you can just use the following python script:

#!/usr/bin/python
"""
Read more about what this script actually does on 

http://blog.motane.lu/2009/09/21/caching-problems-with-mod_expires/

Usage:
	python increment_release.py start_directory static_prefix

Author:
	Tudor Barbu http://blog.motane.lu
"""
import sys, os, re

REGEX = ''
CACHED_DIR = ''

def main():
    global REGEX, CACHED_DIR 

    if sys.argv is not None:
        length = len( sys.argv )
        if length < 3:
            print 'Read the comments in the source code'
            exit()
    start_folder = sys.argv[1]
    CACHED_DIR = sys.argv[2]

    REGEX = re.compile( '=(\'|")' + re.escape( CACHED_DIR ) + '\/((\d+)\/|)([^\'|"]+)(\'|")' )

    parse_files( start_folder )

def parse_files( dir ):
    basedir = dir
    subdirectories = []
    for item in os.listdir( dir ):
        if os.path.isfile( os.path.join( basedir, item ) ):
            perform_replace( os.path.join( basedir, item ) )
        else:
            subdirectories.append( os.path.join( basedir, item ) )
    for subdir in subdirectories:
        parse_files( subdir )

def perform_replace( file ):
    global REGEX
    f = open( file, 'r' )
    contents = f.read()
    f.close()
    if REGEX.search( contents ):
        f = open( file, 'w' )
        f.write( REGEX.sub( handle_match, contents ) )
        f.close()

def handle_match( matches ):
    global CACHED_DIR
    if matches.group(3) is not None:
        revision_number = int( matches.group(3) ) + 1
    else:
        revision_number = 1
    return '=%s%s/%s/%s%s' % ( matches.group(1), CACHED_DIR, revision_number, matches.group(4), matches.group(5) )

if __name__ == '__main__':
    main()

…and, of course, there’s no need to create lots of directories either. A simple .htaccess rewrite rule will do. Just redirect all the URLs like /static/(number)/css/style.css to point to /static/css/style.css, by adding these 2 lines in the /static/.htaccess file:

RewriteEngine On
RewriteRule ^\/static\/(\d+)\/(.*)$ $2 [NC,L]

This should solve all your caching related problems. If you want to look savvy, you can use the version number of the head revision from subversion of whatever versioning system you might be using instead of a simple incremental number.

Yes, I know that the script is a little bit buggy but it works for me. If you have an improved version, post a comment below. Credits will be given.