Advanced Techniques with mod_rewrite

A common technique for distributing the burden of server load or storage space is called "sharding". When using this method, a front-end server will use the url to consistently "shard" users or objects to separate backend servers.

A mapping is maintained, from users to target servers, in external map files. They look like:

user1  physical_host_of_user1
user2  physical_host_of_user2
:      :

We put this into a map.users-to-hosts file. The aim is to map;

/u/user1/anypath

to

http://physical_host_of_user1/u/user/anypath

thus every URL path need not be valid on every backend physical host. The following ruleset does this for us with the help of the map files assuming that server0 is a default server which will be used if a user has no entry in the map:

RewriteEngine on

RewriteMap      users-to-hosts   txt:/path/to/map.users-to-hosts

RewriteRule   ^/u/([^/]+)/?(.*)   http://${users-to-hosts:$1|server0}/u/$1/$2

We wish to dynamically generate content, but store it statically once it is generated. This rule will check for the existence of the static file, and if it's not there, generate it. The static files can be removed periodically, if desired (say, via cron) and will be regenerated on demand.

# This example is valid in per-directory context only
RewriteCond %{REQUEST_FILENAME}   !-s
RewriteRule ^page\.html$          page.cgi   [T=application/x-httpd-cgi,L]

Here a request for page.html leads to an internal run of a corresponding page.cgi if page.html is missing or has filesize null. The trick here is that page.cgi is a CGI script which (additionally to its STDOUT) writes its output to the file page.html. Once it has completed, the server sends out page.html. When the webmaster wants to force a refresh of the contents, he just removes page.html (typically from cron).

We wish to randomly distribute load across several servers using mod_rewrite.

We'll use RewriteMap and a list of servers to accomplish this.

RewriteEngine on
RewriteMap lb rnd:/path/to/serverlist.txt

RewriteRule ^/(.*) http://${lb:servers}/$1 [P,L]

serverlist.txt will contain a list of the servers:

## serverlist.txt

servers one.example.com|two.example.com|three.example.com

If you want one particular server to get more of the load than the others, add it more times to the list.

Apache comes with a load-balancing module - mod_proxy_balancer - which is far more flexible and featureful than anything you can cobble together using mod_rewrite.

Wouldn't it be nice, while creating a complex web page, if the web browser would automatically refresh the page every time we save a new version from within our editor? Impossible?

No! We just combine the MIME multipart feature, the web server NPH feature, and the URL manipulation power of mod_rewrite. First, we establish a new URL feature: Adding just :refresh to any URL causes the 'page' to be refreshed every time it is updated on the filesystem.

RewriteRule   ^(/[uge]/[^/]+/?.*):refresh  /internal/cgi/apache/nph-refresh?f=$1

Now when we reference the URL

/u/foo/bar/page.html:refresh

this leads to the internal invocation of the URL

/internal/cgi/apache/nph-refresh?f=/u/foo/bar/page.html

The only missing part is the NPH-CGI script. Although one would usually say "left as an exercise to the reader" ;-) I will provide this, too.

#!/sw/bin/perl
##
##  nph-refresh -- NPH/CGI script for auto refreshing pages
##  Copyright (c) 1997 Ralf S. Engelschall, All Rights Reserved.
##
$| = 1;

#   split the QUERY_STRING variable
@pairs = split(/&/, $ENV{'QUERY_STRING'});
foreach $pair (@pairs) {
($name, $value) = split(/=/, $pair);
$name =~ tr/A-Z/a-z/;
$name = 'QS_' . $name;
$value =~ s/%([a-fA-F0-9][a-fA-F0-9])/pack("C", hex($1))/eg;
eval "\$$name = \"$value\"";
}
$QS_s = 1 if ($QS_s eq '');
$QS_n = 3600 if ($QS_n eq '');
if ($QS_f eq '') {
print "HTTP/1.0 200 OK\n";
print "Content-type: text/html\n\n";
print "<b>ERROR</b>: No file given\n";
exit(0);
}
if (! -f $QS_f) {
print "HTTP/1.0 200 OK\n";
print "Content-type: text/html\n\n";
print "<b>ERROR</b>: File $QS_f not found\n";
exit(0);
}

sub print_http_headers_multipart_begin {
print "HTTP/1.0 200 OK\n";
$bound = "ThisRandomString12345";
print "Content-type: multipart/x-mixed-replace;boundary=$bound\n";
&print_http_headers_multipart_next;
}

sub print_http_headers_multipart_next {
print "\n--$bound\n";
}

sub print_http_headers_multipart_end {
print "\n--$bound--\n";
}

sub displayhtml {
local($buffer) = @_;
$len = length($buffer);
print "Content-type: text/html\n";
print "Content-length: $len\n\n";
print $buffer;
}

sub readfile {
local($file) = @_;
local(*FP, $size, $buffer, $bytes);
($x, $x, $x, $x, $x, $x, $x, $size) = stat($file);
$size = sprintf("%d", $size);
open(FP, "<$file");
$bytes = sysread(FP, $buffer, $size);
close(FP);
return $buffer;
}

$buffer = &readfile($QS_f);
&print_http_headers_multipart_begin;
&displayhtml($buffer);

sub mystat {
local($file) = $_[0];
local($time);

($x, $x, $x, $x, $x, $x, $x, $x, $x, $mtime) = stat($file);
return $mtime;
}

$mtimeL = &mystat($QS_f);
$mtime = $mtime;
for ($n = 0; $n < $QS_n; $n++) {
while (1) {
    $mtime = &mystat($QS_f);
    if ($mtime ne $mtimeL) {
        $mtimeL = $mtime;
        sleep(2);
        $buffer = &readfile($QS_f);
        &print_http_headers_multipart_next;
        &displayhtml($buffer);
        sleep(5);
        $mtimeL = &mystat($QS_f);
        last;
    }
    sleep($QS_s);
}
}

&print_http_headers_multipart_end;

exit(0);

##EOF##

Some sites with thousands of users use a structured homedir layout, i.e. each homedir is in a subdirectory which begins (for instance) with the first character of the username. So, /~larry/anypath is /home/l/larry/public_html/anypath while /~waldo/anypath is /home/w/waldo/public_html/anypath.

We use the following ruleset to expand the tilde URLs into the above layout.

RewriteEngine on
RewriteRule   ^/~(([a-z])[a-z0-9]+)(.*)  /home/$2/$1/public_html$3

By default, redirecting to an HTML anchor doesn't work, because mod_rewrite escapes the # character, turning it into %23. This, in turn, breaks the redirection.

Use the [NE] flag on the RewriteRule. NE stands for No Escape.

We wish to use mod_rewrite to serve different content based on the time of day.

There are a lot of variables named TIME_xxx for rewrite conditions. In conjunction with the special lexicographic comparison patterns <STRING, >STRING and =STRING we can do time-dependent redirects:

RewriteEngine on
RewriteCond   %{TIME_HOUR}%{TIME_MIN} >0700
RewriteCond   %{TIME_HOUR}%{TIME_MIN} <1900
RewriteRule   ^foo\.html$             foo.day.html [L]
RewriteRule   ^foo\.html$             foo.night.html

This provides the content of foo.day.html under the URL foo.html from 07:01-18:59 and at the remaining time the contents of foo.night.html.

At time, we want to maintain some kind of status when we perform a rewrite. For example, you want to make a note that you've done that rewrite, so that you can check later to see if a request can via that rewrite. One way to do this is by setting an environment variable.

Use the [E] flag to set an environment variable.

RewriteEngine on
RewriteRule   ^/horse/(.*)   /pony/$1 [E=rewritten:1]

Later in your ruleset you might check for this environment variable using a RewriteCond:

RewriteCond %{ENV:rewritten} =1