1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
230
231
232
233
234
235
236
237
238
239
240
241
242
243
244
245
246
247
248
249
250
251
252
253
254
255
256
257
258
259
260
261
262
263
264
265
266
267
268
269
270
271
272
273
274
275
276
277
278
279
280
281
282
283
284
285
286
287
288
289
290
291
292
293
294
295
296
297
298
299
300
301
302
303
304
305
306
307
308
309
310
311
312
313
314
315
316
317
|
<?xml version="1.0" encoding="ISO-8859-1"?>
<!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.0 Strict//EN" "http://www.w3.org/TR/xhtml1/DTD/xhtml1-strict.dtd">
<html xmlns="http://www.w3.org/1999/xhtml" lang="en" xml:lang="en"><head><!--
XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX
This file is generated from xml source: DO NOT EDIT
XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX
-->
<title>mod_firehose - Apache HTTP Server</title>
<link href="../style/css/manual.css" rel="stylesheet" media="all" type="text/css" title="Main stylesheet" />
<link href="../style/css/manual-loose-100pc.css" rel="alternate stylesheet" media="all" type="text/css" title="No Sidebar - Default font size" />
<link href="../style/css/manual-print.css" rel="stylesheet" media="print" type="text/css" /><link rel="stylesheet" type="text/css" href="../style/css/prettify.css" />
<script src="../style/scripts/prettify.js" type="text/javascript">
</script>
<link href="../images/favicon.ico" rel="shortcut icon" /></head>
<body>
<div id="page-header">
<p class="menu"><a href="../mod/">Modules</a> | <a href="../mod/quickreference.html">Directives</a> | <a href="http://wiki.apache.org/httpd/FAQ">FAQ</a> | <a href="../glossary.html">Glossary</a> | <a href="../sitemap.html">Sitemap</a></p>
<p class="apache">Apache HTTP Server Version 2.5</p>
<img alt="" src="../images/feather.gif" /></div>
<div class="up"><a href="./"><img title="<-" alt="<-" src="../images/left.gif" /></a></div>
<div id="path">
<a href="http://www.apache.org/">Apache</a> > <a href="http://httpd.apache.org/">HTTP Server</a> > <a href="http://httpd.apache.org/docs/">Documentation</a> > <a href="../">Version 2.5</a> > <a href="./">Modules</a></div>
<div id="page-content">
<div id="preamble"><h1>Apache Module mod_firehose</h1>
<div class="toplang">
<p><span>Available Languages: </span><a href="../en/mod/mod_firehose.html" title="English"> en </a></p>
</div>
<table class="module"><tr><th><a href="module-dict.html#Description">Description:</a></th><td>Multiplexes all I/O to a given file or pipe.</td></tr>
<tr><th><a href="module-dict.html#Status">Status:</a></th><td>Extension</td></tr>
<tr><th><a href="module-dict.html#ModuleIdentifier">Module�Identifier:</a></th><td>firehose_module</td></tr>
<tr><th><a href="module-dict.html#SourceFile">Source�File:</a></th><td>mod_firehose.c</td></tr></table>
<h3>Summary</h3>
<p><code>mod_firehose</code> provides a mechanism to record data
being passed between the httpd server and the client at the raw
connection level to either a file or a pipe in such a way that the
data can be analysed or played back to the server at a future date.
It can be thought of as "tcpdump for httpd".</p>
<p>Connections are recorded after the SSL has been stripped, and can
be used for forensic debugging.</p>
<p>The <code class="program"><a href="../programs/firehose.html">firehose</a></code> tool can be used to demultiplex
the recorded stream back into individual files for analysis, or
playback using a tool like <code>netcat</code>.</p>
<div class="note"><h3>WARNING</h3>This module IGNORES all request level
mechanisms to keep data private. It is the responsibility of the
administrator to ensure that private data is not inadvertently
exposed using this module.
</div>
</div>
<div id="quickview"><h3 class="directives">Directives</h3>
<ul id="toc">
<li><img alt="" src="../images/down.gif" /> <a href="#firehoseconnectioninput">FirehoseConnectionInput</a></li>
<li><img alt="" src="../images/down.gif" /> <a href="#firehoseconnectionoutput">FirehoseConnectionOutput</a></li>
<li><img alt="" src="../images/down.gif" /> <a href="#firehoseproxyconnectioninput">FirehoseProxyConnectionInput</a></li>
<li><img alt="" src="../images/down.gif" /> <a href="#firehoseproxyconnectionoutput">FirehoseProxyConnectionOutput</a></li>
<li><img alt="" src="../images/down.gif" /> <a href="#firehoserequestinput">FirehoseRequestInput</a></li>
<li><img alt="" src="../images/down.gif" /> <a href="#firehoserequestoutput">FirehoseRequestOutput</a></li>
</ul>
<h3>Topics</h3>
<ul id="topics">
<li><img alt="" src="../images/down.gif" /> <a href="#enable">Enabling a Firehose</a></li>
<li><img alt="" src="../images/down.gif" /> <a href="#format">Stream Format</a></li>
</ul><h3>See also</h3>
<ul class="seealso">
<li><code class="program"><a href="../programs/firehose.html">firehose</a></code></li>
</ul><ul class="seealso"><li><a href="#comments_section">Comments</a></li></ul></div>
<div class="top"><a href="#page-header"><img alt="top" src="../images/up.gif" /></a></div>
<div class="section">
<h2><a name="enable" id="enable">Enabling a Firehose</a></h2>
<p>To enable the module, it should be compiled and loaded
in to your running Apache configuration, and the directives below
used to record the data you are interested in.</p>
<p>It is possible to record both incoming and outgoing data to
the same filename if desired, as the direction of flow is recorded
within each fragment.</p>
<p>It is possible to write to both normal files and fifos (pipes).
In the case of fifos, mod_firehose ensures that the packet size is
no larger than PIPE_BUF to ensure writes are atomic.</p>
<p>If a pipe is being used, something must be reading from the pipe
before httpd is started for the pipe to be successfully opened for
write. If the request to open the pipe fails, mod_firehose will
silently stand down and not record anything, and the server will
keep running as normal.</p>
<p>By default, all attempts to write will block the server. If the
webserver has been built against APR v2.0 or later, and an optional
"nonblock" parameter is specified all file writes will be non
blocking, and buffer overflows will cause debugging data to be lost.
In this case it is possible to prioritise the running of the server
over the recording of firehose data.</p>
</div><div class="top"><a href="#page-header"><img alt="top" src="../images/up.gif" /></a></div>
<div class="section">
<h2><a name="format" id="format">Stream Format</a></h2>
<p>The server typically serves multiple connections simultaneously,
and as a result requests and responses need to be multiplexed before
being written to the firehose.</p>
<p>The fragment format is designed as clear text, so that a firehose
can be opened with and inspected by a normal text editor.
Alternatively, the <code class="program"><a href="../programs/firehose.html">firehose</a></code> tool can be used to
demultiplex the firehose back into individual requests or
connections.</p>
<p>The size of the multiplexed fragments is governed by PIPE_BUF,
the maximum size of write the system is prepared to perform
atomically. By keeping the multiplexed fragments below PIPE_BUF in
size, the module guarantees that data from different fragments does
not interleave. The size of PIPE_BUF varies on different operating
systems.</p>
<p>The BNF for the fragment format is as follows:</p>
<pre>
stream = 0*(fragment)
fragment = header CRLF body CRLF
header = length SPC timestamp SPC ( request | response ) SPC uuid SPC count
length = <up to 16 byte hex fragment length>
timestamp = <up to 16 byte hex timestamp microseconds since 1970>
request = "<"
response = ">"
uuid = <formatted uuid of the connection>
count = <hex fragment number in the connection>
body = <the binary content of the fragment>
SPC = <a single space>
CRLF = <a carriage return, followed by a line feed>
</pre>
<p>All fragments for a connection or a request will share the same
UUID, depending on whether connections or requests are being recorded.
If connections are being recorded, multiple requests may appear within
a connection. A fragment with a zero length indicates the end of the
connection.</p>
<p>Fragments may go missing or be dropped if the process reading the
fragments is too slow. If this happens, gaps will exist in the
connection counter numbering. A warning will be logged in the error
log to indicate the UUID and counter of the dropped fragment, so it
can be confirmed the fragment was dropped.</p>
<p>It is possible that the terminating empty fragment may not appear,
caused by the httpd process crashing, or being terminated ungracefully.
The terminating fragment may be dropped if the process reading the
fragments is not fast enough.</p>
</div>
<div class="top"><a href="#page-header"><img alt="top" src="../images/up.gif" /></a></div>
<div class="directive-section"><h2><a name="FirehoseConnectionInput" id="FirehoseConnectionInput">FirehoseConnectionInput</a> <a name="firehoseconnectioninput" id="firehoseconnectioninput">Directive</a></h2>
<table class="directive">
<tr><th><a href="directive-dict.html#Description">Description:</a></th><td>Capture traffic coming into the server on each connection</td></tr>
<tr><th><a href="directive-dict.html#Syntax">Syntax:</a></th><td><code>FirehoseConnectionInput <var>[ block | nonblock ]</var> <var>filename</var></code></td></tr>
<tr><th><a href="directive-dict.html#Default">Default:</a></th><td><code>none</code></td></tr>
<tr><th><a href="directive-dict.html#Context">Context:</a></th><td>server config</td></tr>
<tr><th><a href="directive-dict.html#Status">Status:</a></th><td>Extension</td></tr>
<tr><th><a href="directive-dict.html#Module">Module:</a></th><td>mod_firehose</td></tr>
<tr><th><a href="directive-dict.html#Compatibility">Compatibility:</a></th><td>FirehoseConnectionInput is only available in Apache 2.5.0 and
later.</td></tr>
</table>
<p>Capture traffic coming into the server on each connection. Multiple
requests will be captured within the same connection if keepalive is
present.</p>
<div class="example"><h3>Example</h3><pre class="prettyprint lang-config">
FirehoseConnectionInput connection-input.firehose
</pre>
</div>
</div>
<div class="top"><a href="#page-header"><img alt="top" src="../images/up.gif" /></a></div>
<div class="directive-section"><h2><a name="FirehoseConnectionOutput" id="FirehoseConnectionOutput">FirehoseConnectionOutput</a> <a name="firehoseconnectionoutput" id="firehoseconnectionoutput">Directive</a></h2>
<table class="directive">
<tr><th><a href="directive-dict.html#Description">Description:</a></th><td>Capture traffic going out of the server on each connection</td></tr>
<tr><th><a href="directive-dict.html#Syntax">Syntax:</a></th><td><code>FirehoseConnectionOutput <var>[ block | nonblock ]</var> <var>filename</var></code></td></tr>
<tr><th><a href="directive-dict.html#Default">Default:</a></th><td><code>none</code></td></tr>
<tr><th><a href="directive-dict.html#Context">Context:</a></th><td>server config</td></tr>
<tr><th><a href="directive-dict.html#Status">Status:</a></th><td>Extension</td></tr>
<tr><th><a href="directive-dict.html#Module">Module:</a></th><td>mod_firehose</td></tr>
<tr><th><a href="directive-dict.html#Compatibility">Compatibility:</a></th><td>FirehoseConnectionOutput is only available in Apache 2.5.0 and
later.</td></tr>
</table>
<p>Capture traffic going out of the server on each connection.
Multiple requests will be captured within the same connection if
keepalive is present.</p>
<div class="example"><h3>Example</h3><pre class="prettyprint lang-config">
FirehoseConnectionOutput connection-output.firehose
</pre>
</div>
</div>
<div class="top"><a href="#page-header"><img alt="top" src="../images/up.gif" /></a></div>
<div class="directive-section"><h2><a name="FirehoseProxyConnectionInput" id="FirehoseProxyConnectionInput">FirehoseProxyConnectionInput</a> <a name="firehoseproxyconnectioninput" id="firehoseproxyconnectioninput">Directive</a></h2>
<table class="directive">
<tr><th><a href="directive-dict.html#Description">Description:</a></th><td>Capture traffic coming into the back of mod_proxy</td></tr>
<tr><th><a href="directive-dict.html#Syntax">Syntax:</a></th><td><code>FirehoseProxyConnectionInput <var>[ block | nonblock ]</var> <var>filename</var></code></td></tr>
<tr><th><a href="directive-dict.html#Default">Default:</a></th><td><code>none</code></td></tr>
<tr><th><a href="directive-dict.html#Context">Context:</a></th><td>server config</td></tr>
<tr><th><a href="directive-dict.html#Status">Status:</a></th><td>Extension</td></tr>
<tr><th><a href="directive-dict.html#Module">Module:</a></th><td>mod_firehose</td></tr>
<tr><th><a href="directive-dict.html#Compatibility">Compatibility:</a></th><td>FirehoseProxyConnectionInput is only available in Apache 2.5.0 and
later.</td></tr>
</table>
<p>Capture traffic being received by mod_proxy.</p>
<div class="example"><h3>Example</h3><pre class="prettyprint lang-config">
FirehoseProxyConnectionInput proxy-input.firehose
</pre>
</div>
</div>
<div class="top"><a href="#page-header"><img alt="top" src="../images/up.gif" /></a></div>
<div class="directive-section"><h2><a name="FirehoseProxyConnectionOutput" id="FirehoseProxyConnectionOutput">FirehoseProxyConnectionOutput</a> <a name="firehoseproxyconnectionoutput" id="firehoseproxyconnectionoutput">Directive</a></h2>
<table class="directive">
<tr><th><a href="directive-dict.html#Description">Description:</a></th><td>Capture traffic sent out from the back of mod_proxy</td></tr>
<tr><th><a href="directive-dict.html#Syntax">Syntax:</a></th><td><code>FirehoseProxyConnectionOutput <var>[ block | nonblock ]</var> <var>filename</var></code></td></tr>
<tr><th><a href="directive-dict.html#Default">Default:</a></th><td><code>none</code></td></tr>
<tr><th><a href="directive-dict.html#Context">Context:</a></th><td>server config</td></tr>
<tr><th><a href="directive-dict.html#Status">Status:</a></th><td>Extension</td></tr>
<tr><th><a href="directive-dict.html#Module">Module:</a></th><td>mod_firehose</td></tr>
<tr><th><a href="directive-dict.html#Compatibility">Compatibility:</a></th><td>FirehoseProxyConnectionOutput is only available in Apache 2.5.0 and
later.</td></tr>
</table>
<p>Capture traffic being sent out by mod_proxy.</p>
<div class="example"><h3>Example</h3><pre class="prettyprint lang-config">
FirehoseProxyConnectionOutput proxy-output.firehose
</pre>
</div>
</div>
<div class="top"><a href="#page-header"><img alt="top" src="../images/up.gif" /></a></div>
<div class="directive-section"><h2><a name="FirehoseRequestInput" id="FirehoseRequestInput">FirehoseRequestInput</a> <a name="firehoserequestinput" id="firehoserequestinput">Directive</a></h2>
<table class="directive">
<tr><th><a href="directive-dict.html#Description">Description:</a></th><td>Capture traffic coming into the server on each request</td></tr>
<tr><th><a href="directive-dict.html#Syntax">Syntax:</a></th><td><code>FirehoseRequestInput <var>[ block | nonblock ]</var> <var>filename</var></code></td></tr>
<tr><th><a href="directive-dict.html#Default">Default:</a></th><td><code>none</code></td></tr>
<tr><th><a href="directive-dict.html#Context">Context:</a></th><td>server config</td></tr>
<tr><th><a href="directive-dict.html#Status">Status:</a></th><td>Extension</td></tr>
<tr><th><a href="directive-dict.html#Module">Module:</a></th><td>mod_firehose</td></tr>
<tr><th><a href="directive-dict.html#Compatibility">Compatibility:</a></th><td>FirehoseRequestInput is only available in Apache 2.5.0 and
later.</td></tr>
</table>
<p>Capture traffic coming into the server on each request. Requests
will be captured separately, regardless of the presence of keepalive.</p>
<div class="example"><h3>Example</h3><pre class="prettyprint lang-config">
FirehoseRequestInput request-input.firehose
</pre>
</div>
</div>
<div class="top"><a href="#page-header"><img alt="top" src="../images/up.gif" /></a></div>
<div class="directive-section"><h2><a name="FirehoseRequestOutput" id="FirehoseRequestOutput">FirehoseRequestOutput</a> <a name="firehoserequestoutput" id="firehoserequestoutput">Directive</a></h2>
<table class="directive">
<tr><th><a href="directive-dict.html#Description">Description:</a></th><td>Capture traffic going out of the server on each request</td></tr>
<tr><th><a href="directive-dict.html#Syntax">Syntax:</a></th><td><code>FirehoseRequestOutput <var>[ block | nonblock ]</var> <var>filename</var></code></td></tr>
<tr><th><a href="directive-dict.html#Default">Default:</a></th><td><code>none</code></td></tr>
<tr><th><a href="directive-dict.html#Context">Context:</a></th><td>server config</td></tr>
<tr><th><a href="directive-dict.html#Status">Status:</a></th><td>Extension</td></tr>
<tr><th><a href="directive-dict.html#Module">Module:</a></th><td>mod_firehose</td></tr>
<tr><th><a href="directive-dict.html#Compatibility">Compatibility:</a></th><td>FirehoseRequestOutput is only available in Apache 2.5.0 and
later.</td></tr>
</table>
<p>Capture traffic going out of the server on each request. Requests
will be captured separately, regardless of the presence of keepalive.</p>
<div class="example"><h3>Example</h3><pre class="prettyprint lang-config">
FirehoseRequestOutput request-output.firehose
</pre>
</div>
</div>
</div>
<div class="bottomlang">
<p><span>Available Languages: </span><a href="../en/mod/mod_firehose.html" title="English"> en </a></p>
</div><div class="top"><a href="#page-header"><img src="../images/up.gif" alt="top" /></a></div><div class="section"><h2><a id="comments_section" name="comments_section">Comments</a></h2><div class="warning"><strong>Notice:</strong><br />This is not a Q&A section. Comments placed here should be pointed towards suggestions on improving the documentation or server, and may be removed again by our moderators if they are either implemented or considered invalid/off-topic. Questions on how to manage the Apache HTTP Server should be directed at either our IRC channel, #httpd, on Freenode, or sent to our <a href="http://httpd.apache.org/lists.html">mailing lists</a>.</div>
<script type="text/javascript"><!--//--><![CDATA[//><!--
var comments_shortname = 'httpd';
var comments_identifier = 'http://httpd.apache.org/docs/trunk/mod/mod_firehose.html';
(function(w, d) {
if (w.location.hostname.toLowerCase() == "httpd.apache.org") {
d.write('<div id="comments_thread"><\/div>');
var s = d.createElement('script');
s.type = 'text/javascript';
s.async = true;
s.src = 'https://comments.apache.org/show_comments.lua?site=' + comments_shortname + '&page=' + comments_identifier;
(d.getElementsByTagName('head')[0] || d.getElementsByTagName('body')[0]).appendChild(s);
}
else {
d.write('<div id="comments_thread">Comments are disabled for this page at the moment.<\/div>');
}
})(window, document);
//--><!]]></script></div><div id="footer">
<p class="apache">Copyright 2013 The Apache Software Foundation.<br />Licensed under the <a href="http://www.apache.org/licenses/LICENSE-2.0">Apache License, Version 2.0</a>.</p>
<p class="menu"><a href="../mod/">Modules</a> | <a href="../mod/quickreference.html">Directives</a> | <a href="http://wiki.apache.org/httpd/FAQ">FAQ</a> | <a href="../glossary.html">Glossary</a> | <a href="../sitemap.html">Sitemap</a></p></div><script type="text/javascript"><!--//--><![CDATA[//><!--
if (typeof(prettyPrint) !== 'undefined') {
prettyPrint();
}
//--><!]]></script>
</body></html>
|
