Saturday, August 2, 2014

Google App Engine: skipping complete directory via skip_files

I forgot all about the skip_files in app.yaml and asked my favorite search engine what to do. A number of the top pages returned made it unnecessarily complicated. Just do this to skip directories (.idea and tools in this example):

- ^(.*/)?#.*#$
- ^(.*/)?.*~$
- ^(.*/)?.*\.py[co]$
- ^(.*/)?.*/RCS/.*$
- ^(.*/)?\..*$
- ^\.idea$
- ^tools$

(These are top-level directories within my app engine project.)

Add -v to your ... update ... invocation to confirm, from messages like

2014-08-02 09:19:20,424 INFO Ignoring directory '.idea': Directory matches ignore regex. 
2014-08-02 09:19:20,425 INFO Processing file 'app.yaml' 
2014-08-02 09:19:20,426 INFO Processing file 'index.yaml' 
2014-08-02 09:19:20,426 INFO Processing file '' 
2014-08-02 09:19:20,426 INFO Ignoring file 'main.pyc': File matches ignore regex. 
2014-08-02 09:19:20,426 INFO Processing file '' 
2014-08-02 09:19:20,427 INFO Ignoring directory 'tools': Directory matches ignore regex. 
2014-08-02 09:19:20,429 INFO Processing file 'static/Chart.min.js' 

Friday, August 1, 2014

Another year (or two), another release of mod_whatkilledus and mod_backtrace

mod_whatkilledus and mod_backtrace V2.01

I have a sinking feeling that this isn't pertinent to a fairly large percentage of the world's inhabitants, but it was a fair amount of hours spread over time and it's finally done, so I'll pretend otherwise.

mod_whatkilledus improvements

  • The name of the log file can now be configured with the WKULogfile directive.
  • On Windows: Add directive WKUDisableWindowsErrorBox to allow disabling the pop-up error handling dialog after a child process crash.
  • On Windows: Handle exceptions in 64-bit builds.
  • With Event MPM on Unix: When building with httpd 2.4.10 or later, properly track request processing across threads with the Event MPM. Previously, the wrong client and/or request might be logged as the trigger for a crash (with the Event MPM only). The change is not effective when building with httpd 2.4.9 or earlier.

mod_backtrace improvements

  • Support libunwind on Linux, FreeBSD, and OS X. This provides better resolution of function names, so a raw address will be provided in lieu of the function name in fewer situations. (Add LIBUNWIND=yes to the make invocation.) OS X note: A libunwind build silently fails to capture a backtrace with 32-bit builds, so use the default mechanism with 32-bit builds on OS X.
  • Filter out internal functions from backtraces so that only the caller's stack frames are reported.
  • Improve formatting of a backtrace for the error log field.

Windows binaries

64-bit binaries are now included for use with httpd 2.2 and 2.4 on Windows. (32-bit binaries are still included.)

Build improvements

  • Support make APXS=/path/to/apxs for building with non-default install layouts.
  • Support Clang on FreeBSD 10.
  • Add -funwind-tables on ARM platform for better backtraces.
  • mod_whatkilledus fails to build in a less mysterious way when using an httpd build that doesn't have exception hooks enabled.

Friday, May 30, 2014

Is a script to run a command within a virtualenv missing from virtualenv?

Sometimes the python command in the virtualenv isn't quite enough.

Contents of script




. $envdir/bin/activate && cd $curdir && exec "$@"

This runs a specified command (plus arguments) in a particular directory within a particular virtualenv.

Example use:

$ crontab -l
30 0 * * * /home/trawick/myhg/apache/bin/ git/edurepo/envs/edurepo git/edurepo/src/edurepo python teachers/

July 4 update:

See also vex.

Monday, May 26, 2014

If you replace the hard drive in your MacBook...

Be sure to check the startup disk setting after you are successfully booting from the new drive.

We just replaced the original drive in a Core 2 Duo White MacBook (2008, 4.1) with a 256GB Crucial M500 SSD, following some instructions on Apple Support Communities (search for Second way) for moving the data*. Our power-button-to-productivity benchmark of bootup+login+Chrome-window-appears+Word-window-appears was 2:36 with the original drive and 1:08 with the SSD drive. But that 1:08 included a surprising 25-30 second delay after the screen lit up on power-on but before the Apple logo appeared. There are multiple causes described in this article, but the simple issue for us was selection of the startup disk in System Preferences. After selecting the Crucial SSD as the startup disk, the annoying delay at power up was gone. Thus our power-button-to-productivity time is more like 0:40, down from the original 2:36.

*Nuance with moving the data via Disk Utility Restore: When initially booting from the original drive over USB I selected Macintosh HD instead of Recovery HD. Disk Utility wouldn't let me restore from the original drive to the new drive. I had to choose Recovery HD instead of Macintosh HD from the boot menu. That led to a simple menu (including Disk Utility) instead of my normal OS X environment, and then I was able to restore from the original drive.

Thursday, May 22, 2014

Which Apache httpd module failed the request?

I was reminded today of a module I wrote a while back for Apache httpd 2.4 when I was debugging a config snippet from a customer and saw

[core:trace3] ... request.c(311): fixups hook gave 400: /

Whatever module set 400 (HTTP_BAD_REQUEST) didn't log anything. If you have Apache httpd 2.4 and build it yourself, this type of issue can be solved with mod_hook_ar. This message from mod_hook_ar significantly shrank the search area:

[hook_ar:error] mod_rewrite.c fixups -> 400

Unfortunately, mod_hook_ar doesn't currently have its own web page. You can download the code and information about building it from, and you can read about it starting at slide 46 in this presentation:

Tuesday, May 20, 2014

Recent fun with mod_whatkilledus and mod_backtrace

After Apache httpd 2.4.10 is released, I'll push out a new release of these modules.

The most critical issue resolved in the next release is a problem mod_whatkilledus has with tracking the active request when using the Event MPM. It doesn't notice when the Event MPM maps the request to a different thread while processing the response, so the request which triggered the crash could be misidentified. By using a new web server API I added to httpd 2.4.10, mod_whatkilledus now notices when request processing moves across threads.

Other useful changes in the next release:

  • Windows: Support 64-bit builds, and add the option of disabling the system pop-up error handling dialog after a child process crash, which allows the server to automatically recover and handle new clients. (An alternative to this httpd-specific setting is a Windows registry setting; it has more capabilities, such as enabling a user dump, but it affects all applications.)
  • Linux, FreeBSD, OS X: Optionally use libunwind, for better backtraces in some circumstances.

Strangely, libunwind's unw_step() returns no stack frames in 32-bit builds on OS X, returning 0 on the first call. I didn't encounter any issues with 64-bit builds. (This is with the libunwind APIs in the system library in both Lion and Mavericks, not with libunwind built from source.)

Monday, May 5, 2014

Comparing performance of SCGI and uwsgi protocols

I've been playing with httpd and nginx using different protocols to route to a Python application, and one of the questions that has arisen is whether or not I should try uwsgi (the protocol, not the uWSGI application) with httpd. The third-party module mod_proxy_uwsgi needs some debugging first, so it isn't as simple as modifying one of my existing .conf snippets and seeing how fast it goes.

For the purposes of reverse proxy behind httpd or nginx, uwsgi is essentially SCGI with a more efficient encoding of the length of the parameters passed over — sending over binary lengths with the strings instead of just the string, that then has to be traversed to find the terminating binary zero. I don't think the work in the web server is significantly different either way, though I think SCGI is cheaper for the web server because when copying over HTTP headers and other data it does a strlen() on the var name and value, then a memcpy() for each, including the terminating '\0' so that the backend knows the extent. uwsgi requires copying both strings (without the '\0') and building a two-byte binary length for each in a particular endian order (i.e., shifts and masks). This extra work for uwsgi more than overcomes the savings with the encoding of a couple of other lengths, for which SCGI requires building a printable string.

So how can I make an approximation of the speed-up without actually debugging the uwsgi support for httpd? First, it seems worthwhile to look at nginx. Here are some numbers with nginx, where the request consists of a POST with 71,453-byte body that is written to a very simple WSGI application running under uWSGI, which simply echoes it back. (8 runs each with ab, 10,000 requests, concurrency 100, throw out high and low runs)

nginx, SCGI over a Unix socket4,332
nginx, uwsgi over a Unix socket4,363

So over a lot of runs and a lot of requests per run, uwsgi overall (web server processing + uWSGI processing) has an edge by a little less than one percent.

I came up with a lame experiment with mod_proxy_scgi to see how an improvement in the efficiency of dealing with the parameters might help. In the following patch, I simply remove the second pass of touching (and copying) the parameters in mod_proxy_scgi. Of course this only works if all requests are exactly the same, as in my little benchmark :)

--- mod_proxy_scgi.c 2014-05-05 13:21:39.253193636 -0400
+++ no-header-building 2014-05-05 13:15:33.785184159 -0400
@@ -247,6 +247,8 @@
     return OK;
+static char *saved_headers;
+static apr_size_t saved_headers_size;
  * Send SCGI header block
@@ -292,6 +294,11 @@
     ns_len = apr_psprintf(r->pool, "%" APR_SIZE_T_FMT ":", headerlen);
     len = strlen(ns_len);
     headerlen += len + 1; /* 1 == , */
+    if (getenv("foo") && saved_headers) {
+        return sendall(conn, saved_headers, saved_headers_size, r);
+    }
     cp = buf = apr_palloc(r->pool, headerlen);
     memcpy(cp, ns_len, len);
     cp += len;
@@ -320,6 +327,15 @@
     *cp++ = ',';
+    if (getenv("foo") && !saved_headers) {
+        char *tmp = malloc(headerlen);
+        memcpy(tmp, buf, headerlen);
+        saved_headers_size = headerlen;
+        saved_headers = tmp;
+        ap_log_rerror(APLOG_MARK, APLOG_WARNING, 0, r,
+                      "Saved headers...");
+    }
     return sendall(conn, buf, headerlen, r);

It doesn't exactly match what uWSGI has to do extra with SCGI, but it does knock off some processing in the web server.

httpd, SCGI over a Unix socket, no optimization9,997
httpd, SCGI over a Unix socket, optimization10,014

(Again, these are the averages of multiple runs after throwing out highs and lows.)

Hmmm... Even getting rid of a lot of strlen() and memcpy() calls (my rough attempt to trade cycles in httpd for the cycles that would have been saved in uWSGI if we used uwsgi) resulted in much less than one percent improvement. I think I'll stick with SCGI for now, and I don't even think it is worthwhile to change httpd's SCGI implementation to build the header in a single pass, which would get back only some of the cycles saved by the benchmark-specific optimization shown above. (And I don't think httpd is suffering by not having a bundled, reliable implementation of uwsgi.)

Friday, May 2, 2014

Lingering close

One of the early pieces of code I tackled in httpd was APR-izing lingering_close(). I recall dean gaudet ensuring that I didn't screw it up. (See an early part of the conversation.)

Perhaps a year or two later, some colleagues in z/OS TCP/IP and SNA, where I had worked before joining the IBM team working on httpd, let me know that a file transfer program I had written long before had stopped working reliably when transferring to or from z/OS after some updates in z/OS TCP/IP. (Why does one reinvent file transfer? I decided to learn sockets programming but got tired of all the ifdefs to support Windows and OS/2 and VM/CMS and MVS, so I did what everyone else did and wrote a high-level library. A file transfer program comes next, right? Anyway, it was used by a number of colleagues for higher level features like printing to a workstation printer from VM/CMS, XEDIT macros that interacted with your PC clipboard, and other fun stuff.) At any rate, it was good that I had learned about lingering close via httpd because otherwise I would have been shocked at the reason behind the intermittent failures in the file transfer program; every indication was that both client and server modes were doing exactly what they needed to do, but one of the peers could get ECONNRESET before it had finished reading the response. IIRC, the lingering close logic was then implemented and some small amount of happiness ensued, but I didn't have time to rework the build for the then-available tools on VM/CMS, and a big use-case died. Sorry, folks!

Fast forward through most of the life of the web... I've been playing recently with httpd and nginx in front of uWSGI and writing up my notes in this in-progress tutorial. After initially encountering a bug where uWSGI doesn't eat the empty FCGI_STDIN record with FastCGI, now that it is fixed I'm left with a familiar scenario: The server (uWSGI in this case) writes the entire response to the client (FastCGI gateway) and then calls close(), and sometimes (more often over a slow network) the gateway gets ECONNRESET before it can read everything.

What is authoritative text on the subject? I don't know. Most discussions about getting ECONNRESET on a read-type call on the Internet do not talk about getting hit by RST jumping ahead of data already copied by the server to the TCP send buffers. Some discussions raise the issue about the client perhaps trying to send data after the server has closed. (The lingering close logic as normally implemented in TCP servers helps a lot in those cases.) This ancient Apache httpd 1.3 comment is as succinct as any I know of in describing the problem:

 * in a nutshell -- if we don't make this effort we risk causing
 * TCP RST packets to be sent which can tear down a connection before
 * all the response data has been sent to the client.

Here is a blog page that covers this more disturbing scenario very clearly:

So can I trigger the issue at will with a simple client-or-server program? At first I couldn't. After quite a bit of experimentation, the answer is definitely yes. It isn't hard to trigger with httpd's mod_proxy_fcgi and uWSGI and a particular application-level flow and uWSGI configured to run the simple application on multiple processes and threads, but it wasn't so easy with this simple program.

The final methodology to see the error:

$ ./a.out server
(no trick there)
Run this in 8 or preferably more different terminal sessions:
$ while ./a.out client; do echo ok; done
# while no-error-on-any-client-terminal-session; do Go clean some room in your house; done

Client and server are on two different machines connected by a relatively slow Wi-Fi.

As long as you see that the server displays no socket call errors but at least one client displays something like read: Connection reset by peer, you've encountered the error. Maybe errors will occur only with more clients in your environment. After seeing a few failures in my setup with around twelve client loops, I went for a long walk and found after my return that seven were still happy (many successful transactions with zero errors).

Can SO_LINGER help any, at least on recent-Ubuntu talking to recent-Ubuntu? I'll try that, though I think the tried and true server logic (shutdown for write then wait for timeout or client to read) is the safest solution.

Massive fail

Early results were incorrect; I had forgotten to set the listen backlog to a high enough number to avoid TCP resetting connections. More results to be posted later...

The moral of the story: If you get ECONNRESET on a read-type call before having read any data, check for an issue with listen backlog first, in which case the server would have never processed the connection before it was reset.

Monday, April 28, 2014

sysdig really is strace plus lsof

What does that mean, anyway? This morning it meant that I could see helpful internal file table information in the context of syscalls, so I could see more than just the file descriptors shown by strace.

The problem

A simple WSGI application which is managed by uWSGI is sitting behind either httpd or nginx, and I've been trying out different protocols, socket types, and both httpd and nginx with it. With httpd and FastCGI over TCP connecting to uWSGI, I get several I/O failures out of a thousand. With httpd and FastCGI over AF_UNIX I get perhaps 40 or 50. I don't see any errors with nginx, and I don't see any errors with HTTP or SCGI. What is going on?

The AF_UNIX scenario has the most frequent I/O errors, so I started debugging that. mod_proxy_fcgi sees EPIPE intermittently when writing 8 bytes to the application (after sending the FastCGI begin and parameters data).

I configured httpd (Event MPM) to use a single child process to serve requests, and configured uWSGI to do the same, to make data analysis easier.

I figured out a relatively short (i.e., limited data collection) ab invocation to trigger the failure (50 requests, concurrency 5 IIRC), and tried it several times until it failed. Each time I ran sudo sysdig -w trace.scap to capture data during the run, hoping for a failure. (The capture was started anew for every run to minimize the data collected.) After a short time, I hit 1 EPIPE scenario and had a matching capture. Now what?

22000 was the httpd child pid and 20881 was the uWSGI pid. Here are a few sysdig invocations used to start getting acquainted with the data. (No, I don't necessarily recommend egrep over sysdig expressions :) ).

$ sudo sysdig -r trace.scap " or" |less
$ sudo sysdig -r trace.scap | egrep '(connect|writev|close|read|PIPE)' |less
$ sudo sysdig -r trace.scap "( or and fd.type=unix" |less
(and more)

The view with both processes and just the operations on AF_UNIX sockets was the most helpful. Searching for EPIPE yields this information:

89254 07:12:54.259984241 6 httpd (22014) > writev fd=26(<u>ffff880108bcc440->ffff880108bcba80 /tmp/helloFastCGI.s) size=8 
89285 07:12:54.260023713 6 httpd (22014) < writev res=-32(EPIPE) data=........

The long hex addresses (ffff880108bcc440 and ffff880108bcba80) are the key to seeing what happens inside both processes, since the file descriptors won't match between processes.

$ sysdig -r trace.scap "( or and fd.type=unix"  | egrep '(ffff880108bcc440|EPIPE)'
87796 07:12:54.244173691 6 httpd (22014) < connect res=0 tuple=ffff880108bcc440->ffff880108bcba80 /tmp/helloFastCGI.s 
87797 07:12:54.244176698 6 httpd (22014) > fcntl fd=26(<u>ffff880108bcc440->ffff880108bcba80 /tmp/helloFastCGI.s) cmd=4(F_GETFL) 
87800 07:12:54.244178198 6 httpd (22014) > fcntl fd=26(<u>ffff880108bcc440->ffff880108bcba80 /tmp/helloFastCGI.s) cmd=5(F_SETFL) 
87802 07:12:54.244180563 6 httpd (22014) > writev fd=26(<u>ffff880108bcc440->ffff880108bcba80 /tmp/helloFastCGI.s) size=16 
88619 07:12:54.259281618 7 uwsgi (20882) < accept fd=10(<u>ffff880108bcc440->ffff880108bcba80 /tmp/helloFastCGI.s) tuple=ffff880108bcc440->ffff880108bcba80 /tmp/helloFastCGI.s queuepct=0 
88627 07:12:54.259287565 7 uwsgi (20882) > read fd=10(<u>ffff880108bcc440->ffff880108bcba80 /tmp/helloFastCGI.s) size=4096 
89021 07:12:54.259721201 6 httpd (22014) > writev fd=26(<u>ffff880108bcc440->ffff880108bcba80 /tmp/helloFastCGI.s) size=699 
89040 07:12:54.259743588 6 uwsgi (20882) > read fd=10(<u>ffff880108bcc440->ffff880108bcba80 /tmp/helloFastCGI.s) size=4096 
89083 07:12:54.259786716 6 httpd (22014) > writev fd=26(<u>ffff880108bcc440->ffff880108bcba80 /tmp/helloFastCGI.s) size=8 
89097 07:12:54.259802288 6 uwsgi (20882) > read fd=10(<u>ffff880108bcc440->ffff880108bcba80 /tmp/helloFastCGI.s) size=4096 
89148 07:12:54.259862936 6 uwsgi (20882) > write fd=10(<u>ffff880108bcc440->ffff880108bcba80 /tmp/helloFastCGI.s) size=8 
89158 07:12:54.259872030 6 uwsgi (20882) > write fd=10(<u>ffff880108bcc440->ffff880108bcba80 /tmp/helloFastCGI.s) size=43 
89161 07:12:54.259874552 6 uwsgi (20882) > write fd=10(<u>ffff880108bcc440->ffff880108bcba80 /tmp/helloFastCGI.s) size=8 
89165 07:12:54.259877021 6 uwsgi (20882) > write fd=10(<u>ffff880108bcc440->ffff880108bcba80 /tmp/helloFastCGI.s) size=11 
89184 07:12:54.259892476 6 uwsgi (20882) > write fd=10(<u>ffff880108bcc440->ffff880108bcba80 /tmp/helloFastCGI.s) size=24 
89188 07:12:54.259895312 6 uwsgi (20882) > close fd=10(<u>ffff880108bcc440->ffff880108bcba80 /tmp/helloFastCGI.s) 
89254 07:12:54.259984241 6 httpd (22014) > writev fd=26(<u>ffff880108bcc440->ffff880108bcba80 /tmp/helloFastCGI.s) size=8 
89285 07:12:54.260023713 6 httpd (22014) < writev res=-32(EPIPE) data=........ 
89314 07:12:54.260054390 6 httpd (22014) > close fd=26(<u>ffff880108bcc440->ffff880108bcba80 /tmp/helloFastCGI.s)

The buffer at ffff880108bcc440 was also used for other socket connections, though not concurrently. I omitted the data for connections that didn't have the EPIPE error.

So httpd writes 16 then 699 then 8 then dies trying to write 8 more. uWSGI reads several times with a 4096-byte buffer, but we have to dig into the trace to find out how much it really read (following the "< read" entry). uWSGI then writes 8 then 43 then 8 then 11 then 24 bytes, then closes before httpd tried to write the last 8.

The missing data showing how much uWSGI actually received when asking for 4K is

88633 07:12:54.259291681 7 uwsgi (20882) < read res=16 data=................
89045 07:12:54.259748834 6 uwsgi (20882) < read res=699 data=..........HTTP_HOST127.0.0.1..HTTP_USER_AGENTApacheBench/2.3..HTTP_ACCEPT*/*.<PA
89104 07:12:54.259806481 6 uwsgi (20882) < read res=8 data=........

So uWSGI read the 16, the 699, and the 8 but never tried to read again before closing.

Looking at the mod_proxy_fcgi source code I see that the final 8, the one that occasionally fails with EPIPE, is for telling the application that there are 0 bytes of FastCGI STDIN (i.e., no request body). So it seems that if mod_proxy_fcgi can get those 8 bytes in the socket buffer before uWSGI closes the socket it won't get a write, and usually it is able to. Next I added a sleep for 15 milliseconds just before that last write. With that it fails every time, since those 8 bytes are never in the socket buffer before uWSGI closes.

So why does it work reliably with nginx? It has all the data (begin request plus params/envvars + end of stdin) in a single buffer, so it doesn't get caught. uWSGI doesn't read it anyway, but there's no EPIPE error.

I looked a bit further at uWSGI, in particular its FastCGI protocol handler. Some higher-level code calls into the protocol handler to perform different work; one call is to read the request body. I found by setting breakpoints in gdb that the function to read the request body is never called. For some protocols that makes sense, since you can tell from the request header fields for this request that there's no body. So perhaps the FastCGI protocol handler needs to be smarter and slurp up the last 8 byte record earlier in this scenario. More investigation is required, presumably with gdb, to understand the flows between the generic code and the FastCGI protocol handler. (It seems very simple, but initial testing with a POST request didn't show what I expected.)

While playing with this I created a workaround in mod_proxy_fcgi:

             rv = send_data(conn, vec, nvec, &len, 0);
             if (rv != APR_SUCCESS) {
-                break;
+                if (last_stdin && writebuflen == 0) {
+                    ap_log_rerror(APLOG_MARK, APLOG_ERR, rv, r, "ignoring error sending empty stdin");
+                }
+                else {
+                    break;
+                }

It isn't harmful, but it still looks like uWSGI needs a fix.

Sometimes you win, sometimes you lose. Recently I was trying to find whether a file was opened before or after a change in identity, but AFAICT sysdig doesn't have a filter for the syscall, and so it continues to report (if you ask for it in a -p format) the original uid/gid.

Tuesday, April 22, 2014

Busy day...

Over the last couple of days I committed my implementation of Certificate Transparency features for httpd (to trunk). The doc is finally in reasonable shape (here). With any luck, someone will actually try it out and provide feedback before I forget everything.

github/trawick/ct-httpd still has a few useful nuggets. I'll put up instructions there soon for using the CT features with httpd 2.4.

I've been thinking about how to pervert the talks I gave at ApacheCon US 2014 for Budapest in the fall (assuming that I can save up enough quarters for aviation fuel). As part of that I queried the local Python user group for information about how they used Apache httpd to front end Python web apps. Apparently Apache httpd isn't very popular, except perhaps to keep ancient mod_rewrite-heavy recipes running. nginx has some very accessible documentation in this space, both in the official documentation as well as within the documentation for various Python web-related projects.

I started writing some notes today, possibly as my attempt at documentation, possibly as a rough draft for a future talk. In it I plan to show httpd and nginx recipes side by side, without preferring one over the other. A lot of work remains.

Monday, April 21, 2014

mod_qos and QS_EventKBytesPerSecLimit

Previously I noted in the mod_qos discussion group some odd testing results for this feature. I didn't receive any advice.

As I noted in a recent follow-up (still stuck in moderation???), I've worked up an alternate implementation for limiting output bandwidth for requests matching a certain event. That can be found at github/trawick/tweak-qos. I have of course solicited advice from the author and offered my changes under the same license as the original.

Saturday, April 12, 2014

Where did that .pod end up? (sysdig)

So I RTFM yesterday and, not having any OpenSSL muscle memory, spelled it STACKOF in my program just like in the man page. Fail.

The .pod file was easy enough to fix, but does it format properly? What happens to it and where does it go?

$ sudo sysdig evt.type=open and contains cert_chain
(wait for command in other terminal)
2901943 13:35:42.114428066 7 sh (24543) < open fd=3(/home/trawick/git/work-on-openssl-master/doc/ssl/SSL_get_peer_cert_chain.pod) name=doc/ssl/SSL_get_peer_cert_chain.pod(/home/trawick/git/work-on-openssl-master/doc/ssl/SSL_get_peer_cert_chain.pod) flags=1(O_RDONLY) mode=0 
2902384 13:35:42.117743930 7 sh (24544) < open fd=3(/home/trawick/inst/omaster/ssl/man/man3/SSL_get_peer_cert_chain.3) name=/home/trawick/inst/omaster/ssl/man/man3/SSL_get_peer_cert_chain.3 flags=262(O_TRUNC|O_CREAT|O_WRONLY) mode=0 
2906809 13:35:42.172268013 5 perl (24547) < open fd=4(/home/trawick/git/work-on-openssl-master/doc/ssl/SSL_get_peer_cert_chain.pod) name=SSL_get_peer_cert_chain.pod(/home/trawick/git/work-on-openssl-master/doc/ssl/SSL_get_peer_cert_chain.pod) flags=1(O_RDONLY) mode=0 
2907040 13:35:42.176303490 7 sh (24548) < open fd=3(/home/trawick/git/work-on-openssl-master/doc/ssl/SSL_get_peer_cert_chain.pod) name=doc/ssl/SSL_get_peer_cert_chain.pod(/home/trawick/git/work-on-openssl-master/doc/ssl/SSL_get_peer_cert_chain.pod) flags=1(O_RDONLY) mode=0
$ make install_docs

That's more satisfying than strace since I don't have to run make install_docs under strace and decide how to separate their outputs. Now to use sysdig as often as possible for a while so I can get used to it...

Friday, April 11, 2014

My httpd-related slides from ApacheCon NA 2014, earlier this week

Apache HTTP Server and CVE-2014-0160, the so called Heartbleed Bug

SSL/TLS-enabled configurations of Apache HTTP Server with OpenSSL 1.0.1 through 1.0.1f are vulnerable to CVE-2014-0160, the so called Heartbleed Bug. (OpenSSL versions smaller than 1.0.1, such as 0.9.8 and 1.0.0, are safe.)

No Apache HTTP Server fix is needed to resolve this; no Apache HTTP Server configuration change besides disabling SSL/TLS completely can resolve this. (SSLv2 and SSLv3 are not vulnerable to CVE-2014-0160, but limiting the configuration to one or both of those protocols is not recommended because of other security concerns.) Instead, a patch to OpenSSL, a rebuild of OpenSSL with the TLS Heartbeat extension disabled, or an upgrade of OpenSSL to 1.0.1g or later is required.

In some cases OpenSSL may be statically linked with mod_ssl. httpd and mod_ssl must be rebuilt with the updated OpenSSL when OpenSSL is statically linked with mod_ssl. Note: The build of mod_spdy may rebuild mod_ssl in this manner.

If you build OpenSSL yourself, refer to the OpenSSL project for further information, including the advisory at

Checking for mod_ssl statically linked with OpenSSL


$ nm /path/to/httpd/modules/ | grep X509_STORE_CTX_free
                 U X509_STORE_CTX_free...

The U in the output means that the relevant OpenSSL library is not statically linked with mod_ssl.

Other platforms

Similar checks for symbols work elsewhere. ldd is another way on many Unix-like platforms to determine where mod_ssl finds OpenSSL. depends can be used on Windows to see if relies on OpenSSL.

When to consult with vendors or other third parties

Contact a third-party supplier in the following situations to determine the applicability of CVE-2014-0160 to your server:

  • You obtain OpenSSL in binary form with or without Apache HTTP Server
  • You are using a commercial product based on Apache HTTP Server
  • You are otherwise using mod_ssl or a replacement for it from a third party
  • Build or installation of a third-party feature (e.g., mod_spdy) rebuilds mod_ssl


Correct the affected versions: 1.0.1 (without the a) is the earliest affected version. Mention explicitly that older versions are not affected. Suggested by Rainer Jung

Wednesday, January 22, 2014

Sorry, the application python2.7 has stopped unexpectedly.

If you notice further problems, try restarting the computer.

Title: python2.7 crashed with Exception in __init__(): invalid httpd install dir

Horse poop! Why the heck are you generating a crash report for this? Some Python code I wrote used an exception to report a setup problem.