Saturday, August 2, 2014

Google App Engine: skipping complete directory via skip_files


I forgot all about the skip_files in app.yaml and asked my favorite search engine what to do. A number of the top pages returned made it unnecessarily complicated. Just do this to skip directories (.idea and tools in this example):

...
skip_files:
- ^(.*/)?#.*#$
- ^(.*/)?.*~$
- ^(.*/)?.*\.py[co]$
- ^(.*/)?.*/RCS/.*$
- ^(.*/)?\..*$
- ^\.idea$
- ^tools$
...

(These are top-level directories within my app engine project.)

Add -v to your appcfg.py ... update ... invocation to confirm, from messages like

2014-08-02 09:19:20,424 INFO appcfg.py:2568 Ignoring directory '.idea': Directory matches ignore regex. 
2014-08-02 09:19:20,425 INFO appcfg.py:2426 Processing file 'app.yaml' 
2014-08-02 09:19:20,426 INFO appcfg.py:2426 Processing file 'index.yaml' 
2014-08-02 09:19:20,426 INFO appcfg.py:2426 Processing file 'main.py' 
2014-08-02 09:19:20,426 INFO appcfg.py:2561 Ignoring file 'main.pyc': File matches ignore regex. 
2014-08-02 09:19:20,426 INFO appcfg.py:2426 Processing file 'not_found.py' 
2014-08-02 09:19:20,427 INFO appcfg.py:2568 Ignoring directory 'tools': Directory matches ignore regex. 
2014-08-02 09:19:20,429 INFO appcfg.py:2426 Processing file 'static/Chart.min.js' 
...

Friday, August 1, 2014

Another year (or two), another release of mod_whatkilledus and mod_backtrace


mod_whatkilledus and mod_backtrace V2.01

I have a sinking feeling that this isn't pertinent to a fairly large percentage of the world's inhabitants, but it was a fair amount of hours spread over time and it's finally done, so I'll pretend otherwise.

mod_whatkilledus improvements

  • The name of the log file can now be configured with the WKULogfile directive.
  • On Windows: Add directive WKUDisableWindowsErrorBox to allow disabling the pop-up error handling dialog after a child process crash.
  • On Windows: Handle exceptions in 64-bit builds.
  • With Event MPM on Unix: When building with httpd 2.4.10 or later, properly track request processing across threads with the Event MPM. Previously, the wrong client and/or request might be logged as the trigger for a crash (with the Event MPM only). The change is not effective when building with httpd 2.4.9 or earlier.

mod_backtrace improvements

  • Support libunwind on Linux, FreeBSD, and OS X. This provides better resolution of function names, so a raw address will be provided in lieu of the function name in fewer situations. (Add LIBUNWIND=yes to the make invocation.) OS X note: A libunwind build silently fails to capture a backtrace with 32-bit builds, so use the default mechanism with 32-bit builds on OS X.
  • Filter out internal functions from backtraces so that only the caller's stack frames are reported.
  • Improve formatting of a backtrace for the error log field.

Windows binaries

64-bit binaries are now included for use with httpd 2.2 and 2.4 on Windows. (32-bit binaries are still included.)

Build improvements

  • Support make APXS=/path/to/apxs for building with non-default install layouts.
  • Support Clang on FreeBSD 10.
  • Add -funwind-tables on ARM platform for better backtraces.
  • mod_whatkilledus fails to build in a less mysterious way when using an httpd build that doesn't have exception hooks enabled.

Friday, May 30, 2014

Is a script to run a command within a virtualenv missing from virtualenv?


Contents of script run_with_env.sh:

#!/bin/sh

envdir=$1
shift

curdir=$1
shift

. $envdir/bin/activate && cd $curdir && exec "$@"

This runs a specified command (plus arguments) in a particular directory within a particular virtualenv.

Example use:

$ crontab -l
...
30 0 * * * /home/trawick/myhg/apache/bin/run_with_env.sh git/edurepo/envs/edurepo git/edurepo/src/edurepo python teachers/pretend_teacher.py
...

July 4 update:

See also vex.

Monday, May 26, 2014

If you replace the hard drive in your MacBook...


Be sure to check the startup disk setting after you are successfully booting from the new drive.

We just replaced the original drive in a Core 2 Duo White MacBook (2008, 4.1) with a 256GB Crucial M500 SSD, following some instructions on Apple Support Communities (search for Second way) for moving the data*. Our power-button-to-productivity benchmark of bootup+login+Chrome-window-appears+Word-window-appears was 2:36 with the original drive and 1:08 with the SSD drive. But that 1:08 included a surprising 25-30 second delay after the screen lit up on power-on but before the Apple logo appeared. There are multiple causes described in this article, but the simple issue for us was selection of the startup disk in System Preferences. After selecting the Crucial SSD as the startup disk, the annoying delay at power up was gone. Thus our power-button-to-productivity time is more like 0:40, down from the original 2:36.

*Nuance with moving the data via Disk Utility Restore: When initially booting from the original drive over USB I selected Macintosh HD instead of Recovery HD. Disk Utility wouldn't let me restore from the original drive to the new drive. I had to choose Recovery HD instead of Macintosh HD from the boot menu. That led to a simple menu (including Disk Utility) instead of my normal OS X environment, and then I was able to restore from the original drive.

Thursday, May 22, 2014

Which Apache httpd module failed the request?


I was reminded today of a module I wrote a while back for Apache httpd 2.4 when I was debugging a config snippet from a customer and saw

[core:trace3] ... request.c(311): fixups hook gave 400: /

Whatever module set 400 (HTTP_BAD_REQUEST) didn't log anything. If you have Apache httpd 2.4 and build it yourself, this type of issue can be solved with mod_hook_ar. This message from mod_hook_ar significantly shrank the search area:

[hook_ar:error] mod_rewrite.c fixups -> 400

Unfortunately, mod_hook_ar doesn't currently have its own web page. You can download the code and information about building it from http://emptyhammock.com/downloads/, and you can read about it starting at slide 46 in this presentation:

Tuesday, May 20, 2014

Recent fun with mod_whatkilledus and mod_backtrace


After Apache httpd 2.4.10 is released, I'll push out a new release of these modules.

The most critical issue resolved in the next release is a problem mod_whatkilledus has with tracking the active request when using the Event MPM. It doesn't notice when the Event MPM maps the request to a different thread while processing the response, so the request which triggered the crash could be misidentified. By using a new web server API I added to httpd 2.4.10, mod_whatkilledus now notices when request processing moves across threads.

Other useful changes in the next release:

  • Windows: Support 64-bit builds, and add the option of disabling the system pop-up error handling dialog after a child process crash, which allows the server to automatically recover and handle new clients. (An alternative to this httpd-specific setting is a Windows registry setting; it has more capabilities, such as enabling a user dump, but it affects all applications.)
  • Linux, FreeBSD, OS X: Optionally use libunwind, for better backtraces in some circumstances.

Strangely, libunwind's unw_step() returns no stack frames in 32-bit builds on OS X, returning 0 on the first call. I didn't encounter any issues with 64-bit builds. (This is with the libunwind APIs in the system library in both Lion and Mavericks, not with libunwind built from source.)