Tuesday, January 27, 2015

Today's Python homework


The card I gave myself today says "Read Python language docs until I have written down 10 new things." Why? Somebody pointed out that var = val or '-' was cleaner than the mess I used instead, and I realized that much more effort is required. (Were it Perl, I'd probably have used or '-', but I can't always keep the features of the different languages straight.)

Here's today's list, which includes some things I'm familiar with in Python or other languages but can't pull out of a hat on demand.

  1. The file is UTF-8: # -*- coding: utf-8 -*- (or UTF-8 byte-order mark or VIM-style coding string)
  2. Identifiers named _* are not imported by from module import *. (I knew that a leading _ is for privacy but I didn't realize it was enforced anywhere besides analysis tools. But I don't think I use from module import * except with cascaded Django settings.)
  3. Identifiers named __* are class-private, and this is essentially enforced.
  4. Wow, combining string prefixes, as in ur or br, is new to me, as is using upper case string prefixes, such as R, bR, etc.
  5. I forgot about escape sequences for Bell, Backspace, Formfeed, and Vertical Tab, and never knew about \N{name-of-Unicode-character}. (I probably didn't know about octal escape sequences in Python either.)
  6. I don't always use whitespace to concatenate string literals, though I should where applicable.
  7. complex literals — 3.14j, 3.14e-10j
  8. ^ operator, for XOR
  9. the NotImplemented single-valued type, for returning from certain methods
  10. the Ellipses single-valued type, for indicating the presence of the ... syntax in a slice

Saturday, August 2, 2014

Google App Engine: skipping complete directory via skip_files


I forgot all about the skip_files in app.yaml and asked my favorite search engine what to do. A number of the top pages returned made it unnecessarily complicated. Just do this to skip directories (.idea and tools in this example):

...
skip_files:
- ^(.*/)?#.*#$
- ^(.*/)?.*~$
- ^(.*/)?.*\.py[co]$
- ^(.*/)?.*/RCS/.*$
- ^(.*/)?\..*$
- ^\.idea$
- ^tools$
...

(These are top-level directories within my app engine project.)

Add -v to your appcfg.py ... update ... invocation to confirm, from messages like

2014-08-02 09:19:20,424 INFO appcfg.py:2568 Ignoring directory '.idea': Directory matches ignore regex. 
2014-08-02 09:19:20,425 INFO appcfg.py:2426 Processing file 'app.yaml' 
2014-08-02 09:19:20,426 INFO appcfg.py:2426 Processing file 'index.yaml' 
2014-08-02 09:19:20,426 INFO appcfg.py:2426 Processing file 'main.py' 
2014-08-02 09:19:20,426 INFO appcfg.py:2561 Ignoring file 'main.pyc': File matches ignore regex. 
2014-08-02 09:19:20,426 INFO appcfg.py:2426 Processing file 'not_found.py' 
2014-08-02 09:19:20,427 INFO appcfg.py:2568 Ignoring directory 'tools': Directory matches ignore regex. 
2014-08-02 09:19:20,429 INFO appcfg.py:2426 Processing file 'static/Chart.min.js' 
...

Friday, August 1, 2014

Another year (or two), another release of mod_whatkilledus and mod_backtrace


mod_whatkilledus and mod_backtrace V2.01

I have a sinking feeling that this isn't pertinent to a fairly large percentage of the world's inhabitants, but it was a fair amount of hours spread over time and it's finally done, so I'll pretend otherwise.

mod_whatkilledus improvements

  • The name of the log file can now be configured with the WKULogfile directive.
  • On Windows: Add directive WKUDisableWindowsErrorBox to allow disabling the pop-up error handling dialog after a child process crash.
  • On Windows: Handle exceptions in 64-bit builds.
  • With Event MPM on Unix: When building with httpd 2.4.10 or later, properly track request processing across threads with the Event MPM. Previously, the wrong client and/or request might be logged as the trigger for a crash (with the Event MPM only). The change is not effective when building with httpd 2.4.9 or earlier.

mod_backtrace improvements

  • Support libunwind on Linux, FreeBSD, and OS X. This provides better resolution of function names, so a raw address will be provided in lieu of the function name in fewer situations. (Add LIBUNWIND=yes to the make invocation.) OS X note: A libunwind build silently fails to capture a backtrace with 32-bit builds, so use the default mechanism with 32-bit builds on OS X.
  • Filter out internal functions from backtraces so that only the caller's stack frames are reported.
  • Improve formatting of a backtrace for the error log field.

Windows binaries

64-bit binaries are now included for use with httpd 2.2 and 2.4 on Windows. (32-bit binaries are still included.)

Build improvements

  • Support make APXS=/path/to/apxs for building with non-default install layouts.
  • Support Clang on FreeBSD 10.
  • Add -funwind-tables on ARM platform for better backtraces.
  • mod_whatkilledus fails to build in a less mysterious way when using an httpd build that doesn't have exception hooks enabled.

Friday, May 30, 2014

Is a script to run a command within a virtualenv missing from virtualenv?


Sometimes the python command in the virtualenv isn't quite enough.

Contents of script run_with_env.sh:

#!/bin/sh

envdir=$1
shift

curdir=$1
shift

. $envdir/bin/activate && cd $curdir && exec "$@"

This runs a specified command (plus arguments) in a particular directory within a particular virtualenv.

Example use:

$ crontab -l
...
30 0 * * * /home/trawick/myhg/apache/bin/run_with_env.sh git/edurepo/envs/edurepo git/edurepo/src/edurepo python teachers/pretend_teacher.py
...

July 4 update:

See also vex.

Monday, May 26, 2014

If you replace the hard drive in your MacBook...


Be sure to check the startup disk setting after you are successfully booting from the new drive.

We just replaced the original drive in a Core 2 Duo White MacBook (2008, 4.1) with a 256GB Crucial M500 SSD, following some instructions on Apple Support Communities (search for Second way) for moving the data*. Our power-button-to-productivity benchmark of bootup+login+Chrome-window-appears+Word-window-appears was 2:36 with the original drive and 1:08 with the SSD drive. But that 1:08 included a surprising 25-30 second delay after the screen lit up on power-on but before the Apple logo appeared. There are multiple causes described in this article, but the simple issue for us was selection of the startup disk in System Preferences. After selecting the Crucial SSD as the startup disk, the annoying delay at power up was gone. Thus our power-button-to-productivity time is more like 0:40, down from the original 2:36.

*Nuance with moving the data via Disk Utility Restore: When initially booting from the original drive over USB I selected Macintosh HD instead of Recovery HD. Disk Utility wouldn't let me restore from the original drive to the new drive. I had to choose Recovery HD instead of Macintosh HD from the boot menu. That led to a simple menu (including Disk Utility) instead of my normal OS X environment, and then I was able to restore from the original drive.

Thursday, May 22, 2014

Which Apache httpd module failed the request?


I was reminded today of a module I wrote a while back for Apache httpd 2.4 when I was debugging a config snippet from a customer and saw

[core:trace3] ... request.c(311): fixups hook gave 400: /

Whatever module set 400 (HTTP_BAD_REQUEST) didn't log anything. If you have Apache httpd 2.4 and build it yourself, this type of issue can be solved with mod_hook_ar. This message from mod_hook_ar significantly shrank the search area:

[hook_ar:error] mod_rewrite.c fixups -> 400

Unfortunately, mod_hook_ar doesn't currently have its own web page. You can download the code and information about building it from http://emptyhammock.com/downloads/, and you can read about it starting at slide 46 in this presentation: