Discuss Here

Here is a video demonstrating some of the features, remember the lights on screen are just simple proxy lights for demo purposes. The main output is realtime DMX – which is an already working feature, I just don’t have it hooked up to light gear yet.

Here is the code listing that created the demo sequence elements:

#!/usr/bin/env python
# encoding: utf-8

from midi_reader import MidiDispatcher
import time
from lights import *
import sys


# create a light show - manages the updating of all lights
show = LightShow()

light_seqA = LightSequence()
light_seqB = LightSequence()
light_seqC = LightSequence()
light_seqD = LightSequence()


# create our 8 lights and add them to each sequence and the show
for i in range(1,9):
    l = LightUnit(channels=[i])
    l.release = .3
    show.lights.append(l)
    light_seqA.elements.append(SequenceElement(l))
    light_seqB.elements.append(SequenceElement(l))
    light_seqC.elements.append(SequenceElement(l))
    light_seqD.elements.append(SequenceElement(l))

# add the sequences themselves to the show
show.lights.append(light_seqA)
show.lights.append(light_seqB)
show.lights.append(light_seqC)
show.lights.append(light_seqD)


# set up the midi communication thread
dispatcher = MidiDispatcher(0)

light_seqA.speed_control = .01
light_seqA.duration = 0
light_seqA.on_off_trigger = "reverse"
# set_message is a way of mapping a message to an attribute
light_seqA.set_message((176, 14),['speed_control'])
light_seqA.max_speed = .5

light_seqB.speed_control = .01
light_seqB.duration = 0
light_seqB.on_off_trigger = "sequence"


light_seqC.speed_control = .3
light_seqC.duration = -1
light_seqC.max_speed = .5
light_seqC.max_duration = .5
light_seqC.set_message((176, 15),['speed_control'])
light_seqC.set_message((176, 16),['duration_control'])



light_seqD.elements.reverse()
light_seqD.speed_control = .3
light_seqD.duration = -1

dispatcher.add_observer((144, 42), light_seqA)
dispatcher.add_observer((176, 14), light_seqA)
dispatcher.add_observer((176, 15), light_seqC)
dispatcher.add_observer((176, 16), light_seqC)
dispatcher.add_observer((144, 44), light_seqB)
dispatcher.add_observer((144, 54), light_seqC)
dispatcher.add_observer((144, 49), light_seqD)



# startup the midi communication
dispatcher.start()

# this defaults to true for certain equipment
show.dmx_keep_alive = False
# run the show
try:
    show.run_live()
except KeyboardInterrupt:
    # cleanup
    dispatcher.stop()
    sys.exit(0)

The lights are controlled via a hardware board that takes signals from a computer and operates multiple dimmers to control the brightness of the lights. These dimmers range in the number of channels, but can be connected together to add more channel capacity. It is not unusual to have hundreds of channels in a residential size show.

The sequencing of these dimmer channels is generally done with software that divides the show into a set of time chunks represented as columns, with each channel being a row – resulting in a grid. To sequence a show, a light intensity level needs to be set for each chunk of time, usually many per second, for many channels. This grid can be huge and time consuming to fill, and doesn’t seem an intuitive canvas to experiment with creativity in response to the music. I thought there should be a better way.

I’m not alone. The problem with the grid and a hope for a better way are well outlined in this piece called “The Death of the Grid“, which I did not discover till after I started down this path.

Other approaches that try to abstract away the grid is in development software called Prancer, and commercial software LightShow Pro which takes an interesting approach of using video images mapped to a light layout called layer transitions.

Being on a Mac and a Python programmer, I wanted to look at coming up with an easy way to create expressive sequences on my gear without too much work.

Now there is an existing protocol for capturing musical expressiveness via computer and that is the MIDI protocol which was designed to capture the act of playing a musical instrument into a digital signal. I felt like this was a good input protocol to adopt, as there is lots of hardware out there to support it, and it was built to capture an expressive range of actions. This would need to be converted into DMX – which is the protocol that controls the lights (there are other lighting protocols, but DMX seems to be the main non-proprietary standard, and the smartest to work with).

Now each protocol has its own features, and they are not a perfect match – but they are good enough. Midi has 16 channels, each with 127 notes, and a whole mess of controller channels (see here) with a message data value range of 0-127. DMX consists of a universe of 512 channels, with values of 0-255.

What I wanted to write was something that abstracts the channels of DMX involved in a set of lights, creating light objects (which may consist of one or more physical light sets), and allow these to be controlled expressively through MIDI.

Now I’m not the first to consider the combination of midi and dmx – the most comprehensive review is this paper (but also see (1,2,3, and I’m sure others). I felt there was still a lot of room of putting a higher level of abstraction between the basics of MIDI notes to DMX channels.

Here are two quick screen captures showing the start of this project in action (if you have quicktime, you can just copy the URLs and open them directly in quicktime player)

The first shows just the basics of a simple light setup:

A single light

The second shows the addition of a couple more lights, and a couple methods of creating simple groups

Three Lights

Here is more information about the A.ttack D.ecay S.ustain R.elease envelope

The remaining features I have implemented or planned are:

• combining lights or light groups into a “sequence” (not to be confused with a show sequence – more of a chase sequence). These sequences have parameters like duration and speed, loop patterns etc.
• Calculating RGB values for Hue, and blending between them over time or space.
• Adding physics tween motion – such as acceleration, bounce, or elastic to sequence groups.

The idea will be to record multiple “live sessions” and then merge them. The complexity may prevent the realtime processing of the midi-track into DMX – and so a MIDI file would be compiled against a show setup to produce either a vixen sequence, or some other similar DMX “player file” (really just binary data of chunks of 512byte DMX data).

This will eventually all be open sourced. I’m not planning on any GUI at the moment, I hope to come up with a clean API that actually makes it pretty easy to write out the scripts.

Here is the snippet:

#!/usr/bin/env python
# encoding: utf-8


import os
import re
from sys import stdout, stdin, exit

display_list = stdin.read()
explicit = 1


def sanitize_enum(s):
    # @@ plenty to improve here
    s = s.upper().strip(' ()')
    s = re.sub('^[-\d ]+','',s)
    s = re.sub('[./() \-]', "_",s)
    while re.search('__',s):
        s = s.replace('__','_')
    return s

def format_choice(t):
    return "    ( %s, '%s' )," % t


choices = []
i = 1

for choice in display_list.split('\n'):
    if choice:
        if explicit:
            sanitized = sanitize_enum(choice)
            print "%s = %s" % (sanitized,i)
            choices.append((sanitized,choice.strip()))
        else:
            choices.append((i,choice.strip()))
        i+=1

print "choices = ("
for c in choices:
    print format_choice(c)
print "    )"

No one part of this solution is all that earth shattering, but it is a simple clever combination of steps.

So you have a bunch of Macs in a lab – you have Network accounts and server based home folders working, and a fast network. Here are the steps to turn that lab into a fast burn factory for optical disks.

• First, create a network home folder user, I call mine burn.

• Set a few settings for that user, such as to ignore blank media when inserted.

• Create your master .img .dmg or .cdr lets call it ‘burnme.img’, on any machine and then copy it to the Desktop of the burn user.

• Create a burn.command file on the Desktop whose contents is:

burn.command (make executable):

#!/bin/bash
hdiutil burn ~/Desktop/burnme.img

Now you can execute two sets of commands through ARD. The first will log a set of machines in as the burn user (you need to enable assistive devices in UniAccess prefs for this to work):

osascript -e 'tell application "System Events" to keystroke "burn"'; \
osascript -e 'tell application "System Events" to keystroke tab'; \
osascript -e 'tell application "System Events" to delay 0.5'; \
osascript -e 'tell application "System Events" to keystroke "burn"'; \
osascript -e 'tell application "System Events" to delay 0.5'; \
osascript -e 'tell application "System Events" to keystroke return'

The next ARD UNIX command is a simple command to execute the burning:

open ~/Desktop/burn.command

Using the combo of the open command with a command file will give you the visual feedback on each machine, as well as return from the ARD task immediately.

The hdiutil command will even wait for media to be inserted if you execute the above before a blank disc is inserted.

I believe hdiutil also uses buffer under-run protection so even if you hit a network bottleneck – you shouldn’t end up with coasters. On a gigabit connection I’ve done up to 13 machines at the same time no problem.

I picked diskutil for starters as its one I use with some regularity on my personal machine (mainly for force unmounting stubborn volumes) as well it has a pretty basic verb set. Other candidates to apply this two would be hdiutil, severadmin, dscl – I’m sure there are a bunch. It would be interesting to work these up into a single package someday if the effort can ever be marshalled.

I borrowed the approach used in Django of having the completion choices be determined by a python script as it is much more pleasant (for me) to put together any logic or algorithms in Python vs Bash. The same approach could be used with Ruby, Perl, etc.

You can find the project here

Click below to watch a micro screencast:

QR Code is a 2 dimensional matrix code invented in Japan that has qualities of easy machine recognition. The idea is that if the paper document is printed with a qrcode, then a computer could later do the work of attaching the scan back to the record. So the qrcode becomes the bridge between the digital and paper steps of the process. The item could even switch between the two multiple times (ie to get multiple signatures in different locations).

I put together a quick django proof of concept that illustrates all the steps.

Briefly:

• a Django view prints a item-document with a generated qrcode in a corner of the page
• an item has a filefield to store a scanned document
• a cron script that loads up the Django environment looks through the scanned documents, decodes the qrcode, and matches them up with the django object

The qrencoding is done with google charts (no hassle of a cached image), and the decode is done with pyqrcode. Note that the pyqrcode project doesn’t have an egg for 10.6, but I put one together here. A little ghostscript love can be used to convert PDF for the decoding step.

Many workplaces now have copiers with built in scanners that can scan a pile of documents with ease, making the effort to attach documents back to the object from where they came pretty easy. There are also iPhone and phone apps that will open the embedded URL, which could be used to pull up a “quick action” list for the item. I haven’t seen anything like that for iSight cameras – which is too bad.

So if you are interested in the proof of concept (it is bare bones, but does cover all the core steps you can grab it here (if you don’t know python/django – you’re going to be a bit lost if you download it – sorry). There is one hardcoded path in the link.py script which is what you run to do the importing (ie called from cron). Now I just have to see if I can find time to actually get some apps that use this written.

Basically he is highlighting some areas for growth when it comes to the process of combining, in an efficient way, multiple reusable Django apps.

He covers the following areas:

• settings
• static media
• upload paths
• Migrations
• cache keys
• translation

Basically – every one of those except the last two is a pain point that I’ve experienced relatively early in my Django life.

The Django community has a strong bent towards caution and letting “convention” pick the best in class. While this works in some cases (ie I think it has happened with South for migrations) it is often slow to be recognized or slow to happen.

Basically – South is the way to go for migrations and it’s time to give the author the support that comes from it being the blessed version. If not then there is the risk that he will burn out, and South will wither, and then we have to begin the process all over again (I know thats not likely – but it is a risk)

For other areas – waiting for a “cohesive” convention is a waste of time, because there are multiple, equally good, ways of doing something, and the problem is not that one is better, just that there exists frustrating variety in the context of predictable combination and reuse.

Take upload paths – it would really take nothing to namespace the upload zone as Kevin is suggesting. This is not a convention that will organically come from the community easily, and should just be something that Django core dictates. There is a reason why Python likes the BDFL tag – because we know that it’s often best in the end to just have certain things dictated.

Same with static media – Django already basically does this with templates, it seems like it makes sense to extend and expand on that approach for static media.

I don’t know much about cache keys and translation – but given how strongly I agree with everything else in the article, the suggestions should probably be strongly considered.

Like the whos down in whoville, just wanted to add one more “we are here, we are here”

This post has a great tool for randomizing terminal colors.

If you have a lot of terminal windows open – this can make expose very handy (I don’t really use tabs).

I have this bound to CMD-Shift-C

We recently got an 8 core Mac Pro, and I wanted to see if I could take more advantage of all those cores when resizing images.

One of the things that confused me when I first opened activity monitor, is that it had 16 processors listed. A little digging turns up that this relates to Intel’s Hyperthreading technology, where they present the OS twice the cores and hand some extra concurrency on the chip. There is some debate out there as to whether or not it makes a difference.

It’s frustrating to have all that power and watch a CPU utilization tool look like this:

But that is what you get when a tool is not written to be parallel across processing units. Much is made over Python’s limit to threading and the GIL, however it seems that multiple threads never take as much advantage of multicore horsepower as do multiple processes.

Thanks to Python’s multiprocessing library it was very easy to create a worker pool to handle the resizing. The results are impressive. The test task was to resize a folder of 350 jpeg files by 50% and save them to a folder.

This shows you the performance gain by doing the image resizing in parallel. It goes from a 6 minute task to a 30 second task.

What is interesting is that even though there are only 8 true cores, there is a 40% increase in speed using all 16 virtual cores, but almost no advantage in going any higher than that. I’d say hyperthreading makes a difference in this case.

Now this is more like it:

A couple assorted notes:

With 8 workers, there is a roughly 10% increase in performance seen when turning off hyperthreading (via processor prefpane from dev-tools). It may be that the overhead of managing threads that aren’t doing much for you detracts from the overall performance?? Not sure.

Photoshop batch automate on the same task takes about 6 minutes, which seems to refute Adobe’s implication that muliprocessing doesn’t often gain you much.

There is one gotcha however, and that is that there does seem to be some sort of memory leak with the multiprocessing module. With just one worker in the pool, you can see a steadily increasing memory use that isn’t present in the same PIL code that isn’t run through the multiprocessing module. This is probably a manifestation of this bug

Finally, the script that I used to do the tests is available here.

So here is a quick script that I keep on my path for viewing hg diff, formated with pygments – I’m sure it would be very easy to adopt this to git.

here is the gist

Dreamhost provides free hosting to nonprofits, and for now they seem to have thrown in a virtual private server. This is hard not to take advantage of, and I’m hoping that the VPS is better performing than the shared hosting – but my concern is that they all share the same mysql servers. So here is how I got pinax up and running in its comfy and modern Python web stack.

At first I tried to go the route of custom install of python2.6 – but got bogged down trying to get either passengerwsgi or modwsgi to work properly – they both seemed to have fatal dead ends in the end.

The first thing you need to do is associate a domain with your PS and set it up as a directory of a user.

so that you have a user and directory like /home/django_user/example.com/

In the Dreamhost panel private server section “Web Server Configuration” uncheck “DreamHost Managed:”. This will prevent any accidental changes you make in the panel from undoing the customizations you need to make to the apache config.

Next you need to create an admin user in the Dreamhost control panel for your PS, lets say djangoadmin, this probably should be a different user than is set up for the domain. This user will just be configuring system level software, namely set the python version to use, install the mysql bindings, PIL, and modwsgi packages.

login as the admin user and then:

mkdir src

# This is to set python 2.5 as the default - if you don't care you can skip this step - but I'm assuming this is done
cd /usr/bin/
sudo rm python
sudo ln -s python2.5 python


cd ~/src
# note that this is specific for 2.5 - use the right egg for your python version
wget http://pypi.python.org/packages/2.5/s/setuptools/setuptools-0.6c11-py2.5.egg#md5=64c94f3bf7a72a13ec83e0b24f2749b2
sudo sh setuptools-0.6c11-py2.5.egg

You next need to download the MySQL-python tarball from http://sourceforge.net/projects/mysql-python/ as there is no easy way to fetch it from the command line. FTP it to your src/ directory of the admin user then:

tar xvzf MySQL-python-1.2.3c1.tar.gz
cd MySQL-python-1.2.3c1
sudo python setup.py install

cd ..
wget http://effbot.org/media/downloads/Imaging-1.1.6.tar.gz
tar xvzf Imaging-1.1.6.tar.gz
cd Imaging-1.1.6
python setup.py build --force
sudo python setup.py install
python selftest.py

You should see: 57 tests passed. Next:

sudo easy_install pip
sudo pip install virtualenv
# virtualenvwrapper is primarily useful if you are going to do 
# dev work or host many projects - I don't assume you fetch this
sudo pip install virtualenvwrapper

cd ~/src
wget http://modwsgi.googlecode.com/files/mod_wsgi-3.0c5.tar.gz
tar xvzf mod_wsgi-3.0c5.tar.gz 

./configure --with-apxs=/usr/local/dh/apache2/template/sbin/apxs --with-python=/usr/bin/python
make
sudo make install

# note that your private server ID replaces the 99999, 
# sub your favorite editor of choice (vi,emacs etc)

sudo pico /usr/local/dh/apache2/apache2-ps99999/etc/httpd.conf

find the section on modules and add:

LoadModule wsgi_module /dh/apache2/template/lib/modules/mod_wsgi.so

Now your base python environment is configured, don’t log out of this user – as we will need to make more changes to the httpd.conf file later. Instead open up another terminal session and login as your django_user

The first thing we need to do is create some folders we will use later

mkdir src envs projects wsgi-scripts

Then we will grab pinax, install it, and activate the virtualenv

cd ~/src
wget http://downloads.pinaxproject.com/Pinax-0.7.1-bundle.tar.gz
tar xvzf Pinax-0.7.1-bundle.tar.gz
python Pinax-0.7.1-bundle/scripts/pinax-boot.py ~/envs/pinax_test/
cd ~
source envs/pinax_test/bin/activate

#optional - update django to current release version
pip install -U django

Now we will create a basic pinax project

pinax-admin clone_project basic_project projects/basictest
cd projects/basictest/

Before you continue, go back to the Dreamhost panel and create a mysql database for your project – note the name and password, and the host name.

Edit the settings.py in your basictest project to use these database settings

Now lets see if we can get pinax up and running:

chmod +x manage.py 
./manage.py syncdb
./manage.py runserver 0.0.0.0:8000

You should now be able to go to http://example.com:8000 and see the pinax welcome screen, ctrl-C to stop the dev server.

To get pinax working with mod_wsgi you need two parts, a wsgi script, and changes to the apache configuration

create a file called pinax_basictest.wsgi in your wsgi-scripts folder. Its contents should look like (cobbled from wsgi docs and pinax deployment sample wsgi file):

import os
import sys
import site
sys.stdout = sys.stderr

# not needed in all environments - but is in others
os.environ['PYTHON_EGG_CACHE'] = '/tmp'

# this forces the virtualenv site-packages to be higher priority than system site-packages
site.addsitedir('/home/django_user/envs/pinax_test/lib/python2.5/site-packages')

sys.path.insert(0,'/home/django_user/projects/')

from django.conf import settings
os.environ['DJANGO_SETTINGS_MODULE'] = 'basictest.settings'

sys.path.insert(0, os.path.join(settings.PINAX_ROOT, "apps"))
sys.path.insert(0, os.path.join(settings.PROJECT_ROOT, "apps"))


import django.core.handlers.wsgi

application = django.core.handlers.wsgi.WSGIHandler()


# uncomment below to do a sanity check on the wsgi setup

# def test_wsgi(environ, start_response):
#     status = '200 OK'
#     output = 'Hello World! wsgi py \n' + sys.version + '\n' + '\n'.join(sys.path)
# 
#     response_headers = [('Content-type', 'text/plain'),
#                         ('Content-Length', str(len(output)))]
#     start_response(status, response_headers)
# 
#     return [output]
# 
# application = test_wsgi

Next you need to switch back to your admin user terminal (you didn’t close it right?) and edit your httpd.conf file again

sudo pico /usr/local/dh/apache2/apache2-ps99999/etc/httpd.conf

go to the very end of the file, and before the last closing /virtualhost tag (so inside the virtualhost) add the following

# wsgi
<Directory /home/django_user/wsgi-scripts>
Order allow,deny
Allow from all
</Directory>


# absolute path
WSGIScriptAlias / /home/django_user/wsgi-scripts/pinax_basictest.wsgi 

<Directory "/home/django_user/projects/basictest/media">
    Allow from all
    Order allow,deny
</Directory>


<Directory "/home/django_user/envs/pinax_test/lib/python2.5/site-packages/django/contrib/admin/media">
    Allow from all
    Order allow,deny
</Directory>

Alias "/media/" "/home/django_user/projects/basictest/media/"

# note some use hyphen - others _
Alias "/admin_media/" "/home/django_user/envs/pinax_test/lib/python2.5/site-packages/django/contrib/admin/media/"

Finally, still while in the admin user terminal – restart apache (remember to change for your private server id):

sudo /etc/init.d/httpd2 restart apache2-ps999999

Now you should be able to go to http://example.com and see the pinax welcome screen

If you get a 500 error:

• look in your apache error log which will be at /home/django_user/logs/example.com/http/error.log
• uncomment the sanity check in the wsgi script to see if you can get basic hello world working, along with some info about what python environment is being used by wsgi

First I check for the user and create it if it doesn’t exist

#!bash
user_exists=`dscl . -read /Users/remote GeneratedUID | grep -c GeneratedUID`
if [ $user_exists -ne 1 ]; then
    echo "creating remote user"
    sudo dscl . -create /Users/remote
    dscl . -create /Users/remote UserShell /bin/bash
    sudo dscl . -create /Users/remote RealName "remote"
    dscl . -create /Users/remote UniqueID 509
    dscl . -create /Users/remote PrimaryGroupID 1000
    dscl . -create /Users/remote NFSHomeDirectory /Local/Users/remote
    dscl . -passwd /Users/remote remote
fi

because these are fully managed machines, I know what UIDs are available. (For a method that checks for available UID I’ve posted a script from Andrew Mortensen below)

The next part of the script will check if the user is logged in as “remote” and on campus using a regular expression (our two subnets are 10.5.5.X and 10.6.6.X). You could also check a router, DHCP server, or internal DNS as other approaches. If they are on campus I use a display utility called BigHonkingText to throw up a message and then kill the loginwindow.

#!bash
user="$1"

if [ "$user" == "remote" ]; then
    IP=`ifconfig | grep "inet " | grep -v 127.0.0.1 | awk 'NR>1{exit};{ print $2 }'`
    echo $IP
    if [[ "$IP" =~ 10.[5,6].[5,6].[0-9]* ]]; then
        echo "on campus"
    /usr/local/bin/BigHonkingText "account not allowed on campus"
    killall loginwindow
        # reboot
    fi
fi

Here is the script from Andrew Mortensen:

#!bash
#! /bin/sh

# create a template user

export PATH=/bin:/usr/bin:/sbin:/usr/sbin

# arbitrary uid
N_UID=501

# arbitrary gid
N_GID=501

# user name
N_USERNAME="$1"

# home
N_HOME="/var/${N_USERNAME}"

# system default user home template
SYSHOMETEMPLATE="/System/Library/User Template/English.lproj"

# make sure the uid and gid are available
while [ true ]; do
    user="`dscl . -search /users UniqueID ${N_UID} 2>/dev/null`"

    if [ -z "${user}" ]; then
    break
    fi

    N_UID=$((${N_UID} + 1));
done

while [ true ]; do
    group=`dscl . -search /groups PrimaryGroupID ${N_GID} 2>/dev/null`

    if [ -z "${group}" ]; then
    break
    fi

    N_GID=$((${N_GID} + 1));
done

# create user
dscl . <<EOF
create "/users/${N_USERNAME}"
create "/users/${N_USERNAME}" AppleMetaNodeLocation /Local/Default
create "/users/${N_USERNAME}" GeneratedUID `uuidgen`
create "/users/${N_USERNAME}" UniqueID ${N_UID}
create "/users/${N_USERNAME}" PrimaryGroupID ${N_GID}
create "/users/${N_USERNAME}" Password "*"
create "/users/${N_USERNAME}" RecordName ${N_USERNAME}
create "/users/${N_USERNAME}" RecordType dsRecTypeNative:users
create "/users/${N_USERNAME}" NFSHomeDirectory ${N_HOME}
create "/users/${N_USERNAME}" RealName "Template User"
create "/users/${N_USERNAME}" UserShell /bin/bash
EOF
if [ $? -ne 0 ]; then
    logger -is Creation of ${N_USERNAME} failed.

    # destroy account
    dscl . -delete "/users/${N_USERNAME}" 2>/dev/null
    exit 2
fi

# create group
dscl . <<EOF
create "/groups/${N_USERNAME}"
create "/groups/${N_USERNAME}" AppleMetaNodeLocation /Local/Default
create "/groups/${N_USERNAME}" GeneratedUID `uuidgen`
create "/groups/${N_USERNAME}" PrimaryGroupID ${N_GID}
create "/groups/${N_USERNAME}" RecordName ${N_USERNAME}
create "/groups/${N_USERNAME}" RecordType dsRecTypeNative:groups
create "/groups/${N_USERNAME}" Password "*"
create "/groups/${N_USERNAME}" GroupMembership ${N_USERNAME}
EOF
if [ $? -ne 0 ]; then
    logger -is Creation of ${N_USERNAME} failed.

    # destroy account
    dscl . -delete "/users/${N_USERNAME}" 2>/dev/null
    dscl . -delete "/groups/${N_USERNAME}" 2>/dev/null
    exit 2
fi

# make home directory
mkdir -m 0700 -p ${N_HOME}
ditto --rsrc "${SYSHOMETEMPLATE}" "${N_HOME}"

if [ $? -ne 0 ]; then
    logger -is Creation of ${N_USERNAME} failed.

    # destroy account
    dscl . -delete "/users/${N_USERNAME}" 2>/dev/null
    dscl . -delete "/groups/${N_USERNAME}" 2>/dev/null
    exit 2
fi

chown -R ${N_USERNAME}:${N_USERNAME} ${N_HOME}

logger -i Creation of template user ${N_USERNAME} succeeded.

exit 0

The crux of the issue is that the version of MySQL that comes with Leopard Server does not include the header and library files needed for linking external libraries with. This applies to 10.5, but a similar process should work with 10.6.

If you attempt to build the MySQLdb python module you’ll get errors like:

building '_mysql' extension
gcc -fno-strict-aliasing -Wno-long-double -no-cpp-precomp -mno-fused-madd -fno-common -dynamic -DNDEBUG -g -Os -Wall -Wstrict-prototypes -DMACOSX -I/usr/include/ffi -DENABLE_DTRACE -arch i386 -arch ppc -pipe -Dversion_info=(1,2,3,'gamma',1) -D__version__=1.2.3c1 -I/usr/include -I/System/Library/Frameworks/Python.framework/Versions/2.5/include/python2.5 -c _mysql.c -o build/temp.macosx-10.5-i386-2.5/_mysql.o -fno-omit-frame-pointer -pipe -D_P1003_1B_VISIBLE -DSIGNAL_WITH_VIO_CLOSE -DSIGNALS_DONT_BREAK_READ -DIGNORE_SIGHUP_SIGQUIT
_mysql.c:36:23: error: my_config.h: No such file or directory
_mysql.c:36:23: error: my_config.h: No such file or directory
_mysql.c:38:19: error: mysql.h: No such file or directory
_mysql.c:38:19: error: mysql.h: No such file or directory

Apple has a http://support.apple.com/kb/TA25017 on the issue – and the solution is to download these separately. You can also just install a different version of mysql – but for a variety of reasons, I wanted to continue to run the installed version.

So here are the commands that will build you a working mysql python binding:

(assumes you have dev tools installed)

grab the binding library source:

http://sourceforge.net/projects/mysql-python/files/

mv path/to/MySQL-python-1.2.3c1.tar.gz /tmp
cd /tmp
tar xvzf MySQL-python-1.2.3c1.tar.gz
curl -O http://www.opensource.apple.com/other/MySQL-43.binaries.tar.gz
tar xzvf MySQL-43.binaries.tar.gz
tar xvzf MySQL-43.binaries/MySQL-43.root.tar.gz
cd usr
sudo ditto usr/include/ /usr/include/
sudo ditto usr/lib/ /usr/lib/
cd /tmp/MySQL-python-1.2.3c1
ARCHFLAGS='-arch i386 -arch x86_64'
export ARCHFLAGS
# edit site.cfg to point to point to /usr/bin/mysql_config
python setup.py build
sudo python setup.py install

To verify this worked – fire up a python console and ‘import MySQLdb’

I’ve been mostly using transmit/SFTP to do editing of remote files with my local editor of choice (TextMate). Before I switched to TextMate I was using TextWrangler. The problem with using Transmit (or any FTP client) for editing is that if you are already where you need to be in a SSH session – you have to navigate to that same location in the FTP client before opening the file for local editing.

One feature TextWrangler has that TextMate does not is the ability to open files from a SFTP URL and save them back remotely when you do CMD-S.

So putting the following in the .bash_profile on the server will let you execute “redit somefile” which will cause a reverse ssh command to be sent you telling TextWrangler to open the file.

Some prereqs:

• You have to have key based SSH auth working in BOTH directions
• You have to have the TextWrangler CLI tool (edit) installed
• Since there is a good chance that the local user you are logged in as is different than the one you are shelling into as, you have to hard code that – so unless your name is preston – you will have to change that.

`

SSH_CLIENT_ADDRESS=`echo $SSH_CONNECTION | cut -f1 -d" "`
export SSH_CLIENT_ADDRESS

SSH_HOST_ADDRESS=`echo $SSH_CONNECTION | cut -f3 -d" "`
export SSH_HOST_ADDRESS

ME=`whoami`

function redit() {
    ssh preston@$SSH_CLIENT_ADDRESS edit \"sftp://$ME@$SSH_HOST_ADDRESS/$PWD/"$@"\"
}

`