Discuss Here

Here is a video demonstrating some of the features, remember the lights on screen are just simple proxy lights for demo purposes. The main output is realtime DMX – which is an already working feature, I just don’t have it hooked up to light gear yet.

Here is the code listing that created the demo sequence elements:

#!/usr/bin/env python
# encoding: utf-8

from midi_reader import MidiDispatcher
import time
from lights import *
import sys


# create a light show - manages the updating of all lights
show = LightShow()

light_seqA = LightSequence()
light_seqB = LightSequence()
light_seqC = LightSequence()
light_seqD = LightSequence()


# create our 8 lights and add them to each sequence and the show
for i in range(1,9):
    l = LightUnit(channels=[i])
    l.release = .3
    show.lights.append(l)
    light_seqA.elements.append(SequenceElement(l))
    light_seqB.elements.append(SequenceElement(l))
    light_seqC.elements.append(SequenceElement(l))
    light_seqD.elements.append(SequenceElement(l))

# add the sequences themselves to the show
show.lights.append(light_seqA)
show.lights.append(light_seqB)
show.lights.append(light_seqC)
show.lights.append(light_seqD)


# set up the midi communication thread
dispatcher = MidiDispatcher(0)

light_seqA.speed_control = .01
light_seqA.duration = 0
light_seqA.on_off_trigger = "reverse"
# set_message is a way of mapping a message to an attribute
light_seqA.set_message((176, 14),['speed_control'])
light_seqA.max_speed = .5

light_seqB.speed_control = .01
light_seqB.duration = 0
light_seqB.on_off_trigger = "sequence"


light_seqC.speed_control = .3
light_seqC.duration = -1
light_seqC.max_speed = .5
light_seqC.max_duration = .5
light_seqC.set_message((176, 15),['speed_control'])
light_seqC.set_message((176, 16),['duration_control'])



light_seqD.elements.reverse()
light_seqD.speed_control = .3
light_seqD.duration = -1

dispatcher.add_observer((144, 42), light_seqA)
dispatcher.add_observer((176, 14), light_seqA)
dispatcher.add_observer((176, 15), light_seqC)
dispatcher.add_observer((176, 16), light_seqC)
dispatcher.add_observer((144, 44), light_seqB)
dispatcher.add_observer((144, 54), light_seqC)
dispatcher.add_observer((144, 49), light_seqD)



# startup the midi communication
dispatcher.start()

# this defaults to true for certain equipment
show.dmx_keep_alive = False
# run the show
try:
    show.run_live()
except KeyboardInterrupt:
    # cleanup
    dispatcher.stop()
    sys.exit(0)

The lights are controlled via a hardware board that takes signals from a computer and operates multiple dimmers to control the brightness of the lights. These dimmers range in the number of channels, but can be connected together to add more channel capacity. It is not unusual to have hundreds of channels in a residential size show.

The sequencing of these dimmer channels is generally done with software that divides the show into a set of time chunks represented as columns, with each channel being a row – resulting in a grid. To sequence a show, a light intensity level needs to be set for each chunk of time, usually many per second, for many channels. This grid can be huge and time consuming to fill, and doesn’t seem an intuitive canvas to experiment with creativity in response to the music. I thought there should be a better way.

I’m not alone. The problem with the grid and a hope for a better way are well outlined in this piece called “The Death of the Grid“, which I did not discover till after I started down this path.

Other approaches that try to abstract away the grid is in development software called Prancer, and commercial software LightShow Pro which takes an interesting approach of using video images mapped to a light layout called layer transitions.

Being on a Mac and a Python programmer, I wanted to look at coming up with an easy way to create expressive sequences on my gear without too much work.

Now there is an existing protocol for capturing musical expressiveness via computer and that is the MIDI protocol which was designed to capture the act of playing a musical instrument into a digital signal. I felt like this was a good input protocol to adopt, as there is lots of hardware out there to support it, and it was built to capture an expressive range of actions. This would need to be converted into DMX – which is the protocol that controls the lights (there are other lighting protocols, but DMX seems to be the main non-proprietary standard, and the smartest to work with).

Now each protocol has its own features, and they are not a perfect match – but they are good enough. Midi has 16 channels, each with 127 notes, and a whole mess of controller channels (see here) with a message data value range of 0-127. DMX consists of a universe of 512 channels, with values of 0-255.

What I wanted to write was something that abstracts the channels of DMX involved in a set of lights, creating light objects (which may consist of one or more physical light sets), and allow these to be controlled expressively through MIDI.

Now I’m not the first to consider the combination of midi and dmx – the most comprehensive review is this paper (but also see (1,2,3, and I’m sure others). I felt there was still a lot of room of putting a higher level of abstraction between the basics of MIDI notes to DMX channels.

Here are two quick screen captures showing the start of this project in action (if you have quicktime, you can just copy the URLs and open them directly in quicktime player)

The first shows just the basics of a simple light setup:

A single light

The second shows the addition of a couple more lights, and a couple methods of creating simple groups

Three Lights

Here is more information about the A.ttack D.ecay S.ustain R.elease envelope

The remaining features I have implemented or planned are:

• combining lights or light groups into a “sequence” (not to be confused with a show sequence – more of a chase sequence). These sequences have parameters like duration and speed, loop patterns etc.
• Calculating RGB values for Hue, and blending between them over time or space.
• Adding physics tween motion – such as acceleration, bounce, or elastic to sequence groups.

The idea will be to record multiple “live sessions” and then merge them. The complexity may prevent the realtime processing of the midi-track into DMX – and so a MIDI file would be compiled against a show setup to produce either a vixen sequence, or some other similar DMX “player file” (really just binary data of chunks of 512byte DMX data).

This will eventually all be open sourced. I’m not planning on any GUI at the moment, I hope to come up with a clean API that actually makes it pretty easy to write out the scripts.

Here is the snippet:

#!/usr/bin/env python
# encoding: utf-8


import os
import re
from sys import stdout, stdin, exit

display_list = stdin.read()
explicit = 1


def sanitize_enum(s):
    # @@ plenty to improve here
    s = s.upper().strip(' ()')
    s = re.sub('^[-\d ]+','',s)
    s = re.sub('[./() \-]', "_",s)
    while re.search('__',s):
        s = s.replace('__','_')
    return s

def format_choice(t):
    return "    ( %s, '%s' )," % t


choices = []
i = 1

for choice in display_list.split('\n'):
    if choice:
        if explicit:
            sanitized = sanitize_enum(choice)
            print "%s = %s" % (sanitized,i)
            choices.append((sanitized,choice.strip()))
        else:
            choices.append((i,choice.strip()))
        i+=1

print "choices = ("
for c in choices:
    print format_choice(c)
print "    )"

No one part of this solution is all that earth shattering, but it is a simple clever combination of steps.

So you have a bunch of Macs in a lab – you have Network accounts and server based home folders working, and a fast network. Here are the steps to turn that lab into a fast burn factory for optical disks.

• First, create a network home folder user, I call mine burn.

• Set a few settings for that user, such as to ignore blank media when inserted.

• Create your master .img .dmg or .cdr lets call it ‘burnme.img’, on any machine and then copy it to the Desktop of the burn user.

• Create a burn.command file on the Desktop whose contents is:

burn.command (make executable):

#!/bin/bash
hdiutil burn ~/Desktop/burnme.img

Now you can execute two sets of commands through ARD. The first will log a set of machines in as the burn user (you need to enable assistive devices in UniAccess prefs for this to work):

osascript -e 'tell application "System Events" to keystroke "burn"'; \
osascript -e 'tell application "System Events" to keystroke tab'; \
osascript -e 'tell application "System Events" to delay 0.5'; \
osascript -e 'tell application "System Events" to keystroke "burn"'; \
osascript -e 'tell application "System Events" to delay 0.5'; \
osascript -e 'tell application "System Events" to keystroke return'

The next ARD UNIX command is a simple command to execute the burning:

open ~/Desktop/burn.command

Using the combo of the open command with a command file will give you the visual feedback on each machine, as well as return from the ARD task immediately.

The hdiutil command will even wait for media to be inserted if you execute the above before a blank disc is inserted.

I believe hdiutil also uses buffer under-run protection so even if you hit a network bottleneck – you shouldn’t end up with coasters. On a gigabit connection I’ve done up to 13 machines at the same time no problem.

I picked diskutil for starters as its one I use with some regularity on my personal machine (mainly for force unmounting stubborn volumes) as well it has a pretty basic verb set. Other candidates to apply this two would be hdiutil, severadmin, dscl – I’m sure there are a bunch. It would be interesting to work these up into a single package someday if the effort can ever be marshalled.

I borrowed the approach used in Django of having the completion choices be determined by a python script as it is much more pleasant (for me) to put together any logic or algorithms in Python vs Bash. The same approach could be used with Ruby, Perl, etc.

You can find the project here

Click below to watch a micro screencast:

QR Code is a 2 dimensional matrix code invented in Japan that has qualities of easy machine recognition. The idea is that if the paper document is printed with a qrcode, then a computer could later do the work of attaching the scan back to the record. So the qrcode becomes the bridge between the digital and paper steps of the process. The item could even switch between the two multiple times (ie to get multiple signatures in different locations).

I put together a quick django proof of concept that illustrates all the steps.

Briefly:

• a Django view prints a item-document with a generated qrcode in a corner of the page
• an item has a filefield to store a scanned document
• a cron script that loads up the Django environment looks through the scanned documents, decodes the qrcode, and matches them up with the django object

The qrencoding is done with google charts (no hassle of a cached image), and the decode is done with pyqrcode. Note that the pyqrcode project doesn’t have an egg for 10.6, but I put one together here. A little ghostscript love can be used to convert PDF for the decoding step.

Many workplaces now have copiers with built in scanners that can scan a pile of documents with ease, making the effort to attach documents back to the object from where they came pretty easy. There are also iPhone and phone apps that will open the embedded URL, which could be used to pull up a “quick action” list for the item. I haven’t seen anything like that for iSight cameras – which is too bad.

So if you are interested in the proof of concept (it is bare bones, but does cover all the core steps you can grab it here (if you don’t know python/django – you’re going to be a bit lost if you download it – sorry). There is one hardcoded path in the link.py script which is what you run to do the importing (ie called from cron). Now I just have to see if I can find time to actually get some apps that use this written.

Basically he is highlighting some areas for growth when it comes to the process of combining, in an efficient way, multiple reusable Django apps.

He covers the following areas:

• settings
• static media
• upload paths
• Migrations
• cache keys
• translation

Basically – every one of those except the last two is a pain point that I’ve experienced relatively early in my Django life.

The Django community has a strong bent towards caution and letting “convention” pick the best in class. While this works in some cases (ie I think it has happened with South for migrations) it is often slow to be recognized or slow to happen.

Basically – South is the way to go for migrations and it’s time to give the author the support that comes from it being the blessed version. If not then there is the risk that he will burn out, and South will wither, and then we have to begin the process all over again (I know thats not likely – but it is a risk)

For other areas – waiting for a “cohesive” convention is a waste of time, because there are multiple, equally good, ways of doing something, and the problem is not that one is better, just that there exists frustrating variety in the context of predictable combination and reuse.

Take upload paths – it would really take nothing to namespace the upload zone as Kevin is suggesting. This is not a convention that will organically come from the community easily, and should just be something that Django core dictates. There is a reason why Python likes the BDFL tag – because we know that it’s often best in the end to just have certain things dictated.

Same with static media – Django already basically does this with templates, it seems like it makes sense to extend and expand on that approach for static media.

I don’t know much about cache keys and translation – but given how strongly I agree with everything else in the article, the suggestions should probably be strongly considered.

Like the whos down in whoville, just wanted to add one more “we are here, we are here”

This post has a great tool for randomizing terminal colors.

If you have a lot of terminal windows open – this can make expose very handy (I don’t really use tabs).

I have this bound to CMD-Shift-C

We recently got an 8 core Mac Pro, and I wanted to see if I could take more advantage of all those cores when resizing images.

One of the things that confused me when I first opened activity monitor, is that it had 16 processors listed. A little digging turns up that this relates to Intel’s Hyperthreading technology, where they present the OS twice the cores and hand some extra concurrency on the chip. There is some debate out there as to whether or not it makes a difference.

It’s frustrating to have all that power and watch a CPU utilization tool look like this:

But that is what you get when a tool is not written to be parallel across processing units. Much is made over Python’s limit to threading and the GIL, however it seems that multiple threads never take as much advantage of multicore horsepower as do multiple processes.

Thanks to Python’s multiprocessing library it was very easy to create a worker pool to handle the resizing. The results are impressive. The test task was to resize a folder of 350 jpeg files by 50% and save them to a folder.

This shows you the performance gain by doing the image resizing in parallel. It goes from a 6 minute task to a 30 second task.

What is interesting is that even though there are only 8 true cores, there is a 40% increase in speed using all 16 virtual cores, but almost no advantage in going any higher than that. I’d say hyperthreading makes a difference in this case.

Now this is more like it:

A couple assorted notes:

With 8 workers, there is a roughly 10% increase in performance seen when turning off hyperthreading (via processor prefpane from dev-tools). It may be that the overhead of managing threads that aren’t doing much for you detracts from the overall performance?? Not sure.

Photoshop batch automate on the same task takes about 6 minutes, which seems to refute Adobe’s implication that muliprocessing doesn’t often gain you much.

There is one gotcha however, and that is that there does seem to be some sort of memory leak with the multiprocessing module. With just one worker in the pool, you can see a steadily increasing memory use that isn’t present in the same PIL code that isn’t run through the multiprocessing module. This is probably a manifestation of this bug

Finally, the script that I used to do the tests is available here.

So here is a quick script that I keep on my path for viewing hg diff, formated with pygments – I’m sure it would be very easy to adopt this to git.

here is the gist

The crux of the issue is that the version of MySQL that comes with Leopard Server does not include the header and library files needed for linking external libraries with. This applies to 10.5, but a similar process should work with 10.6.

If you attempt to build the MySQLdb python module you’ll get errors like:

building '_mysql' extension
gcc -fno-strict-aliasing -Wno-long-double -no-cpp-precomp -mno-fused-madd -fno-common -dynamic -DNDEBUG -g -Os -Wall -Wstrict-prototypes -DMACOSX -I/usr/include/ffi -DENABLE_DTRACE -arch i386 -arch ppc -pipe -Dversion_info=(1,2,3,'gamma',1) -D__version__=1.2.3c1 -I/usr/include -I/System/Library/Frameworks/Python.framework/Versions/2.5/include/python2.5 -c _mysql.c -o build/temp.macosx-10.5-i386-2.5/_mysql.o -fno-omit-frame-pointer -pipe -D_P1003_1B_VISIBLE -DSIGNAL_WITH_VIO_CLOSE -DSIGNALS_DONT_BREAK_READ -DIGNORE_SIGHUP_SIGQUIT
_mysql.c:36:23: error: my_config.h: No such file or directory
_mysql.c:36:23: error: my_config.h: No such file or directory
_mysql.c:38:19: error: mysql.h: No such file or directory
_mysql.c:38:19: error: mysql.h: No such file or directory

Apple has a http://support.apple.com/kb/TA25017 on the issue – and the solution is to download these separately. You can also just install a different version of mysql – but for a variety of reasons, I wanted to continue to run the installed version.

So here are the commands that will build you a working mysql python binding:

(assumes you have dev tools installed)

grab the binding library source:

http://sourceforge.net/projects/mysql-python/files/

mv path/to/MySQL-python-1.2.3c1.tar.gz /tmp
cd /tmp
tar xvzf MySQL-python-1.2.3c1.tar.gz
curl -O http://www.opensource.apple.com/other/MySQL-43.binaries.tar.gz
tar xzvf MySQL-43.binaries.tar.gz
tar xvzf MySQL-43.binaries/MySQL-43.root.tar.gz
cd usr
sudo ditto usr/include/ /usr/include/
sudo ditto usr/lib/ /usr/lib/
cd /tmp/MySQL-python-1.2.3c1
ARCHFLAGS='-arch i386 -arch x86_64'
export ARCHFLAGS
# edit site.cfg to point to point to /usr/bin/mysql_config
python setup.py build
sudo python setup.py install

To verify this worked – fire up a python console and ‘import MySQLdb’

I’ve been mostly using transmit/SFTP to do editing of remote files with my local editor of choice (TextMate). Before I switched to TextMate I was using TextWrangler. The problem with using Transmit (or any FTP client) for editing is that if you are already where you need to be in a SSH session – you have to navigate to that same location in the FTP client before opening the file for local editing.

One feature TextWrangler has that TextMate does not is the ability to open files from a SFTP URL and save them back remotely when you do CMD-S.

So putting the following in the .bash_profile on the server will let you execute “redit somefile” which will cause a reverse ssh command to be sent you telling TextWrangler to open the file.

Some prereqs:

• You have to have key based SSH auth working in BOTH directions
• You have to have the TextWrangler CLI tool (edit) installed
• Since there is a good chance that the local user you are logged in as is different than the one you are shelling into as, you have to hard code that – so unless your name is preston – you will have to change that.

`

SSH_CLIENT_ADDRESS=`echo $SSH_CONNECTION | cut -f1 -d" "`
export SSH_CLIENT_ADDRESS

SSH_HOST_ADDRESS=`echo $SSH_CONNECTION | cut -f3 -d" "`
export SSH_HOST_ADDRESS

ME=`whoami`

function redit() {
    ssh preston@$SSH_CLIENT_ADDRESS edit \"sftp://$ME@$SSH_HOST_ADDRESS/$PWD/"$@"\"
}

`

I use Butler as my quicklaunch keyboard macro tool – but you could use your favorite one to run the following script while reading mail.

#!applescript
tell application "Mail"
    activate
    get message viewer 1's selected mailboxes
    repeat with thisBox in result
        set read status of (messages of thisBox whose read status is false) to true
    end repeat

end tell

I find this almost critical for getting through mailing list traffic.

There are a number of defaults that can be set to make these not appear. The preferred way is to manage these with MCX, but for now I’ve deployed these as a login script (its lazy-easy because its all in just one text file)

This will set defaults so that iWork and iLife don’t bug the user. There are also defaults to turn off auto update check in perian, disable the welcome movie for Safari 4 and to set the standard file format for iWork 09 to use the iWork 08 package style file format which is more efficient when saving over the network. Finally there is a default to tell iTunes not to look for shared music.

Most of these should be pretty straight forward to read – if you have others, please post them in the comments.

TODO: make these into MCX manifests?

# !/bin/bash
echo Setting iPrefs...

sudo -u $1 defaults write org.perian.Perian NextRunDate -date '4000-12-31 16:00:00 -0800'

# iWork

sudo -u $1 defaults -currentHost write com.apple.iWork SFLDefaultsAutoUpdateCheck -bool FALSE
defaults write /Library/Preferences/com.apple.iWork09 DisplayedRegistrationWindowCount -int 3
defaults write /Library/Preferences/com.apple.iWork09 ShouldNotSendRegistration -bool TRUE
chown admin:admin /Library/Preferences/com.apple.iWork09.plist
chmod 644 /Library/Preferences/com.apple.iWork09.plist

sudo -u $1 defaults write com.apple.iWork.Keynote DisplayedRegistrationWindowCount -int 2
sudo -u $1 defaults write com.apple.iWork.Keynote RegistrationHasBeenSent -bool TRUE
sudo -u $1 defaults -currentHost write com.apple.iWork.Keynote FirstRunFlag -bool TRUE
sudo -u $1 defaults write com.apple.iWork.Keynote dontShowWhatsNew -bool TRUE
sudo -u $1 defaults write com.apple.iWork.Keynote DisableSingleFileFormat -bool TRUE

sudo -u $1 defaults write com.apple.iWork.Pages DisplayedRegistrationWindowCount -int 2
sudo -u $1 defaults write com.apple.iWork.Pages RegistrationHasBeenSent -bool TRUE
sudo -u $1 defaults -currentHost write com.apple.iWork.Pages FirstRunFlag -bool TRUE
sudo -u $1 defaults write com.apple.iWork.Pages dontShowWhatsNew -bool TRUE
sudo -u $1 defaults write com.apple.iWork.Pages DisableSingleFileFormat -bool TRUE

sudo -u $1 defaults write com.apple.iWork.Numbers DisplayedRegistrationWindowCount -int 2
sudo -u $1 defaults write com.apple.iWork.Numbers RegistrationHasBeenSent -bool TRUE
sudo -u $1 defaults -currentHost write com.apple.iWork.Numbers FirstRunFlag -bool TRUE
sudo -u $1 defaults write com.apple.iWork.Numbers dontShowWhatsNew -bool TRUE
sudo -u $1 defaults write com.apple.iWork.Numbers DisableSingleFileFormat -bool TRUE

# Garageband
sudo -u $1 defaults write com.apple.garageband dontShowWhatsNew -bool TRUE
sudo -u $1 defaults write com.apple.garageband DfPrefs_NoDocumentPanel -bool TRUE
sudo -u $1 defaults write com.apple.garageband DfPrefs_NextUpgradeCheck -date 'Aug 10, 2012 12:00:00 PM'
sudo -u $1 defaults write com.apple.garageband DfPrefs_AskAboutUpgrades -dict '5.1' 'NO'

# iMovie
sudo -u $1 defaults write com.apple.iMovie8 dontShowWhatsNew -bool TRUE
sudo -u $1 defaults write com.apple.iMovie8 AutoUpgradeCheck -bool FALSE
sudo -u $1 defaults write com.apple.iMovie8 NextUpgradeCheck -date 'Aug 10, 2012 12:00:00 PM'
# no proof that this one is recognized - but it is in the others
sudo -u $1 defaults write com.apple.iMovie8 SFLDefaultsAutoUpdateCheck -bool FALSE

# iPhoto
sudo -u $1 defaults write com.apple.iPhoto CheckForUpdates -bool FALSE
sudo -u $1 defaults write com.apple.iPhoto dontShowWhatsNew -bool TRUE
sudo -u $1 defaults write com.apple.iPhoto GeocodeLookupPreference -int 0
sudo -u $1 defaults write com.apple.iPhoto NextUpgradeCheck -date 'Aug 10, 2012 12:00:00 PM'
# not so sure about these:
sudo -u $1 defaults write com.apple.iPhoto SFLDefaultsAutoUpdateCheck -bool FALSE
sudo -u $1 defaults write com.apple.iPhoto UpdateSplash -int 8

# iTunes
sudo -u $1 defaults write com.apple.iTunes lookForSharedMusic -bool FALSE

# iWeb
sudo -u $1 defaults write com.apple.iWeb dontShowWhatsNew -bool TRUE
sudo -u $1 defaults write com.apple.iWeb dismissDotMacUpsellWindow -bool TRUE
sudo -u $1 defaults write com.apple.iWeb SFLDefaultsAutoUpdateCheck -bool FALSE
sudo -u $1 defaults -currentHost write com.apple.iWeb FirstRunFlag -bool TRUE 

# Safari
sudo -u $1 defaults write com.apple.Safari LastDisplayedWelcomePageVersionString '4.0'