Dreamhost provides free hosting to nonprofits, and for now they seem to have thrown in a virtual private server. This is hard not to take advantage of, and I’m hoping that the VPS is better performing than the shared hosting – but my concern is that they all share the same mysql servers. So here is how I got pinax up and running in its comfy and modern Python web stack.

At first I tried to go the route of custom install of python2.6 – but got bogged down trying to get either passengerwsgi or modwsgi to work properly – they both seemed to have fatal dead ends in the end.

The first thing you need to do is associate a domain with your PS and set it up as a directory of a user.

so that you have a user and directory like /home/django_user/example.com/

In the Dreamhost panel private server section “Web Server Configuration” uncheck “DreamHost Managed:”. This will prevent any accidental changes you make in the panel from undoing the customizations you need to make to the apache config.

Next you need to create an admin user in the Dreamhost control panel for your PS, lets say djangoadmin, this probably should be a different user than is set up for the domain. This user will just be configuring system level software, namely set the python version to use, install the mysql bindings, PIL, and modwsgi packages.

login as the admin user and then:

mkdir src

# This is to set python 2.5 as the default - if you don't care you can skip this step - but I'm assuming this is done
cd /usr/bin/
sudo rm python
sudo ln -s python2.5 python


cd ~/src
# note that this is specific for 2.5 - use the right egg for your python version
wget http://pypi.python.org/packages/2.5/s/setuptools/setuptools-0.6c11-py2.5.egg#md5=64c94f3bf7a72a13ec83e0b24f2749b2
sudo sh setuptools-0.6c11-py2.5.egg

You next need to download the MySQL-python tarball from http://sourceforge.net/projects/mysql-python/ as there is no easy way to fetch it from the command line. FTP it to your src/ directory of the admin user then:

tar xvzf MySQL-python-1.2.3c1.tar.gz
cd MySQL-python-1.2.3c1
sudo python setup.py install

cd ..
wget http://effbot.org/media/downloads/Imaging-1.1.6.tar.gz
tar xvzf Imaging-1.1.6.tar.gz
cd Imaging-1.1.6
python setup.py build --force
sudo python setup.py install
python selftest.py

You should see: 57 tests passed. Next:

sudo easy_install pip
sudo pip install virtualenv
# virtualenvwrapper is primarily useful if you are going to do 
# dev work or host many projects - I don't assume you fetch this
sudo pip install virtualenvwrapper

cd ~/src
wget http://modwsgi.googlecode.com/files/mod_wsgi-3.0c5.tar.gz
tar xvzf mod_wsgi-3.0c5.tar.gz 

./configure --with-apxs=/usr/local/dh/apache2/template/sbin/apxs --with-python=/usr/bin/python
make
sudo make install

# note that your private server ID replaces the 99999, 
# sub your favorite editor of choice (vi,emacs etc)

sudo pico /usr/local/dh/apache2/apache2-ps99999/etc/httpd.conf

find the section on modules and add:

LoadModule wsgi_module /dh/apache2/template/lib/modules/mod_wsgi.so

Now your base python environment is configured, don’t log out of this user – as we will need to make more changes to the httpd.conf file later. Instead open up another terminal session and login as your django_user

The first thing we need to do is create some folders we will use later

mkdir src envs projects wsgi-scripts

Then we will grab pinax, install it, and activate the virtualenv

cd ~/src
wget http://downloads.pinaxproject.com/Pinax-0.7.1-bundle.tar.gz
tar xvzf Pinax-0.7.1-bundle.tar.gz
python Pinax-0.7.1-bundle/scripts/pinax-boot.py ~/envs/pinax_test/
cd ~
source envs/pinax_test/bin/activate

#optional - update django to current release version
pip install -U django

Now we will create a basic pinax project

pinax-admin clone_project basic_project projects/basictest
cd projects/basictest/

Before you continue, go back to the Dreamhost panel and create a mysql database for your project – note the name and password, and the host name.

Edit the settings.py in your basictest project to use these database settings

Now lets see if we can get pinax up and running:

chmod +x manage.py 
./manage.py syncdb
./manage.py runserver 0.0.0.0:8000

You should now be able to go to http://example.com:8000 and see the pinax welcome screen, ctrl-C to stop the dev server.

To get pinax working with mod_wsgi you need two parts, a wsgi script, and changes to the apache configuration

create a file called pinax_basictest.wsgi in your wsgi-scripts folder. Its contents should look like (cobbled from wsgi docs and pinax deployment sample wsgi file):

import os
import sys
import site
sys.stdout = sys.stderr

# not needed in all environments - but is in others
os.environ['PYTHON_EGG_CACHE'] = '/tmp'

# this forces the virtualenv site-packages to be higher priority than system site-packages
site.addsitedir('/home/django_user/envs/pinax_test/lib/python2.5/site-packages')

sys.path.insert(0,'/home/django_user/projects/')

from django.conf import settings
os.environ['DJANGO_SETTINGS_MODULE'] = 'basictest.settings'

sys.path.insert(0, os.path.join(settings.PINAX_ROOT, "apps"))
sys.path.insert(0, os.path.join(settings.PROJECT_ROOT, "apps"))


import django.core.handlers.wsgi

application = django.core.handlers.wsgi.WSGIHandler()


# uncomment below to do a sanity check on the wsgi setup

# def test_wsgi(environ, start_response):
#     status = '200 OK'
#     output = 'Hello World! wsgi py \n' + sys.version + '\n' + '\n'.join(sys.path)
# 
#     response_headers = [('Content-type', 'text/plain'),
#                         ('Content-Length', str(len(output)))]
#     start_response(status, response_headers)
# 
#     return [output]
# 
# application = test_wsgi

Next you need to switch back to your admin user terminal (you didn’t close it right?) and edit your httpd.conf file again

sudo pico /usr/local/dh/apache2/apache2-ps99999/etc/httpd.conf

go to the very end of the file, and before the last closing /virtualhost tag (so inside the virtualhost) add the following

# wsgi
<Directory /home/django_user/wsgi-scripts>
Order allow,deny
Allow from all
</Directory>


# absolute path
WSGIScriptAlias / /home/django_user/wsgi-scripts/pinax_basictest.wsgi 

<Directory "/home/django_user/projects/basictest/media">
    Allow from all
    Order allow,deny
</Directory>


<Directory "/home/django_user/envs/pinax_test/lib/python2.5/site-packages/django/contrib/admin/media">
    Allow from all
    Order allow,deny
</Directory>

Alias "/media/" "/home/django_user/projects/basictest/media/"

# note some use hyphen - others _
Alias "/admin_media/" "/home/django_user/envs/pinax_test/lib/python2.5/site-packages/django/contrib/admin/media/"

Finally, still while in the admin user terminal – restart apache (remember to change for your private server id):

sudo /etc/init.d/httpd2 restart apache2-ps999999

Now you should be able to go to http://example.com and see the pinax welcome screen

If you get a 500 error:

• look in your apache error log which will be at /home/django_user/logs/example.com/http/error.log
• uncomment the sanity check in the wsgi script to see if you can get basic hello world working, along with some info about what python environment is being used by wsgi

First I check for the user and create it if it doesn’t exist

#!bash
user_exists=`dscl . -read /Users/remote GeneratedUID | grep -c GeneratedUID`
if [ $user_exists -ne 1 ]; then
    echo "creating remote user"
    sudo dscl . -create /Users/remote
    dscl . -create /Users/remote UserShell /bin/bash
    sudo dscl . -create /Users/remote RealName "remote"
    dscl . -create /Users/remote UniqueID 509
    dscl . -create /Users/remote PrimaryGroupID 1000
    dscl . -create /Users/remote NFSHomeDirectory /Local/Users/remote
    dscl . -passwd /Users/remote remote
fi

because these are fully managed machines, I know what UIDs are available. (For a method that checks for available UID I’ve posted a script from Andrew Mortensen below)

The next part of the script will check if the user is logged in as “remote” and on campus using a regular expression (our two subnets are 10.5.5.X and 10.6.6.X). You could also check a router, DHCP server, or internal DNS as other approaches. If they are on campus I use a display utility called BigHonkingText to throw up a message and then kill the loginwindow.

#!bash
user="$1"

if [ "$user" == "remote" ]; then
    IP=`ifconfig | grep "inet " | grep -v 127.0.0.1 | awk 'NR>1{exit};{ print $2 }'`
    echo $IP
    if [[ "$IP" =~ 10.[5,6].[5,6].[0-9]* ]]; then
        echo "on campus"
    /usr/local/bin/BigHonkingText "account not allowed on campus"
    killall loginwindow
        # reboot
    fi
fi

Here is the script from Andrew Mortensen:

#!bash
#! /bin/sh

# create a template user

export PATH=/bin:/usr/bin:/sbin:/usr/sbin

# arbitrary uid
N_UID=501

# arbitrary gid
N_GID=501

# user name
N_USERNAME="$1"

# home
N_HOME="/var/${N_USERNAME}"

# system default user home template
SYSHOMETEMPLATE="/System/Library/User Template/English.lproj"

# make sure the uid and gid are available
while [ true ]; do
    user="`dscl . -search /users UniqueID ${N_UID} 2>/dev/null`"

    if [ -z "${user}" ]; then
    break
    fi

    N_UID=$((${N_UID} + 1));
done

while [ true ]; do
    group=`dscl . -search /groups PrimaryGroupID ${N_GID} 2>/dev/null`

    if [ -z "${group}" ]; then
    break
    fi

    N_GID=$((${N_GID} + 1));
done

# create user
dscl . <<EOF
create "/users/${N_USERNAME}"
create "/users/${N_USERNAME}" AppleMetaNodeLocation /Local/Default
create "/users/${N_USERNAME}" GeneratedUID `uuidgen`
create "/users/${N_USERNAME}" UniqueID ${N_UID}
create "/users/${N_USERNAME}" PrimaryGroupID ${N_GID}
create "/users/${N_USERNAME}" Password "*"
create "/users/${N_USERNAME}" RecordName ${N_USERNAME}
create "/users/${N_USERNAME}" RecordType dsRecTypeNative:users
create "/users/${N_USERNAME}" NFSHomeDirectory ${N_HOME}
create "/users/${N_USERNAME}" RealName "Template User"
create "/users/${N_USERNAME}" UserShell /bin/bash
EOF
if [ $? -ne 0 ]; then
    logger -is Creation of ${N_USERNAME} failed.

    # destroy account
    dscl . -delete "/users/${N_USERNAME}" 2>/dev/null
    exit 2
fi

# create group
dscl . <<EOF
create "/groups/${N_USERNAME}"
create "/groups/${N_USERNAME}" AppleMetaNodeLocation /Local/Default
create "/groups/${N_USERNAME}" GeneratedUID `uuidgen`
create "/groups/${N_USERNAME}" PrimaryGroupID ${N_GID}
create "/groups/${N_USERNAME}" RecordName ${N_USERNAME}
create "/groups/${N_USERNAME}" RecordType dsRecTypeNative:groups
create "/groups/${N_USERNAME}" Password "*"
create "/groups/${N_USERNAME}" GroupMembership ${N_USERNAME}
EOF
if [ $? -ne 0 ]; then
    logger -is Creation of ${N_USERNAME} failed.

    # destroy account
    dscl . -delete "/users/${N_USERNAME}" 2>/dev/null
    dscl . -delete "/groups/${N_USERNAME}" 2>/dev/null
    exit 2
fi

# make home directory
mkdir -m 0700 -p ${N_HOME}
ditto --rsrc "${SYSHOMETEMPLATE}" "${N_HOME}"

if [ $? -ne 0 ]; then
    logger -is Creation of ${N_USERNAME} failed.

    # destroy account
    dscl . -delete "/users/${N_USERNAME}" 2>/dev/null
    dscl . -delete "/groups/${N_USERNAME}" 2>/dev/null
    exit 2
fi

chown -R ${N_USERNAME}:${N_USERNAME} ${N_HOME}

logger -i Creation of template user ${N_USERNAME} succeeded.

exit 0

So OE-Cake now has a beta for windows – but not OS X.

We still have the binary from their old version that expired some time ago – we were launching it with ARD with the below trick, but now I’ve moved it into a read-only applescript application that the students can launch themselves:

try
    do shell script "sudo date 1103100008" user name "admin" password "*****" with administrator privileges
end try

do shell script "open /Applications/OE-CAKE\\!.app"
delay 10
do shell script "launchctl stop org.ntp.ntpd" user name "admin" password "*****" with administrator privileges
do shell script "launchctl start org.ntp.ntpd" user name "admin" password "*****" with administrator privileges

I have a mac mini set up as a HTPC. The music end of this was a bit neglected as it was originally set up to use a custom onscreen menu control to switch between several different iTunes libraries and then have basic control within them.

Now that we have iPhones in the house – I consolidated the decade + worth of iTunes libraries into one.

I wanted a way to control the speakers based on iTunes state.

So this little python script runs every 10 seconds, here is a quick outline of what it does.

If iTunes is playing, but the speakers are off – then adjust some volumes on the speakers, turn them on, then rewind the track.

If iTunes is paused for more than 2 minutes, and the speakers are on – and they were turned on by this autodetect system, then turn the speakers off.

The turning on and off is done by my original HTPC applescript which controls an irTrans USB IR transceiver through the iRed software.

Now we just fire up the remote app on the phone, and within a few seconds the speakers are on and the music is playing.

#!/usr/bin/env python
# encoding: utf-8
"""

Created by Preston Holmes on 2009-09-05.
preston@ptone.com
Copyright (c) 2009

Permission is hereby granted, free of charge, to any person obtaining
a copy of this software and associated documentation files (the
"Software"), to deal in the Software without restriction, including
without limitation the rights to use, copy, modify, merge, publish,
distribute, sublicense, and/or sell copies of the Software, and to
permit persons to whom the Software is furnished to do so, subject to
the following conditions:

The above copyright notice and this permission notice shall be included
in all copies or substantial portions of the Software.

THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND,
EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF
MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT.
IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY
CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT,
TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION WITH THE
SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE.
"""

import sys
import os
from appscript import *
from subprocess import Popen, call, STDOUT, PIPE
import plistlib
import time

prefs = '/Library/Preferences/com.ptone.tuneswatcher.plist'
vol_target = '/Users/Preston/Documents/HTPC/targetvolume'

def sh(cmd):
    return Popen(cmd,shell=True,stdout=PIPE,stderr=PIPE).communicate()[0]

def itunes_running_():
    for p in psi.process.ProcessTable().items():
        try:
            if "Applications/iTunes" in p.command:
                return True
        except:
            pass
    return False

def init_prefs ():
    d = {'state':'unknown','last_state_change':time.time(),'auto_on':False}
    plistlib.writePlist (d,prefs)

def itunes_running():
    pcount = sh('ps -xa | grep -c iTunes').strip()
    return int(pcount) > 3

def getSettings():
    settings_file = '/Users/preston/Documents/HTPC/settings'
    f = open(settings_file,'r')
    flines = f.readlines()
    f.close()
    settingsDict = {}
    for aLine in flines:
        aLine = aLine.rstrip('\n')
        #print aLine
        if aLine != '':
            apair = aLine.split(' ')
            settingsDict[apair[0]] = apair[1]
    return settingsDict

def read_prefs ():
    if not os.path.exists (prefs):
        init_prefs()
    d = plistlib.readPlist (prefs)
    htpc_settings = getSettings()
    d.update(htpc_settings)
    return d

def save_prefs(d):
    plistlib.writePlist (d,prefs)

def main():
    d = read_prefs()
    # print d
    is_running = itunes_running()
    if is_running:
        tunes = app("iTunes")
        # print "tunes.running"
        playing = (tunes.player_state() == k.playing)
        # print tunes.player_state() 
        # print playing
        if playing:  
            # print "tunes.playing"
            if d['speakers'] == 'off':
                # turn on speakers
                tunes.pause()
                sh ("""osascript -e 'tell application "HTPC" to power_speakers()'""")
                if int(d['volume']) < 40:
                    open(vol_target,'w').write("50")
                    sh ("""osascript -e 'tell application "HTPC" to setVolume()'""")
                tunes.sound_volume.set(60)
                tunes.player_position.set(0)
                tunes.play()
                d['auto_on'] = True
                save_prefs(d)
                # start streamer here?
            if d['state'] != 'playing':
                d['state'] = 'playing'
                d['last_state_change'] = time.time()
                save_prefs(d)


        else: # iTunes not playing
            if d['state'] == 'playing':
                d['state'] = 'stopped'
                d['last_state_change'] = time.time()
                save_prefs(d)
            # check if last change was 5 minutes ago and turn off speakers
            if d['auto_on'] and ((time.time() - d['last_state_change']) > 10):
                if d['speakers'] == 'on':
                    # turn speakers off only if turned on automatically
                    sh ("""osascript -e 'tell application "HTPC" to power_speakers()'""")
                    sh ("""osascript -e 'tell application "HTPC" to calibrateVolume()'""")
                d['auto_on'] = False
                save_prefs(d)
        # print (tunes.player_state() == k.stopped)


if __name__ == '__main__':
    main()

First off we have a student web proxy that works based on a whitelist concept. So we have to allow any web traffic explicitly. With the help of little snitch and one incomplete help page from google, here is what I’ve added to our global whitelist and seems to be allowing all the features:

• kh.google.com
• www.keyhole.com
• mw2.google.com
• earth.google.com
• auth.keyhole.com
• maps.google.com
• khmdb.google.com

It also want access to www.google.com – but we don’t allow students to full google access – but Google Earth still seems to run fine.

The next annoyance was Google’s softwareupdate which wants to run for all users and update Google Earth on launch.

Adding the following to our loginhook fixed that:

mkdir -P $nethomedir/Library/Google/
touch $nethomedir/Library/Google/GoogleSoftwareUpdate
chown root $nethomedir/Library/Google/GoogleSoftwareUpdate
chmod 644 $nethomedir/Library/Google/GoogleSoftwareUpdate

the $nethomedir var is fetched from dscl earlier in the script

Students need to have stored the proxy password in their keychain – but most have already done that when visiting a web page earlier. They then just need to allow Google Earth to access their keychain.

Note that these are not strong passwords as they are the definition of “dictionary” passwords…

import sys
import os
import random

dfile = '/usr/share/dict/web2'

def main():
    for x in range(130):
        winner = False
        MAX = 234936
        suffix = random.randint (10,100)
        while not winner:
            choice = random.randint(1,MAX)
            f = open(dfile)
            for i,l in enumerate(f):
                l = l.strip()
                length = len(l)
                if i < choice:
                    continue
                if length > 3 and length < 7:
                    winner = '%s%d' % (l,suffix)
                    break
        f.close()
        print winner.lower()


if __name__ == '__main__':
    main()

These are documented here:

http://www.list.org/site.html

They are located in /usr/share/mailman/bin/

so you may want to add that to your path in .bash_profile

most have decent help summary with –help arg at command line

while the remove_members command accepts addresses on the command line itself:

remove_members [options] [listname] [addr1 ...]

remove_members staff frank@foo.com

The add_members command specifically wants file input. Sometimes I just want to add 2-3 so I use this trick:

add_members -r - staff <<-EOF
>sue@foo.com
>bob@bar.com
> <cntl-d>

First for me their was the discovery of cntl-a which jumps one back to the beginning of a line, but just as often I wanted to delete a long path as an opt to a long command.

The first thing is to set your default Editor in your environment variables. I use TextMate – but you could use textwrangler, VI, Emacs.

add a line like this to your ~/.bash_profile

export EDITOR=”mate -w”

then close your terminal session or “source ~/.bash_profile”

Now when you are in the middle of typing a long command, or after hitting the up arrow, press cntl-x and hold it, then hit “e”

boom – your current command opens up in your editor, you can use all the features of that editor, and when you save and close that file – the command will be executed back in your shell.

Since I’ve integrated this tip into my workflow – I find I use it all the time.

Only downside is GUI editors won’t work for SSH since you are in the remote hosts env – there is probably a tricky way to reverse-ssh the editor command back to you, but I haven’t explored that.

The script monitors http://images.apple.com/downloads/macosx/apple/recent.rss and if it has been updated since last check (stores the last check in a plist file) it will check the feed links for any dmg it can find and download it to /downloaded_dmgs/ (which is easy enough to change in the script source)

The script source is below the fold – or download here

#!/usr/bin/env python
# encoding: utf-8
"""
This script attempts to download any dmg files it can find in Apple's RSS feed of updates

requires BeautifulSoup and feedparser modules

Created by Preston Holmes on 2009-07-15.
preston@ptone.com
Copyright (c) 2009

Permission is hereby granted, free of charge, to any person obtaining
a copy of this software and associated documentation files (the
"Software"), to deal in the Software without restriction, including
without limitation the rights to use, copy, modify, merge, publish,
distribute, sublicense, and/or sell copies of the Software, and to
permit persons to whom the Software is furnished to do so, subject to
the following conditions:

The above copyright notice and this permission notice shall be included
in all copies or substantial portions of the Software.

THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND,
EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF
MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT.
IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY
CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT,
TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION WITH THE
SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE.
"""

import sys
import os
import feedparser
import urllib, urllib2
from BeautifulSoup import BeautifulSoup
import plistlib
import re
import pdb
import time

feed_url = 'http://images.apple.com/downloads/macosx/apple/recent.rss'
dmg_location = '/downloaded_dmgs/'
prefs_file = '/Library/Preferences/com.ptone.updatefetcher.plist'

last_done = 0
ticker = 0
current_download = {}
last_msg = ''
debug = False

def download_status(block_ct,block_sz,total):
    """The hook will be passed three arguments; a count of blocks transferred so far, a block size in bytes, and the total size of the file."""
    global last_done, current_download,ticker,last_msg
    now = time.time()
    done = ((block_ct * block_sz)/float(total)) * 100
    if done > last_done + 5 and now-ticker > 1:
        msg = "downloading %s, %smb %s %% done" % (current_download['entry'],total/(1024*1024),int(done)) 
        sys.stderr.write(''.ljust(len(last_msg),'\b'))
        sys.stderr.write(msg)
        last_msg = msg
        sys.stderr.flush()
        last_done = done
        ticker = now

def get_dmg_from_iframepage(url):
    """Sleuths out the dmg url from pages that have an iFrame download form like iTunes and Safari"""
    # pdb.set_trace()
    if not url.endswith('/'):
        return False
    soup = BeautifulSoup(urllib2.urlopen(url).read())
    iframe_url = soup.find('iframe',src=re.compile('.*swdlp.apple.com.*'))
    if not iframe_url: 
        return False
    else:
        iframe_url = iframe_url['src']
    # user agent is needed to get the mac version of safari
    user_agent = 'Mozilla/5.0 (Macintosh; U; Intel Mac OS X 10_5_7; en-us) AppleWebKit/531.2+ (KHTML, like Gecko) Version/4.0.1 Safari/530.18'
    headers = { 'User-Agent' : user_agent }
    req = urllib2.Request(iframe_url, headers=headers)
    page_data = urllib2.urlopen(req).read()
    soup = BeautifulSoup(page_data)
    dmgs = soup.findAll('input',attrs = {'type':'hidden','name':'downloadURL','value':re.compile('.*dmg$')})
    if len(dmgs) > 0:
        return [i['value'] for i in dmgs]
    else:
        return False

def clear_status():
    global current_download,last_done,last_msg
    if last_done > 0:
        sys.stderr.write(''.ljust(len(last_msg),'\b'))
        sys.stderr.write(''.ljust(len(last_msg),' '))
        sys.stderr.write('\r')
        sys.stdout.flush()
    last_done = 0
    last_msg = ''
    
def get_dmg(dmg_url):
    global current_download,last_done,last_msg
    # do fancy clearing of % done
    clear_status()
    fname = dmg_url.split('/')[-1]
    fpath = os.path.join(dmg_location,fname)
    if os.path.exists(fpath): 
        print "%s already downloaded" % fname
    else:
        # save the dmg
        current_download['fname'] = fname
        urllib.urlretrieve (dmg_url,fpath,download_status)
        clear_status()
        print 'downloaded: %s' % current_download['entry']

def main():
    print "============ Script Run %s ============" % time.asctime()
    prefs = {}
    global current_download, last_done, last_msg

    print "Parsing Feed"
    feed = feedparser.parse(feed_url)
    feed_time = time.mktime (feed.feed.updated_parsed)
    if os.path.exists(prefs_file):
        prefs = plistlib.readPlist(prefs_file)
    else:
        prefs = {'last_check':feed_time - 1000}
    
    if debug: prefs = {'last_check':feed_time - 1000}
    
    now = time.mktime(time.gmtime())
    # print feed_time
    # print now
    if feed_time <= prefs['last_check']:
        print "Feed not updated since last check"
        prefs['last_check'] = now
        plistlib.writePlist(prefs, prefs_file)
        sys.exit()
    else:
        print "feed updated %s" % feed.feed.updated
    if not os.path.exists(dmg_location): os.makedirs(dmg_location)
    # loop over the entries
    for entry in feed.entries:
        current_download = {'entry':entry.title}
        dmg_found = False
        url = entry.link
        # load link with urllib2 into beautiful soup
        soup = BeautifulSoup(urllib2.urlopen(url).read())
        # find sidecar link
        downloads = soup.findAll('a',href=re.compile('http://wsidecar.*'))
        if downloads:
            download_url = downloads[0]['href']
            # determine if link is dmg                
            if download_url.endswith('.dmg'):
                dmg_found = True
                # make it a list for compatibility with pages that return multiple dmgs
                dmg_urls = [download_url]
            else:
                dmg_found = dmg_urls = get_dmg_from_iframepage(download_url)            
            if dmg_found:
                for a_dmg in dmg_urls:
                    get_dmg(a_dmg)
            else:
                print "no dmg found in %s" % entry.title
                
    prefs['last_check'] = now
    plistlib.writePlist(prefs, prefs_file)
    
if __name__ == '__main__':
    main()

I work at a school where I’m involved in putting together our yearbook. For several years now we have done a collage of self portraits.

These can be a chore to layout in Illustrator. You want to randomize the grades, scale everything to be in nice rows and line everything up.

Here is what I started with:

Each grade is a different layer (and highlight color)

Its hard to know what size to standardize on to flow these out best onto the cover layout (its a front and back deal) Also I don’t want all of one grade next to another, so they need some shuffling.

So I whipped up this python script to do the grunt work for me.

quick digression on scripting language Note I could have done this in Applescript – and for most of you that would be easier to use. I think that for quick scripts where the bulk of the script is generating apple events – Applescript is generally easier. But as soon as you involve much general programming, Applescript is sorely lacking. It has an awkward and verbose syntax, and has a very limited standard library – so you often have to write all your own utilities. appscript is a Python module that provides access to Apples OSA system and allows you to do anything you can do in Applescript. The syntax takes some getting used to, but if you need to do other things in your script – its a win. For example the “Shuffle” bit in the code below was a single line of python, but would have required a PITA sub-routine in Applescript.

#!/usr/bin/env python
# encoding: utf-8
"""
Created by Preston Holmes on 2009-06-04.
"""

import sys
import os
import appscript
import random
import time
def main():
    il = appscript.app("Adobe Illustrator")
    page_items = il.current_document.page_items()
    random_index = range(len(page_items))
    random.shuffle(random_index)
    left_edge = 15
    curr_x = left_edge
    curr_y = 775
    x_max = 600
    row_height = 80
    print len(random_index)
    for i in random_index:
        item = page_items[i]
        if item.locked() or item.layer.locked():
            continue
        h = item.height()
        w = item.width()
        if h == 0 or w ==0:
            continue
        s = int ((row_height/h) * 100)
        # s2 = row_height/h
        if (curr_x + w) > x_max:
            curr_x = left_edge
            curr_y = curr_y - row_height
        sm = il.get_scale_matrix(horizontal_scale=s,vertical_scale=s)
        # Having some real problems with scale transforms screwing up stroke width - but I'm just working with placed items
        il.transform(item,using=sm,line_scale=100,transforming_stroke_patterns=False)
        # item.stroked_width.set(1)
        # alternate scale approach (also seems to affect stroke width...)
        # item.width.set(w*s2)
        # item.height.set(h*s2)
        left,top,right,bottom = item.geometric_bounds()
        dx = curr_x - left
        dy = curr_y - top
        m = il.get_translation_matrix(delta_x=dx,delta_y=dy)
        # can't use concatenated transform as scale will change the amount to move...
        # full_m = il.concatenate_scale_matrix(m,horizontal_scale=s,vertical_scale=s)
        il.transform(item,using=m)
        curr_x = curr_x + (right - left)

    

if __name__ == '__main__':
    main()

So what does this get me?

I made this to generate one page width – and its easy to split up across pages:

And as you can see, it also shuffled the source layers:

Now if I want to change how it flows – or add or subtract images I can just change the single row_height parameter of the script and get variations like this:

If you realize an image needs 90 degree rotation, it no longer requires having to entirely re-layout the whole project

Of course there is still some massaging needed by hand – but it is a fraction of the effort.

At work I’m involved in laying out our yearbook. Honestly a lot of these pages are just quick photo montages. I’ve made these in both Illustrator and Pages and each have some advantages – I use mostly Illustrator but its harder to have others help in that app.

I wanted to whip up some scripts that could take some of the scale and random rotation repetition out of the process, which these do – the pages version also adds drop shadow – both could be improved to do some repositioning…

The Pages version:

tell application "Pages"

set docName to name of document 1
tell document 1
--mark images:

set imagelist to get every image -- of body text

set an_image to item 1 of imagelist
repeat with an_image in imagelist
set w to width of an_image
set h to height of an_image
if w > h then
set width of an_image to 3
else
set height of an_image to 3
end if
set x to random number from -15 to 15
set rotation of an_image to x
set shadow of an_image to true
set shadow angle of an_image to 58.0
set shadow color of an_image to {0, 0, 0}
set shadow offset of an_image to 7
set shadow opacity of an_image to 75
set shadow blur of an_image to 7

--get properties of an_image
end repeat

end tell
end tell

The Illustrator version:

tell application "Adobe Illustrator"
set s to 2
set maxdim to 250

set page_items to page items of current document

repeat with an_item in page_items
set w to width of an_item
set h to height of an_item
if w > h then
set s to maxdim / w
else
set s to maxdim / h
end if
set width of an_item to w * s
set height of an_item to h * s
set x to random number from -15 to 15
set baseMatrix to get rotation matrix angle x
transform an_item using baseMatrix
end repeat

end tell

First some prerequisites and expectations. You will need a server on your network that supports sharing volumes to OS X, and a way to configure your clients to automount such a sharepoint. Both of these are easily achieved if you have an OS X server that your clients are binding to – but there are other ways. Also I assume that you have some familiarity with what shell scripts are, and file permissions, if not more of a command on how to write them.

At its most basic level, a system of login scripts relies on a single hook provided by the system. In the plist at:

/var/root/Library/Preferences/com.apple.loginwindow.plist

<key>LoginHook</key>
<string>/path/to/script</string>

This just is the path to an executable script that will be run with root privileges at user login. The script is passed a single argument, which is the name of the logging in user.

In this script you can do any number of things you can do in a script – it runs with root privileges (I’ll outline some of those things later on).

Now if you start doing a lot of things in this script, or you want to do some things on some machines but not others, managing the contents of this script can get tricky or tedious. The solution here is the solution for many similar situations. Modularize it into a set of separate scripts. This has the advantage of atomizing different purposes into their own files that can be moved and reused, or even downloaded from other sources and added into your process. It also allows you to use scripts written in multiple languages. You may have some things that are best done in Bash, others that are better done with Python or Perl.

The basic way to modularize this is to have a folder of scripts, and then your login script pointed to by the plist key acts as a wrapper that might look like this:

This basically loops over the directory specified (in this case /etc/loginscripts/) and runs each script in turn – passing along whatever original arguments were sent to the primary loginscript.

This system has been used by several management packages, including both NetRestore, and Radmind/iHook.

This is as good a time as any for a quick digression on iHook. iHook is a fantastic tool that allows you to provide graphical feedback of scripts that are running. If your login scripts are doing things that might take some time – there is no built in way to give the user feedback. iHook provides a UI window that scripts can write their output to (echo) or using iHook’s special commands, can update a progress bar or put up images.

I’m not going into the details here on iHook, but I use a modified version of the multiple scripts system that is distributed with the Radmind Assistant

The new (though I’m not claiming I’m the first/only one to think of or implement this) idea I want to introduce in this post is to take the modular approach another step and use a network folder to hold the scripts to be run at login.

That is, the login script first fetches all the scripts to run from a server, then runs them all locally.

The advantages of this are huge – mainly you no longer have to foresee at imaging time all the possible things you may want to do at login with a script, in order to have those actions written into scripts that are distributed as part of your image. The advantage over via ARD for taking such late breaking actions is that there is no issue if you have many laptops or machines that are often offline on your network, the commands are run at next login.

For me one of the best examples is printer creation. In the Tiger days I dutifully tried to deploy MCX printing – but had any number of problems with it, but primarily:

• Printers that I removed from the list, were not removed from the clients
• Many printer specific features could not be managed via MCX

Now I create printers with a script at login time – if we add or remove a printer from that list, I can make that change to a single copy of the script on the server, and all the machines who participate in this system will see the changes at login.

Another example is that midway through the school year, the administration asked that cameras on the Macs be disabled for students because they were posting to YouTube from campus. I was able to add in a script that did this to the folder on the server and poof, it was implemented instantly on all the machines at next login.

I’m using a combination of locally and remotely modular loginscripts that look something like this:

• login.hook (starts iHook with the contents of the wrapper script as above )
  • locally defined login scripts are each run from /etc/hooks
  • one of these scipts then fetches additional scripts from the server.
    • each local copy of the remotely defined scripts is in turn run

The script that fetches and runs the remote scripts looks like this:

I’m using OS X’s server ability to define an automount at /Network/Library as this is in the search path for several other apps/technology that look in various Libraries. Obviously you want to make sure that such a location is read-only.

OK, so what are some of the useful things you can do with login scripts (local or remote)? Here is a collection of some snippets (see also Mike Bombich’s collection for more ideas). Remember these are snippets and should not just be pasted en-mass into one of your login scripts. Let me know of your own tricks in the comments! (and yes – I know many of these can also be done via MCX – I do use MCX, but sometimes this seems more straightforward to me, and generally works more predictably)

# install that cool little tool you didn’t know about when making your image
# http://duti.sourceforge.net/
if [ ! -e /usr/local/bin/duti ]; then
    ditto /Network/Library/management08/files/duti/usr/local/ /usr/local/
fi

# then use that cool little tool to set quicktime player (not itunes) as the default player for .wav files
# that are generated by that new phone system
echo ‘com.apple.quicktimeplayer com.microsoft.waveform-audio all’ | sudo -u $1 /usr/local/bin/duti

• download and unpack tarball
• su as root

• cd to the unpacked distribution folder and enter the following in terminal:

  python setup.py install
  touch /etc/hosts.deny
  cp /usr/share/denyhosts/denyhosts.cfg-dist /usr/share/denyhosts/denyhosts.cfg
  cp /usr/share/denyhosts/daemon-control-dist /usr/share/denyhosts/daemon-control
  chmod 700 /usr/share/denyhosts/daemon-control
  

read on for configuration

Configure Script

These are the MINIMAL settings for OS X. There are lots of other good options in there including some settings about thresholds and purging, and emailing an admin when a attacker is added to the list. There is also a feature that lets you share your list of attackers with the community, as well as download the community generated blacklist. Using your favorite text editor or the built in pico change the following settings:

pico /usr/share/denyhosts/denyhosts.cfg

look for and change the following:

    SECURE_LOG = /var/log/secure.log
    LOCK_FILE = /tmp/denyhosts.lock

(strictly speaking – you don’t need to do this next part as we will be starting the script with launchd)

pico /usr/share/denyhosts/daemon-control

look for and change the following:

    DENYHOSTS_BIN   = "/usr/local/bin/denyhosts.py"
    DENYHOSTS_LOCK  = "/tmp/denyhosts.lock"

to test out the daemon you can execute the following:

    /usr/bin/env python /usr/local/bin/denyhosts.py --daemon --config=/usr/share/denyhosts/denyhosts.cfg

To launch the monitoring script automatically

create and then edit a launchd plist

    pico /Library/LaunchDaemons/denyhosts.daemon.plist

and then paste the following:

    <?xml version="1.0" encoding="UTF-8"?>
    <!DOCTYPE plist PUBLIC "-//Apple//DTD PLIST 1.0//EN" "http://www.apple.com/DTDs/
    PropertyList-1.0.dtd">
    <plist version="1.0">
    <dict>
            <key>Label</key>
            <string>denyhosts.daemon</string>
            <key>ProgramArguments</key>
            <array>
                    <string>/usr/bin/env</string>
                    <string>python</string>
                    <string>/usr/local/bin/denyhosts.py</string>
                    <string>--daemon</string>
                    <string>--config=/usr/share/denyhosts/denyhosts.cfg</string>
            </array>
            <key>RunAtLoad</key>
            <false/>
    </dict>
    </plist>

then to load the plist without logging out or restarting:

    launchctl load /Library/LaunchDaemons/denyhosts.daemon.plist

try it out – try logging in as a fake user/bad pw and you should see your IP listed in “/etc/hosts.deny”

simply remove a host from that file if you want to gain access again (or see the configuration notes about purging)

back when Jott was free I was one of several people who were using it to get quick voice notes into OmniFocus (OF). A couple of limitations of the Jott solution was that:

• It sometimes had trouble understanding the recipient of “who I wanted to Jott”
• You never got a chance to proofread or edit the transcription before it got sent

There are just times when I think of something on the go, and firing up the iPhone OF app and tapping it in won’t work, so when a friend told me about vlingo for iPhone and twitter, I thought – hmm I’ve been playing a bit with the twitter API and that might be a good route for me to get stuff from voice into OF.

So here are the steps (updated with more detail):

• Set up a new twitter account – make it “private” (assuming you don’t want to world to see your tasks)
• Set up vlingo to post status updates to this twitter account (obvious caveat, you won’t be able to use vlingo for your regular twitter account…)
• download the python script (Click on the link, then do a save as, from here we will assume its on your Desktop, but it is quite likely you may want to keep it somewhere else)
• Rename it oftwitter.py (mind os x’s habit of hiding extensions…)
• open the script in any text editor and set the username and password settings to those of your new account

• open terminal and type the following lines (you will need your admin password)

  chmod +x ~/Desktop/oftwitter.py
  sudo easy_install appscript
  sudo easy_install twitter
  

you will get a warning about a C extension not being compiled, but you can ignore that.

• Now assuming you have already logged some tweets to the twitter account, try it out by typing the following into terminal:

  ~/Desktop/oftwitter.py
  

You should see your tweets as items in your OmniFocus inbox

The script creates a preference file stored at ~/Library/Preferences/com.ptone.oftwitter.plist that stores the most recent tweet and uses that to query twitter more effectively.

to set up automated retrieval of new tweets install Lingon and configure a new launch agent to look something like this:

Now when you are out and about you can fire up vlingo on the iPhone and say: “twitter, order new engine for rocket ship”, you get the chance to fix any quick errors, then update your status. I’m finding it even easier to just hit the “social” button in vlingo, then you don’t have to say “twitter” – just your task.

When you return to your Mac – it should shortly appear in your OF inbox. This script assumes you keep OmniFocus open

A couple other goodies:

• If you say “important” or “flag”, that word will be stripped, and the entry will be flagged in OF
• you can send a @reply message to the twitter account and replies are checked too (with the leading @user stripped from the start)
  • This is handy if you want to give someone a way to submit tasks to your inbox, they can just send a tweet to your new account

To remove all traces:

• delete the oftwitter.py script file
• delete ~/Library/Preferences/com.ptone.oftwitter.plist
• delete /Library/Python/2.5/site-packages/appscript*
• delete /Library/Python/2.5/site-packages/simplejson
• delete /Library/Python/2.5/site-packages/twitter

I try to keep up with them, use them where possible, but don’t put EVERYTHING I do in version control. Since I am on a Mac running Leopard, I do have, and use Time Machine and so wanted to see if it would be pretty easy to use that to do some quick diffs between some source files.

The result is a quick and dirty python script. In the unlikely event that I ever have time, this would be a cool pyObjC project, a file browser panel, date versions picker, and a webkit view (with some better css).

The source to the script is below the fold – it will look for the first attached volume that has time machine backups for the current machine. It will not work with network based time machine backups that are on disk images. This script will run on a stock Leopard install without any extra python modules needed.

#!/usr/bin/env python
# encoding: utf-8
"""
Created by Preston Holmes on 2009-02-23.
Copyright (c) 2009 __MyCompanyName__. All rights reserved.
"""

import sys
import os
import getopt
import difflib
import time
import pdb
from subprocess import Popen, PIPE

# if you set time_machine_path, it should be to the full path of this machines backup drive:
# ie '/Volumes/TM_Drive/Backups.backupsdb/Joes-Mac/'
# if not set explicitly - the script will use the first TM drive it finds, 
# and the first host folder it finds - which should work for most cases
# Will Not work with network or disk image based time machine backups

time_machine_path = None

cmd = 'tell application "Finder" to name of (path to startup disk)'
boot_volume = Popen('osascript -e \'%s\'' % cmd,shell=True,stdout=PIPE,stderr=PIPE).communicate()[0][0:-1]



help_message = '''
Call this script with the path to one or more text based files as arguments
'''
header = '''

<!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.0 Transitional//EN"
          "http://www.w3.org/TR/xhtml1/DTD/xhtml1-transitional.dtd">

<html>

<head>
    <meta http-equiv="Content-Type"
          content="text/html; charset=ISO-8859-1" />
    <title></title>
    <style type="text/css">
        table.diff {font-family:Courier; border:medium;}
        .diff_header {background-color:#e0e0e0}
        td.diff_header {text-align:right}
        .diff_next {background-color:#c0c0c0}
        .diff_add {background-color:#aaffaa}
        .diff_chg {background-color:#ffff77}
        .diff_sub {background-color:#ffaaaa}
    </style>
</head>

<body>
'''
footer = '''
    <table class="diff" summary="Legends">
        <tr> <th colspan="2"> Legends </th> </tr>
        <tr> <td> <table border="" summary="Colors">
                      <tr><th> Colors </th> </tr>
                      <tr><td class="diff_add">&nbsp;Added&nbsp;</td></tr>
                      <tr><td class="diff_chg">Changed</td> </tr>
                      <tr><td class="diff_sub">Deleted</td> </tr>
                  </table></td>
             <td> <table border="" summary="Links">
                      <tr><th colspan="2"> Links </th> </tr>
                      <tr><td>(f)irst change</td> </tr>
                      <tr><td>(n)ext change</td> </tr>
                      <tr><td>(t)op</td> </tr>
                  </table></td> </tr>
    </table>
</body>

</html>
'''
class Usage(Exception):
    def __init__(self, msg):
        self.msg = msg

def find_versions(path):
    #pdb.set_trace()
    print 'looking for versions of %s' % path
    #print time.strftime("%m/%d/%Y %I:%M:%S %p",time.localtime(os.path.getmtime(fname)))
    backups = os.listdir(time_machine_path)
    backups.sort()
    pathlist = [os.path.join(time_machine_path,b,boot_volume,path[1:]) for b in backups]
    pathlist.append(path)
    versions = []
    mod_times = []
    for f in pathlist:
        if os.path.exists(f):
            mod_time = time.localtime(os.path.getmtime(f))
            if not mod_time in mod_times:
                mod_times.append(mod_time)
                versions.append({mod_time:f})
    return versions
    
def getTMLocation():
    global time_machine_path
    if time_machine_path and os.path.exists(time_machine_path):
        return True
    volumes = os.listdir('/Volumes')
    #hostname = os.uname()[1].split('.')[0]
    #cmd = 'scutil --get ComputerName'
    #machine_name = Popen('osascript -e \'%s\'' % cmd,shell=True,stdout=PIPE,stderr=PIPE).communicate()[0][0:-1]
    for v in volumes:
        if os.path.exists(os.path.join('/Volumes',v,'Backups.backupdb')):
            backupsdb = (os.path.join('/Volumes',v,'Backups.backupdb'))
            time_machine_path = os.path.join(backupsdb,os.listdir(backupsdb)[0])
            return True
            # candidate_path = os.path.join('/Volumes',v,'Backups.backupdb',machine_name)
            # if os.path.exists(candidate_path):
            #     time_machine_path = candidate_path
            #     return True
    return False
    
def main(argv=None):
    
    if not getTMLocation():
        print 'No Time Macine Backup Found'
        sys.exit()
    print 'time machine path: ' + time_machine_path
    if argv is None:
        argv = sys.argv
    try:
        try:
            opts, args = getopt.getopt(argv[1:], "ho:v", ["help", "output="])
        except getopt.error, msg:
            raise Usage(msg)
    
        # option processing
        for option, value in opts:
            if option == "-v":
                verbose = True
            if option in ("-h", "–help"):
                raise Usage(help_message)
            if option in ("-o", "–output"):
                output = value
        if not args:
            raise Usage(help_message)
        differ = difflib.HtmlDiff(tabsize=4)
        html = header
        for path in args:
            path = os.path.join(os.getcwd(),path)
            html += '<h1>Changes for %s</h1' % os.path.basename(path)
            versions = find_versions(path)
            if len(versions) < 2:
                html += '<h2>Less than 2 Versions found on Time Machine Backup</h2>'
            else:
                html += '<h2>%s versions found</h2>' % len(versions)
            for i in range(0,len(versions)):
                if i: #skip the first
                    d1 = time.strftime("%m/%d/%Y %I:%M:%S %p",versions[i-1].keys()[0])
                    d2 = time.strftime("%m/%d/%Y %I:%M:%S %p",versions[i].keys()[0])
                    html = html + '<h2>changes from %s to %s</h2>' % (d1,d2)
                    l1 = open(versions[i-1].values()[0]).readlines()
                    l2 = open(versions[i].values()[0]).readlines()
                    table = differ.make_table(l1,l2,context=True,numlines=3)
                    html += table
        html += footer
        o = open('/tmp/diff.html','w')
        o.write(html)
        o.close()
        import webbrowser
        webbrowser.open('/tmp/diff.html')
    except Usage, err:
        print >> sys.stderr, sys.argv[0].split("/")[-1] + ": " + str(err.msg)
        print >> sys.stderr, "\t for help use –help"
        return 2


if __name__ == "__main__":
    sys.exit(main())