My current iPython research setup

This is a summary of how I’ve setup my ipython research environment on a Debian Squeeze computer (the MakeHackVoid Sun computer (Morphia) mentioned in my earlier post)

Using virtualenvwrapper lets me setup different python workspaces without changing the default environment for anyone else using the computer.

I’m running mySQL, Solr and iPython on the remote computer and using the iPython notebook in my local browser via a ssh tunnel.

Install all the system support libraries

These are required by the python modules I install below. If you remove some of the modules, you may not need all of these support libraries.

sudo apt-get install python-dev libfreetype6-dev liblapack-dev libatlas-dev \
libxml2-dev libxslt-dev gfortran uuid-dev libmysqlclient-dev

Install ZeroMQ

iPython needs ZeroMQ for notebooks.

http://www.zeromq.org/intro:get-the-software

wget http://download.zeromq.org/zeromq-2.2.0.tar.gz
tar -zxvf zeromq-2.2.0.tar.gz
cd zeromq-2.2.0
configure
make
make check
sudo make install
sudo ldconfig

Install virtualenv and virtualenvwrapper

http://www.doughellmann.com/docs/virtualenvwrapper/

Not sure if these steps are necessary because I went back and forward trying different approaches:

sudo apt-get install python-virtualenv
sudo apt-get install python-pip

Uninstall debian squeeze default virtualenvwrapper as it is old:

sudo apt-get remove virtualenvwrapper

Make sure to delete /etc/bash_completion.d/virtualenvwrapper which is virtualenvwrapper.sh from the original debian install of virtualenvwrapper. It is not removed when you unistall virtualenvwrapper from debian.

Then:

sudo pip install virtualenvwrapper

to install the latest version.

Add lines to your .bashrc to define where the virtual environments should be

export WORKON_HOME=$HOME/.virtualenvs
export PROJECT_HOME=$HOME/Devel
source /usr/local/bin/virtualenvwrapper.sh

I also created the Devel directory, although I’m not sure if that was necessary.

Install ipython in virtual environment

Create a virtual environment and install ipython and all of the modules I’m using
into it. You may not want all of these modules.

mkvirtualenv --no-site-packages brendatest
pip -E $VIRTUAL_ENV install ipython
pip -E $VIRTUAL_ENV install tornado
pip -E $VIRTUAL_ENV install numpy
pip -E $VIRTUAL_ENV install scipy
pip -E $VIRTUAL_ENV install pandas
pip -E $VIRTUAL_ENV install matplotlib
pip -E $VIRTUAL_ENV install gensim
pip -E $VIRTUAL_ENV install pyzmq
pip -E $VIRTUAL_ENV install sunburnt
pip -E $VIRTUAL_ENV install lxml
pip -E $VIRTUAL_ENV install pytz
pip -E $VIRTUAL_ENV install httplib2
pip -E $VIRTUAL_ENV install MySQL-python

Although I’m now using sunburnt, some of my earlier tests use solrpy so install that too:

pip -E $VIRTUAL_ENV install solrpy

Updating to development version of ipython

Copy the base setup so can go back to it:

cpvirtualenv brendatest brendabase

This will not copy the working directory under

workon brendatest

Upgrade to the most recent pip in the virtual environment

pip -E $VIRTUAL_ENV install --upgrade iphython

For me this changed pip from 0.7.2 to 1.1
New version doesn’t use the -E option anymore. So just use:

pip install git+git://github.com/ipython/ipython.git

Setup virtual environment for every login

I added: workon brendatest
to the bottom of my .bashrc file so the brendatest environment is always activated when I log in directly.

MySQL

As well as putting my database into mySQL on morphia, I need to add a readonly user matching that defined in the Solr db-data-config.xml on morphia for solr access to the mySQL database. I also need to add a user on morphia with a password to give iPython programs access to database.

Remote iPython access

Having finished setting up ipython, it is possible to run it in a ssh terminal window, but it is using the w3m browser. I want to view it in chrome on my local computer instead. So I’ve written some scripts to do this. At the server end I have have a script to start solr:

startSolr.sh

#!/bin/sh
cd java -Xms1024M -Xmx10240M -jar start.jar

and one to start iPython notebook.

Because using -t with ssh doesn’t load your user profile (.bash_rc for me), need to set up the virtualenvwrapper environment in this script.
http://unix.stackexchange.com/questions/9883/how-can-i-run-a-script-immediately-after-connecting-via-ssh

startIPython.sh

#!/bin/bash
export WORKON_HOME=$HOME/.virtualenvs
export PROJECT_HOME=$HOME/Devel
source /usr/local/bin/virtualenvwrapper.sh
workon brendatest
ipython notebook --pylab inline

Then I have a startup and shutdown script on my macbookpro. I’m using certificate authentication for ssh, so my ssh commands don’t have any password information.

Replacein the scripts below with your username and remote computer url for ssh. For example user@foobar.com

The startup script sets up the port redirection for ipython notebook, start solr and ipython remotely and then start chrome with the ipython notebook index page.

The shutdown script relies on the fact that I don’t have anything else running on morphia. This is probably not the best approach.

startupRemotePython.sh

#!/bin/sh
# want to automate setting up of port forwarding from morphia.makehackvoid.com and start up of solr and ipython on morphia before opening ipython notebook page in chrome on my computer.

# setup redirection for ipython notebook
ssh -f -L 8888:localhost:8888 -N

# on morphia...
ssh -f ./startSolr.sh
# delay here to keep solr startup messages seperate from ipython ones - not actually needed just cosmetic
sleep 1
ssh -f b ./startIPython.sh

# open browser for http://localhost:8888/ to access remote notebook
# need to add something here to keep window open until I want to exit ipython
sleep 1
/usr/bin/open -a "/Applications/Google Chrome.app" 'http://localhost:8888/'

shutdownRemotePython.sh

#!/bin/sh
# now shut everything back down.
ssh 'ps h -u brenda -o pid | xargs kill'

I’d like to find a way to have the shutdown happen automatically when I close the terminal window that the startup script has run in (or maybe a way of having the startup script wait to be told to quit?)

2 comments
  1. Hmm, useful setup. I think I’ll try to duplicate this on an EC2 instance and see how that goes.

  2. I’d be interested to hear how that goes.

Leave a Reply

Your email address will not be published.

This site uses Akismet to reduce spam. Learn how your comment data is processed.