Category: Science

16 September 2012

Learning how to install python packages

In an earlier post I described my python setup for Debian, this time I’ve kept notes while working yesterday to get a stable python environment on Mac OS X Mountain Lion (10.8.1). I’ve used the notes to write up the steps that worked, there were a lot of dead ends along the way that I haven’t included.

Since the upgrade to Mountain Lion, I’ve had a lot of problems with my python environment not working, and in trying to sort it out I’d somehow managed to end up with a broken easy_install. I think I achieved that by downloading and installing the latest command line tools from the Apple Developer website instead of through XCode. So I’d recommend sticking with the one installed through XCode. But at that stage I wasn’t taking notes about what I was trying, so I’m not sure.

I fixed easy_install by installing distribute:

curl -O http://python-distribute.org/distribute_setup.py
sudo python distribute_setup.py

1 2	curl -O http://python-distribute.org/distribute_setup.py sudo python distribute_setup.py

I noticed that my path had /usr/bin ahead of /usr/local/bin, but this should probably be the other way around.

/usr/local/git/bin:/usr/bin:/bin:/usr/sbin:/sbin:/usr/local/bin:/opt/X11/bin:/usr/texbin:

1	/usr/local/git/bin:/usr/bin:/bin:/usr/sbin:/sbin:/usr/local/bin:/opt/X11/bin:/usr/texbin:

This is being set by /etc/pathsÂ Â and the ~/.bashrc

Changed /etc/paths to have:

/usr/local/git/bin:/usr/local/bin:/usr/bin:/bin:/usr/local/sbin:/usr/sbin:/sbin:/opt/X11/bin:/usr/texbin

1	/usr/local/git/bin:/usr/local/bin:/usr/bin:/bin:/usr/local/sbin:/usr/sbin:/sbin:/opt/X11/bin:/usr/texbin

Then started setting up my ipython, scipy and pandas environment. Using “distribute” in the virtual environment seems to work better at building some packages.

mkvirtualenv --distribute brmTest2
pip install tornado
pip install ipython
pip install numpy
pip install scipy

mkvirtualenv --distribute brmTest2

pip install tornado

pip install ipython

pip install numpy

pip install scipy

The scipy install failed, and I fixed it by installing the dev version as suggested here – http://www.thisisthegreenroom.com/2012/compiling-scipy-on-mountain-lion/

pip install -e git+https://github.com/scipy/scipy#egg=scipy-dev
pip install pandas
pip install matplotlib

pip install -e git+https://github.com/scipy/scipy#egg=scipy-dev

pip install pandas

pip install matplotlib

The matplotlib install failed, so I then followed these instructions -
https://gist.github.com/1860902Â (but not step 5) to install X11 and pkg-config as follows:

download and install X11 from

http://xquartz.macosforge.org/trac/wiki
sudo xcode-select -switch /Applications/Xcode.app/Contents/Developer

1 2	http://xquartz.macosforge.org/trac/wiki sudo xcode-select -switch /Applications/Xcode.app/Contents/Developer

logged out and in to activate X11.

brew install pkg-config

1	brew install pkg-config

(not sure this actually helped)

The x11 libraries are not on the path (which I thought was what pkg-config was meant to fix?) Fixed by using:

export CFLAGS="-I/usr/X11/include -I/usr/X11/include/freetype2 -I/usr/X11/include/libpng12"

1	export CFLAGS="-I/usr/X11/include -I/usr/X11/include/freetype2 -I/usr/X11/include/libpng12"

then

pip install matplotlib
pip install pyzmq

1 2	pip install matplotlib pip install pyzmq

So I now have a virtual environment with the right Python packages installed, and I understand a bit more about working out how to install packages when they fail. I still need to work out what the difference is between pip, distribute and easy_install.

1 January 2012

Learning Python – iPython, matplotlib and Pandas

As I said in my last post, I was inspired by the talk at OSDC2011Â by Dr Edward Schofield, Python for R&D to try out Python and in particular iPython.Â

So I’ve been learning Python by using iPython for analysing my twitter data. The iPython notebook provides a fantastic environment for doing this by letting you write notes in between blocks of python code, and see the results from running the python on the same page.

I’m starting ipython by opening a terminal window in the directory I have my ipython notebooks and running:
ipython notebook --pylab inline which makes the matplotlib graphics appear inline (on the webpage) instead of in a separate window.

I tried a few different approaches to getting everything working on Mac OSX Lion. The Scipi-Superpack for OS X was the last I tried, and it seems to have got the last piece that I hadn’t got working via the other approaches, Pandas and scikits.statsmodels, working.

I’m using the dev version of iPython from GitHub. It is great that they have it setup so that each time I pull the updates they are available straight away without any extra install just by restarting the notebook.

I began by working out how to get data sets from mySql and from Apache Solr and then draw graphs of them using matplotlib. I used paired lists for this as that was what the matplotlib examples used. When I started trying to add time series of different lengths and with different gaps in the data I started to find the limitations of paired lists. Looking around for python time series libraries I found scikits timeseries which looked good, but then came across scikits.statsmodels and Pandas and decided to try them.

If you want to try running this code, I’ve linked the iPython notebook that these code snippets were taken from at the bottom of the post.

Convert pair of Python lists to Pandas series

Pandas makes it easy to convert the paired lists into a pandas.series object:

def convertListPairToTimeSeries(dList, cList):
    # my dateList had date objects, so convert back to datetime objects
    dListDT = [datetime.datetime.combine(x, datetime.time()) for x in dList]
    # found that NaN didn't work if the cList contained int data
    cListL = [float(x) for x in cList]
    # create the index from the datestimes list
    indx = pandas.Index(dListDT)
    # create the timeseries
    ts = pandas.Series(cListL, index=indx)
    # fill in missing days
    ts = ts.asfreq(pandas.datetools.DateOffset())
    return ts

def convertListPairToTimeSeries(dList, cList):

# my dateList had date objects, so convert back to datetime objects

dListDT = [datetime.datetime.combine(x, datetime.time()) for x in dList]

# found that NaN didn't work if the cList contained int data

cListL = [float(x) for x in cList]

# create the index from the datestimes list

indx = pandas.Index(dListDT)

# create the timeseries

ts = pandas.Series(cListL, index=indx)

# fill in missing days

ts = ts.asfreq(pandas.datetools.DateOffset())

return ts

Adjusting the Pandas series

I then made the two data sets (tweets received per day and tweets limited per day) have the same start and end and filled in all the missing days with 0′s or, where I knew that I had a data collection outage, with NaN.

# my data had lots of gaps that were actually 0 values, not missing data
# So I used this to fix the NaN outside the known outage
startOutage = datetime.datetime(2011,12,7)
endOutage = datetime.datetime(2011,12,8)
# set all NaN values to 0
tsFilled = tSeries.fillna(0)
# set the known outage values back to NAN
tsFilled.ix[startOutage:endOutage] = numpy.NAN

# my data had lots of gaps that were actually 0 values, not missing data

# So I used this to fix the NaN outside the known outage

startOutage = datetime.datetime(2011,12,7)

endOutage = datetime.datetime(2011,12,8)

# set all NaN values to 0

tsFilled = tSeries.fillna(0)

# set the known outage values back to NAN

tsFilled.ix[startOutage:endOutage] = numpy.NAN

If the gaps in my data were all meant to missing values instead of 0, I could have just left the series as they were and used pandas.join instead of + to add them together

Truncating the data

I truncated the data, both to make the series the same length (although there are other ways to do this) and to remove partial days of data at the start and end of the periods.

# use slicing to change length of data
tSeriesSlice = tSeries.ix[startData:endData]
# use truncate instead of slicing to change length of data
tSeriesTruncate = tSeries.truncate(before=startData, after=endData)

# use slicing to change length of data

tSeriesSlice = tSeries.ix[startData:endData]

# use truncate instead of slicing to change length of data

tSeriesTruncate = tSeries.truncate(before=startData, after=endData)

Plotting the data

Basic graphing of a Pandas series is very straight forward:

tsFilled.plot();

1	tsFilled.plot();

It is easy to have multiple lines on the same graph, and to add titles and axis labels.

tsFilled.plot(label="original")
tsNew = tsFilled+(rand(len(tsFilled)))
tsNew.plot(label="adding")
tsNew1 = tsFilled.fillna(1)
tsNew1 = tsNew1 +(rand(len(tsNew1)))
tsNew1.plot(label="fillna+adding")
plt.legend(loc=3)
plt.title("Testing Panda Plotting")
plt.ylabel("Counts");

tsFilled.plot(label="original")

tsNew = tsFilled+(rand(len(tsFilled)))

tsNew.plot(label="adding")

tsNew1 = tsFilled.fillna(1)

tsNew1 = tsNew1 +(rand(len(tsNew1)))

tsNew1.plot(label="fillna+adding")

plt.legend(loc=3)

plt.title("Testing Panda Plotting")

plt.ylabel("Counts");

For more control over the layout, it is possible to pass the matplotlib axis in the Pandas plot statement.

# loc = MonthLocator()
loc = DayLocator(interval=2)
formatter = DateFormatter('%d %b')
fig = plt.figure(figsize=(8,4))
ax = fig.add_subplot(211) 
plt.title("Testing Panda Plotting")
tsFilled.plot(label="original", ax=ax)
plt.ylabel("Counts")
plt.legend()
ax2 = fig.add_subplot(212) 
tsFilled.fillna().plot(label="filled", ax=ax2, color='g')
plt.legend()
plt.ylabel("Counts")
plt.xlabel("2011")
ax2.xaxis.set_major_locator(loc)
ax2.xaxis.set_major_formatter(formatter)
labels = ax2.get_xticklabels()
setp(labels, rotation=80, fontsize=10);

# loc = MonthLocator()

loc = DayLocator(interval=2)

formatter = DateFormatter('%d %b')

fig = plt.figure(figsize=(8,4))

ax = fig.add_subplot(211)

plt.title("Testing Panda Plotting")

tsFilled.plot(label="original", ax=ax)

plt.ylabel("Counts")

plt.legend()

ax2 = fig.add_subplot(212)

tsFilled.fillna().plot(label="filled", ax=ax2, color='g')

plt.legend()

plt.ylabel("Counts")

plt.xlabel("2011")

ax2.xaxis.set_major_locator(loc)

ax2.xaxis.set_major_formatter(formatter)

labels = ax2.get_xticklabels()

setp(labels, rotation=80, fontsize=10);

For plotting multiple lines, it is probably better to add the series into a Pandas dataframe and then plot from that, but I’ll leave that for another day.

I’m new to python, matplotlib and Pandas, so I’d be very happy for any feedback about better ways to do things.

iPython Notebook for these examples

Download pandasTimeSeriesNotes.ipynb_.zip (58k)

Links

29 August 2010

Asteroid Discovery since 1980 – video by British astronomer Scott Manley

An amazing video of Asteroid discovery since 1980. It is best viewed in high resolution at YouTube. There is an interesting article on the Daily Mail UK about it too.