Sonart on Pi (Part I)

Sonart is my project to control Sonos / Spotify directly via a camera and album covers.

Since building the POC (see my last post above), I wanted to make this a permanent capability for my Sonos set up. I decided to build a ‘production’ version, on top of Raspberry Pi hardware. This post is about how I got the pi fully set up for computer vision.

I chose a Raspberry Pi 2 (as I’ll need as much processing power as possible to process the images quickly), and also bought the official Pi Camera, and a case that could hold the whole thing. After having assembled the Pi, camera and case, and plugging in all the relevant cables, I powered it up and installed Raspbian through Noobs, making sure to enable the camera module in raspi-config.

From there I started to follow various guides to install and test the camera, get SimpleCV installed and deploy my Sonart application. I’d like to say things worked perfectly, but unsurprisingly I came up against a number of issues! In this post I’ve detailed those issues, and how I managed to fix them. I’ve also included the happy-path code for anyone who wants to avoid the problems!

Installation Issues

• The first time I tried to use the camera on the pi, it didn’t work. I tried to take a simple still from the command line by running:

  raspistill -n -t 300 -o test.jpg

  Breaking this down:

• Take a still (raspistill)
• Don’t show a preview window because I haven’t loaded the GUI (-n)
• Don’t wait the default 5 seconds before taking the photo, just wait 300 ms to let the camera start (-t 300)
• Write the image to this directory and call it test.jpg (-o test.jpg)Unfortunately, it didn’t work and I got a weird error instead. It turns out I had forgotten to upgrade my raspbian installation, and the version that had come shipped with my pi wasn’t new enough to know about the camera. Nothing a quick ‘sudo apt-get upgrade’ won’t fix, I thought. Unfortunately however, the upgrade process got about half way through before spewing out the message ‘segmentation fault’. At this point my raspbian installation was ruined, and wouldn’t respond or reboot.Fortunately, the solution was fairly quick: Hold shift as you restart the pi, taking you back to the noobs recovery image. From there I reinstalled a fresh raspbian, and on loading that, I ran ‘sudo apt-get update’, then ‘sudo apt-get upgrade’ as the first two commands, which worked this time. After that I tested the camera with ‘raspistill’ and it worked perfectly! I was impressed by the image quality.
• Later in the process, I followed the guide to install simpleCV and ipython, but also supplemented that by installing ipython-notebook, as I knew this would allow me to see images from the pi directly (even though I wasn’t using the GUI), through the notebook web interface.After installing notebook, I opened server, telling it to allow me to connect from anywhere on the network, and supressing the default behaviour to open a browser on the pi:
  ip

  ipython-notebook --ip='*' --no-browser

  Then I tried the boilerplate code to read in the test.jpg image I made with the camera, and display it in the notebook:

  from SimpleCV import *
  disp = Display(displaytype='notebook')
  
  img = Image('test.jpg')
  img.save(disp)

  Unfortunately, the code errored on the ‘disp = Display’ line, with the error that it was ‘unable to open a console terminal’, from line 163 of Display.py in the SimpleCV module. I decided to take a look at the offending code, so went to find the SimpleCV module files on the system:

  cd /usr/local/lib/python2.7/dist-packages/SimpleCV
  sudo nano Display.py

  I navigated to line 163, and could see that it seemed to be trying to set a display icon, which seemed pretty redundant – so I tried the simplest possible solution of commenting it out.

  Having reloaded the module in python, I ran the SimpleCV code again, and hey presto, my image appeared.

• This was the hardest one. After checking that SimpleCV was working, and that I could load and display an image, I next tested SimpleCV’s findKeypoints() function, by trying to open an image, draw on it’s keypoints, and then show it in the notebook:

  from SimpleCV import *<br>disp = Display(displaytype='notebook')<br>
  img = Image('test.jpg')
  kp = img.findKeypoints()
  kp.draw()
  img.save(disp)

  Disaster. I got another Seg Fault, this one originating from pygame. A bit of googling had me try moving to the develop branch of SimpleCV (no luck), and then deciding I needed to upgrade my opencv installation.

  I had installed opencv with a simple ‘sudo apt-get install python-opencv’. I checked the version by running the following in python:

  import cv2
  cv2.__version__

  It returned version 2.4.1. The posts I read were indicated that moving to version 2.4.4 would fix this bug – however this meant building from source. First I ran ‘sudo apt-get remove python-opencv’ to get rid of the old version. Then I followed OpenCV’s build process (you can see the code below in the happy-path).

  Alas, the first time I ran the build, it failed, having run out of space at 42%… I ran ‘build clean’ to remove the progress, and noted that I had tried to do the build with about 500mb space free. I then removed some other files from the Pi, so that there was 2.4Gb free, and ran the build again.

  After several hours, I re-ran the __version__ code above, and following that re-ran the keypoint code. What happened!?

So, finally, here is my happy-path code to get a fresh raspbian install on a Pi 2 ready for some computer vision:

# upgrade the distro so that the pi camera module works
sudo apt-get update
sudo apt-get upgrade

# install all the python basics (Relative to the 'official' instructions, I've added ipython-notebook to this list, and taken away python-opencv)
sudo apt-get install ipython ipython-notebook python-scipy python-numpy python-setuptools python-pip

# install the latest develop branch simplecv
mkdir ~/simplecv_code
cd ~/simplecv_code
git clone git://github.com/sightmachine/SimpleCV.git
cd SimpleCV
sudo pip install -r requirements.txt
sudo python setup.py develop

# build and install the latest stable opencv release
mkdir ~/opencv_code
cd ~/opencv_code
wget http://sourceforge.net/projects/opencvlibrary/files/opencv-unix/2.4.10/opencv-2.4.10.zip/download
unzip download
cd opencv-2.4.10
mkdir build
cd build
cmake -D CMAKE_BUILD_TYPE=RELEASE -D CMAKE_INSTALL_PREFIX=/usr/local -D BUILD_NEW_PYTHON_SUPPORT=ON -D BUILD_EXAMPLES=ON ..
screen
make
sudo make install
sudo sh -c 'echo "/usr/local/lib" > /etc/ld.so.conf.d/opencv.conf'
sudo ldconfig

# test that it's all working by opening up ipython notebook

Note the ‘screen’ command above. I realised the make was going to take a *long* time, when it was at 2% after 10 minutes. I also knew that I was accessing my pi via ssh, and that the connection wasn’t great and had already flaked out a few times. Because I wanted to leave the ‘make’ command to do it’s thing, I quick installed ‘screen‘ (sudo apt-get install screen), and then loaded a session before calling make. This allows me to disconnect from the terminal session, but still leave it running. If I lost connection, it meant I could ssh back in, and then re-attach to the screen session I’d left running.

• ‘screen’, followed by spacebar, opens a new screen.
• ctrl + a, followed by d, detaches the screen
• ‘screen -r’ reattaches to the screen