Using Travis-CI on python-compiletools

Background

If you buy into the concept that Unit Tests and Continuous Integration is a good thing then you will eventually bump into Travis-CI (https://travis-ci.org/).  The concept is that Travis-CI will take your github project (in my case https://github.com/Zomojo/compiletools) and run the unit tests.  All you need to do is add a small .travis.yml file that contains details like the language and commands to run your tests then the rest is supposed to be automagic.

Reality vs Marketing

With hindsight, I bet that for many projects Travis-CI works like a charm.  However I immediately failed. Now that I’ve finally cracked it, I can say that one of the reason I initially failed was due to the python requirements of my project and the other was a difference in hyphen handling by argparse.

My project was developed on Fedora 24 and Arch Linux, however the container that Travis-CI provides is Ubuntu based. As a specific example of dependency issues, I depend on configargparse. On F24 (Oct 2016), installing python-configargparse gives you version 0.10.  Ubuntu doesn’t provide configargparse in a package so you get  configargparse from PyPi (via pip install configargparse) which is now at version 0.11.  It turns out that there is a change in API between those two versions that broke my project.  That wasn’t the only difference between the two environments but it is a good example.

Experienced pythonistas will simply point out that I could have pinned the version of configargparse in my requirements.txt. And while that is true, it’s 1) not obvious to somebody with only hundreds of hours of python experience, and 2) wasn’t my only problem.

At this point a good bit of advice to my future self is “Use Travis-CI from the beginning”. If I had have done that my project would not have evolved into a place where it took almost two days to get running on Travis-CI.

Travis-CI tips

My (finally working) .travis.yml file is

language: python
python:
    - "2.7" 
    - "3.4"
    - "3.5"
addons:
    apt:
        sources:
            - ubuntu-toolchain-r-test
        packages:
            - g++-5
            - gcc-5

# command to install dependencies 
install:
    - pip install -r requirements.txt 
    - export CXX="g++-5" CC="gcc-5" 

# command to run tests
script:
    - nosetests

The language and versions of python are fairly obvious.  The addons section is only necessary because my project (python-compiletools) is about compiling C++ code so I need a C++ compiler available to run the tests.

The install section took a lot to learn.  There seems to be many ways of doing this section and one of my own hangups was that I didn’t want to write a requirements.txt that got out of sync with my setup.py.  I eventually learnt that if you have a requirements.txt that contains simply

-e .

then the requirements are extracted from setup.py.

The export line in the install section is a project specific requirement. That is, I need a compiler that can compile C++11 in order to complete my specific unittests.

Initial Giveup

It’s worth pointing out that at the time I gave up on using Travis-CI my project was (locally) passing unit tests and in active use on Fedora 22, 24, Arch Linux and Centos 7. So in one sense the code was “good”. Looking over my git logs I can see that I had 9 commits trying to make it work on Travis-CI before I threw in the towel. The process of reading Travis errors, making a commit that I think will work, waiting for Travis, reading more errors, was too slow and the big reason that I gave up.

Ubuntu

For unrelated reasons I set up an Ubuntu virtual machine on F24 using virt-manager. Having my hands on this environment then allowed me to fix a lot of the errors that Travis-CI was reporting. One of the stranger errors that occurred seems to be a difference in the algorithm that argparse uses to tell if an option starting with a hyphen is actually an option in its own right or if it was meant to be data for a prior option.  For example,

myexe --foo -bar

Is that meant to be a variable called foo is set to “-bar”  or is -bar another option?   A realworld example that I was facing is passing around options for g++

ct-cake --CXXFLAGS=-std=c++11

so the -std is not an option in its own right but a setting for my CXXFLAGS variable. That worked fine on F24 but failed on Ubuntu.

I eventually got around this particular problem by implementing a quote stripping phase in my program.  I now use a command line like

ct-cake --CXXFLAGS="-std=c++11"

Then my python code looks like

cap = configargparse.getArgumentParser()
args = cap.parse_args()
strip_quotes(args)

And the strip_quotes function removes the extraneous double quotes on any of the arguments. See around line 332 in https://github.com/Zomojo/compiletools/blob/master/ct/apptools.py if you are interested in how I implement the quote stripping.

As mentioned earlier there was also the difference in versions of python modules which I took care of by a try statement around the import.

At this point all my unit tests worked on Ubuntu so I thought I’d give Travis-CI another crack.  Unexpectedly, I still failed!

Docker

A bit more learning led me to the fact that the Travis-CI team make available a docker image that you can use to test on.  This answers my previous objection that I was taking too long in the guess solution, commit, push, wait, blowup, cycle.

I installed docker using

sudo dnf install docker-client docker --allowerasing 

and started docker with

sudo systemctl start docker

and proved that I’d got this far correctly via

sudo docker run hello-world

The docker image from Travis-CI came by running

docker run -it -u travis quay.io/travisci/travis-python /bin/bash

switched to the travis user (tip: user name is also the password)

su - travis

checked out my project

mkdir Zomojo
cd Zomojo/
git clone https://github.com/Zomojo/compiletools.git
cd compiletools/

I can’t remember why or from where I got this line of magic

travis compile

Travis-CI does its unittesting in a python virtualenv so we need to activate that

source ~/virtualenv/python2.7/bin/activate

Remember from my earlier .travis.yml I needed some specific environment variables so time to set those

export CXX="g++-5" CC="gcc-5"

And finally we can test and fix quickly

nosetests

Conclusion

Despite the rather large learning hurdles that needed to be vaulted, overall I’d recommend using Travis-CI.  In particular I recommend setting Travis-CI up very early on in your project so that you never reach the point of a wide divergence between tests that pass on your local environment and tests that pass on Travis-CI. Your code will be more robust by having the different environment running it and you are more likely to encounter and fix problems before your users run into them.

 

Tagged with: , , ,
Posted in Uncategorized

systemd –user, a first look

Version

The information in this post was tested on Fedora 22 which reports

$ systemctl --version 
systemd 219

Two Modes

systemd runs in both system mode and user mode.  In a corporate environment this is amazingly useful because it means you can set up your own user services without having to hunt down a system administrator to set up system services.

Hello World as a bash program

In my ~/bin directory I wrote a simple bash hello world program that once every 10 seconds writes hello world to a file in /tmp/.  Make sure you chmod u+x ~/bin/helloworld.

$ cat ~/bin/hello-world 
#!/bin/bash

rm /tmp/hw.txt
while true; do
 DATE=$(date +%Y%m%dT%H%M%S.%N)
 echo $DATE hello world >> /tmp/hw.txt
 sleep 10
done

Unit file

systemd services are specified in “unit files”. User unit files are placed in ~/.config/systemd/user/.  My first mistake toying about with systemd user mode was that I replaced the characters “user” with my username. Those characters really are meant to be “user”.

$ cat ~/.config/systemd/user/hw.service
[Unit]
Description=Hello World

[Service]
ExecStart=/home/geoff/bin/hello-world

[Install]
WantedBy=default.target

Note that the command that is run by ExecStart must be specified as a full path.  If you do something like

ExecStart=~/bin/hello-world

or try to be smart and use the systemd variables like ‘%h’

ExecStart=%h/bin/hello-world

then when you run

$ systemctl --user status hw

you will get errors like

Sep 16 14:48:27 ws002.zomojo.com systemd[25899]: [/home/geoff/.config/systemd/user/hw.service:6] Executable path is not absolute, ignoring: %h/bin/hello-world
Sep 16 14:48:27 ws002.zomojo.com systemd[25899]: hw.service lacks both ExecStart= and ExecStop= setting. Refusing.
Sep 16 15:39:36 ws002.zomojo.com systemd[25899]: [/home/geoff/.config/systemd/user/hw.service:5] Executable path is not absolute, ignoring: ~/bin/hello-world
Sep 16 15:39:36 ws002.zomojo.com systemd[25899]: hw.service lacks both ExecStart= and ExecStop= setting. Refusing.

Interestingly, when you fix those errors by putting the full path in the command line, then rerun systemctl –user status hw, you will still see all the old errors.  Don’t go bananas trying to figure out why systemd has failed to reload your unit file, all is actually well.  You can see all is well by 1) noting that the timestamps on the errors are exactly the same as they previously were and 2) try starting the service.

$ systemctl --user start hw

Watch the service run via

$ tail -f /tmp/hw.txt

And stop the service by

$ systemctl --user stop hw
Tagged with: ,
Posted in Uncategorized

A premade libcec deb

I’ve run across people who are interested in using libcec and would prefer to get started as soon as possible rather than going down the compile it up route.  For those people, you can download gzipped deb files from

http://www.owltra.com/libcec/index.html

Then gunzip the deb file and install

cd Downloads 
gunzip libcec_20150809-1_armhf.deb.gz 
sudo dpkg -i libcec_20150809-1_armhf.deb 
sudo ldconfig

A prerequisite is to have liblockdev1 installed.

sudo apt-get install liblockdev1
Tagged with: , ,
Posted in Uncategorized

Multiple strange errors involving SE Linux with Fedora 22

At work we are in the process of converting our developer workstations over to using Fedora 22.  I’ve had a few rather strange errors

  • SELinux is preventing sshd from read access on the file authorized_keys.
  • webcam not working
  • sound not working

One of the interesting features of our workstations is that we have the the home area under /data/work/<user> rather than /home/<user>. To make a long story short, I found an idea at http://www.geekheads.net/selinux-and-ssh/

Which lead me to

  • edit /etc/selinux/targeted/contexts/files/file_contexts.homedirs
    • copied the /home rules to new /data/work rules
  • ran restorecon -p -R /data/work
  • restarted, so that processes get the new contexts

The result being that webcam, sound, and ssh are now working.

Tagged with: ,
Posted in Uncategorized

CEC 3.0.0 on the Raspberry Pi

In a previous blog I gave some instructions on how to compile up libcec on the RPi. Those no longer work as of CEC 3.0.0. In this post I’ll give you the new instructions that work as at 11 June 2015. To be specific, I have tested these instructions on an RPi2 running Raspbian. I have to say thankyou to nlrb for leaving excellent instructions on the RPi forums.

The first thing to do is update your RPi to the latest packages.

sudo apt-get update && sudo apt-get dist-upgrade
sudo rpi-update
sudo reboot

Next install the prerequisites for building libcec

sudo apt-get install git cmake g++-4.8 checkinstall liblockdev1-dev libudev-dev libxrandr-dev python-dev swig

Clone the Pulse-Eight libcec repository

Be careful to do the clone recursively otherwise you will be missing the “platform” submodule.

cd [workarea]
git clone --recursive https://github.com/Pulse-Eight/libcec.git

Patch src/libcec/cmake/CheckPlatformSupport.cmake

(remove the line with the -sign and add the lines with the +sign. Other lines are just included for context. Around about line 80) with

- check_library_exists(bcm_host vchi_initialise "" HAVE_RPI_API)
+ check_library_exists(bcm_host bcm_host_init "" HAVE_RPI_API)
 if (HAVE_RPI_API)
 set(LIB_INFO "${LIB_INFO}, 'RPi'")
+ include_directories(/opt/vc/include /opt/vc/include/interface/vcos/pthreads /opt/vc/include/interface/vmcs_host/linux)
 list(APPEND CMAKE_REQUIRED_LIBRARIES "vcos")

Create the cec platform library

cd [workarea]/libcec/src/platform
mkdir build
cd build

Note the dot dot at the end of the following line

cmake -DCMAKE_CXX_COMPILER=g++-4.8 ..
make
sudo checkinstall

Finally build libcec

cd [workarea]/libcec
mkdir build
cd build
cmake -DCMAKE_CXX_COMPILER=g++-4.8 -DCMAKE_INSTALL_PREFIX=/usr -DBUILD_SHARED_LIBS=1 ..
make
sudo -s
export LIBRARY_PATH=/opt/vc/lib
checkinstall
Tagged with: , , ,
Posted in Uncategorized

A simple libCEC example for the Raspberry Pi

After getting the libcec test client (cec-client) to run my thoughts turned to incorporating CEC control into my own code.  The source code for cec-client is 1145 lines long, according to sloccount, which takes a bit of time to get your head around. By tossing away almost all of cec-client’s features I have managed to create a 75 sloc example which is much easier to understand. The source code for this really short example can be found on github . 

The basic idea of the program is

  1. Initialise libcec
  2. Give libcec a callback that will handle the CEC keypresses
  3. Go to sleep until ctrl-c finishes the program

Any keys pressed on your remote control then wake up the callback function which I creatively called on_keypress.

Tagged with: , , ,
Posted in Uncategorized

CEC adventures on the Raspberry Pi

OUTDATED

I’ve written an updated version about compiling at CEC 3.0.0 on the Raspberry Pi

If you are just looking for a precompiled version, a-premade-libcec-deb

TL;DR

Many blogs tell you about how wonderfully easy it is to use the Raspberry Pi to interact with CEC. That was not my story. Two issues hit me: 1) the first HDMI cable I used did not have a CEC wire in it. 2) the cec-utils package in the raspbian repository doesn’t work. Cures are get a HDMI cable with the CEC wire and recompile libcec.

Preparation

To prevent failure due to already fixed bugs, I updated the RPi before doing anything else.

sudo apt-get update
sudo apt-get dist-upgrade
sudo rpi-update
sudo reboot

The cec-client program is in the cec-utils package.

sudo apt-get install cec-utils

First Errors

I’ve tested now with the original bad cable and the newly bought good cable, and the cec-client from the raspbian repositories always gives the following error

geoff@devpi:~$ cec-client 
No device type given. Using 'recording device'
CEC Parser created - libCEC version 1.6.2
no serial port given. trying autodetect: DEBUG:   [               2]	trying to autodetect all CEC adapters
FAILED

And similarly with the -l option to list the devices

geoff@devpi:~$ cec-client -l
DEBUG:   [               2]	trying to autodetect all CEC adapters
Found devices: NONE

Building a new cec package

First remove the cec-utils provided by the rasbian repository.

sudo apt-get remove libcec1 cec-utils

The source code for the cec package can be found on github. You get it by

git clone https://github.com/Pulse-Eight/libcec 

Install some extra packages for building. Make sure to install the libudev-dev. I missed that the first time around which caused another whole host of problems.

sudo apt-get install gcc liblockdev1-dev autoconf automake libtool checkinstall libudev-dev

The actual build is pretty easy but be prepared to wait a few minutes.

cd libcec
./bootstrap
./configure --with-rpi-include-path="/opt/vc/include" --with-rpi-lib-path="/opt/vc/lib/" --enable-rpi
make
sudo checkinstall
sudo ldconfig

Using the new cec-client

The newly packaged version cec-client ends up in /usr/local/bin/. With the dodgy cable in place the output looks like

geoff@devpi:~$ /usr/local/bin/cec-client 
No device type given. Using 'recording device'
CEC Parser created - libCEC version 2.1.4
no serial port given. trying autodetect: 
 path:     Raspberry Pi
 com port: RPI

opening a connection to the CEC adapter...
DEBUG:   [             104]	unregistering all CEC clients
DEBUG:   [             107]	Broadcast (F): osd name set to 'Broadcast'
DEBUG:   [             109]	InitHostCEC - vchiq_initialise succeeded
DEBUG:   [             110]	InitHostCEC - vchi_initialise succeeded
DEBUG:   [             111]	InitHostCEC - vchi_connect succeeded
DEBUG:   [             113]	logical address changed to Broadcast (f)
DEBUG:   [             115]	RegisterLogicalAddress - registering address e
DEBUG:   [             296]	logical address changed to Recorder 1 (1)
DEBUG:   [             297]	logical address changed to Free use (e)
DEBUG:   [             298]	Open - vc_cec initialised
NOTICE:  [             299]	connection opened
DEBUG:   [             301]	<< Broadcast (F) -> TV (0): POLL
DEBUG:   [             303]	initiator 'Broadcast' is not supported by the CEC adapter. using 'Free use' instead
TRAFFIC: [             304]	<< e0
DEBUG:   [             305]	processor thread started
DEBUG:   [             395]	command 'POLL' was not acked by the controller

That ‘POLL’ was not acked by the controller is the first indication that all is not well. Not having prior experience with CEC, I wasn’t sure at first if this was a real error or an indication that I had a dodgy implementation of CEC in my TV. After a few hours more of investigation I learnt that not all HDMI cables come with the CEC wire. A trip to my local supermarket and for only AUD$8 obtained a “high speed” HDMI cable that also advertised that it had the Audio Return Channel.

With the new cable in place the equivalent output looks like

ff@devpi:~$ /usr/local/bin/cec-client 
No device type given. Using 'recording device'
CEC Parser created - libCEC version 2.1.4
no serial port given. trying autodetect: 
 path:     Raspberry Pi
 com port: RPI

opening a connection to the CEC adapter...
DEBUG:   [             104]	unregistering all CEC clients
DEBUG:   [             107]	Broadcast (F): osd name set to 'Broadcast'
DEBUG:   [             108]	InitHostCEC - vchiq_initialise succeeded
DEBUG:   [             109]	InitHostCEC - vchi_initialise succeeded
DEBUG:   [             110]	InitHostCEC - vchi_connect succeeded
DEBUG:   [             112]	logical address changed to Broadcast (f)
DEBUG:   [             113]	RegisterLogicalAddress - registering address e
DEBUG:   [             295]	logical address changed to Recorder 1 (1)
DEBUG:   [             296]	logical address changed to Free use (e)
DEBUG:   [             297]	Open - vc_cec initialised
NOTICE:  [             298]	connection opened
DEBUG:   [             300]	<< Broadcast (F) -> TV (0): POLL
DEBUG:   [             302]	initiator 'Broadcast' is not supported by the CEC adapter. using 'Free use' instead
TRAFFIC: [             304]	<< e0 DEBUG:   [             306]	processor thread started DEBUG:   [             335]	>> POLL sent
DEBUG:   [             336]	TV (0): device status changed into 'present'

Pressing buttons on the TV remote now yields a satisfying

TRAFFIC: [            5836]	>> 01:44:02
DEBUG:   [            5837]	>> TV (0) -> Recorder 1 (1): user control pressed (44)
DEBUG:   [            5838]	key pressed: down (2)
DEBUG:   [            5839]	making Recorder 1 (1) the active source
NOTICE:  [            5841]	>> source activated: Recorder 1 (1)
TRAFFIC: [            5906]	>> 01:8b:02
DEBUG:   [            5907]	>> TV (0) -> Recorder 1 (1): vendor remote button up (8B)
DEBUG:   [            5909]	key released: down (2)
Tagged with: , ,
Posted in Uncategorized