My apologies for the long wait since the last post. I started a PhD in AI and Robotics at Cambridge University, which has been absorbing all my spare time. This post is actually quite old, but I never got around to publishing it. The good news is that I’ve learned a ton about robotics, biomimetics, ROS, MoveIt, deep learning, the Jetson TX1 and more, all of which I hope to share in due course. My apologies also for comments that waited months to be approved. When I finally dragged myself back to comments section I found only 4 real comments out of more than 600 bits of spam.
This rather specific post grew out of a frustrating attempt to write a fairly complex ROS package in Python which was continually stymied by seemingly random import errors when my node was launched in different ways. In fairness, I am neither an experienced Python programmer nor a ROS expert, so my problems were due to ignorance. However, clearing up that ignorance was non-trivial and led me down one rabbit hole in terms of how Python manages packages and another rabbit hole in terms of how ROS does it.
This post is intended to be a shortcut for novice ROS developers who want to develop a Python node that involves more than one source file and imports Python libraries.
My earlier ignorance (at least on this subject) can be summarised as:
- I didn’t really understand how Python’s import statement worked
- I didn’t know how to structure the source files in a Python-based ROS package
- I didn’t really know what the purpose of Catkin was for Python-based ROS packages
- I didn’t know what boilerplate needed to be applied so that Catkin could deal with my ROS package
- I didn’t know how to declare dependencies in my ROS package
Why we structure the source files in a Python-based ROS package the way we do
Python-based ROS packages will generally contain nodes, scripts and launch files.
Launch files are a set of instructions to launch one or more ROS nodes as well as the ROS master if it isn’t already running.
Nodes and Scripts
The difference between nodes and scripts is largely semantic: they are both Python files that can be executed from the command line with
rosrun <package name> <node or script name>
Nodes are assumed to create and launch ROS Nodes, whereas scripts can be used for any purpose. This distinction is a convention to make our packages easier to understand, not a technical difference.
For convenience we put nodes and scripts into folders called <base dir>/nodes and <base dir>/scripts respectively, although this isn’t strictly necessary. They should be given execute permissions with chmod
cd <base dir> chmod +x nodes/mynode1 chmod +x scripts/myscript1
For aesthetics, mynode1 and myscript1 won’t have a .py extension (which means the first line of the file must be labelled with #!/usr/bin/env python). The user can then use
rosrun mypackage mynode1
and not the messier
rosrun mypackage mynode1.py
You can put your entire code in either of these files and move on. That’s enough to get to hello world:
#!/usr/bin/env python
if __name__== '__main__':
print "hello world"
However, it is more likely that you want to break down any non-trivial Python code into various separate files (known as “modules” in Python land). The best way to do this is to keep your nodes and scripts as short as possible and have them simply import and execute the main() function from your real code. For example, mynode1 might contain:
#!/usr/bin/env python from mypackage import main if __name__== '__main__': main()
Why bother with this extra layer of files if the real work is being done elsewhere? The ROS command rosrun hooks into nodes and scripts, as does roslaunch. But we may also want to import our ROS package functionality into a second ROS package. This is done by declaring Catkin dependencies in the second ROS package (for building purposes) and then using standard Python mechanisms for importing the code from the first package into the second one. The “Python mechanisms” bit is important, as it means we should structure our importable code as a Python package, not to be confused with a ROS package!
Now let’s turn to where we put the bulk of our code.
Bundling our code into a Python package
So our ROS package needs to contain a Python package with the bulk of our code, at least if we want to make it importable into other ROS packages. The conventional way to do that is to put the code into a directory called
src/<Python package name>
By convention, the Python package name and the ROS Package name are the same, even if strictly speaking they don’t need to be. So if our ROS Package is called mypackage, then put the bulk of your Python code in
src/mypackage
The import statement in our nodes and scripts will look like this:
from mypackage import main
When Python comes across an import statement of this sort, it assumes that mypackage refers to either a single file called mypackage.py (a “module” in Python land) or a directory containing an __init__.py file and optionally other Python files (collectively known as a “Python package”). We’re going to be using the latter.
So at a minimum, we need a file called
src/mypackage/__init__.py
This can contain all our code if we want, including the main() function called from our nodes and scripts. More likely, we’ll want to split our code into different files with descriptive names. So let’s put main() into src/mypackage/foo.py and have __init__.py contain the following:
from foo import main
This will then be executed whenever a node or script file says
from mypackage import main
So main() from foo.py gets imported into the __init__.py file within mypackage, where it can be re-imported into the node and script files. foo.py itself can look something like this:
#!/usr/bin/env python
# ... The bulk of our code can go here ...
def main():
print "Now executing main() in foo.py"
# Call the bulk of our code...
All done, right? Not quite. We still haven’t told the node and script files where to find the Python package mypackage. Remember that Python will look in the current directory for a module or Python package to import, or on PYTHONPATH. But the directory mypackage is <base dir>/src which is not immediately accessible. Prepare to enter a rabbit hole.
How does a node or script file actually import our main code?
Python looks for modules or Python packages in the current directory or on the path specified by the environment variable PYTHONPATH. We shouldn’t manipulate PYTHONPATH ourselves, but get Catkin to do it for us. To do that, we need to configure a file called setup.py in <base dir>.
setup.py is a standard Python file used for creating a distributable, installable chunks of code (I refuse to use the word “package” with yet another meaning). This process is handled by a Python tool suite called distutils which is documented here, for those that are interested. For our purposes, we simply need to use setup.py to tell Catkin the name of our Python package (“mypackage”) and where it is located (in the directory “src”). Here is an example of setup.py:
#!/usr/bin/env python
from distutils.core import setup
from catkin_pkg.python_setup import generate_distutils_setup
setup_args = generate_distutils_setup(
packages=['mypackage'],
package_dir={'': 'src'}
)
setup(**setup_args)
All you need to do is customise the line packages=… The rest is boilerplate. For those interested in what the boilerplate does: it gets called by Catkin during the catkin_make process, examines package.xml for meta-data like author, license etc, adds the defined packages and package_dir and passes the combined set of information to distutils which does the actual installation into the Catkin development workspace.
To make all this work, there are two more tweaks to do in Catkin’s CMakeLists.txt file.
First, make sure this line is uncommented:
catkin_python_setup()
This tells Catkin to pay attention during the catkin_make process to the setup.py file we just configured.
Second, in the install() function we need to list the node and script files we defined right at the beginning. So:
install(PROGRAMS
scripts/myscript1
nodes/mynode1
nodes/mynode2
DESTINATION ${CATKIN_PACKAGE_BIN_DESTINATION}
)
This tells Catkin to install our executable files into an accessible location in the Catkin development workspace.
Once setup.py and CMakeLists.txt are configured, execute catkin_make:
cd ~/catkin_ws
catkin_make
and you should be able to execute your nodes and scripts, and therefore your code in src/mypackage/foo.py from rosrun:
rosrun my package mynode1
You can pretty much stop here. Add Python files (“modules”) like bar.py, baz.py to src/mypackage and have them import functions into each other with
from bar import fn1, fn2 from baz import fn3, fn4
For Python code of medium complexity that’s probably enough to add some decent structure to your code. Gluttons for punishment may want to further divide the code into sub-packages. Let’s have a look at how to do that.
Adding Sub-packages to our main Python Package
You can add further Python packages underneath our main Python package in src/mypackage. Each subdirectory takes the name of the sub-package and must include an __init_.py file and optionally, some other Python files.
src/mypackage/subpackage/__init__.py src/mypackage/subpackage/something.py
As usual, we could shoehorn all our code into __init__.py, but we’ll likely want to have it into separate files like something.py.
So our sub-package __init__.py might contain:
from something import *
In our main Python package we then import code using the standard Python mechanisms:
from subpackage import fn5, fn6
If also we want to make these Python sub-packages available to import into other ROS packages, we would need to add them to packages=… in setup.py. If they’re purely used internally by our code, then there’s no need.
A Working Example you can download
Here’s a an example I threw together as an illustration, with one node, one script, one launch file, the main Python package and one sub-package.
You can see the code here https://github.com/SimonBirrell/simontest
rosrun simontest simon_node1
Check out the code, paying attention to setup.py and the various __init__.py files. Figuring out what is being called when and how should give you a solid grounding for building robust and complex Python applications.
If you want to download, build it and run the commands yourself:
cd ~/catkin_ws/src git clone https://github.com/SimonBirrell/simontest cd .. catkin_make
Defining Dependencies for our Python-based ROS Package
All the above is sufficient for ROS Packages that don’t call external Python libraries. Generally though, you will be importing and using other Python libraries such as numpy. We can simply install these packages on our development machine using pip, apt-get or some other method and then import them as usual, with
import numpy
The only problem with this approach is when you come to distribute your ROS Package to other users to install their machines. You will have to include instructions to ask them to install the same required Python libraries before they run your ROS Package. It is easy to forget what you have installed on your own development machine, so a better solution is to explicitly define the dependencies, so that Catkin will install them if required when users build your ROS Package.
As far as I can tell, this needs to be done twice, as once would be too easy.
Defining Standard Dependencies
First, you need to define the dependencies the Python level, by informing distutils of the Python packages you require. This is done through setup.py in the <base dir>:
#!/usr/bin/env python
from distutils.core import setup
from catkin_pkg.python_setup import generate_distutils_setup
setup_args = generate_distutils_setup(
packages=['my package'],
package_dir={'': 'src'},
install_requires=[‘required_python_package1’, 'required_python_package2’]
)
The dependencies are listed in the line beginning install_requires=… The names used are the Python names, i.e. the names you would use if you were installing manually with pip.
sudo pip install required_python_package1
The second place you need to define the dependencies is in package.xml. Why on earth do we need to define them twice?
The reason is that Catkin has its own set of names for Python packages that is separate from the Python names. Python dependencies are defined in package.xml with a tag like this:
<exec_depend>ros-python-dependency</exec_depend>
where the name ros-python-dependency comes from the rosdistro list. As an example, on a recent project I used the Python web sockets library ws4py, which is installed manually like this:
sudo pip install ws4py
Searching for ws4py on the rosdistro list we find:
python-ws4py-pip: ubuntu: pip: packages: [ws4py]
The final line references the install target for pip. So we can use the first line for our dependency name in package.xml:
<run_depend>python-ws4py-pip</run_depend>
Once this is added, your ROS Package should be installable by Catkin on other peoples’ machines and the dependencies should be automatically installed and built as required.
The rosdistro list includes most commonly-used Python packages. Occasionally though, you will have some unusual Python dependency.
I should say at this point that the simplest solution is just to ask your users to manually install the non-standard Python dependency, with pip or some other installer. If you want full automation though, you’re going to have to fork the rosdistro list, add the non-standard package and submit a pull request. Assuming your request is accepted, the next people to access the rosdistro list will be able to automatically install the dependency.
There are perfectly adequate instructions for doing this here.
Tip: Testing your Dependency Definitions
It’s typically tough to test whether the various changes above work. As soon as you test once, your machine has changed state; the packages are now installed. Ideally, each test would start off with a pristine ROS installation, then install your ROS Package, run catkin_make and then test the code itself.
This is now quite easy to do using one of the continuous integration services like TravisCI or CircleCI. I use the latter; sign up is quick and usage is free for the sort of workloads you’ll likely be doing.
For example, I pointed CircleCI to my ROS Package’s GitHub repo and added the following circle.yaml file in <base dir>:
dependencies: pre: - sudo sh -c 'echo "deb http://packages.ros.org/ros/ubuntu $(lsb_release -sc) main" > /etc/apt/sources.list.d/ros-latest.list' - sudo apt-key adv --keyserver hkp://ha.pool.sks-keyservers.net --recv-key 0xB01FA116 - sudo apt-get update - sudo apt-get install ros-indigo-ros-base - sudo rosdep init - rosdep update - echo "source /opt/ros/indigo/setup.bash" >> ~/.bashrc - mkdir -p ~/catkin_ws/src && cd ~/catkin_ws/src && catkin_init_workspace - pip install 'gevent==1.0.2' - cd ~/catkin_ws/ && catkin_make
This YAML file tells CircleCI to run a set of instructions before executing the automated tests. These instructions are essentially the standard ROS installation instructions until the line
- mkdir -p ~/catkin_ws/src && cd ~/catkin_ws/src && catkin_init_workspace
This line creates the Catkin workspace. Again, this is standard ROS.
- pip install 'gevent==1.0.2'
This installs gevent, which you’ll recall was a package that wasn’t available in the rosdistro list.
- cd ~/catkin_ws/ && catkin_make
This line runs catkin_make, which will build your ROS package.
Of course, you will ideally now have some tests (perhaps with nose) that run through your code and do the various imports. CircleCI should be configured to run these tests.
If you don’t have tests, you could just do a rosrun command at the end of this section of the circle.yaml file. If it works, fine. If not, CircleCI will show you an error.
So with this, you can be confident that your dependencies will be installed correctly on other people’s machines. At least as confident as you can ever be in Linux Hell.
To conclude, I should say that the above represents the best of my knowledge based on trudging through ROS Answers and StackOverflow. I’m very happy to be corrected by any ROS gurus out there.
Gheorghe 
Alborz Jelvani 
Oleg 
Nikhil