Recently we attended ECC2017 conference. One of topics that we considered was a system for development and validation automation. Unfortunately this talk was not accepted, but we present some research below and plan to provide more soon.

As maintainers of PC Engines platforms in coreboot we debug and fix quite a lot of bugs, but to take full responsibility for our code everything should be validated each time we do release. Limited resources lead us to automation and as Python enthusiasts we decided to evaluate Robot Framework as first candidate.

When preparing to mentioned conference I found that we lack PXE server from which I could run or install needed OSes. Also there was no ready to use configuration that gave ability to utilize diskless boot and try recent Linux. I started to fight with PXE server configuration, but then I realized that without DHCP I have to provide booting information every time typing in iPXE shell or integrate custom script. In long term those were not good solutions.

Robot Framework first try

Project by itself seems to be very popular and at first glance is well designed. It gives ability to leverage enormous amount of Python libraries and has integrated most important ones. I decided to start with installation in virtualenv:

Verification:

Quick start guide

Output should look like this:

What is great about that? Output is clean and can be easily understood. In addition it generates log and report. Both generated files are clean and eye-catching.

Let’s try something real on PC Engines APU2

My typical output on minicom during boot was:

After N for PXE boot show script should send N or n what should send me to iPXE> prompt.

Initially I thought about using robotframework-seriallibrary, but limitation led me to search for different solution. robotframework-seriallibrary was designed to handle single byte communication not serial output streams from operating system. For example read_until function check for termination character, instead of matching string pattern, what was expected during iPXE testing.

ser2net and telnet solution was suggested on mailing list and eventually was much better choice for my use case.

Quick test with config file ser2net_apu.cfg

To prove that ser2net works correctly:

Telnet module for Robot Framework

After playing some time I got to point when I can enter iPXE command prompt. My test looks pretty simple:

Most complex part was related to pushing arrow keys through terminal. Magic is in x1b[A what triggers escape sequence matching arrow up key on keyboard. Other keys are:

Debugging pxelinux booting

Serial console handling in Robot Framework is not trivial task. Especially, if You are doing it first time. What I learned is that below parameters are critical to correct understand what is going on behind the scene:

  • enable debug log by using -b <file> parameter
  • set debug level -L <level>

Command for running framework should look like that:

iPXE on apu2

I tried to create Robot Framework script, but faced weird issue when trying to send more then one character. I described my findings in this email. As I wrote it happens that Telnet.write() function is too fast and iPXE cannot handle incoming characters. It took couple hours to figure it out and without debugging output it would not be possible.

Booting pxelinux on apu2 using Robot Framework

Finally I managed to boot to iPXE shell and reliably send commands. Next step was to provide an address of PXE server for downloading and booting purpose.

Setup PXE server for apu2

Communication with apu2 goes only through serial console or ssh when service is ready, because of that typical Debian netboot had to be modified. In addition to that I decided to create bigger booting menu, based on netboot package, for various systems so You can see little bit different structure in future.

To setup PXE server easy way please follow:

At point of writing this blog post support was very limited and menu had just Debian i386 installer.

Full Robot Framework script

Below is my full script. Please note that I’m using custom method Write Bare Slow. This is because of flaw related to slow iPXE input. To use this code You can utilize our fork of robotframework.

Configuration for ser2net port and PXE server IP address are passed through environment variables.

Automated PXE booting

Assuming Your PXE server works fine You can run:

Then please change port and IP address accordingly:

Please note that port is hardcoded in ser2net_apu.cfg.

Output in terminal should look like this:

This means PXE boot menu is already in Your telnet, to connect to it simply type:

After refreshing Your screen with <CTRL>-L You should see boot menu:

pxe_server_menu

Moreover You can investigate in details what happened using automatically generated HTML page:

ipxe_test_log

Summary

Why bother? Firmware debugging effort consists of tons of repeatable tasks. Lots of them can be automated. Even if debugging is finished and problem was root caused and fixed, we should make sure it will never return. That’s why if we face hacking session with big number of debug-code-test cycle we should think about automation as soon as possible.

I tried to build PXE server for apu2 platform many times and always failed running out of time. Typical booting cycle took me ~90s, automated environment does it in 37s. This is not blasting result, but automation gives me solid ground for growing further test case and keep me away from typing mistakes.

If You have any comments or problems related to above blog post please let us know we would be glad to help You. If You are trying to automate Your embedded device validation we would be glad to hear more about that.