Recently we attended ECC2017 conference. One of topics that we considered was a system for development and validation automation. Unfortunately this talk was not accepted, but we present some research below and plan to provide more soon. As maintainers of PC Engines platforms in coreboot we debug and fix quite a lot of bugs, but to take full responsibility for our code everything should be validated each time we do release. Limited resources lead us to automation and as Python enthusiasts we decided to evaluate Robot Framework as first candidate. When preparing to mentioned conference I found that we lack PXE server from which I could run or install needed OSes. Also there was no ready to use configuration that gave ability to utilize diskless boot and try recent Linux. I started to fight with PXE server configuration, but then I realized that without DHCP I have to provide booting information every time typing in iPXE shell or integrate custom script. In long term those were not good solutions.

Robot Framework first try Project by itself seems to be very popular and at first glance is well designed. It gives ability to leverage enormous amount of Python libraries and has integrated most important ones. I decided to start with installation in virtualenv:


Quick start guide

Let’s try something real on PC Engines APU2 My typical output on minicom during boot was:

N for PXE boot show script should send N or n what should send me to iPXE> prompt. Initially I thought about using robotframework-seriallibrary, but limitation led me to search for different solution. robotframework-seriallibrary was designed to handle single byte communication not serial output streams from operating system. For example read_until function check for termination character, instead of matching string pattern, what was expected during iPXE testing. ser2net and telnet solution was suggested on mailing list and eventually was much better choice for my use case. sudo apt-get install ser2net Quick test with config file

ser2net_apu.cfg 13542:telnet:600:/dev/ttyUSB0:115200 8DATABITS NONE 1STOPBIT To prove that

ser2net works correctly: [0:19:39] pietrushnic:pcengines $ telnet localhost 13542 Trying ::1… Connected to localhost. Escape character is ‘^]’. PC Engines apu2 coreboot build 07/18/2017 BIOS version 4080 MB ECC DRAM

Telnet module for Robot Framework After playing some time I got to point when I can enter iPXE command prompt. My test looks pretty simple:

Most complex part was related to pushing arrow keys through terminal. Magic is in

x1b[A what triggers escape sequence matching arrow up key on keyboard. Other keys are: x1b[B – down key x1b[C – right key x1b[D – left key

Debugging pxelinux booting Serial console handling in Robot Framework is not trivial task. Especially, if You are doing it first time. What I learned is that below parameters are critical to correct understand what is going on behind the scene:

  • enable debug log by using -b <file> parameter
  • set debug level -L <level> Command for running framework should look like that:

    robot -b debug.log -L TRACE

iPXE on apu2 I tried to create Robot Framework script, but faced weird issue when trying to send more then one character. I described my findings in

this email. As I wrote it happens that Telnet.write() function is too fast and iPXE cannot handle incoming characters. It took couple hours to figure it out and without debugging output it would not be possible.

Booting pxelinux on apu2 using Robot Framework Finally I managed to boot to iPXE shell and reliably send commands. Next step was to provide an address of PXE server for downloading and booting purpose.

Setup PXE server for apu2 Communication with apu2 goes only through serial console or ssh when service is ready, because of that typical Debian netboot had to be modified. In addition to that I decided to create bigger booting menu, based on netboot package, for various systems so You can see little bit different structure in future. To setup PXE server easy way please follow:

Full Robot Framework script Below is my full script. Please note that I’m using custom method

Write Bare Slow. This is because of flaw related to slow iPXE input. To use this code You can utilize our fork of robotframework. *** settings *** Library Telnet

ser2net port and PXE server IP address are passed through environment variables.

Automated PXE booting Assuming Your PXE server works fine You can run:

<CTRL>-L You should see boot menu: pxe_server_menu Moreover You can investigate in details what happened using automatically generated HTML page: ipxe_test_log

Summary Why bother? Firmware debugging effort consists of tons of repeatable tasks. Lots of them can be automated. Even if debugging is finished and problem was root caused and fixed, we should make sure it will never return. That’s why if we face hacking session with big number of debug-code-test cycle we should think about automation as soon as possible. I tried to build PXE server for apu2 platform many times and always failed running out of time. Typical booting cycle took me ~90s, automated environment does it in 37s. This is not blasting result, but automation gives me solid ground for growing further test case and keep me away from typing mistakes. If You have any comments or problems related to above blog post please let us know we would be glad to help You. If You are trying to automate Your embedded device validation we would be glad to hear more about that.