Research Wiki | Notes / BBBRTakeTwo

Installing and Running PINT on a Beagle Bone Black, Take Two

October 24, 2015

This time around I'm going to focus on building the RT kernel (with patch) on the BBB itself, instead of attempting to create a bootable SD card. Creating the SD card was turning into a time sink (although this could as well).

There are a few tasks to take care of for a new BeagleBone. First of is changing the USB0 ip address (even if they aren't connected at the same time, it prevents ssh key warnings. Just change all of the occurrences of 192.168.7.X to 192.168.8.X in the following http://ewong.me/changing-usb0-ip-address-on-the-beaglebone-black/:

  edit /etc/network/interfaces
  edit /opt/scripts/boot/am335x_evm.sh
  edit /etc/udhcpd.conf

Update passwd, for both root and debian.

Since I now have a few beagble bones, I've put them in cases with stickers to keep them organized. I'm currently working on hamster:192.168.8.2.

I suppose it's time to download, patch, build, and boot a kernel. Luckily I have done this before: RT Patch. Current kernel is 3.8.13-bone47.

uncomment auto eth0 and iface eth0 inet dhcp in /etc/network/interfaces, then run sudo /etc/init.d/networking restart.

  git clone https://github.com/RobertCNelson/bb-kernel.git
  cd bb-kernel
  git checkout am33x-rt-v4.1
  sudo apt-get update
  sudo apt-get install bc lzma lzop libncurses5-dev
  ./build_dep

Remember to select "kernel features --> Preemption Model --> Fully Preemptable Kernel"

Looks like not enough space on the BBB to build there, so doing on my laptop (again!). Not sure if I want build_deb or build_kernel. Trying build_kernel on the BBB now... not enough space.

Instead, I ran build_dep on my laptop (which has the cross compile tools installed). This built a bunch of .deb files in ./deploy/. I used scp to transfer them to the BBB, and then ran dpkg -i linux....dep on each of them. The dev_cross one through some errors... oh well.

uname -a --- Linux beaglebone 3.8.13-bone47 #1 SMP Fri Apr 11 01:36:09 UTC 2014 armv7l GNU/Linux Linux beaglebone 3.8.13-bone47 #1 SMP Fri Apr 11 01:36:09 UTC 2014 armv7l GNU/Linux

That didn't do it. Even though the files showed up in /boot... maybe it's something with uboot?

Okay, I'm looking in /boot/uboot/, and I am terrified to make any changes. Seems like uEnv.txt might be the place... I have a replacement for the kernel file and initrd, although the kernel file is vmlinuz format instead of zImage... and I don't know about u-boot.img... seems important. Sigh. I have no idea what I'm doing here. Maybe I need to rebuild uboot?

Sigh. Bricked it again.

Recovery... Again.

Do not connect the red pin.

Attach the serial cable as shown, and run screen.

  > screen /dev/ttyUSB0 115200

CCCCCCCCCCCCCCCC

That is all that I am getting on the serial connection.

I'm going to download the latest image (http://beagleboard.org/latest-images) and just try to get everything back to normal (http://beagleboard.org/getting-started#update). That worked just fine, so the BBB is back. I'm not going to bother flashing to the eMMC yet (http://elinux.org/Beagleboard:BeagleBoneBlack_Debian#Flashing_eMMC) since I want to try to try getting an RT kernel again.

Okay, I take that back. After faffing about for a few hours trying to create a new card from scratch, I decided to re-flash the eMMC with the default debian image. Now I'm going to try scp the .deb files over, installing, and them editing uEnv.txt.

I can't believe it. That worked. Just booted into the right kernel. Everything is perfect! Except that the USB serial connection isn't working... But the regular serial connection is sufficient. That took me way too long to figure out.

Installing software.

Let's see, I need PINT and Player. I don't think I actually need Stage.

  > sudo apt-get update
  > sudo apt-get install subversion cmake-curses-gui

Player. My laptop is using version 3.1.0, not sure why. So I think I need to pull from the svn repo and build.

  > svn checkout svn://svn.code.sf.net/p/playerstage/svn/code/player/branches/release-3-1-patches player
  > cd player; mkdir build; cd build
  > ccmake ../

A lot of drivers aren't necessary, and compilation will be much quicker without them. Pretty sure that some things are necessary... (lasertoranger, vmapfile, ?) might need to get SWIG for bindings...

  > make; sudo make install
  > player

If at this point you receive the error: "player: error while loading shared libraries: libplayerdrivers.so.3.1: cannot open shared object file: No such file or directory" then you need to update LD_LIBRARY_PATH to include the location of the Player libraries.

  > echo "export LD_LIBRARY_PATH=$LD_LIBRARY_PATH:/usr/local/lib" >> ~/.bashrc
  > exit

Log back in (or open a new terminal), and try running player. If all went well, then you should get a help message.

PINT

  > git clone https://github.com/jcmarsh/PINT.git

Now need to install libccv if you want to run the Load component. If not, you are going to have to edit the Makefiles to exclude Load.

  > wget https://github.com/liuliu/ccv/tarball/stable
  > tar -xf stable
  > rm stable
  > mv liuliu-ccv-785aba2 ccv (filename may have changed)
  > cd ccv/lib/
  > ./configure - check to see if anything you want is missing (libpng and libjpeg in the LINK FLAGS bit at the end)
  > make

CCV seems to give me no end of troubles when it comes to memory. Trying to make runs out of memory:

  gcc ccv_numeric.c -o ccv_numeric.o -c -O3 -ffast-math -Wall -D HAVE_LIBPNG -D HAVE_LIBJPEG -D HAVE_AVCODEC -D HAVE_AVFORMAT -D HAVE_SWSCALE -I/usr/local/include
  ccv_numeric.c: In function ‘ccv_filter’:   
  ccv_numeric.c:944:3: warning: ‘end_tile_x’ may be used uninitialized in this function [-Wuninitialized]
  ccv_numeric.c:944:3: warning: ‘end_tile_y’ may be used uninitialized in this function [-Wuninitialized]
  ccv_numeric.c:944:3: warning: ‘end_tile_x’ may be used uninitialized in this function [-Wuninitialized]
  ccv_numeric.c:944:3: warning: ‘end_tile_y’ may be used uninitialized in this function [-Wuninitialized]
  ... Many identical seeming warnings ...
  [ 3932.187201] Out of memory: Kill process 3213 (cc1) score 783 or sacrifice child
  [ 3932.204849] Killed process 3213 (cc1) total-vm:408108kB, anon-rss:395716kB, file-rss:4kB

I guess I can either add a swap file on an SD card to provide more memory... but this may just lead to thrashing. The number of identical warnings is troublesome... could there be an include loop?

Mount an SD card for extra memory (http://elinux.org/Beagleboard:MicroSD_As_Extra_Storage):

Use disk utility to format the SD card to use as FAT.
Add a file uEnv.txt to the partition with the following:

    mmcdev=1
    bootpart=1:2
    mmcroot=/dev/mmcblk1p2 ro
    optargs=quiet

Power down BBB, add sd card, power on BBB, and check /deb/mmcblk1p2 for the device, and also /media/<Partition Label>.

Add a swap file on the SD card (https://sheldondwill.wordpress.com/2013/12/14/beaglebone-black-ubuntu-adding-a-swapfile/):

  > cd /media/<Partition Label>
  > sudo mkdir swap
  > sudo dd if=/dev/zero of=./swap/swapfile bs=1M count=512
  > sudo chmod 0600 ./swap/swapfile 
  > sudo mkswap ./swap/swapfile
  > sudo swapon ./swap/swapfile
  > free -m <- if it worked, last like should show available swap: Swap:          511          0        511

I skipped the step to make the swap file permanent, since I only need it for compiling software right now. Now try make again. ARE YOU KIDDING ME? 1GB isn't enough. The process got further along... but gcc ccv_resample.c -o ccv_resample.o -c -O3 -ffast-math -Wall -D HAVE_LIBPNG -D HAVE_LIBJPEG -D HAVE_AVCODEC -D HAVE_AVFORMAT -D HAVE_SWSCALE -I/usr/local/include runs out of memory. I guess I can make a larger swap file? This seems ridiculous. The other option is to cross compile.

  > df -h <- I needed to check how much space is really available on my 1GB SD card.
  > sudo dd if=/dev/zero of=./swap/swapfile bs=1M count=940

Redo steps from above. Hopefully make finishes this time. That worked. Then I got an error about some vgg sample... so I'm just commenting out that part of the makefile: all: libccv.a #../samples/image-net-2012-vgg-d.sqlite3

  > make
  > sudo cp libccv.a /usr/local/lib/

I'm going to need to check about the installation paths... also, I think the easiest way to get set up is to clone my Stage branch, even though I won't be running Stage on the BBB... or will I? I shouldn't have to, but I think the way I have things set up is convoluted and needs it.

Oh right... I have assembly code in PINT. Which means is needs to be changed now that I'm trying to port to an ARM platform.

Back to the Kernel

Looks like I need to enable user mode performance counters. The only part of the PINT code that is assemble, is the bit that is used to track cpu usage. Or rather, how many cycles have elapsed. The code.

This time I should make better notes about what I did in case anyone else wants to try.

Actually... there isn't a kernel option for this. Instead, it must be enabled while in kernel mode, which likely means a kernel module is the easiest fix. SO on the topic: http://stackoverflow.com/a/3250835/1601162. This blog post references the SO answer, and also mentions perf_event as a solution, but perhaps too heavy (system calls involved) and not accurate enough (for small code segments) for my needs http://neocontra.blogspot.com/2013/05/user-mode-performance-counters-for.html. Then again, maybe not...

Kernel Module

The pain about the kernel module is that you need to have the kernel source (or maybe just headers?) to be able to build it. For the BBB, the kernel had to be patched, and cross-compiled. Luckily I found the nifty script to take care of all of the patching... but it also starts the build from scratch each time.

Used https://github.com/thoughtpolice/enable_arm_pmu.git, the ko directory, edited Makefile to:

  export ARCH:=arm
  export CROSS_COMPILE:=arm-linux-gnueabi-
  obj-m := enable_arm_pmu.o
  KDIR := /home/jcmarsh/research/bbb/bb-kernel/deploy/modules/lib/modules/4.1.13-bone-rt-r16/build
  PWD := $(shell pwd)

  all:
        $(MAKE) -C $(KDIR) SUBDIRS=$(PWD) modules
  clean:
        $(MAKE) -C $(KDIR) SUBDIRS=$(PWD) clean

Now I just need to transfer the .ko file to the BBB, insert the module, and try out the test code https://github.com/jcmarsh/Research_Notes/tree/master/misc_test/c_test/cycle_count.

I used scp, over the shared ethernet connection with my laptop. I somehow managed to kill the usb internet connection.

  scp enable_arm_pmu.ko debian@10.42.0.27:~/

On the BBB:

  > gcc -o p15_test p15.c 
  > sudo insmod enable_arm_pmu.ko
  > ./p15_test 
      Test program for cycle count on ARM using p15
      Looped 10000 times in 7055808 cycles.
  > sudo rmmod enable_arm_pmu
  > ./p15_test
      Test program for cycle count on ARM using p15
      Can't read from user mode
      Can't read from user mode
      Looped 10000 times in 0 cycles.

Finally! The BBB should be set for using as a test platform! Now if only that USB connection was working...

CPU Frequency Scaling

I ran cat /proc/cpuinfo to check the processor frequency, which doesn't actually have a MHz field. But there is a BogoMIPS field which reads BogoMIPS : 298.84. I'd never heard of BogoMIPS before, but they roughly equate to MHz.

So I'm running at 300MHz. The chip should be able to run at 1000MHz! Looks like frequency scaling strikes again! Which makes complete sense on an embedded platform where power usage is essential. But my timing code assumes a set speed.

I stumbled across this useful post (I think I've cited this person's posts before). It shows how to check the frequency scaling settings using cpufrequtils, which is already install on my BBB (but not laptop...).

  > cpufreq-info
  > sudo cpufreq-set -f 800MHz     /* Set to 800MHz */
  > cpufreq-info                   /* Should show governor as "userspace" */
  > sudo cpufreq-set -g powersave  /* Set back to powersave scaling */

There is a way this at boot time, but I'm just going to add it to a pre-run script since the kernel module needs to be inserted as well. That should probably make it into the repo.

February 20, 2016

Update for Anon_VMA fix

It looks like the kernel I'm using on the BBB (4.1.13-bone-rt-r16) has a fix for the anon_vma problem. Comment from anon_vma_clone in mm/rmap.c:

  /*                                                                                                                                                     
   * Attach the anon_vmas from src to dst.                                                                                                               
   * Returns 0 on success, -ENOMEM on failure.                                                                                                           
   *                                                                                                                                                     
   * If dst->anon_vma is NULL this function tries to find and reuse existing                                                                             
   * anon_vma which has no vmas and only one child anon_vma. This prevents                                                                               
   * degradation of anon_vma hierarchy to endless linear chain in case of                                                                                
   * constantly forking task. On the other hand, an anon_vma with more than one                                                                          
   * child isn't reused even if there was no alive vma, thus rmap walker has a                                                                           
   * good chance of avoiding scanning the whole hierarchy when it searches where                                                                         
   * page is mapped.                                                                                                                                     
   */

Need to confirm by running this test code (Daniel Frost, same as patch https://lkml.org/lkml/2012/8/15/765), and checking for growth in /proc/slabinfo anon_vma.

This program wasn't really producing the expected results. I think maybe I am not waiting long enough. Regardless, I'm better off checking by running https://github.com/jcmarsh/PINT/blob/master/test/micro_test/empty_restart.c, which evaluates component restart time (which is how the problem was initially discovered).

  #include <unistd.h>

  int main(int argc, char *argv[])
  {
    pid_t pid;
    while (1) {
      pid = fork();
      if (pid == -1) {
        /* error */
        return 1;
      }
      if (pid) {
        /* parent */
        sleep(2);
        break;
      }
      else {
        /* child */
        sleep(1);
      }
    }
    return 0;
  }