Sunday, 24 March 2019

Smooth streaming video from a raspberry pi camera


Here's how to set up a smooth streaming video webcam from a raspberry pi. In my case I have a raspberry pi zero w with the pi camera which streams a feed from a window.

Prerequisites:
  • Raspberry pi (zero, zero w, B, whatever - they all do the heavy h.264 encoding, unsure on A)
  • Raspberry pi camera
  • raspbivid installed and working (with raspi-config). Not covered here.
  • Network connection to raspberry pi

What we'll be doing is:
  • Compiling FFMPEG from source
  • Compiling NGINX with the rtmp module
  • Creating a simple index file to load a javascript hls library
  • Creating a script to start the streaming
  • Systemd units to keep everything going

In part 2, I'll show how to stream this via an external proxy.

FFMPEG

You'll now need to compile ffmpeg on the pi
# install libx246 dev tools
sudo apt-get install build-essential libx264-dev
# Get a copy of the latest ffmpeg
git clone git://source.ffmpeg.org/ffmpeg.git
cd ffmpeg/
# Configure ffmpeg with x246 and non-free codecs
./configure --enable-gpl --enable-nonfree --enable-libx264
# Make and install
make && sudo make install

Nginx with rtmp module

Since Nginx doesn't natively support the rtmp protocol, you have to compile it in.

Here's how:
# Download and install development tools
sudo apt-get install build-essential libpcre3 libpcre3-dev libssl-dev
# Get the rtmp module
git clone git://github.com/arut/nginx-rtmp-module.git
# Get the nginx source (at the time this was 1.14.1, review your versions...)
wget http://nginx.org/download/nginx-1.14.1.tar.gz
# Extract the nginx source and go in to the directory
tar xvzf nginx-1.14.1.tar.gz && cd nginx-1.14.1
# Configure nginx with defaults + ssl + rtmp module
./configure --with-http_ssl_module --add-module=../nginx-rtmp-module --
with-cc-opt=-Wno-error
# Compile, and install
make && sudo make install

Next create a directory to store the rtmp content in for streaming
sudo mkdir /webcam
sudo chown nobody: /webcam

Create an nginx configuration that uses the rtmp module in /usr/local/nginx/conf/nginx.conf:
worker_processes 4;
pid /run/nginx.pid;
error_log  logs/error.log debug;

events {
  worker_connections  512;
}

http {
  include mime.types;
  #default_type application/octet-stream;
  sendfile off;
  keepalive_timeout 65;

  server {
    listen 80;

    root /webcam/;

    location / {
      rewrite ^/webcam/(.*) /$1; # Allows http://site/ or http://site/webcam/
      index index.html;
      add_header Cache-Control no-cache;
      add_header 'Access-Control-Allow-Origin' '*';
      types {
        application/vnd.apple.mpegurl m3u8;
        video/mp2t ts;
        text/html html;
      }
    }
  }

}

rtmp {
  server {
    listen 1935;
    ping 30s;
    notify_method get;
    application video {
      live on;           # Enable live streaming
      meta copy;
      hls on;            # Enable HLS output
      hls_path /webcam; # Where to write HLS files
    }
  }
}

In summary of above:
- Users will connect to http://host/webcam/index.html and this will load a javascript library that loads the stream (next section)
- The rtmp stream will be published from ffmpeg to http://host/video/streamname, it will be served out at http://host/webcam/streamname

HTML file to load stream

Create an /webcam/index.html file that loads a public hls streaming javascript library - this will give you the streaming video interface. Make sure to update the URL to where you are hosting your stream:
<script src="https://cdn.jsdelivr.net/npm/hls.js@latest"></script>
  <video autoplay="" controls="" id="video"></video>
  <script>
    if (Hls.isSupported()) {
      var video = document.getElementById('video');
      var hls = new Hls();
      // bind them together
      hls.attachMedia(video);
      hls.on(Hls.Events.MEDIA_ATTACHED, function () {
        console.log("video and hls.js are now bound together !");
        hls.loadSource("http://raspberrypi.local/webcam/stream.m3u8");
        hls.on(Hls.Events.MANIFEST_PARSED, function (event, data) {
          console.log("manifest loaded, found " + data.levels.length + " quality level");
        });
        video.play();
      });
    }
</script>

Script to start the stream

Next part is to create a script that will launch rasbpivid to do the encoding, then feed that in to ffmpeg to do the rtmp handling and point that output at nginx.

I've put this script in /home/pi/webcam.sh (remember to chmod a+x this file)
#!/bin/bash
/usr/bin/raspivid -o - -t 0 -b 1000000 -w 1280 -h 720 -g 50 | \
/usr/local/bin/ffmpeg -i - -vcodec copy -map 0:0 -strict experimental \
-f flv rtmp://127.0.0.1/video/stream

Quick summary of those commands:
- Start raspivid
- output to STDOUT (-o -)
- Do it forever (-t 0)
- Set the bitrate to 1000000 bps, close enough to 1Mbit for jazz (-b 1000000)
- Set the width and height of the video to 1280x720 (-w 1280 -h 720)
- Set the GOP length to 50 frames (this means the image is completely refreshed every 50 frames)
- Pipe this output in to ffmpeg (| /usr/local/bin/ffmpeg)
- Tell ffmpeg to get input from STDIN
- Set the video codec to copy
- Map stream 0 of the input to to stream 0 of the output
- Enable some experimental features
- Format to flv
- Set the output to the nginx server with the rtmp module.


Use Systemd to keep everything going

Create a systemd service that will start the webcam, after nginx has started:
in /etc/systemd/system/webcam.service
[Unit]
Description=webcam
After=nginx.service
After=systemd-user-sessions.service
After=rc-local.service
Before=getty.target


[Service]
ExecStart=/home/pi/webcam.sh
Type=simple
Restart=always
User=pi
Group=pi

[Install]
WantedBy=multi-user.target

Create an nginx systemd service to start the local install of nginx in /etc/systemd/system/nginx.service:
[Unit]
Description=The NGINX HTTP and reverse proxy server
After=syslog.target network.target remote-fs.target nss-lookup.target

[Service]
Type=forking
PIDFile=/run/nginx.pid
ExecStartPre=/usr/local/nginx/sbin/nginx -t
ExecStart=/usr/local/nginx/sbin/nginx
ExecReload=/usr/local/nginx/sbin/nginx -s reload
ExecStop=/bin/kill -s QUIT $MAINPID
PrivateTmp=true

[Install]
WantedBy=multi-user.target

Start everything up

sudo systemctl daemon-reload
sudo systemctl enable nginx.service
sudo systemctl enable webcam.service
sudo systemctl start nginx.service
sudo systemctl start webcam.service

Now, connect your browser to http://raspberrypi.local/webcam/ and you should see your video stream which will look something like this:

Note that this is not live. It's a sample captured from the camera and hosted elsewhere.

Tuesday, 16 May 2017

Finding deleted files and summing all numbers in a column


Here are two handy things, which can be combined in to one!

First, is there a mismatch between what du says is used on disk and what df is reporting as used? Well you probably have some processes holding open deleted files, the links will be removed so du will be unable to sum their size, but since they're still allocated on disk then df will report it as used space.

You often see this with log files where someone has gone in to clean up files due to a disk space alert, but the process is still writing to those files. As an aside, it's better avoid this whole situation and use >logfile.name to truncate a file rather than rm to delete the file on a process you're unable to restart right away.

Point one, find the deleted files, you can do this easily with:

sudo lsof | grep deleted

Also if you know which process is holding open the files then you can use the proc filesystem and look in the processes fd directory, the links will have (deleted) after their names if they are removed. Historical tip: The flash plugin used to protect streaming videos (such as youtube) by saving the file in /tmp and then deleting the file. You could just go in to proc and retrieve it if you like, that's changed now of course with HTML5 and DRM. Anyway back on topic.

The second part of this is: now I have the list of deleted files how do I sum up all the size values to see how much disk space is used?

Here you can use awk to extract column, paste to remove the newline and put '+' in it's place (awk can probably do this, but this is easier for me to remember), then pipe that in to bc to get a total:

sudo lsof | grep deleted | awk '{ print $7}' | paste -sd+ | bc

Easy. That number should be the same as the mismatch between du and df values.

Wednesday, 10 May 2017

Tar and rename files using substring replacement

In my day to day I'm asked to tar up logs quite often, but often from hosts which have the same log file names. I've got this little snippet saved that can rename the log file paths in the tar file so we don't clobber log files when extracting at the destination end.

find /var/log/jbossas/standalone -mtime -1 -type f -print | \
   xargs tar --transform 'flags=r;s|var/log/jbossas/standalone|${hostname}|' \
   -cvf /var/tmp/logs_$(hostname)_$(date +%Y%m%d).tgz

The first part of the command is running find and looking for any files that are modified in the past 24 hours (use -daystart for the past day). We print any filenames found and pass that to xargs, which will then run tar and add the files to the output tar file. However in the middle is this transform option, it's doing a simple substring replacement of "var/log/jbossas/standalone" with "$(hostname)".

And that's it. Simple tar filename transformation.

This is of course completely ignoring solutions like splunk or greylog, but often vendors want their raw log files to look at.

Extract start/end dates of SSL certificates across a list of servers

This openssl command will extract the certificate start and end dates from a server:

HOST=randomnamehost.com
openssl s_client -showcerts -connect ${HOST}:443 </dev/null 2>/dev/null | \
openssl x509 -noout -dates

This is useful to extract the dates for monitoring/checking (although, about 1000 other solutions could work too).

Here's a quick application of that in a loop to check a list of servers I have saved:

for HOST in $(cat /host/name/file); do
echo "==========="
echo $HOST
openssl s_client -showcerts -connect ${host}:443 </dev/null 2>/dev/null | \
openssl x509 -noout -dates
done
There's most likely a python module to do the same which would make comparing dates easier in a monitoring script. I should look for one.

Sunday, 16 April 2017

Create a generic host in AWS using Terraform and Ansible

For my test projects I've been using Terraform to create a test host within AWS and then apply a configuration with Ansible. This is another less of a snippet and more of a kickstart for creating development hosts.

The full repository is here:
Terraform and Ansible repository

What the terraform script does is:
  1. Create an EC2 instance (of type and region specified)
  2. Assign a public IP and apply a security group to allow SSH
  3. Tag the instance to be found by ansible
  4. Create a DNS entry for the newly created host
Then with ansible it will:
  1. Apply the hostname
  2. Add a user, apply the public key and create a sudoers file
  3. Install required packages
The advantage here is that when I need a new host I can quickly start a new instance (in a few minutes) and be able to address that via an easy DNS entry. I can adjust the security groups to allow web traffic if needed. Terraform and ansible are immutable so I can run the scripts as often as I want and push updates to my host if needed.

The advantage of all this means I can create/destroy/update the environment on demand. For instance I mostly work on my personal projects on weekends. I could schedule the creation of the environment on a Friday afternoon and then it's destruction on a Monday morning. This would save a lot of cost in unused AWS EC2 time - A large advantage if I need a GPU instance for Tensorflow. 

There are of course many ways to achieve the same goal. However by using it in this way I can adjust the Terraform scripts to target an instance in VMware or another cloud provider, or even locally. You do not have to rely on AWS only.

Tuesday, 11 April 2017

Identify which Java threads are consuming CPU time

Here's a little bash snippet that will help identify which of the JVM threads are consuming CPU time.

Theoretical problem #1 (which may have actually occured...):
The JVM is using 100% of the CPU time on a host, what is the JVM doing?
You can see this condition with a cursory glance at top, but assume there is no log message or you have no further information that lets you know why this is occurring. The usual cases are garbage collection, but the GC logs or jstat don't show this. To investigate further we need to find what inside the JVM is causing this CPU usage, so we'll look to thread CPU usage. To begin with. how do I tell how much CPU time a thread is using? This ps command will show all threads for a single JVM sorted by 'Percent CPU' usage (more on that later). For your own use adjust the pgrep line to find whichever JVM you want:

PID=$(pgrep java | head -1)
ps -eLq $PID -o cputime,lwp,%cpu | sort -k3,3 -r

Sample output:

[user@hostname default]# ps -eLq $PID -o cputime,lwp,%cpu | sort -k3,3 -r | head
    TIME   LWP %CPU
00:00:38  9951  0.1
00:00:38  9950  0.1
00:00:26 11384  0.0
00:00:26 10953  0.0
00:00:25 14177  0.0
00:00:25 10954  0.0
00:00:24  9946  0.0
00:00:24 10955  0.0
00:00:23 13721  0.0

This is good, however the %CPU usage column is a little misleading. According to the man page for ps:
CPU usage is currently expressed as the percentage of time spent running during the entire lifetime of a process. This is not ideal, and it does not conform to the standards that ps otherwise conforms to. CPU usage is unlikely to add up to exactly 100%.
This makes the value not immediately useful here, we can identify threads that have consumed CPU usage for the entire lifetime of the JVM, however if it suddenly picks up after a number of days/weeks running the percentage will be very small. We need to focus on the cputime value.

What I need to do here is put the above ps line in to a function, then call that from diff in a loop so I can see the differences over time:

# Function to list threads by PID, Identify PID by adjusting the pgrep line
function java_ps() {
  PID=$(pgrep java | head -1)
  ps -eLq $PID -o cputime,lwp
}

# Wait 30 seconds between two captures of ps output to identify the differences
LOOP_WAIT=30

while true
do
    # Some formatting to make it look nice
    echo "---------------------"
    date
    echo "Waiting ${LOOP_WAIT} seconds..."
    # And this is where we do the magic
    # Provide two inputs to diff, one is the ps output, and the second is a sleep for the LOOP_WAIT
    # and then take another output from ps. Ask diff to show only the new/changed lines between the two inputs
    diff --unchanged-line-format= --old-line-format= --new-line-format='%L' <(java_ps) <(sleep ${LOOP_WAIT};java_ps)
done

Some example output from the above:

---------------------
Mon Apr 10 13:44:30 UTC 2017
Waiting 30 seconds...
00:01:32 20990
---------------------
Mon Apr 10 13:45:00 UTC 2017
Waiting 30 seconds...
00:02:02 20990
00:10:43  5805
00:01:30  8701
00:01:30  8702
00:02:57 30297
00:01:31  2402
00:00:00 19579
00:00:00 19580
00:00:00 19582

What is shown here is LWP 20990 is consuming 30 seconds of CPU between each 30 second sample of CPU time. This will be the thread we want to investigate more closely.

Once I've identified the LWP I can convert the number to hex and extract it from the jstack output of the JVM with this:

LWP=20990
THREADID=$(printf '%x\n' $LWP)
PID=$(pgrep java | head -1)
sudo -u ${javauser} jstack $PID|  pcregrep -M "(?s)nid=0x${THREADID}.*?\t.*?\n\n" | head -n -2

As per a previous post on extracting the stack traces: We extract using pcregrep in multiline mode (M), looking for the nid field (which is native thread ID, the LWP in hex), and then capture everything to the next blank line ('\n\n').

That will give us the stack trace of the problem thread, for example:

"LDAPv3EventService" daemon prio=10 tid=0x00007f6a6ce14000 nid=0xaae runnable [0x00007f6acc2db000]
   java.lang.Thread.State: RUNNABLE
        at java.util.HashMap.put(HashMap.java:494)
        at java.util.HashSet.add(HashSet.java:217)
        at com.sun.identity.idm.server.IdRepoJAXRPCObjectImpl.processEntryChanged_idrepo(IdRepoJAXRPCObjectImpl.java:795)
        at com.sun.identity.idm.server.JAXRPCObjectImplEventListener.identityDeleted(JAXRPCObjectImplEventListener.java:43)
        at com.sun.identity.idm.IdRepoListener.objectChanged(IdRepoListener.java:205)
        at com.sun.identity.idm.plugins.ldapv3.LDAPv3Repo.objectChanged(LDAPv3Repo.java:5308)
        at com.sun.identity.idm.plugins.ldapv3.LDAPv3EventService.dispatchEvent(LDAPv3EventService.java:949)
        at com.sun.identity.idm.plugins.ldapv3.LDAPv3EventService.processSearchResultMessage(LDAPv3EventService.java:1092)
        at com.sun.identity.idm.plugins.ldapv3.LDAPv3EventService.processResponse(LDAPv3EventService.java:718)
        at com.sun.identity.idm.plugins.ldapv3.LDAPv3EventService.run(LDAPv3EventService.java:589)
        at java.lang.Thread.run(Thread.java:745)

In our maybe-not theoretical problem we've identified the stack trace of the problem thread. We can then perform a quick search to see if the problem has been discovered before, and in this case it leads to these bug reports (and probably this post now):

https://bugster.forgerock.org/jira/browse/OPENAM-5324
https://bugster.forgerock.org/jira/browse/OPENAM-119

Now we can form a plan to resolve the issue.

Sunday, 26 March 2017

Using Feature Importance to identify metrics of concern in an application

This is less of a snippet, more of a story.

I've recently been involved in trying to diagnose an intermittent performance problem in a production system of which there were many performance metrics collected. However with the volume of data being recorded for each transaction there were no clear identifiers to what was causing the issues. Every time a batch of transactions would hit a threshold an incident would be raised and then investigations would go on and result in no clear answer on what was the cause. This would keep repeating over and over each time looking at different components that have had historical issues.

In the background I'd been reading about Tensorflow and seeing what machine learning could do. There's a lot of interest at the moment as it's just hit a milestone release, however nothing immediately applied to any problems I was having and it was more of an interest topic. What did come about while trying to think of ways machine learning could be used to apply to the above problems we were seeing was framing my question a bit better: How do I identify which metrics correlate the most to a transaction taking longer than the defined SLA?

Google wasn't so helpful with my new question, however it did lead me to discover more about how to select which data to train your machine learning models with. Within machine learning it's important to identify which features give the best information about what you're classifying so that you don't waste time training your model with unnecessary data, so out of that comes a topic called feature importance.

After reading about this and experimenting with the scikit-learn example for a few hours I was able to set up a notebook in ipython, load our data in and produce a graph which clearly identified which of the components were causing issues. It was a component we hadn't looked at before, and the output clearly showed that any increase in the timings on that component had a massive correlation with the total transaction time exceeding the SLA. I had loaded in a lot of other metrics along with this, such as the time of day and the datacenter the processing took place in, but the clearest measure by a huge margin was this one often ignored component.

What this allowed me to do was help redirect the investigations to this component so we can find out why it would impacted total transaction time so much. Eventually investigations may have ended up looking around this area, but with this technique I'm able to sift through the massive volumes of recorded metrics to short cut those investigations.

I've put together an ipython notebook to show how the application of feature importance could be used in identifying an application performance problem here. There is a lot of other information that I've included in the descriptions also. It should be very easy to pick this up and apply it to some other problem where you need to find what exactly correlates to an expected output.