How to analyze your pipe runs with Bunyan

Introduction

In this post, we'll discuss how our Simple Data Pipe example app uses the Bunyan Node.js logging framework to capture detailed logging information about a pipe run. We'll then show how to analyze the report using the Bunyan viewer tool.

If you've explored our Simple Data Pipe tutorial on Bluemix, you'd know that metadata about your pipe runs is stored in Cloudant as JSON. Cloudant's support for binary attachments within JSON lets us attach the logs from Bunyan right alongside their associated JSON document, which you can then access for further analysis.

A word about Bunyan

Bunyan is a simple and fast JSON logging library for Node.js services. It can be configured to output the data to streams that can be stored anywhere. Simple Data Pipe uses this library to capture log information about a particular run, then attach the report to the pipe run document stored in the pipe_db database in your Cloudant account.

This logging framework supports many log levels: trace, debug, info, warn, and error. Bunyan also provides a CLI utility to pretty-print its output, with the ability to filter by logging group and level. See https://github.com/trentm/node-bunyan for more information.

How to locate the log for a particular run

Here's the scenario: You attempted a pipe run and something went wrong. You now need to locate the log, download it from the pipe_db Cloudant database, and analyze it for troubleshooting.

###Follow these steps:###

1. Locate the log: Go to Bluemix and click on your pipe app instance
2. Click on the pipe-cloudant-service box to open the Cloudant dashboard, then click on the Launch button
3. In the Cloudant dashboard, click on the pipe_db database
4. On the left tab, click on _design/application, then Views, then all_runs
5. On the left-hand side of the all_runs view, you can see that the Map function logic that defines this view indexes pipe run documents sorted in increasing order by date. Therefore, the run document you're looking for is the last one in the view. (You may need to page through the results a few times if you have performed a lot of runs.)
6. Click on the pencil icon to open the run document. You should be able to see the JSON metadata for the run.
7. Click on the View Attachments dropdown button and right-click on run.log. Then click on the Save Link As... option. This will download the file to a directory of your choice.
8. The next step will be to use the Bunyan CLI tool to analyze run.log.

Analyze run.log

If you have not already done so, install Bunyan on your local machine using these simple steps:

npm install bunyan -g

Note: use sudo if you are on a Linux-based system like Mac OS X.

You can view the entire log in pretty-printed format using this command:

 bunyan <path_to_run.log>

Note: use q to quit long output.

You can also filter the log to only view errors using the following command:

 bunyan <path_to_run.log> -l error

Note: you can use any log level you want, e.g., error, info, warn, etc.