PR feedback, especially on .md files

Author: idigdoug

LinuxTraceLogCapture.md

@@ -97,10 +97,46 @@ $ sudo lttng stop
 $ sudo lttng destroy
 ```
 
-# Perf
-Perf is used to collect tracepoint events.
+# Perf.data
 
-[perf](https://perf.wiki.kernel.org/)
+You can collect and view Linux [kernel-mode](https://www.kernel.org/doc/html/latest/trace/tracepoints.html) and [user-mode](https://docs.kernel.org/trace/user_events.html) tracepoints in the `perf.data` file format.
+
+- Select existing tracepoints that you want to collect, or write your own programs that generate tracepoint events.
+- Use the Linux `perf` tool or some other tool to collect tracepoint events into `perf.data` files.
+- Use the PerfData extension to view the tracepoints.
+
+## Selecting tracepoints
+
+The Linux kernel and the kernel modules generate many useful tracepoints that you can collect. Look in `/sys/kernel/tracing/events` for the tracepoints that are available to you.
+
+In addition, Linux 6.4 adds support for generating
+[user_events](https://docs.kernel.org/trace/user_events.html)
+tracepoints from user-mode programs.
+
+- [LinuxTracepoints](https://github.com/microsoft/LinuxTracepoints) contains support for generating `user_events` from C/C++ programs.
+- [LinuxTracepoints-Rust](https://github.com/microsoft/LinuxTracepoints-Rust) contains support for generating `user_events` from Rust programs.
+
+## Collecting tracepoints
+
+The Linux [perf](https://perf.wiki.kernel.org/) tool supports collecting tracepoint events using `perf record`.
+
+- Install `perf` from the `linux-perf` or `linux-tools` package.
+  - Note that some `perf` packages use a wrapper script to help you match the running kernel version with a version-specific build of the `perf` tool, e.g. `perf_VERSION`. For collecting tracepoints, the version doesn't need to match. If you have version mismatch problems, you can safely bypass the wrapper script and directly use the `perf_VERSION` tool.
+- Install the `libtraceevent1` package to enable `perf` support for tracepoints.
+- Use [perf record](https://www.man7.org/linux/man-pages/man1/perf-record.1.html) to collect traces, e.g. `perf record -o MyFile.perf.data -k monotonic -e "event1_group:event1_name,event2_group:event2_name"`
+  - Use `-k monotonic` to include clock offset information in the data file.
+
+You can also use other tools that generate `perf.data`-compatible files.
+
+- [libtracepoint-control](https://github.com/microsoft/LinuxTracepoints/tree/main/libtracepoint-control-cpp) includes a library for configuring tracepoint collection sessions and collecting `perf.data` files.
+- [tracepoint-collect](https://github.com/microsoft/LinuxTracepoints/blob/main/libtracepoint-control-cpp/tools/tracepoint-collect.cpp) is a simple tool that collects tracepoint events into `perf.data` files.
+
+# Perf.data.txt
+Perf is used to collect CPU Sampling (cpu-clock) events as LTTng doesn't support capturing these yet. Note: Stacks may require symbol setup.
+
+The perf CPU Sampling analysis plugin uses perf.data.txt files as input.
+
+[perf](https://perf.wiki.kernel.org/) CPU Sampling(cpu-clock)
 
 If you want to trace .NET Core then you need [perfcollect](http://aka.ms/perfcollect) which capture CPU sampling and more
 
@@ -109,6 +145,11 @@ If you want to trace .NET Core then you need [perfcollect](http://aka.ms/perfcol
 $ sudo apt-get install linux-tools-common
 ```
 
+## User-Mode (UM) Symbols Install
+KM symbols are automatically resolved. If you wish to resolve UM cpu sample functions and stacks, you may need to install debug packages for the binary you are profiling
+
+For example, [Debug Symbol Packages on Ubuntu](https://wiki.ubuntu.com/Debug%20Symbol%20Packages)
+
 ## Record a trace
 ```bash
 $ sudo /usr/bin/perf record -g -a -F 999 -e cpu-clock,sched:sched_stat_sleep,sched:sched_switch,sched:sched_process_exit -o perf_cpu.data
@@ -119,19 +160,50 @@ $ sudo /usr/bin/perf record -g -a -F 999 -e cpu-clock,sched:sched_stat_sleep,sch
 $ Ctrl-C
 ```
 
+## Convert trace to text format
+This is to useful along-side the CTF trace to resolve UM IP/Symbols. Similar to what [perfcollect](https://raw.githubusercontent.com/microsoft/perfview/master/src/perfcollect/perfcollect) uses
+
+```bash
+$ sudo perf inject -v -s -i perf_cpu.data -o perf.data.merged
+
+# There is a breaking change where the capitalization of the -f parameter changed.
+$ sudo perf script -i perf.data.merged -F comm,pid,tid,cpu,time,period,event,ip,sym,dso,trace > perf.data.txt
+
+if [ $? -ne 0 ]
+then
+    $ sudo perf script -i perf.data.merged -f comm,pid,tid,cpu,time,period,event,ip,sym,dso,trace > perf.data.txt
+fi
+
+# If the dump file is zero length, try to collect without the period field, which was added recently.
+if [ ! -s perf.data.txt ]
+then
+    $ sudo perf script -i perf.data.merged -f comm,pid,tid,cpu,time,event,ip,sym,dso,trace > perf.data.txt
+fi
+```
+
+## Capture trace timestamp start 
+Perf.data.txt only contains relative timestamps. If you want correct absolute timestamps in UI then you will need to know the trace start time.
+
+```bash
+$ sudo perf report --header-only -i perf_cpu.data | grep "captured on"
+```
+
+Place the "captured on" timestamp for example "Thu Oct 17 15:37:36 2019" in a timestamp.txt file next to the trace folder. The timestamp will be interpreted as UTC
+
 # Transferring the files to Windows UI (optional)
 You then need to transfer the perf files to a Windows box where WPA runs. The most important file is perf.data.txt
 
 ```bash
-$ sudo chmod 777 -R perf_cpu.data
+$ sudo chmod 777 -R perf*
 ```
 
 - Copy files from Linux to Windows box with WinSCP/SCP OR 
 ```bash
-$ tar -czvf perf_cpu.tar.gz perf_cpu.data
+$ tar -czvf perf_cpu.tar.gz perf*
 ```
+- (Optional if you want absolute timestamps) Place timestamp.txt next to perf.data.txt
+- Open perf.data.txt with WPA
 
-- Open perf_cpu.data with WPA
 
 # Presentations
 

PerfDataExtension/pluginManifest.json

@@ -5,7 +5,7 @@
     "version": "0.1.2"
   },
   "displayName": "Linux - Perf",
-  "description": "Enables viewing Linux tracepoint events recorded with the perf tool",
+  "description": "Enables loading Linux events from perf.data files (e.g. generated by the perf tool)",
   "owners": [
     {
       "name": "LinuxTracepoints",

README.md

@@ -8,7 +8,7 @@
 
 > Tracing supported: 
 - [LTTng](https://lttng.org) (Kernel CPU scheduling, Processes, Threads, Block IO/Disk, Syscalls, File events, etc)
-- [perf](https://perf.wiki.kernel.org/) (Generic events)
+- [perf](https://perf.wiki.kernel.org/) Generic Events, CPU Sampling (cpu-clock)
 - [Perfetto](https://perfetto.dev/) Android & Chromium (CPU Scheduling, CPU Sampling, CPU Frequency, FTrace, Android Logs, Generic Events / Default Tracks, GPU Counters, Jank Detection, Processes, Android Packages)
 
 > Logs supported: 
@@ -57,7 +57,8 @@ The tools can be run in several modes:
   - Used as a library to process traces / logs programatically in a .NET Core language like C#
     - Examples: 
     - [LTTng 1](LTTngDriver/Program.cs), [LTTng 2](LTTngDataExtUnitTest/LTTngUnitTest.cs)
-    - [Perf](PerfDataUnitTest/PerfDataUnitTest.cs)
+    - [Perf.data.txt](PerfUnitTest/PerfUnitTest.cs) (CPU Sampling analysis)
+    - [Perf.data](PerfDataUnitTest/PerfDataUnitTest.cs) (Generic events)
     - [LinuxLogs](LinuxLogParsers/LinuxLogParsersUnitTest/LinuxLogParsersUnitTest.cs)
     - [Perfetto](PerfettoUnitTest/PerfettoUnitTest.cs)
   - With a driver program for example dumping to screen or text format
@@ -113,7 +114,7 @@ The tools can be run in several modes:
 - Unified (LTTng, Perfetto, or other multiple different logs files together)
   - Once you gather the data, there is a tiny bit of prep needed to open them in a single unified timeline (like the screenshot above)
   - If you want to open multiple logs together in single timeline - Copy all trace files and logs you want to open to single folder
-  - Example: You want to open in the same timeline: LTTng, Perf, Dmesg
+  - Example: You want to open in the same timeline: LTTng, Perf CPU Sampling, Dmesg
     - Ensure that the Linux CTF folder/trace is zipped and renamed to .ctf in the same folder (hack so open Unified works)
   - WPA -> File -> Open -> Multi-select all files and choose "Open Unified"
 

azure-pipelines.yml

@@ -75,6 +75,8 @@ extends:
               LTTngDataExtensions\LTTngDataExtensions.csproj
               LTTngDataExtUnitTest\LTTngDataExtUnitTest.csproj
               LTTngDriver\LTTngDriver.csproj
+              PerfDataExtension\PerfDataExtension.csproj
+              PerfDataUnitTest\PerfDataUnitTest.csproj
               PerfDataTxtExtension\PerfDataTxtExtension.csproj
               PerfettoCds\PerfettoCds.csproj
               PerfUnitTest\PerfUnitTest.csproj
@@ -92,7 +94,14 @@ extends:
           inputs:
             command: 'test'
             projects: '$(System.DefaultWorkingDirectory)/Microsoft-Perf-Tools-Linux-Android.sln'
-        
+
+        - task: CopyFiles@2
+          displayName: Copy PerfDataExtension Build to Output Artifacts
+          inputs:
+            SourceFolder: 'PerfDataExtension/bin/$(BuildConfiguration)/netstandard2.1'
+            Contents: '**'
+            TargetFolder: '$(Build.ArtifactStagingDirectory)/Microsoft-Performance-Tools-Linux/MicrosoftPerfToolkitAddins/PerfDataExtension'
+
         - task: CopyFiles@2
           displayName: Copy PerfDataTxtExtension Build to Output Artifacts
           inputs: