section 含义详细解释：https://docs.nvidia.com/nsight-compute/ProfilingGuide/index.html#sections-and-rules。
就算你用了 --set full，NCU GUI 或者 NCU 命令行也不会把所有 section 都显示出来。
$ ncu --version
NVIDIA (R) Nsight Compute Command Line Profiler
Copyright (c) 2018-2024 NVIDIA Corporation
Version 2025.1.0.0 (build 35237751) (public-release)
--set arg Identifier of section set to collect. **If not specified, the basic set is
collected.**
--section arg Collect the section by providing section identifier in one of the following
ways:
<section identifier> Set the section identifier for an exact match.
regex:<expression> Set the regex to use for matching the section identifier.
**If option is not specified, the default section set is collected.**
Section metrics that cannot be collected will usually generate a warning only.
$ ncu --list-sets
---------- --------------------------------------------------------------------------- ------- -----------------
Identifier Sections Enabled Estimated Metrics
---------- --------------------------------------------------------------------------- ------- -----------------
basic LaunchStats, Occupancy, SpeedOfLight, WorkloadDistribution yes 202
detailed ComputeWorkloadAnalysis, LaunchStats, MemoryWorkloadAnalysis, MemoryWorkloa no 569
dAnalysis_Chart, Occupancy, SourceCounters, SpeedOfLight, SpeedOfLight_Roof
lineChart, WorkloadDistribution
full ComputeWorkloadAnalysis, InstructionStats, LaunchStats, MemoryWorkloadAnaly no 5907
sis, MemoryWorkloadAnalysis_Chart, MemoryWorkloadAnalysis_Tables, NumaAffin
ity, Nvlink_Tables, Nvlink_Topology, Occupancy, PmSampling, SchedulerStats,
SourceCounters, SpeedOfLight, SpeedOfLight_HierarchicalDoubleRooflineChart
, SpeedOfLight_HierarchicalHalfRooflineChart, SpeedOfLight_HierarchicalSing
leRooflineChart, SpeedOfLight_HierarchicalTensorRooflineChart, SpeedOfLight
_RooflineChart, WarpStateStats, WorkloadDistribution
nvlink Nvlink, Nvlink_Tables, Nvlink_Topology no 52
pmsampling PmSampling, PmSampling_WarpStates no 186
roofline SpeedOfLight, SpeedOfLight_HierarchicalDoubleRooflineChart, SpeedOfLight_Hi no 5260
erarchicalHalfRooflineChart, SpeedOfLight_HierarchicalSingleRooflineChart,
SpeedOfLight_HierarchicalTensorRooflineChart, SpeedOfLight_RooflineChart, W
orkloadDistribution
$ ncu --list-sections
-------------------------------------------- ----------------------------------------------------------------- ------- --------------------------------------------------
Identifier Display Name Enabled Filename
-------------------------------------------- ----------------------------------------------------------------- ------- --------------------------------------------------
C2CLink C2CLink no ...sight Compute/2025.1.0/Sections/C2CLink.section
ComputeWorkloadAnalysis Compute Workload Analysis no ...25.1.0/Sections/ComputeWorkloadAnalysis.section
InstructionStats Instruction Statistics no ...2025.1.0/Sections/InstructionStatistics.section
LaunchStats Launch Statistics yes ...pute/2025.1.0/Sections/LaunchStatistics.section
MemoryWorkloadAnalysis Memory Workload Analysis no ...025.1.0/Sections/MemoryWorkloadAnalysis.section
MemoryWorkloadAnalysis_Chart Memory Workload Analysis Chart no ...0/Sections/MemoryWorkloadAnalysis_Chart.section
MemoryWorkloadAnalysis_Tables Memory Workload Analysis Tables no .../Sections/MemoryWorkloadAnalysis_Tables.section
NumaAffinity NUMA Affinity no ... Compute/2025.1.0/Sections/NumaAffinity.section
Nvlink NVLink no ...Nsight Compute/2025.1.0/Sections/Nvlink.section
Nvlink_Tables NVLink Tables no ...Compute/2025.1.0/Sections/Nvlink_Tables.section
Nvlink_Topology NVLink Topology no ...mpute/2025.1.0/Sections/Nvlink_Topology.section
Occupancy Occupancy yes ...ght Compute/2025.1.0/Sections/Occupancy.section
PmSampling PM Sampling no ...ht Compute/2025.1.0/Sections/PmSampling.section
PmSampling_WarpStates PM Sampling: Warp States no ...2025.1.0/Sections/PmSampling_WarpStates.section
SchedulerStats Scheduler Statistics no ...e/2025.1.0/Sections/SchedulerStatistics.section
SourceCounters Source Counters no ...ompute/2025.1.0/Sections/SourceCounters.section
SpeedOfLight GPU Speed Of Light Throughput yes ... Compute/2025.1.0/Sections/SpeedOfLight.section
SpeedOfLight_HierarchicalDoubleRooflineChart GPU Speed Of Light Hierarchical Roofline Chart (Double Precision) no ...OfLight_HierarchicalDoubleRooflineChart.section
SpeedOfLight_HierarchicalHalfRooflineChart GPU Speed Of Light Hierarchical Roofline Chart (Half Precision) no ...edOfLight_HierarchicalHalfRooflineChart.section
SpeedOfLight_HierarchicalSingleRooflineChart GPU Speed Of Light Hierarchical Roofline Chart (Single Precision) no ...OfLight_HierarchicalSingleRooflineChart.section
SpeedOfLight_HierarchicalTensorRooflineChart GPU Speed Of Light Hierarchical Roofline Chart (Tensor Core) no ...OfLight_HierarchicalTensorRooflineChart.section
SpeedOfLight_RooflineChart GPU Speed Of Light Roofline Chart no ...1.0/Sections/SpeedOfLight_RooflineChart.section
WarpStateStats Warp State Statistics no ...e/2025.1.0/Sections/WarpStateStatistics.section
WorkloadDistribution GPU and Memory Workload Distribution yes .../2025.1.0/Sections/WorkloadDistribution.section
Section 文件就是 NCU GUI Details 页的模版，文件中定义了 NCU GUI 对应的地方该显示什么。
Section 文件同时定义了自己所属的 set。
Identifier: "SpeedOfLight"
DisplayName: "GPU Speed Of Light Throughput"
Description: "High-level overview of the throughput for compute and memory resources of the GPU. For each unit, the throughput reports the achieved percentage of utilization with respect to the theoretical maximum. Breakdowns show the throughput for each individual sub-metric of Compute and Memory to clearly identify the highest contributor."
Order: 10
Sets {
Identifier: "basic"
}
Sets {
Identifier: "detailed"
}
Sets {
Identifier: "full"
}
Sets {
Identifier: "roofline"
}
Header {
Metrics {
Label: "Compute (SM) Throughput"
Name: "sm__throughput.avg.pct_of_peak_sustained_elapsed"
}
Metrics {
Label: "Duration"
Name: "gpu__time_duration.sum"
}
Metrics {
Label: "Memory Throughput"
Name: "gpu__compute_memory_throughput.avg.pct_of_peak_sustained_elapsed"
}
Metrics {
Label: "Elapsed Cycles"
Name: "gpc__cycles_elapsed.max"
}
Metrics {
Label: "L1/TEX Cache Throughput"
Name: "l1tex__throughput.avg.pct_of_peak_sustained_active"
}
Metrics {
Label: "SM Active Cycles"
Name: "sm__cycles_active.avg"
}
Metrics {
Label: "L2 Cache Throughput"
Name: "lts__throughput.avg.pct_of_peak_sustained_elapsed"
}
Metrics {
Label: "SM Frequency"
Name: "gpc__cycles_elapsed.avg.per_second"
}
Metrics {
Label: "DRAM Throughput"
Name: "gpu__dram_throughput.avg.pct_of_peak_sustained_elapsed"
Filter {
MaxArch: CC_70
}
Options {
Name: "gpu__dram_throughput.avg.pct_of_peak_sustained_elapsed"
Filter {
MinArch: CC_75
MaxArch: CC_86
}
}
Options {
Name: "gpu__dram_throughput.avg.pct_of_peak_sustained_elapsed"
Filter {
MinArch: CC_89
MaxArch: CC_100
}
}
Options {
Name: "gpu__dram_throughput.avg.pct_of_peak_sustained_elapsed"
Filter {
MinArch: CC_120
}
}
}
Metrics {
Label: "DRAM Frequency"
Name: "dram__cycles_elapsed.avg.per_second"
Filter {
MaxArch: CC_70
}
Options {
Name: "dram__cycles_elapsed.avg.per_second"
Filter {
MinArch: CC_75
MaxArch: CC_86
}
}
Options {
Name: "dram__cycles_elapsed.avg.per_second"
Filter {
MinArch: CC_89
MaxArch: CC_100
}
}
Options {
Name: "dram__cycles_elapsed.avg.per_second"
Filter {
MinArch: CC_120
}
}
}
}
Metrics {
Metrics {
Label: "Waves Per SM"
Name: "launch__waves_per_multiprocessor"
Filter {
CollectionFilter {
CollectionScopes: CollectionScope_Launch
}
}
}
}
Body {
DisplayName: "GPU Throughput Chart"
SetDefault: true
Items {
BarChart {
Label: "GPU Throughput"
ValueAxis {
Label: "Speed Of Light (SOL) [%]"
Range {
Min: 0
Max: 100
}
TickCount: 11
}
Metrics {
Label: "Compute (SM) [%]"
Name: "sm__throughput.avg.pct_of_peak_sustained_elapsed"
}
Metrics {
Label: "Memory [%]"
Name: "gpu__compute_memory_throughput.avg.pct_of_peak_sustained_elapsed"
}
}
}
}
Body {
DisplayName: "GPU Throughput Breakdown"
Items {
HorizontalContainer {
Items {
Table {
Label: "Compute Throughput Breakdown"
Columns: 1
SortKey: ByValue
SortDirection: Descending
Metrics {
Label: "{UNIT}: {COMPOUND_NAME}"
Name: "breakdown:sm__throughput.avg.pct_of_peak_sustained_elapsed"
}
}
}
Items {
Table {
Label: "Memory Throughput Breakdown"
Columns: 1
SortKey: ByValue
SortDirection: Descending
Metrics {
Label: "{UNIT}: {COMPOUND_NAME}"
Name: "breakdown:gpu__compute_memory_throughput.avg.pct_of_peak_sustained_elapsed"
}
}
}
}
}
}
--list-metrics List all metrics to be collected based on **selected sections**.
--query-metrics Query available metrics for devices on the system. Use --devices and --chips to
filter which devices to query. By default, metrics reported by this option
require a suffix to be collected. See --query-metrics-mode for details.
--query-metrics-mode arg (=base) Set the mode for querying metrics. Implies --query-metrics.
Available modes:
base (default)
(base names for metrics)
suffix
(suffix names for metrics. Use --metrics to specify the base metrics to
query)
all
(full names for metrics)
--query-metrics-collection arg (=profiling)
Set which metric collection kind to query. Implies --query-metrics.
Available modes:
device
(query CUDA device attributes)
groups
(query metric groups)
launch
(query launch attributes)
numa
(query NUMA topology metrics)
nvlink
(query metrics related to NVLink)
pmsampling
(query metrics available for pm sampling)
profiling (default)
(query metrics available for profiling)
source
(query source metrics)
stats
(query metrics generated by the profiler to inform about profiling
statistics)
warpsampling
(query metrics available for periodic sampling of the warp program counter
and warp scheduler state)
--metrics arg Specify all metrics to be profiled, separated by comma.
Names passed to this option support the following prefixes:
regex:<expression> Expands to all metrics that partially match the
expression. Enclose the regular expression in
^...$ to force a full match.
group:<name> Lists all metrics of the metric group with that
name. See section files for valid group names.
breakdown:<metric> Expands to the input metrics of the high-level
throughput metric. If the specified metric does
not support a breakdown, no metrics are added.
If a metric requires a suffix to be valid, and no prefix is used this option
automatically expands the name to all available first-level sub-metrics.
Metrics that cannot be collected will generate an error.
--list-chips List all supported chips that can be used with --chips.
--chips arg Specify the chips for querying metrics, separated by comma.
-
Collection
Metric 的来源（collection）如下所示，有：launch、nvlink、numa、device、warpsampling、source、groups、stats、pmsampling、profiling。我们平时一般关注 profiling。
这个 group 有点奇怪，help 里面说 See section files for valid group names. 但是下面 8 个 group name，我只在 section files 里面找到最后两个。
这 8 个 group 的子 metric 列表估计只能 profile 的时候得到，不 profile 估计得不到。
- launch (query launch attributes) - nvlink (query metrics related to NVLink) - numa (query NUMA topology metrics) - device (query CUDA device attributes) - warpsampling (query metrics available for periodic sampling of the warp program counter and warp scheduler state) - Doc Warp Stall Reasons - Doc Warp Stall Reasons (Not Issued) - source (query source metrics) - Doc Source Metrics - Doc L2 Cache Eviction Metrics - Doc Instructions Per Opcode Metrics - Doc SASS Unit-Level Instructions Executed Metrics - groups (query metric groups) - group:memory__chart - group:memory__dram_table - group:memory__first_level_cache_table - group:memory__l2_cache_evict_policy_table - group:memory__l2_cache_table - group:memory__shared_table - group:smsp__pcsamp_warp_stall_reasons - group:smsp__pcsamp_warp_stall_reasons_not_issued - stats (query metrics generated by the profiler to inform about profiling statistics) - pmsampling (query metrics available for pm sampling) - profiling (default) (query metrics available for profiling) -
命令
-
查询 default collection、default section 的所有 metric
ncu --query-metrics --chip ga102 -
查询 suffix
-
counter
ncu --query-metrics-mode suffix --metrics sm__inst_executed --chip ga100 -
ratio
ncu --query-metrics-mode suffix --metrics smsp__average_warp_latency --chip ga100 -
throughput
ncu --query-metrics-mode suffix --metrics sm__throughput --chip ga100
-
-
查询某个 collection 的 metric
ncu --query-metrics-collection launch --chip ga102
-
-
elapsed 和 active 区别
elapsed 意思是，从 kernel 开始到结束的整个时间跨度，不管这个硬件单元当时有没有在工作。它是个「全程」基准。
active 意思是，只算这个单元实际处于活跃状态（被分配了工作、在运行）的那部分时间/周期。单元空着的时候不计入。
-
类型
Metric 基本上只有三类：counter、ratio、throughput。
-
Counter
counter 类 suffix 一般有 4 x (1+10) 个。
Every counter has associated peak rates in the database, to allow computing its throughput as a percentage.
-
Rollup metric
- sum
- avg
- min
- max
-
sub metric
.peak_sustained.peak_sustained_active.peak_sustained_active.per_second.peak_sustained_elapsed.peak_sustained_elapsed.per_second.per_second.per_cycle_active.per_cycle_elapsed.pct_of_peak_sustained_active.pct_of_peak_sustained_elapsed
-
例子：
sm__inst_executed.avgTODO
关于 submetric，官方文档有含义描述。
ncu --query-metrics-mode suffix --metrics sm__inst_executed --chip ga102也有含义描述。但是好像都不太准确。https://forums.developer.nvidia.com/t/how-are-pct-of-peak-metrics-calculated/330187/4
ncu --metrics regex:sm__inst_executed\.avg\..*,sm__cycles_active.max,sm__cycles_elapsed.max ... Section: Command line profiler metrics ------------------------------------------------------- ----------- ------------ Metric Name Metric Unit Metric Value ------------------------------------------------------- ----------- ------------ sm__inst_executed.avg.peak_sustained inst/cycle 4 sm__inst_executed.avg.peak_sustained_active inst 2,297,660.79 sm__inst_executed.avg.peak_sustained_active.per_second inst/ns 8.45 sm__inst_executed.avg.peak_sustained_elapsed inst 2,398,115.26 sm__inst_executed.avg.peak_sustained_elapsed.per_second inst/ns 8.82 sm__inst_executed.avg.per_second inst/ns 2.96 sm__inst_executed.avg.per_cycle_active inst/cycle 1.40 sm__inst_executed.avg.per_cycle_elapsed inst/cycle 1.34 sm__inst_executed.avg.pct_of_peak_sustained_active % 35.03 sm__inst_executed.avg.pct_of_peak_sustained_elapsed % 33.57 sm__cycles_active.max cycle 597,149 sm__cycles_elapsed.max cycle 599,534sm__inst_executed.avg inst # of warp instructions executed sm__inst_executed.avg inst/cycle # of insts executed per cycle .peak_sustained 这是个理论值，一个 SM 4 个 SMSP，所以是 4 sm__inst_executed.avg inst peak_sustained x sm__cycles_active .peak_sustained_active 平均在 sm__cycles_active 上取，和结果上取，一样 sm__inst_executed.avg inst/s peak_sustained_active / time .peak_sustained_active.per_second 这个 time 是什么还不清楚，可能是 kernel duration sm__inst_executed.avg inst peak_sustained x sm__cycles_elapsed .peak_sustained_elapsed 平均在 sm__cycles_elapsed 上取，和结果上取，一样 sm__inst_executed.avg inst/s peak_sustained_elapsed / time .peak_sustained_elapsed.per_second 这个 time 是什么还不清楚，可能是 kernel duration sm__inst_executed.avg inst/s sm__inst_executed / time .per_second 这个 time 是什么还不清楚，可能是 kernel duration sm__inst_executed.avg inst/cycle avg (sm__inst_executed / sm__cycles_active) .per_cycle_active sm__inst_executed.avg inst/cycle avg (sm__inst_executed / sm__cycles_elapsed) .per_cycle_elapsed sm__inst_executed.avg % .pct_of_peak_sustained_active sm__inst_executed.avg % .pct_of_peak_sustained_elapsed
-
-
Ratio
ratio 类的一般只有 3 个 suffix。
.pct.ratio.max_rate
-
Throughput
throughput 类的一般有 4x2 个 suffix。throughput 类的 metric 必须跟
.pct_of_peak_sustained_active或者.pct_of_peak_sustained_elapsed。throughput 的子 metric 列表估计只能 profile 的时候得到，不 profile 估计得不到。
Throughput metrics return the maximum percentage value of their constituent counters. These constituents have been carefully selected to represent the sections of the GPU pipeline that govern peak performance.
Throughputs have a breakdown of underlying metrics from which the throughput value is computed. You can collect
breakdown:<throughput-metric>to collect a throughput’s breakdown metrics.-
Rollup metric
- sum
- avg
- min
- max
-
Sub metric
.pct_of_peak_sustained_active.pct_of_peak_sustained_elapsed
-
例子：
sm__throughput.avg.pct_of_peak_sustained_elapsed该值为 breakdown 中最大的值，所以具体含义由 breakdown 中最大的值决定。
该值为所有 sm 的值的平均。
-
-
-
Naming
Collection 不是 profiling 的 metric 可能不遵守下面的命令规则。
-
Unit-Level Counter
unit__(subunit?)_(pipestage?)_quantity_(qualifiers?) -
Interface Counter
unit__(subunit?)_(pipestage?)_(interface)_quantity_(qualifiers?) -
Unit Metric
(counter_name).(rollup_metric) -
Sub-Metric
(counter_name).(rollup_metric).(submetric)
-
-
Units
- dram
- fe
- gpc
- gpu
- l1tex
- ltc
- lts
- pm
- sm
- smsp
- tpc
-
Subunits
- global
- local
- lsu
- shared
-
Pipelines
-
Logical
- alu: Bit / Logic / integer (w/o IMAD and IMUL) / FP32-to-FP16 conversion
- aluheavy
- alulite
- fma
- fmaheavy: FP32 (FADD, FMAD, FMUL) / FP16 (HADD2, HMUL2, HFMA2) / IMAD and IMUL
- fmalite: FP32 (FADD, FMUL, FMA) / FP16 (HADD2, HMUL2, HFMA2)
- alu: Bit / Logic / integer (w/o IMAD and IMUL) / FP32-to-FP16 conversion
-
Physical
A40 的 metric 中没有 aluheavy 和 alulite，只有 alu。有 fma、fmaheavy、fmalite。和 physical 结构比较符合。
- fmalite: FP32 (FADD, FMUL, FMA) / FP16 (HADD2, HMUL2, HFMA2)
- Shared FMA Heavey
- fmaheavy: FP32 (FADD, FMAD, FMUL) / FP16 (HADD2, HMUL2, HFMA2) / IMAD and IMUL
- alulite
-
-
Quantities
- instruction
- request
- sector
- tag
- wavefront
sm__inst_executed_pipe_alu
sm__inst_executed_pipe_fma
sm__inst_executed_pipe_fma_type_fp16
sm__inst_executed_pipe_fmaheavy
sm__inst_executed_pipe_fmalite
sm__inst_executed_pipe_tensor
sm__inst_executed_pipe_tensor_op_hmma
sm__inst_executed_pipe_tensor_op_imma
sm__ops_path_tensor_src_bf16_dst_fp32
sm__ops_path_tensor_src_bf16_dst_fp32_sparsity_off
sm__ops_path_tensor_src_bf16_dst_fp32_sparsity_on
sm__ops_path_tensor_src_fp16_dst_fp16
sm__ops_path_tensor_src_fp16_dst_fp16_sparsity_off
sm__ops_path_tensor_src_fp16_dst_fp16_sparsity_on
sm__ops_path_tensor_src_fp16_dst_fp32
sm__ops_path_tensor_src_fp16_dst_fp32_sparsity_off
sm__ops_path_tensor_src_fp16_dst_fp32_sparsity_on
sm__ops_path_tensor_src_fp64
sm__ops_path_tensor_src_int1
sm__ops_path_tensor_src_int4
sm__ops_path_tensor_src_int4_sparsity_off
sm__ops_path_tensor_src_int4_sparsity_on
sm__ops_path_tensor_src_int8
sm__ops_path_tensor_src_int8_sparsity_off
sm__ops_path_tensor_src_int8_sparsity_on
sm__ops_path_tensor_src_tf32_dst_fp32
sm__ops_path_tensor_src_tf32_dst_fp32_sparsity_off
sm__ops_path_tensor_src_tf32_dst_fp32_sparsity_on
sm__pipe_alu_cycles_active
sm__pipe_fma_cycles_active
sm__pipe_fmaheavy_cycles_active
sm__pipe_fmalite_cycles_active
sm__pipe_fp64_cycles_active
sm__pipe_tensor_cycles_active
sm__pipe_tensor_op_hmma_cycles_active
sm__pipe_tensor_op_imma_cycles_active