Class 3：RTMPose关键点检测

主讲人：张子豪（同济子豪兄）https://space.bilibili.com/1900783

课程地址：RTMPose关键点检测-安装MMDetection和MMPose_哔哩哔哩_bilibili

MMPose主页：https://github.com/open-mmlab/mmpose

教程地址：TommyZihao/MMPose_Tutorials: Jupyter notebook tutorials for mmpose (github.com)

概述

以三角板关键点检测场景为例，结合OpenMMLab开源目标检测算法库MMDetection、开源关键点检测算法库MMPose、开源模型部署算法库MMDeploy，全面讲解项目全流程：

• 数据集：Labelme标注数据集、整理标注格式至MS COCO
• 目标检测：分别训练Faster R CNN和RTMDet-Tiny目标检测模型、训练日志可视化、测试集评估、对图像、摄像头画面预测
• 关键点检测：训练RTMPose-S关键点检测模型、训练日志可视化、测试集上评估、分别对“图像、视频、摄像头画面”预测
• 模型终端部署：转ONNX格式，终端推理

视频教程合集：https://space.bilibili.com/1900783/channel/collectiondetail?sid=1316981

安装相关库

为了方便使用者快速上手 MMPose，这次课程有着丰富的示例代码脚本，以及详细的技术文档，包括安装、数据集准备、使用教程、常见问题解答等。通过这些示例和文档，我们可以迅速了解 MMPose 的主要功能和特性，以及在具体场景中的使用方法。

根据：MMPose_Tutorials/【A1】安装MMPose.ipynb at main · TommyZihao/MMPose_Tutorials · GitHub

安装MMDetection

按照顺序逐行运行下面的代码，即可安装配置 MMDetection 环境

https://github.com/TommyZihao/MMPose_Tutorials/blob/main/2023/0524/%E3%80%90A2%E3%80%91%E5%AE%89%E8%A3%85MMDetection.ipynb

预训练模型推理

模型库预训练模型

• 目标检测模型

MMDetection模型库：https://github.com/open-mmlab/mmdetection/blob/master/docs/en/model_zoo.md

demo/mmdetection_cfg/faster_rcnn_r50_fpn_coco.py

https://download.openmmlab.com/mmdetection/v2.0/faster_rcnn/faster_rcnn_r50_fpn_1x_coco/faster_rcnn_r50_fpn_1x_coco_20200130-047c8118.pth

• MMPose人体姿态估计模型

configs/body_2d_keypoint/topdown_heatmap/coco/td-hm_hrnet-w32_8xb64-210e_coco-256x192.py

https://download.openmmlab.com/mmpose/top_down/hrnet/hrnet_w32_coco_256x192-c78dce93_20200708.pth

• RTMPose人体姿态估计模型

RTMPose主页：https://github.com/open-mmlab/mmpose/tree/dev-1.x/projects/rtmpose

RTMPose-S

projects/rtmpose/rtmpose/body_2d_keypoint/rtmpose-s_8xb256-420e_coco-256x192.py

https://download.openmmlab.com/mmpose/v1/projects/rtmpose/rtmpose-s_simcc-aic-coco_pt-aic-coco_420e-256x192-fcb2599b_20230126.pth

RTMPose-L

projects/rtmpose/rtmpose/body_2d_keypoint/rtmpose-l_8xb256-420e_coco-384x288.py

https://download.openmmlab.com/mmpose/v1/projects/rtmpose/rtmpose-l_simcc-aic-coco_pt-aic-coco_420e-384x288-97d6cb0f_20230228.pth

预测单张图像

In [2]:

# HRNet
!python demo/topdown_demo_with_mmdet.py \
        demo/mmdetection_cfg/faster_rcnn_r50_fpn_coco.py \
        https://download.openmmlab.com/mmdetection/v2.0/faster_rcnn/faster_rcnn_r50_fpn_1x_coco/faster_rcnn_r50_fpn_1x_coco_20200130-047c8118.pth \
        configs/body_2d_keypoint/topdown_heatmap/coco/td-hm_hrnet-w32_8xb64-210e_coco-256x192.py \
        https://download.openmmlab.com/mmpose/top_down/hrnet/hrnet_w32_coco_256x192-c78dce93_20200708.pth \
        --input data/test/multi-person.jpeg \
        --output-root outputs/B1_HRNet_1 \
        --device cuda:0 \
        --bbox-thr 0.5 \
        --kpt-thr 0.2 \
        --nms-thr 0.3 \
        --radius 8 \
        --thickness 4 \
        --draw-bbox \
        --draw-heatmap \
        --show-kpt-idx

预测视频：

直接将--input换成视频路径即可

MMPose官方可视化工具visualizer

img = mmcv.imread(img_path)
img = mmcv.imconvert(img, 'bgr', 'rgb')

img_output = visualizer.add_datasample(
            'result',
            img,
            data_sample=data_samples,
            draw_gt=False,
            draw_heatmap=True,
            draw_bbox=True,
            show_kpt_idx=True,
            show=False,
            wait_time=0,
            out_file='outputs/B2.jpg'
)

展示可视化效果

MMDetection三角板目标检测

下载三角板关键点检测数据集

下载用于测试的图像和视频

!mkdir data/test_triangle

# 图像-30度直角三角板，拍摄：同济子豪兄、田文博
!wget https://zihao-openmmlab.obs.myhuaweicloud.com/20220610-mmpose/triangle_dataset/test_img/triangle_1.jpg -P data/test_triangle
!wget https://zihao-openmmlab.obs.myhuaweicloud.com/20220610-mmpose/triangle_dataset/test_img/triangle_2.jpg -P data/test_triangle
!wget https://zihao-openmmlab.obs.myhuaweicloud.com/20220610-mmpose/triangle_dataset/test_img/triangle_3.jpg -P data/test_triangle
!wget https://zihao-openmmlab.obs.myhuaweicloud.com/20220610-mmpose/triangle_dataset/test_img/triangle_4.jpg -P data/test_triangle

# 视频-30度直角三角板，拍摄：同济子豪兄，田文博
!wget https://zihao-openmmlab.obs.myhuaweicloud.com/20220610-mmpose/triangle_dataset/videos/triangle_6.mp4 -P data/test_triangle
!wget https://zihao-openmmlab.obs.myhuaweicloud.com/20220610-mmpose/triangle_dataset/videos/triangle_7.mp4 -P data/test_triangle
!wget https://zihao-openmmlab.obs.myhuaweicloud.com/20220610-mmpose/triangle_dataset/videos/triangle_9.mp4 -P data/test_triangle

查看数据集中的图片

# from PIL import Image
# Image.open('data/Triangle_215_Keypoint_coco/images/DSC_0373.jpg')

三角板目标检测-训练

进入mmdetection主目录

import os
os.chdir('mmdetection')

目标检测算法：Faster R CNN

# 建议在命令行中运行
!python tools/train.py data/faster_r_cnn_triangle.py

目标检测算法：RTMDet

# 建议在命令行中运行
!python tools/train.py data/rtmdet_tiny_triangle.py

在work_dirs目录下，查看训练日志和训练得到的模型权重文件

测试集上评估模型精度

!python tools/test.py data/rtmdet_tiny_triangle.py \
                      work_dirs/rtmdet_tiny_triangle/epoch_200.pth

训练集损失函数

df_train.columns

Index(['lr', 'data_time', 'loss', 'loss_rpn_cls', 'loss_rpn_bbox', 'loss_cls',
       'acc', 'loss_bbox', 'time', 'epoch', 'memory', 'step'],
      dtype='object')

metrics = ['loss', 'loss_bbox', 'loss_cls', 'loss_rpn_cls', 'loss_rpn_bbox']

plt.figure(figsize=(16, 8))

x = df_train['step']
for y in metrics:
    plt.plot(x, df_train[y], label=y, **get_line_arg())

plt.tick_params(labelsize=20)
plt.xlabel('step', fontsize=20)
plt.ylabel('loss', fontsize=20)
plt.title('训练集损失函数', fontsize=25)
plt.savefig('训练集损失函数.pdf', dpi=120, bbox_inches='tight')

plt.legend(fontsize=20)

plt.show()

训练集准确率

metrics = ['acc']

plt.figure(figsize=(16, 8))

x = df_train['step']
for y in metrics:
    plt.plot(x, df_train[y], label=y, **get_line_arg())

plt.tick_params(labelsize=20)
plt.xlabel('step', fontsize=20)
plt.ylabel('loss', fontsize=20)
plt.title('训练集准确率', fontsize=25)
plt.savefig('训练集准确率.pdf', dpi=120, bbox_inches='tight')

plt.legend(fontsize=20)

plt.show()

​

​

测试集评估指标-MS COCO Metric

df_test.columns

Index(['coco/bbox_mAP', 'coco/bbox_mAP_50', 'coco/bbox_mAP_75',
       'coco/bbox_mAP_s', 'coco/bbox_mAP_m', 'coco/bbox_mAP_l',
       'pascal_voc/mAP', 'pascal_voc/AP50', 'data_time', 'time', 'step'],
      dtype='object')

metrics = ['coco/bbox_mAP', 'coco/bbox_mAP_50', 'coco/bbox_mAP_75', 'coco/bbox_mAP_s', 'coco/bbox_mAP_m', 'coco/bbox_mAP_l']

plt.figure(figsize=(16, 8))

x = df_test['step']
for y in metrics:
    plt.plot(x, df_test[y], label=y, **get_line_arg())

plt.tick_params(labelsize=20)
# plt.ylim([0, 100])
plt.xlabel('Epoch', fontsize=20)
plt.ylabel(y, fontsize=20)
plt.title('测试集评估指标', fontsize=25)
plt.savefig('测试集分类评估指标.pdf', dpi=120, bbox_inches='tight')

plt.legend(fontsize=20)

plt.show()

​

测试集评估指标-PASCAL VOC Metric

metrics = ['pascal_voc/mAP', 'pascal_voc/AP50']

plt.figure(figsize=(16, 8))

x = df_test['step']
for y in metrics:
    plt.plot(x, df_test[y], label=y, **get_line_arg())

plt.tick_params(labelsize=20)
# plt.ylim([0, 100])
plt.xlabel('Epoch', fontsize=20)
plt.ylabel(y, fontsize=20)
plt.title('测试集评估指标', fontsize=25)
plt.savefig('测试集分类评估指标.pdf', dpi=120, bbox_inches='tight')

plt.legend(fontsize=20)

plt.show()

三角板目标检测-预测

三角板目标检测-模型权重文件精简转换

经过精简转换之后，模型.pth权重文件大小缩小为原来的一半以上，但不影响推理结果和推理速度

进入 mmdetection 主目录

import os
os.chdir('mmdetection')

模型轻量化转换

# Faster R CNN
!python tools/model_converters/publish_model.py \
        work_dirs/faster_r_cnn_triangle/epoch_50.pth \
        checkpoint/faster_r_cnn_triangle_epoch_50_202305120846.pth

05/12 08:46:49 - mmengine - [4m[37mINFO[0m - Key `message_hub` will be removed because it is not in save_keys. If you want to keep it, please set --save-keys.
05/12 08:46:49 - mmengine - [4m[37mINFO[0m - Key `optimizer` will be removed because it is not in save_keys. If you want to keep it, please set --save-keys.
05/12 08:46:49 - mmengine - [4m[37mINFO[0m - Key `param_schedulers` will be removed because it is not in save_keys. If you want to keep it, please set --save-keys.
05/12 08:46:50 - mmengine - [4m[37mINFO[0m - The published model is saved at checkpoint/faster_r_cnn_triangle_epoch_50_202305120846-76d9dde3.pth.

# RTMDet-tiny
!python tools/model_converters/publish_model.py \
        work_dirs/rtmdet_tiny_triangle/epoch_200.pth \
        checkpoint/rtmdet_tiny_triangle_epoch_200_202305120847.pth

05/12 08:47:08 - mmengine - [4m[37mINFO[0m - Key `message_hub` will be removed because it is not in save_keys. If you want to keep it, please set --save-keys.
05/12 08:47:08 - mmengine - [4m[37mINFO[0m - Key `optimizer` will be removed because it is not in save_keys. If you want to keep it, please set --save-keys.
05/12 08:47:08 - mmengine - [4m[37mINFO[0m - Key `param_schedulers` will be removed because it is not in save_keys. If you want to keep it, please set --save-keys.
05/12 08:47:08 - mmengine - [4m[37mINFO[0m - Key `ema_state_dict` will be removed because it is not in save_keys. If you want to keep it, please set --save-keys.
05/12 08:47:08 - mmengine - [4m[37mINFO[0m - The published model is saved at checkpoint/rtmdet_tiny_triangle_epoch_200_202305120847-3cd02a8f.pth.

模型权重文件保存在checkpoint目录

进入mmdetection主目录

import os
os.chdir('mmdetection')

目标检测预测-单张图像

# Faster R CNN
!python demo/image_demo.py \
        data/test_triangle/triangle_3.jpg \
        data/faster_r_cnn_triangle.py \
        --weights checkpoint/faster_r_cnn_triangle_epoch_50_202305120846-76d9dde3.pth \
        --out-dir outputs/E2_faster_r_cnn \
        --device cuda:0 \
        --pred-score-thr 0.3

Loads checkpoint by local backend from path: checkpoint/faster_r_cnn_triangle_epoch_50_202305120846-76d9dde3.pth
05/18 10:35:17 - mmengine - [5m[4m[33mWARNING[0m - Failed to search registry with scope "mmdet" in the "function" registry tree. As a workaround, the current "function" registry in "mmengine" is used to build instance. This may cause unexpected failure when running the built modules. Please check whether "mmdet" is a correct scope, or whether the registry is initialized.
05/18 10:35:17 - mmengine - [5m[4m[33mWARNING[0m - `Visualizer` backend is not initialized because save_dir is None.
[2KInference [91m━[0m[35m━[0m[90m━[0m[90m━[0m[90m━[0m[90m━[0m[90m━[0m[90m━[0m[90m━[0m[90m━[0m[90m━[0m[90m━[0m[90m━[0m[35m━[0m[91m━[0m[91m━[0m[91m━[0m[91m━[0m[91m━[0m[91m━[0m[91m━[0m[35m━[0m[90m━[0m[90m━[0m[90m━[0m[90m━[0m[90m━[0m[90m━[0m[90m━[0m[90m━[0m[90m━[0m[90m━[0m[90m━[0m[35m━[0m[91m━[0m[91m━[0m[91m━[0m[91m━[0m[91m━[0m[91m━[0m  [36m [0m
[?25hresults have been saved at outputs/E2_faster_r_cnn

# RTMDet
!python demo/image_demo.py \
        data/test_triangle/triangle_3.jpg \
        data/rtmdet_tiny_triangle.py \
        --weights checkpoint/rtmdet_tiny_triangle_epoch_200_202305120847-3cd02a8f.pth \
        --out-dir outputs/E2_rtmdet \
        --device cuda:0 \
        --pred-score-thr 0.3

Loads checkpoint by local backend from path: checkpoint/rtmdet_tiny_triangle_epoch_200_202305120847-3cd02a8f.pth
05/18 10:35:32 - mmengine - [5m[4m[33mWARNING[0m - Failed to search registry with scope "mmdet" in the "function" registry tree. As a workaround, the current "function" registry in "mmengine" is used to build instance. This may cause unexpected failure when running the built modules. Please check whether "mmdet" is a correct scope, or whether the registry is initialized.
05/18 10:35:32 - mmengine - [5m[4m[33mWARNING[0m - `Visualizer` backend is not initialized because save_dir is None.
[2K/environment/miniconda3/lib/python3.7/site-packages/torch/functional.py:445: [0m[91m━[0m[91m━[0m[35m━[0m[90m━[0m[90m━[0m[90m━[0m[90m━[0m[90m━[0m[90m━[0m[90m━[0m[90m━[0m[90m━[0m[90m━[0m[90m━[0m[35m━[0m[91m━[0m[91m━[0m[91m━[0m[91m━[0m[91m━[0m[91m━[0m[91m━[0m[35m━[0m[90m━[0m[90m━[0m[90m━[0m[90m━[0m[90m━[0m[90m━[0m[90m━[0m[90m━[0m[90m━[0m[90m━[0m  [36m [0m
UserWarning: torch.meshgrid: in an upcoming release, it will be required to pass
the indexing argument. (Triggered internally at  
../aten/src/ATen/native/TensorShape.cpp:2157.)
  return _VF.meshgrid(tensors, **kwargs)  # type: ignore[attr-defined]
[2KInference [91m━[0m[91m━[0m[91m━[0m[35m━[0m[90m━[0m[90m━[0m[90m━[0m[90m━[0m[90m━[0m[90m━[0m[90m━[0m[90m━[0m[90m━[0m[90m━[0m[90m━[0m[35m━[0m[91m━[0m[91m━[0m[91m━[0m[91m━[0m[91m━[0m[91m━[0m[91m━[0m[35m━[0m[90m━[0m[90m━[0m[90m━[0m[90m━[0m[90m━[0m[90m━[0m[90m━[0m[90m━[0m[90m━[0m[90m━[0m[90m━[0m[35m━[0m[91m━[0m[91m━[0m[91m━[0m[91m━[0m  [36m [0m
[?25hresults have been saved at outputs/E2_rtmdet

目标检测预测-视频

# Faster R CNN
!python demo/video_demo.py \
        data/test_triangle/triangle_9.mp4 \
        data/faster_r_cnn_triangle.py \
        checkpoint/faster_r_cnn_triangle_epoch_50_202305120846-76d9dde3.pth \
        --device cuda:0 \
        --score-thr 0.96 \
        --out outputs/E2_out_video_faster_r_cnn.mp4

Loads checkpoint by local backend from path: checkpoint/faster_r_cnn_triangle_epoch_50_202305120846-76d9dde3.pth
05/18 10:35:47 - mmengine - [5m[4m[33mWARNING[0m - `Visualizer` backend is not initialized because save_dir is None.
[>>>>>>>>>>>>>                 ] 142/318, 3.4 task/s, elapsed: 42s, ETA:    52s/environment/miniconda3/lib/python3.7/site-packages/mmengine/visualization/visualizer.py:759: UserWarning: Warning: The bbox is out of bounds, the drawn bbox may not be in the image
  ' the drawn bbox may not be in the image', UserWarning)
/environment/miniconda3/lib/python3.7/site-packages/mmengine/visualization/visualizer.py:830: UserWarning: Warning: The polygon is out of bounds, the drawn polygon may not be in the image
  ' the drawn polygon may not be in the image', UserWarning)
[>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>] 318/318, 3.4 task/s, elapsed: 93s, ETA:     0s

# RTMDet
!python demo/video_demo.py \
        data/test_triangle/triangle_9.mp4 \
        data/rtmdet_tiny_triangle.py \
        checkpoint/rtmdet_tiny_triangle_epoch_200_202305120847-3cd02a8f.pth \
        --device cuda:0 \
        --score-thr 0.6 \
        --out outputs/E2_out_video_rtmdet.mp4

Loads checkpoint by local backend from path: checkpoint/rtmdet_tiny_triangle_epoch_200_202305120847-3cd02a8f.pth
05/18 10:37:30 - mmengine - [5m[4m[33mWARNING[0m - `Visualizer` backend is not initialized because save_dir is None.
[                                                  ] 0/318, elapsed: 0s, ETA:/environment/miniconda3/lib/python3.7/site-packages/torch/functional.py:445: UserWarning: torch.meshgrid: in an upcoming release, it will be required to pass the indexing argument. (Triggered internally at  ../aten/src/ATen/native/TensorShape.cpp:2157.)
  return _VF.meshgrid(tensors, **kwargs)  # type: ignore[attr-defined]
[>>>>>>>>>>>>>                 ] 142/318, 4.0 task/s, elapsed: 36s, ETA:    45s/environment/miniconda3/lib/python3.7/site-packages/mmengine/visualization/visualizer.py:759: UserWarning: Warning: The bbox is out of bounds, the drawn bbox may not be in the image
  ' the drawn bbox may not be in the image', UserWarning)
/environment/miniconda3/lib/python3.7/site-packages/mmengine/visualization/visualizer.py:830: UserWarning: Warning: The polygon is out of bounds, the drawn polygon may not be in the image
  ' the drawn polygon may not be in the image', UserWarning)
[>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>] 318/318, 4.0 task/s, elapsed: 80s, ETA:     0s

三角板关键点检测

过程和detection相似，这选取一些不同的地方：

训练RTMPose

同济子豪兄 2023-4-3 5-23

进入 mmpose 主目录

import os
os.chdir('mmpose')

训练（建议在命令行中运行）

!python tools/train.py data/rtmpose-s-triangle.py

测试集上评估模型精度

!python tools/test.py data/rtmpose-s-triangle.py \
                      work_dirs/rtmpose-s-triangle/epoch_300.pth

in the image’, UserWarning)
[>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>] 318/318, 4.0 task/s, elapsed: 80s, ETA: 0s

三角板关键点检测

过程和detection相似，这选取一些不同的地方：

训练RTMPose

同济子豪兄 2023-4-3 5-23

进入 mmpose 主目录

import os
os.chdir('mmpose')

训练（建议在命令行中运行）

!python tools/train.py data/rtmpose-s-triangle.py

测试集上评估模型精度

!python tools/test.py data/rtmpose-s-triangle.py \
                      work_dirs/rtmpose-s-triangle/epoch_300.pth