smplify-x

Expressive Body Capture: 3D Hands, Face, and Body from a Single Image

2,038

365

2,038

View on GitHub

Top Related Projects

VIBE

3,137

Official implementation of CVPR2020 paper "VIBE: Video Inference for Human Body Pose and Shape Estimation"

frankmocap

2,259

A Strong and Easy-to-use Single View 3D Hand+Body Pose Estimator

openpose

33,406

OpenPose: Real-time multi-person keypoint detection library for body, face, hands, and foot estimation

Quick Overview

SMPLify-X is a Python-based framework for estimating 3D human pose and shape from a single image. It extends the SMPL body model to include face and hand details, providing a more comprehensive full-body reconstruction. The project aims to improve the accuracy and realism of 3D human pose estimation in various applications.

Pros

Offers full-body 3D reconstruction, including face and hand details
Provides state-of-the-art results in human pose and shape estimation
Supports both 2D and 3D joint positions as input
Includes pre-trained models for immediate use

Cons

Requires significant computational resources for processing
May struggle with complex poses or occlusions
Limited to single-person reconstruction
Depends on several external libraries and models

Code Examples

Loading the SMPLify-X model:

from smplify_x import SMPLify_X

smplify_x = SMPLify_X(
    step_size=1e-2,
    batch_size=1,
    num_iters=100,
    device='cuda'
)

Estimating pose and shape from 2D keypoints:

import torch

keypoints_2d = torch.tensor(...)  # 2D keypoints
camera = torch.tensor(...)  # Camera parameters

output = smplify_x(
    keypoints_2d=keypoints_2d,
    camera_translation=camera
)

Visualizing the results:

from smplify_x.utils import render_mesh

vertices = output['vertices']
faces = smplify_x.faces

render_mesh(vertices, faces, output['camera_translation'])

Getting Started

Clone the repository:

git clone https://github.com/vchoutas/smplify-x.git
cd smplify-x

Install dependencies:
```
pip install -r requirements.txt
```
Download pre-trained models:
```
./fetch_models.sh
```

Run the demo:

python demo.py --img_path path/to/image.jpg

Competitor Comparisons

VIBE

3,137

Official implementation of CVPR2020 paper "VIBE: Video Inference for Human Body Pose and Shape Estimation"

Pros of VIBE

Real-time performance: VIBE can process videos in real-time, making it suitable for live applications
End-to-end approach: Utilizes a temporal network for more coherent predictions across video frames
Easier to use: Requires less manual intervention and parameter tuning compared to SMPLify-X

Cons of VIBE

Limited to body pose estimation: Focuses primarily on body pose, while SMPLify-X includes hand and face models
Less detailed output: May not capture fine-grained details as accurately as SMPLify-X's optimization-based approach
Dependency on training data: Performance can be affected by the quality and diversity of the training dataset

Code Comparison

SMPLify-X (optimization-based approach):

optimizer = optim.Adam([body_pose, betas, global_orient, transl], lr=step_size)
for i in range(num_iters):
    loss = camera_fitting_loss(model_joints, camera_joints, camera_center, focal_length)
    optimizer.zero_grad()
    loss.backward()
    optimizer.step()

VIBE (learning-based approach):

model = VIBE_MODEL(seqlen=16)
pred_pose, pred_betas = model(input_frames)
pred_vertices = smpl(pred_pose, pred_betas)

Both repositories aim to estimate 3D human pose and shape, but they use different approaches. SMPLify-X employs an optimization-based method, while VIBE utilizes a learning-based approach with temporal coherence.

frankmocap

2,259

A Strong and Easy-to-use Single View 3D Hand+Body Pose Estimator

Pros of FrankMocap

Supports real-time motion capture for both body and hands
Includes pre-trained models for easier implementation
Offers a more user-friendly interface and documentation

Cons of FrankMocap

Less accurate for complex poses and occlusions
Limited customization options compared to SMPLify-X
Focuses primarily on single-person scenarios

Code Comparison

SMPLify-X:

smplx_model = smplx.create(model_path, model_type='smplx')
optimizer = optim.Adam(smplx_model.parameters(), lr=0.01)
for _ in range(num_iterations):
    loss = compute_loss(smplx_model, observations)
    optimizer.zero_grad()
    loss.backward()
    optimizer.step()

FrankMocap:

bodymocap = BodyMocap(regressor_checkpoint, smpl_dir)
handmocap = HandMocap(hand_regressor_checkpoint)
body_pose, body_bbox = bodymocap.regress(img)
hand_pose, hand_bbox = handmocap.regress(img, body_bbox)

SMPLify-X offers more flexibility and control over the optimization process, allowing for fine-tuning of parameters and custom loss functions. FrankMocap provides a simpler, more streamlined approach with pre-trained models, making it easier to implement but potentially less adaptable to specific use cases.

openpose

33,406

OpenPose: Real-time multi-person keypoint detection library for body, face, hands, and foot estimation

Pros of OpenPose

Real-time multi-person keypoint detection
Supports 2D and 3D pose estimation
Extensive documentation and community support

Cons of OpenPose

Limited to keypoint detection, doesn't provide full body mesh
Higher computational requirements for real-time processing

Code Comparison

OpenPose:

// Configuring OpenPose
op::Wrapper opWrapper{op::ThreadManagerMode::Asynchronous};
opWrapper.configure(opConfig);
opWrapper.start();

// Process and display image
opWrapper.emplaceAndPop(datumProcessed, imageToProcess);
cv::imshow("OpenPose", datumProcessed->cvOutputData);

SMPLify-X:

# Configuring SMPLify-X
smplify = SMPLify(step_size=1e-2, batch_size=1, num_iters=100)

# Process image and optimize SMPL model
camera_translation, opt_pose, opt_betas = smplify(
    keypoints_2d, conf_2d, init_pose, init_betas, camera_translation)

OpenPose focuses on real-time keypoint detection, while SMPLify-X aims to fit a full 3D body model to 2D keypoints. OpenPose is more suitable for applications requiring fast pose estimation, whereas SMPLify-X provides a more detailed body representation at the cost of increased processing time.

Convert designs to code with AI

Introducing Visual Copilot: A new AI model to turn Figma designs to high quality code using your components.

Try Visual Copilot

README

Expressive Body Capture: 3D Hands, Face, and Body from a Single Image

[Project Page] [Paper] [Supp. Mat.]

SMPL-X Examples

License
Description
Dependencies
Citation
Acknowledgments
Contact

License

Software Copyright License for non-commercial scientific research purposes. Please read carefully the terms and conditions and any accompanying documentation before you download and/or use the SMPL-X/SMPLify-X model, data and software, (the "Model & Software"), including 3D meshes, blend weights, blend shapes, textures, software, scripts, and animations. By downloading and/or using the Model & Software (including downloading, cloning, installing, and any other use of this github repository), you acknowledge that you have read these terms and conditions, understand them, and agree to be bound by them. If you do not agree with these terms and conditions, you must not download and/or use the Model & Software. Any infringement of the terms of this agreement will automatically terminate your rights under this License.

Disclaimer

The original images used for the figures 1 and 2 of the paper can be found in this link. The images in the paper are used under license from gettyimages.com. We have acquired the right to use them in the publication, but redistribution is not allowed. Please follow the instructions on the given link to acquire right of usage. Our results are obtained on the 483 Ã 724 pixels resolution of the original images.

Description

This repository contains the fitting code used for the experiments in Expressive Body Capture: 3D Hands, Face, and Body from a Single Image.

Fitting

Run the following command to execute the code:

python smplifyx/main.py --config cfg_files/fit_smplx.yaml 
    --data_folder DATA_FOLDER 
    --output_folder OUTPUT_FOLDER 
    --visualize="True/False"
    --model_folder MODEL_FOLDER
    --vposer_ckpt VPOSER_FOLDER
    --part_segm_fn smplx_parts_segm.pkl

where the DATA_FOLDER should contain two subfolders, images, where the images are located, and keypoints, where the OpenPose output should be stored.

Different Body Models

To fit SMPL or SMPL+H, replace the yaml configuration file with either fit_smpl.yaml or fit_smplx.yaml, i.e.:

for SMPL:

python smplifyx/main.py --config cfg_files/fit_smpl.yaml 
   --data_folder DATA_FOLDER 
   --output_folder OUTPUT_FOLDER 
   --visualize="True/False"
   --model_folder MODEL_FOLDER
   --vposer_ckpt VPOSER_FOLDER

for SMPL+H:

python smplifyx/main.py --config cfg_files/fit_smplh.yaml 
   --data_folder DATA_FOLDER 
   --output_folder OUTPUT_FOLDER 
   --visualize="True/False"
   --model_folder MODEL_FOLDER
   --vposer_ckpt VPOSER_FOLDER

Visualizing Results

To visualize the results produced by the method you can run the following script:

python smplifyx/render_results.py --mesh_fns OUTPUT_MESH_FOLDER

where OUTPUT_MESH_FOLDER is the folder that contains the resulting meshes.

Dependencies

Follow the installation instructions for each of the following before using the fitting code.

Optional Dependencies

PyTorch Mesh self-intersection for interpenetration penalty
- Download the per-triangle part segmentation: smplx_parts_segm.pkl (ctrl/cmd +F for the filename)
Trimesh for loading triangular meshes
Pyrender for visualization

The code has been tested with Python 3.6, CUDA 10.0, CuDNN 7.3 and PyTorch 1.0 on Ubuntu 18.04.

Citation

If you find this Model & Software useful in your research we would kindly ask you to cite:

@inproceedings{SMPL-X:2019,
  title = {Expressive Body Capture: 3D Hands, Face, and Body from a Single Image},
  author = {Pavlakos, Georgios and Choutas, Vasileios and Ghorbani, Nima and Bolkart, Timo and Osman, Ahmed A. A. and Tzionas, Dimitrios and Black, Michael J.},
  booktitle = {Proceedings IEEE Conf. on Computer Vision and Pattern Recognition (CVPR)},
  year = {2019}
}

Acknowledgments

LBFGS with Strong Wolfe Line Search

The LBFGS optimizer with Strong Wolfe Line search is taken from this Pytorch pull request. Special thanks to Du Phan for implementing this. We will update the repository once the pull request is merged.

Contact

The code of this repository was implemented by Vassilis Choutas and Georgios Pavlakos.

For questions, please contact smplx@tue.mpg.de.

For commercial licensing (and all related questions for business applications), please contact ps-licensing@tue.mpg.de.

Top Related Projects

Convert designs to code with AI

Introducing Visual Copilot: A new AI model to turn Figma designs to high quality code using your components.

Try Visual Copilot

Top Related Projects

Quick Overview

Pros

Cons

Code Examples

Getting Started

Competitor Comparisons

Pros of VIBE

Cons of VIBE

Code Comparison

Pros of FrankMocap

Cons of FrankMocap

Code Comparison

Pros of OpenPose

Cons of OpenPose

Code Comparison

Convert designs to code with AI

README

Expressive Body Capture: 3D Hands, Face, and Body from a Single Image

Table of Contents

License

Disclaimer

Description

Fitting

Different Body Models

Visualizing Results

Dependencies

Optional Dependencies

Citation

Acknowledgments

LBFGS with Strong Wolfe Line Search

Contact

Top Related Projects

Convert designs to code with AI