VToonify is an AI software that converts a video with facial movement to a stylized animation. It is built on StyleGAN. You can find the original paper here. This tutorial gives you a step-by-step guide on how to use the code in GitHub to turn your video into a cartoon. Note: VToonify works only with face videos.
1. First download the code from VToonify GitHub repository. Open a dos prompt, in the directory where you want to install, type the command:
>git clone https://github.com/williamyang1991/VToonify.git
2. Download checkpoints. You should read README.md in the “VToonify\checkpoint” directory. It tells you the folder structure of checkpoints.
3. Go to hugging face or Google drive to download “directions.npy”, “encoder.pt”, “faceparsing.pth” and “shape_predictor_68_face_landmarks.dat.” Put them under “checkpoint” root directory.
4. There are a few pre-trained models for different styles, such as cartoon, comic, illustration, pixar, etc. You can see the samples of each style at huggingface space. I suggest you try one of them first. “vtoonify_d_cartoon” has the best results. Go to vtoonify_d_cartoon at hugging face, download four files there and put under your “checkpoint\vtoonify_d_cartoon” directory.
5. You need to configure a virtual environment to run the code. If you haven’t installed Anaconda3, go to install Anaconda3.
6. Setup a conda environment with the instruction.
7. If you are going to run with Nvidia GPU on windows, you need to install VS2022 buildtools. Set the path of cl.exe in the environment variables. The instruction and issues are explained here.
8. I suggest you run the code using CPU. The speed is acceptable. In order to run on CPU, you change 3 files: “VToonify\util.py”, “VToonify\model\simple_augment.py”, “VToonify/model/stylegan/model.py.” In these files, change “model.stylegan.op” to “model.stylegan.op_cpu.” See the detailed instruction here.
9. Prepare a short video mp4 or gif file with a clear face movement. Put it in “data” directory. You can also use the existing input file in “data” as well, for example “651.mp4.”
10. Open an Anaconda Prompt. Run command:
>conda activate stylegan3_env
11. Still in the Anaconda prompt, go to the directory “VToonify” and run command:
>python style_transfer.py --scale_image --content ./data/651.mp --video
12. When it finishes, the new video is saved at “output” directory.
13. If you cannot play the mp4 file due to encoding differences, import the image sequence to After Effects or other video editing tools to render as a video.
14. (Optional) Now you can download other pre-trained models and check out their results.
