A repository for generating stylized talking 3D and 3D face

Overview

style_avatar

A repository for generating stylized talking 3D faces and 2D videos. This is the repository for paper Imitating Arbitrary Talking Style for Realistic Audio-Driven Talking Face Synthesis, MM 2021 The demo video can be viewed in this link: https://hcsi.cs.tsinghua.edu.cn/demo/MM21-HAOZHEWU.mp4.

framework


Quick start

Installation

conda create -n python36 python=3.6 
conda activate python36
  • Install necessary packages through pip install -r requirements.txt
  • Download the pretrained deepspeech model from the Link, and then unzip the zipped file to ./deepspeech folder.
  • Same as the instructions of Deep 3D Face Reconstruction.
    • Download the Basel Face Model. Due to the license agreement of Basel Face Model, you have to download the BFM09 model after submitting an application on its home page. After getting the access to BFM data, download "01_MorphableModel.mat" and put it into ./deep_3drecon/BFM subfolder.
    • Download Download the Expression Basis provided by Guo et al. You can find a link named "CoarseData" in the first row of Introduction part in their repository. Download and unzip the Coarse_Dataset.zip. Put "Exp_Pca.bin" into ./deep_3drecon/BFM subfolder. The expression basis are constructed using Facewarehouse data and transferred to BFM topology. Download the pre-trained reconstruction network, unzip it and put "FaceReconModel.pb" into ./deep_3drecon/network subfolder.
    • Run git lfs checkout ./deep_3drecon/BFM/BFM_model_front.mat
  • Download the pretrained audio2motion model, put it into ./audio2motion/model
  • Download the pretrained texture encoder and render, put it into ./render/model

Run

To run our demo, you need at least one GPU with 11G GPU memory.

python demo.py --in_img [*.png] --in_audio [*.wav] --output_path [path]

We provide 10 example talking styles in style.npy, you can also calculate your own style codes with the following code. Where the exp is the 3DMM series and pose is the pose matrix reconstructed from Deep 3D Face Reconstruction. Usually we calculate style codes with videos of 5-20 seconds.

def get_style_code(exp, pose):
  exp_mean_std = pkl.load(open("./data/ted_hd/exp_mean_std.pkl", 'rb'))
  exp_std_mean = exp_mean_std['s_m']
  exp_std_std = exp_mean_std['s_s']
  exp_diff_std_mean = exp_mean_std['d_s_m']
  exp_diff_std_std = exp_mean_std['d_s_s']

  pose_mean_std = pkl.load(open("./data/ted_hd/pose_mean_std.pkl", 'rb'))
  pose_diff_std_mean = pose_mean_std['d_s_m']
  pose_diff_std_std = pose_mean_std['d_s_s']

  diff_exp = exp[:-1, :] - exp[1:, :]
  exp_std = (np.std(exp, axis = 0) - exp_std_mean) / exp_std_std
  diff_exp_std = (np.std(diff_exp, axis = 0) - exp_diff_std_mean) / exp_diff_std_std

  diff_pose = pose[:-1, :] - pose[1:, :]
  diff_pose_std = (np.std(diff_pose, axis = 0) - pose_diff_std_mean) / pose_diff_std_std

  return np.concatenate((exp_std, diff_exp_std, diff_pose_std))

Notice that the pose of each talking face is static in current demo, you can control the pose of face by modifying the coeff_array in demo.py in line 93. The coeff_array has shape of $N * 257$ , where $N$ is framesize, vector of $257$ dimensions has same definition as deep 3d face reconstruction, where $254-257$ dim controls the translation, and $224-227$ dim controls euler angles for pose.


Project Overview

Our project organizes the files as follows:

├── README.md
├── data_process
├── deepspeech
├── face_alignment
├── deep_3drecon
├── render
├── audio2motion

Data process

The data process folder contains processing code of several datasets.

DeepSpeech

We leverage the DeepSpeech project to extract audio related features. Please download the pretrained deepspeech model from the Link. In deepspeech/evaluate.py, we implement the funtion get_prob to get the latent deepspeech features with input audio path. The latent deepspeech features have 50 frames per second. We should align the deepspeech features to 25 fps videos in subsequent implementations.

Face Alignment

We modify Face Alignment for data preprocess. Different from the original project, we enforce the face alignment to detect only the largest face in each frame for speed-up.

Deep 3D Face Reconstruction

We modify Deep 3D Face Reconstruction for data preprocess. We add batch-api, uv-texture unwarpping api and uv coodinate image generation api in deep_3drecon/utils.py.

Render

We implement our texture encoder and rendering model in the render folder. We also implement some other renders like neural voice puppertry.

Audio to Motion

We implement our stylized audio to facial motion model in audio2motion folder.


Data

Ted-HD data

We leverage lmdb to store the fragmented data. The data can be downloaded from link, and then run cat xa* > data.mdb. You can obtain the train/test video with the code bellow. We use the Ted-HD data to train the audio2motion model. We also provide the reconstructed 3D param and landmarks in the lmdb.

import lmdb

def test():
    lmdb_path = "./lmdb"
    env = lmdb.open(lmdb_path, map_size=1099511627776, max_dbs = 64)

    train_video = env.open_db("train_video".encode())
    train_audio = env.open_db("train_audio".encode())
    train_lm5 = env.open_db("train_lm5".encode())
    test_video = env.open_db("test_video".encode())
    test_audio = env.open_db("test_audio".encode())
    test_lm5 = env.open_db("test_lm5".encode())

    with env.begin(write = False) as txn:
        video = txn.get(str(0).encode(), db=test_video)
        audio = txn.get(str(0).encode(), db=test_audio)
        video_file = open("test.mp4", "wb")
        audio_file = open("test.wav", "wb")
        video_file.write(video)
        audio_file.write(audio)
        video_file.close()
        audio_file.close()
        print(txn.stat(db=train_video))
        print(txn.stat(db=test_video)) # we can obtain the database size here  

For the training of render, we will not provide the processed dataset due to the license of LRW.


Citation

@inproceedings{wu2021imitating,
  title={Imitating Arbitrary Talking Style for Realistic Audio-Driven Talking Face Synthesis},
  author={Wu, Haozhe and Jia, Jia and Wang, Haoyu and Dou, Yishun and Duan, Chao and Deng, Qingshan},
  booktitle={Proceedings of the 29th ACM International Conference on Multimedia},
  pages={1478--1486},
  year={2021}
}

Further works

  • Current render is still buggy, there are noisy dots in the synthesized videos, we will fix this problem.
  • We will optimize the rendering results of particular person with video footage of only 2-3 seconds.
  • We will blend the synthesized results with backgrounds.
  • We will add controllable dynamic textures and light control.
Comments
  • tex_encode.pkl load failed

    tex_encode.pkl load failed

    hi, i have loaded 'backbone.pkl' successfully, but failed to load tex_encode.pkl in the same way, the error is "./render/model/tex_encoder.pkl is a zip archive (did you mean to use torch.jit.load()?)", is the model upload error ? @wuhaozhe

    opened by liyuanyaun 11
  • Getting the UV mapping from UVAtlas

    Getting the UV mapping from UVAtlas

    @wuhaozhe Thank you very much for sharing your code! I have a question on how you generate the UV mappings In the paper, you have mentioned that you use UVAtlas, but as I am going through your demo.py code, it seems that the UV maps are created by passing the 3DMM face model to the google Mesh_UV renderer. Just to be exact: In this method of the Face3D, the mesh_uv from mesh_renderer is called. Could you elaborate more on this, on how the UV maps of shape (H, W, 2) are generated.

    opened by Armen-J 9
  • unknown mat file type, version 49, 50

    unknown mat file type, version 49, 50

    when i load "BFM_model_front.mat", the error "unknown mat file type, version 49, 50" happened? can you share what version scipy you install @wuhaozhe

    opened by liyuanyaun 3
  • question

    question

    This is amazing work. I was wondering, how would I add eye blinking to the generated output video? Would I need to train a new talking style?

    thank you

    opened by skunkwerk 2
  • The code implementation is inconsistent with the paper

    The code implementation is inconsistent with the paper

    Thank you share awesome work! In the code, you extract the speech features and energy features from the audio clips. However, in your paper, you only mentioned leveraging the DeepSpeech model to extract speech features. Could you give me some advice for the above situation?

    opened by xiao-keeplearning 2
  • Any ideas for imitating the expression, head and body movement at the same time?

    Any ideas for imitating the expression, head and body movement at the same time?

    Thanks for your nice work. I have a question: If I want to imitate the expression, head and body movement at the same time (Given a source full-body image, a driving full-body video and a corresponding audio), any good ideas?

    opened by aishoot 0
  • RuntimeError: CUDA error: out of memory CUDA kernel errors might be asynchronously reported at some other API call,so the stacktrace below might be incorrect. For debugging consider passing CUDA_LAUNCH_BLOCKING=1.

    RuntimeError: CUDA error: out of memory CUDA kernel errors might be asynchronously reported at some other API call,so the stacktrace below might be incorrect. For debugging consider passing CUDA_LAUNCH_BLOCKING=1.

    Hi! Thanks for your great work! Hovewer, I faced problem when I run the python demo.py --in_img ./example/example.png --in_audio ./example/example.wav --output_path ./output Can, you please, give me some tips to overcome this issue?

    opened by muxiddin19 2
  • Reading the LMDB data issue

    Reading the LMDB data issue

    I have download the xa* data and followed the commands to create "data.mdb", but I run this code:

    env = lmdb.open(lmdb_path, map_size=1099511627776, max_dbs = 64)
    train_video = env.open_db("train_video".encode())
    

    I get this error: "lmdb.PageNotFoundError: mdb_dbi_open: MDB_PAGE_NOTFOUND: Requested page not found"

    Well, it seems strange because the data.mdb is almost 17 gigabytes

    When I print the env.stat() I get this: {'psize': 4096, 'depth': 1, 'branch_pages': 0, 'leaf_pages': 1, 'overflow_pages': 0, 'entries': 35}

    opened by Armen-J 6
  • error

    error

    Downloading: "https://www.adrianbulat.com/downloads/python-fan/2DFAN4-11f355bf06.pth.tar" to /root/.cache/torch/hub/checkpoints/2DFAN4-11f355bf06.pth.tar
    100% 91.2M/91.2M [00:05<00:00, 19.1MB/s]
    WARNING: Logging before flag parsing goes to stderr.
    W1124 05:18:17.878163 140281471424384 module_wrapper.py:139] From /content/style_avatar/deep_3drecon/utils.py:68: The name tf.placeholder is deprecated. Please use tf.compat.v1.placeholder instead.
    
    W1124 05:18:17.879213 140281471424384 module_wrapper.py:139] From /content/style_avatar/deep_3drecon/utils.py:14: The name tf.gfile.GFile is deprecated. Please use tf.io.gfile.GFile instead.
    
    W1124 05:18:17.879425 140281471424384 module_wrapper.py:139] From /content/style_avatar/deep_3drecon/utils.py:15: The name tf.GraphDef is deprecated. Please use tf.compat.v1.GraphDef instead.
    
    W1124 05:18:18.932533 140281471424384 module_wrapper.py:139] From /content/style_avatar/deep_3drecon/face_decoder.py:129: The name tf.cross is deprecated. Please use tf.linalg.cross instead.
    
    W1124 05:18:18.933541 140281471424384 deprecation.py:506] From /content/style_avatar/deep_3drecon/face_decoder.py:131: calling l2_normalize (from tensorflow.python.ops.nn_impl) with dim is deprecated and will be removed in a future version.
    Instructions for updating:
    dim is deprecated, use axis instead
    W1124 05:18:19.503320 140281471424384 deprecation.py:323] From /content/style_avatar/deep_3drecon/mesh_renderer/mesh_renderer.py:165: where (from tensorflow.python.ops.array_ops) is deprecated and will be removed in a future version.
    Instructions for updating:
    Use tf.where in 2.0, which has the same broadcast rule as np.where
    W1124 05:18:19.969749 140281471424384 module_wrapper.py:139] From /content/style_avatar/deep_3drecon/utils.py:85: The name tf.GPUOptions is deprecated. Please use tf.compat.v1.GPUOptions instead.
    
    W1124 05:18:19.970036 140281471424384 module_wrapper.py:139] From /content/style_avatar/deep_3drecon/utils.py:86: The name tf.Session is deprecated. Please use tf.compat.v1.Session instead.
    
    W1124 05:18:19.970237 140281471424384 module_wrapper.py:139] From /content/style_avatar/deep_3drecon/utils.py:86: The name tf.ConfigProto is deprecated. Please use tf.compat.v1.ConfigProto instead.
    
    2021-11-24 05:18:19.983109: I tensorflow/core/platform/profile_utils/cpu_utils.cc:94] CPU Frequency: 2299995000 Hz
    2021-11-24 05:18:19.983801: I tensorflow/compiler/xla/service/service.cc:168] XLA service 0x5565daed5d40 initialized for platform Host (this does not guarantee that XLA will be used). Devices:
    2021-11-24 05:18:19.983842: I tensorflow/compiler/xla/service/service.cc:176]   StreamExecutor device (0): Host, Default Version
    2021-11-24 05:18:19.987771: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcuda.so.1
    2021-11-24 05:18:19.992980: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:983] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
    2021-11-24 05:18:19.993832: I tensorflow/compiler/xla/service/service.cc:168] XLA service 0x5565daed5b80 initialized for platform CUDA (this does not guarantee that XLA will be used). Devices:
    2021-11-24 05:18:19.993868: I tensorflow/compiler/xla/service/service.cc:176]   StreamExecutor device (0): Tesla K80, Compute Capability 3.7
    2021-11-24 05:18:19.994966: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:983] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
    2021-11-24 05:18:19.995591: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1639] Found device 0 with properties: 
    name: Tesla K80 major: 3 minor: 7 memoryClockRate(GHz): 0.8235
    pciBusID: 0000:00:04.0
    2021-11-24 05:18:20.022505: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudart.so.10.1
    2021-11-24 05:18:20.211194: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcublas.so.10
    2021-11-24 05:18:20.237375: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcufft.so.10
    2021-11-24 05:18:20.260865: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcurand.so.10
    2021-11-24 05:18:20.509169: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcusolver.so.10
    2021-11-24 05:18:20.528118: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcusparse.so.10
    2021-11-24 05:18:20.896984: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudnn.so.7
    2021-11-24 05:18:20.897234: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:983] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
    2021-11-24 05:18:20.898117: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:983] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
    2021-11-24 05:18:20.898852: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1767] Adding visible gpu devices: 0
    2021-11-24 05:18:20.902326: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudart.so.10.1
    2021-11-24 05:18:20.903938: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1180] Device interconnect StreamExecutor with strength 1 edge matrix:
    2021-11-24 05:18:20.903995: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1186]      0 
    2021-11-24 05:18:20.904026: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1199] 0:   N 
    2021-11-24 05:18:20.905518: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:983] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
    2021-11-24 05:18:20.906512: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:983] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
    2021-11-24 05:18:20.907194: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1325] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:0 with 10199 MB memory) -> physical GPU (device: 0, name: Tesla K80, pci bus id: 0000:00:04.0, compute capability: 3.7)
    2021-11-24 05:18:24.483044: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:983] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
    2021-11-24 05:18:24.483631: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1639] Found device 0 with properties: 
    name: Tesla K80 major: 3 minor: 7 memoryClockRate(GHz): 0.8235
    pciBusID: 0000:00:04.0
    2021-11-24 05:18:24.483758: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudart.so.10.1
    2021-11-24 05:18:24.483833: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcublas.so.10
    2021-11-24 05:18:24.483898: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcufft.so.10
    2021-11-24 05:18:24.483972: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcurand.so.10
    2021-11-24 05:18:24.484039: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcusolver.so.10
    2021-11-24 05:18:24.484101: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcusparse.so.10
    2021-11-24 05:18:24.484165: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudnn.so.7
    2021-11-24 05:18:24.484282: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:983] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
    2021-11-24 05:18:24.484860: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:983] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
    2021-11-24 05:18:24.485335: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1767] Adding visible gpu devices: 0
    2021-11-24 05:18:24.486420: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:983] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
    2021-11-24 05:18:24.486925: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1639] Found device 0 with properties: 
    name: Tesla K80 major: 3 minor: 7 memoryClockRate(GHz): 0.8235
    pciBusID: 0000:00:04.0
    2021-11-24 05:18:24.487000: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudart.so.10.1
    2021-11-24 05:18:24.487065: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcublas.so.10
    2021-11-24 05:18:24.487127: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcufft.so.10
    2021-11-24 05:18:24.487192: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcurand.so.10
    2021-11-24 05:18:24.487254: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcusolver.so.10
    2021-11-24 05:18:24.487315: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcusparse.so.10
    2021-11-24 05:18:24.487376: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudnn.so.7
    2021-11-24 05:18:24.487489: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:983] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
    2021-11-24 05:18:24.488060: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:983] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
    2021-11-24 05:18:24.488592: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1767] Adding visible gpu devices: 0
    2021-11-24 05:18:24.488651: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1180] Device interconnect StreamExecutor with strength 1 edge matrix:
    2021-11-24 05:18:24.488687: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1186]      0 
    2021-11-24 05:18:24.488714: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1199] 0:   N 
    2021-11-24 05:18:24.488897: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:983] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
    2021-11-24 05:18:24.489464: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:983] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
    2021-11-24 05:18:24.489974: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1325] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:0 with 10199 MB memory) -> physical GPU (device: 0, name: Tesla K80, pci bus id: 0000:00:04.0, compute capability: 3.7)
    /content/style_avatar/align_img.py:21: FutureWarning: `rcond` parameter will change to the default of machine precision times ``max(M, N)`` where M and N are the input matrix dimensions.
    To use the future default and silence this warning we advise to pass `rcond=None`, to keep using the old, explicitly pass `rcond=-1`.
      k,_,_,_ = np.linalg.lstsq(A,b)
    /content/style_avatar/align_img.py:97: VisibleDeprecationWarning: Creating an ndarray from ragged nested sequences (which is a list-or-tuple of lists-or-tuples-or ndarrays with different lengths or shapes) is deprecated. If you meant to do this, you must specify 'dtype=object' when creating the ndarray
      trans_params = np.array([w0,h0,102.0/s,t[0],t[1]])
    2021-11-24 05:18:27.965252: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudnn.so.7
    2021-11-24 05:18:30.431787: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcublas.so.10
    2021-11-24 05:18:32.221454: I tensorflow/stream_executor/cuda/cuda_driver.cc:831] failed to allocate 4.00G (4294967296 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory
    2021-11-24 05:18:32.233011: I tensorflow/stream_executor/cuda/cuda_driver.cc:831] failed to allocate 3.60G (3865470464 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory
    2021-11-24 05:18:32.244708: I tensorflow/stream_executor/cuda/cuda_driver.cc:831] failed to allocate 3.24G (3478923264 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory
    2021-11-24 05:18:32.255462: I tensorflow/stream_executor/cuda/cuda_driver.cc:831] failed to allocate 2.92G (3131030784 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory
    2021-11-24 05:18:32.268217: I tensorflow/stream_executor/cuda/cuda_driver.cc:831] failed to allocate 2.62G (2817927680 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory
    2021-11-24 05:18:32.280419: I tensorflow/stream_executor/cuda/cuda_driver.cc:831] failed to allocate 2.36G (2536134912 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory
    2021-11-24 05:18:32.280486: W tensorflow/core/common_runtime/bfc_allocator.cc:305] Garbage collection: deallocate free memory regions (i.e., allocations) so that we can re-allocate a larger region to avoid OOM due to memory fragmentation. If you see this message frequently, you are running near the threshold of the available device memory and re-allocation may incur great performance overhead. You may try smaller batch sizes to observe the performance impact. Set TF_ENABLE_GPU_GARBAGE_COLLECTION=false if you'd like to disable this feature.
    2021-11-24 05:18:34.034494: W tensorflow/core/framework/cpu_allocator_impl.cc:81] Allocation of 75497472 exceeds 10% of system memory.
    rm: cannot remove '/content/outt/*.png': No such file or directory
    2021-11-24 05:18:35.471827: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:983] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
    2021-11-24 05:18:35.472222: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1639] Found device 0 with properties: 
    name: Tesla K80 major: 3 minor: 7 memoryClockRate(GHz): 0.8235
    pciBusID: 0000:00:04.0
    2021-11-24 05:18:35.472343: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudart.so.10.1
    2021-11-24 05:18:35.472443: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcublas.so.10
    2021-11-24 05:18:35.472528: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcufft.so.10
    2021-11-24 05:18:35.472621: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcurand.so.10
    2021-11-24 05:18:35.472738: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcusolver.so.10
    2021-11-24 05:18:35.472816: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcusparse.so.10
    2021-11-24 05:18:35.472911: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudnn.so.7
    2021-11-24 05:18:35.473096: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:983] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
    2021-11-24 05:18:35.476636: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:983] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
    2021-11-24 05:18:35.477987: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1767] Adding visible gpu devices: 0
    2021-11-24 05:18:35.479073: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1180] Device interconnect StreamExecutor with strength 1 edge matrix:
    2021-11-24 05:18:35.479127: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1186]      0 
    2021-11-24 05:18:35.479157: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1199] 0:   N 
    2021-11-24 05:18:35.479720: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:983] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
    2021-11-24 05:18:35.480132: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:983] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
    2021-11-24 05:18:35.480518: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1325] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:0 with 10199 MB memory) -> physical GPU (device: 0, name: Tesla K80, pci bus id: 0000:00:04.0, compute capability: 3.7)
    2021-11-24 05:18:36.163937: W tensorflow/core/framework/cpu_allocator_impl.cc:81] Allocation of 57802752 exceeds 10% of system memory.
    FATAL Flags parsing error: Unknown command line flag 'in_img'
    Pass --helpshort or --helpfull to see help on flags.
    
    opened by loboere 7
  • 如何调整成对中文输入也支持?

    如何调整成对中文输入也支持?

    大佬您好,感谢您的分享! 如果想尝试调整成对中文输入也适用的方案,不知道要调整哪些内容? 这边也看到其他的有类似的疑问: https://github.com/wuhaozhe/style_avatar/issues/1#issuecomment-968499818 “The deepspeech is trained on English, you can test it in Chinese, but the result wouldn't be satisfactory.” 还有一个疑问,输入的音频是训练集中不存在的,那么音画同步效果如何?

    opened by DWCTOD 1
Releases(0.1)
Owner
Haozhe Wu
Research interests in Computer Vision and Machine Learning.
Haozhe Wu
2 Jul 19, 2022
This repository contains the code for the paper "PIFu: Pixel-Aligned Implicit Function for High-Resolution Clothed Human Digitization"

PIFu: Pixel-Aligned Implicit Function for High-Resolution Clothed Human Digitization News: [2020/05/04] Added EGL rendering option for training data g

Shunsuke Saito 1.5k Jan 03, 2023
A deep learning network built with TensorFlow and Keras to classify gender and estimate age.

Convolutional Neural Network (CNN). This repository contains a source code of a deep learning network built with TensorFlow and Keras to classify gend

Pawel Dziemiach 1 Dec 19, 2021
SCAN: Learning to Classify Images without Labels, incl. SimCLR. [ECCV 2020]

Learning to Classify Images without Labels This repo contains the Pytorch implementation of our paper: SCAN: Learning to Classify Images without Label

Wouter Van Gansbeke 1.1k Dec 30, 2022
Code Impementation for "Mold into a Graph: Efficient Bayesian Optimization over Mixed Spaces"

Code Impementation for "Mold into a Graph: Efficient Bayesian Optimization over Mixed Spaces" This repo contains the implementation of GEBO algorithm.

Jaeyeon Ahn 2 Mar 22, 2022
Plover-tapey-tape: an alternative to Plover’s built-in paper tape

plover-tapey-tape plover-tapey-tape is an alternative to Plover’s built-in paper

7 May 29, 2022
Build tensorflow keras model pipelines in a single line of code. Created by Ram Seshadri. Collaborators welcome. Permission granted upon request.

deep_autoviml Build keras pipelines and models in a single line of code! Table of Contents Motivation How it works Technology Install Usage API Image

AutoViz and Auto_ViML 102 Dec 17, 2022
A library for Deep Learning Implementations and utils

deeply A Deep Learning library Table of Contents Features Quick Start Usage License Features Python 2.7+ and Python 3.4+ compatible. Quick Start $ pip

Achilles Rasquinha 1 Dec 12, 2022
Grammar Induction using a Template Tree Approach

Gitta Gitta ("Grammar Induction using a Template Tree Approach") is a method for inducing context-free grammars. It performs particularly well on data

Thomas Winters 36 Nov 15, 2022
This is a Image aid classification software based on python TK library development

This is a Image aid classification software based on python TK library development.

EasonChan 1 Jan 17, 2022
Range Image-based LiDAR Localization for Autonomous Vehicles Using Mesh Maps

Range Image-based 3D LiDAR Localization This repo contains the code for our ICRA2021 paper: Range Image-based LiDAR Localization for Autonomous Vehicl

Photogrammetry & Robotics Bonn 208 Dec 15, 2022
An unreferenced image captioning metric (ACL-21)

UMIC This repository provides an unferenced image captioning metric from our ACL 2021 paper UMIC: An Unreferenced Metric for Image Captioning via Cont

hwanheelee 14 Nov 20, 2022
GeDML is an easy-to-use generalized deep metric learning library

GeDML is an easy-to-use generalized deep metric learning library

Borui Zhang 32 Dec 05, 2022
Unofficial implementation of MLP-Mixer: An all-MLP Architecture for Vision

MLP-Mixer: An all-MLP Architecture for Vision This repo contains PyTorch implementation of MLP-Mixer: An all-MLP Architecture for Vision. Usage : impo

Rishikesh (ऋषिकेश) 175 Dec 23, 2022
Annotated notes and summaries of the TensorFlow white paper, along with SVG figures and links to documentation

TensorFlow White Paper Notes Features Notes broken down section by section, as well as subsection by subsection Relevant links to documentation, resou

Sam Abrahams 437 Oct 09, 2022
SMCA replication There are no extra compiled components in SMCA DETR and package dependencies are minimal

Usage There are no extra compiled components in SMCA DETR and package dependencies are minimal, so the code is very simple to use. We provide instruct

22 May 06, 2022
Aerial Single-View Depth Completion with Image-Guided Uncertainty Estimation (RA-L/ICRA 2020)

Aerial Depth Completion This work is described in the letter "Aerial Single-View Depth Completion with Image-Guided Uncertainty Estimation", by Lucas

ETHZ V4RL 70 Dec 22, 2022
AI-UPV at IberLEF-2021 EXIST task: Sexism Prediction in Spanish and English Tweets Using Monolingual and Multilingual BERT and Ensemble Models

AI-UPV at IberLEF-2021 EXIST task: Sexism Prediction in Spanish and English Tweets Using Monolingual and Multilingual BERT and Ensemble Models Descrip

Angel de Paula 1 Jun 08, 2022
Urban mobility simulations with Python3, RLlib (Deep Reinforcement Learning) and Mesa (Agent-based modeling)

Deep Reinforcement Learning for Smart Cities Documentation RLlib: https://docs.ray.io/en/master/rllib.html Mesa: https://mesa.readthedocs.io/en/stable

1 May 15, 2022
(CVPR 2021) PAConv: Position Adaptive Convolution with Dynamic Kernel Assembling on Point Clouds

PAConv: Position Adaptive Convolution with Dynamic Kernel Assembling on Point Clouds by Mutian Xu*, Runyu Ding*, Hengshuang Zhao, and Xiaojuan Qi. Int

CVMI Lab 228 Dec 25, 2022