Local Environment building

What you need

Hardware

  • 可用CUDA的设备(因此MACOS不能run 3DGS)
  • 足够的显存(最好24G)

Software

  • Conda
  • C++ Compiler for Pytorch extensions(eg. Visual Studio)
  • CUDA SDK 11 for Pytorch extensions
  • 所有的环境要版本适配

My environment version

  • conda 4.12.0
  • cuda 11.7
  • torch 1.12.1
  • torchaudio 0.12.1
  • torchvision 0.13.1
  • Visual Studio 2022
  • 8G显存

How to SETUP

1
2
3
SET DISTUTILS_USE_SDK=1 # Windows only
conda env create --file environment.yml
conda activate gaussian_splatting

  • Execute the above command, then it can automatically help you setup your environment, but maybe because my cuda version doesnot match with the provided official tutorial or I didnot install c++ compiler for visual environment, then I face with some problems to fix.’

  • Error message is mainly about the installation of submodule diff_gaussian_rasterization, simple_knn error

  • How to solve (install visual studio 2022 by hand)

    • First

      1
      conda install -c conda-forge vs2022_win-64

    • Then

      1
      2
      pip install submodules\diff-gaussian-rasterization
      pip install submodules\simple_knn

    • Or

      1
      conda env update --file environment.yml --name gaussian_splatting

  • Check

    1
    2
    3
    python
    import torch
    torch.cuda.is_available()

After all executions, then you can have your own coding environment to run 3DGS.

Other Extensions

  • Colmap 一款可以帮助你生成初始点云的工具, 帮助你生成起始的sfm points进而init你的3D Gaussian Colmap

    • You can directly download COLMAP-3.9.1-windows-cuda.zip
    • After downloading, you need to add it to env var. Add it to your path for using in your sys var (系统变量!!!).

  • ImageMagick 一款可以帮你resize image的工具, 让你的图片更适合训练 ImageMagicksys

    • You can directly download ImageMagick-7.1.1-34-Q16-HDRI-x64-dll.exe
    • After downloading, you need to add it to env var. Add it to your path for using in your sys var (系统变量!!!).

  • FFMPEG 一款可以帮你将视频抽帧成图片的工具, 允许你的video来进行训练 FFMPEG

    • You can just download ffmpeg-git-essentials.7z .ver .sha256
    • After downloading, you need to add it to env var. Add its bin to your path for using in your usr var (用户变量!!!).

  • Official Viewer官方提供的可视化工具，帮助你更好的观察最终的重建结果 Viewer

    • You can just download viewers
    • After downloading, you need to move it to your gaussian_splatting folder.

How to Run

Train

  • Step 1: activate your corresponding conda env
  • Step 2: cd to your gaussian-splatting folder
  • Step 3: python train.py -s your_input_Data_relative_path(注意你需要自己创建一个data folder，并且确保你的image集是undistorted)
1
2
3
conda activate gaussian_splatting
cd gaussian_splatting
python train.py -s data/input_data

Then waiting for a while, the result will be stored in the output folder with a random name.

Convert

For rasterization, the camera models must be either a SIMPLE_PINHOLE or PINHOLE camera. We provide a converter script convert.py, to extract undistorted images and SfM information from input images. Optionally, you can use ImageMagick to resize the undistorted images. This rescaling is similar to MipNeRF360, i.e., it creates images with 1/2, 1/4 and 1/8 the original resolution in corresponding folders. To use them, please first install a recent version of COLMAP (ideally CUDA-powered) and ImageMagick. Put the images you want to use in a directory /input.

(Need to focus on that for your source images, you need to put them under input folder!!!)

1
python convert.py -s <location> [--resize] # If doesnot need resizing, ImageMagick is not need

Video to Image

First, maybe you need to cd to your source video path, so that you can get your target images at the same level.

Then

1
ffmpeg -i absolute_path_to_your_source_video -qscale:v 1 -qmin 1 -vf fps=your_target_fps %04d.jpg

Then wait for a while, you can get your target image data set.(There are also many ffmpeg command for you to choose to get your target images. For example you can check your video relevant info and also you can choose you final image quality)

Viewer

1
2
cd viewers/bin
SIBR_gaussianViewer_app.exe -m absolute_path_to_your_output_result

Then it will open an exe, you can do many interations in it.

How to Prepare Your Own Scene

Expected Type

1
2
3
4
5
6
7
8
9
10
<location>
|---images
|   |---<image 0>
|   |---<image 1>
|   |---...
|---sparse
    |---0
        |---cameras.bin
        |---images.bin
        |---points3D.bin

Preparation (Using Colmap)

  1. 创建一个新的工程，为Database新建Database.db, 将Image路径设置为你image集合所在的folder
  2. 提取特征值: Processing -> Feature extraction, 将shared for all images勾选, first_octave设置为0, others by default.
  3. 特征匹配: Processing -> Feature matching (不同mode 的选择, eg. Sequential 顺序模式 or Exhaustive 穷举模式)
  4. 重建: Reconstruction -> Reconstruction options, 取消勾选 multiple_models
  5. 开始重建: Reconstruction -> Start reconstruction
  6. 导出模型: file -> Export model
  7. Remake folder structure
    1
    2
    3
    4
    5
    6
    7
    8
    9
    10
    <location>
    |---input
    |   |---<image 0>
    |   |---<image 1>
    |   |---...
    |---distorted
        |---database.db
        |---sparse
            |---0
                |---...
  8. 消除distortion: Run convert.py

Tricky

  1. 若显存不足的情况下，可以进行参数的修改:
    • Reduce --iteration
    • Increase the --densify_grad_threshold, --densification_interval
    • Reduce the --densify_until_iter
1
2
3
|---GAUSSIAN-SPLATTING
|   |---arguments
|       |---__init__.py
  1. 若显存不足/系统为MACOS, 可以在colab上部署代码然后运行，接着将output保存，从本地观看

Reference