Optimizes image files by converting them to webp while also updating all references.

Overview

About

Optimizes images by (re-)saving them as webp. For every file it replaced it automatically updates all references. Works on single files as well as on folders.

Installation: Windows

Step 1: Get the exe

Option 1: Download

Download one of the pre-built fvttoptimizer.exe files from Releases

Go to Step 2: Install program

Option 2: Build it yourself

Install python3 (version >= 3.8) and add it to your PATH system environment variable.

You can check the version via CMD or powershell with:

python --version

Download or clone the repo.

Open a CMD instance inside the project root folder and create a virtual environment.

python -m venv .\venv

Activate the venv:

.\venv\Scripts\activate

Install the Pillow package:

pip install pillow==9.1.0

Install pyInstaller package:

pip install pyInstaller==4.10

Download the fvttmv python package (whl file) from here: https://github.com/watermelonwolverine/fvttmv/releases/tag/0.2.4 move it into the folder and install it:

pip install fvttmv-0.2.4-py3-none-any.whl

Run build file:

.\scripts\build_for_windows.cmd

You should now have a fvttoptimizer.exe file under dist. After the build succeeded you can delete the venv you previously created.

Go to Step 2: Install program

Step 2: Install program

Create an empty folder where you want to install the program for example C:\fvttoptimizer

Copy fvttoptimizer.exe into that folder.

Create a fvttoptimizer.conf text file in that folder.

Copy {"absolute_path_to_foundry_data":"INSERT_PATH_HERE"} into fvttoptimizer.conf

Replace INSERT_PATH_HERE with the path to the Data folder inside your foundrydata (Not the foundrydata folder itself!).

IMPORTANT: Escape all \ with \\ in that path.

It should look something like this:

{"absolute_path_to_foundry_data":"C:\\Users\\user\\foundrydata\\Data"}

Add the installation path to your PATH system environment variable.

Uninstallation: Windows

Delete fvttoptimizer.exe and fvttoptimizer.conf files from the installation directory.

Remove the path to the installation directory from the PATH system environment variable.

Installation: Ubuntu 16.04 -20.04

Optional: Install Python

An up-to-date Ubuntu 20.04 should have python >= 3.8. Check it with:

python3 --version

From here on pythonX will be used as placeholder for the python you should use. Depending on your system you need to replace X with 3, 3.8, 3.9 or 3.10 .

Install python version>=3.8 if not yet installed.

sudo apt install pythonX

Step 1: Get the executable

Option 1: Download

Download one of the pre-built fvttoptimizer files from Releases

Go to Step 2: Install the files

Option 2: Build it yourself

Install python if haven't already.

Install Pillow dependencies (read more here https://pillow.readthedocs.io/en/stable/installation.html):

sudo apt install pythonX-dev libtiff5-dev libjpeg8-dev libopenjp2-7-dev zlib1g-dev libfreetype6-dev liblcms2-dev libwebp-dev tcl8.6-dev tk8.6-dev python3-tk libharfbuzz-dev libfribidi-dev libxcb1-dev

Download or clone repo.

Create venv inside the repo:

sudo apt install pythonX-venv

pythonX -m venv ./venv

Activate venv:

source venv/bin/activate

Install Pillow (should succeed if you installed all the dependencies):

pythonX -m pip install Pillow==9.1.0

Install pyInstaller:

pythonX -m pip install pyInstaller==4.10

Download the fvttmv python package (whl file) from here: https://github.com/watermelonwolverine/fvttmv/releases/tag/0.2.4 move it into the folder and install it:

pythonX -m pip install fvttmv-0.2.4-py3-none-any.whl

Now run the build script:

./scrips/build_for_ubuntu.sh

Step 2: Install the files

Option 1: Automatic installation

Install python if you haven't already.

Clone the repo if you haven't already.

Go into the project folder.

Build the project if you haven't already. If you downloaded a pre-built executable. Create a folder named dist and move the file there.

Inside the project folder run:

sudo pythonX scripts/install_on_ubuntu.sh

Option 2: Manual installation

Copy the fvttoptimizer file either from dist from where you downloaded it to usr/bin/fvttoptimizer

Make the file executable:

sudo chmod ugo=rx usr/bin/fvttoptimizer

Create a fvttoptimizer.conf file at /etc/

Copy {"absolute_path_to_foundry_data":"INSERT_PATH_HERE"} into fvttoptimizer.conf

Replace INSERT_PATH_HERE with the path to the Data folder inside your foundrydata (Not the foundrydata folder itself!).

It should look something like this:

{"absolute_path_to_foundry_data":"/home/user/foundrydata/Data"}

Uninstallation: Ubuntu

Delete the files /etc/fvttoptimizer.conf and /usr/bin/fvttoptimizer

Usage

IMPORTANT:

Shut down Foundry VTT before optimizing any files.

Use this program at your own risk. It has been tested extensively but there are no guarantees. Always keep backup of your Foundry VTT data for cases where something goes wrong.

Syntax

fvttoptimizer [--verbose-info, --verbose-debug, --version, --help, --quality value, --override-percent value, --skip-webp, --skip-existing, --recursive] target

target: Path to file or directory which should be optimized

Options

--verbose-info: Enables verbose output to console

--verbose-debug: Enables very verbose output to console

--version: Prints version and exits

--help: Display help and exit

--skip-existing: Ignores files of which a webp already exists. For example image.png will not be converted if a image.webp is already in the same folder

--skip-webp: Don't touch webp files at all

--quality: The quality setting for the webp compression. Default is 75

--override-percent: How much smaller the new file needs to be to replace the old one. For example if this value is 25 the file size after optimizing needs to be 25% smaller than the original. Default is 25

--recursive: If the optimization should be done recursively to all sub folders of the target folder.

Examples

Optimizing a file:

Ubuntu: fvttoptimizer path/to/file.png

Windows: fvttoptimizer path\to\file.png

Optimizing a folder and all it's sub folders:

Ubuntu: fvttoptimizer --recursive pathto/folder

Windows: fvttoptimizer.exe --recursive path\to\folder

Use single quotes when moving a file with a space:

Ubuntu: fvttoptimizer 'some folder/some file.jpg'

Windows: fvttoptimizer.exe 'some folder\some file.jpg'

Change parameters:

Ubuntu: fvttoptimizer --quality 50 --override-percent 50 file.jpg

Windows: fvttoptimizer --quality 50 --override-percent 50 file.jpg

Known Issues and Quirks

Windows

The program only works in powershell not in cmd.

When one of the paths has \' at the end, the arguments will get mixed up. This is a problem with how python handles arguments and probably can't be fixed. For example on Windows fvttoptimizer.exe '\folder name with spaces\' will fail but fvttoptimizer.exe '\folder name with spaces' will succeed.

Comments
  • Program fails on special character [ in path

    Program fails on special character [ in path

    First off, thank you so much for making this.

    I am trying to use the following command: fvttoptimizer.exe --recursive 'C:\Users\cnguy\AppData\Local\FoundryVTT\Data\Battle Maps'

    and I am getting this return: An unexpected error occurred: expected str, bytes or os.PathLike object, not NoneType

    Your documentation implies this is supported, as this command works without the "--recursive" flag, so maybe there is an issue with spaces in paths and the recursive function?

    Thank you.

    opened by coboba123 5
  • Trying to create a Docker image - Keep getting 'image.jpg is not in the configured foundry Data folder.'

    Trying to create a Docker image - Keep getting 'image.jpg is not in the configured foundry Data folder.'

    As the title say, I'm trying to get this tool to work in a Docker container, but I'm having an issue that I can't figure out...

    I'm building the image like so:

    FROM python:3.10.0
    
    ARG FVTT_OPTIMIZER_VERSION=0.1.3
    
    RUN curl -L https://github.com/watermelonwolverine/fvttoptimizer/releases/download/$FVTT_OPTIMIZER_VERSION/fvttoptimizer --output usr/bin/fvttoptimizer
    RUN chmod +x usr/bin/fvttoptimizer
    RUN echo '{"absolute_path_to_foundry_data":"/app/Data"}' > /etc/fvttoptimizer.conf
    
    CMD ["fvttoptimizer"]
    

    Which I then configure with docker-compose:

    version: "3.7"
    
    services:
        fvttoptimizer:
            build: python
            volumes:
                - ${FOUNDRY_DATA_VOLUME}:/app
    
    # FOUNDRY_DATA_VOLUME is the directory that contains `Config`, `Data` and `Logs`
    

    However, when ran (docker-compose run --rm fvttoptimizer fvttoptimizer --verbose-debug ./Barney.jpg) I get the error in the title. Running docker-compose run --rm fvttoptimizer ls -la /app/Data, I can confirm the image exists.

    The debug output looks like this:

    root - DEBUG - Got arguments ['fvttoptimizer', '--verbose-debug', '--recursive', './Barney.jpg']
    root - DEBUG - Reading config from: /etc/fvttoptimizer.conf
    root - DEBUG - Running with target_path='./Barney.jpg', config='{'_RunConfig__program_config': {'_ProgramConfigImpl__abs_path_to_foundry_data': '/app/Data'}, 'skip_existing': False, 'skip_webp': False, 'recursive': True, 'quality': 75, 'override_percent': 25}'
    root - ERROR - /Barney.jpg is not in the configured foundry Data folder.
    /Barney.jpg is not in the configured foundry Data folder.
    

    I pushed a branch on my repo where it should be possible to reproduce the issue, or at the very least look at my configuration: https://github.com/gCardinal/foundrty-vtt/blob/feature/use-fvttoptimizer

    support 
    opened by gCardinal 3
  • Missing config file

    Missing config file

    When running on an image file I get the error:

    Missing config file. Could not find /etc/fvttoptimizer.conf

    Tried running all three release versions on Arch Linux with Python 3.10.4 and get the same error every time.

    I'm not seeing any mention of a config file in the readme. Am I missing something?

    support 
    opened by tonydero 2
  • Suggestion: Print which file it is working on during the conversion process.

    Suggestion: Print which file it is working on during the conversion process.

    I think it would be nice to have some kind of progress indicator to show what its working on in the command prompt window. The easiest way to have this would simply just print the name of the file that is being worked on when it gets to it. When working on folders with lots of images, its nice to know its not frozen or hanging on something.

    Thank you!

    opened by coboba123 2
  • getting root - ERROR - [path] is not in the configured foundry Data folder.

    getting root - ERROR - [path] is not in the configured foundry Data folder.

    So I am running on a Google Cloud server, and have manually installed fvttoptimizer with Python 3.8.

    I created the /etc/fvttoptimizer.conf with the following line:

    {"absolute_path_to_foundry_data":"/home/tehjoester/foundrydata/Data"}

    and here is my output with the --verbose-debug option:

    [email protected]:~$ sudo fvttoptimizer --verbose-debug --skip-webp --recursive 'foundrydata/Data/'
    root - DEBUG - Got arguments ['fvttoptimizer', '--verbose-debug', '--skip-webp', '--recursive', 'foundrydata/Data/']
    root - DEBUG - Reading config from: /etc/fvttoptimizer.conf
    root - DEBUG - Running with target_path='foundrydata/Data/', config='{'_RunConfig__program_config': {'_ProgramConfigImpl__abs_path_to_foundry_data': '/home/tehjoester/foundrydata/Data'}, 'skip_existing': False, 'skip_webp': True, 'recursive': True, 'quality': 75, 'override_percent': 25}'
    root - ERROR - /home/tehjoester/foundrydata/Data is not in the configured foundry Data folder.
    /home/tehjoester/foundrydata/Data is not in the configured foundry Data folder.
    
    opened by thejoester 1
  • Argument --skip-webp does not seem to work

    Argument --skip-webp does not seem to work

    It's entirely possible that I am doing something wrong, but when I try to use --skip-webp it still tries to process webp files. Running the optimizer twice in a row shows that it will attempt the same files again even if they've already been converted.

    I am running Windows 11. This is the command I tried to run in Windows Terminal/PowerShell. Variations on this command have not resolved the issue.

    fvttoptimizer.exe --verbose-info --skip-webp --quality 100 --override-percent 10 --recursive 'C:\Games\Foundry Virtual Tabletop Data\Data\worlds\mitera'

    opened by seanms1991 0
  • Some actors are not getting the directory to their new avatar image updated properly

    Some actors are not getting the directory to their new avatar image updated properly

    While most actors have their pictures updated as expected, some are skipped. From what I can tell, it's mostly the avatars of the actors and not the tokens. The images are updated but the reference is not updated in the actors.db. Attached are some examples of actors that have this issue. Their tokens were created with Tokenizer, but those converted successfully and their main picture did not, which was not created with tokenizer. I've uploaded the pictures that were converted, the actor data exported as json files, and the actors.db files they were in both before and after the optimizer processed it.

    I am running Windows 11 and have tried to run this on two separate computers with the same result.

    If you need more information, let me know.

    Example.zip

    opened by seanms1991 0
  • Feature Request

    Feature Request

    First off, this script is amazing. It's greatly reduced the sizes of my images directories.

    It would be really cool though if I could maybe put in the config file the path to personal modules that it will update the file names. I have several shared compendiums in my own module that I move between worlds.

    enhancement 
    opened by Jonwh25 1
  • Program fails when a db file contains a byte UTF-8 can't decode.

    Program fails when a db file contains a byte UTF-8 can't decode.

    I got the following message in powershell:

    `PS C:\FVTTOptimizer> .\fvttoptimizer.exe --verbose-info --skip-webp --recursive "C:\Users\Scott-Server\AppData\Local\FoundryVTT\Data\Worlds\bakja" root - INFO - Optimizing directory 'C:\Users\Scott-Server\AppData\Local\FoundryVTT\Data\worlds\bakja' root - INFO - Optimizing directory 'C:\Users\Scott-Server\AppData\Local\FoundryVTT\Data\worlds\bakja\Assets' root - INFO - Optimizing directory 'C:\Users\Scott-Server\AppData\Local\FoundryVTT\Data\worlds\bakja\Assets\Map' root - INFO - Optimizing C:\Users\Scott-Server\AppData\Local\FoundryVTT\Data\worlds\bakja\Assets\Map\Animal Market Stall.png. Decreased file size by 83% root - ERROR - Traceback (most recent call last): File "cli_wrapper\main.py", line 235, in main File "cli_wrapper\main.py", line 229, in do_run File "cli_wrapper\main.py", line 81, in perform_optimization_with File "fvttoptimizer\optimizer.py", line 42, in optimize File "fvttoptimizer__directory_optimizer.py", line 38, in optimize File "fvttoptimizer__directory_optimizer.py", line 38, in optimize File "fvttoptimizer__directory_optimizer.py", line 35, in optimize File "fvttoptimizer__file_optimizer.py", line 34, in maybe_optimize File "fvttoptimizer__file_optimizer.py", line 48, in __maybe_optimize2 File "fvttoptimizer__file_optimizer.py", line 68, in __optimize File "fvttoptimizer__file_optimizer.py", line 100, in __replace_file_with_webp File "fvttmv\update\references_updater.py", line 22, in replace_references File "fvttmv\update\references_updater.py", line 34, in _do_replace_references File "fvttmv\update__references_updater_file.py", line 21, in replace_references_in_file File "codecs.py", line 322, in decode UnicodeDecodeError: 'utf-8' codec can't decode byte 0xd0 in position 0: invalid continuation byte

    An internal error occurred: Traceback (most recent call last): File "cli_wrapper\main.py", line 235, in main File "cli_wrapper\main.py", line 229, in do_run File "cli_wrapper\main.py", line 81, in perform_optimization_with File "fvttoptimizer\optimizer.py", line 42, in optimize File "fvttoptimizer__directory_optimizer.py", line 38, in optimize File "fvttoptimizer__directory_optimizer.py", line 38, in optimize File "fvttoptimizer__directory_optimizer.py", line 35, in optimize File "fvttoptimizer__file_optimizer.py", line 34, in maybe_optimize File "fvttoptimizer__file_optimizer.py", line 48, in __maybe_optimize2 File "fvttoptimizer__file_optimizer.py", line 68, in __optimize File "fvttoptimizer__file_optimizer.py", line 100, in __replace_file_with_webp File "fvttmv\update\references_updater.py", line 22, in replace_references File "fvttmv\update\references_updater.py", line 34, in _do_replace_references File "fvttmv\update__references_updater_file.py", line 21, in replace_references_in_file File "codecs.py", line 322, in decode UnicodeDecodeError: 'utf-8' codec can't decode byte 0xd0 in position 0: invalid continuation byte

    Please file a bug report on https://github.com/watermelonwolverine/fvttoptimizer/issues`

    bug 
    opened by YourDNDPleasesMe 10
Releases(0.1.3)
Owner
Watermelon Wolverine
Watermelon Wolverine
Calibrate your listeners! Robust communication-based training for pragmatic speakers. Findings of EMNLP 2021.

Calibrate your listeners! Robust communication-based training for pragmatic speakers Rose E. Wang, Julia White, Jesse Mu, Noah D. Goodman Findings of

Rose E. Wang 3 Apr 02, 2022
Official PyTorch code of DeepPanoContext: Panoramic 3D Scene Understanding with Holistic Scene Context Graph and Relation-based Optimization (ICCV 2021 Oral).

DeepPanoContext (DPC) [Project Page (with interactive results)][Paper] DeepPanoContext: Panoramic 3D Scene Understanding with Holistic Scene Context G

Cheng Zhang 66 Nov 16, 2022
Code for the KDD 2021 paper 'Filtration Curves for Graph Representation'

Filtration Curves for Graph Representation This repository provides the code from the KDD'21 paper Filtration Curves for Graph Representation. Depende

Machine Learning and Computational Biology Lab 16 Oct 16, 2022
Transformer Tracking (CVPR2021)

TransT - Transformer Tracking [CVPR2021] Official implementation of the TransT (CVPR2021) , including training code and trained models. We are revisin

chenxin 465 Jan 06, 2023
Official PyTorch Implementation of Hypercorrelation Squeeze for Few-Shot Segmentation, arXiv 2021

Hypercorrelation Squeeze for Few-Shot Segmentation This is the implementation of the paper "Hypercorrelation Squeeze for Few-Shot Segmentation" by Juh

Juhong Min 165 Dec 28, 2022
Advanced yabai wooting scripts

Yabai Wooting scripts Installation requirements Both https://github.com/xiamaz/python-yabai-client and https://github.com/xiamaz/python-wooting-rgb ne

Max Zhao 3 Dec 31, 2021
This YoloV5 based model is fit to detect people and different types of land vehicles, and displaying their density on a fitted map, according to their coordinates and detected labels.

This YoloV5 based model is fit to detect people and different types of land vehicles, and displaying their density on a fitted map, according to their

Liron Bdolah 8 May 22, 2022
This project is for a Twitter bot that monitors a bird feeder in my backyard. Any detected birds are identified and posted to Twitter.

Backyard Birdbot Introduction This is a silly hobby project to use existing ML models to: Detect any birds sighted by a webcam Identify whic

Chi Young Moon 71 Dec 25, 2022
Official implementation of the Implicit Behavioral Cloning (IBC) algorithm

Implicit Behavioral Cloning This codebase contains the official implementation of the Implicit Behavioral Cloning (IBC) algorithm from our paper: Impl

Google Research 210 Dec 09, 2022
An updated version of virtual model making

Model-Swap-Face v2   这个项目是基于stylegan2 pSp制作的,比v1版本Model-Swap-Face在推理速度和图像质量上有一定提升。主要的功能是将虚拟模特进行环球不同区域的风格转换,目前转换器提供西欧模特、东亚模特和北非模特三种主流的风格样式,可帮我们实现生产资料零成

seeprettyface.com 62 Dec 09, 2022
A Repository of Community-Driven Natural Instructions

A Repository of Community-Driven Natural Instructions TLDR; this repository maintains a community effort to create a large collection of tasks and the

AI2 244 Jan 04, 2023
Code for MentorNet: Learning Data-Driven Curriculum for Very Deep Neural Networks

MentorNet: Learning Data-Driven Curriculum for Very Deep Neural Networks This is the code for the paper: MentorNet: Learning Data-Driven Curriculum fo

Google 302 Dec 23, 2022
MIMIC Code Repository: Code shared by the research community for the MIMIC-III database

MIMIC Code Repository The MIMIC Code Repository is intended to be a central hub for sharing, refining, and reusing code used for analysis of the MIMIC

MIT Laboratory for Computational Physiology 1.8k Dec 26, 2022
This repository accompanies our paper “Do Prompt-Based Models Really Understand the Meaning of Their Prompts?”

This repository accompanies our paper “Do Prompt-Based Models Really Understand the Meaning of Their Prompts?” Usage To replicate our results in Secti

Albert Webson 64 Dec 11, 2022
OBBDetection: an oriented object detection toolbox modified from MMdetection

OBBDetection note: If you have questions or good suggestions, feel free to propose issues and contact me. introduction OBBDetection is an oriented obj

MIXIAOXIN_HO 3 Nov 11, 2022
Convert Mission Planner (ArduCopter) Waypoint Missions to Litchi CSV Format to execute on DJI Drones

Mission Planner to Litchi Convert Mission Planner (ArduCopter) Waypoint Surveys to Litchi CSV Format to execute on DJI Drones Litchi doesn't support S

Yaros 24 Dec 09, 2022
Grounding Representation Similarity with Statistical Testing

Grounding Representation Similarity with Statistical Testing This repo contains code to replicate the results in our paper, which evaluates representa

26 Dec 02, 2022
Causal Influence Detection for Improving Efficiency in Reinforcement Learning

Causal Influence Detection for Improving Efficiency in Reinforcement Learning This repository contains the code release for the paper "Causal Influenc

Autonomous Learning Group 21 Nov 29, 2022
Tooling for GANs in TensorFlow

TensorFlow-GAN (TF-GAN) TF-GAN is a lightweight library for training and evaluating Generative Adversarial Networks (GANs). Can be installed with pip

803 Dec 24, 2022