Optimizes image files by converting them to webp while also updating all references.

Overview

About

Optimizes images by (re-)saving them as webp. For every file it replaced it automatically updates all references. Works on single files as well as on folders.

Installation: Windows

Step 1: Get the exe

Option 1: Download

Download one of the pre-built fvttoptimizer.exe files from Releases

Go to Step 2: Install program

Option 2: Build it yourself

Install python3 (version >= 3.8) and add it to your PATH system environment variable.

You can check the version via CMD or powershell with:

python --version

Download or clone the repo.

Open a CMD instance inside the project root folder and create a virtual environment.

python -m venv .\venv

Activate the venv:

.\venv\Scripts\activate

Install the Pillow package:

pip install pillow==9.1.0

Install pyInstaller package:

pip install pyInstaller==4.10

Download the fvttmv python package (whl file) from here: https://github.com/watermelonwolverine/fvttmv/releases/tag/0.2.4 move it into the folder and install it:

pip install fvttmv-0.2.4-py3-none-any.whl

Run build file:

.\scripts\build_for_windows.cmd

You should now have a fvttoptimizer.exe file under dist. After the build succeeded you can delete the venv you previously created.

Go to Step 2: Install program

Step 2: Install program

Create an empty folder where you want to install the program for example C:\fvttoptimizer

Copy fvttoptimizer.exe into that folder.

Create a fvttoptimizer.conf text file in that folder.

Copy {"absolute_path_to_foundry_data":"INSERT_PATH_HERE"} into fvttoptimizer.conf

Replace INSERT_PATH_HERE with the path to the Data folder inside your foundrydata (Not the foundrydata folder itself!).

IMPORTANT: Escape all \ with \\ in that path.

It should look something like this:

{"absolute_path_to_foundry_data":"C:\\Users\\user\\foundrydata\\Data"}

Add the installation path to your PATH system environment variable.

Uninstallation: Windows

Delete fvttoptimizer.exe and fvttoptimizer.conf files from the installation directory.

Remove the path to the installation directory from the PATH system environment variable.

Installation: Ubuntu 16.04 -20.04

Optional: Install Python

An up-to-date Ubuntu 20.04 should have python >= 3.8. Check it with:

python3 --version

From here on pythonX will be used as placeholder for the python you should use. Depending on your system you need to replace X with 3, 3.8, 3.9 or 3.10 .

Install python version>=3.8 if not yet installed.

sudo apt install pythonX

Step 1: Get the executable

Option 1: Download

Download one of the pre-built fvttoptimizer files from Releases

Go to Step 2: Install the files

Option 2: Build it yourself

Install python if haven't already.

Install Pillow dependencies (read more here https://pillow.readthedocs.io/en/stable/installation.html):

sudo apt install pythonX-dev libtiff5-dev libjpeg8-dev libopenjp2-7-dev zlib1g-dev libfreetype6-dev liblcms2-dev libwebp-dev tcl8.6-dev tk8.6-dev python3-tk libharfbuzz-dev libfribidi-dev libxcb1-dev

Download or clone repo.

Create venv inside the repo:

sudo apt install pythonX-venv

pythonX -m venv ./venv

Activate venv:

source venv/bin/activate

Install Pillow (should succeed if you installed all the dependencies):

pythonX -m pip install Pillow==9.1.0

Install pyInstaller:

pythonX -m pip install pyInstaller==4.10

Download the fvttmv python package (whl file) from here: https://github.com/watermelonwolverine/fvttmv/releases/tag/0.2.4 move it into the folder and install it:

pythonX -m pip install fvttmv-0.2.4-py3-none-any.whl

Now run the build script:

./scrips/build_for_ubuntu.sh

Step 2: Install the files

Option 1: Automatic installation

Install python if you haven't already.

Clone the repo if you haven't already.

Go into the project folder.

Build the project if you haven't already. If you downloaded a pre-built executable. Create a folder named dist and move the file there.

Inside the project folder run:

sudo pythonX scripts/install_on_ubuntu.sh

Option 2: Manual installation

Copy the fvttoptimizer file either from dist from where you downloaded it to usr/bin/fvttoptimizer

Make the file executable:

sudo chmod ugo=rx usr/bin/fvttoptimizer

Create a fvttoptimizer.conf file at /etc/

Copy {"absolute_path_to_foundry_data":"INSERT_PATH_HERE"} into fvttoptimizer.conf

Replace INSERT_PATH_HERE with the path to the Data folder inside your foundrydata (Not the foundrydata folder itself!).

It should look something like this:

{"absolute_path_to_foundry_data":"/home/user/foundrydata/Data"}

Uninstallation: Ubuntu

Delete the files /etc/fvttoptimizer.conf and /usr/bin/fvttoptimizer

Usage

IMPORTANT:

Shut down Foundry VTT before optimizing any files.

Use this program at your own risk. It has been tested extensively but there are no guarantees. Always keep backup of your Foundry VTT data for cases where something goes wrong.

Syntax

fvttoptimizer [--verbose-info, --verbose-debug, --version, --help, --quality value, --override-percent value, --skip-webp, --skip-existing, --recursive] target

target: Path to file or directory which should be optimized

Options

--verbose-info: Enables verbose output to console

--verbose-debug: Enables very verbose output to console

--version: Prints version and exits

--help: Display help and exit

--skip-existing: Ignores files of which a webp already exists. For example image.png will not be converted if a image.webp is already in the same folder

--skip-webp: Don't touch webp files at all

--quality: The quality setting for the webp compression. Default is 75

--override-percent: How much smaller the new file needs to be to replace the old one. For example if this value is 25 the file size after optimizing needs to be 25% smaller than the original. Default is 25

--recursive: If the optimization should be done recursively to all sub folders of the target folder.

Examples

Optimizing a file:

Ubuntu: fvttoptimizer path/to/file.png

Windows: fvttoptimizer path\to\file.png

Optimizing a folder and all it's sub folders:

Ubuntu: fvttoptimizer --recursive pathto/folder

Windows: fvttoptimizer.exe --recursive path\to\folder

Use single quotes when moving a file with a space:

Ubuntu: fvttoptimizer 'some folder/some file.jpg'

Windows: fvttoptimizer.exe 'some folder\some file.jpg'

Change parameters:

Ubuntu: fvttoptimizer --quality 50 --override-percent 50 file.jpg

Windows: fvttoptimizer --quality 50 --override-percent 50 file.jpg

Known Issues and Quirks

Windows

The program only works in powershell not in cmd.

When one of the paths has \' at the end, the arguments will get mixed up. This is a problem with how python handles arguments and probably can't be fixed. For example on Windows fvttoptimizer.exe '\folder name with spaces\' will fail but fvttoptimizer.exe '\folder name with spaces' will succeed.

Comments
  • Program fails on special character [ in path

    Program fails on special character [ in path

    First off, thank you so much for making this.

    I am trying to use the following command: fvttoptimizer.exe --recursive 'C:\Users\cnguy\AppData\Local\FoundryVTT\Data\Battle Maps'

    and I am getting this return: An unexpected error occurred: expected str, bytes or os.PathLike object, not NoneType

    Your documentation implies this is supported, as this command works without the "--recursive" flag, so maybe there is an issue with spaces in paths and the recursive function?

    Thank you.

    opened by coboba123 5
  • Trying to create a Docker image - Keep getting 'image.jpg is not in the configured foundry Data folder.'

    Trying to create a Docker image - Keep getting 'image.jpg is not in the configured foundry Data folder.'

    As the title say, I'm trying to get this tool to work in a Docker container, but I'm having an issue that I can't figure out...

    I'm building the image like so:

    FROM python:3.10.0
    
    ARG FVTT_OPTIMIZER_VERSION=0.1.3
    
    RUN curl -L https://github.com/watermelonwolverine/fvttoptimizer/releases/download/$FVTT_OPTIMIZER_VERSION/fvttoptimizer --output usr/bin/fvttoptimizer
    RUN chmod +x usr/bin/fvttoptimizer
    RUN echo '{"absolute_path_to_foundry_data":"/app/Data"}' > /etc/fvttoptimizer.conf
    
    CMD ["fvttoptimizer"]
    

    Which I then configure with docker-compose:

    version: "3.7"
    
    services:
        fvttoptimizer:
            build: python
            volumes:
                - ${FOUNDRY_DATA_VOLUME}:/app
    
    # FOUNDRY_DATA_VOLUME is the directory that contains `Config`, `Data` and `Logs`
    

    However, when ran (docker-compose run --rm fvttoptimizer fvttoptimizer --verbose-debug ./Barney.jpg) I get the error in the title. Running docker-compose run --rm fvttoptimizer ls -la /app/Data, I can confirm the image exists.

    The debug output looks like this:

    root - DEBUG - Got arguments ['fvttoptimizer', '--verbose-debug', '--recursive', './Barney.jpg']
    root - DEBUG - Reading config from: /etc/fvttoptimizer.conf
    root - DEBUG - Running with target_path='./Barney.jpg', config='{'_RunConfig__program_config': {'_ProgramConfigImpl__abs_path_to_foundry_data': '/app/Data'}, 'skip_existing': False, 'skip_webp': False, 'recursive': True, 'quality': 75, 'override_percent': 25}'
    root - ERROR - /Barney.jpg is not in the configured foundry Data folder.
    /Barney.jpg is not in the configured foundry Data folder.
    

    I pushed a branch on my repo where it should be possible to reproduce the issue, or at the very least look at my configuration: https://github.com/gCardinal/foundrty-vtt/blob/feature/use-fvttoptimizer

    support 
    opened by gCardinal 3
  • Missing config file

    Missing config file

    When running on an image file I get the error:

    Missing config file. Could not find /etc/fvttoptimizer.conf

    Tried running all three release versions on Arch Linux with Python 3.10.4 and get the same error every time.

    I'm not seeing any mention of a config file in the readme. Am I missing something?

    support 
    opened by tonydero 2
  • Suggestion: Print which file it is working on during the conversion process.

    Suggestion: Print which file it is working on during the conversion process.

    I think it would be nice to have some kind of progress indicator to show what its working on in the command prompt window. The easiest way to have this would simply just print the name of the file that is being worked on when it gets to it. When working on folders with lots of images, its nice to know its not frozen or hanging on something.

    Thank you!

    opened by coboba123 2
  • getting root - ERROR - [path] is not in the configured foundry Data folder.

    getting root - ERROR - [path] is not in the configured foundry Data folder.

    So I am running on a Google Cloud server, and have manually installed fvttoptimizer with Python 3.8.

    I created the /etc/fvttoptimizer.conf with the following line:

    {"absolute_path_to_foundry_data":"/home/tehjoester/foundrydata/Data"}

    and here is my output with the --verbose-debug option:

    [email protected]:~$ sudo fvttoptimizer --verbose-debug --skip-webp --recursive 'foundrydata/Data/'
    root - DEBUG - Got arguments ['fvttoptimizer', '--verbose-debug', '--skip-webp', '--recursive', 'foundrydata/Data/']
    root - DEBUG - Reading config from: /etc/fvttoptimizer.conf
    root - DEBUG - Running with target_path='foundrydata/Data/', config='{'_RunConfig__program_config': {'_ProgramConfigImpl__abs_path_to_foundry_data': '/home/tehjoester/foundrydata/Data'}, 'skip_existing': False, 'skip_webp': True, 'recursive': True, 'quality': 75, 'override_percent': 25}'
    root - ERROR - /home/tehjoester/foundrydata/Data is not in the configured foundry Data folder.
    /home/tehjoester/foundrydata/Data is not in the configured foundry Data folder.
    
    opened by thejoester 1
  • Argument --skip-webp does not seem to work

    Argument --skip-webp does not seem to work

    It's entirely possible that I am doing something wrong, but when I try to use --skip-webp it still tries to process webp files. Running the optimizer twice in a row shows that it will attempt the same files again even if they've already been converted.

    I am running Windows 11. This is the command I tried to run in Windows Terminal/PowerShell. Variations on this command have not resolved the issue.

    fvttoptimizer.exe --verbose-info --skip-webp --quality 100 --override-percent 10 --recursive 'C:\Games\Foundry Virtual Tabletop Data\Data\worlds\mitera'

    opened by seanms1991 0
  • Some actors are not getting the directory to their new avatar image updated properly

    Some actors are not getting the directory to their new avatar image updated properly

    While most actors have their pictures updated as expected, some are skipped. From what I can tell, it's mostly the avatars of the actors and not the tokens. The images are updated but the reference is not updated in the actors.db. Attached are some examples of actors that have this issue. Their tokens were created with Tokenizer, but those converted successfully and their main picture did not, which was not created with tokenizer. I've uploaded the pictures that were converted, the actor data exported as json files, and the actors.db files they were in both before and after the optimizer processed it.

    I am running Windows 11 and have tried to run this on two separate computers with the same result.

    If you need more information, let me know.

    Example.zip

    opened by seanms1991 0
  • Feature Request

    Feature Request

    First off, this script is amazing. It's greatly reduced the sizes of my images directories.

    It would be really cool though if I could maybe put in the config file the path to personal modules that it will update the file names. I have several shared compendiums in my own module that I move between worlds.

    enhancement 
    opened by Jonwh25 1
  • Program fails when a db file contains a byte UTF-8 can't decode.

    Program fails when a db file contains a byte UTF-8 can't decode.

    I got the following message in powershell:

    `PS C:\FVTTOptimizer> .\fvttoptimizer.exe --verbose-info --skip-webp --recursive "C:\Users\Scott-Server\AppData\Local\FoundryVTT\Data\Worlds\bakja" root - INFO - Optimizing directory 'C:\Users\Scott-Server\AppData\Local\FoundryVTT\Data\worlds\bakja' root - INFO - Optimizing directory 'C:\Users\Scott-Server\AppData\Local\FoundryVTT\Data\worlds\bakja\Assets' root - INFO - Optimizing directory 'C:\Users\Scott-Server\AppData\Local\FoundryVTT\Data\worlds\bakja\Assets\Map' root - INFO - Optimizing C:\Users\Scott-Server\AppData\Local\FoundryVTT\Data\worlds\bakja\Assets\Map\Animal Market Stall.png. Decreased file size by 83% root - ERROR - Traceback (most recent call last): File "cli_wrapper\main.py", line 235, in main File "cli_wrapper\main.py", line 229, in do_run File "cli_wrapper\main.py", line 81, in perform_optimization_with File "fvttoptimizer\optimizer.py", line 42, in optimize File "fvttoptimizer__directory_optimizer.py", line 38, in optimize File "fvttoptimizer__directory_optimizer.py", line 38, in optimize File "fvttoptimizer__directory_optimizer.py", line 35, in optimize File "fvttoptimizer__file_optimizer.py", line 34, in maybe_optimize File "fvttoptimizer__file_optimizer.py", line 48, in __maybe_optimize2 File "fvttoptimizer__file_optimizer.py", line 68, in __optimize File "fvttoptimizer__file_optimizer.py", line 100, in __replace_file_with_webp File "fvttmv\update\references_updater.py", line 22, in replace_references File "fvttmv\update\references_updater.py", line 34, in _do_replace_references File "fvttmv\update__references_updater_file.py", line 21, in replace_references_in_file File "codecs.py", line 322, in decode UnicodeDecodeError: 'utf-8' codec can't decode byte 0xd0 in position 0: invalid continuation byte

    An internal error occurred: Traceback (most recent call last): File "cli_wrapper\main.py", line 235, in main File "cli_wrapper\main.py", line 229, in do_run File "cli_wrapper\main.py", line 81, in perform_optimization_with File "fvttoptimizer\optimizer.py", line 42, in optimize File "fvttoptimizer__directory_optimizer.py", line 38, in optimize File "fvttoptimizer__directory_optimizer.py", line 38, in optimize File "fvttoptimizer__directory_optimizer.py", line 35, in optimize File "fvttoptimizer__file_optimizer.py", line 34, in maybe_optimize File "fvttoptimizer__file_optimizer.py", line 48, in __maybe_optimize2 File "fvttoptimizer__file_optimizer.py", line 68, in __optimize File "fvttoptimizer__file_optimizer.py", line 100, in __replace_file_with_webp File "fvttmv\update\references_updater.py", line 22, in replace_references File "fvttmv\update\references_updater.py", line 34, in _do_replace_references File "fvttmv\update__references_updater_file.py", line 21, in replace_references_in_file File "codecs.py", line 322, in decode UnicodeDecodeError: 'utf-8' codec can't decode byte 0xd0 in position 0: invalid continuation byte

    Please file a bug report on https://github.com/watermelonwolverine/fvttoptimizer/issues`

    bug 
    opened by YourDNDPleasesMe 10
Releases(0.1.3)
Owner
Watermelon Wolverine
Watermelon Wolverine
ESGD-M - A stochastic non-convex second order optimizer, suitable for training deep learning models, for PyTorch

ESGD-M - A stochastic non-convex second order optimizer, suitable for training deep learning models, for PyTorch

Katherine Crowson 53 Dec 29, 2022
This repository provides some of the code implemented and the data used for the work proposed in "A Cluster-Based Trip Prediction Graph Neural Network Model for Bike Sharing Systems".

cluster-link-prediction This repository provides some of the code implemented and the data used for the work proposed in "A Cluster-Based Trip Predict

Bárbara 0 Dec 28, 2022
Implementation of Vision Transformer, a simple way to achieve SOTA in vision classification with only a single transformer encoder, in Pytorch

Implementation of Vision Transformer, a simple way to achieve SOTA in vision classification with only a single transformer encoder, in Pytorch

Phil Wang 12.6k Jan 09, 2023
Large scale PTM - PPI relation extraction

Large-scale protein-protein post-translational modification extraction with distant supervision and confidence calibrated BioBERT The silver standard

1 Feb 25, 2022
(CVPR2021) Kaleido-BERT: Vision-Language Pre-training on Fashion Domain

Kaleido-BERT: Vision-Language Pre-training on Fashion Domain Mingchen Zhuge*, Dehong Gao*, Deng-Ping Fan#, Linbo Jin, Ben Chen, Haoming Zhou, Minghui

248 Dec 04, 2022
ConE: Cone Embeddings for Multi-Hop Reasoning over Knowledge Graphs

ConE: Cone Embeddings for Multi-Hop Reasoning over Knowledge Graphs This is the code of paper ConE: Cone Embeddings for Multi-Hop Reasoning over Knowl

MIRA Lab 33 Dec 07, 2022
Noise Conditional Score Networks (NeurIPS 2019, Oral)

Generative Modeling by Estimating Gradients of the Data Distribution This repo contains the official implementation for the NeurIPS 2019 paper Generat

451 Dec 26, 2022
Incremental Transformer Structure Enhanced Image Inpainting with Masking Positional Encoding (CVPR2022)

Incremental Transformer Structure Enhanced Image Inpainting with Masking Positional Encoding by Qiaole Dong*, Chenjie Cao*, Yanwei Fu Paper and Supple

Qiaole Dong 190 Dec 27, 2022
Code for: https://berkeleyautomation.github.io/bags/

DeformableRavens Code for the paper Learning to Rearrange Deformable Cables, Fabrics, and Bags with Goal-Conditioned Transporter Networks. Here is the

Daniel Seita 121 Dec 30, 2022
Paper: Cross-View Kernel Similarity Metric Learning Using Pairwise Constraints for Person Re-identification

Cross-View Kernel Similarity Metric Learning Using Pairwise Constraints for Person Re-identification T M Feroz Ali, Subhasis Chaudhuri, ICVGIP-20-21

T M Feroz Ali 3 Jun 17, 2022
Research code for Arxiv paper "Camera Motion Agnostic 3D Human Pose Estimation"

GMR(Camera Motion Agnostic 3D Human Pose Estimation) This repo provides the source code of our arXiv paper: Seong Hyun Kim, Sunwon Jeong, Sungbum Park

Seong Hyun Kim 1 Feb 07, 2022
Implementation of Perceiver, General Perception with Iterative Attention in TensorFlow

Perceiver This Python package implements Perceiver: General Perception with Iterative Attention by Andrew Jaegle in TensorFlow. This model builds on t

Rishit Dagli 84 Oct 15, 2022
A framework that constructs deep neural networks, autoencoders, logistic regressors, and linear networks

A framework that constructs deep neural networks, autoencoders, logistic regressors, and linear networks without the use of any outside machine learning libraries - all from scratch.

Kordel K. France 2 Nov 14, 2022
NeurIPS'21 Tractable Density Estimation on Learned Manifolds with Conformal Embedding Flows

NeurIPS'21 Tractable Density Estimation on Learned Manifolds with Conformal Embedding Flows This repo contains the code for the paper Tractable Densit

Layer6 Labs 4 Dec 12, 2022
Domain Generalization with MixStyle, ICLR'21.

MixStyle This repo contains the code of our ICLR'21 paper, "Domain Generalization with MixStyle". The OpenReview link is https://openreview.net/forum?

Kaiyang 208 Dec 28, 2022
A Python library for differentiable optimal control on accelerators.

A Python library for differentiable optimal control on accelerators.

Google 80 Dec 21, 2022
This repository contains the source code for the paper "DONeRF: Towards Real-Time Rendering of Compact Neural Radiance Fields using Depth Oracle Networks",

DONeRF: Towards Real-Time Rendering of Compact Neural Radiance Fields using Depth Oracle Networks Project Page | Video | Presentation | Paper | Data L

Facebook Research 281 Dec 22, 2022
A web application that provides real time temperature and humidity readings of a house.

About A web application which provides real time temperature and humidity readings of a house. If you're interested in the data collected so far click

Ben Thompson 3 Jan 28, 2022
Deep Reinforcement Learning for Keras.

Deep Reinforcement Learning for Keras What is it? keras-rl implements some state-of-the art deep reinforcement learning algorithms in Python and seaml

Keras-RL 0 Dec 15, 2022
Parameterized Explainer for Graph Neural Network

PGExplainer This is a Tensorflow implementation of the paper: Parameterized Explainer for Graph Neural Network https://arxiv.org/abs/2011.04573 NeurIP

Dongsheng Luo 89 Dec 12, 2022