metamaker
Simple command line tool to train and deploy your machine learning models with AWS SageMaker
Features
metamaker enables you to:
- Build a docker image for training and inference with poetry and FastAPI
- Train your own machine learning model with SageMaker
- Deploy inference endpoint with SageMaker
Usage
- Create poetry project and install metamaker
❯ poetry new your_module
❯ cd your_module
❯ poetry add metamaker
- Define scripts for traning and inference in
main.py
Output: # ... from your_module import Model, Input, Output app = MetaMaker[Model, Input, Output]() @app.trainer def train( dataset_path: Path, artifact_path: Path, hyperparameters: Dict[str, Any], ) -> None: model = Model(**hyperparameters) model.train(dataset_path / "train.csv") model.save(artifact_path / "model.tar.gz") @app.loader def load(artifact_path: Path) -> Model: return Model.load(artifact_path / "model.tar.gz") @app.predictor def predict(model: Model, data: Input) -> Output: return model.predict(data) ">
from pathlib import Path from typing import Any, Dict from metamaker import MetaMaker # Import your model, and input/output data classs: # # Model ... machine learning model class you want to use # Input ... input data class for inference # Output ... ouput data class for inference # # Note that the Input and Output are used as type hints to # create API endpoint with FastAPI like below: # # @fastapi_app.post("/invocations") # def predict(data: Input) -> Output: # ... from your_module import Model, Input, Output app = MetaMaker[Model, Input, Output]() @app.trainer def train( dataset_path: Path, artifact_path: Path, hyperparameters: Dict[str, Any], ) -> None: model = Model(**hyperparameters) model.train(dataset_path / "train.csv") model.save(artifact_path / "model.tar.gz") @app.loader def load(artifact_path: Path) -> Model: return Model.load(artifact_path / "model.tar.gz") @app.predictor def predict(model: Model, data: Input) -> Output: return model.predict(data)
- Write metamaker configs in
metamaker.yaml
# Specify metamaker handler like: `path.to.module:app_name`
handler: main:app
# dataset_path and artifact_path should be directories and end with '/'
dataset_path: s3://your-bucket/path/to/dataset/
artifact_path: s3://your-bucket/path/to/artifacts/
hyperparameter_path: ./hparams.yaml
image:
name: metamaker
includes:
- your_module/
- main.py
excludes:
- __pycache__/
- '*.py[cod]'
training:
execution_role: arn:aws:iam::xxxxxxxxxxxx:role/SageMakerExecutionRole
instance:
type: ml.m5.large
count: 1
inference:
endpoint_name: your_endpoint
instance:
type: ml.t2.meduim
count: 1
- Build docker image and push to ECR
metamaker build --deploy .
- Train your model with SageMaker and deploy endpoint
metamker sagemaker train --deploy
3.4k Dec 30, 2022
5.9k Dec 30, 2022
3 Feb 14, 2022
13 Oct 14, 2022
5 Feb 20, 2022
10 Aug 01, 2022
2 Oct 31, 2021
2 Jun 22, 2022
6 Dec 23, 2022
73 Oct 12, 2022
8 Aug 16, 2022
21 Nov 03, 2022
2 Jul 18, 2022
23 Jul 04, 2022
4 Apr 11, 2022
2 Apr 04, 2022
1 Oct 24, 2021
1 Dec 10, 2021
332 Dec 26, 2022
104 Dec 31, 2022