Optimus's transformation plugins are implementations of Task and Hook interfaces that allows execution of arbitrary jobs in optimus.
brew tap odpf/taps
brew install optimus-plugins-odpf
curl -sL ${PLUGIN_RELEASE_URL} | tar xvz
chmod +x optimus-*
mv optimus-* /usr/bin/
Hello, Currently, for any query, we try to find the dependancy and ignoredependancy with FindDependenciesWithRegex and then we again pull the Refereced table with big query dry run.
If query contains view which is marked with /* @ignoreupstream */ helper, then ignoredependancy will contain the view name but not the table referenced by view.
The change here is to revise ignoredependancy list with table referenced by view.
I kept the loop execution in sequential manner, please let me know if should add concurrency here
enhancementWe have a query referencing to table as well as view.
select * from proj.dataset.table t1 left join proj.dataset.view v1 on t1.date = v1.date and t1.id = v1.id
proj.dataset.table then it correctly ignores to create upstream dependancy for this table.proj.dataset.view ( note the view query refers to 2 more tables ) then it does not ignore view or table referenced by view.As per Optimus PR, the executor boot process is standardised and maintained at optimus. Plugin devs need no longer have to wrap the executor image. closes odpf/optimus#405
Hi team,
I have a bq2bq job with window configuration
window:
size: 720h
offset: -48h
truncate_to: M
I expect to have transformation for date 01 to last day of the month, e.g on April, I expect got transformation from date 01 - 30. but currently only got transformation from date 01 - 29
[2022-06-13 15:08:12,323] {pod_launcher.py:149} INFO - [2022-06-13 15:08:12] INFO:bumblebee.transformation: create transformation for partition: 2022-04-26 00:00:00+00:00
[2022-06-13 15:08:12,323] {pod_launcher.py:149} INFO - [2022-06-13 15:08:12] INFO:bumblebee.transformation: create transformation for partition: 2022-04-27 00:00:00+00:00
[2022-06-13 15:08:12,323] {pod_launcher.py:149} INFO - [2022-06-13 15:08:12] INFO:bumblebee.transformation: create transformation for partition: 2022-04-28 00:00:00+00:00
[2022-06-13 15:08:12,323] {pod_launcher.py:149} INFO - [2022-06-13 15:08:12] INFO:bumblebee.transformation: create transformation for partition: 2022-04-29 00:00:00+00:00
[2022-06-13 15:08:12,324] {pod_launcher.py:149} INFO - [2022-06-13 15:08:12] INFO:bumblebee.transformation: start transformation job
[2022-06-13 15:08:12,324] {pod_launcher.py:149} INFO - [2022-06-13 15:08:12] INFO:bumblebee.transformation: sql transformation query:
after checking, I suspect the logic may related to this line, where the last day generated by windows class not included as the transformation partition.
https://github.com/odpf/transformers/blob/ea1de4f0de3d17d9be7ccefb1e2f3beab1a685f1/task/bq2bq/executor/bumblebee/transformation.py#L393
please kindly check it, and release the fix. thank you
With this we are adding support for using secrets in macros, we do not want to print the env vars in the logs, so exporting them as a separate file from optimus.
Plugins can export this extra file to get env vars.
As per https://github.com/odpf/optimus/pull/425, the executor boot process is standardised and maintained at optimus. Plugin devs need no longer have to wrap the executor image. closes https://github.com/odpf/optimus/issues/405
The most intuitive way is to parse the query and hit the metadata apis instead of going through the dry run which should be definitly costly then the metadata fetch apis.
enhancement performanceOptions
aa14f19 feat: load method for partition replace using aggregation queries (#15)
docker pull docker.io/odpf/optimus-task-bq2bq:0.0.9cb26e8f feat: add destination type when generating destination of bq2bq task to support backup feature (#7)
docker pull docker.io/odpf/optimus-task-bq2bq:0.0.882d75fb build(deps): bump sqlparse from 0.4.1 to 0.4.2 in /task/bq2bq/executor (#6) 7b6ea2d feat: add execution_project configuration (#9)
docker pull docker.io/odpf/optimus-task-bq2bq:0.0.7047c3ba fix(bq2bq): handle BQ dry run error caused by self table SQL reference (#5)
docker pull docker.io/odpf/optimus-task-bq2bq:0.0.6631527a refactor: implement plugin mods 621dc7d refactor: update help for bigquery prompts to ask for destination (#3)
docker pull docker.io/odpf/optimus-task-bq2bq:0.0.52122157 build(deps): bump rsa from 4.6 to 4.7 in /task/bq2bq/executor (#1) 5f4d0d2 build(deps): bump urllib3 from 1.25.9 to 1.26.5 in /task/bq2bq/executor (#2) db8307c feat(bq2bq): support default values inferred from job name
docker pull docker.io/odpf/optimus-task-bq2bq:0.0.4365796e fix: dataset/table name passed during config gen
docker pull docker.io/odpf/optimus-task-bq2bq:0.0.3
Natural Language Processing for Adverse Drug Reaction (ADR) Detection This repo contains code from a project to identify ADRs in discharge summaries a
The SVO-Probes Dataset for Verb Understanding This repository contains the SVO-Probes benchmark designed to probe for Subject, Verb, and Object unders
Fast (GAN Based Neural) Vocoder Chinese README Todo Submit demo Support NHV Discription Include MelGAN, HifiGAN and Multiband-HifiGAN, maybe include N
Luga A blazing fast language detection using fastText's language models Luga is a Swahili word for language. fastText provides a blazing fast language
Japanese-LUW-Tokenizer Japanese Long-Unit-Word (国語研長単位) Tokenizer for Transformers based on 青空文庫 Basic Usage from transformers import RemBertToken
Live Demo (currently not running, on it) Espial is an engine for automated organization and discovery in knowledge bases. It can be adapted to run wit
Rule-Based-Classification-in-a-Banking-Case. A CRM department in a local bank works on classify their lost customers with their past datas. So they wa
Mesh TensorFlow - Model Parallelism Made Easier Introduction Mesh TensorFlow (mtf) is a language for distributed deep learning, capable of specifying
Feature_CRF_AE Feature_CRF_AE provides a implementation of Bridging Pre-trained Language Models and Hand-crafted Features for Unsupervised POS Tagging
Schrodinger's Cat Schrodinger'sCat is a Shellcode antivirus evasion framework Technical principle Please visit my blog https://idiotc4t.com/ How to us
Full Spectrum Bioinformatics is a free online text designed to introduce key topics in Bioinformatics using the Python programming language. The text is written in interactive Jupyter Notebooks, whic
Modern-Python-Doc-Example mkdocs + material + cool stuff Doc is live here Features out of the box amazing good looking website thanks to mkdocs.org an
DAG_Generator Random Directed Acyclic Graph Generator verison1.0 简介 工作流通常由DAG（有向无环图）来定义，其中每个计算任务$T_i$由一个顶点(node,task,vertex)表示。同时，任务之间的每个数据或控制依赖性由一条加权
Table of contents Mesh Transformer JAX Updates Pretrained Models GPT-J-6B Links Acknowledgments License Model Details Zero-Shot Evaluations Architectu
BERT Convolutions Code for the paper Convolutions and Self-Attention: Re-interpreting Relative Positions in Pre-trained Language Models. Contains expe
smaller-LaBSE LaBSE(Language-agnostic BERT Sentence Embedding) is a very good method to get sentence embeddings across languages. But it is hard to fi
pretrain4ir_tutorial NLPIR tutorial: pretrain for IR. pre-train on raw textual corpus, fine-tune on MS MARCO Document Ranking 用作NLPIR实验室, Pre-training
Wordle_Bot Python bot created with Selenium that can guess the daily Wordle word correct 96.8% of the time. It will log onto the wordle website and en
GDAP The code of paper "Code for "Generating Disentangled Arguments with Prompts: a Simple Event Extraction Framework that Works"" Event Datasets Prep
Deduplication is the task to combine different representations of the same real world entity. This package implements deduplication using active learning. Active learning allows for rapid training wi
21 Aug 05, 2022
20 Nov 30, 2022
134 Dec 16, 2022
18 Dec 20, 2022
3 Dec 22, 2021
159 Dec 30, 2022
4 Mar 20, 2022
1.3k Dec 26, 2022
6 Apr 29, 2022
27 Jul 09, 2022
33 Dec 28, 2022
61 Oct 26, 2022
17 Dec 27, 2022
4.9k Jan 04, 2023
21 Jul 18, 2022
13 Sep 02, 2022
12 Apr 07, 2022
15 Dec 11, 2022
45 Oct 29, 2022
63 Nov 17, 2022