Optimus's transformation plugins are implementations of Task and Hook interfaces that allows execution of arbitrary jobs in optimus.
brew tap odpf/taps
brew install optimus-plugins-odpf
curl -sL ${PLUGIN_RELEASE_URL} | tar xvz
chmod +x optimus-*
mv optimus-* /usr/bin/
Hello, Currently, for any query, we try to find the dependancy and ignoredependancy with FindDependenciesWithRegex and then we again pull the Refereced table with big query dry run.
If query contains view which is marked with /* @ignoreupstream */ helper, then ignoredependancy will contain the view name but not the table referenced by view.
The change here is to revise ignoredependancy list with table referenced by view.
I kept the loop execution in sequential manner, please let me know if should add concurrency here
enhancementWe have a query referencing to table as well as view.
select * from proj.dataset.table t1 left join proj.dataset.view v1 on t1.date = v1.date and t1.id = v1.id
proj.dataset.table then it correctly ignores to create upstream dependancy for this table.proj.dataset.view ( note the view query refers to 2 more tables ) then it does not ignore view or table referenced by view.As per Optimus PR, the executor boot process is standardised and maintained at optimus. Plugin devs need no longer have to wrap the executor image. closes odpf/optimus#405
Hi team,
I have a bq2bq job with window configuration
window:
size: 720h
offset: -48h
truncate_to: M
I expect to have transformation for date 01 to last day of the month, e.g on April, I expect got transformation from date 01 - 30. but currently only got transformation from date 01 - 29
[2022-06-13 15:08:12,323] {pod_launcher.py:149} INFO - [2022-06-13 15:08:12] INFO:bumblebee.transformation: create transformation for partition: 2022-04-26 00:00:00+00:00
[2022-06-13 15:08:12,323] {pod_launcher.py:149} INFO - [2022-06-13 15:08:12] INFO:bumblebee.transformation: create transformation for partition: 2022-04-27 00:00:00+00:00
[2022-06-13 15:08:12,323] {pod_launcher.py:149} INFO - [2022-06-13 15:08:12] INFO:bumblebee.transformation: create transformation for partition: 2022-04-28 00:00:00+00:00
[2022-06-13 15:08:12,323] {pod_launcher.py:149} INFO - [2022-06-13 15:08:12] INFO:bumblebee.transformation: create transformation for partition: 2022-04-29 00:00:00+00:00
[2022-06-13 15:08:12,324] {pod_launcher.py:149} INFO - [2022-06-13 15:08:12] INFO:bumblebee.transformation: start transformation job
[2022-06-13 15:08:12,324] {pod_launcher.py:149} INFO - [2022-06-13 15:08:12] INFO:bumblebee.transformation: sql transformation query:
after checking, I suspect the logic may related to this line, where the last day generated by windows class not included as the transformation partition.
https://github.com/odpf/transformers/blob/ea1de4f0de3d17d9be7ccefb1e2f3beab1a685f1/task/bq2bq/executor/bumblebee/transformation.py#L393
please kindly check it, and release the fix. thank you
With this we are adding support for using secrets in macros, we do not want to print the env vars in the logs, so exporting them as a separate file from optimus.
Plugins can export this extra file to get env vars.
As per https://github.com/odpf/optimus/pull/425, the executor boot process is standardised and maintained at optimus. Plugin devs need no longer have to wrap the executor image. closes https://github.com/odpf/optimus/issues/405
The most intuitive way is to parse the query and hit the metadata apis instead of going through the dry run which should be definitly costly then the metadata fetch apis.
enhancement performanceOptions
aa14f19 feat: load method for partition replace using aggregation queries (#15)
docker pull docker.io/odpf/optimus-task-bq2bq:0.0.9cb26e8f feat: add destination type when generating destination of bq2bq task to support backup feature (#7)
docker pull docker.io/odpf/optimus-task-bq2bq:0.0.882d75fb build(deps): bump sqlparse from 0.4.1 to 0.4.2 in /task/bq2bq/executor (#6) 7b6ea2d feat: add execution_project configuration (#9)
docker pull docker.io/odpf/optimus-task-bq2bq:0.0.7047c3ba fix(bq2bq): handle BQ dry run error caused by self table SQL reference (#5)
docker pull docker.io/odpf/optimus-task-bq2bq:0.0.6631527a refactor: implement plugin mods 621dc7d refactor: update help for bigquery prompts to ask for destination (#3)
docker pull docker.io/odpf/optimus-task-bq2bq:0.0.52122157 build(deps): bump rsa from 4.6 to 4.7 in /task/bq2bq/executor (#1) 5f4d0d2 build(deps): bump urllib3 from 1.25.9 to 1.26.5 in /task/bq2bq/executor (#2) db8307c feat(bq2bq): support default values inferred from job name
docker pull docker.io/odpf/optimus-task-bq2bq:0.0.4365796e fix: dataset/table name passed during config gen
docker pull docker.io/odpf/optimus-task-bq2bq:0.0.3
Unifying Cross-Lingual Semantic Role Labeling with Heterogeneous Linguistic Resources Description This is the repository for the paper Unifying Cross-
Code for "Semantic Role Labeling as Dependency Parsing: Exploring Latent Tree Structures Inside Arguments".
sentello Sentello is a python script that simulates the anti-evasion and anti-analysis techniques used by malware. For techniques that are difficult t
Creates random strings and checks if any of them are a real words. Mostly a waste of time ngl but it is cool to see it work and the fact that it can generate a real random word within10sec
rJAM splitscreen message reader for MysticBBS A46+
End-to-end neural table-text understanding models.
YouTokenToMe YouTokenToMe is an unsupervised text tokenizer focused on computational efficiency. It currently implements fast Byte Pair Encoding (BPE)
Google Text-To-Speech Batch Prompt File Maker Are you in the need of IVR prompts, but you have no voice actors? Let Google talk your prompts like a pr
transformers A website which allows you to play with the GPT-2 model Built with ❤️ by raphtlw Table of contents Model Setup About Contributors Model T
StyleSpeech - PyTorch Implementation PyTorch Implementation of Meta-StyleSpeech : Multi-Speaker Adaptive Text-to-Speech Generation. Status (2021.06.09
Manglish Text Editor This is a simple transiteration (romanization ) program which is used to convert manglish to malayalam (converts njaan to ഞാൻ ).
SciBERT is a BERT model trained on scientific text.
Headliner Headliner is a sequence modeling library that eases the training and in particular, the deployment of custom sequence models for both resear
SAVI2I: Continuous and Diverse Image-to-Image Translation via Signed Attribute Vectors [Paper] [Project Website] Pytorch implementation for SAVI2I. We
NLP-Models-Tensorflow, Gathers machine learning and tensorflow deep learning models for NLP problems, code simplify inside Jupyter Notebooks 100%. Tab
Chimera: Learning Shared Semantic Space for Speech-to-Text Translation This is a Pytorch implementation for the "Chimera" paper Learning Shared Semant
Traditional Chinese Text Recognition Dataset: Synthetic Dataset and Labeled Data Authors: Yi-Chang Chen, Yu-Chuan Chang, Yen-Cheng Chang and Yi-Ren Ye
Biterm Topic Model Bitermplus implements Biterm topic model for short texts introduced by Xiaohui Yan, Jiafeng Guo, Yanyan Lan, and Xueqi Cheng. Actua
ruCLIP-SB RuCLIP-SB (Russian Contrastive Language–Image Pretraining SWIN-BERT) is a multimodal model for obtaining images and text similarities and re
Harvis Harvis is designed to automate your C2 Infrastructure, currently using Mythic C2. 📌 What is it? Harvis is a python tool to help you create mul
16 Sep 09, 2022
50 Nov 08, 2022
62 Oct 02, 2022
1 Jan 06, 2022
4 Nov 22, 2022
914 Jan 07, 2023
847 Dec 19, 2022
1 Aug 19, 2021
2 Jan 27, 2022
142 Jan 06, 2023
1 May 11, 2022
1.2k Dec 24, 2022
231 Nov 18, 2022
44 Dec 30, 2022
1.7k Dec 30, 2022
43 Dec 28, 2022
5 Dec 15, 2022
49 Dec 30, 2022
5 Apr 13, 2022
99 Oct 06, 2022