DorisWriter The plugin documentation

1 Quick introduction

DorisWriter Support writing large quantities of data Doris in .

2 Realization principle

DorisWriter adopt Doris Native support Stream load How to import data , DorisWriter Will reader The read data is cached in memory , Stitching into Json Text , Then batch import to Doris.

3 Functional specifications

3.1 Sample configuration

Here is a copy from Stream After reading the data, import it to Doris Configuration file for .

{
    "job": {
        "setting": {
            "speed": {
                "channel": 1
            },
            "errorLimit": {
                "record": 0,
                "percentage": 0
            }
        },
        "content": [
            {
                "reader": {
                    "name": "streamreader",
                    "parameter": {
                        "column": [
                            {
                                "value": " Preserved egg 1",
                                "type": "string"
                            },
                            {
                                "value": " Preserved egg 2",
                                "type": "string"
                            },
                            {
                                "value": "111",
                                "type": "long"
                            },
                            {
                                "value": "222",
                                "type": "long"
                            }
                        ],
                        "sliceRecordCount": 100
                    }
                },
                "writer": {
                    "name": "doriswriter",
                    "parameter": {
                        "feLoadUrl": ["127.0.0.1:8030", "127.0.0.2:8030", "127.0.0.3:8030"],
                        "beLoadUrl": ["192.168.10.1:8040", "192.168.10.2:8040", "192.168.10.3:8040"],
                        "jdbcUrl": "jdbc:mysql://127.0.0.1:9030/",
                        "database": "db1",
                        "table": "t1",
                        "column": ["k1", "k2", "v1", "v2"],
                        "username": "root",
                        "password": "",
                        "postSql": [],
                        "preSql": [],
                        "loadProps": {
                        },
                        "maxBatchRows" : 500000,
                        "maxBatchByteSize" : 104857600,
                        "labelPrefix": "my_prefix",
                        "lineDelimiter": "\n"
                    }
                }
            }
        ]
    }
}

3.2 Parameter description

• jdbcUrl

  • describe ：Doris Of JDBC Connection string , User execution preSql or postSQL.
  • Mandatory ： yes
  • The default value is ： nothing

• feLoadUrl

  • describe ： and beLoadUrl A choice . As Stream Load Connection target of . The format is “ip:port”. among IP yes FE node IP,port yes FE Node http_port. You can fill in more than one ,doriswriter Will be accessed by polling .
  • Mandatory ： no
  • The default value is ： nothing

• beLoadUrl

  • describe ： and feLoadUrl A choice . As Stream Load Connection target of . The format is “ip:port”. among IP yes BE node IP,port yes BE Node webserver_port. You can fill in more than one ,doriswriter Will be accessed by polling .
  • Mandatory ： no
  • The default value is ： nothing

• username

  • describe ： visit Doris The user name of the database
  • Mandatory ： yes
  • The default value is ： nothing

• password

  • describe ： visit Doris Database password
  • Mandatory ： no
  • The default value is ： empty

• database

  • describe ： What needs to be written Doris Database name .
  • Mandatory ： yes
  • The default value is ： nothing

• table

  • describe ： What needs to be written Doris The name of the table .
  • Mandatory ： yes
  • The default value is ： nothing

• column

  -  describe ： Purpose table ** Need to write data ** Field of , These fields will be generated as  Json  Field name of data . Fields are separated by commas . for example : "column": ["id","name","age"].
  -  Mandatory ： yes 
  -  The default value is ： no 
  

• timeZone

  • describe ：Doris The time zone .
  • Mandatory ： no
  • The default value is ：+08:00

• preSql

  • describe ： Before writing data to the destination table , The standard statements here will be executed first .
  • Mandatory ： no
  • The default value is ： nothing

• postSql

  • describe ： After writing data to the destination table , Will execute the standard statements here .
  • Mandatory ： no
  • The default value is ： nothing

• maxBatchRows

  • describe ： The maximum number of rows of imported data per batch . and maxBatchByteSize Jointly control the import quantity of each batch . Each batch of data reaches one of two thresholds , Start importing this batch of data .
  • Mandatory ： no
  • The default value is ：500000

• maxBatchByteSize

  • describe ： The maximum amount of imported data per batch . and ** maxBatchRows** Jointly control the import quantity of each batch . Each batch of data reaches one of two thresholds , Start importing this batch of data .
  • Mandatory ： no
  • The default value is ：104857600

• labelPrefix

  • describe ： Of each batch of import tasks label Prefix . The final label There will be labelPrefix + UUID + Serial number form
  • Mandatory ： no
  • The default value is ：datax_doris_writer_

• lineDelimiter

  • describe ： Each batch of data contains multiple rows , Every act Json Format , The separator of each line is lineDelimiter. Support multiple bytes , for example ’\x02\x03’.
  • Mandatory ： no
  • The default value is ：\n

• loadProps

  • describe ：StreamLoad Request parameters for , Details refer to StreamLoad Introduction page .
  • Mandatory ： no
  • The default value is ： nothing

• connectTimeout

  • describe ：StreamLoad The timeout for a single request , Unit millisecond (ms).
  • Mandatory ： no
  • The default value is ：-1