Impala advanced settings of broadcast_ BYTES_ LIMIT

Official website address

https://impala.apache.org/docs/build/html/topics/impala_broadcast_bytes_limit.html

Sets the limit for the size of the broadcast input based on estimated size.

The Impala planner may in rare cases make a bad choice to broadcast a large table or intermediate result and encounter performance problems due to high memory pressure. Setting this limit will make the planner pick a partition based hash join instead of broadcast and avoid such performance problems.

Type: integer

Default:

The default value is 34359738368 (32 GB). A value of 0 causes the option to be ignored.

Examples:

-- Change the limit to 16GB.
set broadcast_bytes_limit=17179869184;

-- Disable the BROADCAST_BYTES_LIMIT.
set broadcast_bytes_limit=0;

It's nothing to say about the broadcast table Limit its size  .

What are the benefits of this setting . There is something in the front . In very few cases impala Will make a wrong choice , That is, our various join When , He will broadcast a huge middle watch or a big watch , At this time, a broadcast uses more memory ！！！ It's easy to hang up .

So let's say Setting this value smaller can avoid the occurrence of this problem , But it's small , It will affect some normal queries .

Unfortunately, I'm set The parameter is still not found in .

In Impala 2.0 and later, you can set query options directly through the JDBC and ODBC interfaces by using the SET statement. Formerly, SET was only available as a command within the impala-shell interpreter.

In Impala 2.11 and later, you can set query options for an impala-shell session by specifying one or more command-line arguments of the form --query_option=option=value. See impala-shell Configuration Options for details.

But I mentioned these two sentences May be able to jdbc Add this parameter to the connection string

[[email protected] ~]$ impala-shell -i test-flimpala.woa.com:25003 -d default -k -Q broadcast_bytes_limit=17179869184
Starting Impala Shell using Kerberos authentication
Using service name 'impala'
Opened TCP connection to test-flimpala.woa.com:25003
Connected to test-flimpala.woa.com:25003
Server version: impalad version 3.2.0-cdh6.3.2 RELEASE (build 1bb9836227301b839a32c6bc230e35439d5984ac)
BROADCAST_BYTES_LIMIT is not supported for the impalad being connected to, ignoring.
Query: use `default`
***********************************************************************************
Welcome to the Impala shell.
(Impala Shell v3.2.0-cdh6.3.2 (1bb9836) built on Fri Nov  8 07:22:06 PST 2019)

Press TAB twice to see a list of available commands.
***********************************************************************************
Query: use `default`
[test-flimpala.woa.com:25003] default> set
 

  To no avail . He said yes set You can see which values can set Did it turn out . It's all supplemented by knowledge