搭建Hue环境

0) Prerequisite
## install hue dependencies: hue dependencies

## use PostgreSQL as desktop db
$ brew services restart postgresql
$ createdb hue_d --lc-collate='en_US.UTF-8' -T "template0"
$ psql -s hue_d
hue_d=# create user hue_u with password 'huepassword';
hue_d=# grant all privileges on database hue_d to hue_u;
hue_d=# \q

1) Build
## download code
$ git config --global --add remote.origin.proxy ""
$ git clone https://github.com/cloudera/hue.git

## set env variables

$ cat setenv
export PYTHON_VER=python3.8.2
export SKIP_PYTHONDEV_CHECK=true
export CFLAGS="-I$(xcrun --show-sdk-path)/usr/include/sasl"
export LDFLAGS="-L/usr/local/opt/libffi/lib -L/usr/local/opt/openssl/lib"
export CPPFLAGS="-I/usr/local/opt/libffi/include -I/usr/local/opt/openssl/include"

export PATH=$PATH:~/Library/Python/3.8/bin
export DYLD_LIBRARY_PATH="/usr/local/mysql/lib:$PATH"

## modify make file
$ diff -u Makefile.vars.orig Makefile.vars      # specify a python3 manually

$ diff -u Makefile.vars.orig Makefile.vars
--- Makefile.vars.orig	2022-05-24 10:20:06.000000000 +0800
+++ Makefile.vars	2022-05-25 16:03:27.000000000 +0800
@@ -71,6 +71,7 @@
   SYS_PYTHON ?= $(shell which $(PYTHON_VER))
 endif
 
+SYS_PYTHON := python3
 ifeq ($(SYS_PYTHON),)
   $(error "Error: Need python version 2.7 or >= 3.5")
 else

## make
$ make apps
## note
## During pip install desktop/core/requirements.txt, 2 git clone operations always timeout
## try to split and install separately
$ diff -u Makefile.sdk.orig Makefile.sdk

$ diff -u Makefile.sdk.orig Makefile.sdk  
--- Makefile.sdk.orig	2022-05-24 10:20:06.000000000 +0800
+++ Makefile.sdk	2022-05-27 09:16:25.000000000 +0800
@@ -116,7 +116,7 @@
 .PHONY: ext-env-pip-install
 ext-env-pip-install:
 	@echo '--- Installing $(REQUIREMENT_FILE) into virtual-env via $(ENV_PIP)'
-	@$(ENV_PIP) install -r $(REQUIREMENT_FILE)
+	# @$(ENV_PIP) install -r $(REQUIREMENT_FILE)
 	@echo '--- Finished $(REQUIREMENT_FILE) into virtual-env'
 	@$(ENV_PIP) install $(NAVOPTAPI_WHL)
 	@echo '--- Finished $(NAVOPTAPI_WHL) into virtual-env'

$ diff -u desktop/core/requirements.txt.orig desktop/core/requirements.txt

$ diff -u desktop/core/requirements.txt.orig desktop/core/requirements.txt
--- desktop/core/requirements.txt.orig	2022-05-24 10:20:13.000000000 +0800
+++ desktop/core/requirements.txt	2022-05-27 09:03:18.000000000 +0800
@@ -69,8 +69,6 @@
 tabulate==0.8.9
 thrift==0.13.0
 thrift-sasl==0.4.2
-git+https://github.com/gethue/django-babel.git
-git+https://github.com/gethue/django-mako.git
 werkzeug==2.0.1  # Should remove it here and from devtools.mk
 django-utils-six==2.0
 six==1.16.0

$ cat desktop/core/r2.txt

$ cat desktop/core/r2.txt
git+https://github.com/gethue/django-babel.git
git+https://github.com/gethue/django-mako.git

$ build/env/bin/pip install -r desktop/core/requirements.txt
$ build/env/bin/pip install -r desktop/core/r2.txt
$ make apps     # do again

# install db clients
$ build/env/bin/pip install psycopg2        # maybe needs reinstall libpq-dev
$ build/env/bin/pip install mysqlclient

# update db schema for hue
$ build/env/bin/hue makemigrations      # if need
$ build/env/bin/hue migrate

3) Configure in [desktop]
$ diff -u desktop/conf/pseudo-distributed.ini.tmpl desktop/conf/pseudo-distributed.ini

$ diff -u desktop/conf/pseudo-distributed.ini.tmpl desktop/conf/pseudo-distributed.ini
--- desktop/conf/pseudo-distributed.ini.tmpl	2022-05-24 10:20:07.000000000 +0800
+++ desktop/conf/pseudo-distributed.ini	2022-06-05 17:10:11.000000000 +0800
@@ -22,33 +22,33 @@
 
   # Set this to a random string, the longer the better.
   # This is used for secure hashing in the session store.
-  secret_key=
+  secret_key=a_random_string
 
   # Execute this script to produce the Django secret key. This will be used when
   # 'secret_key' is not set.
   ## secret_key_script=
 
   # Webserver listens on this address and port
-  http_host=0.0.0.0
+  http_host=localhost
   http_port=8000
 
   # A comma-separated list of available Hue load balancers
   ## hue_load_balancer=
 
   # Time zone name
-  time_zone=America/Los_Angeles
+  time_zone=Asia/Shanghai
 
   # Enable or disable debug mode.
-  ## django_debug_mode=true
+  django_debug_mode=false

   # Enable development mode, where notably static files are not cached.
-  dev=true
+  dev=false
 
   # Enable or disable database debug mode.
   ## database_logging=false
 
   # Whether to send debug messages from JavaScript to the server logs.
-  send_dbug_messages=true
+  send_dbug_messages=false
 
   # Enable or disable backtrace for server error
   ## http_500_debug_mode=true
@@ -74,11 +74,11 @@
   ## gunicorn_number_of_workers=1
 
   # Webserver runs as this user
-  ## server_user=hue
-  ## server_group=hue
+  server_user=sun_xo
+  server_group=sun_xo
 
   # This should be the Hue admin and proxy user
-  ## default_user=hue
+  default_user=sun_xo

   # This should be the hadoop cluster admin
   ## default_hdfs_superuser=hdfs
@@ -662,17 +662,17 @@
     # Note for Oracle, options={"threaded":true} must be set in order to avoid crashes.
     # Note for Oracle, you can use the Oracle Service Name by setting "host=" and "port=" and then "name=<host>:<port>/<service_name>".
     # Note for MariaDB use the 'mysql' engine.
-    ## engine=sqlite3
-    ## host=
-    ## port=
-    ## user=
-    ## password=
+    engine=postgresql_psycopg2
+    host=localhost
+    port=5432
+    user=hue_u
+    password=huepassword
     # conn_max_age option to make database connection persistent value in seconds
     # https://docs.djangoproject.com/en/1.9/ref/databases/#persistent-connections
     ## conn_max_age=0
     # Execute this script to produce the database password. This will be used when 'password' is not set.
     ## password_script=/path/script
-    ## name=desktop/desktop.db
+    name=hue_d
     ## options={}

## to browse hue remotely, we should:
## restore http_host=0.0.0.0
## configure gunicorn to avoid following socket issue:

[2022-06-21 10:28:40 +0800] [35077] [ERROR] Socket error processing request.
Traceback (most recent call last):
  File "~/work/hue/build/env/lib/python3.8/site-packages/gunicorn/workers/base_async.py", line 66, in handle
    six.reraise(*sys.exc_info())
...
  File "~/work/hue/build/env/lib/python3.8/site-packages/gunicorn/http/wsgi.py", line 393, in sendfile
    sent += sendfile(sockno, fileno, offset + sent, count)
BlockingIOError: [Errno 11] Resource temporarily unavailable

$ cd desktop/core/src/desktop/management/commands
$ diff -u rungunicornserver.py.orig rungunicornserver.py

--- rungunicornserver.py.orig	2022-06-20 14:54:02.078402272 +0800
+++ rungunicornserver.py	2022-06-21 11:19:54.266358329 +0800
@@ -131,7 +131,7 @@
       'raw_paste_global_conf': None,
       'reload': None,
       'reload_engine': None,
-      'sendfile': None,
+      'sendfile': False,
       'spew': None,
       'ssl_version': ssl.PROTOCOL_TLSv1_2,    # SSL version to use (see stdlib ssl module) [ssl.PROTOCOL_SSLv23, ssl.PROTOCOL_TLSv1]
       'statsd_host': None,

## perform "supervisor" instead of "hue runserver"

4) Start
$ cd ~/work/hue
$ build/env/bin/hue runserver   # or build/env/bin/supervisor

$ build/env/bin/hue runserver
Performing system checks...

System check identified no issues (0 silenced).
June 06, 2022 - 11:01:51
Django version 3.2.13, using settings 'desktop.settings'
Starting development server at http://127.0.0.1:8000/
Quit the server with CONTROL-C.

## browse hue on http://localhost:8000
## register hue_u / huepassword as admin user
## login as hue_u, create a new user sun_xo / sun_xo
## re-login as sun_xo

5) Set dev environment with PyCharm
## open directory ~/work/hue as a new project
## Preferences -> Project: hue ->  Python interpreter: Python 3.8 (env) ~/work/hue/build/env/python
## Run/Debug Configurations -> Configuration
##     Script path: ~/work/hue/build/env/bin/hue
##     Parameters: runserver

## Note: I encounterred a import issue between 2 modules:
ImportError: cannot import name 'TGetCrossReferenceReq' from 'TCLIService.ttypes' (~/work/hue/apps/impala/gen-py/TCLIService/ttypes.py)
Looks it should import TCLIService from hue/apps/beeswax/gen-py
    hue/apps/beeswax/gen-py/TCLIService
The workaround is in Project:hue -> Project Structure -> Cuntent Root, remove apps/impala/gen-py from list, and then add it back, ensure it is behind of apps/beeswas/gen-py
Debugging with PyCharm