Hue介绍入门--Hadoop可视化分析利_yefengzhichen的博客-CSDN博客
Hue介绍入门--Hadoop可视化分析利
yefengzhichen
于 2016-03-29 14:19:25 发布
20045
收藏
分类专栏：
Web
文章标签：
可视化
Hue
hadoop
Web
专栏收录该内容
1 篇文章
0 订阅
订阅专栏
先来看下hue的架构图：
（1）Hue是什么？
Hue是一个可快速开发和调试Hadoop生态系统各种应用的一个基于浏览器的图形化用户接口。
（2）Hue能干什么？
1，访问HDFS和文件浏览
2，通过web调试和开发hive以及数据结果展示
3，查询solr和结果展示，报表生成
4，通过web调试和开发impala交互式SQL Query
5，spark调试和开发
6，Pig开发和调试
7，oozie任务的开发，监控，和工作流协调调度
8，Hbase数据查询和修改，数据展示
9，Hive的元数据（metastore）查询
10，MapReduce任务进度查看，日志追踪
11，创建和提交MapReduce，Streaming，Java job任务
12，Sqoop2的开发和调试
13，Zookeeper的浏览和编辑
14，数据库（MySQL，PostGres，SQlite，Oracle）的查询和展示
（3）Hue怎么用或者什么时候应该用？
如果你们公司用的是CDH的hadoop，那么很幸运，Hue也是出自CDH公司，自家的东西用起来当然很爽。
如果你们公司用的是Apache Hadoop或者是HDP的hadoop，那么也没事，Hue是开源的，而且支持任何版本的hadoop。
关于什么时候用，这纯属一个锦上添花的功能，你完全可以不用hue，因为各种开源项目都有自己的使用方式和开发接口，hue只不过是统一了各个项目的开发方式在一个接口里而已，这样比较方便而已，不用你一会准备使用hive，就开一个hive的cli终端，一会用pig，你就得开一个pig的grunt，或者你又想查Hbase，又得需要开一个Hbase的shell终端。如果你们使用hadoop生态系统的组件很多的情况下，使用hue还是比较方便的，另外一个好处就是hue提供了一个web的界面来开发和调试任务，不用我们再频繁登陆Linux来操作了。
你可以在任何时候，只要能上网，就可以通过hue来开发和调试数据，不用再装Linux的客户端来远程登陆操作了，这也是B/S架构的好处。
（4）如何下载，安装和编译Hue？
centos系统，执行命令：
yum install -y asciidoc cyrus-sasl-devel cyrus-sasl-gssapi gcc gcc-c++ krb5-devel libtidy libxml2-devel libxslt-devel make mysql mysql-devel openldap-devel python-devel sqlite-devel openssl-devel gmp-devel
1，hue的依赖（centos系统）
Java代码
ant asciidoc cyrus-sasl-devel cyrus-sasl-gssapi gcc gcc-c++ krb5-devel libtidy (for unit tests only) libxml2-devel libxslt-devel make mvn (from maven package or maven3 tarball) mysql mysql-devel openldap-devel python-devel sqlite-devel openssl-devel (for version 7+)
2，散仙的在安装hue前，centos上已经安装好了，jdk，maven，ant，hadoop，hive，oozie等，环境变量如下：
Java代码
user="search" # java export JAVA_HOME="/usr/local/jdk" export CLASSPATH=.:$CLASSPATH:$JAVA_HOME/lib:$JAVA_HOME/jre/lib export PATH=$PATH:$JAVA_HOME/bin # ant export ANT_HOME=/usr/local/ant export CLASSPATH=$CLASSPATH:$ANT_HOME/lib export PATH=$PATH:$ANT_HOME/bin # maven export MAVEN_HOME="/usr/local/maven" export CLASSPATH=$CLASSPATH:$MAVEN_HOME/lib export PATH=$PATH:$MAVEN_HOME/bin ##Hadoop2.2的变量设置 export PATH=$PATH:$HADOOP_HOME/bin:$HADOOP_HOME/sbin export HADOOP_HOME=/home/search/hadoop export HADOOP_MAPRED_HOME=$HADOOP_HOME export HADOOP_COMMON_HOME=$HADOOP_HOME export HADOOP_HDFS_HOME=$HADOOP_HOME export YARN_HOME=$HADOOP_HOME export HADOOP_CONF_DIR=$HADOOP_HOME/etc/hadoop export YARN_CONF_DIR=$HADOOP_HOME/etc/hadoop export PATH=$PATH:$HADOOP_HOME/bin:$HADOOP_HOME/sbin export CLASSPATH=.:$CLASSPATH:$HADOOP_COMMON_HOME:$HADOOP_COMMON_HOMEi/lib:$HADOOP_MAPRED_HOME:$HADOOP_HDFS_HOME:$HADOOP_HDFS_HOME # Hive export HIVE_HOME=/home/search/hive export HIVE_CONF_DIR=/home/search/hive/conf export CLASSPATH=$CLASSPATH:$HIVE_HOME/lib export PATH=$PATH:$HIVE_HOME/bin:$HIVE_HOME/conf export OOZIE_HOME="/home/search/oozie-4.1.0" export PATH=$PATH:$OOZIE_HOME/sbin:$OOZIE_HOME/bin
3,本文散仙主要是采用tar包的方式安装hue，除了tar包的方式，hue还能采用cm安装，当然这就与cdh的系统依赖比较大了。
hue最新的版本是3.8.1，散仙这里用的3.7.0的版本
下载地址：
https://github.com/cloudera/hue/releases
hue的github地址：
https://github.com/cloudera/hue
4，下载完后，解压tar包，并进入hue的根目录执行命令
make apps编译
5，编译成功后，需要配置/home/search/hue/desktop/conf/pseudo-distributed.ini文件，里面包含了hdfs，yarn，mapreduce，hive，oozie，pig，spark，solr等的ip地址和端口号配置，可根据自己的情况设置，如果没有安装某个应用，那就无须配置，只不过这个应用在web上不能使用而已，并不会影响其他框架的使用。
一个例子如下：
Java代码
##################################### # DEVELOPMENT EDITION ##################################### # Hue configuration file # =================================== # # For complete documentation about the contents of this file, run # $ <hue_root>/build/env/bin/hue config_help # # All .ini files under the current directory are treated equally. Their # contents are merged to form the Hue configuration, which can # can be viewed on the Hue at # http://<hue_host>:<port>/dump_config ########################################################################### # General configuration for core Desktop features (authentication, etc) ########################################################################### [desktop] send_dbug_messages=1 # To show database transactions, set database_logging to 1 database_logging=0 # Set this to a random string, the longer the better. # This is used for secure hashing in the session store. secret_key=search # Webserver listens on this address and port http_host=0.0.0.0 http_port=8000 # Time zone name time_zone=Asia/Shanghai # Enable or disable Django debug mode ## django_debug_mode=true # Enable or disable backtrace for server error ## http_500_debug_mode=true # Enable or disable memory profiling. ## memory_profiler=false # Server email for internal error messages ## django_server_email='hue@localhost.localdomain' # Email backend ## django_email_backend=django.core.mail.backends.smtp.EmailBackend # Webserver runs as this user server_user=search server_group=search # This should be the Hue admin and proxy user default_user=search # This should be the hadoop cluster admin default_hdfs_superuser=search # If set to false, runcpserver will not actually start the web server. # Used if Apache is being used as a WSGI container. ## enable_server=yes # Number of threads used by the CherryPy web server ## cherrypy_server_threads=10 # Filename of SSL Certificate ## ssl_certificate= # Filename of SSL RSA Private Key ## ssl_private_key= # List of allowed and disallowed ciphers in cipher list format. # See http://www.openssl.org/docs/apps/ciphers.html for more information on cipher list format. ## ssl_cipher_list=DEFAULT:!aNULL:!eNULL:!LOW:!EXPORT:!SSLv2 # LDAP username and password of the hue user used for LDAP authentications. # Set it to use LDAP Authentication with HiveServer2 and Impala. ## ldap_username=hue ## ldap_password= # Default encoding for site data ## default_site_encoding=utf-8 # Help improve Hue with anonymous usage analytics. # Use Google Analytics to see how many times an application or specific section of an application is used, nothing more. ## collect_usage=true # Support for HTTPS termination at the load-balancer level with SECURE_PROXY_SSL_HEADER. ## secure_proxy_ssl_header=false # Comma-separated list of Django middleware classes to use. # See https://docs.djangoproject.com/en/1.4/ref/middleware/ for more details on middlewares in Django. ## middleware=desktop.auth.backend.LdapSynchronizationBackend # Comma-separated list of regular expressions, which match the redirect URL. # For example, to restrict to your local domain and FQDN, the following value can be used: # ^\/.*$,^http:\/\/www.mydomain.com\/.*$ ## redirect_whitelist= # Comma separated list of apps to not load at server startup. # e.g.: pig,zookeeper ## app_blacklist= # The directory where to store the auditing logs. Auditing is disable if the value is empty. # e.g. /var/log/hue/audit.log ## audit_event_log_dir= # Size in KB/MB/GB for audit log to rollover. ## audit_log_max_file_size=100MB #poll_enabled=false # Administrators # ---------------- [[django_admins]] ## [[[admin1]]] ## name=john ## email=john@doe.com # UI customizations # ------------------- [[custom]] # Top banner HTML code #banner_top_html=Search Team Hadoop Manager # Configuration options for user authentication into the web application # ------------------------------------------------------------------------ [[auth]] # Authentication backend. Common settings are: # - django.contrib.auth.backends.ModelBackend (entirely Django backend) # - desktop.auth.backend.AllowAllBackend (allows everyone) # - desktop.auth.backend.AllowFirstUserDjangoBackend # (Default. Relies on Django and user manager, after the first login) # - desktop.auth.backend.LdapBackend # - desktop.auth.backend.PamBackend # - desktop.auth.backend.SpnegoDjangoBackend # - desktop.auth.backend.RemoteUserDjangoBackend # - libsaml.backend.SAML2Backend # - libopenid.backend.OpenIDBackend # - liboauth.backend.OAuthBackend # (New oauth, support Twitter, Facebook, Google+ and Linkedin ## backend=desktop.auth.backend.AllowFirstUserDjangoBackend # The service to use when querying PAM. ## pam_service=login # When using the desktop.auth.backend.RemoteUserDjangoBackend, this sets # the normalized name of the header that contains the remote user. # The HTTP header in the request is converted to a key by converting # all characters to uppercase, replacing any hyphens with underscores # and adding an HTTP_ prefix to the name. So, for example, if the header # is called Remote-User that would be configured as HTTP_REMOTE_USER # # Defaults to HTTP_REMOTE_USER ## remote_user_header=HTTP_REMOTE_USER # Ignore the case of usernames when searching for existing users. # Only supported in remoteUserDjangoBackend. ## ignore_username_case=false # Ignore the case of usernames when searching for existing users to authenticate with. # Only supported in remoteUserDjangoBackend. ## force_username_lowercase=false # Users will expire after they have not logged in for 'n' amount of seconds. # A negative number means that users will never expire. ## expires_after=-1 # Apply 'expires_after' to superusers. ## expire_superusers=true # Configuration options for connecting to LDAP and Active Directory # ------------------------------------------------------------------- [[ldap]] # The search base for finding users and groups ## base_dn="DC=mycompany,DC=com" # URL of the LDAP server ## ldap_url=ldap://auth.mycompany.com # A PEM-format file containing certificates for the CA's that # Hue will trust for authentication over TLS. # The certificate for the CA that signed the # LDAP server certificate must be included among these certificates. # See more here http://www.openldap.org/doc/admin24/tls.html. ## ldap_cert= ## use_start_tls=true # Distinguished name of the user to bind as -- not necessary if the LDAP server # supports anonymous searches ## bind_dn="CN=ServiceAccount,DC=mycompany,DC=com" # Password of the bind user -- not necessary if the LDAP server supports # anonymous searches ## bind_password= # Pattern for searching for usernames -- Use <username> for the parameter # For use when using LdapBackend for Hue authentication ## ldap_username_pattern="uid=<username>,ou=People,dc=mycompany,dc=com" # Create users in Hue when they try to login with their LDAP credentials # For use when using LdapBackend for Hue authentication ## create_users_on_login = true # Synchronize a users groups when they login ## sync_groups_on_login=false # Ignore the case of usernames when searching for existing users in Hue. ## ignore_username_case=false # Force usernames to lowercase when creating new users from LDAP. ## force_username_lowercase=false # Use search bind authentication. ## search_bind_authentication=true # Choose which kind of subgrouping to use: nested or suboordinate (deprecated). ## subgroups=suboordinate # Define the number of levels to search for nested members. ## nested_members_search_depth=10 [[[users]]] # Base filter for searching for users ## user_filter="objectclass=*" # The username attribute in the LDAP schema ## user_name_attr=sAMAccountName [[[groups]]] # Base filter for searching for groups ## group_filter="objectclass=*" # The username attribute in the LDAP schema ## group_name_attr=cn [[[ldap_servers]]] ## [[[[mycompany]]]] # The search base for finding users and groups ## base_dn="DC=mycompany,DC=com" # URL of the LDAP server ## ldap_url=ldap://auth.mycompany.com # A PEM-format file containing certificates for the CA's that # Hue will trust for authentication over TLS. # The certificate for the CA that signed the # LDAP server certificate must be included among these certificates. # See more here http://www.openldap.org/doc/admin24/tls.html. ## ldap_cert= ## use_start_tls=true # Distinguished name of the user to bind as -- not necessary if the LDAP server # supports anonymous searches ## bind_dn="CN=ServiceAccount,DC=mycompany,DC=com" # Password of the bind user -- not necessary if the LDAP server supports # anonymous searches ## bind_password= # Pattern for searching for usernames -- Use <username> for the parameter # For use when using LdapBackend for Hue authentication ## ldap_username_pattern="uid=<username>,ou=People,dc=mycompany,dc=com" ## Use search bind authentication. ## search_bind_authentication=true ## [[[[[users]]]]] # Base filter for searching for users ## user_filter="objectclass=Person" # The username attribute in the LDAP schema ## user_name_attr=sAMAccountName ## [[[[[groups]]]]] # Base filter for searching for groups ## group_filter="objectclass=groupOfNames" # The username attribute in the LDAP schema ## group_name_attr=cn # Configuration options for specifying the Desktop Database. For more info, # see http://docs.djangoproject.com/en/1.4/ref/settings/#database-engine # ------------------------------------------------------------------------ [[database]] # Database engine is typically one of: # postgresql_psycopg2, mysql, sqlite3 or oracle. # # Note that for sqlite3, 'name', below is a a path to the filename. For other backends, it is the database name. # Note for Oracle, options={'threaded':true} must be set in order to avoid crashes. # Note for Oracle, you can use the Oracle Service Name by setting "port=0" and then "name=<host>:<port>/<service_name>". ## engine=sqlite3 ## host= ## port= ## user= ## password= ## name=desktop/desktop.db ## options={} # Configuration options for specifying the Desktop session. # For more info, see https://docs.djangoproject.com/en/1.4/topics/http/sessions/ # ------------------------------------------------------------------------ [[session]] # The cookie containing the users' session ID will expire after this amount of time in seconds. # Default is 2 weeks. ## ttl=1209600 # The cookie containing the users' session ID will be secure. # Should only be enabled with HTTPS. ## secure=false # The cookie containing the users' session ID will use the HTTP only flag. ## http_only=false # Use session-length cookies. Logs out the user when she closes the browser window. ## expire_at_browser_close=false # Configuration options for connecting to an external SMTP server # ------------------------------------------------------------------------ [[smtp]] # The SMTP server information for email notification delivery host=localhost port=25 user= password= # Whether to use a TLS (secure) connection when talking to the SMTP server tls=no # Default email address to use for various automated notification from Hue ## default_from_email=hue@localhost # Configuration options for Kerberos integration for secured Hadoop clusters # ------------------------------------------------------------------------ [[kerberos]] # Path to Hue's Kerberos keytab file ## hue_keytab= # Kerberos principal name for Hue ## hue_principal=hue/hostname.foo.com # Path to kinit ## kinit_path=/path/to/kinit # Configuration options for using OAuthBackend (Core) login # ------------------------------------------------------------------------ [[oauth]] # The Consumer key of the application ## consumer_key=XXXXXXXXXXXXXXXXXXXXX # The Consumer secret of the application ## consumer_secret=XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX # The Request token URL ## request_token_url=https://api.twitter.com/oauth/request_token # The Access token URL ## access_token_url=https://api.twitter.com/oauth/access_token # The Authorize URL ## authenticate_url=https://api.twitter.com/oauth/authorize ########################################################################### # Settings to configure SAML ########################################################################### [libsaml] # Xmlsec1 binary path. This program should be executable by the user running Hue. ## xmlsec_binary=/usr/local/bin/xmlsec1 # Entity ID for Hue acting as service provider. # Can also accept a pattern where '<base_url>' will be replaced with server URL base. ## entity_id="<base_url>/saml2/metadata/" # Create users from SSO on login. ## create_users_on_login=true # Required attributes to ask for from IdP. # This requires a comma separated list. ## required_attributes=uid # Optional attributes to ask for from IdP. # This requires a comma separated list. ## optional_attributes= # IdP metadata in the form of a file. This is generally an XML file containing metadata that the Identity Provider generates. ## metadata_file= # Private key to encrypt metadata with. ## key_file= # Signed certificate to send along with encrypted metadata. ## cert_file= # A mapping from attributes in the response from the IdP to django user attributes. ## user_attribute_mapping={'uid':'username'} # Have Hue initiated authn requests be signed and provide a certificate. ## authn_requests_signed=false # Have Hue initiated logout requests be signed and provide a certificate. ## logout_requests_signed=false # Username can be sourced from 'attributes' or 'nameid'. ## username_source=attributes # Performs the logout or not. ## logout_enabled=true ########################################################################### # Settings to configure OpenId ########################################################################### [libopenid] # (Required) OpenId SSO endpoint url. ## server_endpoint_url=https://www.google.com/accounts/o8/id # OpenId 1.1 identity url prefix to be used instead of SSO endpoint url # This is only supported if you are using an OpenId 1.1 endpoint ## identity_url_prefix=https://app.onelogin.com/openid/your_company.com/ # Create users from OPENID on login. ## create_users_on_login=true # Use email for username ## use_email_for_username=true ########################################################################### # Settings to configure OAuth ########################################################################### [liboauth] # NOTE: # To work, each of the active (i.e. uncommented) service must have # applications created on the social network. # Then the "consumer key" and "consumer secret" must be provided here. # # The addresses where to do so are: # Twitter: https://dev.twitter.com/apps # Google+ : https://cloud.google.com/ # Facebook: https://developers.facebook.com/apps # Linkedin: https://www.linkedin.com/secure/developer # # Additionnaly, the following must be set in the application settings: # Twitter: Callback URL (aka Redirect URL) must be set to http://YOUR_HUE_IP_OR_DOMAIN_NAME/oauth/social_login/oauth_authenticated # Google+ : CONSENT SCREEN must have email address # Facebook: Sandbox Mode must be DISABLED # Linkedin: "In OAuth User Agreement", r_emailaddress is REQUIRED # The Consumer key of the application ## consumer_key_twitter= ## consumer_key_google= ## consumer_key_facebook= ## consumer_key_linkedin= # The Consumer secret of the application ## consumer_secret_twitter= ## consumer_secret_google= ## consumer_secret_facebook= ## consumer_secret_linkedin= # The Request token URL ## request_token_url_twitter=https://api.twitter.com/oauth/request_token ## request_token_url_google=https://accounts.google.com/o/oauth2/auth ## request_token_url_linkedin=https://www.linkedin.com/uas/oauth2/authorization ## request_token_url_facebook=https://graph.facebook.com/oauth/authorize # The Access token URL ## access_token_url_twitter=https://api.twitter.com/oauth/access_token ## access_token_url_google=https://accounts.google.com/o/oauth2/token ## access_token_url_facebook=https://graph.facebook.com/oauth/access_token ## access_token_url_linkedin=https://api.linkedin.com/uas/oauth2/accessToken # The Authenticate URL ## authenticate_url_twitter=https://api.twitter.com/oauth/authorize ## authenticate_url_google=https://www.googleapis.com/oauth2/v1/userinfo?access_token= ## authenticate_url_facebook=https://graph.facebook.com/me?access_token= ## authenticate_url_linkedin=https://api.linkedin.com/v1/people/~:(email-address)?format=json&oauth2_access_token= # Username Map. Json Hash format. # Replaces username parts in order to simplify usernames obtained # Example: {"@sub1.domain.com":"_S1", "@sub2.domain.com":"_S2"} # converts 'email@sub1.domain.com' to 'email_S1' ## username_map={} # Whitelisted domains (only applies to Google OAuth). CSV format. ## whitelisted_domains_google= ########################################################################### # Settings for the RDBMS application ########################################################################### [librdbms] # The RDBMS app can have any number of databases configured in the databases # section. A database is known by its section name # (IE sqlite, mysql, psql, and oracle in the list below). [[databases]] # sqlite configuration. ## [[[sqlite]]] # Name to show in the UI. ## nice_name=SQLite # For SQLite, name defines the path to the database. ## name=/tmp/sqlite.db # Database backend to use. ## engine=sqlite # Database options to send to the server when connecting. # https://docs.djangoproject.com/en/1.4/ref/databases/ ## options={} # mysql, oracle, or postgresql configuration. ## [[[mysql]]] # Name to show in the UI. ## nice_name="My SQL DB" # For MySQL and PostgreSQL, name is the name of the database. # For Oracle, Name is instance of the Oracle server. For express edition # this is 'xe' by default. ## name=mysqldb # Database backend to use. This can be: # 1. mysql # 2. postgresql # 3. oracle ## engine=mysql # IP or hostname of the database to connect to. ## host=localhost # Port the database server is listening to. Defaults are: # 1. MySQL: 3306 # 2. PostgreSQL: 5432 # 3. Oracle Express Edition: 1521 ## port=3306 # Username to authenticate with when connecting to the database. ## user=example # Password matching the username to authenticate with when # connecting to the database. ## password=example # Database options to send to the server when connecting. # https://docs.djangoproject.com/en/1.4/ref/databases/ ## options={} ########################################################################### # Settings to configure your Hadoop cluster. ########################################################################### [hadoop] # Configuration for HDFS NameNode # ------------------------------------------------------------------------ [[hdfs_clusters]] # HA support by using HttpFs [[[default]]] # Enter the filesystem uri fs_defaultfs=hdfs://h1:8020 # NameNode logical name. logical_name=h1 # Use WebHdfs/HttpFs as the communication mechanism. # Domain should be the NameNode or HttpFs host. # Default port is 14000 for HttpFs. webhdfs_url=http://h1:50070/webhdfs/v1 # Change this if your HDFS cluster is Kerberos-secured security_enabled=false # Default umask for file and directory creation, specified in an octal value. umask=022 hadoop_conf_dir=/home/search/hadoop/etc/hadoop # Configuration for YARN (MR2) # ------------------------------------------------------------------------ [[yarn_clusters]] [[[default]]] # Enter the host on which you are running the ResourceManager resourcemanager_host=h1 # The port where the ResourceManager IPC listens on resourcemanager_port=8032 # Whether to submit jobs to this cluster submit_to=True # Resource Manager logical name (required for HA) ## logical_name= # Change this if your YARN cluster is Kerberos-secured ## security_enabled=false # URL of the ResourceManager API resourcemanager_api_url=http://h1:8088 # URL of the ProxyServer API proxy_api_url=http://h1:8088 # URL of the HistoryServer API history_server_api_url=http://h1:19888 # HA support by specifying multiple clusters # e.g. # [[[ha]]] # Resource Manager logical name (required for HA) ## logical_name=my-rm-name # Configuration for MapReduce (MR1) # ------------------------------------------------------------------------ [[mapred_clusters]] [[[default]]] # Enter the host on which you are running the Hadoop JobTracker jobtracker_host=h1 # The port where the JobTracker IPC listens on #jobtracker_port=8021 # JobTracker logical name for HA ## logical_name= # Thrift plug-in port for the JobTracker ## thrift_port=9290 # Whether to submit jobs to this cluster submit_to=False # Change this if your MapReduce cluster is Kerberos-secured ## security_enabled=false # HA support by specifying multiple clusters # e.g. # [[[ha]]] # Enter the logical name of the JobTrackers # logical_name=my-jt-name ########################################################################### # Settings to configure the Filebrowser app ########################################################################### [filebrowser] # Location on local filesystem where the uploaded archives are temporary stored. ## archive_upload_tempdir=/tmp ########################################################################### # Settings to configure liboozie ########################################################################### [liboozie] # The URL where the Oozie service runs on. This is required in order for # users to submit jobs. Empty value disables the config check. ## oozie_url=http://localhost:11000/oozie oozie_url=http://h1:11000/oozie # Requires FQDN in oozie_url if enabled ## security_enabled=false # Location on HDFS where the workflows/coordinator are deployed when submitted. remote_deployement_dir=/user/hue/oozie/deployments ########################################################################### # Settings to configure the Oozie app ########################################################################### [oozie] # Location on local FS where the examples are stored. local_data_dir=apps/oozie/examples/ # Location on local FS where the data for the examples is stored. ## sample_data_dir=...thirdparty/sample_data # Location on HDFS where the oozie examples and workflows are stored. remote_data_dir=apps/oozie/workspaces # Maximum of Oozie workflows or coodinators to retrieve in one API call. oozie_jobs_count=100 # Use Cron format for defining the frequency of a Coordinator instead of the old frequency number/unit. ## enable_cron_scheduling=true enable_cron_scheduling=true ########################################################################### # Settings to configure Beeswax with Hive ########################################################################### [beeswax] # Host where HiveServer2 is running. # If Kerberos security is enabled, use fully-qualified domain name (FQDN). hive_server_host=h1 # Port where HiveServer2 Thrift server runs on. hive_server_port=10000 # Hive configuration directory, where hive-site.xml is located hive_conf_dir=/home/search/hive/conf # Timeout in seconds for thrift calls to Hive service server_conn_timeout=120 # Set a LIMIT clause when browsing a partitioned table. # A positive value will be set as the LIMIT. If 0 or negative, do not set any limit. browse_partitioned_table_limit=250 # A limit to the number of rows that can be downloaded from a query. # A value of -1 means there will be no limit. # A maximum of 65,000 is applied to XLS downloads. download_row_limit=1000000 # Hue will try to close the Hive query when the user leaves the editor page. # This will free all the query resources in HiveServer2, but also make its results inaccessible. ## close_queries=false # Thrift version to use when communicating with HiveServer2 ## thrift_version=5 [[ssl]] # SSL communication enabled for this server. ## enabled=false # Path to Certificate Authority certificates. ## cacerts=/etc/hue/cacerts.pem # Path to the private key file. ## key=/etc/hue/key.pem # Path to the public certificate file. ## cert=/etc/hue/cert.pem # Choose whether Hue should validate certificates received from the server. ## validate=true ########################################################################### # Settings to configure Pig ########################################################################### [pig] # Location of piggybank.jar on local filesystem. local_sample_dir=/home/search/hue/apps/pig/examples # Location piggybank.jar will be copied to in HDFS. remote_data_dir=/home/search/pig/examples ########################################################################### # Settings to configure Sqoop ########################################################################### [sqoop] # For autocompletion, fill out the librdbms section. # Sqoop server URL server_url=http://h1:12000/sqoop ########################################################################### # Settings to configure Proxy ########################################################################### [proxy] # Comma-separated list of regular expressions, # which match 'host:port' of requested proxy target. ## whitelist=(localhost|127\.0\.0\.1):(50030|50070|50060|50075) # Comma-separated list of regular expressions, # which match any prefix of 'host:port/path' of requested proxy target. # This does not support matching GET parameters. ## blacklist= ########################################################################### # Settings to configure Impala ########################################################################### [impala] # Host of the Impala Server (one of the Impalad) ## server_host=localhost # Port of the Impala Server ## server_port=21050 # Kerberos principal ## impala_principal=impala/hostname.foo.com # Turn on/off impersonation mechanism when talking to Impala ## impersonation_enabled=False # Number of initial rows of a result set to ask Impala to cache in order # to support re-fetching them for downloading them. # Set to 0 for disabling the option and backward compatibility. ## querycache_rows=50000 # Timeout in seconds for thrift calls ## server_conn_timeout=120 # Hue will try to close the Impala query when the user leaves the editor page. # This will free all the query resources in Impala, but also make its results inaccessible. ## close_queries=true # If QUERY_TIMEOUT_S > 0, the query will be timed out (i.e. cancelled) if Impala does not do any work # (compute or send back results) for that query within QUERY_TIMEOUT_S seconds. ## query_timeout_s=600 ########################################################################### # Settings to configure HBase Browser ########################################################################### [hbase] # Comma-separated list of HBase Thrift servers for clusters in the format of '(name|host:port)'. # Use full hostname with security. ## hbase_clusters=(Cluster|localhost:9090) # HBase configuration directory, where hbase-site.xml is located. ## hbase_conf_dir=/etc/hbase/conf # Hard limit of rows or columns per row fetched before truncating. ## truncate_limit = 500 # 'buffered' is the default of the HBase Thrift Server and supports security. # 'framed' can be used to chunk up responses, # which is useful when used in conjunction with the nonblocking server in Thrift. ## thrift_transport=buffered ########################################################################### # Settings to configure Solr Search ########################################################################### [search] # URL of the Solr Server solr_url=http://172.21.50.41:8983/solr/ # Requires FQDN in solr_url if enabled ## security_enabled=false ## Query sent when no term is entered ## empty_query=*:* ########################################################################### # Settings to configure Solr Indexer ########################################################################### [indexer] # Location of the solrctl binary. ## solrctl_path=/usr/bin/solrctl # Location of the solr home. ## solr_home=/usr/lib/solr # Zookeeper ensemble. ## solr_zk_ensemble=localhost:2181/solr # The contents of this directory will be copied over to the solrctl host to its temporary directory. ## config_template_path=/../hue/desktop/libs/indexer/src/data/solr_configs ########################################################################### # Settings to configure Job Designer ########################################################################### [jobsub] # Location on local FS where examples and template are stored. ## local_data_dir=..../data # Location on local FS where sample data is stored ## sample_data_dir=...thirdparty/sample_data ########################################################################### # Settings to configure Job Browser ########################################################################### [jobbrowser] # Share submitted jobs information with all users. If set to false, # submitted jobs are visible only to the owner and administrators. ## share_jobs=true ########################################################################### # Settings to configure the Zookeeper application. ########################################################################### [zookeeper] [[clusters]] [[[default]]] # Zookeeper ensemble. Comma separated list of Host/Port. # e.g. localhost:2181,localhost:2182,localhost:2183 host_ports=zk1:2181 # The URL of the REST contrib service (required for znode browsing) ## rest_url=http://localhost:9998 ########################################################################### # Settings to configure the Spark application. ########################################################################### [spark] # URL of the REST Spark Job Server. server_url=http://h1:8080/ ########################################################################### # Settings for the User Admin application ########################################################################### [useradmin] # The name of the default user group that users will be a member of ## default_user_group=default ########################################################################### # Settings for the Sentry lib ########################################################################### [libsentry] # Hostname or IP of server. ## hostname=localhost # Port the sentry service is running on. ## port=8038 # Sentry configuration directory, where sentry-site.xml is located. ## sentry_conf_dir=/etc/sentry/conf
编译好的目录如下：
Java代码
-rw-rw-r-- 1 search search 2782 5月 19 06:04 app.reg -rw-rw-r-- 1 search search 2782 5月 19 05:41 app.reg.bak drwxrwxr-x 22 search search 4096 5月 20 01:05 apps drwxrwxr-x 3 search search 4096 5月 19 05:41 build drwxr-xr-x 2 search search 4096 5月 19 05:40 data drwxrwxr-x 7 search search 4096 5月 20 01:29 desktop drwxrwxr-x 2 search search 4096 5月 19 05:41 dist drwxrwxr-x 7 search search 4096 5月 19 05:40 docs drwxrwxr-x 3 search search 4096 5月 19 05:40 ext -rw-rw-r-- 1 search search 11358 5月 19 05:38 LICENSE.txt drwxrwxr-x 2 search search 4096 5月 20 01:29 logs -rw-rw-r-- 1 search search 8121 5月 19 05:41 Makefile -rw-rw-r-- 1 search search 8505 5月 19 05:41 Makefile.sdk -rw-rw-r-- 1 search search 3093 5月 19 05:40 Makefile.tarball -rw-rw-r-- 1 search search 3498 5月 19 05:41 Makefile.vars -rw-rw-r-- 1 search search 2302 5月 19 05:41 Makefile.vars.priv drwxrwxr-x 2 search search 4096 5月 19 05:41 maven -rw-rw-r-- 1 search search 801 5月 19 05:40 NOTICE.txt -rw-rw-r-- 1 search search 4733 5月 19 05:41 README.rst -rw-rw-r-- 1 search search 52 5月 19 05:38 start.sh -rw-rw-r-- 1 search search 65 5月 19 05:41 stop.sh drwxrwxr-x 9 search search 4096 5月 19 05:38 tools -rw-rw-r-- 1 search search 932 5月 19 05:41 VERSION
6，启动hue，执行命令：build/env/bin/supervisor
Java代码
[search@h1 hue]$ build/env/bin/supervisor [INFO] Not running as root, skipping privilege drop starting server with options {'ssl_certificate': None, 'workdir': None, 'server_name': 'localhost', 'host': '0.0.0.0', 'daemonize': False, 'threads': 10, 'pidfile': None, 'ssl_private_key': None, 'server_group': 'search', 'ssl_cipher_list': 'DEFAULT:!aNULL:!eNULL:!LOW:!EXPORT:!SSLv2', 'port': 8000, 'server_user': 'search'}
然后我们就可以访问安装机ip+8000端口来查看了：
工具箱界面：
hive的界面：
在配置hive（散仙这里是0.13的版本）的时候，需要注意以下几个方面：
hive的metastrore的服务和hiveserver2服务都需要启动
执行下面命令
bin/hive --service metastore
bin/hiveserver2
除此之外，还需要关闭的hive的SAL认证，否则，使用hue访问会出现问题。
注意下面三项的配置
Java代码
<property> <name>hive.metastore.warehouse.dir</name> <value>/user/hive/warehouse</value> <description>location of default database for the warehouse</description> </property> <property> <name>hive.server2.thrift.port</name> <value>10000</value> <description>Port number of HiveServer2 Thrift interface. Can be overridden by setting $HIVE_SERVER2_THRIFT_PORT</description> </property> <property> <name>hive.server2.thrift.bind.host</name> <value>h1</value> <description>Bind host on which to run the HiveServer2 Thrift interface. Can be overridden by setting $HIVE_SERVER2_THRIFT_BIND_HOST</description> </property> <property> <name>hive.server2.authentication</name> <value>NOSASL</value> <description> Client authentication types. NONE: no authentication check LDAP: LDAP/AD based authentication KERBEROS: Kerberos/GSSAPI authentication CUSTOM: Custom authentication provider (Use with property hive.server2.custom.authentication.class) PAM: Pluggable authentication module. </description> </property>
除了上面的配置外，还需要把hive.server2.long.polling.timeout的参数值，默认是5000L给改成5000,否则使用beenline连接时候，会出错，这是hive的一个bug。
pig的界面：
solr的界面如下：
最后需要注意一点，hue也需要在hadoop的core-site.xml里面配置相应的代理用户，示例如下：
Java代码
<property> <name>hadoop.proxyuser.hue.hosts</name> <value>*</value> </property> <property> <name>hadoop.proxyuser.hue.groups</name> <value>*</value> </property>
ok至此，我们的hue已经能完美工作了，我们可以根据自己的需要，定制相应的app插件，非常灵活！
转载自：http://qindongliang.iteye.com/blog/2212619
yefengzhichen
关注
关注
点赞
收藏
评论
Hue介绍入门--Hadoop可视化分析利
先来看下hue的架构图： （1）Hue是什么？ Hue是一个可快速开发和调试Hadoop生态系统各种应用的一个基于浏览器的图形化用户接口。 （2）Hue能干什么？ 1，访问HDFS和文件浏览 2，通过web调试和开发hive以及数据结果展示 3，查询solr和结果展示，报表生成 4，通过web调试和开发impala交互式SQL Query
复制链接
扫一扫
专栏目录
Hive可视化客户端工具
marko39的专栏
07-21
1万+
目前市面上的Hive可视化客户端工具，大都是C/S模式的，安装使用都不是太方便，目前有一款基于WEB的可视化工具TreeSoft, 通过浏览器就可以访问使用了，并且可以同时管理、维护、监控MySQL,Oracle等关系数据库，值得一用。
1、配置数据库的连接参数，可以Hive，MySQL同时配置，方便查看原始数据，以及回写到MySQL的数据。
2、切库换数据库，查看及维护...
hadoop平台+低保数据采集+分析+可视化
weixin_40903057的博客
05-27
2107
1. 需求：
1.1 背景描述
随着中国经济的迅速发展，广大人民群众的收入也在不断提高，但是仍有
一部分人群需要在经济上获得帮助，每月领取城市居民最低生活保障，为了更
好的实现精准扶贫，你所在的小组需要通过给出的“某市低保人群信息表”，
通过采集其中低保人口收入，低保人口中失业人数，视力或听力残疾人数等信
息，分析各地区低保人群数量，人均收入情况，近几年收入总体情况等信息，
为实现精准扶贫给予...
参与评论
您还未登录，请先
登录
后发表或查看评论
大数据Hadoop之——Hadoop图形化管理系统Hue（Hue环境部署）
匠人精神，持之以恒！
04-19
3440
文章目录
一、概述
Hue是一个开源的Apache Hadoop UI系统，最早是由Cloudera Desktop演化而来，由Cloudera贡献给开源社区，它是基于Python Web框架Django实现的。通过使用Hue我们可以在浏览器端的Web控制台上与Hadoop集群进行交互来分析处理数据，例如操作HDFS上的数据，运行MapReduce Job等等。
Hue官方网站：https://gethue.com/
官方安装文档：https://docs.gethue.com/administrato
hue的安装和hadoop集群整合
自律
11-17
571
一、Hue概述
二、环境准备
Jdk1.8
Root用户启动hdfs、yarn
正常连接外网
hue源码包
下载地址:http://archive.cloudera.com/cdh5/cdh/5/hue-3.9.0-cdh5.5.4.tar.gz
maven环境
https://archive.apache.org/dist/maven/maven-3/3.3.9/binaries/
下载apache-maven-3.3.9-bin.tar.gz
三、maven安装
(1) 上传到software并解压
Python绘图总结(seaborn篇)之概况
weixin_42029738的博客
05-09
1328
导入seaborn会改变matlotlib默认的颜色和绘图样式，提高可读性和美感。即使不适用seaborn的API，也可以利用seaborn来提高可视化的效果。# 使用该魔法，不用写plt.show()，以及可以边写边运行# %matplotlib notebookimport matplotlib.pyplot as pltimport numpy as npimport pandas as p...
一起动手学数据分析 task04 数据可视化
weixin_43356993的博客
12-19
71
一起动手学数据分析~
【转载】Hadoop可视化分析利器之Hue
weixin_30586085的博客
02-24
741
http://qindongliang.iteye.com/blog/2212619
先来看下hue的架构图：（1）Hue是什么？Hue是一个可快速开发和调试Hadoop生态系统各种应用的一个基于浏览器的图形化用户接口。（2）Hue能干什么？1，访问HDFS和文件浏览2，通过web调试和开发hive以及数据结果展示3，查询solr和结果展示，报表生成4，通过web调试和开发imp...
大数据Hadoop之——智能数据分析可视化BI软件 FineBI
最新发布
匠人精神，持之以恒！
10-30
1395
FineBI 是帆软软件有限公司推出的一款商业智能（Business Intelligence）产品。FineBI 是定位于自助大数据分析的BI工具，能够帮助企业的业务人员和数据分析师，开展以问题导向的探索式分析。上面是官方的介绍，其实一般我们会将原始数据通过ETL（抽取（extract）、转换（transform）、加载（load））数据处理，当然也可以直接通过BI分析处理原数据，根据情况而定，最后会通过一款BI产品将处理好的数据或者原数据进行报表展示实时分析数据。
HUE上hive编辑器快捷键说明
liuwei0376的专栏
07-02
1312
一、引言
Cloudera的Hue组件提供了可视化的开发界面，可以方便进行大数据相关功能的便捷开发。
比如针对Hive开发的hive编辑器，就内置了非常多的快捷操作，善用这些操作，开发时势必会事半功倍。
二、如何查看快捷键
2.1 入口
2.2 Hive编辑器帮助文档
三、Hive编辑器的详细用法
3.1 hql注释添加方式
A comment is text that is not executed. It can be of two types:
注释不会被执行。hive编辑器支持两种注释：
单行注
hue的简介及基本原理
FlizhN的博客
10-09
4169
hue定义：
HUE=Hadoop User Experience
个人理解：可视图的webui界面，方便大数据技术之间的CRUD操作。
官方定义：Hue是一个能够与Apache Hadoop交互的Web应用程序。一个开源的Apache Hadoop UI。
特性：一个HDFS的文件浏览器，一个MapReduce/YARN的Job浏览器，一个 HBas的浏览器，Hive，Pig，Cloudera Impala 和 Sqoop2 的查询编辑器。它还附带了一个Oozie的应用程序，用于创建和监控工作流程
大数据技术可视化之Hue
05-05
1455
大数据技术可视化之Hue
【python】seaborn可视化之基础图
小白水手的博客
01-14
849
Python 中，数据可视化一般是通过较底层的 Matplotlib 库和较高层的 Seaborn (之前并不知道，留着学)库实现的，本文主要介绍一些常用的图的绘制方法。
离线数据系统之Hue详解
lixinkuan的博客
09-16
1036
一、Hue简介
Hue是一个开源的Apache Hadoop UI系统，最早是由Cloudera Desktop演化而来，由Cloudera贡献给开源社区，它是基于Python Web框架Django实现的。通过使用Hue我们可以在浏览器端的Web控制台上与Hadoop集群进行交互来分析处理数据，例如操作HDFS上的数据，运行MapReduce Job等等。
总结：
Hue是一个开源的Apa...
数据可视化3_Hadoop
niki__的博客
11-21
1351
Hadoop:
HDFS:Hadoop Distributed File System
MapReduce—>Spark—>PySpark
filter–>map—>reduce
数据专员—>Excel—>透视表
业务数据分析师 Python/SQL/Hive/Excel
数据挖掘工程师 SPSS/Excel/Python/R
算法工程师 编程+数学+算法+大数据
python测试代码性能
timeit
python -m timeit 代码
cprof
Hadoop大数据综合案例6--数据可视化(SpringBoot+ECharts)
CDHong.it的技术分享博客
05-17
9047
Spring Boot 简介
由于Spring是一个轻量级的企业开发框架，主要的功能就是用于整合和管理其他框架。但随着整合的框架越来越多，Spring的整合配置也日益繁琐，一度被人认为“配置地狱”。随着Spring 3.0的发布，Spring IO团队逐渐开始摆脱XML配置文件，并且在开发过程中大量使用约定优先配置的思想来摆脱Spring框架中各类繁复纷杂的配置（即时是Java Config）。
SpringBoot 正是在这样的一个背景下被抽象出来的开发框架，它本身并不提供Spring框架的核心特性以及扩
Hadoop之Hue
fang19970714的博客
05-23
511
Hadoop User Experience
一个图形化界面，提供一个Web界面可以供我们查看执行我们其他的框架
Hue是一个开源的Apache Hadoop UI系统，最早是由Cloudera Desktop演化而来，由Cloudera贡献给开源社区，它是基于Python Web框架Django实现的。通过使用Hue我们可以在浏览器端的Web控制台上与hadoop集群进行交互来分析处理数据，例如操作HDFS上的数据，运行MapReduce Job等等。简单就理解就是hadoop可视化工具
Hue集成H
【seaborn】散点图
热门推荐
学渣的博客
05-05
1万+
Seaborn是一个基于matplotlib的Python数据可视化库。它提供了一个高层次的界面，用于绘制有吸引力和信息丰富的统计图形。
引入数据集
iris是sklearn的一个4特征的数据集，其标签有三种取值：0，1，2
import seaborn as sns
import matplotlib
from sklearn import datasets
import pandas as p...
Hue大数据可视化分析
梦想天空
09-19
1645
目录
一、环境说明
二、Hue下载安装
1. 下载
2. 编译安装
3. 基础配置
三、框架集成
1. HDFS集成
2. YARN集成
3. Hive集成
4. Mysql集成
5. Hbase集成
四、测试
Hue是一个开源的Apache Hadoop UI系统，由Cloudera Desktop演化而来，最后Cloudera公司将其贡献给Apache基金会的Hado...
基于hadoop的气象数据可视化分析
文档源码
09-09
2384
也提供了气象数据的查询方式，如查询当天气象情况，当月情况，当年情况等。1）利用一些气象数据网站提供的API在线获取信息，将这些信息保存到Hbase.每天定时更新，方便快捷，便于城市气象数据的实时获取与维护。（2）分析全国各城市每个月的气象情况，如：当月平均气象，最高气象，最高风速情况等。提供选择指定城市，展现选定时间范围内气象情况的走势情况，展现形式为折线图。提供选择时间范围，选择分析数据方式，如：客户端分析，Hive分析等功能。2）将待上传的数据文件进行信息提取（气象，风速，云量，气象总体概况等）。
Hue 可视化的Hadoop架构工具
ZeroTeam_麒麟
05-11
1776
转载地址：http://www.datawhy.net/index.php?m=blog&f=view&id=42
1）简介
大数据可视化工具
2）编译
下载地址
编译帮助文档
##解压$ tar -zxvf hue-3.7.0-cdh5.3.6.tar.gz -C /opt/cdh/$ cd /opt/cdh$ mv hue-3.7.0-cdh5.3.6hue-3
“相关推荐”对你有帮助么？
非常没帮助
没帮助
一般
有帮助
非常有帮助
提交
©️2022 CSDN
皮肤主题：大白
设计师：CSDN官方博客
返回首页
yefengzhichen
CSDN认证博客专家
CSDN认证企业博客
码龄10年
暂无认证
118
原创
8万+
周排名
143万+
总排名
40万+
访问
等级
3854
积分
72
粉丝
137
获赞
62
评论
401
收藏
私信
关注
热门文章
python给scatter设置颜色渐变条colorbar
77204
python指数、幂数拟合curve_fit
61026
二分查找各种情况大总结
31295
Mysql tinyint长度为1时在java中被转化成boolean型
24524
Hue介绍入门--Hadoop可视化分析利
20044
分类专栏
linux
7篇
POJ
13篇
算法
9篇
STL源码系列
2篇
leetcode
38篇
C++
4篇
分布式系统
5篇
mysql和ORM
4篇
APUE
4篇
剑指offer
1篇
测试
2篇
JAVA
12篇
Web
1篇
面试金典
3篇
JAVAWEB
8篇
求职经历
1篇
Python
4篇
matlab
1篇
视频服务器
1篇
Nginx
1篇
golang
8篇
数据库
1篇
nsq
1篇
消息队列
1篇
源码分析
1篇
最新评论
Ubuntu14.04上配置hadoop
m0_73671972:
cd怎样进入源代码目录啊
python指数、幂数拟合curve_fit
小不点儿是仙女:
print('Y = '+str(a0) + '*exp(-t/'+str(a1)+")+" + str(a2))
类似这种
Ubuntu16上Nginx安装教程
Tisfy:
这让我想起了先贤的一句话：自是人生长恨，水长东。
python指数、幂数拟合curve_fit
weixin_49587842:
up,如何打印出拟合好的方程
从蓝光到4K，腾讯视频高码率下载背后的技术
南棱笑笑生:
既然腾讯视频这么NB，那为什么我只是在WIN7下使用腾讯视频的客户端看一个蓝光视频，平均2.5分钟卡顿一次？马化腾做的也太差劲了吧？
您愿意向朋友推荐“博客详情页”吗？
强烈不推荐
不推荐
一般般
推荐
强烈推荐
提交
最新文章
go消息队列nsq源码调用分析
golang mutex源码详细解析
MongoDB索引-设置记录超时自动删除
2019年12篇
2018年4篇
2017年1篇
2016年45篇
2015年76篇
目录
目录
分类专栏
linux
7篇
POJ
13篇
算法
9篇
STL源码系列
2篇
leetcode
38篇
C++
4篇
分布式系统
5篇
mysql和ORM
4篇
APUE
4篇
剑指offer
1篇
测试
2篇
JAVA
12篇
Web
1篇
面试金典
3篇
JAVAWEB
8篇
求职经历
1篇
Python
4篇
matlab
1篇
视频服务器
1篇
Nginx
1篇
golang
8篇
数据库
1篇
nsq
1篇
消息队列
1篇
源码分析
1篇
目录
评论
被折叠的 条评论
为什么被折叠?
到【灌水乐园】发言
查看更多评论
实付元
使用余额支付
点击重新获取
扫码支付
钱包余额
抵扣说明：
1.余额是钱包充值的虚拟货币，按照1:1的比例进行支付金额的抵扣。 2.余额无法直接购买下载，可以购买VIP、C币套餐、付费专栏及课程。
余额充值