Apache Ranger 编译安装

开源开发者笔记：DevOps，微服务，分布式，大数据，高可用，区块链，白皮书，算法，黑客，设计模式，面试题。求 star:star:️

Apache Ranger是大数据领域的一个集中式安全管理框架，目的是通过制定策略（policies）实现对Hadoop组件的集中式安全管理。用户可以通过Ranger实现对集群中数据的安全访问。

环境

源码编译

环境准备

• 下载 Ranger 源码

wget https://downloads.apache.org/ranger/2.0.0/apache-ranger-2.0.0.tar.gz
或使用镜像
wget https://mirrors.tuna.tsinghua.edu.cn/apache/ranger/2.0.0/apache-ranger-2.0.0.tar.gz
复制代码

• 下载 Maven

wget https://mirrors.tuna.tsinghua.edu.cn/apache/maven/maven-3/3.6.3/binaries/apache-maven-3.6.3-bin.tar.gz
复制代码

• 安装 java、python及c++编译环境

apt install openjdk-8-jdk
apt install python2.7
apt install gcc g++
ln -s /usr/bin/python2.7 /usr/bin/python
复制代码

• 配置环境变量

export MAVEN_HOME=/opt/app/apache-maven-3.6.3
export PATH=$PATH:$MAVEN_HOME/bin
export MAVEN_OPTS=-Xmx2048m
复制代码

• 使用 ali maven 源

/opt/app/apache-maven-3.6.2/conf/settings.xml

<mirrors>
   <mirror>
     <id>nexus-aliyun</id>
     <mirrorOf>central</mirrorOf>
     <name>Nexus aliyun</name>
     <url>http://maven.aliyun.com/nexus/content/groups/public</url>
   </mirror>

   <mirror>
     <id>CN</id>
     <name>OSChina Central</name>
     <url>http://maven.oschina.net/content/groups/public/</url>
     <mirrorOf>central</mirrorOf>
   </mirror>

   <mirror>
     <id>alimaven</id>
     <mirrorOf>central</mirrorOf>
     <name>aliyun maven</name>
     <url>https://maven.aliyun.com/nexus/content/repositories/central/</url>
   </mirror>

   <mirror>
     <id>jboss-public-repository-group</id>
     <mirrorOf>central</mirrorOf>
     <name>JBoss Public Repository Group</name>
     <url>https://repository.jboss.org/nexus/content/groups/public</url>
   </mirror>
</mirrors>
复制代码

编译

tar xf apache-ranger-2.0.0.tar.gz
cd apache-ranger-2.0.0
mvn clean compile package assembly:assembly install -DskipTests -Drat.skip=true

编译成功打开 target 文件夹
apache-ranger-2.0.0/target# ll
total 1605220
drwxr-xr-x  5 root root      4096 Jun 29 11:04 ./
drwxr-xr-x 59 root root      4096 Jun 29 03:20 ../
drwxr-xr-x  2 root root      4096 Jun 29 03:20 antrun/
drwxr-xr-x  2 root root      4096 Jun 29 11:04 archive-tmp/
drwxr-xr-x  3 root root      4096 Jun 29 03:20 maven-shared-archive-resources/
-rw-r--r--  1 root root        30 Jun 29 11:04 .plxarc
-rw-r--r--  1 root root 248635508 Jun 29 11:01 ranger-2.0.0-admin.tar.gz
-rw-r--r--  1 root root 249666712 Jun 29 11:02 ranger-2.0.0-admin.zip
-rw-r--r--  1 root root  27787895 Jun 29 11:03 ranger-2.0.0-atlas-plugin.tar.gz
-rw-r--r--  1 root root  27832012 Jun 29 11:03 ranger-2.0.0-atlas-plugin.zip
-rw-r--r--  1 root root  31555304 Jun 29 11:04 ranger-2.0.0-elasticsearch-plugin.tar.gz
-rw-r--r--  1 root root  31605841 Jun 29 11:04 ranger-2.0.0-elasticsearch-plugin.zip
-rw-r--r--  1 root root  26638377 Jun 29 10:58 ranger-2.0.0-hbase-plugin.tar.gz
-rw-r--r--  1 root root  26665982 Jun 29 10:58 ranger-2.0.0-hbase-plugin.zip
-rw-r--r--  1 root root  23971392 Jun 29 10:58 ranger-2.0.0-hdfs-plugin.tar.gz
-rw-r--r--  1 root root  23997623 Jun 29 10:58 ranger-2.0.0-hdfs-plugin.zip
-rw-r--r--  1 root root  23825995 Jun 29 10:58 ranger-2.0.0-hive-plugin.tar.gz
-rw-r--r--  1 root root  23854522 Jun 29 10:58 ranger-2.0.0-hive-plugin.zip
-rw-r--r--  1 root root  39930681 Jun 29 10:59 ranger-2.0.0-kafka-plugin.tar.gz
-rw-r--r--  1 root root  39983878 Jun 29 10:59 ranger-2.0.0-kafka-plugin.zip
-rw-r--r--  1 root root  90982883 Jun 29 11:02 ranger-2.0.0-kms.tar.gz
-rw-r--r--  1 root root  91106270 Jun 29 11:02 ranger-2.0.0-kms.zip
-rw-r--r--  1 root root  28380704 Jun 29 10:58 ranger-2.0.0-knox-plugin.tar.gz
-rw-r--r--  1 root root  28411022 Jun 29 10:58 ranger-2.0.0-knox-plugin.zip
-rw-r--r--  1 root root  23940874 Jun 29 11:03 ranger-2.0.0-kylin-plugin.tar.gz
-rw-r--r--  1 root root  23980053 Jun 29 11:03 ranger-2.0.0-kylin-plugin.zip
-rw-r--r--  1 root root     34223 Jun 29 11:02 ranger-2.0.0-migration-util.tar.gz
-rw-r--r--  1 root root     37740 Jun 29 11:02 ranger-2.0.0-migration-util.zip
-rw-r--r--  1 root root  26388071 Jun 29 11:00 ranger-2.0.0-ozone-plugin.tar.gz
-rw-r--r--  1 root root  26421136 Jun 29 11:00 ranger-2.0.0-ozone-plugin.zip
-rw-r--r--  1 root root  40302042 Jun 29 11:04 ranger-2.0.0-presto-plugin.tar.gz
-rw-r--r--  1 root root  40341626 Jun 29 11:04 ranger-2.0.0-presto-plugin.zip
-rw-r--r--  1 root root  22232050 Jun 29 11:02 ranger-2.0.0-ranger-tools.tar.gz
-rw-r--r--  1 root root  22248747 Jun 29 11:02 ranger-2.0.0-ranger-tools.zip
-rw-r--r--  1 root root     42667 Jun 29 11:02 ranger-2.0.0-solr_audit_conf.tar.gz
-rw-r--r--  1 root root     45636 Jun 29 11:02 ranger-2.0.0-solr_audit_conf.zip
-rw-r--r--  1 root root  26964416 Jun 29 11:00 ranger-2.0.0-solr-plugin.tar.gz
-rw-r--r--  1 root root  27010058 Jun 29 11:00 ranger-2.0.0-solr-plugin.zip
-rw-r--r--  1 root root  23952732 Jun 29 11:03 ranger-2.0.0-sqoop-plugin.tar.gz
-rw-r--r--  1 root root  23986050 Jun 29 11:03 ranger-2.0.0-sqoop-plugin.zip
-rw-r--r--  1 root root   4081388 Jun 29 11:03 ranger-2.0.0-src.tar.gz
-rw-r--r--  1 root root   6257752 Jun 29 11:03 ranger-2.0.0-src.zip
-rw-r--r--  1 root root  37230628 Jun 29 10:59 ranger-2.0.0-storm-plugin.tar.gz
-rw-r--r--  1 root root  37268719 Jun 29 10:59 ranger-2.0.0-storm-plugin.zip
-rw-r--r--  1 root root  32772036 Jun 29 11:02 ranger-2.0.0-tagsync.tar.gz
-rw-r--r--  1 root root  32782070 Jun 29 11:02 ranger-2.0.0-tagsync.zip
-rw-r--r--  1 root root  16256778 Jun 29 11:02 ranger-2.0.0-usersync.tar.gz
-rw-r--r--  1 root root  16280886 Jun 29 11:02 ranger-2.0.0-usersync.zip
-rw-r--r--  1 root root  23953849 Jun 29 10:59 ranger-2.0.0-yarn-plugin.tar.gz
-rw-r--r--  1 root root  23992362 Jun 29 10:59 ranger-2.0.0-yarn-plugin.zip
-rw-r--r--  1 root root         5 Jun 29 11:04 version
复制代码

编译时发生错误

• 没安装c++编译环境

[INFO] Unix Native Authenticator .......................... FAILURE [  1.387 s]
[INFO] ------------------------------------------------------------------------
[INFO] BUILD FAILURE
[INFO] ------------------------------------------------------------------------
[INFO] Total time: 08:51 min
[INFO] Finished at: 2020-06-030T12:00:43-02:00
[INFO] Final Memory: 96M/420M
[INFO] ------------------------------------------------------------------------
[ERROR] Failed to execute goal org.codehaus.mojo:native-maven-plugin:1.0-alpha-8:compile (default-compile) on project credValidator: Error executing command line. Exit code:127 -> [Help 1]
org.apache.maven.lifecycle.LifecycleExecutionException: Failed to execute goal org.codehaus.mojo:native-maven-plugin:1.0-alpha-8:compile (default-compile) on project credValidator: Error executing command line. Exit code:127
复制代码

• 没安装python

[INFO] ------------------------------------------------------------------------
[ERROR] Failed to execute goal org.apache.maven.plugins:maven-antrun-plugin:1.7:run (generate-version-annotation) on project ranger-util: An Ant BuildException has occured: exec returned: 1
[ERROR] around Ant part ...<exec failonerror="true" executable="python">... @ 4:48 in /data/apache-ranger-2.0.0/ranger-util/target/antrun/build-main.xml
[ERROR] -> [Help 1]
org.apache.maven.lifecycle.LifecycleExecutionException: Failed to execute goal org.apache.maven.plugins:maven-antrun-plugin:1.7:run (generate-version-annotation) on project ranger-util: An Ant BuildException has occured: exec returned: 1
around Ant part ...<exec failonerror="true" executable="python">... @ 4:48 in /data/apache-ranger-2.0.0/ranger-util/target/antrun/build-main.xml
复制代码

安装 ranger-admin

解压

tar zxvf ranger-2.0.0-admin.tar.gz -C /opt/ranger
cd /opt/ranger
复制代码

修改 ranger-2.0.0-admin/install.properties

# 数据库连接
SQL_CONNECTOR_JAR=/opt/ranger/mysql-connector-java-8.0.18.jar
db_root_user=root
db_root_password=root
db_host=localhost

db_name=ranger
db_user=ranger
db_password=rangeradmin

# 日志审计
audit_store=solr
audit_solr_urls=http://localhost:6083/solr/ranger_audits
audit_solr_user=solr
复制代码

执行 ranger-2.0.0-admin/setup.sh 进行安装

成功则输出。 install.properties 配置中需要的 solr 、 mysql 、 mysql驱动包 如果没有可参考下面

2020-06-30 07:55:44,211  [I] Checking connection passed.
Installation of Ranger PolicyManager Web Application is completed.
复制代码

执行 ranger-2.0.0-admin/set_globals.sh 配置软连接

执行 ranger-admin start 或者 ranger-2.0.0-admin/ews/ranger-admin-services.sh start 启动服务

浏览器访问 http://ip:6080/ admin/admin

安装 solr

修改 ranger-2.0.0-admin/contrib/solr_for_audit_setup/install.properties

SOLR_INSTALL=true

JAVA_HOME=/usr/lib/jvm/java-8-openjdk-amd64
SOLR_DOWNLOAD_URL=https://mirrors.tuna.tsinghua.edu.cn/apache/lucene/solr/8.5.2/solr-8.5.2.tgz

SOLR_INSTALL_FOLDER=/data/solr   #安装目录，这个随意
SOLR_RANGER_HOME=/data/solr/ranger_audit_server
SOLR_RANGER_PORT=6083   #默认端口
SOLR_DEPLOYMENT=standalone  #部署模式(单节点部署)，还有solrcloud(集群)模式
SOLR_RANGER_DATA_FOLDER=/data/solr/ranger_audit_server/data   #数据存放目录
复制代码

执行 ranger-2.0.0-admin/contrib/solr_for_audit_setup/setup.sh

Tue Jun 30 06:58:31 UTC 2020|INFO|Installed Solr in /opt/solr
Tue Jun 30 06:58:31 UTC 2020|INFO|Configuring standalone instance
Tue Jun 30 06:58:31 UTC 2020|INFO|Copying Ranger Audit Server configuration to /opt/solr/ranger_audit_server
Tue Jun 30 06:58:31 UTC 2020|INFO|Creating group solr
Tue Jun 30 06:58:31 UTC 2020|INFO|Creating user solr
Tue Jun 30 06:58:31 UTC 2020|INFO|Done configuring Solr for Apache Ranger Audit
Tue Jun 30 06:58:31 UTC 2020|INFO|Solr HOME for Ranger Audit is /opt/solr/ranger_audit_server
Tue Jun 30 06:58:31 UTC 2020|INFO|Data folder for Audit logs is /opt/solr/ranger_audit_server/data
Tue Jun 30 06:58:31 UTC 2020|INFO|To start Solr run /opt/solr/ranger_audit_server/scripts/start_solr.sh
Tue Jun 30 06:58:31 UTC 2020|INFO|To stop Solr run /opt/solr/ranger_audit_server/scripts/stop_solr.sh
Tue Jun 30 06:58:31 UTC 2020|INFO|After starting Solr for RangerAudit, it will listen at 6083. E.g http://weihai-2:6083
Tue Jun 30 06:58:31 UTC 2020|INFO|Configure Ranger to use the following URL http://weihai-2:6083/solr/ranger_audits
Tue Jun 30 06:58:31 UTC 2020|INFO| ** NOTE: If Solr is Secured then solrclient JAAS configuration has to be added to Ranger Admin and Ranger Plugin properties
Tue Jun 30 06:58:31 UTC 2020|INFO| ** Refer documentation on how to configure Ranger for audit to Secure Solr
########## Done ###################
Created file /opt/solr/ranger_audit_server/install_notes.txt with instructions to start and stop
###################################
复制代码

启动/停止 solr

/opt/solr/ranger_audit_server/scripts/start_solr.sh
/opt/solr/ranger_audit_server/scripts/stop_solr.sh
复制代码

安装 MySQL

• 下载 mysql 驱动包 https://downloads.mysql.com/archives/c-j/

• 下载 mysql-server

  • 官网下载

  https://dev.mysql.com/downloads/mysql/
  复制代码

  或

  • 更新APT源

  wget https://dev.mysql.com/get/mysql-apt-config_0.8.15-1_all.de
  pkg -i mysql-apt-config_0.8.15-1_all.deb
  
  apt search mysql-server
  Sorting... Done
  Full Text Search... Done
  auth2db/xenial 0.2.5-2+dfsg-5ubuntu1 all
    Powerful and eye-candy IDS logger, log viewer and alert generator
  
  mysql-server/unknown,now 8.0.20-1ubuntu16.04 amd64 [installed]
    MySQL Server meta package depending on latest version
    
    
  apt install mysql-server
  复制代码

• 配置 mysql

create user 'ranger'@'%' identified by 'rangeradmin';
create database ranger;
grant all privileges on ranger.* to 'ranger'@'%';
set global log_bin_trust_function_creators=TRUE;
SET GLOBAL innodb_lock_wait_timeout=500;
复制代码