ambari源码分析 -----ambari-server启动流程

news/2024/7/5 8:16:58

一、启动脚本分析

 

1、ambari的启动脚本为:service ambari-server start 或者 ambari-server start。分别对应脚本文件/etc/init.d/ambari-server 和 /usr/sbin/ambari-server,其中/usr/sbin/ambari-server文件是一个快捷方式,指向/etc/init.d/ambari-server。脚本文件/etc/init.d/ambari-server。

对应工程源码ambari-server/sbin/ambari-server,启动关键代码如下:

PYTHON_WRAP="$ROOT/usr/bin/ambari-python-wrap"
AMBARI_ENV="$ROOT/var/lib/ambari-server/ambari-env.sh"
AMBARI_PYTHON_EXECUTABLE="$ROOT/usr/sbin/ambari-server.py"
AMBARI_EXECUTABLE="$ROOT/usr/sbin/ambari-server"

case "${1:-}" in
  start)
        echo -e "Starting ambari-server"
        $PYTHON "$AMBARI_PYTHON_EXECUTABLE" "$@"
        ;;
  stop)
        echo -e "Stopping ambari-server"
        $PYTHON "$AMBARI_PYTHON_EXECUTABLE" "$@"
        ;;
  reset)
        echo -e "Resetting ambari-server"
        $PYTHON "$AMBARI_PYTHON_EXECUTABLE" "$@"
        ;;
  restart)
        echo -e "Restarting ambari-server"
        $PYTHON "$AMBARI_PYTHON_EXECUTABLE" "$@"
        ;;
  upgrade)
        echo -e "Upgrading ambari-server"
        $PYTHON "$AMBARI_PYTHON_EXECUTABLE" "$@"
        ;;
  status)
        echo -e "Ambari-server status"
        $PYTHON "$AMBARI_PYTHON_EXECUTABLE" "$@"
        ;;
  upgradestack)
        echo -e "Upgrading stack of ambari-server"
        $PYTHON "$AMBARI_PYTHON_EXECUTABLE" "$@"
        ;;
  setup)
        echo -e "Setup ambari-server"
        $PYTHON "$AMBARI_PYTHON_EXECUTABLE" "$@"
        ;;
  setup-jce)
        echo -e "Updating jce policy"
        $PYTHON "$AMBARI_PYTHON_EXECUTABLE" "$@"
        ;;
  setup-pam)
        echo -e "Setting up PAM properties..."
        $PYTHON "$AMBARI_PYTHON_EXECUTABLE" "$@"
        ;;
  migrate-ldap-pam)
        echo -e "Migration LDAP to PAM"
        $PYTHON "$AMBARI_PYTHON_EXECUTABLE" "$@"
        ;;
  setup-ldap)
        $PYTHON "$AMBARI_PYTHON_EXECUTABLE" "$@"
        ;;
  sync-ldap)
        echo -e "Syncing with LDAP..."
        $PYTHON "$AMBARI_PYTHON_EXECUTABLE" "$@"
        ;;
  set-current)
        echo -e "Setting current version..."
        $PYTHON "$AMBARI_PYTHON_EXECUTABLE" "$@"
        ;;
  setup-security)
        echo -e "Security setup options..."
        $PYTHON "$AMBARI_PYTHON_EXECUTABLE" "$@"
        ;;
  refresh-stack-hash)
        echo -e "Refreshing stack hashes..."
        $PYTHON "$AMBARI_PYTHON_EXECUTABLE" "$@"
        ;;
  backup)
        echo -e "Backing up Ambari File System state... *this will not backup the server database*"
        $PYTHON "$AMBARI_PYTHON_EXECUTABLE" "$@"
        ;;
  restore)
        echo -e "Restoring Ambari File System state"
        $PYTHON "$AMBARI_PYTHON_EXECUTABLE" "$@"
        ;;
  update-host-names)
        echo -e "Updating host names"
        $PYTHON "$AMBARI_PYTHON_EXECUTABLE" "$@"
        ;;
  check-database)
        echo -e "Checking database"
        $PYTHON "$AMBARI_PYTHON_EXECUTABLE" "$@"
        ;;
  enable-stack)
        echo -e "Enabling stack(s)..."
        $PYTHON "$AMBARI_PYTHON_EXECUTABLE" "$@"
        ;;
  setup-sso)
        echo -e "Setting up SSO authentication properties..."
        $PYTHON "$AMBARI_PYTHON_EXECUTABLE" "$@"
        ;;
  db-purge-history)
        echo -e "Purge database history..."
        $PYTHON "$AMBARI_PYTHON_EXECUTABLE" "$@"
        ;;
  install-mpack)
        echo -e "Installing management pack"
        $PYTHON "$AMBARI_PYTHON_EXECUTABLE" "$@"
        ;;
  uninstall-mpack)
        echo -e "Uninstalling management pack"
        $PYTHON "$AMBARI_PYTHON_EXECUTABLE" "$@"
        ;;
  upgrade-mpack)
        echo -e "Upgrading management pack"
        $PYTHON "$AMBARI_PYTHON_EXECUTABLE" "$@"
        ;;
  setup-kerberos)
        echo -e "Setting up Kerberos authentication"
        $PYTHON "$AMBARI_PYTHON_EXECUTABLE" "$@"
        ;;
  *)
        echo "Usage: $AMBARI_EXECUTABLE
        {start|stop|reset|restart|upgrade|status|upgradestack|setup|setup-jce|setup-ldap|sync-ldap|set-current|setup-security|refresh-stack-hash|backup|restore|update-host-names|check-database|enable-stack|setup-sso|db-purge-history|install-mpack|uninstall-mpack|upgrade-mpack|setup-kerberos|setup-pam|migrate-ldap-pam} [options]
        Use $AMBARI_PYTHON_EXECUTABLE <action> --help to get details on options available.
        Or, simply invoke ambari-server.py --help to print the options."
        exit 1
esac

2. 从上图中可以看出,执行/etc/init.d/ambari-server start 命令,脚本会调用python执行/usr/sbin/ambari-server.py 脚本文件,对应源码ambari-server/src/main/python/ambari-server.py,ambari-server.py 脚本解析传入的命令行参数,然后执行 main(options, args, parser) 方法,核心代码如下:

@OsFamilyFuncImpl(OsFamilyImpl.DEFAULT)
def create_user_action_map(args, options):
  action_map = {
        SETUP_ACTION: UserAction(setup, options),
        START_ACTION: UserAction(start, options),
        STOP_ACTION: UserAction(stop, options),
  }
  return action_map
 
def main(options, args, parser):
  action_map = create_user_action_map(args, options)
  action = args[0]
 
  try:
    action_obj = action_map[action]
  except KeyError:
    parser.error("Invalid action: " + action)
 
 action_obj.execute() 


此时args值是['start'],options值为空,START_ACTION是一个常量,其值在setupActions.py文件中定义:START_ACTION = "start".调用action_obj.execute()方法实际是调用了 UserAction(start, options) 对象的方法:

class UserActionPossibleArgs(object):
  def __init__(self, i_fn, i_possible_args_numbers, *args, **kwargs):
    self.fn = i_fn
    self.possible_args_numbers = i_possible_args_numbers
    self.args = args
    self.kwargs = kwargs
    self.need_restart = False
 
  def execute(self):
    self.fn(*self.args, **self.kwargs)
 
class UserAction(UserActionPossibleArgs):
  def __init__(self, i_fn, *args, **kwargs):
    super(UserAction, self).__init__(i_fn, [1], *args, **kwargs)


从上面代码中可以看出,execute()方法去调用fn()方法,但是fn变量是由外部传入的,传入值为start,因此会去调用start()方法,start()方法代码如下:

@OsFamilyFuncImpl(OsFamilyImpl.DEFAULT)
def start(args):
  status, pid = is_server_runing()
  if status:
    err = "Ambari Server is already running."
    raise FatalException(1, err)
 
  server_process_main(args)


3. server_proecess_main()方法在/usr/sbin/ambari-server-main.py文件中定义,对应源码ambari-server/src/main/python/ambari-server-main.py。server_process_main()方法源码如下:

CHECK_DATABASE_HELPER_CMD = "{0} -cp {1} org.apache.ambari.server.checks.DatabaseConsistencyChecker"
 
def server_process_main(options, scmStatus=None):
   jdk_path = find_jdk()
  properties = get_ambari_properties() 
  ensure_jdbc_driver_is_installed(options, properties)
  ensure_dbms_is_running(options, properties, scmStatus)
  refresh_stack_hash(properties)
  java_exe = get_java_exe_path()
  environ = generate_env(options, ambari_user, current_user)
  class_path=serverClassPath.get_full_ambari_classpath_escaped_for_shell(
    validate_classpath=True)
 
  command = CHECK_DATABASE_HELPER_CMD.format(java_exe, class_path)
   (retcode, stdout, stderr) = run_os_command(command, env=environ)
 
   update_properties(properties)
   param_list = generate_child_process_param_list(ambari_user, java_exe, 
     class_path, debug_start, suspend_mode,options)
 
   print_info_msg("Running server: " + str(param_list))
   procJava = subprocess.Popen(param_list, env=environ, 
        preexec_fn=make_process_independent)
 
  pidfile = os.path.join(configDefaults.PID_DIR, PID_NAME)
   save_pid(pidJava, pidfile) 
  wait_for_server_start(pidfile, scmStatus)
  return procJava


get_ambari_properties()方法去加载ambari.properties文件,默认位于/etc/ambari-server/conf目录下,将其转化为 Properties 对象.

4. refresh_stack_hash()方法是去刷新stack目录下文件夹hash值:在 /var/lib/ambari-server/resources/stacks/xxx/xxx/services 目录下,集成了关于各个组件的元信息,控制脚本及配置文件,以HDFS服务为例,其文件夹内容如下:

该目录下的package文件夹存放了组件启停控制脚本文件,在agent端执行组件启停命令时,也是先将该文件夹拷贝到自己节点的cache目录下,然后执行对应脚本,package文件夹内容如下:


该文件夹有两个特别的文件archive.zip和.hash,archive.zip文件是对alerts,files,scripts,templates4个文件夹压缩而产生的压缩包,.hash文件存放archive.zip文件的hash值.agent端拷贝package文件夹时,只拷贝了archive,zip文件和.hash文件.当修改了package文件夹下的文件内容时,ambari-server进程重启会重新生成archive.zip文件和.hash文件,而agent端在执行命令时,会先去比对两个.hash文件内容是否一致,如果不一致,则重新拷贝这两个文件到cache目录下.

5. 接下来的代码是去执行数据库的数据一致性检测:

command = CHECK_DATABASE_HELPER_CMD.format(java_exe, class_path)
(retcode, stdout, stderr) = run_os_command(command, env=environ)
它调用了org.apache.ambari.server.checks.DatabaseConsistencyChecker类的main()方法进行数据库一致性检测,防止数据错乱的问题.

6. 接着方法调用了generate_child_param_list()方法生成命令参数列表,该方法主要逻辑如下:

SERVER_START_CMD = "{0} " \
    "-server -XX:NewRatio=3 " \
    "-XX:+UseConcMarkSweepGC " + \
    "-XX:-UseGCOverheadLimit -XX:CMSInitiatingOccupancyFraction=60 " \
    "-XX:+CMSClassUnloadingEnabled " \
    "-Dsun.zip.disableMemoryMapping=true " + \
    "{1} {2} " \
    "-cp {3} "\
    "org.apache.ambari.server.controller.AmbariServer {4}" \
    "> {5} 2>&1 || echo $? > {6}"
 
@OsFamilyFuncImpl(OsFamilyImpl.DEFAULT)
def generate_child_process_param_list(ambari_user, java_exe, class_path,
                                      debug_start, suspend_mode,options=None):
  from ambari_commons.os_linux import ULIMIT_CMD
 
  properties = get_ambari_properties()
 
  command_base = SERVER_START_CMD_DEBUG if debug_start else SERVER_START_CMD
 
  command = command_base.format(java_exe,
          ambari_provider_module_option,
          jvm_args,
          class_path,
          startMode,
          configDefaults.SERVER_OUT_FILE,
          os.path.join(configDefaults.PID_DIR, EXITCODE_NAME),
          suspend_mode)
  param_list.append(cmd)
  return param_list


generate_child_param_list()方法里最主要的是格式化SERVER_START_CMD命令来生成启动子进程的param_list,从该命令中得出Java程序入口类是org.apache.ambari.server.controller.AmbariServer。

接下来便是启动java子进程的代码:

print_info_msg("Running server: " + str(param_list))
procJava = subprocess.Popen(param_list, env=environ, 
        preexec_fn=make_process_independent)
 
pidfile = os.path.join(configDefaults.PID_DIR, PID_NAME)
save_pid(pidJava, pidfile) 
wait_for_server_start(pidfile, scmStatus)
return procJava


方法调用python的subprocess模块启动子进程,然后保存pid到文件中,等待进程成功启动后返回,至此整个启动流程结束。

二、java启动流程


http://lihuaxi.xjx100.cn/news/1010980.html

相关文章

Ubuntu常用环境配置

配置软件源 切换清华源 sudo sed -i "shttp://.*archive.ubuntu.comhttps://mirrors.tuna.tsinghua.edu.cng" /etc/apt/sources.list sudo sed -i "shttp://.*security.ubuntu.comhttps://mirrors.tuna.tsinghua.edu.cng" /etc/apt/sources.list sudo ap…

Scrapy-爬虫多开技能

我们知道&#xff0c;现在运行Scrapy项目中的爬虫文件&#xff0c;需要一个一个地运行&#xff0c;那么是否可以将对应的爬虫文件批量运行呢&#xff1f;如果可以&#xff0c;又该怎么实现呢&#xff1f;在Scrapy中&#xff0c;如果想批量运行爬虫文件&#xff0c;常见的有两种…

IPv4 和 IPv6 的组成结构和对比

IPv4 和 IPv6 的组成结构和对比IPv4IPv6互联网协议 (IP) 是互联网通信的基础&#xff0c;IP 地址是互联网上每个设备的唯一标识符。目前最常用的 IP 协议是 IPv4&#xff0c;它已经有近 30 年的历史了。然而&#xff0c;IPv4 存在一些问题&#xff0c;例如: 地址空间不足:IPv4 …

向量的内积外积哈达玛积

1.向量的内积 1.1 定义 从代数角度看&#xff0c;先对两个数字序列中的每组对应元素求积&#xff0c;再对所有积求和&#xff0c;结果即为点积。从几何角度看&#xff0c;点积则是两个向量的长度与它们夹角余弦的积。 表示形式&#xff1a;ATBA^TBATB、<A,B><A,B&g…

[学习笔记]金融风控实战

参考资料&#xff1a; 零基础入门金融风控-贷款违约预测 导包 import pandas as pd import matplotlib.pyplot as plt# 读取数据 train pd.read_csv(train.csv) testA pd.read_csv(testA.csv) print(Train data shape:, train.shape) print(testA data shape:, testA.shape…

我的面试八股(JAVA并发)

程序计数器为什么是线程私有的? 程序计数器主要有下面两个作用&#xff1a; 字节码解释器通过改变程序计数器来依次读取指令&#xff0c;从而实现代码的流程控制&#xff0c;如&#xff1a;顺序执行、选择、循环、异常处理。在多线程的情况下&#xff0c;程序计数器用于记录…

Redis 客户端连接服务器失败

公司项目开发环境需要使用到 Redis&#xff0c;申请基础技术支撑平台的 Redis 中间件比较麻烦&#xff0c;项目组也不知道具体流程&#xff0c;而且时间可能比较长。 现在的情况是&#xff0c;项目因为 Redis 启动报错。 这种情况下&#xff0c;我们项目组就自行在虚拟机上临…

开心档之C++ 多态

目录 C 多态 实例 虚函数 纯虚函数 多态按字面的意思就是多种形态。当类之间存在层次结构&#xff0c;并且类之间是通过继承关联时&#xff0c;就会用到多态。 C 多态意味着调用成员函数时&#xff0c;会根据调用函数的对象的类型来执行不同的函数。 下面的实例中&#x…