大數據運維圖解教程

目 錄

第1 章 大數據運維概述

1.1 從大數據說起 ····················································································· 1

1.1.1 大數據產生、影響及挑戰 ······································································ 1

1.1.2 大數據概念、特徵及價值 ····································································· 14

1.1.3 大數據技術、產業及應用 ····································································· 18

1.2 大數據技術生態 ················································································· 24

1.2.1 分佈式集群概述 ················································································· 24

1.2.2 Hadoop 生態系統 ················································································ 30

1.3 大數據運維 ······················································································· 37

第2 章 集群基礎環境搭建與運維

2.1 集群規劃 ·························································································· 41

2.2 虛擬機與Linux ·················································································· 44

2.3 網絡配置與時鐘同步 ··········································································· 55

2.4 免密 ································································································ 60

2.5 JDK 和MySQL 部署 ··········································································· 62

第3 章 分佈式集群Hadoop 運維

3.1 Hadoop 平臺安裝與部署 ····································································· 69

3.1.1 Hadoop 安裝準備 ················································································ 69

3.1.2 Hadoop 安裝與部署 ············································································· 71

3.2 分佈式存儲組件HDFS ········································································ 75

3.2.1 HDFS 結構 ······················································································· 75

3.2.2 HDFS 讀寫原理 ················································································· 83

3.2.3 HDFS 操作與監控 ·············································································· 86

3.3 離線分佈式計算引擎MapReduce ·························································· 90

3.3.1 MapReduce 結構與原理 ········································································ 90

3.3.2 MapReduce 部署與優化 ······································································ 100

3.4 集群資源管理YARN ········································································· 101

3.4.1 YARN 的結構與原理 ········································································· 101

3.4.2 YARN 部署、優化與監控 ··································································· 105

第4 章 分佈式系統協調器Zookeeper

4.1 Zookeeper 結構 ··············································································· 108

IV 大數據運維圖解教程

4.2 Zookeeper 原理 ··············································································· 111

4.3 Zookeeper 部署與優化 ······································································ 117

第5 章 數據採集組件運維

5.1 日誌採集組件Flume ········································································· 121

5.2 數據遷移組件Sqoop ········································································ 128

5.3 發布訂閱消息組件Kafka ··································································· 132

第6 章 數據處理組件運維

6.1 NoSQL 數據庫HBase ······································································ 137

6.1.1 NoSQL ·························································································· 137

6.1.2 HBase 架構與原理 ············································································ 141

6.1.3 HBase 部署與優化 ············································································ 148

6.2 數據倉庫引擎Hive ··········································································· 153

6.2.1 Hive 架構與原理 ·············································································· 154

6.2.2 Hive 部署與優化 ·············································································· 166

6.3 內存計算引擎Spark ········································································· 169

6.3.1 Spark 生態 ······················································································ 169

6.3.2 Spark 架構與原理 ············································································· 173

6.3.3 Spark 部署與優化 ············································································· 179

第7 章 大數據平臺運維工具

7.1 Ambari 大數據環境搭建利器 ······························································ 187

7.1.1 安裝 ······························································································ 187

7.1.2 使用 ······························································································ 192

7.2 Hive SQL 監控系統—Hive Falcon ···················································· 198

7.2.1 安裝 ······························································································ 198

7.2.2 使用 ······························································································ 199

7.3 統一日誌監控系統EKL ····································································· 202

7.3.1 安裝 ······························································································ 202

7.3.2 使用 ······························································································ 207

參考文獻

附錄A Web 監控埠

附錄B 大數據平臺運維任務

附錄C 大數據運維三十六計