發表於 程式分享

三大GitOps Solution

三大GitOps Solution,分別說明如下

一、Flux (https://fluxcd.io)
1.依k8s design, auditable而設計
2.out of the box Integrations、Extensible,故可支援原生yaml檔以外的設定格式
3.請使用v2版本,v1已不支援使用
4.沒有GUI設定畫面,為CLI based (Command Line操作介面)
5.特點: container image有變動,也可通知k8s做變動,並寫一筆至repository並commit

二、Rancher Fleet (https://rancher.com)
1.依GitOps及大規模cluster而設計
2.Rancher UI,繼承了Rancher的好處,有SSO (LDAP)及授權(RBAC)的控管
3.Multi-Cluster: 以Fleet Controller cluster 內建可管理多套cluster

三、Argo (https://argoproj.github.io/argo-cd/)
1.知名度高
2.可管理多套cluster,但不是內建,故需取得kubernetes cluster的控制權
3.SSO: 支援OIDC,OAuth2,LDAP,SAML2.0,GitHub,GitLab,MS,LinkedIn
4.授權: 支援多組權限設定、RBAC政策
5.Web UI是三套架構內最好的
6.WebHook push整合佳,如GitHub、BitBucket、GitLab可主動呼叫ArgoCD有異動,
7.可自動、手動同步應用程式至其想達到的狀態
8.自動設定檔漂移偵測(drift detection)且可視覺化查看(Visualization)
9.有三大controller: API、Repository Service、Application Controller

發表於 程式分享

在ubuntu下安裝Spark

1.下載檔案
https://downloads.apache.org/spark/spark-3.1.2/spark-3.1.2-bin-hadoop3.2.tgz

2.上傳至ubuntu的/usr/local/路徑下
3.解壓縮
tar xvf spark-3.1.2-bin-hadoop3.2.tgz
mv spark-3.1.2-bin-hadoop3.2 spark

4.新增檔案/usr/local/spark/djt.log

hadoop hadoop hadoop
spark spark spark

5.執行spark並進行字頻統計
/usr/local/spark/bin/spark-shell

val line = sc.textFile("/usr/local/spark/djt.log")
line.flatMap(_.split(" ")).map((_,1)).reduceByKey(_+_).collect().foreach(println)

6.standalone安裝
1) spark-env.sh

cd /usr/local/spark/conf
cp spark-env.sh.template spark-env.sh
mkdir /usr/local/spark/my-data

vi spark-env.sh

export JAVA_HOME=/usr/local/jdk
export HADOOP_CONF_DIR=/usr/local/hadoop/etc/hadoop
export HADOOP_HOME=/usr/local/hadoop
SPARK_Master_WEBUI_PORT=8888
SPARK_DAEMON_JAVA_OPTS="-Dspark.deploy.recoveryMode=ZOOKEEPER -Dspark.deploy.zookeeper.url=ubuntu-VirtualBox:2181 -Dspark.deploy.zookeeper.dir=/usr/local/spark/my-data"

2) slaves
cd /usr/local/spark/conf
vi slaves

ubuntu-VirtualBox

3) 啟動spark
因spark cluster需依賴zookeeper cluster,請先啟動zookeeper
/usr/local/zookeeper/bin/zkServer.sh start (需先啟動zookeeper)
/usr/local/spark/sbin/start-all.sh

jps
有以下process

Worker
Jps
QuorumPeerMain
Master

4) 查看瀏覽器
http://ubuntu-virtualbox:8888/

發表於 程式分享

在ubuntu下安裝Hive

需先安裝mysql
1.安裝mysql

apt install mysql-server
systemctl status mysql

2.改mysql root密碼 (預設無密碼)
1) 先查看mysql版本

mysql -V

2) mysql 8以前
mysql -u root -p

use mysql;

UPDATE user SET Password=PASSWORD("8888") WHERE User='root';
或
UPDATE user SET authentication_string=password('8888') WHERE User='root';

FLUSH PRIVILEGES;

另先開放權限讓root在任何主機都可執行(這個不安全)

CREATE USER 'root'@'%' IDENTIFIED BY '8888';
GRANT ALL PRIVILEGES ON *.* TO 'root'@'%' WITH GRANT OPTION;
FLUSH PRIVILEGES;

3) mysql 8以後
mysql -u root -p

use mysql;
ALTER USER 'root'@'localhost' IDENTIFIED WITH mysql_native_password BY '8888';
flush privileges;

3.建mysql / hive帳號
mysql -u root -p

CREATE user 'hive' IDENTIFIED BY '8888';
GRANT ALL PRIVILEGES ON *.* to 'hive'@'%' WITH GRANT OPTION;
FLUSH PRIVILEGES;

4.下載hive
https://ftp.tsukuba.wide.ad.jp/software/apache/hive/hive-3.1.2/apache-hive-3.1.2-bin.tar.gz

5.上傳至ubuntu的/usr/local/路徑下
6.解壓縮

cd /usr/local
tar xvf apache-hive-3.1.2-bin.tar.gz
mv apache-hive-3.1.2-bin hive

7.新增/usr/local/hive/conf/hive-site.xml路徑下設定檔
cd /usr/local/hive/conf/
cp hive-default.xml.template hive-site.xml

vi hive-site.xml

<property>
<name>javax.jdo.option.ConnectionDriverName</name>
<value>com.mysql.jdbc.Driver</value>
<description>Driver class name for a JDBC metastore</description>
</property>
...
<property>
<name>javax.jdo.option.ConnectionURL</name>
<value>jdbc:mysql://ubuntu-VirtualBox:3306/hive?createDatabaseIfNotExit=true</value>
<description>
JDBC connect string for a JDBC metastore.
</description>
</property>
...
<property>
<name>javax.jdo.option.ConnectionUserName</name>
<value>hive</value>
<description>Username to use against metastore database</description>
</property>
...
<property>
<name>javax.jdo.option.ConnectionPassword</name>
<value>8888</value>
<description>password to use against metastore database</description>
</property>

8.配置環境變數
vi /etc/profile

export HIVE_HOME=/usr/local/hive
export PATH=$PATH:$JAVA_HOME/bin:$HADOOP_HOME/bin:$HADOOP_HOME/sbin:$HBASE_HOME/bin;$HIVE_HOME/bin

source /etc/profile

9.下載mysql driver複製mysql driver至/usr/local/hive/lib
https://downloads.mysql.com/archives/get/p/3/file/mysql-connector-java-8.0.25.tar.gz

10.修改hive數據目錄
cd /usr/local/hive/conf/
vi hive-site.xml

<property>
<name>hive.querylog.location</name>
<value>/usr/local/hive/iotmp</value>
<description>Location of Hive run time structured log file</description>
</property>
...
<property>
<name>hive.exec.local.scratchdir</name>
<value>/usr/local/hive/iotmp</value>
<description>Local scratch space for Hive jobs</description>
</property>
<property>
<name>hive.downloaded.resources.dir</name>
<value>/usr/local/hive/iotmp</value>
<description>Temporary local directory for added resources in the remote file system.</description>
</property>

11.啟動hive
/usr/local/hive/bin/hive
註: 須先啟動hadoop及mysql才能啟動hive

發表於 程式分享

在ubuntu下裝Flume日誌採集系統

1.下載
http://www.apache.org/dyn/closer.lua/flume/1.9.0/apache-flume-1.9.0-bin.tar.gz

2.上傳至ubuntu的/usr/local/路徑下
3.解壓縮
cd /usr/local
tar xvf apache-flume-1.9.0-bin.tar.gz
mv apache-flume-1.9.0-bin flume

4.修改/usr/local/flume/conf 路徑下設定檔
mv flume-conf.properties.template flume-conf.properties
vi flume-conf.properties

agent.sources = seqGenSrc
agent.channels = memoryChannel
agent.sinks = loggerSink
agent.sources.seqGenSrc.type = seq
agent.sources.seqGenSrc.channels = memoryChannel
agent.sinks.loggerSink.type = logger
agent.sinks.loggerSink.channel = memoryChannel
agent.channels.memoryChannel.type = memory
agent.channels.memoryChannel.capacity = 100

5.啟動Flume
/usr/local/flume/bin/flume-ng agent -n agent -c conf -f /usr/local/flume/conf/flume-conf.properties -Dflume.root.logger=INFO,console

發表於 程式分享

在ubuntu下裝kafka

1.下載
https://downloads.apache.org/kafka/2.8.0/kafka_2.13-2.8.0.tgz

2.上傳至ubuntu的/usr/local/路徑下
3.解壓縮
cd /usr/local
tar xvf kafka_2.13-2.8.0.tgz
mv kafka_2.13-2.8.0 kafka

4.修改/usr/local/kafka/config/路徑下設定檔
1) zookeeper.properties

dataDir=/usr/local/hadoop/data/zookeeper/zkdata
clientPort=2181

2) consumer.properties

zookeeper.connect=ubuntu-VirtualBox:2181

3) producer.properties

metadata.broker.list=ubuntu-VirtualBox:9092

4) server.properties

broker.id=1
zookeeper.connect=ubuntu-VirtualBox:2181

5.啟動kafka
/usr/local/kafka/bin/kafka-server-start.sh /usr/local/kafka/config/server.properties
註: 須先啟動zookeeper cluster才能啟動kafka cluster

發表於 程式分享

於ubuntu裝zookeeper (分布式協調服務)

1.下載zookeeper
https://www.apache.org/dyn/closer.lua/zookeeper/zookeeper-3.7.0/apache-zookeeper-3.7.0-bin.tar.gz

2.上傳至ubuntu的/usr/local/路徑下
3.解壓縮
cd /usr/local
tar xvf apache-zookeeper-3.7.0-bin.tar.gz
mv apache-zookeeper-3.7.0-bin zookeeper

4.建立路徑
/usr/local/hadoop/data/zookeeper/zkdata
/usr/local/hadoop/data/zookeeper/zkdatalog

新增檔案/usr/local/hadoop/data/zookeeper/zkdata/myid
內儲值1

5.新增/usr/local/zookeeper/conf/zoo.cfg

tickTime=2000
initLimit=10
syncLimit=5
dataDir=/usr/local/hadoop/data/zookeeper/zkdata
dataLogDir=/usr/local/hadoop/data/zookeeper/zkdatalog
clientPort=2181
server.1=ubuntu-VirtualBox:2888:3888

6.啟動zookeeper
/usr/local/zookeeper/bin/zkServer.sh start
/usr/local/zookeeper/bin/zkServer.sh status

jps
會有process

QuorumPeerMain

7.查看訊息
/usr/local/zookeeper/bin/zkCli.sh -server ubuntu-VirtualBox:2181

68.208.3:2181, session id = 0x100002e0cfa0000, negotiated timeout = 30000

WATCHER::

WatchedEvent state:SyncConnected type:None path:null
[zk: ubuntu-VirtualBox:2181(CONNECTED) 0] ls /
[zookeeper]
[zk: ubuntu-VirtualBox:2181(CONNECTED) 0] ls /
[zookeeper]
[zk: ubuntu-VirtualBox:2181(CONNECTED) 1] create /test helloworld
Created /test
[zk: ubuntu-VirtualBox:2181(CONNECTED) 2] get /test
helloworld
[zk: ubuntu-VirtualBox:2181(CONNECTED) 3] get /test
helloworld
[zk: ubuntu-VirtualBox:2181(CONNECTED) 4] set /test zookeeper
[zk: ubuntu-VirtualBox:2181(CONNECTED) 5] get /test
zookeeper
[zk: ubuntu-VirtualBox:2181(CONNECTED) 6] delete /test
[zk: ubuntu-VirtualBox:2181(CONNECTED) 7]

7.啟動hadoop
start-all.sh
jps
有以下process

DataNode
NodeManager
SecondaryNameNode
ResourceManager
NameNode

8.啟動hbase
start-hbase.sh
jps
有以下process

HMaster
HRegionServer

9.查看HBase Web介面
http://ubuntu-virtualbox:16010/master-status

註: 啟動順序 zookeeper -> hadoop (hdfs) -> hbase

發表於 程式分享

於ubuntu安裝hBase

1.下載檔案
https://www.apache.org/dyn/closer.lua/hbase/2.3.6/hbase-2.3.6-bin.tar.gz

2.上傳至ubuntu的/usr/local/路徑下
3.解壓縮
cd /usr/local
tar xvf hbase-2.3.6-bin.tar.gz
mv hbase-2.3.6 hbase

4.配置環境變數
vi /etc/profile

export HBASE_HOME=/usr/local/hbase
export PATH=$PATH:$JAVA_HOME/bin:$HADOOP_HOME/bin:$HADOOP_HOME/sbin:$HBASE_HOME/bin

source /etc/profile

5.調整設定檔
cd /usr/local/hbase/conf
1) vi hbase-env.sh

export JAVA_HOME=/usr/local/jdk
export HBASE_MANAGES_ZK=true

2) vi hbase-site.xml

</configuration>
<property>
<name>hbase.rootdir</name>
<value>hdfs://ubuntu-VirtualBox:9000/hbase</value>
</property>
<property>
<name>hbase.cluster.distributed</name>
<value>true</value>
</property>
<property>
<name>hbase.zookeeper.quorum</name>
<value>ubuntu-VirtualBox</value>
</property>
</configuration>

6.啟動hbase
start-hbase.sh