TA???????????????????

Chengsoul

???????????????????????????????????????????????????????????????????????????????????????????????

?????????????Hadoop?Presto?Kudu?Kafka????????????????????????????��Thinking Analytics???�TA??�?

TA????????????????????????????????????????????????????????????????

???????????????????TA????????????????TA???????????????????????????????????

??TA?????????

????????? ?????????????????????TA??????????????????????????????????????????

?????????????????????????????????????????????????????????????

TA???????????

???????HA???????????????????????????????????????????????????????????????????????????????

TA?????????????????????????????TA????????????????????��????????????????????????HA??????????????????TA?????????????

??TA????????????

?????????????????????????????????????????????

????????????????????????????????????????????????????????????

??????????????????TA??????????????????????

• ?????????????????????????
• ????????????????TA???????????????
• ?????????????TA???????????????
• ??????????TA?????????????????????
• ????????????TA??????????????????????
  

????????????????????????????????????????????TA???????????????????TA???????????????????????????????????????????

???100%??????TA????????

• ?????crash???????????
• ??????????????HA??
• ????+???????????
  

????????????????????????????????????TA??????????????????????????????????????????????????

????????TA?????��??????????????????????

??TA???????????

??TA????????????????????????????????????????????????????????????????????

??TA??????????Hadoop?Kudu??????Kafka???????????????????????????????????

// 3.1 MySQL??????

?????????MySQL????????????????MHA????MGR?

MHA?Master High Availability???????MySQL???????????????????MySQL??????????????????????????????????????????????????????????????MHA???????????????MHA??????????????????????????????????master??????????

MHA?????????????????crash??????????????????????????????

??MySQL???????????????TA????????????????????TA????????????????????????????MHA?????????????

MGR?MySQL Group Replication?MySQL????MySQL???????shared nothing????????paxos???????????????

MGR?MySQL???????GTID???MySQL??????????????????????????????????

?MGR????????????????????????????TA?????

MGR??????????

• ??????????????????
• ???????????????
• ????????????????? READ COMMITTED ?????????Gap Locks?????????????????????
• ???????????????????????????????
  

??TA????????????????????????

MySQL???MGR?MySQL??????????????????MySQL Router?????MySQL InnoDB Cluster????MySQL Router?????failover?????????MGR??????????????MySQL InnoDB Cluster???

??MySQL InnoDB Cluster?MySQL????????MySQL NDB Cluster?????????????????????????

?????????MySQL 5.6???MySQL InnoDB Cluster??5.7??????????MySQL 5.7???InnoDB Cluster??????????????MySQL????????????????????????????????????????????????????

????????MySQL?????????????????????????????MySQL???????????????????????????recover????????????MySQL???????????????relay log?????????????slave thread????relay log?????????????slave-parallel-type=LOGICAL_CLOCK?????????????????????relay log????

// 3.2 Presto??????

?????Presto?PrestoSQL???2020???PrestoSQL?????????????Trino??Presto????????????????????????????????Presto???PrestoSQL/Trino?

Presto??????????????????Presto?????????

Presto???????Coordinator?Discovery Server?Worker?Coordinator??query???????Discovery Server??????????Worker???????

Worker??????Discovery Server?????Coordinator?Discovery Server?????????Worker???

???SQL?????client???SQL????????http POST???????????Coordinator???????????????????????????????????Worker??????

Presto?????????????Coordinator?Discovery Server?Worker?????????????????????????Coordinator?Discovery Server???????????

Worker?????????????Worker???????????Coordinator?????

Presto??????????Coordinator/Discovery Server??????

Presto????????????????HAProxy+Keepalived?????????lb??????????????????VRRP????IP??????????????????????????????????????????????????????

?????????Presto????????

Presto?discovery????airlift discovery????????????????HA????Presto?????????????

??Coordinator??????????????dispatcher???????

Coordinator?????????shared queue???????????????Presto????????????????????????????????????System Pool ?Reserved Pool?General Pool?

System Pool ????????????Reserved Pool?General Pool ?????query????????????????General Pool????????query???????Reserved Pool????query???????

???????????????2?Coordinator???????Presto????????????????????????

??????????????????Reserved Pool??????????Presto??????Reserved Pool???????????????query??????????????????????Reserved Pool?????OOM???Presto?????????????????????

????TA??????????Reserved Pool?????????Presto????????2?Coordinator?

????????presto-jdbc???????presto failover??????????Presto Driver?????????TaFailoverPrestoDriver????????????????????????Presto?????????????

presto-jdbc???PrestoConnection???presto-cli?http????????????

??????????????java???????????TaFailoverPrestoConnection???????????????PrestoConnection??PrestoConnection??????????failover?

?????????????????current PrestoConnection????????????????????????????????????

????????????????TaFailoverPrestoConnection?PrestoConnection???TaPrestoConnection????????????????presto?????presto failover??????

?????Presto?SystemConnector?????Coordinator????????Unknown transaction ID????????????????Coordinator???????????????????? ? https://github.com/ prestosql/presto/pull/6328?

// 3.3 Redis??????

Redis????Redis Sentinel????????????????TA???Redis???????????????????????????

Redis???????????????????????????????????????????????????????Redis???????????????????????????????????????

????????Redis??????????????????????????????

???Redis????????????????????????????Redis????????????????zookeeper??????????????

?????Redis???????????????????????????????????????????????Redis??????????Redis???????????????????????Redis?TA???????

zookeeper?????????????????+???????????????????????????????????????????????????????????????????????

// 3.4 ??????????

????????????????????????????????????????????????????????????????????????????????????????????�??�???????�??�???????????????????

???????????????ganglia-gmond???????agent??????????????????????????TA?????????udp?????????ganglia-gmetad???????????rrd???????

????????admin-service???????????????????MySQL?????????????admin-service?????????????????????????????????admin-service??????????????????????????????????????

????????????????????gmetad/rrd??????????????????????????????????????gmetad/rrd?????????admin-service????????????????????????????????????????????????????????????admin-service???????

??????zookeeper????zookeeper????????admin-service???zookeeper???????admin-service???zookeeper??????????????????????????????????????????admin-service??????admin-service?????????????????????????????admin-service?????TA?????2?????�????�??????????????????admin-service????????????????????????????????????

// 3.5 ?????????ETL????????

????????????????????Kafka-common????ETL??data-etl??????Kafka????Kudu?HDFS?

??????data-etl??????TA????????????????ETL???????????????????????????????????????????????????????

????????????????????�????�?????????????????Presto?????Kudu?HDFS??????????????HDFS????????Kudu?????????????????????????????????????????????�????�?????????????????????HDFS????????????????????????????????Kudu?????

??????????????????????????????????HDFS?????????????

??data-etl???????Kafka-common???offset??MySQL/Redis??????????data-etl??????consumer group.id???Kafka?Kafka??partition???????data-etl????????????Kafka???????/??????????????data-etl??partition?

Kafka???????????????????????????Kafka consumer??Kafka??????????????Kafka???auto.offset.reset??????????earliest?latest?none?earliest??????????????????latest???????????????????none??????????????????????none????????Kafka??????????????????????????????

???????????Kafka Assignor??????Kafka???????????lag????????????log size?????????offset?????????????????data-etl???????partition?????????????

??????????????????????????????data-etl????Kafka???

// 3.6 ??????????????????

TA??????????receiver????????????????receiver??crash???????????????????receiver?????http?????????http???Kafka?????

???TA???????receiver????????load balance??load balance???????????????????receiver???????????????

?????load balance??????????????????lvs+nginx??????????lb???????F5????????

receiver????Kafka??????Kafka?????????/??????????receiver??????????????????Kafka???????????????Kafka?

// 3.7 web/common-service?????????

web??????????????????????????????????????????????????????????????????????????????LB?

web?????????????????????web???????????Ribbon????????????????????????????

common-service???????????web???????????????????????????????????????????????????common-service????????????????

???????????????????????HA?????????TA???????????TA?????????????????????????????????????????????????????????

???????????????????????????????????????????????????????TA????????????????????????

?/????????
???????
???https://mp.weixin.qq.com/s/UM6PjFw-1KiHdRqcGdkTjw