Introduce
ClickHouse adapter/proxy for Kibana
ClickHouse visualization tool
CKibana is a service for convenient analysis of ClickHouse data using native Kibana.
A typical use case is when nginx log storage is migrated from ElasticSearch to ClickHouse,users can seamlessly continue their log query and analysis with Kibana.
Featuresβ
- Version Support: Compatible with ElasticSearch and Kibana versions 6.x, 7.x, and 8.x.
- Syntax Support: Compatible with common ElasticSearch syntax (note: ip_range and date_range can only be queried using the querystring syntax in the search box).
- Sampling Function: For queries with hit results exceeding the threshold, it supports dynamic calculation of sampling and result restoration to improve query performance and ensure that the basic trend of the chart is consistent with the real data trend (the larger the flow control threshold, the closer the real chart trend is).
- Cache Function: Supports using ElasticSearch to cache results to improve the performance of repeated queries.
- Time Round Function: Supports rounding query time, for example, round 20 seconds (example: if the query time is in seconds 0-19, it will automatically round to 0; if it is in seconds 20-39, it will automatically round to 20 seconds, for example, if the query time is from 01:50:15 to 05:52:47, it will automatically adjust to 01:50:00 to 05:52:40). Used in conjunction with the cache function, it can effectively alleviate the pressure on ClickHouse caused by concurrent queries of the same statement by multiple users.
- Blacklist Function: Supports setting a blacklist to prevent complex statements from causing pressure on ClickHouse.
- Query Template Function: Supports syntax monitoring after removing time to facilitate quick problem location. Used in conjunction with the blacklist function, it can effectively ensure the stability of ClickHouse.
- Query Circuit Breaker: Supports advanced features such as configuring the maximum query time range.
Supported Aggregation Syntax:β
Get startedβ
quick startβ
To quickly experience Kibana, you can deploy it quickly using docker-compose.
# Enter the docker-compose directory.
cd ckibana/docker-compose
# Deploy
docker-compose up -d
After deployment is complete, access Kibana in a browser atοΌ http://127.0.0.1:5601/
A process with built-in mock data has been included. After importing the Kibana configuration file (dashboard, index-pattern, etc.), you can start experiencing it!
Configuration file pathοΌdocker-compose/quickstart-export.json Effect displayοΌ
Local run CKibanaβ
CKibana can be run on all major operating systems and requires Java JDK version 17 or higher to be installed. To check, please run:java -version
:
$ java -version
java version "17.0.5"
The proxy service depends on the ES, CK, and Kibana services, which need to be prepared in advance. The following steps only include querying data from CK and do not include writing data to CK.
1οΌCreate a database and a table
To initialize database and table structures in ClickHouse, please refer to:api-docs Detailed explanation of creating a table
2οΌStart CKibana
To modify the ES configuration in CKibana, you need to change it to your own ES information. The configuration file path is src/main/resources/application.yml
metadata-config:
hosts: your es metadata cluster hosts
headers:
headerKey: yourHeaderValue
Package and run or run locally directly com.ly.ckibana.Bootstrap
### start ckibana
$ nohup java -jar ckibana.jar > run.out 2>&1 &
### check whether ckibana is successfully started
$ tail -f ~/logs/app.log
Tomcat started on port(s): 8080 (http) with context path ''
Started Bootstrap in 1.474 seconds
3οΌstart kibana
Configure the elasticsearchHosts parameter in Kibana to the CKibana proxy address, so that it can be routed to the proxy service.
egοΌelasticsearchHosts=http://ip:port
4οΌConfigure the index pattern whitelist
Only after configuring the corresponding whitelist can you proceed to create it in the next step index pattern
οΌThe configured whitelist needs to be consistent with the table nameοΌplease refer to:api-docs Update the whitelist index interface
5οΌCreate index pattern
in Kibana
egοΌIf the CK table name is `table_test`, then create an index pattern `table_test`
After completing the above 5 steps, you can create a dashboard and start using it.
TODOβ
- Segmented query: Supports automatically splitting query time, which can fully utilize the cache and improve query performance.
- IDS query: Currently v8 version of several places do not support data Json display and Doc View, for example: Json, Single document, Surrounding documents and so on.
Licenseβ
Apache License, Version 2.0 Copyright (C) Apache Software Foundation