Skip to main content

api-docs

application.yaml​

This file contains some basic configurations such as ports, logging, etc. The core parameter is the metadata-config parameter.

eg：
metadata-config:
  hosts: your es metadata cluster hosts
  headers:
    Authorization: 'username password'

Detailed explanation of table creation​

Table creation instructions​

The time field is required in the table creation statement, such as the @timestamp field in the table, which is used as the time option when creating an index pattern in Kibana. Other fields should be added as needed based on the query conditions of the business itself.

The logic of matching time field is as follows, just satisfy one of them:

1）Fields of type Date, such as DateTime64, will be recognized as time fields.

2）Alternatively, if the field name contains "time", such as (@timestamp UInt64), it will be recognized as a time field.

Here is an example of a table creation statement.

create database ckibana;
use ckibana;

CREATE TABLE ckibana.table_local
(
    `request_method` LowCardinality(String),
    `request_time` Float64,
    `http_host` String,
    `@timestamp` UInt64,
    `ck_assembly_extension` String,
    INDEX timestamp_index `@timestamp` TYPE minmax GRANULARITY 8192
)
ENGINE = MergeTree
PARTITION BY (toYYYYMMDD(toDateTime(`@timestamp` / 1000, 'Asia/Shanghai')), toHour(toDateTime(`@timestamp` / 1000, 'Asia/Shanghai')))
ORDER BY (`http_host`)
SETTINGS in_memory_parts_enable_wal = 0, index_granularity = 8192;

CREATE TABLE IF NOT EXISTS ckibana.table_all AS ckibana.table_local ENGINE Distributed('clickhouse_cluster', ckibana, table_local, rand());

When using the sampling feature, you need to modify the ORDER BY and SAMPLE BY statements in the table creation statement. For more details, please refer to the following resources:clickhouse doc 。

Note that the sampling field can be a user-defined field, It is recommended to use the @timestamp field as the sampling field instead of creating an additional field.


create database ckibana;
use ckibana;

CREATE TABLE ckibana.table_local
(
    `request_method` LowCardinality(String),
    `request_time` Float64,
    `http_host` String,
    `@timestamp` UInt64,
    `ck_assembly_extension` String,
    INDEX timestamp_index `@timestamp` TYPE minmax GRANULARITY 8192
)
ENGINE = MergeTree
PARTITION BY (toYYYYMMDD(toDateTime(`@timestamp` / 1000, 'Asia/Shanghai')), toHour(toDateTime(`@timestamp` / 1000, 'Asia/Shanghai')))
ORDER BY (`http_host`, intHash64(`@timestamp`))
SAMPLE BY intHash64(`@timestamp`)
SETTINGS in_memory_parts_enable_wal = 0, index_granularity = 8192;

CREATE TABLE IF NOT EXISTS ckibana.table_all AS ckibana.table_local ENGINE Distributed('clickhouse_cluster', ckibana, table_local, rand());

ES data types and CK data types mapping:​

system indexes.​

proxy-settings​

By default, the index name for this is proxy-settings.

Detailed explanation of parameters​

To update the Elasticsearch metadata cluster configuration, the following interface is provided by CKibana.

1）Updating ClickHouse configuration，It will not take effect immediately. You need to wait for 10 seconds for the task to be executed.

curl --location --request POST 'localhost:8080/config/updateCk?url=ip:Port&user=yourUserName&pass=yourPass&defaultCkDatabase=yourCkDatabaseName'

2）Updating Elasticsearch configuration，It will not take effect immediately. You need to wait for 10 seconds for the task to be executed.

curl --location --request POST 'localhost:8080/config/updateEs?host=esHost&headers=key:value,key2:value2'

3）Updating whitelist index

curl --location --request POST 'localhost:8080/config/updateWhiteIndexList?list=index1,index2'

4）Updating blacklist index

curl --location --request POST 'localhost:8080/config/updateBlackIndexList?list=index1,index2'

5）Updating sampling index

curl --location --request POST 'localhost:8080/config/updateSampleIndexList?list=index1,index2'

6）Updating sampleCountMaxThreshold

curl --location --request POST 'localhost:8080/config/updateSampleCountMaxThreshold?sampleCountMaxThreshold=1500000'

7）Updating useCache

curl --location --request POST 'localhost:8080/config/updateUseCache?useCache=true'

8）Updating maxResultRow

curl --location --request POST 'localhost:8080/config/updateMaxResultRow?maxResultRow=40000'

9）Updating roundAbleMinPeriod

curl --location --request POST 'localhost:8080/config/updateRoundAbleMinPeriod?roundAbleMinPeriod=120000'

10）Updating round

curl --location --request POST 'localhost:8080/config/updateRound?round=10'

11）Updating maxTimeRange

curl --location --request POST 'localhost:8080/config/updateMaxTimeRange?maxTimeRange=864000000'

12）Updating enableMonitoring

curl --location --request POST 'localhost:8080/config/updateEnableMonitoring?enableMonitoring=false'

13）Updating msearchThreadPoolCoreSize

curl --location --request POST 'localhost:8080/config/updateMsearchThreadPoolCoreSize?msearchThreadPoolCoreSize=4'

14）Get configuration

curl --location 'localhost:8080/config/all'

15）Full update configuration interface

curl --location 'localhost:8080/config/all' \
--header 'Content-Type: application/json' \
--data '{
    "query":{
        "sampleIndexPatterns":[
            "index1",
            "index2"
        ],
        "sampleCountMaxThreshold":"1500000",
        "useCache":false
    },
    "proxy":{
        "roundAbleMinPeriod":120000,
        "maxTimeRange":86400000,
        "enableMonitoring":true,
        "blackIndexList":[
            "index1",
            "index2"
        ],
        "whiteIndexList":[
            "index1",
            "index2"
        ],
        "es":{
            "host":"esHost
            "headers":[
                {
                    "Authorization":"yourEsClusterAuthorization"
                }
            ]
        },
        "ck":{
            "url":"ip:Port",
            "user":"yourUserName",
            "pass":"yourPass",
            "defaultCkDatabase":"yourCkDatabaseName"
        }
    }
}'

proxy-monitor​

Provide monitoring index. Can be enabled through configuration.

This index can be used for SQL monitoring on Kibana.

proxy-black-list​

Index query SQL blacklist. Can be enabled through configuration.

This index can be directly operated through Kibana proxy：

GET​

curl --location 'http://localhost:8080/proxy/_black_list'

Insert​

range: Query interval (in milliseconds)

sql: Query statement with time range exclusion. Can be obtained in monitor logs.

curl --location 'http://localhost:8080/proxy/_black_list' \
--header 'Content-Type: application/json' \
--data-raw '{
    "range": "900000",
    "sql": "((`@timestamp` = 1698293120000))"
}'

Delete​

curl --location --request DELETE 'http://localhost:8080/proxy/_black_list/{id}'

Contents