Skip to content

Commit e4d7325

Browse files
morningmanmorningman
committed
docs(lakehouse): add Apache Ozone storage docs and catalog examples (4.0.4)
1 parent 218d63f commit e4d7325

File tree

9 files changed

+243
-1
lines changed

9 files changed

+243
-1
lines changed

‎docs/lakehouse/catalogs/hive-catalog.mdx‎

Lines changed: 16 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -104,6 +104,7 @@ Hive transactional tables are supported from version 3.x onwards. For details, r
104104
* [AWS S3](../storages/s3.md)
105105
* [Google Cloud Storage](../storages/gcs.md)
106106
* [Azure Blob](../storages/azure-blob.md)
107+
* [Apache Ozone](../storages/ozone.md) (supported since 4.0.4)
107108
* [Alibaba Cloud OSS](../storages/aliyun-oss.md)
108109
* [Tencent Cloud COS](../storages/tencent-cos.md)
109110
* [Huawei Cloud OBS](../storages/huawei-obs.md)
@@ -359,6 +360,21 @@ Hive transactional tables are supported from version 3.x onwards. For details, r
359360
);
360361
```
361362
</TabItem>
363+
<TabItem value='Ozone' label='Ozone'>
364+
Supported since 4.0.4
365+
```sql
366+
CREATE CATALOG `hive_hms_on_ozone_new_catalog` PROPERTIES (
367+
'type' = 'hms',
368+
'hive.metastore.uris' = 'thrift://127.0.0.1:9383',
369+
'fs.ozone.support' = 'true',
370+
'ozone.endpoint' = 'http://ozone-s3g:9878',
371+
'ozone.access_key' = '<ak>',
372+
'ozone.secret_key' = '<sk>',
373+
'ozone.region' = 'us-east-1',
374+
'ozone.use_path_style' = 'true'
375+
);
376+
```
377+
</TabItem>
362378
<TabItem value='MinIO' label='MinIO'>
363379
```sql
364380
CREATE CATALOG test_hive_on_hms_minio_catalog PROPERTIES (

‎docs/lakehouse/catalogs/iceberg-catalog.mdx‎

Lines changed: 17 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -127,6 +127,7 @@ CREATE CATALOG [IF NOT EXISTS] catalog_name PROPERTIES (
127127
* [AWS S3](../storages/s3.md)
128128
* [Google Cloud Storage](../storages/gcs.md)
129129
* [Azure Blob](../storages/azure-blob.md)
130+
* [Apache Ozone](../storages/ozone.md) (supported since 4.0.4)
130131
* [Aliyun OSS](../storages/aliyun-oss.md)
131132
* [Tencent COS](../storages/tencent-cos.md)
132133
* [Huawei OBS](../storages/huawei-obs.md)
@@ -409,6 +410,22 @@ Support for Nested Namespace needs to be explicitly enabled. For details, please
409410
);
410411
```
411412
</TabItem>
413+
<TabItem value='Ozone' label='Ozone'>
414+
Supported since 4.0.4
415+
```sql
416+
CREATE CATALOG iceberg_fs_on_ozone_catalog PROPERTIES (
417+
'type' = 'iceberg',
418+
'iceberg.catalog.type' = 'hadoop',
419+
'warehouse' = 's3a://bucket/iceberg_warehouse',
420+
'fs.ozone.support' = 'true',
421+
'ozone.endpoint' = 'http://ozone-s3g:9878',
422+
'ozone.access_key' = '<ak>',
423+
'ozone.secret_key' = '<sk>',
424+
'ozone.region' = 'us-east-1',
425+
'ozone.use_path_style' = 'true'
426+
);
427+
```
428+
</TabItem>
412429
<TabItem value='MinIO' label='MinIO'>
413430
```sql
414431
CREATE CATALOG test_iceberg_on_hms_minio_catalog PROPERTIES (

‎docs/lakehouse/catalogs/paimon-catalog.mdx‎

Lines changed: 19 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -114,6 +114,8 @@ The currently dependent Paimon version is 1.0.0.
114114

115115
* [Google Cloud Storage](../storages/gcs.md)
116116

117+
* [Apache Ozone](../storages/ozone.md) (supported since 4.0.4)
118+
117119
* [Alibaba Cloud OSS](../storages/aliyun-oss.md)
118120

119121
* [Tencent Cloud COS](../storages/tencent-cos.md)
@@ -261,6 +263,23 @@ Supported since version 4.0.3, `timestamp_with_local_time_zone` can be mapped to
261263
);
262264
```
263265
</TabItem>
266+
<TabItem value='Ozone' label='Ozone'>
267+
Supported since 4.0.4
268+
```sql
269+
CREATE CATALOG paimon_hms_on_ozone_catalog PROPERTIES (
270+
'type' = 'paimon',
271+
'paimon.catalog.type' = 'hms',
272+
'warehouse' = 's3a://test-bucket/paimon-warehouse',
273+
'hive.metastore.uris' = 'thrift://127.0.0.1:9383',
274+
'fs.ozone.support' = 'true',
275+
'ozone.endpoint' = 'http://ozone-s3g:9878',
276+
'ozone.access_key' = '<ak>',
277+
'ozone.secret_key' = '<sk>',
278+
'ozone.region' = 'us-east-1',
279+
'ozone.use_path_style' = 'true'
280+
);
281+
```
282+
</TabItem>
264283
<TabItem value='MinIO' label='MinIO'>
265284
```sql
266285
CREATE CATALOG paimon_hms_on_minio_catalog PROPERTIES (

‎docs/lakehouse/storages/ozone.md‎

Lines changed: 68 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,68 @@
1+
---
2+
{
3+
"title": "Apache Ozone | Storages",
4+
"language": "en",
5+
"description": "Starting from version 4.0.4, Doris supports accessing Apache Ozone through the S3 Gateway.",
6+
"sidebar_label": "Apache Ozone"
7+
}
8+
---
9+
10+
# Apache Ozone
11+
12+
Starting from version 4.0.4, Doris supports accessing Apache Ozone through the S3 Gateway.
13+
14+
This document describes the parameters required to access Apache Ozone. These parameters apply to:
15+
16+
- Catalog properties
17+
- Table Valued Function properties
18+
- Broker Load properties
19+
- Export properties
20+
- Outfile properties
21+
22+
**To use Ozone as a dedicated storage type, configure `"fs.ozone.support" = "true"` explicitly.**
23+
24+
## Parameter Overview
25+
26+
| Property Name | Legacy Name | Description | Default Value | Required |
27+
| --- | --- | --- | --- | --- |
28+
| ozone.endpoint | s3.endpoint | Ozone S3 Gateway endpoint, for example `http://ozone-s3g:9878` | None | Yes |
29+
| ozone.region | s3.region | Region of Ozone S3 Gateway | `us-east-1` | No |
30+
| ozone.access_key | s3.access_key, s3.access-key-id | Access key used for authentication | None | No* |
31+
| ozone.secret_key | s3.secret_key, s3.secret-access-key | Secret key used for authentication | None | No* |
32+
| ozone.session_token | s3.session_token, s3.session-token | Session token | None | No |
33+
| ozone.connection.maximum | s3.connection.maximum | Maximum number of connections | `100` | No |
34+
| ozone.connection.request.timeout | s3.connection.request.timeout | Request timeout in milliseconds | `10000` | No |
35+
| ozone.connection.timeout | s3.connection.timeout | Connection timeout in milliseconds | `10000` | No |
36+
| ozone.use_path_style | use_path_style, s3.path-style-access | Whether to use path-style access | `true` | No |
37+
| ozone.force_parsing_by_standard_uri | force_parsing_by_standard_uri | Whether to force standard URI parsing | `false` | No |
38+
| fs.ozone.support | | Whether to enable Ozone as storage type | `false` | Yes |
39+
40+
Notes:
41+
42+
- `ozone.access_key` and `ozone.secret_key` must be configured together.
43+
- `fs.s3a.*` keys are not parsed directly by Ozone properties. Use `ozone.*` or compatible `s3.*` keys.
44+
- Ozone supports `s3://`, `s3a://`, and `s3n://` URI schemas.
45+
46+
## Example Configuration
47+
48+
Using `ozone.*` keys:
49+
50+
```properties
51+
"fs.ozone.support" = "true",
52+
"ozone.endpoint" = "http://ozone-s3g:9878",
53+
"ozone.access_key" = "hadoop",
54+
"ozone.secret_key" = "hadoop",
55+
"ozone.region" = "us-east-1",
56+
"ozone.use_path_style" = "true"
57+
```
58+
59+
Using compatible `s3.*` aliases:
60+
61+
```properties
62+
"fs.ozone.support" = "true",
63+
"s3.endpoint" = "http://ozone-s3g:9878",
64+
"s3.access_key" = "hadoop",
65+
"s3.secret_key" = "hadoop",
66+
"s3.region" = "us-east-1",
67+
"s3.path-style-access" = "true"
68+
```

‎i18n/zh-CN/docusaurus-plugin-content-docs/current/lakehouse/catalogs/hive-catalog.mdx‎

Lines changed: 17 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -112,6 +112,8 @@ CREATE CATALOG [IF NOT EXISTS] catalog_name PROPERTIES (
112112

113113
* [Azure Blob](../storages/azure-blob.md)
114114

115+
* [Apache Ozone](../storages/ozone.md)（自 4.0.4 起支持）
116+
115117
* [阿里云 OSS](../storages/aliyun-oss.md)
116118

117119
* [腾讯云 COS](../storages/tencent-cos.md)
@@ -370,6 +372,21 @@ CREATE CATALOG [IF NOT EXISTS] catalog_name PROPERTIES (
370372
);
371373
```
372374
</TabItem>
375+
<TabItem value='Ozone' label='Ozone'>
376+
自 4.0.4 起支持
377+
```sql
378+
CREATE CATALOG `hive_hms_on_ozone_new_catalog` PROPERTIES (
379+
'type' = 'hms',
380+
'hive.metastore.uris' = 'thrift://127.0.0.1:9383',
381+
'fs.ozone.support' = 'true',
382+
'ozone.endpoint' = 'http://ozone-s3g:9878',
383+
'ozone.access_key' = '<ak>',
384+
'ozone.secret_key' = '<sk>',
385+
'ozone.region' = 'us-east-1',
386+
'ozone.use_path_style' = 'true'
387+
);
388+
```
389+
</TabItem>
373390
<TabItem value='MinIO' label='MinIO'>
374391
```sql
375392
CREATE CATALOG test_hive_on_hms_minio_catalog PROPERTIES (
@@ -1101,4 +1118,3 @@ DROP DATABASE [IF EXISTS] hive_ctl.hive_db;
11011118
| -------- | ------------------------------------ |
11021119
| 2.1.6 | 支持 Hive 表数据写回 |
11031120
| 3.0.4 | 支持 JsonSerDe 格式的 Hive 表。支持 Hive4 的事务表。 |
1104-

‎i18n/zh-CN/docusaurus-plugin-content-docs/current/lakehouse/catalogs/iceberg-catalog.mdx‎

Lines changed: 18 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -140,6 +140,8 @@ CREATE CATALOG [IF NOT EXISTS] catalog_name PROPERTIES (
140140

141141
* [Azure Blob](../storages/azure-blob.md)
142142

143+
* [Apache Ozone](../storages/ozone.md)（自 4.0.4 起支持）
144+
143145
* [阿里云 OSS](../storages/aliyun-oss.md)
144146

145147
* [腾讯云 COS](../storages/tencent-cos.md)
@@ -1125,6 +1127,22 @@ Iceberg 的元数层级关系是 Catalog -> Namespace -> Table。其中 Namespac
11251127
);
11261128
```
11271129
</TabItem>
1130+
<TabItem value='Ozone' label='Ozone'>
1131+
自 4.0.4 起支持
1132+
```sql
1133+
CREATE CATALOG iceberg_fs_on_ozone_catalog PROPERTIES (
1134+
'type' = 'iceberg',
1135+
'iceberg.catalog.type' = 'hadoop',
1136+
'warehouse' = 's3a://bucket/iceberg_warehouse',
1137+
'fs.ozone.support' = 'true',
1138+
'ozone.endpoint' = 'http://ozone-s3g:9878',
1139+
'ozone.access_key' = '<ak>',
1140+
'ozone.secret_key' = '<sk>',
1141+
'ozone.region' = 'us-east-1',
1142+
'ozone.use_path_style' = 'true'
1143+
);
1144+
```
1145+
</TabItem>
11281146
<TabItem value='MinIO' label='MinIO'>
11291147
```sql
11301148
CREATE CATALOG test_iceberg_fs_on_minio PROPERTIES (

‎i18n/zh-CN/docusaurus-plugin-content-docs/current/lakehouse/catalogs/paimon-catalog.mdx‎

Lines changed: 19 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -114,6 +114,8 @@ CREATE CATALOG [IF NOT EXISTS] catalog_name PROPERTIES (
114114

115115
* [Google Cloud Storage](../storages/gcs.md)
116116

117+
* [Apache Ozone](../storages/ozone.md)（自 4.0.4 起支持）
118+
117119
* [阿里云 OSS](../storages/aliyun-oss.md)
118120

119121
* [腾讯云 COS](../storages/tencent-cos.md)
@@ -263,6 +265,23 @@ CREATE CATALOG [IF NOT EXISTS] catalog_name PROPERTIES (
263265
);
264266
```
265267
</TabItem>
268+
<TabItem value='Ozone' label='Ozone'>
269+
自 4.0.4 起支持
270+
```sql
271+
CREATE CATALOG paimon_hms_on_ozone_catalog PROPERTIES (
272+
'type' = 'paimon',
273+
'paimon.catalog.type' = 'hms',
274+
'warehouse' = 's3a://test-bucket/paimon-warehouse',
275+
'hive.metastore.uris' = 'thrift://127.0.0.1:9383',
276+
'fs.ozone.support' = 'true',
277+
'ozone.endpoint' = 'http://ozone-s3g:9878',
278+
'ozone.access_key' = '<ak>',
279+
'ozone.secret_key' = '<sk>',
280+
'ozone.region' = 'us-east-1',
281+
'ozone.use_path_style' = 'true'
282+
);
283+
```
284+
</TabItem>
266285
<TabItem value='MinIO' label='MinIO'>
267286
```sql
268287
CREATE CATALOG paimon_hms_on_minio_catalog PROPERTIES (

‎i18n/zh-CN/docusaurus-plugin-content-docs/current/lakehouse/storages/ozone.md‎

Lines changed: 68 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,68 @@
1+
---
2+
{
3+
"title": "Apache Ozone | Storages",
4+
"language": "zh-CN",
5+
"description": "自 4.0.4 版本起，Doris 支持通过 S3 Gateway 访问 Apache Ozone。",
6+
"sidebar_label": "Apache Ozone"
7+
}
8+
---
9+
10+
# Apache Ozone
11+
12+
自 4.0.4 版本起，Doris 支持通过 S3 Gateway 访问 Apache Ozone。
13+
14+
本文档介绍访问 Apache Ozone 所需的参数，这些参数适用于以下场景：
15+
16+
- Catalog 属性
17+
- Table Valued Function 属性
18+
- Broker Load 属性
19+
- Export 属性
20+
- Outfile 属性
21+
22+
**如果要将 Ozone 作为独立存储类型使用，需要显式配置 `"fs.ozone.support" = "true"`。**
23+
24+
## 参数总览
25+
26+
| 属性名称 | 曾用名 | 描述 | 默认值 | 是否必须 |
27+
| --- | --- | --- | --- | --- |
28+
| ozone.endpoint | s3.endpoint | Ozone S3 Gateway 访问端点，例如 `http://ozone-s3g:9878` | 无 | 是 |
29+
| ozone.region | s3.region | Ozone S3 Gateway 区域 | `us-east-1` | 否 |
30+
| ozone.access_key | s3.access_key, s3.access-key-id | 用于认证的 Access Key | 无 | 否* |
31+
| ozone.secret_key | s3.secret_key, s3.secret-access-key | 用于认证的 Secret Key | 无 | 否* |
32+
| ozone.session_token | s3.session_token, s3.session-token | Session Token | 无 | 否 |
33+
| ozone.connection.maximum | s3.connection.maximum | 最大连接数 | `100` | 否 |
34+
| ozone.connection.request.timeout | s3.connection.request.timeout | 请求超时时间（毫秒） | `10000` | 否 |
35+
| ozone.connection.timeout | s3.connection.timeout | 连接超时时间（毫秒） | `10000` | 否 |
36+
| ozone.use_path_style | use_path_style, s3.path-style-access | 是否使用 path-style 访问 | `true` | 否 |
37+
| ozone.force_parsing_by_standard_uri | force_parsing_by_standard_uri | 是否强制使用标准 URI 解析 | `false` | 否 |
38+
| fs.ozone.support | | 是否启用 Ozone 存储类型 | `false` | 是 |
39+
40+
说明：
41+
42+
- `ozone.access_key` 和 `ozone.secret_key` 需要成对配置。
43+
- `fs.s3a.*` 参数不会被 Ozone 属性直接解析，请使用 `ozone.*` 或兼容的 `s3.*` 参数。
44+
- Ozone 支持 `s3://`、`s3a://`、`s3n://` 三种 URI 协议。
45+
46+
## 配置示例
47+
48+
使用 `ozone.*` 参数：
49+
50+
```properties
51+
"fs.ozone.support" = "true",
52+
"ozone.endpoint" = "http://ozone-s3g:9878",
53+
"ozone.access_key" = "hadoop",
54+
"ozone.secret_key" = "hadoop",
55+
"ozone.region" = "us-east-1",
56+
"ozone.use_path_style" = "true"
57+
```
58+
59+
使用兼容的 `s3.*` 参数：
60+
61+
```properties
62+
"fs.ozone.support" = "true",
63+
"s3.endpoint" = "http://ozone-s3g:9878",
64+
"s3.access_key" = "hadoop",
65+
"s3.secret_key" = "hadoop",
66+
"s3.region" = "us-east-1",
67+
"s3.path-style-access" = "true"
68+
```

‎sidebars.ts‎

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -528,6 +528,7 @@ const sidebars: SidebarsConfig = {
528528
items: [
529529
'lakehouse/storages/hdfs',
530530
'lakehouse/storages/s3',
531+
'lakehouse/storages/ozone',
531532
'lakehouse/storages/azure-blob',
532533
'lakehouse/storages/gcs',
533534
'lakehouse/storages/aliyun-oss',

0 commit comments

Comments
 (0)