# System Architecture
Source: https://docs.ryft.io/administration/architecture
Learn how Ryft's architecture is designed, including full separation between the control plane and the data plane to keep your data secure.
## Overview
Ryft's architecture consists of two separate planes:
* **Control Plane** - hosts the UI, authentication, and the metadata store
* **Data Plane** - executes optimizations and compute workloads on your Iceberg lakehouse
The two planes are strictly separated by design. The control plane has **read-only access to metadata only** - it never sees or touches any raw data files. All data-level operations are handled exclusively by the data plane.
Ryft offers two deployment models: a **Managed** option, and a **BYOC (Bring Your Own Cloud)** model, where the data plane runs in your cloud environment.
***
## Control Plane
The control plane is responsible for management and metadata operations, such as:
* UI and API services
* Metadata store
* Authentication and access control
* Job orchestration and monitoring
It is stateless and reads metadata (table schemas, partition layouts, optimization status) via connectors configured with read-only permissions.
It also coordinates with the data plane to trigger optimization jobs and apply configuration changes.
## Data Plane
The data plane is responsible for data operations and compute-intensive workloads, such as:
* Data compaction
* File layout optimization
* Table maintenance
The data plane executes compute-intensive workloads close to where your data lives. All access to Iceberg data files occurs exclusively in the data plane.
***
## Deployment Models
### Managed
In this deployment model, both the control and data planes are hosted and operated by Ryft. This is the fastest way to get started with lakehouse optimization.
Although fully managed, the planes remain logically and physically isolated:
* The **control plane** runs in a multi-tenant environment managed by Ryft.
* The **data plane** runs in a **dedicated cloud account** provisioned and managed by Ryft.
**Highlights:**
* Both metadata and data operations are performed in Ryft cloud.
* The data plane runs in its own dedicated and isolated VPC.
* Ryft manages all infrastructure, including provisioning, upgrades, monitoring, and scaling.
***
### BYOC (Bring Your Own Cloud)
In the **BYOC** model, the control plane is hosted by Ryft, while the data plane is deployed in **your cloud environment**.
You provision a dedicated cloud account, and Ryft manages and operates the data plane infrastructure within it.
**Highlights:**
* Your data never leaves your environment - all data operations are executed within your cloud environment.
* You retain full control over access policies, networking, and resource usage.
* Data operations are triggered and monitored via the Ryft-managed control plane, but are executed within your cloud account.
# Managing Users
Source: https://docs.ryft.io/administration/managing-users
Invite users, assign roles, and control access across Ryft organizations. Covers the different roles, permissions, and how to manage them.
Access to Ryft is managed at the organization level.
A Ryft account may contain multiple organizations, which are commonly used to represent different environments (for example: dev, staging, prod) or distinct business units within the same company. Each organization is fully isolated in terms of configuration, metadata, and access control.
User access is granted per organization and assigned through roles, which determine the actions a user is permitted to perform within that organization.
## Roles
| Role | Description |
| ------------- | ------------------------------------------------------------------- |
| Administrator | Full access to all features and settings, including user management |
| Member | Read and write access; cannot access Settings or manage users |
| Read-Only | View-only access; cannot modify configurations or data |
## Managing Users
Administrators can manage users directly from the **Settings** page.
Only users with the Administrator role can access the Settings page and manage users.
### Inviting a User
1. Navigate to **Settings** and open the **Users** tab.
2. Click **Invite**.
3. Enter the user's email address and select their role.
4. Click **Send Invitation** - the user will receive an email invitation to join the organization.
### Changing a User's Role
1. Navigate to **Settings** and open the **Users** tab.
2. Find the user in the list and select a new role from the dropdown.
### Removing a User
1. Navigate to **Settings** and open the **Users** tab.
2. Find the user in the list and click **Remove User**.
## Provisioning Organizations
To provision additional organizations, contact your account manager or reach out to Ryft Support at [support@ryft.io](mailto:support@ryft.io).
# Configuring Okta SSO
Source: https://docs.ryft.io/administration/okta-sso
Step-by-step guide to set up Okta single sign-on for Ryft. Create an OIDC app, configure redirect URIs, and share credentials with Ryft.
To connect Okta as an identity provider in Ryft, create an OIDC application.
### Creating an application
1. Log in to your Okta Admin Console.
2. Navigate to **Applications** > **Applications**, and click on **Create App Integration**.
3. Select **OIDC - OpenID Connect** as the Sign-in method.
4. Select **Web Application** as the Application type.
5. Set **App Integration Name** to "Ryft" (or any name you prefer).
6. Only select **Authorization Code** under the **Grant type** section.
7. Set **Sign-in redirect URIs** to `https://auth.ryft.io/login/callback`
8. Choose the relevant **Assignments** for your organization.
9. Send your Ryft representative the **Client ID**, **Client Secret** and **Okta Domain**.
10. Download the following Ryft logo and edit the application to add the logo so it's easily recognizable by users:
To limit access to specific IP ranges, you can configure your Okta application using the following [guide](https://support.okta.com/help/s/article/How-to-Allow-Access-to-the-Okta-Applications-Only-From-a-Specific-IP-Range?language=en_US).
### Creating an application link (optional)
1. Click on the Ryft application you just created.
2. Navigate to **General settings** and click **Edit**
3. Set **Login initiated by** to **Either Okta or App**
4. Select **Display application icon to users**
5. Select **Redirect to app to initiate login (OIDC Compliant)** under **Login Flow**
6. Set **Initiate login URI** to **[https://app.ryft.io/auth/login?connection=\{myorg}-sso](https://app.ryft.io/auth/login?connection=\{myorg}-sso)**
* Note - replace `{myorg}` with your Ryft organization name. If your organization name is "acme", the URI would be: `https://app.ryft.io/auth/login?connection=acme-sso`
7. Click **Save**
Done! You have successfully configured Okta SSO for your Ryft organization. Users can now log in to Ryft using their Okta profile.
# Security & Privacy
Source: https://docs.ryft.io/administration/security-and-privacy
Ryft security overview covering encryption, least-privilege access, SSO, MFA, and compliance certifications including SOC2 and ISO-27001.
At Ryft, when it comes to your data, security and privacy are top priorities. That’s why we’ve built our platform securely from the ground up, drawing on years of experience from a team with deep expertise in building secure, enterprise-grade systems.
## Security Highlights
* All data is encrypted at rest and in transit using leading industry standards.
* Ryft only extracts statistics, metadata and query logs.
* Ryft uses service accounts and authentication tokens with **read-only access** to ensure secure integration. We enable **granular permissions**, allowing you to grant access only to the necessary datasets and resources, adhering to the principle of least privilege.
## Building on Secure Foundations
Ryft's security practices are aligned with industry-leading standards to safeguard your data against unauthorized access, breaches, and threats. Our approach includes:
* **End-to-End Encryption** - data is always encrypted at transit & at rest using AES-256.
* **Network communication** - All connections to Ryft are encrypted by default in both directions, using modern ciphers and cryptographic protocols. We ensure data is encrypted in transit with TLS 1.2 or higher. Any HTTP connection attempts are automatically redirected to HTTPS for added security
* **Strong authentication & authorization** - Ryft enforces secured SSO & Multi-factor authentication for all of its employees, and use a least privilege access model for every system access.
* **Real-Time Monitoring and Incident Response** - Ryft leverages comprehensive logging and alerting across its infrastructure to detect threats in real-time.
* **Secure Software Development Lifecycle** - Ryft ensures the security of its product by implementing the leading security standards including automated image scanning, mandatory peer review, vulnerability scanning and more.
* We conduct annual **penetration tests** to assess and strengthen Ryft's security posture by identifying and addressing vulnerabilities. Our latest penetration test and remediation reports are available in our Trust Center upon request.
* All employees use encrypted laptops and undergo regular security awareness training to ensure best practices.
## Compliance
Ryft maintains comprehensive audits to uphold industry standards, regulatory requirements, and data protection laws worldwide.
Ryft is:
* **SOC2 (Type II)** compliant
* **ISO-27001** compliant
* **GDPR** compliant
Ryft will provide complete reports, and will sign NDAs and/or DPAs if requested.
## Sub-processors
| Name | Description |
| :---- | :---------------------------------------- |
| AWS | Cloud hosting and infrastructure provider |
| Auth0 | Authentication provider |
To receive notifications of any updates made to our subprocessors, please email [privacy@ryft.io](mailto:privacy@ryft.io).
# BYOC Deployment
Source: https://docs.ryft.io/deployment-guides/byoc-deployment
Deploy the Ryft data plane in your own AWS account with Bring Your Own Cloud. Configure IAM roles via CloudFormation or Terraform for secure access.
## Overview
Ryft's Bring Your Own Cloud (BYOC) deployment model lets you deploy the platform's data plane infrastructure in a cloud environment that you own and control.
This ensures that all data remains entirely within your cloud boundaries, while Ryft's control plane connects securely to this account and handles provisioning, ongoing updates, and maintenance.
In this guide, we will use the following terms for the different accounts:
The dedicated account you will create to host the data plane components will be referred to as the ***data plane account***
Your existing account that hosts your lakehouse components (catalog, storage, etc.) will be referred to as the ***lakehouse account***
## Step 1: Allocate a Dedicated AWS Account
Following AWS best practices, the preferred approach is to deploy Ryft in a dedicated AWS account, providing clear isolation for identity, resources, quotas, and access control.
This account will be managed by Ryft’s control plane and used to provision and operate the data plane components, including services like VPCs, EMR, and S3 buckets.
Create a new AWS sub-account in your organization. We recommend naming the account `ryft-data-plane`, and placing it in `OU=Infrastructure/OU=Managed`.
## Step 2: Request Your External IDs
Reach out to your Ryft representative to begin your BYOC setup. They will provide you with unique AWS external IDs used to establish secure cross-account access via IAM.
## Step 3: Configure IAM Access to Data Plane Account
Create cross-account IAM roles in the **data plane account**. This allows Ryft to securely manage infrastructure with scoped permissions.
The easiest way to get started is using our provided CloudFormation template.
Click [here](https://console.aws.amazon.com/cloudformation/home#/stacks/quickcreate?stackName=Ryft-Access\&templateURL=https://ryft-public-us-east-1.s3.us-east-1.amazonaws.com/integrations-templates/byoc/RyftBYOC.yaml) to launch the CloudFormation stack and deploy the required IAM role into your AWS account.
If you prefer to use Terraform, you can download and use the module [here](https://ryft-public-us-east-1.s3.us-east-1.amazonaws.com/integrations-templates/byoc/RyftBYOC.tf) to create the IAM role.
Once the IAM setup is done, contact Ryft with the ARNs of the 3 roles created to provision the account to finalize the setup.
## Step 4: Create the Data Plane Access Role in Your Lakehouse Account
In your **lakehouse account**, create an IAM role:
The easiest way to get started is using the data plane role CloudFormation template.
Click [here](https://console.aws.amazon.com/cloudformation/home#/stacks/quickcreate?stackName=Ryft-DataPlaneRole\&templateURL=https://ryft-public-us-east-1.s3.us-east-1.amazonaws.com/integrations-templates/RyftDataPlaneRole.yaml) to launch the CloudFormation stack and deploy the required IAM role into your AWS account.
If you prefer to use Terraform, you can download and use the data plane role module [here](https://ryft-public-us-east-1.s3.us-east-1.amazonaws.com/integrations-templates/RyftDataPlaneRole.tf) to create the IAM role.
Alternatively, you can create the role manually:
1. Log in to your AWS account
2. Navigate to **IAM -> Roles**
3. Click **Create Role**
4. Give the role a name, e.g. `Ryft-DataPlaneRole`
5. Click **Next: Permissions**
6. Click **Attach existing policies**
Attach the following policies:
#### Glue Catalog Access
```json theme={null}
{
"Sid": "RyftGlueDataPlaneAccess",
"Effect": "Allow",
"Action": [
"glue:GetTable",
"glue:GetTables",
"glue:GetDatabases",
"glue:GetDatabase",
"glue:UpdateTable",
"glue:CreateTable",
"glue:DeleteTable"
],
"Resource": [
"arn:aws:glue:::catalog",
"arn:aws:glue:::database/*",
"arn:aws:glue:::table/*"
]
}
```
**Note:** Replace `` and `` with your AWS region and account ID.
#### S3 Bucket Access
```json theme={null}
{
"Sid": "AllowRyftExecutionRoleAccess",
"Effect": "Allow",
"Action":[
"s3:GetObject",
"s3:PutObject",
"s3:DeleteObject",
"s3:ListBucket",
"s3:GetBucketLocation",
"s3:GetObjectVersion",
"s3:GetObjectTagging",
"s3:PutObjectTagging"
],
"Resource": [
"arn:aws:s3:::/*",
"arn:aws:s3:::"
]
}
```
**Important:** Replace `` with your actual S3 bucket name.
Configure a trust relationship for the role, use the ExecutionRole ARN and Data Plane Role External ID provided by Ryft:
```json theme={null}
{
"Version": "2012-10-17",
"Statement": [
{
"Effect": "Allow",
"Principal": {
"AWS": "arn:aws:iam:::role/Ryft-ExecutionRole"
},
"Action": "sts:AssumeRole",
"Condition": {
"StringEquals": {
"sts:ExternalId": ""
}
}
}
]
}
```
Once the IAM role is created, share the ARN of the role with your Ryft representative.
## Step 5: Create the Control Plane Access Role in Your Lakehouse Account
If you did not yet integrate the Ryft control plane with your **lakehouse account**, make sure to create it by following the steps [here](/integrations/aws-role-setup).
This article contains instructions for setting up the BYOC data plane deployment.
For a general overview of the Ryft architecture and deployment models, refer to [System Architecture](/administration/architecture).
When using a BYOC deployment, it's recommended to turn off event recording for `AWS::EC2::NetworkInterface` in AWS Config
Follow the recommendation of AWS as described [here](https://repost.aws/knowledge-center/emr-serverless-reduce-costs)
AWS Config -> Settings -> Edit Recorder Settings -> Override settings -> Exclude AWS::EC2::NetworkInterface from recording
# AWS PrivateLink Setup
Source: https://docs.ryft.io/deployment-guides/private-link
Set up AWS PrivateLink for secure private connectivity between your VPC and Ryft services. Covers endpoint creation and load balancer configuration.
## Overview
Ryft supports AWS PrivateLink to enable secure and private connectivity between your VPC and Ryft services without exposing traffic to the public internet.
This guide provides step-by-step instructions to set up AWS PrivateLink for Ryft.
#### AWS PrivateLink
1. Follow the instructions [here](https://docs.aws.amazon.com/vpc/latest/privatelink/create-endpoint-service.html) to setup the PrivateLink endpoint in your AWS account, and share the endpoint details with Ryft support.
2. Ensure that `us-east-1` is one of the supported regions in your PrivateLink Endpoint Service configuration
3. Add the Ryft AWS account ID `864981746572` as a permitted principal for your PrivateLink endpoint. If you are setting up a BYOC deployment, add the account ID of your data plane account as well.
4. If you setup the PrivateLink to require manual approval, please approve the connection request from Ryft after the setup.
5. Ensure that `Enforce inbound rules on PrivateLink traffic` is set to **Off** on the Load Balancer associated with the PrivateLink endpoint service.
6. Check "Enable cross-zone load balancing" in the configuration of the Load Balancer associated with the PrivateLink endpoint service.
Provide the service name of your PrivateLink endpoint to Ryft to finalize the setup.
# FAQ
Source: https://docs.ryft.io/faq
Find answers to common questions about Ryft, including integrations, features, security, and scale.
### What is Ryft?
Ryft is the Intelligent Iceberg Management Platform. It automatically maintains and optimizes your Iceberg tables based on usage, while keeping your lakehouse fully GDPR-compliant, resilient and secure.
### Who is Ryft for?
Ryft is designed for data engineers and data platform leaders operating large-scale Iceberg environments who want to reduce operational overhead, improve query performance, and keep their lakehouse reliable and compliant over time.
### How quickly can I connect Ryft to my environment?
Ryft integrates with Iceberg catalogs, object storage, and query engines in minutes - no migration or rewrites required.
### How is Ryft deployed?
Ryft is built on a control plane / data plane model: orchestration and metadata live in the *control plane*, while all data processing runs in the *data plane*.
The data plane can be managed by Ryft (Managed) or deployed in your environment (BYOC).
For more details, see [System Architecture](/administration/architecture).
### Do I have to use a specific catalog or query engine?
No. Ryft works with any Iceberg catalog, object storage, and query engine you use. If an integration is missing, contact Ryft Support at [support@ryft.io](mailto:support@ryft.io).
### Does Ryft require changes to my existing pipelines?
No. Ryft analyzes usage patterns and applies optimizations seamlessly without changing your upstream pipelines or query workloads.
### How frequently does Ryft update its data?
Ryft updates in near real time based on the latest data from your data sources. This means that you can always see the most up-to-date information and insights.
### Can I control the optimization behavior?
Yes. In addition to automatic optimization, Ryft supports custom settings for fine-grained control of the maintenance and optimization procedures.
For more information, check out [Managed Tables](/lakehouse-management/table-management/managed-tables).
### What scale does Ryft support?
Ryft is built to handle large-scale, high-throughput environments, supporting petabytes of data and hundreds of thousands of tables. Its distributed architecture enables efficient processing and optimization across large datasets.
### How does Ryft protect my data?
Ryft adheres to industry standards and best practices for security: encryption at rest and in transit, role-based access control, audit logging, and regular security assessments.
# AWS Athena
Source: https://docs.ryft.io/integrations/athena
Connect AWS Athena to Ryft for query history analysis and usage-based Iceberg optimizations. Set up IAM roles via CloudFormation or manually.
The best way to set up Athena integration is by running CloudFormation/Terraform directly from the
[AWS Role Setup](/integrations/aws-role-setup#option-a%3A-create-iam-role-with-policies-attached-with-cloudformation-%2F-terraform).
For manual setup, proceed below.
### Manual Setup
Follow the manual steps to create a role and attach the Athena policy to it, as described [here](/integrations/aws-role-setup#aws-athena):
1. Create the "**Ryft-ControlPlaneRole**" (or reuse if already exists)
2. Create and attach the Athena access policy
You are done! Locate the ARN of the role you have just created ("**Ryft-ControlPlaneRole**") and provide it to Ryft.
The ARN should look similar to: `arn:aws:iam:::role/Ryft-ControlPlaneRole`
# AWS
Source: https://docs.ryft.io/integrations/aws-role-setup
Integrate AWS services to Ryft with an IAM role. Use CloudFormation, Terraform, or configure the role manually in your account.
Ryft accesses metadata through a single IAM Role. Use CloudFormation / Terraform or create the role manually.
## Option A: Create IAM Role with CloudFormation / Terraform
For the simplest integration, we recommend using our CloudFormation template to create the role. Click [here](https://console.aws.amazon.com/cloudformation/home#/stacks/quickcreate?param_EnableAthenaAccess=true\&stackName=Ryft-Access\&templateURL=https://ryft-public-us-east-1.s3.us-east-1.amazonaws.com/integrations-templates/RyftControlPlaneRole.yaml)
to launch the CloudFormation template.
If you prefer to use Terraform, you can download and use the module [here](https://ryft-public-us-east-1.s3.us-east-1.amazonaws.com/integrations-templates/RyftControlPlaneRole.tf) to create the IAM role.
* Make sure to enter the External ID provided to you by Ryft.
* Before launching the stack, make sure to log in to the AWS account where you want to create the access role.
* To disable Glue access, set `EnableGlueAccess` to `false`.
* To disable Athena access, set `EnableAthenaAccess` to `false`.
## Option B: Create the IAM Role manually
1. Login to your AWS account
2. Security → Identity & Compliance → IAM → Roles
3. Click **Create Role**
4. Select **AWS Account**
5. Select **Another AWS Account**
6. Enter Ryft’s AWS account ID: **864981746572**
7. Enable **Require External ID**, and enter the External ID received from Ryft
8. For permissions, we are going to create the policies in the next steps
9. Give the role a name, e.g - `Ryft-ControlPlaneRole`
10. Click **Create Role**
### AWS Policies
#### AWS S3
1. IAM → Roles → Search for “**Ryft-ControlPlaneRole”** (or the name you used)
2. Add permissions → Create inline policy → Select the **JSON** tab
3. Add the following policy to provide S3 access
4. Give the policy a name and save it (e.g - `RyftS3AccessPolicy`)
5. Attach the created policy to the **Ryft-ControlPlaneRole**
```json {14,24} theme={null}
{
"Version": "2012-10-17",
"Statement": [
{
"Version": "2012-10-17",
"Statement": [
{
"Sid": "AllowS3MetadataAccess",
"Effect": "Allow",
"Action": [
"s3:GetObject"
],
"Resource": [
"arn:aws:s3:::/*/metadata/*"
]
},
{
"Sid": "AllowListBucket",
"Effect": "Allow",
"Action": [
"s3:ListBucket"
],
"Resource": [
"arn:aws:s3:::"
]
}
]
}
]
}
```
#### AWS Glue
1. IAM → Roles → Search for “**Ryft-ControlPlaneRole”** (or the name you used)
2. Add permissions → Create inline policy → Select the **JSON** tab
3. Add the following policy to provide Glue API access
4. Give the policy a name and save it (e.g - `RyftGlueAccessPolicy`)
5. Attach the created policy to the **Ryft-ControlPlaneRole**
```json {19-21} theme={null}
{
"Version": "2012-10-17",
"Statement": [
{
"Version": "2012-10-17",
"Statement": [
{
"Sid": "AllowGlueCatalogTableAccess",
"Effect": "Allow",
"Action": [
"glue:GetTable",
"glue:GetTables",
"glue:GetDatabase",
"glue:GetDatabases",
"glue:GetCatalog",
"glue:GetCatalogs"
],
"Resource": [
"arn:aws:glue:::table/*/*",
"arn:aws:glue:::catalog",
"arn:aws:glue:::catalog/*",
"arn:aws:glue:::database/*"
]
}
]
}
]
}
```
#### AWS Athena
1. IAM → Roles → Search for “**Ryft-ControlPlaneRole”** (or the name you used)
2. Add permissions → Create inline policy → Select the **JSON** tab
3. Add the following policy to provide Athena API access
4. Name the policy “**Ryft-AthenaMetadataAccess”**
5. Attach the created policy to the **Ryft-ControlPlaneRole**
```json theme={null}
{
"Version": "2012-10-17",
"Statement": [
{
"Sid": "Ryft-AthenaMetadataAccess",
"Effect": "Allow",
"Action": [
"athena:ListWorkGroups",
"athena:GetWorkGroup",
"athena:GetQueryExecution",
"athena:GetQueryRuntimeStatistics",
"athena:BatchGetQueryExecution",
"athena:ListQueryExecutions",
"athena:ListDataCatalogs",
"athena:GetDataCatalog"
],
"Resource": "*"
}
]
}
```
You are done! Locate the ARN of the role you have just created (”**Ryft-ControlPlaneRole”)** and provide it to Ryft, we will now finish setting up the integration.
The ARN should look similar to: `arn:aws:iam:::role/Ryft-ControlPlaneRole`
## Glue Metadata Encryption
If you are using Glue Metadata Encryption, ensure that the role you created above has the necessary KMS permissions to the encryption key as well.
```json {11} theme={null}
{
"Version": "2012-10-17",
"Statement": [
{
"Effect": "Allow",
"Action": [
"kms:Decrypt",
"kms:Encrypt",
"kms:GenerateDataKey"
],
"Resource": "arn:aws:kms:us-east-1:111122223333:key/key-id"
}
]
}
```
## AWS Lake Formation
If you are using AWS Lake Formation to manage access to your Glue Catalog, ensure that the role you created above has the necessary Lake Formation permissions as well.
Follow the AWS documentation [here](https://docs.aws.amazon.com/lake-formation/latest/dg/granting-database-permissions.html) to grant the required permissions to the role.
## Amazon S3 Access Points
If you are using Amazon S3 Access Points as the storage location for your tables, ensure that the role you created above has the necessary permissions for the access point.
The policy for access points should include the same S3 actions mentioned above, with the resource in the format of `arn:aws:s3::::accesspoint/`.
For more details, see AWS documentation [here](https://docs.aws.amazon.com/AmazonS3/latest/userguide/access-points-policies.html).
# BigLake Metastore
Source: https://docs.ryft.io/integrations/biglake-metastore
Connect Google BigLake Metastore to Ryft for Iceberg table management and visibility.
Integrating with BigLake Metastore is currently in private preview.
For more details, contact Ryft Support at [support@ryft.io](mailto:support@ryft.io).
# Custom Engine
Source: https://docs.ryft.io/integrations/custom-engine
Send query data from any custom engine to Ryft via the Ingest API. Covers authentication, payload format, and bulk ingestion.
For query engines not natively supported by Ryft, you can send query data directly using the Ryft Ingest API.
### Get your Ryft Ingest Token
Ryft will provide a unique ingest token for your environment. This token is required to authenticate requests to the Ingest API.
### Send Query Data
Send a `POST` request to the ingest endpoint with your query data as JSON:
```bash{2} theme={null}
curl -X POST https://ingest.ryft.io/ingest/push/queries \
-H "X-Ryft-Token: " \
-H "Content-Type: application/json" \
-d '{
"context": {
"catalog": "iceberg",
"schema": "gold",
"queryType": "SELECT",
"user": "my-service",
"userAgent": "my-client"
},
"timestamp": "2026-03-10T12:00:00.000Z",
"duration_ms": 100,
"query": "SELECT * FROM gold.my_table",
"queryId": "test-001",
"queryState": "FINISHED",
"tags": ["source:my-engine"]
}'
```
A `200` response means the message was accepted and will be processed.
### Bulk Ingestion
You can send multiple queries in a single request by passing a JSON array instead of a single object:
```bash{2} theme={null}
curl -X POST https://ingest.ryft.io/ingest/push/queries \
-H "X-Ryft-Token: " \
-H "Content-Type: application/json" \
-d '[
{
"context": { "catalog": "iceberg" },
"queryId": "q-123",
"query": "SELECT * FROM my_table",
"queryState": "FINISHED",
"duration_ms": 1230,
"timestamp": "2026-03-10T12:00:00.000Z"
},
{
"context": { "catalog": "iceberg" },
"queryId": "q-124",
"query": "SELECT count(*) FROM events",
"queryState": "FINISHED",
"duration_ms": 450,
"timestamp": "2026-03-10T12:01:00.000Z"
}
]'
```
### Payload Schema
| Field | Type | Required | Description |
| ---------------------------- | ------------------- | -------------------------- | ----------------------------------------------------------------------------------------------------------------------------------- |
| `context` | object | **yes** | Query environment context. All fields within are optional — pass `{}` if no context is available. |
| `context.catalog` | string | no | Catalog name (e.g. `iceberg`) |
| `context.schema` | string | no | Default namespace for unqualified table names |
| `context.queryType` | string | no | One of: `SELECT`, `EXPLAIN`, `DESCRIBE`, `INSERT`, `UPDATE`, `DELETE`, `ANALYZE`, `DATA_DEFINITION`, `ALTER_TABLE_EXECUTE`, `MERGE` |
| `context.serverVersion` | string | no | Query engine version |
| `context.user` | string | no | User or service account that ran the query |
| `context.userAgent` | string | no | Client identifier |
| `query` | string | **yes** | The SQL query text |
| `queryId` | string | **yes** | Unique query identifier |
| `queryState` | string | **yes** | Terminal query state: `FINISHED`, `SUCCEEDED`, or `FAILED` |
| `timestamp` | string | **yes** | ISO 8601 timestamp of query execution |
| `duration_ms` | number | **yes** | Execution time in milliseconds |
| `tags` | string\[] | no | Free-form labels for filtering in Query Explorer |
| `failureInfo` | object or object\[] | no | Only needed for failed queries (`queryState: "FAILED"`). Can be a single object or an array. |
| `failureInfo.errorCode.type` | string | yes (within `failureInfo`) | One of: `USER_ERROR`, `INTERNAL_ERROR`, `INSUFFICIENT_RESOURCES`, `EXTERNAL` |
| `failureInfo.errorCode.code` | number | no | Numeric error code (free-form) |
| `failureInfo.errorCode.name` | string | no | Error name (free-form) |
| `failureInfo.failureMessage` | string | no | Human-readable error description |
| `failureInfo.failureType` | string | no | Exception or error class name |
Make sure outbound connectivity is enabled to `ingest.ryft.io` on port `443` (HTTPS).
# Databricks
Source: https://docs.ryft.io/integrations/databricks
Connect Databricks to Ryft to sync Iceberg query history and table metadata.
Databricks integration is currently in private preview.
For more details, contact Ryft Support at [support@ryft.io](mailto:support@ryft.io).
# AWS Glue Data Catalog
Source: https://docs.ryft.io/integrations/glue-catalog
Connect AWS Glue Data Catalog to Ryft for Iceberg table management and optimization. Requires Glue and S3 access - set up via CloudFormation, Terraform, or manually.
**The AWS Glue Data Catalog integration requires both AWS Glue access and S3 metadata access.**
The best way to set up the AWS Glue integration is by running CloudFormation/Terraform directly from the
[AWS Role Setup](/integrations/aws-role-setup#option-a%3A-create-iam-role-with-policies-attached-with-cloudformation-%2F-terraform).
For manual setup, proceed below.
### Manual Setup
Follow the manual steps to create a role and attach Glue and S3 policies, as described [here](/integrations/aws-role-setup#aws-policies):
1. Create the "**Ryft-ControlPlaneRole**" (or reuse if already exists)
2. Create and attach the Glue access policy
3. Create and attach the S3 access policy
You are done! Locate the ARN of the role you have just created ("**Ryft-ControlPlaneRole**") and provide it to Ryft.
The ARN should look similar to: `arn:aws:iam:::role/Ryft-ControlPlaneRole`
## Glue Metadata Encryption
If you are using Glue Metadata Encryption, ensure that the role you created above has the necessary KMS permissions to the encryption key as well.
```json {11} theme={null}
{
"Version": "2012-10-17",
"Statement": [
{
"Effect": "Allow",
"Action": [
"kms:Decrypt",
"kms:Encrypt",
"kms:GenerateDataKey"
],
"Resource": "arn:aws:kms:us-east-1:111122223333:key/key-id"
}
]
}
```
## AWS Lake Formation
If you are using AWS Lake Formation to manage access to your Glue Catalog, ensure that the role you created above has the necessary Lake Formation permissions as well.
Follow the AWS documentation [here](https://docs.aws.amazon.com/lake-formation/latest/dg/granting-database-permissions.html) to grant the required permissions to the role.
## Amazon S3 Access Points
If you are using Amazon S3 Access Points as the storage location for your tables, ensure that the role you created above has the necessary permissions for the access point.
The policy for access points should include the same S3 actions mentioned above, with the resource in the format of `arn:aws:s3::::accesspoint/`.
For more details, see AWS documentation [here](https://docs.aws.amazon.com/AmazonS3/latest/userguide/access-points-policies.html).
# Google Cloud
Source: https://docs.ryft.io/integrations/google-cloud
Connect Google Cloud data lakes to Ryft for Iceberg table management and optimization.
Integrating with Google Cloud data lakes is currently in private preview.
For more details, contact Ryft Support at [support@ryft.io](mailto:support@ryft.io).
# Hive Metastore
Source: https://docs.ryft.io/integrations/hive-metastore
Connect Hive Metastore to Ryft via the Thrift API. Set up PrivateLink or IP access lists for network connectivity and configure S3 metadata access.
## Step 1: Connect with Hive Metastore through AWS PrivateLink
Ryft integrates with Hive Metastore using the default Thrift API.
To integrate with Hive Metastore, obtain the Thrift URI of the Hive Metastore server and provide it to Ryft support.
The URI should look similar to: `thrift://:9083`
### Network connectivity
Ensure that the Ryft instance can connect to the Hive Metastore server over the network. The default port for Hive Metastore is `9083`, but this may vary based on your configuration.
#### AWS PrivateLink
If your Hive Metastore requires PrivateLink connectivity, follow the [PrivateLink setup guide](/deployment-guides/private-link) to set it up.
Note: The only port needed is `9083`, which is the default port for Hive Metastore.
Once your PrivateLink endpoint is set up, provide the endpoint details to Ryft, including the AZs where the endpoint is available.
#### IP Access Lists
If you are restricting network access to specific IP addresses, please allow the following IP addresses:
```
18.211.150.201
52.45.62.176
35.174.30.119
```
## Step 2: Create "**Ryft-ControlPlaneRole**" for metadata access
Ryft needs access to metadata files as stored in S3 in addition to the catalog access.
The easiest way to setup this role is with CloudFormation / Terraform, as described [here](/integrations/aws-role-setup#option-a%3A-create-iam-role-with-policies-attached-with-cloudformation-%2F-terraform).
For manual setup, use the [manual guide](/integrations/aws-role-setup#option-b%3A-create-the-iam-role-manually), and
make sure to define the S3 policy.
Locate the ARN of the role you have just created (”**Ryft-ControlPlaneRole”)** and provide it to Ryft.
The ARN should look similar to: `arn:aws:iam:::role/Ryft-ControlPlaneRole`
# Iceberg REST Catalog
Source: https://docs.ryft.io/integrations/iceberg-rest-catalog
Integrate any Iceberg REST Catalog with Ryft.
## Step 1: Connect with Iceberg REST Catalog
Ryft supports integration with Iceberg REST Catalog. To integrate, obtain the catalog URL and provide it to Ryft support.
### Network connectivity
Ensure that the Ryft instance can connect to the catalog over the network.
#### AWS PrivateLink
If your REST catalog requires PrivateLink connectivity, follow the [PrivateLink setup guide](/deployment-guides/private-link) to set it up.
Note: The only port needed is `8181`, which is the default port for Iceberg REST catalog, or a different one if you are using a different port.
Once your PrivateLink endpoint is set up, provide the endpoint details to Ryft, including the AZs where the endpoint is available.
#### IP Access Lists
If you are restricting network access to specific IP addresses, please allow the following IP addresses:
```
18.211.150.201
52.45.62.176
35.174.30.119
```
Once you have this information, contact your Ryft representative to complete the setup.
# Managed Deployment
Source: https://docs.ryft.io/integrations/managed-deployment
Set up Ryft managed deployment where Ryft hosts and operates everything. Create a data plane IAM role for catalog and S3 access in your account.
## Overview
Ryft offers a fully managed deployment model, for customers who want the fastest and simplest onboarding experience. In this model, Ryft hosts and operates your deployment, provisioned and fully managed on your behalf.
### Step 1: Obtain your Execution Role & External ID
Reach out to the Ryft team to begin your setup. We’ll provide you with a unique IAM role & External ID used to establish secure cross-account access via IAM.
### Step 2: Create the Data Plane Role
Create an IAM role in your AWS account:
The easiest way to get started is using our provided CloudFormation template.
Click [here](https://console.aws.amazon.com/cloudformation/home#/stacks/quickcreate?stackName=Ryft-DataPlaneRole\&templateURL=https://ryft-public-us-east-1.s3.us-east-1.amazonaws.com/integrations-templates/RyftDataPlaneRole.yaml) to launch the CloudFormation stack and deploy the required IAM role into your AWS account.
If you prefer to use Terraform, you can download and use the module [here](https://ryft-public-us-east-1.s3.us-east-1.amazonaws.com/integrations-templates/RyftDataPlaneRole.tf) to create the IAM role.
Alternatively, you can create the role manually:
1. Login to your AWS account
2. Navigate to **IAM -> Roles**
3. Click **Create Role**
4. Give the role a name, e.g - `Ryft-DataPlaneRole`
5. Click **Next: Permissions**
6. Click **Attach existing policies**
Attach the following policies:
#### Glue Catalog Access
```json theme={null}
{
"Sid": "RyftGlueDataPlaneAccess",
"Effect": "Allow",
"Action": [
"glue:GetTable",
"glue:GetTables",
"glue:GetDatabases",
"glue:GetDatabase",
"glue:UpdateTable",
"glue:CreateTable",
"glue:DeleteTable"
],
"Resource": [
"arn:aws:glue:::catalog",
"arn:aws:glue:::database/*",
"arn:aws:glue:::table/*"
]
}
```
**Note:** Replace `` and `` with your AWS region and account ID.
#### S3 Bucket Access
```json theme={null}
{
"Sid": "AllowRyftExecutionRoleAccess",
"Effect": "Allow",
"Action":[
"s3:GetObject",
"s3:PutObject",
"s3:DeleteObject",
"s3:ListBucket",
"s3:GetBucketLocation",
"s3:GetObjectVersion",
"s3:GetObjectTagging",
"s3:PutObjectTagging"
],
"Resource": [
"arn:aws:s3:::/*",
"arn:aws:s3:::"
]
}
```
**Important:** Replace `` with your actual S3 bucket name.
### Step 3: Create the Control Plane Role
If you did not yet integrate the Ryft control plane with your **lakehouse account**, make sure to create it by following the steps [here](/integrations/aws-role-setup).
### Step 4: Finalize integration
Once the IAM roles are created, share the ARNs of the roles with your Ryft representative to finalize the setup.
## Support
If you run into any issues during setup, contact your Ryft support team.
This article contains instructions for setting up the Managed deployment type.
For a general overview of the Ryft architecture and deployment models, refer to [System Architecture](/administration/architecture).
# Microsoft Azure
Source: https://docs.ryft.io/integrations/microsoft-azure
Connect Microsoft Azure data lakes to Ryft for Iceberg table management and visibility.
Integrating with Microsoft Azure data lakes is currently in private preview.
For more details, contact Ryft Support at [support@ryft.io](mailto:support@ryft.io).
# Microsoft OneLake
Source: https://docs.ryft.io/integrations/onelake
Connect Microsoft Fabric OneLake to Ryft via Iceberg REST Catalog.
Ryft connects to your Microsoft Fabric OneLake lakehouse via its Iceberg REST Catalog API to collect table metadata for visibility and optimization.
## Prerequisites
* A Microsoft Fabric workspace with a lakehouse containing Iceberg tables
* A Microsoft Entra ID App Registration (service principal) with access to the workspace
* Your Azure Tenant ID, Workspace ID, and Lakehouse ID
* The following tenant admin settings must be enabled (Fabric Portal -> Settings (gear icon) -> Admin portal -> Tenant settings):
* **Service principals can call Fabric public APIs** — found under **Developer settings**
* **Users can access data stored in OneLake with apps external to Fabric** — found under **OneLake settings**
## Step 1: Create a Microsoft Entra ID App Registration
Register an application in Microsoft Entra ID that Ryft will use to authenticate against your OneLake lakehouse.
1. Navigate to **Azure Portal -> Microsoft Entra ID -> Manage -> App registrations**
2. Click **New registration**
3. Enter a name (e.g. `Ryft-OneLake-Integration`), select **Single tenant only** for supported account types, and register the application
4. Note the **Application (client) ID** and **Directory (tenant) ID**
5. Navigate to **Manage -> Certificates & secrets -> New client secret**
6. Create a secret and note its **Value**
## Step 2: Grant access to your Fabric workspace
The service principal must have read access to your Microsoft Fabric workspace and lakehouse.
1. Navigate to [**Microsoft Fabric**](https://app.fabric.microsoft.com) and open your workspace from the left sidebar
2. Click **Manage access**
3. Click **Add people or groups**
4. Search for the App Registration name you created (e.g. `Ryft-OneLake-Integration`)
5. Assign the **Contributor** role (minimum required for metadata access via the OneLake API)
6. Click **Add**
## Step 3: Collect the required identifiers
Open your lakehouse in the Fabric portal — you should see the Explorer view with your tables and files:
You will need the following identifiers from your Fabric environment:
| Parameter | Where to find it |
| ----------------- | ------------------------------------------------------------ |
| **Tenant ID** | Azure Portal -> Microsoft Entra ID -> Overview |
| **Workspace ID** | Fabric Portal -> Workspace settings -> URL or details pane |
| **Lakehouse ID** | Fabric Portal -> Lakehouse -> URL or details pane |
| **Client ID** | Azure Portal -> App registrations -> Application (client) ID |
| **Client Secret** | The secret value from Step 1 |
The Workspace ID and Lakehouse ID are GUIDs visible in the Fabric portal URL when you open the workspace or lakehouse. For example:
`https://app.fabric.microsoft.com/groups//lakehouses/`
## Step 4: Provide credentials to Ryft
Share the following with your Ryft representative:
* **Tenant ID** — your Microsoft Entra ID directory ID
* **Workspace ID** — the Fabric workspace GUID
* **Lakehouse ID** — the Fabric lakehouse GUID
* **Client ID** — the App Registration application ID
* **Client Secret** — the App Registration secret value
Ryft will securely store these credentials and use them to authenticate against the OneLake Iceberg REST API at `https://onelake.table.fabric.microsoft.com/iceberg`.
# Apache Polaris
Source: https://docs.ryft.io/integrations/polaris-catalog
Connect Apache Polaris to Ryft using the Iceberg REST API.
Apache Polaris is fully compatible with the Iceberg REST API, to integrate see [Iceberg REST Catalog](/integrations/iceberg-rest-catalog/).
In addition, please share the following information as well:
* Polaris Authentication token if exists
* Warehouse (Catalog name)
# Amazon Redshift
Source: https://docs.ryft.io/integrations/redshift
Connect Amazon Redshift to Ryft to sync Iceberg query history and table metadata.
Amazon Redshift integration is currently in private preview.
For more details, contact Ryft Support at [support@ryft.io](mailto:support@ryft.io).
# Ryft MCP
Source: https://docs.ryft.io/integrations/ryft-mcp
Enrich AI agents with lakehouse context using the Ryft MCP server. Supports any agent framework.
Ryft MCP is currently in **Private Preview**. To get early access, contact your Ryft representative.
Ryft MCP is a [Model Context Protocol](https://modelcontextprotocol.io/) server that enriches your AI agents with full lakehouse context, including table metadata, query history, and domain knowledge - so they can write accurate queries and answer business questions without guessing.
## Prerequisites
* One of the supported AI tools: [Claude](https://claude.ai), [Claude Code](https://docs.anthropic.com/en/docs/claude-code), or [Cursor](https://www.cursor.com/)
## Setup
**1. Add MCP configuration**
**2. Connect**
Click **Install** in the prompt that appears, then click **Connect**. A browser window will open for you to log in with your Ryft credentials. After authenticating, you can close the browser tab and return to Cursor.
Add the following to `.cursor/mcp.json` in your project root (or `~/.cursor/mcp.json` for global access):
```json theme={null}
{
"mcpServers": {
"ryft": {
"type": "http",
"url": "https://app.ryft.io/mcp"
}
}
}
```
Then open Cursor Settings (`Cmd/Ctrl + Shift + J`) > **Tools & MCP** > find **Ryft** > **Enable** > **Connect**. A browser window will open for you to log in with your Ryft credentials.
**1. Add MCP server**
Run the following command in your terminal:
```bash theme={null}
claude mcp add --transport http ryft https://app.ryft.io/mcp
```
**2. Connect and authenticate**
In Claude Code, run `/mcp` to load the server. Select Ryft and choose **Authenticate**. A browser window will open for you to log in with your Ryft credentials.
Custom connectors are available on Claude Pro, Max, Team, and Enterprise plans.
**1. Add a custom connector**
Go to **Settings** > **Connectors** > **Add custom connector** and enter the following:
| Field | Value |
| ------------------------- | ------------------------- |
| **Name** | `Ryft` |
| **Remote MCP server URL** | `https://app.ryft.io/mcp` |
Click **Add** to save the connector.
**2. Authenticate**
A browser window will open for you to log in with your Ryft credentials. Once authenticated, the Ryft connector will be available in your conversations. You can close the browser tab and return to Claude.
**3. Enable in conversations**
In a new conversation, click the **+** button at the bottom of the chat, then **Connectors**, and toggle **Ryft** on.
## Verify
To confirm everything is working, ask your AI tool any question about your lakehouse, for example:
> "Search for tables related to orders"
If the connection is set up correctly, the assistant will use the `search_tables` tool and return results from your lakehouse.
## Troubleshooting
Your AI tool will prompt you to re-authenticate via your browser. Follow the prompts to log in again.
* Verify your network can reach `app.ryft.io`. Check if your organization requires a VPN.
* For Claude: go to Settings > Connectors and check that Ryft is enabled
* For Claude Code: run `/mcp` to check server status
* For Cursor: open Cursor Settings (`Cmd/Ctrl + Shift + J`) > Tools & MCP to check server status
Restart your AI tool after adding the MCP configuration. For Claude Code, run `/mcp` to reconnect.
# S3 Tables
Source: https://docs.ryft.io/integrations/s3-tables
Connect Amazon S3 Tables to Ryft for Iceberg table management and storage optimization.
Integrating with S3 Tables is currently in private preview.
For more details, contact Ryft Support at [support@ryft.io](mailto:support@ryft.io).
# Snowflake
Source: https://docs.ryft.io/integrations/snowflake
Connect Snowflake to Ryft by creating a dedicated user, role, and warehouse. Grant permissions for query history and Iceberg metadata access.
### Creating the User and Role
Please use the SQL snippet below to create a user for Ryft. Before using the snippet, you will need to populate the following parameters:
* `ryft_username`
* `ryft_warehouse_size`
* `ryft_warehouse_name`
* `ryft_role_name`
Run the following commands as an account admin:
```sql theme={null}
set ryft_username='RYFT';
set ryft_warehouse_size='XSMALL';
set ryft_warehouse_name='RYFT_WH';
set ryft_role_name='RYFT_ROLE';
USE ROLE ACCOUNTADMIN;
CREATE WAREHOUSE IF NOT EXISTS identifier($ryft_warehouse_name) WAREHOUSE_SIZE=$ryft_warehouse_size INITIALLY_SUSPENDED=TRUE
AUTO_SUSPEND = 1 AUTO_RESUME = TRUE;
-- Create Ryft role
CREATE ROLE IF NOT EXISTS identifier($ryft_role_name);
-- Create Ryft user
CREATE USER IF NOT EXISTS identifier($ryft_username) DEFAULT_ROLE=$ryft_role_name TYPE=SERVICE;
GRANT ROLE identifier($ryft_role_name) TO USER identifier($ryft_username);
-- Grant permissions to the WH
GRANT USAGE ON WAREHOUSE identifier($ryft_warehouse_name) TO ROLE identifier($ryft_role_name);
-- Grant privileges to allow access to query history and metadata
GRANT IMPORTED PRIVILEGES ON DATABASE "SNOWFLAKE" TO ROLE identifier($ryft_role_name);
GRANT MONITOR ON ACCOUNT TO ROLE identifier($ryft_role_name);
-- Grant privileges to to audit the current access state
GRANT DATABASE ROLE SNOWFLAKE.SECURITY_VIEWER TO ROLE identifier($ryft_role_name);
```
### Creating a key-pair for the service user
Follow Snowflake's key-pair [authentication guide](https://docs.snowflake.com/en/user-guide/key-pair-auth#generate-the-private-key) to create a private and public key for the service user you just created.
* The private key will be needed during the onboarding.
* Ryft supports using an encrypted or unencrypted private key.
Use this SQL command to assign the public key to the service user in Snowflake:
```jsx theme={null}
ALTER USER identifier($ryft_username) SET RSA_PUBLIC_KEY='';
```
### Providing user credentials during onboarding:
* **User**: The name of the user created, ***RYFT*** by default
* **Private key** and optional **Private key passphrase** for the encrypted private key
* **Account**: This would be the part that precedes *snowflakecomputing.com* in your Snowflake console URL. For example, if you use the URL [*https://xyz12345.us-east-1.snowflakecomputing.com/*](https://xyz12345.us-east-1.snowflakecomputing.com/) to access Snowflake, your account name will be `xyz12345.us-east-1`
* **Organization** - your snowflake organization ID
* **Warehouse**: Please provide the name of the warehouse you chose (default: ***RYFT\_WH***).
* **Role** - Please provide the name of the role you chose (default: **RYFT\_ROLE**)
### IP Access Lists
If you are restricting network access to specific IP addresses, please allow the following IP addresses:
```
18.211.150.201
52.45.62.176
35.174.30.119
```
# Apache Spark
Source: https://docs.ryft.io/integrations/spark
Install the Ryft Spark plugin to capture real-time execution plans and job history, for app visibility and usage-based optimization.
The Ryft Spark plugin listens to Spark events in real-time, which provide detailed information about the execution of Spark jobs, and writes the logs to a dedicated S3 bucket.
### Spark Plugin Configuration
1. Create an S3 bucket in your account that will store Spark event logs, or contact your Ryft representative if you prefer to use a Ryft-managed bucket.
It's best to set a retention policy of at least 7 days.Note: Verify that your Spark execution role has sufficient permissions to write to the chosen bucket.
2. Add the spark plugin dependency to the Spark Application. This is done differently depending on the deployment:
```bash spark-submit {2} theme={null}
spark-submit \
--packages io.ryft:spark-plugin-3.5_2.13:0.3.6
```
```json Spark EMR {6} theme={null}
"configurationOverrides": {
"applicationConfiguration": [
{
"classification": "spark-defaults",
"properties": {
"spark.jars.packages": "io.ryft:spark-plugin-3.5_2.13:0.3.6",
}
}
]
}
```
```bash AWS Glue Spark Job {3} theme={null}
aws glue start-job-run \
--job-name \
--arguments '{"--extra-jars":"s3:///jars/spark-plugin-3.5_2.13-0.3.6.jar"}'
```
```bash spark-submit {2} theme={null}
spark-submit \
--packages io.ryft:spark-plugin-3.3_2.12:0.3.6
```
```json Spark EMR {6} theme={null}
"configurationOverrides": {
"applicationConfiguration": [
{
"classification": "spark-defaults",
"properties": {
"spark.jars.packages": "io.ryft:spark-plugin-3.3_2.12:0.3.6",
}
}
]
}
```
```bash AWS Glue Spark Job {3} theme={null}
aws glue start-job-run \
--job-name \
--arguments '{"--extra-jars":"s3:///jars/spark-plugin-3.3_2.12-0.3.6.jar"}'
```
3. Register the Ryft Plugin and set the `spark.eventLog.ryft.dir` config to the bucket defined
```bash spark-submit {3-4} theme={null}
spark-submit \
--packages io.ryft:spark-plugin-3.5_2.13:0.3.6 \
--conf spark.eventLog.ryft.dir=s3:// \
--conf spark.plugins=io.ryft.spark.RyftSparkEventLogPlugin
```
```json Spark EMR {7-8} theme={null}
"configurationOverrides": {
"applicationConfiguration": [
{
"classification": "spark-defaults",
"properties": {
"spark.jars.packages": "io.ryft:spark-plugin-3.5_2.13:0.3.6",
"spark.eventLog.ryft.dir": "s3:///",
"spark.plugins": "io.ryft.spark.RyftSparkEventLogPlugin"
}
}
]
}
```
```bash AWS Glue Spark Job {3} theme={null}
aws glue start-job-run \
--job-name \
--arguments '{"--extra-jars":"s3:///jars/spark-plugin-3.5_2.13-0.3.6.jar","--conf":"spark.eventLog.ryft.dir=s3:///"}'
```
```bash spark-submit {3-4} theme={null}
spark-submit \
--packages io.ryft:spark-plugin-3.3_2.12:0.3.6 \
--conf spark.eventLog.ryft.dir=s3:// \
--conf spark.plugins=io.ryft.spark.RyftSparkEventLogPlugin
```
```json Spark EMR {7-8} theme={null}
"configurationOverrides": {
"applicationConfiguration": [
{
"classification": "spark-defaults",
"properties": {
"spark.jars.packages": "io.ryft:spark-plugin-3.3_2.12:0.3.6",
"spark.eventLog.ryft.dir": "s3:///",
"spark.plugins": "io.ryft.spark.RyftSparkEventLogPlugin"
}
}
]
}
```
```bash AWS Glue Spark Job {3} theme={null}
aws glue start-job-run \
--job-name \
--arguments '{"--extra-jars":"s3:///jars/spark-plugin-3.3_2.12-0.3.6.jar","--conf":"spark.eventLog.ryft.dir=s3:///"}'
```
## AWS Glue Spark Job Configuration
### Adding the Plugin to Your Spark Session
Configure your Spark session with the Ryft plugin by adding the following configuration:
```python AWS Glue Spark Job theme={null}
import sys
from awsglue.utils import getResolvedOptions
from awsglue.context import GlueContext
from awsglue.job import Job
args = getResolvedOptions(sys.argv, ['JOB_NAME'])
spark = SparkSession.builder \
.config("spark.plugins", "io.ryft.spark.RyftSparkEventLogPlugin") \
.config("spark.eventLog.ryft.dir", "s3://") \
.getOrCreate()
job = Job(GlueContext(spark.sparkContext))
job.init(args['JOB_NAME'], args)
```
```python AWS Glue Spark Job theme={null}
import sys
from awsglue.utils import getResolvedOptions
from awsglue.context import GlueContext
from awsglue.job import Job
args = getResolvedOptions(sys.argv, ['JOB_NAME'])
spark = SparkSession.builder \
.config("spark.plugins", "io.ryft.spark.RyftSparkEventLogPlugin") \
.config("spark.eventLog.ryft.dir", "s3://") \
.getOrCreate()
job = Job(GlueContext(spark.sparkContext))
job.init(args['JOB_NAME'], args)
```
Glue jobs only support a single SparkSession - make sure only one is initialized. Initializing more than one SparkSession can prevent the plugin from being loaded.
### Uploading the Plugin JAR to S3
AWS Glue jobs require the plugin JAR to be available in S3. You can upload it directly from Maven Central using this command:
```bash Upload JAR to S3 theme={null}
curl -L https://repo1.maven.org/maven2/io/ryft/spark-plugin-3.5_2.13/0.3.6/spark-plugin-3.5_2.13-0.3.6.jar | \
aws s3 cp - s3://YOUR_BUCKET_NAME/jars/spark-plugin-3.5_2.13-0.3.6.jar
```
```bash Upload JAR to S3 theme={null}
curl -L https://repo1.maven.org/maven2/io/ryft/spark-plugin-3.3_2.12/0.3.6/spark-plugin-3.3_2.12-0.3.6.jar | \
aws s3 cp - s3://YOUR_BUCKET_NAME/jars/spark-plugin-3.3_2.12-0.3.6.jar
```
Replace `YOUR_BUCKET_NAME` with your actual bucket name. Ensure your Glue job has the necessary IAM permissions to read from this S3 location.
### Configuring the Extra JARs Parameter
The Glue job needs to include the plugin JAR using the `--extra-jars` parameter. This can be configured in several ways:
1. Navigate to **AWS Glue** → **Jobs** → Select your job
2. Go to the **Job details** tab
3. Scroll to **Advanced properties**
4. In the **Job parameters** section, add:
* **Key:** `--extra-jars`
* **Value:** `s3://your-bucket/jars/spark-plugin-3.5_2.13-0.3.6.jar`
1. Navigate to **AWS Glue** → **Jobs** → Select your job
2. Go to the **Job details** tab
3. Scroll to **Advanced properties**
4. In the **Job parameters** section, add:
* **Key:** `--extra-jars`
* **Value:** `s3://your-bucket/jars/spark-plugin-3.3_2.12-0.3.6.jar`
Start a job run with the extra JARs parameter:
```bash Start Job Run theme={null}
aws glue start-job-run \
--job-name \
--arguments '{"--extra-jars":"s3://your-bucket/jars/spark-plugin-3.5_2.13-0.3.6.jar"}'
```
Start a job run with the extra JARs parameter:
```bash Start Job Run theme={null}
aws glue start-job-run \
--job-name \
--arguments '{"--extra-jars":"s3://your-bucket/jars/spark-plugin-3.3_2.12-0.3.6.jar"}'
```
When creating a job programmatically:
```python Create Job with boto3 theme={null}
import boto3
glue_client = boto3.client('glue')
response = glue_client.create_job(
Name='your-job-name',
Role='your-glue-service-role',
Command={
'Name': 'glueetl',
'ScriptLocation': 's3://your-bucket/scripts/your-script.py'
},
DefaultArguments={
'--extra-jars': 's3://your-bucket/jars/spark-plugin-3.5_2.13-0.3.6.jar'
}
)
```
When creating a job programmatically:
```python Create Job with boto3 theme={null}
import boto3
glue_client = boto3.client('glue')
response = glue_client.create_job(
Name='your-job-name',
Role='your-glue-service-role',
Command={
'Name': 'glueetl',
'ScriptLocation': 's3://your-bucket/scripts/your-script.py'
},
DefaultArguments={
'--extra-jars': 's3://your-bucket/jars/spark-plugin-3.3_2.12-0.3.6.jar'
}
)
```
Another way to include the plugin in Java or Scala Spark applications is by packaging it directly into your application's uberjar. This embeds the plugin as a dependency, removing the need to reference the plugin JAR separately at runtime.
## Add the plugin dependency
Add this to your `pom.xml`:
```xml pom.xml theme={null}
io.ryftspark-plugin-3.5_2.130.3.6compile
```
Add this to your `pom.xml`:
```xml pom.xml theme={null}
io.ryftspark-plugin-3.3_2.120.3.6compile
```
## Include in your uberjar
Configure the Maven Shade plugin to include the dependency:
```xml pom.xml theme={null}
org.apache.maven.pluginsmaven-shade-plugin3.6.0io.ryft:spark-plugin-3.5_2.13packageshade
```
```xml pom.xml theme={null}
org.apache.maven.pluginsmaven-shade-plugin3.6.0io.ryft:spark-plugin-3.3_2.12packageshade
```
This approach eliminates the need to configure external JARs in your Spark setup, which simplifies application bootstrap and gives you better control over version conflicts. However, it also means you'll need to recompile your application whenever the plugin is updated.
## Supported Artifacts and Compatibility
## Choosing the Right Plugin Version
We publish two Spark plugin variants. Use the table below to select the one that matches your environment.
| Artifact | Java Version | Spark Version | Scala Versions | Iceberg Version |
| ----------------------- | ------------ | ------------- | -------------- | --------------- |
| `spark-plugin-3.3_2.12` | Java 8+ | Spark 3.3 | 2.12 | 1.2.0+ |
| `spark-plugin-3.3_2.12` | Java 8+ | Spark 3.3 | 2.13 | 1.2.0+ |
| `spark-plugin-3.5_2.12` | Java 17+ | Spark 3.5 | 2.12 | 1.7.1+ |
| `spark-plugin-3.5_2.13` | Java 17+ | Spark 3.5 | 2.13 | 1.7.1+ |
***
### 📌 Notes
**Java Compatibility**
* `3.3_x` plugins are compiled for Java 8 and run on Java 8+
* `3.5_x` plugins require Java 17+
**Iceberg Compatibility**
* Minimum supported version is **Iceberg 1.2.0**
* ⚠️ Using older versions is unsupported
* ✅ For best results, use the latest Iceberg version officially supported by your Spark distribution
**Scala Versions**
* Each plugin is published for **Scala 2.12 and 2.13** - match your Spark distribution’s Scala version
***
### ✅ Recommended Usage
* Use **`spark-plugin-3.3_x`** with **Spark 3.3** and **Iceberg 1.2.0+**
* Use **`spark-plugin-3.5_x`** with **Spark 3.5+** - this is the **preferred and actively maintained** version
### IAM Permissions
Ensure your Glue job's IAM role has permissions to access the S3 bucket containing the JAR file:
```json IAM Policy theme={null}
{
"Version": "2012-10-17",
"Statement": [
{
"Effect": "Allow",
"Action": [
"s3:GetObject"
],
"Resource": "arn:aws:s3:::your-bucket/jars/*"
}
]
}
```
If you are using a Ryft-managed bucket, skip the next steps
### Setup S3 to SQS notifications
1. Create a new SQS queue that will receive notifications on new files created in your S3 bucket.
2. Add the following policy to the queue access policy to enable receiving notifications:
```json theme={null}
{
"Version": "2012-10-17",
"Id": "S3Notifications",
"Statement": [
{
"Sid": "S3Notifications-statement",
"Effect": "Allow",
"Principal": {
"Service": "s3.amazonaws.com"
},
"Action": [
"SQS:SendMessage"
],
"Resource": "",
"Condition": {
"ArnLike": {
"aws:SourceArn": "arn:aws:s3:*:*:"
},
"StringEquals": {
"aws:SourceAccount": ""
}
}
}
]
}
```
1. [Configure S3 notifications](https://docs.aws.amazon.com/AmazonS3/latest/userguide/enable-event-notifications.html) for new files created (choose "All object create events") in the event logs bucket to be sent to the newly created SQS.
### **Add Ryft access policy to S3 and SQS**
Add the following access policy to the **Ryft-ControlPlaneRole** you already created, to allow reading notifications from the queue.
1. IAM → Roles → Search for "**Ryft-ControlPlaneRole"** (or the name you used)
2. Add permissions → Create inline policy → Select the **JSON** tab
3. Add the following policy, fill in the bucket and the queue parameters
```jsx theme={null}
{
"Version": "2012-10-17",
"Statement": [
{
"Sid": "AllowSparkEventLogAccess",
"Effect": "Allow",
"Action": [
"s3:ListBucket",
"s3:GetObject"
],
"Resource": [
"arn:aws:s3:::/test/*",
"arn:aws:s3:::"
]
},
{
"Sid": "SparkEventLogsSqsAccess",
"Effect": "Allow",
"Action": [
"sqs:ReceiveMessage",
"sqs:DeleteMessage"
],
"Resource": ""
}
]
}
```
You are done! Locate the URL of the queue you just created and provide it to Ryft, we will now finish setting up the integration.
The URL should look similar to: `https://sqs.us-east-1.amazonaws.com//`
# Starburst Galaxy
Source: https://docs.ryft.io/integrations/starburst-galaxy
Connect Starburst Galaxy to Ryft to sync query history, tables, and views.
### Prerequisites
Before beginning the setup, make sure you have the following privileges (usually, you should reach out to an administrator with access to the `accountadmin` role):
* `Manage service account`
* `Create role`
* `Manage security`
* `Create cluster`
***
## Step 1: Creating the Role
**1.** Navigate to **Admin** > **Access** > **Roles and privileges** in the Starburst Galaxy sidebar
**2.** Click **Add role** to create a new role
**3.** Configure the role:
* **Role name**: `ryft_access`
* Click **Add role**
***
## Step 2: Granting Role Access
In this step, you will configure Ryft’s role to access Starburst metadata and logs, namely:
* Tables and views schema information
* Query logs
**1.** Click on the **ryft\_access** role, then navigate to the **Policies** tab
**2.** Click **Add policy** to create a new policy
**3.** Under "Define policy details", set the policy name and description:
* **Policy name**: `ryft_access_policy`
* **Description**: Access for Ryft's Starburst integration
* Click **Next**
**4.** Under "Define policy scopes", select the data scope for the following metadata tables:
* **Catalogs**: Select `galaxy_telemetry`
* **Schemas**: Select the following schemas:
* `galaxy_telemetry.information_schema`
* `galaxy_telemetry.public`
* **Tables**: Select the following tables:
* `galaxy_telemetry.information_schema.schemata`
* `galaxy_telemetry.information_schema.tables`
* `galaxy_telemetry.information_schema.columns`
* `galaxy_telemetry.information_schema.views`
* `galaxy_telemetry.public.query_history`
* Choose **Match everything in scope**
* Click **Next**
**5.** Configure access permissions:
* **Grant**: ALLOW
* **Privileges**: Select (for read access)
* Click **Create policy**
***
**6.** Navigate to the **Privileges** tab and choose **Add privilege**
**7.** Choose the **Account** tab and click on **View all query history**
**8.** Press **Save privileges**
## Step 3: Creating the Service Account
**1.** Navigate to **Admin** > **Access** > **Service accounts**
**2.** Click **Create new service account**
**3.** Configure the service account:
* **User name**: `ryft-access-sa`
* **Default role**: `ryft_access`
* Check **Generate password**
* Click **Create**
**4.** Set a password description (e.g., `ryft-sa-password`) and click **Generate password**
**Important**: Save the generated password immediately. You will not be able to view it again after this step. It needs to be shared with your Ryft representative after initial setup is complete.
**5.** Note the **username** of the new service account, to share it with Ryft later
***
## Step 4: Creating the Cluster
**1.** Navigate to **Admin** > **Clusters** in the sidebar and click **Create cluster**
**2.** Under "Create new cluster", set up the following configuration:
* **Name**: `ryft-access-cluster`
* **Catalogs**: Select `galaxy_telemetry (us-east-1)`
* **Cloud provider region**: AWS US East (N. Virginia)
* **Access**: Select the `ryft_access` role
* **Execution mode**: Standard
* **Auto suspend**: 5 Minutes
* **Autoscale cluster workers**: Enable with range 1-4 workers
* Click **Create cluster**
`us-east-1` is the region of the `galaxy_telemetry` catalog, regardless of where your lake is stored.
***
## Step 5: Getting Connection Information
Once your cluster is configured, you will need to share its connection details with Ryft.
**1.** Click on the new cluster’s name and then "Partner Connect"
**2.** Click on “Connection info” and then copy the connection information from the `Port` and `Host` blocks.
***
## Step 6: Extract the Catalog name that is used to access external Iceberg tables
Click on "Catalogs" tab (Under "Data" section), and locate the catalog that is connected to your Iceberg external catalog.
Provide the catalog name (from the "name" column) to Ryft.
**Setup Complete!**
In order to finalize the integration with Starburst Galaxy, share the following information with Ryft:
* Ryft’s service account **username** and **password** from step **3**
* Ryft’s cluster **host** and **port** from step **5**
* Starburst **catalog name** for the catalog that is used to access external Iceberg tables from step **6**
# StarRocks
Source: https://docs.ryft.io/integrations/starrocks
Connect StarRocks to Ryft to sync Iceberg query history and table metadata.
StarRocks integration is currently in private preview.
For more details, contact Ryft Support at [support@ryft.io](mailto:support@ryft.io).
# Trino
Source: https://docs.ryft.io/integrations/trino
Connect Trino to Ryft by configuring the HTTP event listener on your coordinator, for full query visibility and usage-based optimizations.
### Get your Ryft Ingest Token
Ryft will provide a unique ingest token for your Trino instance. This token is required to authenticate the HTTP event listener that sends query information to Ryft.
### Configure Trino event listener
1. To integrate Trino query information to Ryft, you will need to configure an HTTP event listener.
On your Trino coordinator node, edit the config file of your event listeners. By default it is in `/etc/trino/event-listener.properties`. The required content of the file should look like the following:
```bash{5} theme={null}
### HTTP
event-listener.name=http
http-event-listener.log-completed=true
http-event-listener.connect-ingest-uri=https://ingest.ryft.io/ingest/trino/queries
http-event-listener.connect-http-headers=x-ryft-token:
```
Make sure `http-event-listener.log-completed` is set to true, and to replace `` with the one provided by Ryft.
2. Restart the coordinator node to apply changes.
Make sure outbound connectivity is enabled to `ingest.ryft.io` on port `443` (HTTPS) for the HTTP event listener to work correctly.
# Welcome to Ryft
Source: https://docs.ryft.io/introduction
Ryft is the Intelligent Iceberg Management Platform - continuously monitoring, managing and optimizing your data.
# Lake Formation
Source: https://docs.ryft.io/lakehouse-management/access-management/lake-formation
## Setup
Lake Formation permissions are managed across both Ryft planes. The control plane reads existing LF grants to audit the current access state, while the data plane applies grants on Ryft's behalf.
The required IAM permissions for both roles are provisioned as part of the initial Ryft deployment. If access management was not enabled at that time, contact your Ryft representative - they will provide a CloudFormation stack to update your existing control plane and data plane roles with the necessary permissions.
## Prerequisites
### IdP Group to IAM Role Mapping
Lake Formation enforces access using IAM roles. For Ryft to apply policies defined in terms of IdP groups, a mapping between IdP groups and IAM roles must be established. Ryft supports two approaches:
**AWS IAM Identity Center (recommended)**
Ryft can automatically sync group-to-role mappings from AWS IAM Identity Center. This supports `AWSReservedSSO_*` roles, which are automatically created by Identity Center and mapped to permission sets. Enabling this requires read access to your AWS management account.
**Custom static mapping**
If Identity Center is not available, you can provide a static mapping of IdP groups to IAM roles. Ryft uses this mapping at policy enforcement time to resolve which IAM role a given group corresponds to.
Contact your Ryft representative to configure either option.
# Access Management
Source: https://docs.ryft.io/lakehouse-management/access-management/overview
Ryft provides a single control plane for managing permissions across all query engines that read and write your Iceberg tables - Trino, Snowflake, Athena, and others.
Rather than configuring grants separately in each engine, you define access policies once in Ryft - in terms of IdP groups, not individual users. Ryft continuously reconciles these policies with every connected engine, keeping access controls consistent as your data and teams evolve.
## Policy Scope
* **Catalog** - applies to all tables (including future ones) within that catalog
* **Namespace** - applies to all tables (including future ones) within that namespace and expands catalog-level policies
* **Table** - applies to a specific table and expands namespace-level policies
## Security Considerations
All grant and policy changes are applied exclusively through the Ryft data plane. The control plane only reads the current state to detect drift - it never writes to any engine directly.
Access management is currently additive-only - it applies grants defined in its policies but does not revoke grants that were set outside of Ryft. Engine-native permissions configured independently remain in place.
# Polaris
Source: https://docs.ryft.io/lakehouse-management/access-management/polaris
## Setup
Follow the [Polaris catalog integration setup](/integrations/polaris-catalog) to connect your catalog to Ryft. Then, create a dedicated service principal for access management with [`CATALOG_MANAGE_ACCESS`](https://polaris.apache.org/in-dev/unreleased/managing-security/access-control/#catalog-privileges) on the catalog - this allows Ryft to read and manage catalog role assignments.
Once created, share the following with your Ryft representative:
* **Client ID** and **Client Secret** of the new service principal
* **Warehouse** (Catalog name)
## Supported Engines
The following query engines support per-user identity forwarding to Polaris:
* [Trino](/lakehouse-management/access-management/trino)
* [Starburst](/lakehouse-management/access-management/starburst)
## Prerequisites
### Principal Roles & Principals Sync
Polaris uses **principal roles** to represent groups - each principal role maps to a corresponding IdP group. For Ryft to enforce access policies, the principal roles referenced by those policies must already exist in Polaris, mapped to any **principals** (users) in sync with the IdP.
# Snowflake
Source: https://docs.ryft.io/lakehouse-management/access-management/snowflake
## Setup
### Grant Metadata Access
The [Snowflake integration setup](/integrations/snowflake#creating-the-user-and-role) already includes the required `SNOWFLAKE.SECURITY_VIEWER` grant for reading role and grant metadata. Make sure the integration is configured before continuing.
### Grants Management
The data plane applies grants in Snowflake on Ryft's behalf, using [Workload Identity Federation (WIF)](https://docs.snowflake.com/en/user-guide/workload-identity-federation) - a passwordless mechanism that allows the Ryft AWS IAM role to authenticate directly to Snowflake without storing credentials.
Please use the SQL snippet below to create the role. Before using the snippet, you will need to populate the following parameters:
* `ryft_role_manager_name` - name for the Ryft grant management role
* `ryft_wif_username` - name for the WIF service user
* `ryft_aws_role_arn` - ARN of the Ryft data plane IAM role (provided by Ryft)
Run the following commands as an account admin:
```sql theme={null}
set ryft_role_manager_name = 'RYFT_ROLE_MANAGER';
set ryft_wif_username = 'RYFT_WIF_USER';
set ryft_aws_role_arn = ':role/Ryft-DataPlaneRole>';
USE ROLE ACCOUNTADMIN;
CREATE ROLE IF NOT EXISTS identifier($ryft_role_manager_name);
GRANT MANAGE GRANTS ON ACCOUNT TO ROLE identifier($ryft_role_manager_name);
GRANT CREATE ROLE ON ACCOUNT TO ROLE identifier($ryft_role_manager_name);
CREATE USER identifier($ryft_wif_username)
WORKLOAD_IDENTITY = (TYPE = AWS ARN = $ryft_aws_role_arn)
TYPE = SERVICE
DEFAULT_ROLE = PUBLIC;
GRANT ROLE identifier($ryft_role_manager_name) TO USER identifier($ryft_wif_username);
ALTER USER identifier($ryft_wif_username) SET DEFAULT_ROLE = $ryft_role_manager_name;
```
## Prerequisites
### SCIM Integration
Ryft enforces access policies using IdP group names. For Snowflake to recognize these groups and correctly evaluate group-based role assignments, a SCIM integration must be active between your IdP and Snowflake. This ensures group memberships are synchronized so that the grants Ryft applies take effect as expected. See [Snowflake SCIM documentation](https://docs.snowflake.com/en/user-guide/scim-intro) for setup instructions.
### Network Connectivity
The Ryft data plane Lambda runs in your BYOC AWS account and connects to Snowflake over the network.
**AWS PrivateLink (recommended)**
Keeps all traffic private with no internet exposure. Note that PrivateLink is a [Business Critical](https://docs.snowflake.com/en/user-guide/admin-security-privatelink) feature in Snowflake. To set this up:
1. Enable PrivateLink on your Snowflake account
2. Ryft creates a VPC endpoint in your BYOC account targeting the Snowflake PrivateLink endpoint
If PrivateLink is not an option for your setup, please reach out to your Ryft representative to discuss alternatives.
# Starburst
Source: https://docs.ryft.io/lakehouse-management/access-management/starburst
Starburst access management is currently only supported with the [Polaris catalog](/lakehouse-management/access-management/polaris). RBAC is defined and enforced at the catalog level in Polaris - Starburst acts as a query engine that forwards user identity, while Polaris evaluates the access policies.
## Setup
Starburst supports per-user identity forwarding to the Iceberg REST catalog via OAuth2 passthrough (see [Starburst OAuth2 passthrough docs](https://docs.starburst.io/latest/security/oauth2-passthrough.html)). Two configuration files need to be updated.
First, enable delegated OAuth2 authentication in `config.properties`:
```properties theme={null}
http-server.authentication.type=DELEGATED-OAUTH2
web-ui.authentication.type=DELEGATED-OAUTH2
```
Then configure the Polaris catalog properties file:
```properties theme={null}
connector.name=iceberg
iceberg.catalog.type=rest
iceberg.rest-catalog.uri=/api/catalog
iceberg.rest-catalog.warehouse=
iceberg.rest-catalog.security=OAUTH2
iceberg.rest-catalog.oauth2.server-uri=/api/catalog/v1/oauth/tokens
iceberg.rest-catalog.oauth2.credential=:
iceberg.rest-catalog.oauth2.scope=PRINCIPAL_ROLE:ALL
```
# Trino
Source: https://docs.ryft.io/lakehouse-management/access-management/trino
Trino access management is currently only supported with the [Polaris catalog](/lakehouse-management/access-management/polaris). RBAC is defined and enforced at the catalog level in Polaris - Trino acts as a query engine that forwards user identity, while Polaris evaluates the access policies.
## Setup
To enable per-user identity forwarding, add `iceberg.rest-catalog.session=USER` to your Polaris catalog properties file (`trino/catalog/polaris.properties`). This tells Trino to pass each authenticated user's credentials to Polaris on every query, so access is evaluated per user rather than as the catalog service account (see [Trino REST catalog docs](https://trino.io/docs/current/object-storage/metastores.html#rest-catalog)).
```properties theme={null}
connector.name=iceberg
iceberg.catalog.type=rest
iceberg.rest-catalog.uri=/api/catalog
iceberg.rest-catalog.warehouse=
iceberg.rest-catalog.security=OAUTH2
iceberg.rest-catalog.oauth2.server-uri=/api/catalog/v1/oauth/tokens
iceberg.rest-catalog.oauth2.credential=:
iceberg.rest-catalog.oauth2.scope=PRINCIPAL_ROLE:ALL
iceberg.rest-catalog.session=USER
```
The `session=USER` passthrough has known compatibility issues with some IdPs. If you encounter this, users can manually pass their IdP JWT to trino-cli via `--extra-credential "token="` - Trino's Iceberg REST client will use it directly as a Bearer credential.
# Asset Explorer
Source: https://docs.ryft.io/lakehouse-management/asset-explorer
View all Iceberg tables in one workbench with storage metrics, usage stats, and saved views. Filter, sort, and browse to answer questions about your lakehouse.
### Overview
The **Asset Explorer** provides visibility into all Iceberg tables in your lakehouse in an easy-to-use workbench, including all their properties, storage metrics and usage statistics.
It empowers teams to:
* Analyze usage - learn which tables are very popular and require additional optimizations, and which tables aren't used and can be cleaned up
* Inspect storage utilization - identify who are the largest tables, how much of their storage is used for active snapshots vs. old snapshots, etc.
* Validate configurations - ensure all tables are managed properly, and have the right configurations and properties set up
### Saved Views
The Asset Explorer includes several saved views that can assist in common use cases.
* **New Tables** - all the new tables added to Ryft - useful for monitoring lakehouse usage and validating proper configuration
* **Stale Tables** - all tables that weren't updated in the past month - useful for detecting errors or storage waste
* **Unoptimized Tables** - tables that aren't currently managed by Ryft
### Columns & Filters
The Asset Explorer supports dynamic viewing, filtering and sorting of many fields, allowing you to analyze your lakehouse assets from different angles. You can view, filter and sort by:
* Table configuration, including **Default Partition Keys**, **Default Sort Order**, **Write Strategy**, and any table property defined for the table
* Storage metrics, including **Total Storage**, **Active Storage**, **Average Partition Size** and **Average File Size**
* Usage metrics, including **Query Count** and **Last Queried**
* And many more…
### Additional Capabilities
* **Export** - export filtered results to a CSV file. All exported values are **unformatted** for easier post-processing - durations in milliseconds, sizes in raw bytes, etc.
* **Shareable URLs** - the current view can be shared with team members simply by sending the current URL. All filters, columns and sorts are encoded in the URL.
# Catalog
Source: https://docs.ryft.io/lakehouse-management/catalog
Browse all data assets across catalogs and namespaces. View storage totals, table counts, write activity, and configure table management settings.
The Catalog page provides a centralized view of all data assets across your organization, including Iceberg tables and other types of tables and views from data warehouses.
It allows teams to gain full visibility into their multi-engine lakehouse, making it easier to monitor and manage all of their assets in one place.
## Key Features
* **Catalog Browsing:** View all available tables and views across the lakehouse, in one unified interface.
* **Table Management:** Configure management for individual tables, specific namespaces or entire catalogs - including data compaction, orphan file cleanup, data lifecycle and more. See more under [Managed Tables](/lakehouse-management/table-management/managed-tables).
* **Asset Visibility:** Access essential information for any asset, such as:
* Schema: Details on the columns, column types and column usage.
* Writes: Insights into how and when data is being written to the table.
* Storage: Visibility into table storage, which can be used to monitor infrastructure and utilization.
* Queries: View past queries executed on the asset and understand usage patterns.
## Catalog Overview
For each Iceberg catalog, you can view catalog-level metrics that summarize the health, scale, and activity of all tables in that catalog.
This provides a high-level operational view for platform teams and helps quickly identify hotspots, growth trends, and potential cost or performance issues.
The catalog overview includes:
* **Total Storage:** The total storage footprint of all tables in the catalog across all tracked snapshots.
* **Active Storage:** The portion of storage currently referenced by the latest snapshots. A low active storage ratio may indicate excessive historical data, unexpired snapshots, or suboptimal retention policies.
* **Table Count:** The number of tables in the catalog, including their growth over time.
* **Write Activity:** Aggregated write activity across all tables, highlighting ingestion-heavy catalogs.
## Namespace Overview
The Namespace Overview provides a focused view of all tables within a specific namespace. This is useful for understanding how individual domains, teams, or projects are utilizing the lakehouse.
At the namespace level, you can analyze:
* **Total Storage:** The combined storage footprint of all tables in the namespace.
* **Active Storage:** The portion of storage actively referenced by current snapshots across tables in the namespace.
* **Table Count:** The number of tables in the namespace and how it changes over time.
* **Write Activity:** Aggregated write operations and ingestion volume for all tables in the namespace.
## Table Overview
Each Iceberg table has its own page, providing valuable information to understand table behavior and usage.
The table overview is designed for both day-to-day operations and deeper analysis. It combines metadata from Iceberg with engine-level metrics to help you understand how data is written, stored, queried, and maintained over time. From this page, you can also navigate to management features such as lifecycle policies, compaction, and cleanup.
### Storage
The Storage tab provides detailed insights into the physical layout and storage efficiency of a given Iceberg table. This includes analysis of data file sizes, partition distribution, and the breakdown of different file types.
The following metrics and graphs are available to help understand how storage is used by the table and identify potential inefficiencies:
* **Total Storage:** Shows the total storage footprint of the table over time. This includes all data files across all tracked snapshots.
* **Active Storage:** Displays the percentage of storage actively used by the current snapshot. Lower percentages may indicate data bloat or suboptimal snapshot retention configurations.
* **File Size Distribution:** Shows the distribution of data file sizes in the table. Since Iceberg performs best with optimally sized files, this chart helps detect issues such as small files or oversized files that can degrade performance.
* **Partition Size Distribution:** Shows the size distribution across partitions to identify partition skew, which can affect query performance and parallelism.
* **Partitions Table:** Displays information (file count, last updated, etc.) on all partitions in the table. This can be used to identify partitions with suboptimal sizes, analyze recently updated partitions, and more.
* **Files:** Breaks down the number and size of files by type - data files, metadata files, and delete files - providing visibility into the table’s storage structure and trends.
### Writes
The Writes tab provides deep visibility into the write patterns for a given table. This includes details on the write operations, ingress rate, and ingestion pipeline.
The information in this tab is based on Iceberg metadata, as well as metrics from Spark apps (when available).
The following metrics are currently available:
* **Added Rows:** The total number of rows added to the table in the selected time range, and the average rate in which they're added.
* **Row Count:** The total row count in the table and its growth over time.
* **Added Bytes:** The total number of bytes added to the table in the selected time range, and the average rate in which data is being ingested into the table.
* **Write Operations:** The amount of write operations (i.e. commits) to the table over time, and its breakdown to different Iceberg operation types.
Organizations with the [Ryft Spark integration](/integrations/spark) can also observe different Spark metrics that help analyze ingestion patterns to the table, like batch duration, processing rate, total rows written, etc.
### Snapshots
The Snapshots tab provides full visibility into the snapshot history of an Iceberg table. Each snapshot represents the complete state of the table at a specific point in time.
This view can help you understand the full timeline of snapshot creation, identify changes between different snapshots, and analyze the status of the table in a specific point of time. You can use the column picker to view additional metrics for each snapshot, including the operation type, the amount of bytes added or removed in that snapshot, etc.
For teams that use Iceberg tags or branches, these can be tracked in the "References" column.
You can also view daily and weekly snapshots created as part of [Snapshot Lifecycle](/lakehouse-management/table-management/snapshot-lifecycle).
# Query Explorer
Source: https://docs.ryft.io/lakehouse-management/query-explorer
Analyze every query across your lakehouse engines. Filter by 50+ fields to identify slow queries, spot usage anomalies, and audit data access patterns.
## Overview
Ryft continuously collects and analyzes the query history from all engines connected to your lakehouse.
This information powers Ryft's **intelligent optimization engine**, allowing it to detect inefficiencies and optimize data layout based on your organization's usage patterns.
Queries are also visible in the Ryft platform, providing deep visibility into how data is accessed and used, and enabling your team to identify inefficiencies and optimize performance.
Explore query activity and trends in the Query Explorer, or directly in the Queries tab of specific tables.
## Query Explorer
The **Query Explorer** provides visibility into every query executed across your lakehouse and query engines. It enables flexible filtering, sorting, and analysis capabilities for troubleshooting, performance tuning, and optimization.
You can use the Query Explorer to:
* Identify slow or expensive queries
* Analyze trends in usage across users, engines and tables
* Spot outliers or anomalies
* Audit data access and monitor governance
### Columns & Filters
The Query Explorer supports dynamic filtering and sorting on many fields, allowing you to slice and dice your query history. You can filter and sort by:
* **Engine** (e.g. Snowflake, Trino, Spark)
* **Owner**
* **Status**
* **Query Duration**
* **Assets Referenced** (tables/views mentioned explicitly in query text)
* **Tables Used** (resolved physical tables accessed)
* **Columns Referenced**
* **Query Tags**
* And many more…
The **Column Picker** lets you customize which fields are visible in the table view. Over 50 fields are available, including engine-specific fields for the query engines that you connect.
### Analysis
The **Analysis** tab lets you go beyond the table view:
* View aggregate stats like total queries, average duration, and total duration
* Group and pivot queries by dimensions like engine, user, team, day, or table
* Export pivot results for external analysis
### Additional Capabilities
* **Export** - export filtered results to a CSV file. All exported values are **unformatted** for easier post-processing - durations in milliseconds, sizes in raw bytes, etc.
* **Shareable URLs** - the current view can be shared with team members simply by sending the current URL. All filters, columns and sorts are encoded in the URL.
# Compliance Cleanup
Source: https://docs.ryft.io/lakehouse-management/table-management/compliance-cleanup
Ensure physical removal of deleted Iceberg data for GDPR and CCPA compliance. Merges delete files with data files to eliminate non-compliant records.
In Iceberg, deleting a record doesn’t guarantee its physical removal from storage. Data may still exist in:
* Old snapshots
* Orphan files
* Delete files (created when writing in Merge-on-Read mode)
**Compliance Cleanup** ensures that deleted data is physically removed from your Iceberg tables, meeting the requirements of GDPR, CCPA, and other data privacy regulations.
It does so by merging delete files and data files, validating that no deleted data remains in the data files.
When enabled together with snapshot expiration and orphan file cleanup, it ensures that deleted data is fully removed from the table's storage footprint.
Enable Compliance Cleanup for any table that requires compliance with data privacy regulations.
# Data Compaction
Source: https://docs.ryft.io/lakehouse-management/table-management/data-compaction
Rewrite small Iceberg data files into larger, optimized ones. Choose automatic self-tuning or customize rewrite strategy when needed.
Ryft's **Data Compaction** is responsible for improving performance, reducing costs, and keeping your Apache Iceberg tables healthy by rewriting small, inefficient data files into larger, more optimal ones.
This helps minimize metadata overhead, improve scan efficiency, and reduce the number of S3 list operations required for query planning and execution.
## Why Compaction Matters
Apache Iceberg is a file-based table format. As data lands in the lake - whether through batch jobs, streaming ingestion, or change data capture (CDC) - it often arrives in small files. Over time, these small files accumulate, leading to:
* Excessive metadata growth (manifest files, manifest lists)
* Expensive planning and scanning (too many file reads per query)
* Unbalanced partition sizes (e.g., one partition with 10 files of 5MB, another with 1 file of 1GB)
* Increased S3 API costs
Compaction solves this by continuously merging small data and delete files into larger, more efficient files.
## Configuration Modes
Ryft supports two modes for compaction: **Automatic** and **Custom**.
### Automatic Compaction (Self-tuning)
In automatic mode, Ryft continuously monitors table and partition health and dynamically decides when and how to run compaction. This includes:
* Detecting file size skew or small file accumulation
* Learning from actual query patterns and write frequencies
* Avoiding unnecessary rewrites for old or rarely queried partitions
* Scaling resources dynamically based on table size and performance needs
There's no need to configure anything - Ryft selects the optimal file size, compaction strategy, and scheduling frequency, based on real-world usage.
This is the recommended mode of operation for most tables, unless you have specific requirements.
### Custom Configuration
Custom mode gives users full control over how compaction is executed. You can configure:
* **Compaction strategy:** bin-packing, sorting or Z-ordering
* **Target file size:** e.g. 256MB, 512MB, 1GB
* **Columns to sort by:** when applying a sorting strategy, you must define the columns to sort by
## Key Features
### Adaptive File Size
For highly compressed data, a common pitfall in Iceberg is that compaction output files are smaller than the defined target size. Ryft automatically monitors output file sizes
and adapts the target file size based on the actual compression ratio of the data. This ensures that compaction jobs produce files that are as close to the target size as possible.
This can result in dramatically fewer files, and reduced storage sizes.
### Partition Selection
Ryft identifies the partitions that would benefit most from compaction and prioritizes them, to improve compaction efficiency and improve query performance.
Ryft tracks:
* Number of files per partition
* Average file size
* Query activity per partition
* Time since last compaction
This ensures that hot partitions are kept in optimal shape, while cold partitions are left untouched unless necessary (e.g. for retention or tiering).
### Delete File Handling
Iceberg supports delete files (position and equality deletes) to enable row-level deletes without rewriting data. However, too many delete files can degrade query performance.
Ryft compaction optimizes delete files by:
* Rewriting deleted rows into new data files to eliminate excessive delete files
* Grouping and consolidating equality deletes into fewer, larger delete files
* Adapting to delete-intensive workloads, ensuring that delete files do not accumulate excessively
### Resource Optimization
Compaction jobs run on optimized compute and automatically allocate CPU and memory resources based on the table size and workload characteristics.
Data which is already optimized does not get rewritten, and unused data will not be prioritized for compaction to reduce resource usage.
Compaction is optimized for performance and cost, and runs in parallel, adapting to the amount of tables available.
### Workload-Aware Compaction
Ryft optimizes compaction differently depending on the ingestion and access pattern, and differentiates between the following workload types:
* **Batch**
* **Streaming**
* **Change Data Capture (CDC)**
* **Delete-intensive workloads**
* **Hybrid**
* **Full table rewrites**
# Data Lifecycle
Source: https://docs.ryft.io/lakehouse-management/table-management/data-lifecycle
Manage Iceberg data retention and storage tiering. Automatically delete old partitions or archive them to cold storage.
**Data Lifecycle** enables data retention and storage tiering capabilities on Iceberg tables.
It operates at the partition level, automatically moving old data to reduce storage costs and support compliance requirements.
Data Lifecycle supports two complementary capabilities:
* Retention - permanently delete old partitions based on age
* Tiering - archive old partitions to low-cost storage
Both capabilities are optional and can be enabled independently or together.
## Requirements
To enable Data Lifecycle on a table, the following requirements must be met:
* The table must be partitioned
* A partition key must be selected to represent partition age. Supported data types are `date`, `time`, `timestamp`, `timestamptz`.
* Lifecycle policies apply only to entire partitions, not individual rows
## Data Retention
**Retention** permanently deletes partitions that exceed a configured age threshold.
Retention is commonly used to:
* Enforce compliance policies (e.g. GDPR, data minimization)
* Cap table growth
* Remove data that is no longer operationally or analytically useful
Partitions are removed from the table once they pass the threshold.
The underlying data files are physically deleted only after all referencing snapshots are expired, based on the table's snapshot lifecycle configuration.
This ensures correctness and consistency across readers and engines.
## Data Tiering
**Tiering** archives old partitions by moving their data files to lower-cost cold storage, reducing storage costs by up to 95%.
The partition's data is not readable until it is restored.
Tiering is designed for data that:
* Must be retained for compliance or resilience reasons
* Is rarely accessed (e.g. once or twice per year)
* Does not need to remain immediately queryable
Tiering behavior and considerations:
* Tiering moves data files only; Iceberg metadata remains intact. Archived partitions continue to appear in Iceberg metadata tables.
* Queries that attempt to read archived partitions will fail until the data is restored.
* Write operations that scan archived data (including updates, merges, and most deletes) will fail.
* Tiering runs daily. If new data is written to a partition that is already archived, it may remain accessible briefly before being archived.
* Restoring older snapshots does not automatically restore archived partitions; archived data remains archived until explicitly restored.
* Shortening the archiving period (e.g. from 240 days to 180 days) does not un-archive partitions that were already archived.
### Restoring Archived Partitions
Archived partitions can be temporarily restored to make their data readable again.
All archived partitions are listed in the Storage tab under Archived Partitions.
When restoring, users can select:
* Which partitions to restore (by time range)
* How long to restore them for (1-30 days)
* Restoration speed
* Standard - takes up to 12 hours to restore. Cloud provider cost: \$0.10 per 1000 objects, \$0.02 per GiB.
* Bulk - takes up to 48 hours to restore. Cloud provider cost: \$0.025 per 1000 objects, \$0.0025 per GiB.
Before confirming the restoration, you'll be able to see the estimated cloud costs for the operation.
This is the total cost of restoring (based on total object count and size), and the cost of temporarily storing the restored version of these files.
Note: restoration is temporary. Restored partitions are automatically re-archived when the restoration period expires.
If restored data needs to remain permanently accessible, the recommended approach is to copy it to a new table.
Example:
```sql theme={null}
CREATE TABLE raw_events_2024_01_to_2024_12
AS SELECT *
FROM raw_events
WHERE day(event_time) >= '2024-01-01'
AND day(event_time) <= '2024-12-31'
```
# Managed Tables
Source: https://docs.ryft.io/lakehouse-management/table-management/managed-tables
Enable Ryft management at the table, namespace, or catalog level. Intelligent self-tuning adapts to workload patterns with full customization options.
Ryft seamlessly manages your Iceberg lakehouse and ensures it stays in top-notch performance and health.
You can enable management at any level — individual tables, namespaces, or entire catalogs — and Ryft will handle maintenance and optimization automatically. Customize settings where needed, and inherit smart defaults everywhere else.
### Intelligent Self-Tuning
Ryft continuously adapts to real-time workload patterns, ensuring that table structure and performance remain optimal as data evolves, without manual work.
By dynamically adjusting compaction operations and metadata maintenance based on usage patterns, Ryft minimizes performance bottlenecks and maximizes query efficiency.
### Customization & Visibility
All management settings are fully customizable, so teams that require fine-tuned adjustments can configure parameters to match specific workload needs.
Ryft also provides detailed visibility into management operations, allowing you to monitor the impact on query performance and resource utilization.
## Management Scope
Management can be enabled at three levels, with settings flowing down from broader to narrower scopes:
### Catalog
Enabling management at the catalog level applies to **all current and future tables in the catalog**.
This is the simplest way to get started - one action covers your entire catalog.
### Namespace
Enabling management at the namespace level applies to **all current and future tables in the selected namespace**.
Namespace-level settings take precedence over catalog-level settings.
### Table
You can enable or customize management for a **specific table**.
Table-level settings always take precedence over namespace-level and catalog-level settings.
### Example
* Enable management at the **catalog** level with automatic settings across the board.
* Override **Snapshot Lifecycle** settings for the `bronze` namespace to use a longer retention window.
* Override **Data Compaction** for the `bronze.enriched_events` table to use a custom sort strategy.
* **Result:** `bronze.enriched_events` uses custom compaction but inherits all other settings from the `bronze` namespace. The `bronze` namespace uses custom snapshot lifecycle but inherits everything else from the catalog. All other tables use the catalog defaults.
As seen in the example above, overrides are granular — you can override individual settings while inheriting the rest from the parent scope.
You can revert an override by resetting the setting to inherit from the parent.
## Management Settings
Each managed scope includes configuration for the following procedures:
* [**Data Compaction**](/lakehouse-management/table-management/data-compaction) — merges small files into larger, more efficient ones
* [**Snapshot Lifecycle**](/lakehouse-management/table-management/snapshot-lifecycle) — manages snapshot retention, recovery points, and expiration
* [**Orphan File Cleanup**](/lakehouse-management/table-management/orphan-files) — removes unreferenced files from storage
* [**Compliance Cleanup**](/lakehouse-management/table-management/compliance-cleanup) — physically applies pending deletes to data files
* [**Data Lifecycle**](/lakehouse-management/table-management/data-lifecycle) — manages data retention and tiering policies
## Using Managed Tables
Enable management by clicking "Enable Management" on any table, namespace, or catalog page. You can accept the automatic defaults or customize settings before applying.
Once enabled, you can manage and monitor your tables through:
* **Management tab** — view and configure settings, and track recent runs for any table, namespace, or catalog.
* **Tables tab** — available on namespace and catalog pages, showing all tables with their management status.
* **Asset Explorer** — filter and sort tables by management status across your entire lakehouse.
# Orphan File Cleanup
Source: https://docs.ryft.io/lakehouse-management/table-management/orphan-files
Automatically scan for and remove unreferenced Iceberg files left by failed ingestion or expired snapshots. Reduces storage costs safely.
Iceberg tables can accumulate "orphan files" - files that are no longer necessary for the proper functioning of the table but remain in storage, leading to increased costs over time.
Common causes of orphan files include:
* Files that are no longer referenced by any table snapshot.
* Files that were written to storage but not committed due to ingestion failures.
* Other scenarios where unreferenced data persists in storage.
**Orphan File Cleanup** periodically scans and removes orphan files from the table, helping to reduce storage costs.
This process does not affect any data associated with the table.
# Snapshot Lifecycle
Source: https://docs.ryft.io/lakehouse-management/table-management/snapshot-lifecycle
Manage Iceberg snapshots end-to-end with automated expiration, daily and weekly recovery points, and safe restore. Control retention and storage growth.
Apache Iceberg tables maintain a complete history of changes via snapshots - each representing a consistent view of the table at a specific point in time. Snapshots enable time travel, rollback, and incremental processing, but without structured lifecycle management they quickly become expensive and difficult to manage.
**Snapshot Lifecycle** is Ryft’s unified framework for managing Iceberg snapshots end-to-end. It combines automated expiration, calendar-aware recovery points (daily / weekly snapshots), and a first-class restore experience - so you always have meaningful recovery points without unbounded storage growth.
Snapshot Lifecycle includes three tightly integrated capabilities:
* **Intelligent expiration** — Automatically removes snapshots that are no longer useful
* **Structured recovery points** — Preserves daily and weekly snapshots for reliable historical restores
* **Safe restore experience** — View and restore tables to recovery points with a single action
Together, these replace ad-hoc snapshot retention and rigid expiration policies with a reliable and predictable system.
## Configuring Snapshot Lifecycle
Snapshot Lifecycle manages snapshot cleanup and restore points using a single policy that balances short-term flexibility with longer-term recovery guarantees.
A policy can define:
* A minimum number of snapshots to always retain
* How long to keep **continuous snapshots** (recent, high-resolution history)
* Whether to keep **daily** and **weekly** recovery snapshots (longer-term restore points)
### Continuous snapshots (recent history)
Recent snapshots are retained continuously for a configurable window (for example, the last 12 hours or 3 days).
These are useful for debugging, fast iteration, and short-term rollbacks.
### Daily and weekly recovery snapshots
In addition to continuous retention, Snapshot Lifecycle can preserve structured recovery points over time, such as:
* One snapshot per day for a fixed number of days
* Optional weekly snapshots for longer-term recovery
These snapshots are automatically selected, tagged, and retained so meaningful restore points remain available even as high-frequency snapshots are expired.
### Expiration
Snapshots that do not fall into any active retention window - and are not required to satisfy minimum snapshot guarantees - are safely expired automatically. This keeps storage usage predictable while preserving the recovery points you care about.
## Configuration Modes
Snapshot Lifecycle supports two configuration approaches:
### Automatic (Recommended)
Ryft manages snapshot lifecycle end-to-end using safe defaults and observed table behavior.
This mode is recommended for most tables and requires no manual tuning.
### Custom
For tighter control, users can define explicit snapshot lifecycle rules, including:
* Retention windows for recent continuous snapshot retention
* Daily and weekly snapshot retention periods
* Minimum snapshots to retain
Custom settings can be useful for tables with specific compliance or recovery requirements.
## Considerations and Limitations
* Snapshot Lifecycle uses Iceberg-native snapshot tagging to mark and manage recovery snapshots (daily and weekly). These tags are visible and accessible through any query engine, while Ryft manages expiration and safe restoration.
* Some compute engines (for example, Amazon Data Firehose) do not currently support writing to Iceberg tables with tagged snapshots. For such engines, Snapshot Lifecycle should not be enabled.
* Daily and weekly snapshots are tagged only after the protected period has fully elapsed, based on UTC time.
* Daily and weekly snapshots follow wall-clock time rather than table activity. If a table has no updates during a configured retention window, the corresponding daily or weekly snapshots may be expired.
* Disabling Snapshot Lifecycle removes all lifecycle-managed recovery snapshots. If only a specific retention tier (daily or weekly) is disabled, only snapshots associated with that tier are expired.
# Snapshot Restoration
Source: https://docs.ryft.io/lakehouse-management/table-management/snapshot-restoration
Restore an Iceberg table to a previous snapshot directly from Ryft. Uses native Iceberg semantics with lineage-safe guardrails and cooldown limits.
All tracked snapshots are also restorable from the Ryft platform. You can view all available snapshots in the Snapshots tab for every table, and easily restore the table to a selected snapshot with a single action.
Restoration uses native Iceberg snapshot semantics and is compatible with all engines querying the table.
To keep restores safe and predictable, some guardrails are applied:
* **Lineage-safe restores only** - A snapshot can be restored only if it is an ancestor of the current snapshot. Snapshots outside the current lineage may reflect incompatible schema or metadata changes, so they are visible but not restorable.
* **Restore cooldown** - After a restore completes, Ryft enforces a short cooldown of 10 minutes before another restore can run. This prevents overlapping or accidental repeated restores. Once the cooldown expires, restores are available again.
# Platform Overview
Source: https://docs.ryft.io/platform
Monitor, manage, and optimize your Apache Iceberg lakehouse from a single platform. Reduce costs by up to 50% and accelerate queries up to 10x.
## Lakehouse Management
[Apache Iceberg](https://iceberg.apache.org/) is an open-source table format designed for big data analytics, offering significant advantages over traditional data lake storage.
It provides schema evolution, hidden partitioning, time-travel queries, and ACID transactions, making it a powerful foundation for modern lakehouses.
While Iceberg brings major flexibility and performance benefits, it also introduces challenges in **monitoring, management, and optimization**.
As datasets grow, organizations struggle with small file accumulation, inefficient partitioning, and suboptimal sort orders, leading to slower queries and rising costs.
## The Ryft Platform
Ryft is the Intelligent Iceberg Management Platform, designed to continuously monitor, manage, and optimize your Iceberg Lakehouse.
Ryft provides full visibility into your lakehouse performance and usage, identifying inefficiencies in data layout and query patterns.
It then automatically tunes your environment to improve query performance and reduce costs, ensuring your lakehouse operates at peak efficiency with minimal manual maintenance.
#### With Ryft, data teams can:
* **Monitor and audit** their entire Iceberg lakehouse from a single dashboard.
* **Reduce operational overhead** by automating maintenance & optimization.
* **Accelerate queries by up to 10x** by optimizing table layouts based on query and ingestion patterns.
* **Cut infrastructure costs by up to 50%** by identifying lakehouse waste and automating table maintenance and cleanup procedures.