Dremio

Important Capabilities

Capability	Status	Notes
Data Profiling	✅	Optionally enabled via configuration
Detect Deleted Entities	✅	Optionally enabled via stateful_ingestion.remove_stale_metadata
Domains	✅	Supported via the domain config field
Platform Instance	✅	Enabled by default
Table-Level Lineage	✅	Enabled by default

This plugin extracts the following:

• Metadata for databases, schemas, views and tables
• Column types associated with each table
• Table, row, and column statistics via optional SQL profiling
• Lineage information for views and datasets

CLI based Ingestion

Install the Plugin

The dremio source works out of the box with acryl-datahub.

Starter Recipe

Check out the following recipe to get started with ingestion! See below for full configuration options.

For general pointers on writing and running a recipe, see our main recipe guide.

source:
  type: dremio
  config:
    # Coordinates
    hostname: localhost
    port: 9047
    tls: true

    # Credentials
    authentication_method: password
    username: user
    password: pass

    include_query_lineage: True

    source_mappings:
      - platform: s3
        platform_name: samples

    schema_pattern:
      allow:
        - ".*"

sink:
  # sink configs

Config Details

Options

Note that a . is used to denote nested fields in the YAML recipe.

Field	Description
authentication_method string	Authentication method: 'password' or 'PAT' (Personal Access Token) Default: password
collect_pds boolean	Whether to collect physical datasets Default: False
collect_system_tables boolean	Whether to collect Dremio system tables Default: False
disable_certificate_verification boolean	Disable TLS certificate verification Default: False
dremio_cloud_region string	Dremio Cloud region ('US' or 'EMEA')
hostname string	Hostname or IP Address of the Dremio server
include_copy_lineage boolean	Whether to include copy lineage Default: True
include_query_lineage boolean	Whether to include query-based lineage information. Default: False
include_table_rename_lineage boolean	Whether to include table rename lineage Default: True
is_dremio_cloud boolean	Whether this is a Dremio Cloud instance Default: False
max_workers integer	Maximum number of worker threads for parallel processing. Default: 20
password string	Dremio password or Personal Access Token
path_to_certificates string	Path to SSL certificates Default: /vercel/path0/metadata-ingestion/venv/lib/python3....
platform_instance string	Platform instance for the source. Default:
port integer	Port of the Dremio REST API Default: 9047
tls boolean	Whether the Dremio REST API port is encrypted Default: True
username string	Dremio username
env string	Environment to use in namespace when constructing URNs. Default: PROD
schema_pattern AllowDenyPattern	Regex patterns for schemas to filter Default: {'allow': ['.*'], 'deny': [], 'ignoreCase': True}
schema_pattern.ignoreCase boolean	Whether to ignore case sensitivity during pattern matching. Default: True
schema_pattern.allow array	List of regex patterns to include in ingestion Default: ['.*']
schema_pattern.allow.string string
schema_pattern.deny array	List of regex patterns to exclude from ingestion. Default: []
schema_pattern.deny.string string
source_mappings array	Mappings from Dremio sources to DataHub platforms and datasets.
source_mappings.DremioSourceMapping DremioSourceMapping
source_mappings.DremioSourceMapping.platform ❓ string
source_mappings.DremioSourceMapping.platform_name ❓ string
source_mappings.DremioSourceMapping.databaseName string
source_mappings.DremioSourceMapping.dremio_source_type string
source_mappings.DremioSourceMapping.platform_instance string
source_mappings.DremioSourceMapping.rootPath string
source_mappings.DremioSourceMapping.env string	Default: PROD
stateful_ingestion StatefulIngestionConfig	Stateful Ingestion Config
stateful_ingestion.enabled boolean	Whether or not to enable stateful ingest. Default: True if a pipeline_name is set and either a datahub-rest sink or datahub_api is specified, otherwise False Default: False

Schema

The JSONSchema for this configuration is inlined below.

{
  "title": "DremioSourceConfig",
  "description": "Base configuration class for stateful ingestion for source configs to inherit from.",
  "type": "object",
  "properties": {
    "stateful_ingestion": {
      "title": "Stateful Ingestion",
      "description": "Stateful Ingestion Config",
      "allOf": [
        {
          "$ref": "#/definitions/StatefulIngestionConfig"
        }
      ]
    },
    "hostname": {
      "title": "Hostname",
      "description": "Hostname or IP Address of the Dremio server",
      "type": "string"
    },
    "port": {
      "title": "Port",
      "description": "Port of the Dremio REST API",
      "default": 9047,
      "type": "integer"
    },
    "username": {
      "title": "Username",
      "description": "Dremio username",
      "type": "string"
    },
    "authentication_method": {
      "title": "Authentication Method",
      "description": "Authentication method: 'password' or 'PAT' (Personal Access Token)",
      "default": "password",
      "type": "string"
    },
    "password": {
      "title": "Password",
      "description": "Dremio password or Personal Access Token",
      "type": "string"
    },
    "tls": {
      "title": "Tls",
      "description": "Whether the Dremio REST API port is encrypted",
      "default": true,
      "type": "boolean"
    },
    "disable_certificate_verification": {
      "title": "Disable Certificate Verification",
      "description": "Disable TLS certificate verification",
      "default": false,
      "type": "boolean"
    },
    "path_to_certificates": {
      "title": "Path To Certificates",
      "description": "Path to SSL certificates",
      "default": "/vercel/path0/metadata-ingestion/venv/lib/python3.10/site-packages/certifi/cacert.pem",
      "type": "string"
    },
    "is_dremio_cloud": {
      "title": "Is Dremio Cloud",
      "description": "Whether this is a Dremio Cloud instance",
      "default": false,
      "type": "boolean"
    },
    "dremio_cloud_region": {
      "title": "Dremio Cloud Region",
      "description": "Dremio Cloud region ('US' or 'EMEA')",
      "type": "string"
    },
    "env": {
      "title": "Env",
      "description": "Environment to use in namespace when constructing URNs.",
      "default": "PROD",
      "type": "string"
    },
    "platform_instance": {
      "title": "Platform Instance",
      "description": "Platform instance for the source.",
      "default": "",
      "type": "string"
    },
    "source_mappings": {
      "title": "Source Mappings",
      "description": "Mappings from Dremio sources to DataHub platforms and datasets.",
      "type": "array",
      "items": {
        "$ref": "#/definitions/DremioSourceMapping"
      }
    },
    "collect_pds": {
      "title": "Collect Pds",
      "description": "Whether to collect physical datasets",
      "default": false,
      "type": "boolean"
    },
    "collect_system_tables": {
      "title": "Collect System Tables",
      "description": "Whether to collect Dremio system tables",
      "default": false,
      "type": "boolean"
    },
    "schema_pattern": {
      "title": "Schema Pattern",
      "description": "Regex patterns for schemas to filter",
      "default": {
        "allow": [
          ".*"
        ],
        "deny": [],
        "ignoreCase": true
      },
      "allOf": [
        {
          "$ref": "#/definitions/AllowDenyPattern"
        }
      ]
    },
    "max_workers": {
      "title": "Max Workers",
      "description": "Maximum number of worker threads for parallel processing.",
      "default": 20,
      "type": "integer"
    },
    "include_query_lineage": {
      "title": "Include Query Lineage",
      "description": "Whether to include query-based lineage information.",
      "default": false,
      "type": "boolean"
    },
    "include_table_rename_lineage": {
      "title": "Include Table Rename Lineage",
      "description": "Whether to include table rename lineage",
      "default": true,
      "type": "boolean"
    },
    "include_copy_lineage": {
      "title": "Include Copy Lineage",
      "description": "Whether to include copy lineage",
      "default": true,
      "type": "boolean"
    }
  },
  "additionalProperties": false,
  "definitions": {
    "DynamicTypedStateProviderConfig": {
      "title": "DynamicTypedStateProviderConfig",
      "type": "object",
      "properties": {
        "type": {
          "title": "Type",
          "description": "The type of the state provider to use. For DataHub use `datahub`",
          "type": "string"
        },
        "config": {
          "title": "Config",
          "description": "The configuration required for initializing the state provider. Default: The datahub_api config if set at pipeline level. Otherwise, the default DatahubClientConfig. See the defaults (https://github.com/datahub-project/datahub/blob/master/metadata-ingestion/src/datahub/ingestion/graph/client.py#L19).",
          "default": {},
          "type": "object"
        }
      },
      "required": [
        "type"
      ],
      "additionalProperties": false
    },
    "StatefulIngestionConfig": {
      "title": "StatefulIngestionConfig",
      "description": "Basic Stateful Ingestion Specific Configuration for any source.",
      "type": "object",
      "properties": {
        "enabled": {
          "title": "Enabled",
          "description": "Whether or not to enable stateful ingest. Default: True if a pipeline_name is set and either a datahub-rest sink or `datahub_api` is specified, otherwise False",
          "default": false,
          "type": "boolean"
        }
      },
      "additionalProperties": false
    },
    "DremioSourceMapping": {
      "title": "DremioSourceMapping",
      "type": "object",
      "properties": {
        "platform": {
          "title": "Platform",
          "type": "string"
        },
        "platform_name": {
          "title": "Platform Name",
          "type": "string"
        },
        "platform_instance": {
          "title": "Platform Instance",
          "type": "string"
        },
        "dremio_source_type": {
          "title": "Dremio Source Type",
          "type": "string"
        },
        "env": {
          "title": "Env",
          "default": "PROD",
          "type": "string"
        },
        "rootPath": {
          "title": "Rootpath",
          "type": "string"
        },
        "databaseName": {
          "title": "Databasename",
          "type": "string"
        }
      },
      "required": [
        "platform",
        "platform_name"
      ],
      "additionalProperties": false
    },
    "AllowDenyPattern": {
      "title": "AllowDenyPattern",
      "description": "A class to store allow deny regexes",
      "type": "object",
      "properties": {
        "allow": {
          "title": "Allow",
          "description": "List of regex patterns to include in ingestion",
          "default": [
            ".*"
          ],
          "type": "array",
          "items": {
            "type": "string"
          }
        },
        "deny": {
          "title": "Deny",
          "description": "List of regex patterns to exclude from ingestion.",
          "default": [],
          "type": "array",
          "items": {
            "type": "string"
          }
        },
        "ignoreCase": {
          "title": "Ignorecase",
          "description": "Whether to ignore case sensitivity during pattern matching.",
          "default": true,
          "type": "boolean"
        }
      },
      "additionalProperties": false
    }
  }
}

Code Coordinates

• Class Name: datahub.ingestion.source.dremio.dremio_source.DremioSource
• Browse on GitHub

Questions

If you've got any questions on configuring ingestion for Dremio, feel free to ping us on our Slack.