Shadowfax Quickstart

Before running shadowfax, you need to create a shadowfax.yml and shadowfax_profiles.yml file. The shadowfax.yml lets you configure workspace settings such as environment variables, secrets, and special instructions. The shadowfax_profiles.yml lets you configure the dbt profiles that shadowfax will use to connect to your data. These two files should be placed in the root of your GitHub repository.

shadowfax.yml

Here is an example of a shadowfax.yml file:

setup:
  - echo "Running setup commands"
  - echo "Installing dependencies"

env_vars:
  FIRST_ENV_VAR: first_env_var_value
  SECOND_ENV_VAR: second_env_var_value

secrets:
  - name: MY_FIRST_SECRET
    as: first_secret_env_name
  - name: MY_SECOND_SECRET
    as: second_secret_env_name

dbt_root_path: transforms/dbt-project

instruction: |
  Make sure to add tests to any new models you create.

Let's break down the file:

• setup: The commands that will be run inside the sandboxed workspace before the agent begins.
• env_vars: The environment variables and their values in the workspace.
• secrets: The secrets that will be set as environment variables.
• dbt_root_path: The path to the dbt project. This is useful if your data engineering team has a monorepo setup, with just one subpath for the dbt project.
• instruction: A special instruction that the agent will follow when it is running.

shadowfax_profiles.yml

Here is an example of a shadowfax_profiles.yml file (see example on GitHub):

default:
  target: dev
  outputs:
  dev:
    type: snowflake
    account: "{{ env_var('SNOWFLAKE_ACCOUNT') }}"
    user: "{{ env_var('SNOWFLAKE_USER') }}"
    password: "{{ env_var('SNOWFLAKE_PASSWORD') }}"
    role: "{{ env_var('SNOWFLAKE_ROLE') }}"
    database: "{{ env_var('SNOWFLAKE_DATABASE') }}"
    warehouse: "{{ env_var('SNOWFLAKE_WAREHOUSE') }}"
    schema: "{{ env_var('SNOWFLAKE_SCHEMA') }}"
    threads: 1
    client_session_keep_alive: True

This file is just the dbt profiles.yml file that gets copied into the workspace's /root/.dbt/profiles.yml. You should use environment variables for the credentials, which our system will inject into the workspace. Here are the important environment variables to use:

• SNOWFLAKE_ACCOUNT
• SNOWFLAKE_ROLE
• SNOWFLAKE_DATABASE
• SNOWFLAKE_WAREHOUSE
• SNOWFLAKE_SCHEMA

• SNOWFLAKE_USER
• SNOWFLAKE_PASSWORD

• SNOWFLAKE_USER
• SNOWFLAKE_PRIVATE_KEY

• SNOWFLAKE_USER
• SNOWFLAKE_TOKEN