Skip to main content

Snowpark integration

Hex lets Snowflake users take advantage of Snowpark, directly from your Hex projects.

Snowpark lets users operate on Snowpark DataFrames. Unlike with standard Pandas DataFrames, these are not loaded into the memory of a Hex project. Operations on Snowpark DataFrames are executed on Snowflake’s infrastructure. This enables Hex users to operate on large datasets without requiring additional memory in Hex.

To learn more about Snowpark, visit the Snowpark Developer Guide

Use Snowpark in a Hex project​

Enable Snowpark​

Snowpark can be enabled on both workspace and project-level data connections. To enable Snowpark on a workspace data connection, a Hex Admin will need to update the connection in the Workspace assets page of the Admin panel. To enable Snowpark in a project data connection, select the Snowflake connection from the Data sources section of the left sidebar.

Once you've selected your Snowflake data connection, scroll to the Integrations section, and toggle on Snowpark.

Use the Snowpark image​

Snowpark is only compatible with Python 3.8. To use Snowpark in a Hex project, ensure you’re using the Snowpark (Python 3.8) image. You can select an image when creating a project, or update an existing project’s image from the Compute profile section of the Environment sidebar.

Return a Snowpark DataFrame from a SQL cell​

To return query results as a Snowpark DataFrame, set the Output type to Snowpark in the bottom left corner of a SQL cell.

Note that if this DataFrame is being referenced downstream, changing the output type may cause breaking changes as Snowpark DataFrames are returned to Hex as an object and not a full Pandas DataFrame. Often, an effective way to continue to use the DataFrame is to call the .to_pandas() method, though this may have memory implications for your project.

caution

Snowpark does not support query parameterization, which is required for Jinja templating. Because of this, Jinja syntax is not supported in Snowpark queries.

Create a Snowpark session from a Python cell​

To create a Snowpark session from a Python cell, create a cell with the following code replacing the argument of the get_data_connection method with the name of your connection.

import hextoolkit
hex_snowflake_conn = hextoolkit.get_data_connection('My data connection')
hex_snowpark_session = hex_snowflake_conn.get_snowpark_session()

You can also generate a Python cell with this code from the Data browser menu:

Creating a Snowpark session from the Data browser

On this page