All the features shipping today are highly requested— I’ve personally fielded many requests from you all for some of these, so I hope you enjoy! The Data Manager is a particularly impactful new feature, but I personally am irrationally excited for the CSV UPLOAD CELL!!!1111
📖 Data Manager public beta
Today we’re opening public beta access to the Data Manager, a metadata tool that’s purpose-built to enhance Hex Magic’s SQL writing skills and custom tune them to your organization’s data.
The Data Manager provides a nice, simple interface for you to access and edit four different kinds of metadata about your data connections:
- Schema, table, and column information from your data warehouse.
- All your dbt documentation, live synced from dbt Cloud or manually added from dbt Core.
- Priority rankings that determine which schemas, tables, and columns should be included, excluded, or prioritized for Magic AI.
- Any additional descriptions or custom metadata you want to provide.
Hex uses this metadata to create vector embeddings to power the semantic search algorithms that are the secret sauce in Magic’s SQL generation engine. Better metadata = better embeddings, and better embeddings = better SQL generation.
The coolest part is that you can provide really custom guidance using normal natural language. Magic is incredibly responsive to things like “only use this table if the prompt explicitly requires raw stripe data, otherwise use fct_orders”, or “this column contains a stringified array of items”. You can essentially educate it as you would a human member of your data team!
I spent about 10 minutes setting up the Data Manager on an internal connection yesterday, and was able to improve SQL generation success rates from 82% to a dazzling 96% on our prompt evaluation suite!
We just published a very detailed blog that introduces the Data Manager and highlights some real examples of using it to tune SQL generation, with specific advice on how to leverage it yourself.
Data Manager is available for workspaces on the Professional plan and above. Workspace admins can find the Data Manager from the bottom left of the homepage sidebar, or at https://app.hex.tech/your-workspace-id/data.
