Changelog

Today we’re making some updates to our Magic AI assistant that make it more powerful, intuitive, and faster 🤖.

The first version of Magic has been available for almost a year now, and we’re excited about how useful it’s been for our users. Thousands of people each week are using these features to get more done with data: editing complex SQL queries, debugging confusing Python code, and speeding up the most tedious parts of their jobs (cough pandas cough regex).

This round of updates makes these AI tools even more accurate, updates and polishes their UX, and introduces some powerful new functionality. Check out the blog for more (and higher quality) demos, watch this 3 minute video summary, or read on for the quick notes!

➕ Generate mode​

Magic can now generate multiple cells at a time in your notebook, chaining together SQL, Python, and Chart cells to answer complex questions or kickstart a new analysis.

It’s easy to invoke this from anywhere: just hit Cmd+G, use ↓ from an edit mode prompt bar, or click the “Add with Magic” button between cells.

New cells are all created as drafts, so you can inspect, edit, and validate AI generated code before accepting it. Once accepted, the new cells seamlessly integrate into your notebook.

💬 More powerful, streamlined prompting​

To support new cell generation and more complex instructions, we’ve rebuilt the prompt bar, unifying all Magic actions into one simple interface.

You can activate Magic on a cell with Cmd+Shift+M, and seamlessly switch between editing existing cells or generating new cells with the ↑ and ↓ keys.

The new design also supports more complex multi-line instructions and gives more flexibility to auto-fixes, letting you provide specific instructions instead of relying on Magic to guess what kind of fix you'd like to make.

🎯 Mentioning data​

The prompt bar now supports "@ mentioning" datasets to specify what resources it should use.

In any prompt bar, you can hit @ and then keep typing to search for resources.

This is a simple addition, with a huge impact on prompting efficiency and accuracy. Use this to point at a specific database table (handy if you're off-roading and using some unusual, undocumented resources) or if you're in a complex project, to direct Hex towards a particular dataframe.

🤓 Accuracy and context​

In addition to these big changes, we’ve also been putting in lots of work on our state-of-the-art retrieval pipelines, and seen steady improvement in Hex’s ability to accurately generate SQL queries. But it’s important to note that no matter how good our engineering is, your context is critical! The more metadata you can provide about your tables and columns, the better Hex can effectively prompt the model and maximize completion quality.

You can do this in three main ways:

• dbt Docs: if you’re using dbt Cloud, all you have to do is update the metadata in your models, and it’ll automatically flow into Hex and be used to help inform query generation.
• Support for warehouse metadata: ditto if you update information about your columns and tables in Snowflake, BigQuery, or Redshift.
• via Hex’s built-in Data Manager, which lets you add and edit metadata, as well as promote and exclude schemas and tables.

Alright, that's what's hot off the presses! I can't wait to hear your feedback on these updates, and to see what you build with a little help from Magic 🧙.

🏡 New homepage​

We’ve rebuilt the Hex homepage to make finding important and frequently accessed projects easy.

An early beta user put it best: “all the stuff I wanted to access easier is now easier to access”. Mission accomplished 🫡.

The new homepage has 5 sections. Watch this quick loom tour, or read about them below:

• Jump back in aka recently viewed projects. This is basically what the entire old homepage was.
• Popular projects this week. This is across your entire workspace, so you can easily find the most important Hex apps for your company. This is my new favorite section.
• Collections with new additions. Any collection that has a new project in it will show up here.
• Recently published. This is a “news feed” of recently published projects from your workspace.
• The Sidebar. We’ve moved Favorites to the sidebar, for both Projects and Collections. They’re easier to access, and out of the way of all the other sections.

You’ll also notice that each project in these sections displays a lot more metadata! Often a title isn’t enough to know if this is the thing you’re looking for. How many views does it have? Are they recent views? When was it published? Who made it?

These are all questions that the new project cards answer at a glance:

There’s a ton of careful thought that went into this redesign, and it’s an extensible framework we’re going to continue building and iterating on. Read more about the updates in the blog!

🗣️ Project @ mentions​

You can now @ mention projects from comments and text cells.

This is a small but mighty feature that enables all kinds of useful organizational patterns. We are already using it internally to create some cross-linking between our core company metrics apps, and it’s a delightful experience to reference a project right in a comment.

📊 Chart fixes galore!​

And I mean galore. Here’s a sample of them, which is limited only because my cmd+c & cmd+v fingers were getting tired.

• You can now filter on x-axes that have dots in column names
• Line charts can be grouped by double-quoted column names
• Visual filtering now works when there are brackets in column names
• Charts now show correct dates when column names contained dots
• Tooltips no longer display NaN if a column name contains dots
• Line charts with odd characters in column names now work
• A large and diverse variety of manual tooltip related issues have been fixed
• Color attributes now appear in tooltips even if they’re used in aggregations
• Tooltips render properly when there are 2 series based on the same column
• Tooltips should not fall back to UNIX timestamps when you show a non-axis column
• Manual tooltips now support aggregated columns for double-pivoted-dual-axis charts.
• Custom ordering of legend items now works as expected
• Much, much more.

Charts have a lot of surface area! There’s so many config combinations, and input data can be incredibly complex.

We’re hard at work expanding chart functionality (check out facets if you haven’t yet!), but we’re equally devoted to making sure that basic chart functionality is… functional, no matter what data or chart config you throw at it.

Other improvements​

• Download as PDF now respects your current theme in the download.
• If you’ve ever noticed strange duplicated outputs in a Hex project that went away upon refresh— congratulations! You saw a ghost that has now been captured, disciplined, and banished back to the astral plane. You will never see it again.

Fresh round of Hex features plummeting towards you at relativistic speeds. (wait, does that mean they’re coming really fast, or slow? I guess it depends on your frame of reference… 😎)

💎 Trellis charts​

Chart cells now support faceting, so you can build beautiful, information-dense trellis charts with just a few clicks.

You might know these as trellis charts, small multiples, faceted charts, lattice charts, or just “that thing where the viz is split into a bunch of plots”. Whatever you call them, the premise is pretty simple: Choose a column to "facet" by, and your plot becomes a grid of subplots, broken out by that column.

This lets you display a lot more information without compromising— and often actually enhancing— the readability of a visualization. Done right, you can display 4 (or even 5, if you want to get really loco) dimensions of data at once.

Here's a really quick example of improving a cluttered, confusing viz by turning it into a trellis chart. The initial state here makes it really tough to notice any year-over-year patterns in item sales:

Breaking it out as a trellis chart faceted by year, it's immediately obvious that there's a crazy spike in two specific menu items in 2022. The entire chart is more readable, but the patterns in particular really leap out:

Sold yet? No? Trellis charts are really easy to configure too! Here's a 9 second gif of me going from screenshot 1 to screenshot 2 there, in real time. I count only 5 clicks!

And if you're still hungry for EVEN MORE trellis chart content... Here's a 3 minute video of me walking through the thought process behind that 9 second gif, which of course itself is an elongation of those two screenshots:

🗃️ Google Drive integration​

Access Google Drive files directly from Hex, just like a locally uploaded file.

Once a Google Drive connection is configured in Workspace Assets settings, you’ll be able to browse and import files right from the Files sidebar of any Hex project.

You’re also able to natively write files to GDrive without any configuration or manual upload. Using a standard file.write() or dataframe.to_csv() with an external-files/googledrive/<your_drive>/<your_filename.csv> path will just work.

An important note: Because this is a workspace level connection, it's not possible to connect to a user's entire "My Drive". Instead, a Subdirectory ID is required to connect to a specific folder in Drive. To find your Subdirectory ID, navigate to the directory in Google Drive, and copy the ID after folders/ in the URL.

See the docs for more.

🩺 Magic for PHI ​

HIPAA compliant customers of Hex can now make full use of our Magic AI features.

Hex’s Magic AI (Artifical Intelligence) features are powered by OpenAI, who we now have a BAA (Business Associate Agreement) in place with that allows them to process PHI (Protected Health Information) data for our customers.

This means customers of Hex that must maintain HIPAA (Health Insurance Portability and Accountability Act) compliance are now able to use Magic!

Workspace admins can enable Magic and configure the Data Manager from the Workspace Settings > Magic page. For more about our commitments to data privacy and security, visit trust.hex.tech.

Other improvements​

• We’ve added some handy new keyboard shortcuts to make working in large projects easier. Go try these right now so you bake in the muscle memory! I promise they will make you faster at using Hex.
  • Option + ⬆︎ and Option + ⬇︎: Jump to the top / bottom of the notebook
  • Cmd + [ and Cmd + ]: Step forward / backward through cells in the order you most recently edited them.
  • You can also access these from the command palette (Cmd + P)
• The Bigquery DataFrames (bigframes) package is now installed by default, and users of BigQuery can create a new BigFrames session using hextoolkit. Stay tuned for more here 😉.

All the features shipping today are highly requested— I’ve personally fielded many requests from you all for some of these, so I hope you enjoy! The Data Manager is a particularly impactful new feature, but I personally am irrationally excited for the CSV UPLOAD CELL!!!1111

📖 Data Manager public beta​

Today we’re opening public beta access to the Data Manager, a metadata tool that’s purpose-built to enhance Hex Magic’s SQL writing skills and custom tune them to your organization’s data.

The Data Manager provides a nice, simple interface for you to access and edit four different kinds of metadata about your data connections:

• Schema, table, and column information from your data warehouse.
• All your dbt documentation, live synced from dbt Cloud or manually added from dbt Core.
• Priority rankings that determine which schemas, tables, and columns should be included, excluded, or prioritized for Magic AI.
• Any additional descriptions or custom metadata you want to provide.

Hex uses this metadata to create vector embeddings to power the semantic search algorithms that are the secret sauce in Magic’s SQL generation engine. Better metadata = better embeddings, and better embeddings = better SQL generation.

The coolest part is that you can provide really custom guidance using normal natural language. Magic is incredibly responsive to things like “only use this table if the prompt explicitly requires raw stripe data, otherwise use fct_orders”, or “this column contains a stringified array of items”. You can essentially educate it as you would a human member of your data team!

I spent about 10 minutes setting up the Data Manager on an internal connection yesterday, and was able to improve SQL generation success rates from 82% to a dazzling 96% on our prompt evaluation suite!

We just published a very detailed blog that introduces the Data Manager and highlights some real examples of using it to tune SQL generation, with specific advice on how to leverage it yourself.

Data Manager is available for workspaces on the Professional plan and above. Workspace admins can find the Data Manager from the bottom left of the homepage sidebar, or at https://app.hex.tech/your-workspace-id/data.

🆙 CSV upload input​

The CSV upload input parameter… lets app users upload CSV files! Need I say more?

It also automatically parses the uploaded file and returns a DataFrame for use in downstream cells 🙂.

I’m willfully choosing to not look at the data and am just going off of ✨vibes ✨ here, but I think this has to be one of, if not the most requested feature in our backlog.

And I can see why! Letting end users of apps upload their own data lets you do some pretty cool stuff. A neat hack I threw together: If you pair it with a table input, you can even give app users a full spreadsheet editing experience that culminates in visualization, or data transformation / writeback.

Learn more in the docs.

💼 Presentation mode​

Pop your app into Presentation mode to get a full screen experience that splits tabbed apps into individual slides that remain fully interactive.

For years now, we’ve run our regular metrics reviews and Board of Directors meetings out of Hex apps, rather than slide decks. Scrolling through an app always worked just okay. Tabs were a big improvement, but for a real deal meeting, you can’t beat a good old fashioned slide deck.

Unless, of course, you turn an interactive data app into a slide deck, while maintaining the ability to interact with and explore data live, right from the slides 😉. Next time you’re presenting complex findings or doing a metrics review, split your app into tabs and give presentation mode a spin!

Presentation mode is available for workspaces on the Professional plan and above.

🧟 Auto Archive​

Workspace admins can now configure auto-archiving of projects based on time elapsed since they were last edited or viewed.

This is a pretty straightforward addition, and we hope it will help keep your workspace tidy. What happens when projects get archived?

• They are excluded from search.
• Any associated scheduled runs are disabled.
• They are NOT deleted, and can be unarchived if needed.

This is a way to gain some ground back from all the Untitled projects and half-done explorations lying around your workspace, as well as old masterpieces that were useful at one point, but aren’t in active service anymore.

Auto archive is available for workspaces on the Team plan and above. Find the config in Settings > Organization.

Other improvements​

• Users that have the Slack integration installed can now respond to access requests to projects from within Slack! And if a coworker gets to the request before you do, the Slack message will be updated in-line to let you know. (Note: for now, this only works for users on our multitenant stack, i.e. if your Hex URL starts with app.hex.tech.)
• We’ve improved how chart cells are displayed in the outline sidebar! They are no longer (quite as) squished.
• Single value visualizations now support standard Python negative indexing, so you can select the -1th row to reliably grab the last row in a dataframe of dynamic length.
• Assorted improvements and bug fixes to in app notifications.

Here’s one of the founding principles of Hex: You can do the best analysis in the world, but it won’t be useful unless the right person can see it.

Until they see it, it is Schrödingers data analysis. Useful? Not useful? A radioactive, poisoned cat? It is actually all three.

There’s no way to know for sure until someone sees your work, understands it, and makes a decision based on it. This is why Hex lets you turn your notebooks into reports that others can see, understand, and action off of.

And today we’ve got a bunch of improvements to the way you and your team consume data from those Hex apps you build! Caching, filters, notifications, exporting projects— it’s all a little bit easier now.

⌛ App freshness​

You can now set a cache timeout for entire published apps. When a viewer opens a “stale” app, it will automatically begin refreshing in the background, and alert the viewer when the new results are available.

Previously, updating results in an app required a scheduled run. With the new “Rerun app if stale” setting, any time a viewer opens an app, Hex will check if the results are fresh or stale. If they’re stale, Hex will automatically start refreshing, and let the viewer know when there’s new data.

This means less configuration, fresher data for everyone, and no scheduled runs on that app people stopped looking at months ago— potentially saving you some warehouse $$!

Editors can configure the time-to-stale and refresh behavior from the App settings sidebar. We’ve defaulted the "Rerun app if stale" setting to off, but we recommend you enable it for apps going forward!

Learn more in the docs.

🔍 Project filters​

Project filters make apps dynamic without the need to parameterize SQL and Python by hand. Just create a new filter, attach it to app elements and dataframes by pointing & clicking, and you’re done.

We teased Project filters last month, and they are now fully available for everyone to use! You can get started by choosing “Add project filter” from the notebook or app builder in any project.

Project filters work especially well on apps with many tabs, or in complex projects where parameterizing tons of SQL queries would be a chore. They also make it easier for viewers to reason about how apps are being filtered, and they work seamlessly with the cross-filters that viewers can configure themselves!

Looking for more guidance on how to use project filters alongside filter cells and input parameters? Check out the docs, or watch this 5 minute Loom video where I explain the difference:

🔔 Notification center​

Editors and viewers can now manage all Hex notifications and access requests from one central place.

This new notification center is available in the sidebar of every page in Hex, so you can easily triage failed runs, project access requests, and new comments without having to slog through your email or Slack.

📜 PDF export​

You can now schedule PDF captures of an app, and viewers can export PDFs right from the published app.

You know who you are, PDF lovers. And I know you have your mysterious reasons. Presumably with this feature you are now one step closer to global domination. I just hope you remember me when you seize control.

Quick facts: PDFs are exported with the visible app layout at time of publish, so build your apps for export if you know they’ll be consumed that way! Anonymous users (if your project is shared publicly) cannot export PDFs.

PDF Export is available for Teams + Enterprise customers.

🤳Mobile improvements​

We heard you, subway riders, busy parents, and the rest of the diverse compulsive Hex app checker community. We’ve updated our mobile styles to make Hex apps nice to use on your phone!

These improvements only apply to the published app view for now, and we know there’s a lot more work to do on the homepage and list views.

Other improvements​

• We’ve patched up some issues with the dbt integration in the data browser not showing column-level information.
• Tidied up some of the styles on table display cells on large screens
• We removed the 'day' option from the chart cell Time Unit picker. This is a tiny thing, but it always peeved me to click it and get the Day of Year 😑. You always want 'date' instead, trust me!