Livebook v0.14 is here with new features that make navigating your code easier than ever.

No more endless scrolling through a notebook to find module or function definitions — Livebook v0.14 has you covered.

Module definitions in the Outline panel

When you define an Elixir module in a code cell, it will now be listed in the Outline panel in the sidebar.

The Outline panel, previously known as the Sections panel, now gives you a structured view of your code. It lists all the modules defined within each section of your notebook.

You can also use this panel to navigate to module definitions. Just click on a module's name on the Outline, and your notebook will automatically scroll to the position where it is defined.

Here's a video showing how it works:

Want to see how we built this? Here's the pull request.

Go to definition of modules and functions

Not only can you quickly navigate to the definition of a module, but you can also quickly go to the definition of a function.

To do that, hover over a module or function you defined in your notebook, hold Command ⌘ (for macOS) or Ctrl (for Windows/Linux), and click on the module or function call.

Here's a video showing how it works:

You can also access this feature through the link shown in the intellisense for a module or function call:

Want to see how we built this? Here are the pull requests: #2730, #2741.

Conclusion

We're dedicated to continually improving the Livebook developer experience, and these new features are another step in that journey.

Feel free to check out the v0.14 changelog to discover all the features and changes in this newest release.

Happy hacking!

We are thrilled to announce the release of a new version of Livebook!

This version includes numerous new features and improvements. Today, we will highlight one of the most significant additions: Kino.Proxy.

By using Kino.Proxy, you can now expose API endpoints directly from your notebook or Livebook app.

Let's take a look at how it works.

How to expose an API endpoint from your Livebook notebook

Here's a simple "hello world" example:

Kino.Proxy.listen(fn conn ->
    Plug.Conn.send_resp(conn, 200, "hello world!")
end)

The Kino.Proxy.listen/1 function makes Livebook expose API endpoints and will proxy requests to the handler function passed to it.

That handler function receives a Plug.Conn as an argument and should use the Plug API to send a response to the client.

Let's see a video with a simple example working: 

Leveraging Plug for APIs with Kino.Proxy

You can also provide a module plug as an argument to Kino.Proxy.listen/1, like this:

defmodule MyPlug do
  def init([]), do: false

  def call(conn, _opts) do
    Plug.Conn.send_resp(conn, 200, "hello world!")
  end
end

Kino.Proxy.listen(MyPlug)

Since our API handler is a plug, we can leverage other plugs. For example, here's how to use Plug.Router to handle multiple endpoints:

defmodule ApiRouter do
  use Plug.Router

  plug :match
  plug :dispatch

  get "/hello" do
    send_resp(conn, 200, "hello from router")
  end

  get "/echo/:message" do
    send_resp(conn, 200, String.upcase(message))
  end

  match _ do
    send_resp(conn, 404, "oops, not found")
  end
end

Kino.Proxy.listen(ApiRouter)

Livebook apps + Kino.Proxy = handling requests from external apps

This new feature enables an exciting new possibility: a Livebook app can now use Kino.Proxy to handle external HTTP requests, allowing code execution triggered by an external system.

Let’s see an example.

We built a Livebook app that tracks how many days we don't have a broken build in a GitHub repository:

Whenever a new build finishes in the configured Github Repo, GitHub sends a webhook to the Livebook app, which processes the request to update its state.

To handle webhooks inside the Livebook app, we could combine Plug.Router with Kino.listen/1 like this:

defmodule ApiRouter do
  use Plug.Router

  plug(:match)
  plug(Plug.Parsers, parsers: [:json], json_decoder: Jason)
  plug(:dispatch)

  post "/webhook" do
    process_webhook(conn.body_params)

    conn
    |> put_resp_content_type("application/json")
    |> send_resp(200, ~s({"message": "ok"}))
  end

  match _ do
    conn
    |> put_resp_content_type("application/json")
    |> send_resp(404, ~s({"message": "not found"}))
  end

  defp process_webhook(webhook_payload) do
    # Logic to do something with the webhook payload
  end
end

Kino.Proxy.listen(ApiRouter)

Here's a quick video with a demo of that app:

Here's the source code for that Livebook app so you can understand how it works under the hood. 

If you want to run the app by yourself, click here.

Wrapping up

We believe that APIs with Kino.Proxy open Livebook to a new category of use cases.

For example, you can quickly build an API that receives a request from Zapier to automate internal processes.

We're looking forward to seeing what you will build it with!

One last thing: Livebook 0.13 has plenty more, and you can view all of them in our changelog.

We plan to share quicker demos of other Livebook 0.13 features on our X/Twitter, so feel free to follow us there.

Happy hacking!

Welcome to the 5th day of Livebook Launch Week 2! 🎉

For those of us who found ourselves instinctively trying to use Vim or Emacs shortcuts in Livebook, only to realize that these weren’t supported, feeling frustrated. We hear you.

We’re excited to announce that now you can use Vim and Emacs key bindings while coding in Livebook.

Let’s see how it works.

Using Vim or Emacs key bindings in Livebook

Go to the settings page in Livebook and choose the “Key bindings” you prefer:

Once that’s done, you can use it in any Livebook cell with a code editor, like in a code cell or a markdown cell. Here’s an example using Vim keybindings:

From now on, you can bring your Vim/Emacs muscle memory to Livebook for a more pleasant and productive experience. Thanks to Kenichi Nakamura, who contributed that feature.

What now?

To start using that new feature, install the latest version of Livebook and have fun!

More of Launch Week 2

• Day 1: Remote execution Smart cell
• Day 2: Speech-to-text with Whisper: timestamping, streaming, and parallelism, oh-my!
• Day 3: Introducing File Integration
• Day 4: Integration with SnowFlake and Microsoft SQL Server

And if you want to discover everything that was added/changed in this release, here’s the changelog.

Welcome to the 4th day of Livebook Launch Week 2! 🎉

Today’s post is about our two new database integrations: Snowflake and SQL Server.

Together with those new ones, Livebook now comes with built-in integrations for seven databases and data warehouses:

• PostgreSQL
• MySQL
• SQL Server
• SQLite
• Google BigQuery
• Amazon Athena
• Snowflake

Let’s see how the new integrations work.

Connecting to Snowflake

The Snowflake integration is available through the Database Connection Smart cell. All you need to do is configure the Smart cell with your connection credentials, and you’ll be ready to start executing queries.

Let’s see how that works.

Under the hood, this is different from the other database integrations we already have. Livebook connects to Snowflake through ADBC (Arrow Database Connectivity), using the ADBC hex package, which contains ADBC bindings for Elixir.

The Arrow format is highly efficient, and it’s also integrated with Explorer, so now you can easily query a database and load the result into an Explorer dataframe.

Good news for those of us doing data analysis with Elixir and Livebook. 🎉

Connecting to Microsoft SQL Server

Connecting to SQL Server is also super simple. All you need to do is add a new Database Connection Smart cell, select SQL Server, and fill in your database credentials.

Here’s how it works.

This was a community contribution from Simon McConnell. Curious fact: the pull request contains less than 250 LOC! Cool, right?! 😎

What now?

To start playing with those new features, install the latest version of Livebook and have fun!

More of Launch Week 2

• Day 1: Remote execution Smart cell
• Day 2: Speech-to-text with Whisper: timestamping, streaming, and parallelism, oh-my!
• Day 3: Introducing File Integration
• Day 5: Vim and Emacs key bindings

Welcome to the 3rd day of Livebook Launch Week 2! 🎉

Today, we’re excited to announce Livebook v0.11’s most prominent feature, file integration.

Although file integration may not sound exciting at first glance, bear with us; we have some elegant features to show.

Imagine code generation when you drag and drop a file based on the file type. Or quickly do a lazy reading of a multiple gigabyte parquet file stored in S3. That’s what we’re talking about! 😎

Watch the video where José Valim shows a demo of what this feature can do.

Why?

Throughout the last releases, we have been improving how Livebook helps to work with data.

Data comes from different sources. One common source is a database, which Livebook already has good support for. Another source is a file, either on your machine or on S3, which Livebook hasn’t natively integrated with yet.

Don’t get us wrong, you could already write some Elixir code in your notebook that reads a file from the file system or the web. But the Livebook UI had no understanding of it. And that’s what changed in this release.

What’s the Livebook file integration?

The concept is simple. When you’re editing a notebook, you can add a file to it. All files are listed in the sidebar, so everyone using your notebook can quickly glance at your file dependencies. Once you do that, Livebook exposes an API for your notebook to read that file via the Kino.FS module.

But that’s not the only thing. Now that Livebook natively understands files, it can offer code suggestions based on the file type.

For example, when you drag and drop a CSV file to Livebook, it can generate code that creates an Explorer Dataframe from that data:

Or when you drag and drop a SQLite database file, Livebook can generate code that describes that database:

Or when you drag and drop an audio file, Livebook can generate code that uses a Machine Learning model to generate a transcription for you:

One thing you’ll notice in common with all those drag-and-drop examples is that Livebook doesn’t automatically execute the code. Instead, it emits the code and allows you to run it yourself.

This may look like a small detail, but it’s part of our vision for Livebook. We don’t want it to be a magic box that executes some task behind the scenes that you can’t know what it is. We want to enable you to introspect the code, learn from it, and customize it to your needs.

We took that approach with Smart cells. And that’s the approach we’re continuing to apply.

File references and file attachments

When you start adding files to your notebook, you’ll notice that there are two types of files: References and Attachments.

References are files that point to existing resources on your disk, a remote storage, or a URL. It’s like a symbolic link; Livebook doesn’t store the file, it just has a pointer to it.

Attachments are different. They are files stored in the files/ directory alongside your notebook source. So, a file attachment will be kept wherever you save your notebook.

So, if you’re storing a file in an S3-compatible cloud service, in a web server, or if you want to save it and version it alongside your notebook source, Livebook has you covered.

Lazy data reading and manipulation

When you have a dataset with multiple gigabytes of data, you don’t want to load all of it at once to the memory. A common approach for that is lazy loading the data. And this new Livebook release supports that workflow as well.

For example, let’s say your dataset is stored as a Parquet file in S3. When you add that file to your notebook, Livebook will give the option to load that data into a dataframe lazily:

Not only that, but you can also lazily manipulate that data:

When needed, Livebook will download all the data, like when plotting a chart based on the data or when exporting it after some data wrangling:

When we announced Bumblebee, a collection of pre-trained models inspired by Hugging Face Transformers, the Whisper speech-to-text model quickly became one of the favorite and most used models within the Elixir community.

Thanks to advancements in the overall Numerical Elixir ecosystem, Livebook v0.11 includes a highly improved integration with Whisper, which we will detail in this article.

If you want to skip ahead and give it a try, install Livebook and start a new notebook. Then click “+ Smart cell” and choose “Neural Network task.” You will find Whisper as Speech-to-text under Audio.

New features

There are three new features in our Whisper integration:

1. Timestamping: we now include timestamps on audio segments.
2. Streaming: our previous version of Whisper was limited to 30 seconds of audio, leaving it up to users to break their audio apart. This new version is capable of streaming both inputs and outputs. You can give arbitrarily long files to the model, which will be streamed as input, and the model will proceed to merge and stream transcriptions as they arrive.
3. Parallelism: in addition to streaming, files with more than 30 seconds will be split and batched according to the Neural Network batch size. For example, with a batch size of 10, up to 5 minutes of audio can be processed in parallel. Thanks to this, we expect our models to perform inference an order of magnitude faster compared to Open AI’s implementation when transcribing larger files on the GPU.

Of course, all of those features work together, providing a delightful experience as you can see below, where we transcribe on the fly one of Thinking Elixir episodes:

When you combine the features above with Nx’s ability to run neural networks distributed across multiple machines and GPUs, Elixir developers now have a first-class, state-of-the-art, speech-to-text model ready to run, enjoy, and scale.

What now?

Try for yourself!

Transcribe an audio file using our built-in Neural Network Task Smart cell. Maybe start with a small file to quickly see the result. Then you can try a bigger one.

Download the latest Livebook version and have fun!

More of Launch Week 2

• Day 1: Remote execution Smart cell
• Day 3: Introducing File Integration
• Day 4: Integration with SnowFlake and Microsoft SQL Server
• Day 5: Vim and Emacs key bindings

Welcome to the second Livebook Launch Week! 🎉

If this is your first Launch Week with us, let us explain how this works. Starting today, each day of this week, we’ll announce a new feature of this release, Livebook v0.11.

Today, we’ll discuss our newest built-in Smart cell: Remote execution.

Calling code from other nodes before the new Smart cell

Livebook is a multi-purpose tool. And two of the most common use cases we’ve been seeing are debugging and internal tools.

A common requirement for both use cases is to write code that calls functions from a remote node, probably your main Elixir app in the production environment.

One way to do that is by connecting your notebook to the other node using the “Attached node” runtime:

That said, that approach has a limitation. When you use it, all of the code in your notebook will run in the context of that remote node. So, you can’t add Mix dependencies to your notebook, like VegaLite for charts or Kino for user interactions.

The other way we have to run remote functions is by using the default runtime (Elixir standalone) and using the erpc Erlang module, like this:

But, we noticed people have been using the “Attached node” by default when calling a function from a remote node, even when they need to add dependencies to their notebook. That’s a problem because although the “Attached node” setting is powerful for debugging, we believe that most of the time you need to interact with a remote node, you should use the erpc approach.

So, we decided to make the erpc approach a first-class citizen in Livebook. It’s now available as a new Smart cell that will streamline the process of calling functions from another node.

Calling code from other nodes using the Remote execution Smart cell

Here’s how the new Smart cell works:

As with any other Smart cell, one of the things it helps with is generating boilerplate code for you. There is no need to write the code that connects your notebook to a remote node over and over again.

But it goes beyond that.

The Smart cell can store the remote node’s cookie as a Livebook secret, so you don’t need to hardcode sensitive values inside your notebook. And with the upcoming Livebook Teams, it will be easy to share this secret with everyone on the team in a secure manner.

Last, this new Smart cell gives you autocompletion and docs preview of the modules defined in the remote node! Isn’t that awesome?! 🤯

We hope this feature will simplify the development of internal tools that need to communicate with existing nodes.

As an example, here’s a video showing how to use the new feature to connect a Livebook notebook to a Phoenix app, get some data from it, display a chart with those metrics, and share that as an interactive Livebook app:

What now?

All of the new features of this new release are already available today!

To start playing with them, install the latest version of Livebook and have fun!

More of Launch Week 2

• Day 2: Speech-to-text with Whisper: timestamping, streaming, and parallelism, oh-my!
• Day 3: Introducing File Integration
• Day 4: Integration with SnowFlake and Microsoft SQL Server
• Day 5: Vim and Emacs key bindings

Today we’re launching Livebook 0.10! 🎉

This major update brings many exciting features, with the spotlight being the introduction of multi-session Livebook apps.

We’ve also added a presentation view, initial Erlang support, Live Doctests, and dataframe file export. Let’s dive in and explore these new features.

Multi-Session Livebook Apps

Livebook 0.9 introduced Livebook apps. This is a way to turn your notebook into an interactive web application. Now, we’re expanding that further.

Initially, Livebook Apps was designed for long-running applications. Behind the scenes, only one instance of a Livebook app could run at any given moment. Since Livebook has built-in support for multiple users, all users accessing an app would be sharing the same instance of the app. We’re now calling that single-session Livebook apps.

This new version introduces multi-session Livebook Apps. What’s different is that when you join a multi-session application, you get a version of that app exclusively for you. Like single-session apps, multi-session apps can run for as long as they want, but most often, they will receive user input, execute several instructions, and then exit.

We believe they are an excellent fit for automating technical and business workflows. You can think of them as something similar to scripts, but instead of running in a terminal, they are interactive web applications accessed through the browser.

For example, instead of repeatedly being asked to run one-off scripts, you can package that script as a Livebook app and makes it accessible to other team members to run it, at any time, by themselves.

Let’s see how that works.

Presentation View

We noticed many people use Livebook for presentations. However, it can be frustrating to switch between Livebook and your slides. Also, showing the whole notebook can distract your audience. But there’s good news!

Franklin Rakotomalala contributed a Presentation view feature that hides the sidebar and focuses on the part of your notebook you want to present. Here’s how it works.

Shout out to Franklin!

Initial Erlang Support

You can now write Erlang code inside Livebook. Not only that, but you can combine it with Elixir in the same notebook. If you define a function or a variable inside one language, you can easily use it in the other.

Watch the video below for an example.

This was a community contribution by Benedikt Reinartz. Thanks to him, Livebook now supports multiple BEAM languages.

Live Doctests

Starting from version 0.8, doctests are integrated with Livebook. This new version comes with exciting improvements in that area.

When you evaluate a cell with doctests, a traffic light-like status appears on the left of each doctest. This gives you a visual indication if it passed or not. Additionally, when the doctest fails, the failing result is directly inlined in the code editor.

Here’s how it works:

This is a step towards bringing Live Programming concepts into Livebook, the idea that for “programming to be more fluid, editing and debugging should occur concurrently as you write code.”

This one started with the code contribution of Jose Vargas and the research work of Szymon Kaliski.

Dataframe File Export

When working on a data analysis task using Livebook, you might need to access the analysis result from another tool or share them with someone who prefers opening it in a spreadsheet.

Now, you can easily do that by exporting your dataframe to a CSV, NDJSON, or Parquet file. Here’s how it works:

Try it!

To play with the new features, follow these steps:

• Install the latest Livebook version
• Import the demo notebook that showcases the new features by clicking the badge below

And if you want to discover everything that changed in 0.10, here’s the changelog.

Happy hacking!

Welcome to the fifth and last day of the first Livebook Launch Week!

Today we will talk about data wrangling with Livebook and the new data capabilities with this new release.

Watch the video where José Valim shows a demo of those new features.

You can also read an overview of the new features below.

Explorer: series and dataframes for fast data exploration in Elixir

Explorer is a project that brings series (one-dimensional data) and data frames (two-dimensional data) right into Elixir.

It is implemented on top of the Polars project, a highly performant dataframe Rust library. And it’s highly inspired by the dplyr project from R, which is quite expressive. We aim to bring ideas from both communities into Elixir to provide a powerful and elegant tool for data processing.

Let’s play with Explorer a little bit.

The first step of a data exploration project is to import the data. Let’s see how you can import a dataset into an Explorer data frame:

After importing the data, it’s usually a good idea to explore it a little bit. To do that, we’ll use a new feature from Livebook that helps us visualize an Explorer Dataframe as a table.

Visualizing an Explorer dataframe as an interactive table

Kino is the library used by Livebook to render rich and interactive outputs directly from your Elixir code. Livebook has multiple built-in Kinos, but anyone can also build custom Kinos as a way to extend Livebook.

We built a new Kino called Kino Explorer for this release to improve the integration between Explorer and Livebook. Let’s see how we can use it to display an Explorer data frame as an interactive table:

Data transformation using the new Data Transform Smart cell

In Livebook, a Smart cell is a UI-based cell that helps you to accomplish a specific task like creating a database connection, sending a message to a Slack channel, or running a Machine Learning model.

Livebook has various built-in Smart cells, and anyone can build a Smart cell to extend Livebook to their needs.

For this release, we built a new built-in Smart cell; the Data Transform one. Let’s see how we can use it for some data transformation.

What now?

Want to play with all of those new features?

First, ensure you’ve installed the latest version of Livebook.

Then, click the button below to run the notebook that José Valim built in the demo video:

If you have any comments or want to share what you’ve built using Livebook, you can tweet using the #LivebookLaunchWeek hashtag.

I hope you got as excited as we did with this new Livebook 0.9 release.

Besides that, we’re already working on much more exciting stuff we’re looking forward to sharing—for example, the upcoming Livebook Teams.

If you use or want to use Livebook at work with your colleagues, you can fill in our form to help us to inform Livebook Team’s roadmap and get updates about it.

Thank you very much for being with us these last five days. This is all for this first Livebook Launch Week!

Building Livebook and sharing what it’s capable of is a joy for us. We hope you can have fun with it too. 😄

More Launch Week

• Day 1: Deploy notebooks as apps & quality-of-life upgrades
• Day 2: Distributed² Machine Learning notebooks with Elixir and Livebook
• Day 3: Hubs and secret management
• Day 4: Build and deploy a Whisper chat app to Hugging Face in 15 minutes

Welcome to the fourth day of Livebook Launch Week! 🎉

Today we will build a Whisper chat app with Livebook and deploy it to Hugging Face.

In this chat app, users can communicate only by sending audio messages, which are then automatically converted to text by the Whisper Machine Learning model:

You can access the demo over the next few days on José Valim's Livebook instance on HuggingFace.

This demo covers a lot of cool features from Livebook, Nx, and Bumblebee, like:

• Deploy Livebook notebooks as apps

  On day 1 of our Launch Week, we saw how to deploy your notebook as an interactive web app. This time we’ll go further and deploy a Machine Learning-based app to Hugging Face.

• Realtime and multiplayer apps

  Livebook’s modern approach to code notebooks brings interactivity and collaboration. So, apps built with Livebook are also interactive and multiplayer by default! The Whisper chat app we’ll be an excellent example of that.

• Concurrent Machine Learning

  On day 2 of our Launch Week, we talked about the new Machine Learning models you can use with just a few clicks through Livebook’s Neural Network Smart Cell. In today’s video, you’ll see how to customize the code generated by the Smart cell and build an app on top of it.

  We’ll add concurrency and batching to improve the app’s scalability and performance, allowing multiple users to submit audio messages simultaneously and efficiently process them using the Whisper model.

What now?

Want to run the Whisper chat app by yourself?

First, ensure you’ve installed the latest version of Livebok.

Then, click the button below to run the notebook yourself:

If you have any comments or want to share what you’ve built using Livebook, you can tweet using the #LivebookLaunchWeek hashtag.

Stay tuned for the final announcement of the Livebook Launch Week!

More Launch Week

• Day 1: Deploy notebooks as apps & quality-of-life upgrades
• Day 2: Distributed² Machine Learning notebooks with Elixir and Livebook
• Day 3: Hubs and secret management
• Day 5: Data wrangling in Elixir with Explorer, the power of Rust, the elegance of R