Introduction

vidformer enables people to create videos without worrying about performance or efficiency. vidformer does this via its novel data-oriented declarative video editor.

We target these use cases:

  • Creating videos as results to queries (e.g., "show me every time zebras fight, and overlay the time, date, location, and animal IDs")
  • Visualization of computer vision models (e.g., Drawing bounding over objects in videos in a python notebook)

For these use cases, vidformer strives to:

  • Create the resulting videos instantly and efficiently
  • Allow easy and idiomatic editing of videos combined with data
  • Interact with existing technologies, data, and workloads
  • Serve both exploratory ad hoc use cases and embedded use in production web applications, VDBMSs, and IaaS deployments.

vidformer is NOT:

  • A conventional video editor (like Premiere Pro or Final Cut)
  • A video database/VDBMS
  • A natural language query interface for video
  • A computer vision library (like OpenCV)
  • A computer vision AI model (like CLIP or Yolo)

However, vidformer is highly complementary to each of these. If you're working on any of the later four, vidformer may be for you.

Quick Start

See Getting Started

Next Steps

vidformer is a highly modular suite of tools that work together. The tools overview provides an overview and guidance on the full vidformer architecture.

Getting Started

This is a walkthrough of getting started with vidformer-py.

Installation

Getting vidformer-py:

pip3 install vidformer

Additionally you need a vidformer server. We distibute the locally-running version, yrden, through vidformer-cli. Currently, vidformer-cli only runs on Linux, so using the vidformer docker image is highly advised:

docker pull dominikwinecki/vidformer:latest
docker run --rm -it -p 8000:8000 dominikwinecki/vidformer:latest yrden --print-url

This launches a vidformer yrden server, which is our reference server implementation for local usage, on port 8000. If you want to read or save video files locally add -v /my/local/dir:/data and then reference them as /data in the code.

Hello, world!

We assume this is in a Jupyter notebook. If not then .play() won't work and you have to use .save() instead.

We start by connecting to a server and registering a source:

import vidformer as vf
from fractions import Fraction

server = vf.YrdenServer(domain='localhost', port=8000)
example_vids = vf.StorageService(
    "http",
    endpoint="https://f.dominik.win",
    root="/data/dve2/")

tos = vf.Source(
    server,
    "tos_720p",     # name (for pretty printing)
    "tos_720p.mp4", # path
    stream=0,       # index of the video stream we want to use
    service=example_vids)

print(tos.ts())
print(tos.fmt())

This will print the timestamps of all the frames in the video, and then format information: This may take a few seconds the first time, but frame times are cached afterwords.

> [Fraction(0, 1), Fraction(1, 24), Fraction(1, 12), Fraction(1, 8), ...]
> {'width': 1280, 'height': 720, 'pix_fmt': 'yuv420p'}

Now lets create a 30 second clip starting at the 5 minute mark. The source video is at at a constant 24 FPS, so lets create a 24 FPS output as well:

domain = [Fraction(i, 24) for i in range(24 * 30)]

Now we need to render each of these frames, so we define a render function.

def render(t: Fraction, i: int):
    clip_start_point = Fraction(5 * 60, 1) # start at 5 * 60 seconds
    return tos[t + clip_start_point]

We used timestamp-based indexing here, but you can also use integer indexing (tos.iloc[i + 5 * 60 * 24]).

Now we can create a spec and play it in the browser. We create a spec from the resulting video's frame timestamps (domain), a function to construct each output frame (render), and the output videos format (matching tos.fmt()).

spec = vf.Spec(domain, render, tos.fmt())
spec.play(server)

This plays this result:

Some Jupyter environments are weird (i.e., VS Code), so .play() might not work. Using .play(..., method="iframe") may help.

It's worth noting that we are playing frames in order here and outputing video at the same framerate we recieved, but that doesn't need to be the case. Here are some things other things you can now try:

  • Reversing the video
  • Double the speed of the video
    • Either double the framerate or sample every other frame
  • Shuffle the frames into a random order
  • Combining frames from multiple videos
  • Create a variable frame rate video
    • Note: .play() will not work with VFR, but .save() will.

Bounding Boxes

Now let's overlay some bouding boxes over the entire clip:

# Load some data
import urllib.request, json 
with urllib.request.urlopen("https://f.dominik.win/data/dve2/tos_720p-objects.json") as r:
    detections_per_frame = json.load(r)

bbox = vf.Filter("BoundingBox") # load the built-in BoundingBox filter

domain = tos.ts() # output should have same frame timestamps as our example clip

def render(t, i):
    return bbox(
        tos[t],
        bounds=detections_per_frame[i])

spec = vf.Spec(domain, render, tos.fmt())
spec.play(server)

This plays this result (video is just a sample clip):

Composition

We can place frames next to each other with the HStack and VStack filters. For example, HStack(left_frame, middle_frame, right_frame, width=1280, height=720, format="yuv420p") will place three frames side-by-side.

As a larger example, we can view a window function over frames as a 5x5 grid:

hstack = vf.Filter("HStack")
vstack = vf.Filter("VStack")

w, h = 1920, 1080

def create_grid(tos, i, N, width, height, fmt="yuv420p"):
    grid = []
    for row in range(N):
        columns = []
        for col in range(N):
            index = row * N + col
            columns.append(tos.iloc[i + index])
        grid.append(hstack(*columns, width=width, height=height//N, format=fmt))
    final_grid = vstack(*grid, width=width, height=height, format=fmt)
    return final_grid

domain = [Fraction(i, 24) for i in range(0, 5000)]

def render(t, i):
    return create_grid(tos, i, 5, w, h)

fmt = {'width': w, 'height': h, 'pix_fmt': 'yuv420p'}

spec = vf.Spec(domain, render, fmt)
spec.play(server)

This plays this result (video is just a sample clip):

Viewing Telemetry (and User-Defined Filters)

This notebook shows how to build custom filters to overlay data.

This plays this result (video is just a sample clip):

Concepts & Data Model

vidformer builds on the data model introduced in the V2V paper.

  • Frames are a single image. Frames are represented as their resolution and pixel format (the type and layout of pixels in memory, such as rgb24, gray8, or yuv420p).

  • Videos are sequences of frames represented as an array. We index these arrays by rational numbers corresponding to their timestamp.

  • Filters are functions which construct a frame. Filters can take inputs, such as frames or data. For example, DrawText may draw some text on a frame.

  • Specs declarativly represent a video synthesis task. They represent the construction of a result videos, which is itself modeled as an array.

    • Specs primairly contan domain and render functions.
      • A spec's domain function returns the timestamps of the output frames.
      • A spec's render function returns a composition of filters used to construct a frame at a spesific timestamp.

  • Data Arrays allow using data in specs symbolically, as opposed to inserting constants directly into the spec. These allow for deduplication and loading large data blobs efficiently.

    • Data Arrays can be backed by external data sources, such as SQL databases.

The vidformer Tools

vidformer is a highly modular suite of tools that work together:

  • vidformer-py: A Python 🐍 client for declarative video synthesis

    • Provides an easy-to-use library for symbolically representing transformed videos
    • Acts as a client for a VoD server (i.e., for yrden)
    • Using vidformer-py is the best place to get started

  • libvidformer: The core data-oriented declarative video editing library

    • An embedded video processing execution engine with low-level interfaces
    • Systems code, written in Rust 🦀
    • You should use if: You are building a VDBMS or other multimodal data-system infrastructure.
    • You should not use if: You just want to use vidformer in your workflows or projects.

  • yrden: A vidformer Video-on-Demand server

    • Provides vidformer services over a REST-style API
    • Allows for client libraries to be written in any language
    • Serves video results via HLS streams
    • Designed for local single-tenant use
    • You should use if: You want to create faster video results in your workflows or projects.
    • Note that yrden servers may be spun up transparently by client libraries, so you might use yrden without realizing it.

  • igni: A planned scale-out Video-on-Demand server

    • Will allow for scalable and secure public-facing VOD endpoints

Client libraries in other languages: Writing a vidformer client library for other languages is simple. It's a few hundred lines of code, and you just have to construct some JSON. Contributions or suggestions for other languages are welcome.

Other VoD servers: We provide yrden as a simple reference VoD server implementation. If you want to scale-out deployments, multi-tenant deployments, or deep integration with a specific system, writing another VoD server is needed. (In progress work)

vidformer-py

vidformer-py is our Python 🐍 interface for vidformer. Our getting started guide explains how to use it.

The docs are here.

The source code is here.

libvidformer

libvidformer is our core video synthesis/transformation library. It handles the movement, control flow, and processing of video and conventional (non-video) data.

Here are the source code and docs.

  • It's written in Rust 🦀
    • So it does some fancy parallel processing and does so safely
  • Uses the FFmpeg libav libraries for multimedia stuff
    • So it should work with nearly every video file ever made
  • Uses Apache OpenDAL for I/O
    • So it can access videos in a bunch of storage services
  • Implements some filters using OpenCV

Built-in Filters

While most applications will use user-defined filters, vidformer ships with a handful of built-in filters to get you started:

DrawText

DrawText does exactly what it sounds like: draw text on a frame.

For example:

DrawText(frame, text="Hello, world!", x=100, y=100, size=48, color="white")

BoundingBox

BoundingBox draws bounding boxes on a frame.

For example:

BoundingBox(frame, bounds=obj)

Where obj is JSON with this schema:

[
  {
    "class": "person",
    "confidence": 0.916827917098999,
    "x1": 683.0721842447916,
    "y1": 100.92174338626751,
    "x2": 1006.863525390625,
    "y2": 720
  },
  {
    "class": "dog",
    "confidence": 0.902531921863556,
    "x1": 360.8750813802083,
    "y1": 47.983140622720974,
    "x2": 606.76171875,
    "y2": 717.9591837897462
  }
]

Scale

The Scale filter transforms one frame type to another. It changes both resolution and pixel format. This is the most important filter and is essential for building with vidformer.

Arguments:

Scale(
    frame: Frame,
    width: int = None,
    height: int = None,
    pix_fmt: str = None)

By default missing width, height and format values are set to match frame. pix_fmt must match ffmpeg's name for a pixel format.

For example:

frame = Scale(frame, width=1280, height=720, pix_fmt="rgb24")

IPC

IPC allows for calling User-Defined Filters (UDFs) running on the same system. It is an infrastructure-level filter and is used to implement other filters. It is configured with a socket and func, the filter's name, both strings.

The IPC filter can not be directly invoked, rather IPC filters are constructed by a server upon request. This can be difficult, but vidformer-py handles this for you. As of right now IPC only supports rgb24 frames.

HStack & VStack

HStack & VStack allow for composing multiple frames together, stacking them either horizontally or vertically. It tries to automatically find a reasonable layout.

Arguments:

HStack(
    *frames: list[Frame],
    width: int,
    height: int,
    format: str)

At least one frame is required, along with a width, height and format.

For example:

compilation = HStack(left_frame, right_frame, width=1280, height=720, format="rgb24")

User-Defined Filters

To implement a new user-defined filter (UDF) you need to host a filter server over a UNIX Domain Socket. The vidformer-py library makes this easy.

Filters take some combination of frames and data (string, int, bool) and return a single frame result. The vidformer project uses Python-style arguments, allowing ordered and named arguments (*args and **kwargs style).

To do this we define a new filter class and host it:

import vidformer as vf
import cv2

class MyFilter(vf.UDF):

    def filter(self, frame: vf.UDFFrame, name: str):
        """Return the result frame."""

        text = f"Hello, {name}!"

        image = frame.data().copy()
        cv2.putText(
		    image,
            text, 
            (100,100),
            cv2.FONT_HERSHEY_SIMPLEX,
            1,
            (255, 0, 0),
            1,
        )
        return vf.UDFFrame(image, frame.frame_type())

    def filter_type(self, frame: vf.UDFFrameType, _name: str):
        """Returns the type of the output frame."""
        return frame

mf_udf = MyFilter("MyFilter") # name used for pretty printing

my_filter = mf_udf.into_filter() # host the UDF in a subprocess, returns a vf.Filter

Now we can use our newly-created filter in specs: my_filter(some_frame, "vidformer").

There is a catch, UDFs currently only support rgb24 pixel formats. So invoking my_filter will need to convert around this:

scale = vf.Filter('Scale')

def render(t, i):
    f = scale(tos[t], pix_fmt="rgb24", width=1280, height=720)
    f = my_filter(f, "world")
    f = scale(f, pix_fmt="yuv420p", width=1280, height=720)
    return f

FAQ

What video formats does vidformer support?

In short, essentially everything. vidformer uses the FFmpeg/libav* libraries internally, so any media FFmpeg works with should work in vidformer as well. We support many container formats (e.g., mp4, mov) and codecs (e.g., H.264, VP8).

A full list of supported codecs enabled in a vidformer build can be found by running:

vidformer-cli codecs

Can I access remote videos on the internet?

Yes, vidformer uses Apache OpenDAL for I/O, so most common data/storage access protocols are supported. However, not all storage services are enabled in distributed binaries. We guarantee that HTTP, S3, and the local filesystem always work.

How does vidformer compare to FFmpeg?

vidformer is far more expressive than the FFmpeg filter interface. Mainly, vidformer is designed for work around data, so edits are created programatically and edits can reference data. Also, vidformer enables serving resut videos on demand.

vidformer uses the FFmpeg/libav* libraries internally, so any media FFmpeg works with should also work in vidformer.

How does vidformer compare to OpenCV/cv2?

vidformer orchestrates data movment in video synthesis tasks, but does not implement image processing directly. Most use cases will still use OpenCV for this