DBOS

DBOS (Formerly Database-Oriented Operating System, now just DBOS) is an open source durable workflow execution software library written for the Python, TypeScript, Java, and Go programming languages.

DBOS arose from a joint open source project from MIT and Stanford, after a discussion between Michael Stonebraker and Matei Zaharia on how to scale and improve scheduling and performance of millions of Apache Spark tasks.¹ Today it is a commercial company that offers an open source system to add durable computing to any software, built on concepts derived from the joint research project.

DBOS originated in 2020 as a joint open source project between MIT, Stanford, and Carnegie Mellon. The project explored the idea of operating system services built atop a distributed database - a database-oriented operating system meant to simplify and improve the scalability, security and resilience of large-scale distributed applications.²

The basic concept was to run a multi-node multi-core, transactional, highly-available distributed database, such as VoltDB, as the only application for a microkernel, and then to implement scheduling, messaging, file systems and other operating system services on top of the database.

The architectural philosophy is described by this quote from the abstract of their initial preprint:

All operating system state should be represented uniformly as database tables, and operations on this state should be made via queries from otherwise stateless tasks. This design makes it easy to scale and evolve the OS without whole-system refactoring, inspect and debug system state, upgrade components without downtime, manage decisions using machine learning, and implement sophisticated security features.²

A prototype was built with competitive performance to existing systems.³

• PICK OS, another implementation of an operating system based on a DB.

DBOS, Inc. was incorporated in 2023 to commercialize technologies developed from the DBOS research project.¹

In March of 2024, the company launched the DBOS Transact open source durable execution library, which allows a program to resume running after a system failure, without loss of data or having to repeat work.⁴. The first DBOS library provided was for TypeScript; libraries for Python, Java, and Go have since been developed. Workflow orchestration semantics and time-travel debugging were other capabilities carried forward from the DBOS academic project.

The open-source DBOS Transact library provides a set of building blocks that makes applications resilient to failures. Library functionality includes durable workflows, workflow steps, database transactions, durable queueing, debouncing, messaging, and cron scheduling. The library also includes tooling for workflow management, versioning, patching, and observability features such as logging, tracing, and time-travel debugging.⁵

Application workflows are made durable via annotations that indicate the beginning and end of a workflow (@DBOS.workflow()) and the steps executed within (@DBOS.step()). As the workflow executes, the DBOS library saves state in a PostgreSQL database. If a workflow is interrupted for any reason, DBOS automatically recovers its execution from the last completed step.

@DBOS.step()
def step_one():
    print("Step one completed!")

@DBOS.step()
def step_two():
    print("Step two completed!")

@DBOS.workflow()
def workflow():
    step_one()
    step_two()

Task queues enable the execution of many workflows at once with managed concurrency. The DBOS library makes queues durable so that they survive system failures. The example below shows the creation of a queue of workflows with a worker concurrency of 5, so each process will run at most 5 workflows from this queue simultaneously:

DBOS.register_queue("example_queue", worker_concurrency=5)

@DBOS.workflow()
def process_task(task):
  ...

handle = DBOS.enqueue_workflow("example_queue", process_task, task)