“dat: A Streaming Interface for Data,” Max Ogden (SF Node.JS Meetup)

Dat is an open-source tool, funded by the Sloan Foundation in the US as part of their Science Tools research funding, that seeks to enable collaboration workflows on top of datasets of any size. The high-level goal of the Dat project is to make it easier to work with large scientific datasets in an automated way, which both saves time and also makes reproducibility easier. The core dat tool is a streaming dataset versioning + replication system developed with a heavy Unix philosophy designed to encourage extreme modularity and enable many third-party applications to be built on top. In addition to the core tool, we are also developing tools for building and distributing streaming, cross-platform data pipelines based on Node.js and Docker. About Max Ogden: Max Ogden is an open-source software developer who works full time on the Dat project at the United States Open Data Institute. He previously worked at Code for America, a US-based not-for-profit dedicated to improving technology in cities. Intro music: "IDM" by Vlaaad