Launch: TerminusHub – The Distributed Data Revolution

In the age of data, TerminusHub is arming the rebels to fight back against the internet giants

Dublin, Ireland. TerminusDB today announced the launch of TerminusHub, a distributed data collaboration platform. Powerful versioned data collaboration is a radical departure in the way we manage our data. TerminusHub is dedicated to the democratization of data and empowering people to take back control from the internet giants. It is like GitHub, but for data.

With TerminusHub, you can clone all of Wikipedia to your local machine in less than 2 minutes - and then collaboratively integrate it with another data source to see the links and connections. This functionality is available without cost for all users - TerminusDB is open source now and forever.

The public has been demanding a better solution to collaborative data management for years. With TerminusHub, you can share, build and integrate databases with your team and the wider world. Your project can keep data synchronized and make the latest data accessible no matter where you are. In a suddenly remote world where data is the driver of decision making, TerminusHub is the new paradigm in public benefit data management.

Starting in Trinity College Dublin, TerminusHub is the culmination of 6 years of research and development. TerminusHub has the potential to be a European data management champion to rival the predatory US and Chinese alternatives. TerminusDB was first financed by a €4 million European Commission Horizon 2020 grant.

Kevin Feeney, CEO of TerminusDB, said: “We live in the age of data. Everywhere we look, big data rules our lives. Amazon, Google, Microsoft, Facebook and the other internet giants have too much control, TerminusHub is arming the rebels with advanced data management tools - let's start the fightback”.

In the last 15 years, there has been a revolution in software engineering. For coders, the Git Revision control system changed everything. Distributed teams could collaborate on common code bases. But people who work with data didn’t have an easy and secure way to manage their pipelines and workflows. Until now.

TerminusHub is revolutionizing database collaboration and management.

8 September 2020

END

TerminusHub

Distributed Revision Control for Structured Data

So the cat is out of the bag… TerminusHub, which links TerminusDBs to provide collaborative revision control, is in private beta and will be generally available at the beginning of September. It will be forever free for the open-source and public interest communities.

Here is a quick video (made for our heroic beta testers) of TerminusHub in action:

With TerminusDB and TerminusHub we want to empower collaborative data integration:

  • Curate datasets

    • Load, clean, test / verify

    • Catalog with as much structural information as possible

    • Critical operations: ingest, catalog development, schema check

  • Share datasets easily

    • Using a distributed approach to sharing curated data sets

    • Critical collaboration operations: Push, pull, fetch, clone, fork

  • Integrate datasets

    • Fierce git-like integration

    • Critical operations: branch, merge, rebase

  • Control

    • Greater safety and oversight: time-travel, auditing, rollback

    • Never worry about making a mistake again

This is an intro video that covers some of the background of TerminusDB and Hub.

We want feedback. Tell us if this is cool/lame/confusing.

Leave a comment


Become a Beta Hero

Drop us a line if you want to be one of the first people on the planet to use TerminusHub. You can choose the unique username that will be with you forever. Think of the bragging rights when we get to user #1,232,843.

Give Email for Beta

Master to Main

TerminusDB and Hub will no longer use ‘master’ to designate the principal branch of your databases. As of our forthcoming 3.0 release, we will switch to ‘main’. We are Open Source and will always be open to all users. We stand with the oppressed.

Image

Super DataBase

You probably don’t know this, but Nicki Minaj loves an elegant data model and might be a huge fan of TerminusDB.

In honor of that, we made Super DataBase. Enjoy.

Stickers and Socks

We have Stickers and we have Socks. Do you want? We will post anywhere in the world. A little light relief from the tedium of lockdown!

Form to request (or you can join our Discord and DM ‘LukeEF’

Get Swag


Blog Highlights

Continuous Everything as Code - a look back at DevOps practices and a new path for schema migrations and managing everything as code

Be like the mammals when the asteroid struck - the effects of the pandemic on TerminusDB

TerminusDB v MongoDB - head to head

TerminusDB: Importing, Reordering & Exporting a CSV - the conclusion of our development episode. CSVs rule the world.

TerminusDB Graph Basics - a look at some of the TerminusDB basics


Join our Discord

It is where the magic happens.

Join Discord

Whisper it... TerminusDB 2.0 is Coming

Now in Canary Release

‘There are decades where nothing happens; and there are weeks where decades happen’

Vladimir Ilyich Lenin

And then there is TerminusDB where every week feels like a decade!


TerminusDB 2.0 Canary Release

Breaking with the fashion for lots of minor versions, we are smashing through to 2.0 after 1.0 last October and 1.1 in January. Too many software and database projects get stuck in the 0.18.7 minor version hell.

Yesterday was the Canary Release of TerminusDB 2.0. A big day for us.

The core team’s work over the last number of months has been focused on the delivery of collaborative Git-like functionality for TerminusDB. We want to enable the types of CI/CD used extensively in software engineering to be used with data. TerminusDB 2.0 will allow for the whole suite of revision control features: branch, merge, squash, rollback, blame, and time-travel. Want to reorganize the structure of the database without breaking all the applications which are using it? Branch first, make sure it works, then you can rebase master in confidence.

Time travel in action:

This is the first step towards TerminusHub linking TerminusDBs and allowing users to seamlessly collaborate on complex data. Want to share the data you are working on? Just push it to the Hub, have your buddy clone the DB and pull it down to her local machine.

Like Git, but for data.

Lots of things in the Canary Release will not work as well as we would like. There are many rough edges and a few rough centers. Let us know!

This release is for Terminators - so come join our Discord Server.

Join TerminusDB Discord

Terminators first, always.

(Quickstart is here)


Hacker News

TerminusDB featured on Hacker News! A solid history of the project and great questions from the community.

Separately on HN, there was this interesting article demanding ‘Decentralized Version Control for Structured Data’. Which was great as that’s what TerminusHub is all about.


Product Videos

We have started using this cool tool called Vyond to make videos. Really for the product folk to make sure they can see the idea of what we are building at this point in time.

Here is the first one: ‘Nathan Gets A Spreadsheet In The End'

We are getting better!


Twitch

The quality of live coding and interview content is relentless.

TerminusDB

Cheuk (TerminusDB Avocado)

Secondary mention to the great YouTube videos featuring all of the team


What’s Up?

Lots of Animal Crossing around our way and a little bit of CIV6

We’ve also invested a fair bit of time investigating collaboration tools. Our remote work stack is now:

  • Discord as a virtual office (voice channels are great for drop-in meeting and pair programming areas)

  • Zoom for external calls

  • Milanote for kanbans, boards and internal organization

  • Notion as a wiki

  • Mural for workshops and live agenda meetings

Dmytri from the core team runs a weekly remote collaboration study group on Discord. Hit us up if you want to join. We’re always on the Community Server.

Join TerminusDB Community Server


Next Episode will be the full release of TerminusDB 2.0 and TerminusHub

Bring it on.

Share

Terminators, COVID19 and Community

Our Asabiyyah Grows

This is a strange time for us all. The COVID19 crisis will leave an indelible mark – we’ll never forget the Spring and Summer spent fighting the virus. We send love to everybody out there, Terminator and non-Terminator alike. This is hard and it’s going to be hard and there is no way around that.

TerminusDB is small and tightly knit core team. We were mostly remote before the crisis hit and are 100% remote now. It has been a difficult time for us all, but we are united and hope we will grow stronger in the face of the pandemic. The primary principle of TerminusDB is asabiyyah. It’s an Islamic term described by Ibn Khaldoun in his book Muqaddimah – he calls it the bond of cohesion among humans in a group that forms a society. Khaldoun lived in Tunisia in the 14th century and observed the cohesiveness of the nomadic tribes that lived in the hinterland of Tunis. That cohesion was born out of shared struggle.

We have a saying in Ireland - Ní neart go cur le chéile - there is no strength without unity. TerminusDB hopes to grow the unity of our community through this difficult time.


So in that spirit, we are really excited to introduce the TerminusDB Community. It is great to bring everybody together in one place. Check it out here.

The beating heart of the community is our Discord Server. Since the crisis started, we’ve been using Discord and we love the platform. We can pair program, live code to an audience, we can have drop-in voice meetings - it just feels flexible and open. If you don't know Discord (I didn't until recently), I'm sure you'll find it amazing. Really intuitive and built for collaboration - gamers and streamers do it better obviously. We'll be doing mini-MeetUps with live code sessions, we'll get into the guts of terminus-store, we'll announce features and much more. We want to make the community server the beating heart of our world. We’ll build asabiyyah there.

We also have a forum, but it will take a little longer to get enough content up to make it useful (we will be putting effort in there as well as on StackOverflow etc.).  Come join us to chat on our Discord channel. All the core engineers are there and are happy to help!

Join Discord


In other exciting pivots to internet and gaming platform news, we have a twitch channel! Our DevRel Champion Cheuk is streaming on her channel too. The artwork for ‘World of WoqlCraft’ (the web object query language) is lit:

We are going to have a regular weekly webinar and podcast called ‘Terminators on Tech’ at 4pm BST/IST every Thursday.

The last couple of webinars - DataOps 101 and The Anatomy of a Knowledge Graph are available over on our new YouTube channel

As you can see, a whole bunch of new things.

Since we started TerminusDB, we’ve been hosting weekly discussions led by a different team member. They can be about anything really, but have a tendency to be tech-focused - we really value these discussions and thought the broader TerminusDB Community might also enjoy. We’ve started recording the discussions and uploading them to our YouTube channel (playlist). Blog about how it all began.

Share TerminusDB Newsletter


As I write this, I realise that it has been too long since the last one as I have too many new things to announce - we have a new blog site, we will also be syndicating to our Dev.to community page and our Medium page.

Notable blog highlights:

Why Graph will Win kicked off a furious discussion on r/programming on Reddit - some people were not happy.

Reluctantly Remote, and Loving It about our recently changed circumstances.

Great series of introductory tutorials:

My First TerminusDB Graph Visualisation - Bike Share Data

Are you Supporting the right Politician?


Finally, after all the community and communications, we have been furiously building the database. All is there to see over on GitHub. It is in great shape.

We now have a one-click deploy with heroku. We also have our updated console and TerminusHub just around the corner. You’ll be able to easily push a dataset to a collaborator (across a decentralized network) or pull a database from the central TerminusHub. Branching live databases then merging back to the master. Data like code - and a ‘Git for Data’ (white paper on what we’re talking about)

A few tasters to whet your appetite:

TerminusDB on Twitter

TerminusDB - Crazy Times

The Big Babushka

Hello Terminators!

Happy New Year. 2020 is the year of the Terminator.

It has been a crazy few weeks. We’ve been burning both ends of the candle to get TerminusDB 1.1 and supporting material out into the world.

Big news is the release of TerminusDB 1.1 yesterday - we are calling it The Big Babushka release as it is enourmous. We have a completely new backend, a new Console, extended Web Object Query Language (WOQL), a python client and a pile of new documentation.

As with everything around here, The Big Babushka is a labour of love and we are all really proud of what we’ve built. Check it out, download it, fork it, open a bug report, or contribute some code over on GitHub.

Overview blog on the release.

Remember TerminusDB is open source now and forever.


Terminus-store

The major development is our new storage backend, terminus-store. With the introduction of the new store, TerminusDB takes a radical departure from traditional database architectures. Our approach has three parts:

  1. We have a graph database with strong schema to retain simplicity and generality

  2. The graph is implemented using succinct immutable data structures which enable more sparing use of main memory resources

  3. We adopt a delta encoding approach to updates (‘like git, but for data’) which provides the whole suite of revision control features: branch, merge, squash, rollback, blame, and time-travel facilitating CI/CD approaches on data

This final point is crucial — with TerminusDB we can now use advanced CI/CD workflows in data operations. The large impact of git on the software development world can now be envisioned in the world of data. This is made possible by synergies between an immutable layered approach and the succinct data structure approach that we’ve used for encoding. The DevOps revolution becomes the DataOps revolution.

TerminusDB is a practical tool for enabling branch, merge, rollback, and the various automated and manual testing regimes which they facilitate on a transactional database management system while providing sophisticated query support.


Succinct Data Structures and Delta Encoding for Modern Databases (or How we built a Git for Data and Learned to Stop Worrying and Love the Bomb)

The other big release of the week was our technical paper on our new backend, which (as mentioned above) is like git, but for data.

It is very impressive step forward in core database architecture.

Full paper is over here.


Tutorials

We released a couple of tutorials to get you started on TerminusDB. Firstly, a getting started guide using bike share data from Washington D.C.:

My First TerminusDB Graph Visualisation — Bike Share Data

As it is election season here in Ireland (and in the US), we thought doing a little voting pattern investigation might be interesting:

Are you supporting the right politician?

Share


CowDuck

We were delighted to welcome CowDuck to the TerminusDB family. We think it is a perfect partnership. And so does CowDuck. Mwak, Mwak.

Share TerminusDB Newsletter

Loading more posts…