SpacetimeDB and BitCraft
This is going to be a relatively short blog post! I just wanted to talk to the BitCraft community a little bit about SpacetimeDB. In this post, I’m going to explain what SpacetimeDB is, what problems it solves for BitCraft, what it means for BitCraft development, and how it should allow us to do super interesting things with player mods and game extensions in the future.
What is SpacetimeDB?
SpacetimeDB is the server-side game engine that BitCraft is built on. You can think of SpacetimeDB as both a game server and database combined into one.
The normal architecture for MMORPG server-side backends is to have client applications (the player’s machine) connect to game servers (big computers running in a data center) which run the logic of the game simulation and control how and what data gets stored in the database (other big computers running the same data center). The data stored on the game servers is wiped when the servers restart (i.e. the RAM of the MMORPG) and the database stores all persistent data (i.e. hard drive of the MMORPG).
SpacetimeDB turns this architecture inside out. Rather than game servers sitting in between the clients and the database, in SpacetimeDB you upload your game’s logic into the database itself and have clients connect directly to the database.
Okay, so what is the point of doing this?
Well, SpacetimeDB is a “serverless” platform. This may seem like a silly word to use since it runs on servers, but it’s actually an important point. What “serverless” means is that the logic of the game does not reference or even know about the fact that it is running on a physical server machine.
In a sense, this is the same thing that your PC operating system does. It makes it so that programs that run on your PC don’t have to know about the specific hard drives or RAM modules or network cards that exist on your computer, and thereby dramatically simplifies the task of writing PC programs.
In that context you could think of SpacetimeDB as being a distributed cloud operating system. It takes the many machines, devices, and systems of a cloud datacenter and makes them look like a single giant logical computer.
The point being always to simplify the job of the developer. Because your game’s logic is running inside the database itself, the database can act like an operating system. Rather than having to wrangle a vast array of game servers and databases, you instead just need to upload your program to SpacetimeDB and that’s it.
How does it help BitCraft?
That’s all great, but how does this help us develop our game, BitCraft?
SpacetimeDB solves seven MMORPG game development challenges, each of which could otherwise cripple a game development team of our size.
- Persistence
One of the things that makes MMORPGs so difficult to build is that so much of the data of the game world is persistent. Typically persisting data is a complicated and finicky process involving connecting to an external database like Postgres or MySQL and sending data back and forth from the game servers. What makes it so complicated is that the game programmer always needs to be synchronizing data stored in the game server with data that is stored in the database. Serializing that data back and forth with the database is not only awful for performance, but it’s an enormous source of bugs, since it’s not clear what data needs to be saved and when. If the programmer is not incredibly careful about using database transactions then you can easily have game breaking bugs like gold dupes or other nasty data corruption or data loss issues.
Additionally, BitCraft has waaaaaaaaay more persistence than your average MMORPG. Because the whole world is editable we need to persist everything from the player inventories to the positions of buildings and resources to the height of the terrain itself.
In SpacetimeDB all data is persisted by default.[1] That means BitCraft developers don’t even have to think about this problem at all. Gone. - Permissions
MMORPGs are essentially complicated 3D rendered social networks, and therefore have all of the complicated permissions issues that arise when implementing a social network. Facebook/Meta for example has spent an enormous amount of engineering resources to make it easy for their developers to specify the rules for what users are allowed to do to manipulate the social graph.
We take inspiration from the engineering done on both social networks and blockchains where permissions are also critical to the safe functioning of the system. In SpacetimeDB, for any function that is accessible to clients you can assert that the caller of the function has permission to do the thing that they’re trying to do, or otherwise fail the transaction and rollback any changes that were made.
This paradigm means that complicated access control lists and server side logic become a simple assert at the start of your function. This vastly simplifies the structure of our code for BitCraft. - Deployment
SpacetimeDB simplifies deployment massively for the BitCraft developers. All they have to do is run `spacetime publish bitcraft` and the game server is deployed to the cloud. They don’t ever have to think about game servers or web servers or networks or docker or kubernetes or databases or any of that.
On top of that they will be able to redeploy fixes to the server code in real-time while clients are still connected to the game! It’s a real game changer. - Indexing and memory management
A key aspect of performance in any game (and indeed in any program at all), is laying out your data in memory in a way which optimizes the use of the CPU cache. [2] Another key aspect is being able to access that data quickly via the usage of indexing techniques. In SpacetimeDB, this is built into the database so that the programmer doesn’t have to spend time implementing this themselves. - Multithreading
Multithreading and concurrency control is notoriously one of the most difficult aspects of computer programming. With SpacetimeDB, the application developer doesn’t need to think about it at all. Instead they can write normal functions and let SpacetimeDB employ 60 years of database research to run their functions in parallel.
This is something that databases like Postgres have been doing for years to be able to achieve hundreds of thousands or even millions of transactions per second. Now this power can be brought to bear on your game server logic as well and for free no less. - State synchronization
Yet another complicated aspect of developing an MMORPG is synchronizing all of the state of the game world with client applications. To burden every game developer with having to think about what state to synchronize with clients grossly slows down development of the game. Game developers should focus on how to implement game features and make them fun, not how to ship data around to clients.
SpacetimeDB handles state synchronization automatically for developers. The only thing that game developers need to do is specify a set of queries for the data they would like to be accessible in client programs and SpacetimeDB does the rest. - Error handling
Perhaps the most important aspect of SpacetimeDB is its ability to maintain correctness in the face of errors. All logic that the game developer writes takes place inside an atomic transactional context. This means that if there are any errors or anything goes wrong, none of the changes are committed to the database. It is atomic in the sense that it is indivisible. It either all happens or it does not.
Consider the case where you are executing a trade transaction and only half of the trade is committed when an error occurred. Let’s say the items were transferred to the other person, but they were not yet removed from the senders inventory. Now you’ve got a gold dupe bug on your hands. Atomic transactions eliminate a whole class of bugs that BitCraft developers just don’t even have to think about anymore.
Does it slow BitCraft development?
Each of the above problems are significant challenges that, with the advent of SpacetimeDB, the BitCraft engineers basically don’t have to think about at all. This is the reason that SpacetimeDB is so critical to the development of BitCraft. It separates the concerns of the engine developer from the concerns of the game developer. This allows us to have entirely separate teams for the game and database. The BitCraft team is 100% focused on BitCraft and the SpacetimeDB team is 100% focused on SpacetimeDB, which allows us to build both in parallel even though BitCraft has a dependency on SpacetimeDB. Every bit of effort that goes into SpacetimeDB makes both BitCraft and the SpacetimeDB product better.
There is of course a cost to developing a product which you are selling externally rather than just using as an internal engine, however 80% of what we’ve worked on developing this product is stuff that we would have needed for BitCraft anyway and having the product be battle tested by a whole set of third parties just makes the platform even stronger.
Our goal for SpacetimeDB is to make it so that an indie developer can create an MMORPG or at least a large multiplayer game easily. It just so happens that one of those developers is the BitCraft development team and it’s already been a game changer for them in terms of speed of development.
Does it fix all our problems?
There are many additional challenges in developing BitCraft that SpacetimeDB doesn’t solve. Creating an MMORPG, it turns out, is really difficult.
One thing SpacetimeDB does not really address is the problem of having an infinitely scalable single world for players to play in. This is because each SpacetimeDB database is currently [3] limited in size to fitting in-memory on a single machine. Achieving this in BitCraft requires that BitCraft actually be implemented as a set of many SpacetimeDB databases which handle a spatial partition in the world. This aspect of BitCraft must be implemented (at least currently) on top of SpacetimeDB and not within it. This part of BitCraft deserves an entire blog post though, so keep an eye out for it in the future!
Why does the SpacetimeDB website reference Ethereum and blockchains?
SpacetimeDB shares several concepts and data structures with smart-contracts and blockchains. If you think about it, the idea of a SpacetimeDB module is actually quite similar to the idea of a smart-contract, except we don’t have decentralized consensus, poor performance, cryptocurrency, fraud, scams, and with any luck we can skip cryptobros too. There’s actually some really powerful ideas there and no the data structures they use are not inherently evil. Indeed Git, the source control program [4], employed Merkle trees long before any blockchain did, and Merkle trees are what makes blockchains chains at all. The fact that blockchains share some of these architectural concepts and ideas of course doesn’t imply anything about the direction we’re going in with BitCraft from a game design standpoint.
Fun stuff for BitCraft
While SpacetimeDB does vastly improve our ability to create a game like BitCraft with a small team, it also allows us to do certain things that would not just have been hard to do without SpacetimeDB, but would have been impossible.
One such thing is player created add-ons and mods. With SpacetimeDB, there is no reason that we can’t allow players to program their own modules which interface with the game and build on top of it. One example I give for something like this is player programmed guilds. With SpacetimeDB, when a player applies to join a guild, Clockwork Labs could call out to a guild module created by players to check to see if that player should be allowed to join the guild. You could check that they’ve reached a certain level in a skill, or that they’ve been to a certain region, or that they have a certain item.
It starts to get pretty silly when you think of the types of things that we could allow people to create in the game. User-generated content is all the rage these days, but we’ve never explored what we could do with user-generated logic!
— Tyler (3Blave, Cofounder of Clockwork Labs)
[1] Although all data is persisted by default, it is processed in a low latency way in computer memory, so don’t think of it as being a long slow database transaction. We use a write-ahead-log to achieve this. Incidentally, persisting everything ends up being less data than snapshotting if you want to store historical data for data science analysis.
[2] I think every programmer who is interested in writing high performance code for modern CPUs should read this book: https://www.dataorienteddesign.com/dodmain/
[3] This is not an inherent limitation of the SpacetimeDB architecture, just a temporary one that we have chosen specifically because it’s important for the performance of BitCraft. In the future we will allow users to specify that tables can be stored on disk or distributed across multiple computers. In this case the data will be larger, but the latency for accessing them will be much higher.
[4] Git is, in my humble opinion, the most important and well designed program ever written. It’s on my list as #1 best program, followed by Linux at #2. It’s hard to think of a #3. Open to suggestions.
