Doctacosa

In previous Game Dev entries, I've talked about my reasoning behind the introduction of the chat server and how I managed to apply improved formatting on the messages. The whole system has been extremely reliable since the beginning, with a solid design and self-repair capabilities when the connection between the core and one of the channels (game server, IRC or Discord) is lost. Still, there's always room for improvements.

The majority of my development time in the past week or so has been dedicated to adding an extra layer of polish and functionality to the chat system. This notably includes:

• Deaths, kicks and bans are now visible network-wide to all, helping to explain what's going on.
• An annoying display bug introduced a few weeks ago where newly registered players would still appear as visitors in-game has been squashed.
• The minigames (currently Parkour and PvP) now use the chat core as well. While their chat remains limited to the players in each location, without a global link, they benefit from the richer display formatting and custom titles. As a bonus, private messages can now be sent and received from the rest of the network!
• After erratic reports of the quick private reply, /r, sometimes sending messages to the wrong person, I decided to investigate once it happened to me and I had a small test case to study. It turns out that, if a private message was first sent without capitalizing the receiving player's name properly, later /r would go to the person contacted before then. This was rare as most players use tab-completion, which takes care of the capitalization on its own! I've now fixed this.
• Players who bounce off servers as they're denied entry (dead in UHC, or using the wrong game version) would emit annoying network-wide notifications whenever that happened. Clueless players liked to punch the buttons again and again, creating chat noise. Now, if the player's transfer to another server is denied, no message is sent.
• A small but tricky issue when using the .players command to get the list of online players sometimes caused the chat to delink, later messages being lost until someone talked on the server itself. This turned out to be an issue on how HTTP calls weren't firing from the correct thread. Let's stay light on the technical details of multi-threading...!

None of these were critical, but they're part of some of the first impression that new players get from the Creeper's Lab and its community. I think a layer of polish is important to help people understand that the various systems are stable and reliable, and so I took the initiative of getting some long-time targets and annoyances out of the way.

#gamedev #projects

– Doctacosa

Last time, I explained the path I took when it comes to player stats, and what the next challenges were. Due to our previous plugin no longer being able to work under 1.14, I had to replace it with another option. Rather than try and find something suitable, I chose to expand my own stats plugin to cover everything that'd be needed.

This was completed last weekend, as the Minecraft servers got updated to 1.14 and the new plugin took over. The way I see it, there were three critical parts to this project: track everything that we need, keep the data of the previous plugin, and make it efficient. There were some challenges along the way, so here's how that played out.

Track everything that we need I replicated everything already tracked, which is used for the display of the player profiles and the achievements, among other things, In the future, when new achievements get added, it'll only be a matter of tracking more gameplay events and save them to the database. The framework to do this is already in place now, as the saving methods are shared for all stats, making new additions relatively simple.

Keep the data of the previous plugin The great aspect of doing everything on your own is that you have full control over it, and it's designed the way that works best for you. The worst aspect of doing everything on your own is that you need to do everything on your own. And that takes time. In this case, I needed to bring in pre-1.13 item names to the post-1.13 format. For example, a “bed” of type 15 is now known as a “black_bed”. Thankfully, Mojang provided a file with the list of naming changes between the two game versions, so I could use that directly as a reference to write a converter. I had accounted for most of the challenges related to this, except for one that I didn't foresee...

The plugin in place hadn't been updated in quite a while, and its data was stored using old item names, from several game versions ago. There was no crafting table to be found, it was a “workbench”. Slabs were absents, I had “step”s. The spelling of various names was different. And so on. This meant that I had to create a whole additional conversion layer to bring items to their 1.12 format before I could bring those over to post-1.13. Not an impossible task by any means, but it took more time than I wanted it to!

Make it efficient Stats tracking involves running constant checks on all players to know what they're doing. Things like emptying buckets or changing worlds happen once in a while and are easy to do. Tracking movement is a continuous job; counting the amount of blocks broken goes quick if you've got Efficiency V and beacons to help you! You don't want to save every single individual action, as it would drag the performance of the entire server down very quickly.

What I did instead was batch all actions in groups (for example, player Doctacosa has broken 25 stone blocks and 20 dirt blocks in the Laurasia overworld so far), then push these to the database every minute. This minimises the number of calls done to the database, as we save the total values created in a minute instead of every individual action. Additionally, the saving process is done in a separate thread, so the server's activity isn't on hold while it happens. The end result is super efficient, as a sample analysis revealed that the plugin is accounting for only 0.0019% of Laurasia's workload!

The work on that is completed, the player profiles have been adjusted to pull from the new source, and so have the achievements. Time to move onward!

#gamedev #projects

– Doctacosa

My work on updating the Creeper's Lab to Minecraft 1.14 is underway right now. One of the big (bad) surprises I encountered quickly is that our stats-tracking plugin is breaking in the new game version, and there's no clear upgrade path available. The stats tracking is critical to our setup: it's part of the player profiles on the forums, the achievements targets, griefing tracking, and more. There is no easy way out of this one, and this unfortunately isn't the first time this happens either. Here's a recap of all the gameplay stats solutions I've used in the past.

Stats/TNO-Stats Our first stats plugin, selected because it had a wide range of values being tracked, in a simple format, and was easy enough to use. Initially, nothing relied on this, but I integrated it early on so we could record as many gameplay events as possible.

BeardStats Quickly after that initial setup, the first Stats plugin went abandoned. Someone else made a follow-up that was compatible with the same structure, so I migrated to this. The initial version of key features like the achievements were made using this one.

Vanilla When Minecraft 1.8 rolled around, a lot of functionality was rewritten, breaking a lot of compatibility. Unfortunately, BeardStats stopped getting updated around that time, so I had to look for another option. The Mojang team had implemented their own vanilla statistics, tracked server-side and viewable through your game menu. I rewrote everything to support this. However, at the next game update, I realized that Mojang's implementation caused any stat with an ID change (for example, roses becoming poppies) to completely ERASE the old values and start counting anew, which wasn't acceptable for our uses. Another change was needed.

Stats2 At the time, Stats2 was the leading plugin to track player events, so I once again rewrote all my logic to be based on this. A negative effect was that all stats had to be counted from 0, but it seemed like a necessary evil to get a stable base to work with.

Stats3 The author of Stats2 created a new version, Stats3, with a different data structure and better performance. Thankfully, this one properly converted the values already stored to the new format. While I had to do some updates to the processing of achievements to account for the new format, this was a rather smooth change.

IOBattleStats Stats3 didn't count everything I wanted for the sake of some achievements, like the famous “Sailor Scout – Defeat an enemy with the power of Love”. To complement it, I wrote my own plugin, focused on a more detailed tracking of any damage taken or dealt (to both mobs and other players), along with death causes on all sides. This took care of PvP and PvE events, while everything else still relied on Stats3.

And that's where we stood, up until 1.14 showed up. My initial tests revealed that Stats3 was completely incompatible with the new game version, as it was still trying to use numerical item IDs instead of names. Its author stopped updating it, and instead chose to start building Stats5, once again incompatible with the previous versions, and this time with no data converter available. That'd have meant restarting counting the values from 0 all over again, and that's where I decided to draw the line. No more!

I figured that, if everyone on this front is going to let me down every few years, I might as well do it myself, so that's what I'm doing now. IOBattleStats is being expanded on to track all relevant gameplay values, from block breaking to minecart distance passing by portal uses. I'll also write a converter from Stats3 to IOBattleStats' own data format, so that everyone working toward specific achievements don't need to restart from zero. Then, everything based on these stats (see above: player profiles, achievements, griefing tracking) will be updated to the new data format, and this time should finally be the one.

This takes more time to do upfront, but will save me from having to rewrite significant parts of the existing systems every year or two, so I see it as very much worthwhile!

#gamedev #projects #plugins

– Doctacosa

I've written before on how updating the guts of the various systems takes time and needs to be done carefully.

I had another example of that last week: while doing a basic code review on a page that displays the server logs, I realized that some of the function calls I used will be going away under PHP 7.2. The PHP language has been taking several steps in recent years to improve its design and performance and that gave some very nice results. The downside, of course, is that some older functionality is being removed outright, so scripts and programs need to be adapted accordingly.

In this case, the fix was rather easy but critical, as it's related to fetching files remotely. Without that, various web tools would break when the time came to update, including the achievements check. Fixing this now removes an obstacle from the future, avoiding bad surprises while ensuring smooth operations for everyone!

– Doctacosa

With the bulk of the Minecraft 1.13 update behind us, let's look back at an unexpected issue I faced while moving between the two dedicated servers.

I used the opportunity of migrating the software on a new server to optimize the various config files a bit. One of the changes I did for the Minecraft servers is to have them running on localhost only, as they don't need to be visible to the outside world: only Bungee, the point that ties it all together and is used by the players to connect, needs to be accessible. As such, instead of having the game servers use a public IP address, they only run on a private one. An added benefit is that this matches what I'm using on my PC to run tests, so I can move config files back and forth without editing anything!

A problem I discovered, however, is that this only exposed the related services to the local environment as well. It's possible to use special calls to do things like get the list of players online, the maximum amount of players allowed, or even run commands remotely. However, if the ports for the services aren't opened to the outside world, it's impossible to use them! Things as simple as displaying the list of connected players on the website would no longer work.

To work around this, I ended up writing a simple PHP page on the dedicated server itself that waits for requests. Whenever one is received, it translates it into a game server command and runs it before returning the result. It's able to make the connection as this page is visible both internally and externally, acting as a bridge between the two worlds while keeping the security model intact!

With this new setup, no game ports are visible to the outside. Once in a while, I'll notice that random people (script kiddies?) try to scan the dedicated server for weaknesses. As of now, nothing is opened or accessible without the proper authorization, so these attempts won't even connect!

#gamedev #projects

– Doctacosa

As I've alluded to in my past two updates, I've been dedicating a lot of my time recently on background updates. Part of what allows everything to keep on running smoothly (that prized long-term stability) requires me to review and upgrade some components over time. It's thankless work that's usually completely invisible, but it needs to happen to prevent downtimes and disasters! Here's a quick tour of some key elements that I've been focused on lately.

PHP 7

Everything I've got available on the web runs on PHP. The version currently running on the servers is 5.6; however, PHP 7 has been available for some time now. Upgrading isn't a simple matter of switching versions, as 7 comes with a lot of improvements along with retiring old, outdated methods that I still rely on in several places. Notably, all database connections and queries need a review.

I've chosen to do this now rather than later so I don't get surprised by a forced update announced on short notice. I'll be able to make the change on my own schedule. Some elements, like the website for the Creeper's Lab and its control panel, have already been silently updated about two weeks ago. Others, like the forums, are still being worked on. I've got an internal prototype of the latter up and running.

Dedicated server

The dedicated server that hosts the Minecraft and Starbound servers is currently running Ubuntu 14.04 LTS, which is four years old at this point. It's still supported and receives security fixes, but it's time to bring it up to a more modern version. I'll be doing that next Sunday, taking all services offline for a few hours (at least) so that the dedi ends up running the latest and greatest 2018 version of Ubuntu. Sticking to the LTS (Long Term Support) editions allows me to go through this only every few years rather than sticking to their twice-a-year schedule.

Minecraft 1.13

That's not available yet, but it's coming, and soon. Part of the reason why I'm doing the above right now is to clear the way to properly handle the new Minecraft version when it gets released. Between all the new content and the technical updates it holds, it's likely to require a lot of plugins work to get fully working and stable, so I'm clearing up everything I can before we get there!

– Doctacosa

In the same vein as last month's post about backups, I also need to manage the disk space taken by the various servers. Laurasia is the biggest concern, as it is the largest server by far. A world border was introduced relatively late for it, so people already had time to explore far and wide. A large circle now delimits the areas available for building, but there's a lot of unvisited areas still.

Once in a while, I'll take Laurasia down for several hours and set off to delete areas that have been accessed yet never built, or that have been left alone for a long time. This makes it easier for the players to find new, untouched places to explore while it also decreases the disk space used by the server. It's a slow process that needs to be handled carefully, but the end result is worth it.

Of course, it's important to avoid accidentally deleting players' creations! As I'm preparing a new trim that should be announced soon, I developed a tool to help me identify what can be done, and I decided to give you patrons an idea on how that looks!

Here's a special version of the map for Laurasia – overworld (link removed). It's generated off the images created by DynMap, to which I've added a layer that displays the coordinates of each server file along with the amount of days that a specific area has been untouched. That data is fetched from the servers files themselves: just like any computer file, the system automatically tracks when they were last modified. A simple color code helps to preview the information:

• Green: recently updated
• Yellow: untouched for a year
• Red: untouched for 2+ years

For something that's easier to see and understand, here's Rodinia's overworld (link removed), which is much smaller. The North-West, South-West and South-East corners haven't actually been visited for about 200 days, while the core areas see a much higher rate of updates.

Going back to the Laurasia display, it's now easier to notice some areas that are good candidates at being deleted that might not have been obvious otherwise!

#gamedev #projects

– Doctacosa

Backups. Always important, and always neglected, right? Minecraft tends to be stable... most of the time, but it also runs into its problems: chunk corruption, unexpected crashes, large-scale griefing, or even the server disk filling out. As such, it's important to keep backups on a regular basis – they've been useful before, so not doing them isn't even an option.

For a while now, my main backup procedure consisted to having a copy of each separate world compressed and stored away, waiting to be used. If the file hasn't been fetched within a few days, it'd automatically get deleted. I'd make sure, once in a while, to download a copy of everything to my own computer as well. I used to keep several days worth of backups on the server at once, but as the worlds started using more disk space, I've had to scale this down quite a bit.

In the meantime, I've learned that my host has a free backup location available for each account. To prevent people from using this to simply extend their disk space, this backup spot is only available through FTP, so you need to connect to it explicitly to upload and download files. I decided to make use of this, and wrote a new Python script to automatically run on a regular basis.

The new process is simple:

1. For each world: a) Get a copy of the world and compress it. b) Upload the compressed copy to the backup space. c) Delete the local archive.
2. Backup the database the same way.

Should I ever need to fetch a file from storage, this simple command downloads it back into place:

python3 backup.py -d FILENAME

Moving the files to a different storage location has the big advantage of mitigating risks: if the main dedi's drive fails for some reason, I won't lose both the live games and their backups at the same time.

Another motivation for this improvement is the simple fact that the dedi's disk is starting to fill out. I've got 100 GB to work with, and between the Minecraft worlds, the Starbound universe and various web sections like the images gallery, it's starting to fill out nicely. Ideally, I'd have two servers or drives: one powered by a SSD to run the games, and one on mechanical drives to have a lot more storage. We're not quite there yet, though, and I need to work with what's available. Keeping the backups off location will help to save on disk space as well as make things better organized.

Got ideas on what I should write about in these game dev entries? Please comment with your suggestions!

#gamedev #projects #minecraft

– Doctacosa

Once in a blue moon, I'll go out on vacations, with limited to no Internet access. Of course, during that time, I want things to keep running smoothly for everyone. The websites and forums do fine, as they're stable and need little maintenance. The game servers are another story! I make it a point to keep things running even if I'm not available, but at the same time, you can only plan so much for the unexpected. To keep the odds in my favor, I avoid making any major changes right before leaving. Sometimes, this will go as far as delaying version upgrades. I don't like doing this as waiting too long to update versions can cause us to lose some players, but at the same time, I feel that overall stability is more important.

If something breaks

That's when things get interesting. I keep an eye on things, of course, but not as often when I'm busy visiting somewhere. The moment I either notice that something is not right, or get a notification from another of the staff members, I jump in however I can to set things right. This can be through any available wi-fi hotspot, data access through my website, or even sometimes through someone else's computer.

I've installed the required tools to connect remotely through my computers, my tablet or my phone, so I can login to the dedi and run any commands necessary. If it's something more minor, any web browser will allow me to check on each server through the consoles.

The northern Canada example

A fun example of this happened back in 2013, when I took a long train trip to the western coast. A mere day or so after leaving, Laurasia went down without warning. Nowadays, if that happens, people can easily move on to other servers through the lobby. At the time, though, Laurasia was the entirety of the Creeper's Lab, so losing it was pretty critical. There was one big problem for me: I was in the middle of nowhere. I barely received a text message notification that the server was down, then I promptly lost all network connection to my phone, even for calls. Cell coverage in Northern Canada can be quite spotty...

I had to wait about two full days for a longer stop in that trip, where I was able to find a public wi-fi hotspot. From there, I logged in to the server's web interface and restarted it. Thankfully, it went back online without a hitch and remained that way for the rest of my vacations!

– Doctacosa

As you've seen by now, the custom ranks feature has been put online thanks to the latest wave of improvements to the chat server. To allow this to take place, the messages are now generated by the chat server itself and sent pre-formatted to all Minecraft instances; these only need to display it through a /tellraw command. A side effect is that all public chat now needs to be relayed through said chat server: if it fails for any reason, including bugs and crashes, players in-game will no longer be able to view messages at all. Thankfully, the system has proven to be solid and reliable since it first came online, so I'm comfortable with this change.

Now, how does this works? That chat server maintains a direct connection to the MySQL database, so it's able to read the access level of each player (admin, op, helper, ...) along with their status (contributor, Patreon backer) and any custom titles set. It takes that information, the received message, and puts it all together for display.

There are still a few aspects that I want to add now that the foundation work is solid:

• At this time, the chat sent from Discord and IRC only displays the [Discord] and [IRC] tags respectively. Why not go the extra step and have them display the actual rank of the chatter? We already have matching groups on Discord for all staff positions, so that could be reflected in-game. Likewise, authenticated IRC users could be identified based on their user status (aop, hop, etc.).

• One thing I want to investigate is to allow people to /msg others cross-servers. I already overrode the standard chat and /me actions, so I believe that /msg is doable as well. This would make it more convenient to stay in touch with others while hopping around worlds and servers.

• As a longer term target, possibly have death messages carried over to Discord and IRC. That would provide more context to the chatters; how many times has someone said something like “Why are you yelling? Oh, creeper got you. Nevermind!”? Carrying the cause of kicks could also help the staff identify people who might be using flight mods. And hey, if I can read and propagate death messages, why not replace them with some custom ones at the same time...? Maybe even get some community input on new messages to make things interesting! How does that sound?

#gamedev #projects #minecraft

– Doctacosa