Alright, it’s time for a blog update and this one might be a bit of a doozy but here goes nothing. Originally, I wanted to do a post of a rant about some “micro optimizations” being rejected from Mirror, but that’ll be another day.
For those who aren’t familiar with Mirror Networking, it’s a network library that’s free and plugs straight into Unity, and it just so happens that I’m part of the Mirror Networking development team.
We’ve had the usual share of drama, some internal development conflict that is always present in every development group where you butt heads with each other. And the only way to overcome those matters is backing down or making a compromise and approach problems a different way.
However, this issue that I’m about to discuss revolves around how the networking logic is handled and what can be done, from my point of view, to improve it.
Now, in Mirror Networking, there’s two versions of Mirror. One is Mirror which GitHub and Unity Asset Store releases are created from, and then there’s the cutting room floor that is MirrorNG where Paul, one of the developers uses for his own game (Cubica).
MirrorNG is supposed to be “Next Generation” and is refactored heavily, however since it’s literally a cauldron and unstable, there’s very little user base using it. To be honest, and this is going to sound harsh, but I don’t really care about MirrorNG.
That’s Paul’s baby and if he’s busy chopping limbs off it and stitching up improved or rewritten components to replace those limbs, then good for him. To me, it feels like it’s being developed with only one person in mind (himself). The same thing happens with the Master version of Mirror. Unfortunately this leaves me with fog in my mind that the “lead” developers aren’t thinking of the boarder community for networking and instead are focusing too much on their own needs. This is apparent when micro-optimizations that would enhance and speed up the core get rejected – it’s rejected because of the needs/wants of the developer who pulls the trigger and merges the PRs.
In short, sometimes the “lead” developers have a goal that isn’t in the best interests of the other development team members.
For what I do though, my Mirror tree is forked from the master branch; which is mainly where the development team improves, fixes bugs and accept PRs into. For clarity’s sake, we’ll be focusing on the master fork, which is the “original” in this post.
So, let’s get into the meat and potatoes of this blog post how the network is pumped. Disclaimer: I am not a networking professional and I could be talking out of my ass here. Feel free to correct me in the comments.
Right now, Mirror’s network operation is mainly done inside LateUpdate, which if we look at the Unity Script Life Cycle, we can see it’s the last thing in the stack before the Scene Rendering phase of the lifecycle begins.
However, if we look at the definition of LateUpdate from the Unity Scripting Reference:
LateUpdate is called every frame, if the Behaviour is enabled.
LateUpdate is called after all Update functions have been called. This is useful to order script execution. For example, a follow camera should always be implemented in LateUpdate because it tracks objects that might have moved inside Update.
That alone should tell you that Mirror is doing all the network processing in LateUpdate. Which means that Mirror is always 1 frame behind the eight ball, since the actual scene may already have changed before it goes and processes the network data to keep everything in sync.
I’ve spoken to other network engineers including one member from Flying Squirrel Entertainment and another networking guru who currently works at a director-level in a company, who both agree that LateUpdate is too late (hah, see what I did there?) for networking.
They suggest to both do FixedUpdate processing for networking, so that the networking side of things is done before the Rendering side of things takes place.
So, what’s the fix?
Well, my proposed solution is to harness the power of FixedUpdate, and if possible, use PreFixedUpdate and PostFixedUpdate to process and apply things to the scene, then let the rest of Mirror do its job. This also makes sense for networking physics objects too.
You want to ensure the network is running at a constant steady rate, as LateUpdate is very frame-rate dependent and you’d get network slowdown in the event you start shitting frames in your game instance (i.e. 60 FPS one moment, then 5 FPS the next because the game is loading something). That spike can cause you to get a “turbo boost” as the network tries hard to catch up.
When talking to vis2k and Paul about this, vis2k replied: “unity uses catch-up fixedupdate, meaning that it gets called multiple times if system is currently lagging behind”. His argument was that “an mmo server that is temporarily under heavy load could end up in a deadlock”. Fair enough, because it would always be playing catch up if it was lagging the current frame due to whatever reason. Paul seemed to be silent and/or not interested into entering into the conversation. His choice.
After more discussion between other developers, it was apparent that they weren’t going to budge with my idea and my idea was shelved. However, I wasn’t giving up so easily and thought, how would this be solved?
Solving the deadlock
Let’s solve the deadlock – one solution that both network engineers I talked to agree on, and from what I can see is the easiest solution, is to set a Boolean that controls if the network will be processed this frame. Let’s call this new Boolean networkIsDone.
This Boolean would be set to true at the end of the FixedUpdate to signal that the network has been updated, and any same-frame additional FixedUpdates due to the game lagging behind are skipped, by checking to ensure networkIsDone is false before doing any networking related duties.
In LateUpdate, networkIsDone is reset to false, so the next frame is ready to receive network data. LateUpdate again is called once the frame is done, and since one frame might take a long-ass time to render, networking shouldn’t be updating continuously as that’ll cause the deadlock.
So, now with this implemented, in theory networking would be now based on the FixedUpdate cycle. When FixedUpdate comes in, it immediately checks to see if we’re done with the networking for this frame. If so, skip doing anything related to the networking. If not, then continue and update the networking subsystem and apply changes to the scene. Finally, get all the shit it needs to out the door to the transport mechanism, ready for the next round.
Afterthought
Of course, this is far from the magic bullet that’ll have to solve everything. I would have to go though and ensure that Mirror itself can adapt to these changes and see how deeply rooted LateUpdate is inside Mirror’s core.
But for me, I personally want to see Mirror become a more tick-based networking solution, as it’s a great Networking solution. However, on that note, we can’t always be a frame behind.
Maybe in an MMORPG that’s acceptable, but for my game which is a shooter, a lot of things can happen in one frame.
As always, thanks for reading and let me know what you think in the comments. Stay safe.