I’ve yet to receive any meaningful response to my first post on how to build certain gameplay setups in an ECS system. This is a follow-up post with more questions but a simpler use case that once again highlights issues with the core model when it comes to gameplay object dependencies.
The scenario is as follows, we are building a futuristic tank game which offers the ability to customize the weapons on your tank. You can swap out the main turret as well as the secondary turret mounted on the main turret. An example of this sort of setup is shown below:
Assuming the common “only one component of type X per entity” ECS model, the tank would be broken up into 3 entities: The tank chassis, the main turret and the secondary turret.
The main turret is attached to a bone on the tank chassis skeletal mesh and the secondary turret is attached to a bone on the primary turret skeletal mesh.
The two turrets operate independently
The gameplay operations we need to perform each frame per entity are:
- Tank Chassis
- Perform basic movement i.e. move from a to b in the world
- Resolve physics collision for chassis and update (tilting of mesh)
- Perform animation update for tank
- Perform IK pass so that tank threads match environment collisions
- Main Turret (assuming it’s tracking a target)
- Calculate aim vector needed for turret to aim at target
- Perform animation update and IK to update turret and gun positions
- Secondary Turret (assuming it’s tracking a target)
- Calculate aim vector needed for turret to aim at target
- Perform animation update and IK to update turret and gun positions
Seems like a relatively simple problem right? The operations for the two turrets are identical so we can probably re-use the same system for them. So let’s try to model this in an ECS…
Firstly what are the dependencies?
We have to fully complete the tank chassis entity update (or at least every aspect of it that updates the transform of the mesh) as well as the animation update in-case the turret bone is animated and has moved.
Only then can we even begin to calculate the aim vector of the main turret, since that turret’s position is based on the chassis world position, so we cant do anything with the turret until we are fully finished with the chassis.
The same is true for the secondary turret except our dependency is on the primary turret. Now this is where things get interesting. The code for the target vector calculation is the same for both turrets. The anim update and IK pass code is also the same for both turrets. Let’s assume we have components for targeting (containing horizontal/vertical tracking speeds, etc…) and for animation (anim graph, IK rig, etc…). We then have a system that calculates the target vector for a turret and forwards it to animation. The problem is that we cant update all turrets sequentially, we have an intermediate spatial dependencies we need to resolve.
Here are threes approaches to performing the system updates for the above example. This all assumes that the tank chassis was updated first! And I’ve left out all the spatial transform update stuff.
Option A is plain BROKEN, the position for the secondary turret will be wrong since the primary turret hasn’t updated. Order of updates of components internally in the systems doesn’t matter since the result will be wrong no matter what.
Option B will produce the correct result but requires us to run the same systems multiple times. Furthermore we need to identify which components go in which update so we can create a “dependency component” that specifies that an entity (secondary turret) has a dependency on another entity (main turret). In a production code base we would probably need several dependency component types since we might have multiple different dependencies (spatial, data, etc…)
Option C is a potentially cleaner version of option B, here we remove the need for dependency components and instead create a new component type and a new system type (probably through inheritance) just so that we can order the updates of the systems and components separately. This will work and if we have lots of tanks we will always update the main turret before the secondary turret and we can do all the updates sequentially. It also means that we may have an explosion of component and system types as our gameplay scenarios grow.
As we end up duplicating systems and component types, we start to negate the supposed performance benefits of ECS. You know the mythical: “We can update all our stuff sequentially and the cache god will bestow amazing performance upon us”. Reality is that the targeting system will probably do more than just calculate a vector, it will probably have to check all targets available for a given entity and then find the best one based on some parameters (distance, angles, type, etc…). This will incur cache misses per entity target update, why that is should be obvious so I’m not going to go into it here.
Furthermore there is also the fact that someone will have to constantly maintain and reorder the system updates once you go down the route of either A or B. As the various gameplay rules and exception start trickling in, my gut feeling is that there will be an explosion of components/systems which will become unmanageable at some point especially which still in the pre-production/prototyping phase of the project.
Is there a better solution for this?
Taking Initiative 
If you perform systems’ updates eagerly you will always run into problems with execution that should be interleaved with another system’s update. The naive approach is to try and split the update calls into several phases but this will never scale. There can be an arbitrary amount of these dependencies and no fixed amount of execution phases is guaranteed to be enough for your use-case. The main issue here is that we are trying to push responsibility of handling the execution dependencies on to the ecs implementation in the first place. Ecs is very bad at handling relationships within the data and has no concept of execution dependencies.
A more proper way to solve this is to not run the system updates eagerly, but define the update of a system as a step that constructs tasks to your scheduler and define the data+execution dependencies of the tasks. You perform no updates during this step. You first construct all the tasks and dependencies, and when the description of what must be executed is complete, you can start your worker threads to crunch the numbers.
So use the scheduler to solve execution dependencies, since that is what the scheduler is supposed to do. Use the ecs to access your data when you know it is proper to do so.
You can pass an execution barrier from main turret system to the secondary turret systems, and condition the execution of secondary turret update tasks on the barrier being complete.
I agree – Method A is broken. Method C seems unscalable – and can still be broken by data. Method B is basically the only way to go.
Game Engine Architecture 2nd talks about this in 15.6.3 “Object and Subsystem Interdependencies” — and specifically 15.6.3.2 in which they assign a bucket to a set of components based on where they are in the hierarchy. (Admittedly, it discusses “game objects” and not “components”, but the idea applies)
Chassis of your tank would be in bucket 0 (assuming it has fixed gun and does targeting), main turret in bucket 1, secondary turret in bucket 2, and the soldier sticking his head and gun out the hatch and firing at enemies would be in bucket 2 (if hatch is on the main turret).
Then run your component/systems like this in your game code:
foreach bucket {
TargetingSystem.update(bucket)
TurretAnimSystem.update(bucket)
}
Now think about the “many” case here. If you have 100 tanks, the components of all 100 chassis will be updated in bucket 0 coherently. The main turret of all 100 tanks will be updated in the bucket 1. Also key is that anything that is not a tank and also has a targeting system will also be updated in bucket 0.
You’d want to ensure that components that are of the same bucket are packed sequentially in memory so that the cache god will continue to bestow amazing performance upon us – at least in so much as data cache is concerned.
The instruction cache coherency is regrettably not something you can do better at here as you will end up re-fetching/decoding instructions across buckets. But it really isn’t a penalty compared to any other implementation (such as the atomic entity approach) that implements the same correct behavior too.
There are a few tricky things with this approach though.
– keeping components in the same bucket contiguous with each other could be tricky, but could be managed with a single array with locators for the start of each bucket within the array. Using some padding and per-bucket counters between buckets will allow a bucket to shrink/stretch without necessitating moving large blocks of memory when adding a single component — and even then you could with prior knowledge grow by N items at a time, which could mean a move of at worst N*B items (and remapping their component handle/map/whatever you use for tracking them)
– assigning a bucket for a component will require some knowledge of your hierarchy at spawn time
– reassigning buckets dynamically imply memory moves (eg. due to that soldier deciding to get out the hatch and start firing while standing on the chassis)
Ultimately there will be a practical limit on the number of buckets — and even then you’ll see the majority of your components will be processed in bucket 0, less in bucket 1, less in bucket 2 etc. so you can optimize your solutions to the tricky things according to the data frequencies you actually see.
Would really love to see Mike Acton’s response on this question 😉
You hit all the issues exactly! I completely agree with everything you said.
There’s a lot of caveats and lots of complexity introduced for a relatively simple use case, I would also love to see Mike’s response to this.
“The instruction cache coherency is regrettably not something you can do better at here as you will end up re-fetching/decoding instructions across buckets.”
I just wanted to follow up on this point because there’s something subtle in there that probably needs to be explicitly said. The bucketed ECS case behavior is likely significantly better than monolithic atomic entity model in terms of instruction coherency as the number of dependent systems increase.
In the ECS case you are processing multiple items for each bucket, so the system instruction footprint ends up being loaded once per bucket (assuming your system code is small enough to fit – and it should be) and therefore once for N items in that bucket.
In the monolithic entity case, you end up loading your instruction footprint for all your systems for each entity that is processed.
For each entity {
Targetsystem. Update (entity)
TurretAnimSystem.Update (entity)
}
This is the thing, for the bulk of actual gameplay code, we don’t often have 1000s of the same entities that all require the exact same update code. The target system update specifically, would probably be relatively complex in terms of what targets are available, how much aggro do they have, do I have line of sight, what are preferred targets… then lets sort that list/pick the best one from the list, etc…
Then in terms of the animation update. well every turret could use a different anim graph, a different IK solver, maybe a different set of procedural bones to solve secondary motion since the mesh might be different, etc… Realistically it feels like I would have flushed the I-Cache long before I get to the next component in the system’s list.
I’ve yet to see the case I can run the exact same 100 instructions on every game object in a real world game outside of a particle system or something super trivial. In a real tank game, there would also be a whole set of other parameters to look up or calculate to feed into the anim system and targeting system -> Angular velocities, firing arcs, etc… Those would likely be different per tank and have to be either looked up or calculated. Once designers get involved the complexity and data management grows exponentially, especially in AAA, so it definitely should not be underestimated.
“This is the thing, for the bulk of actual gameplay code, we don’t often have 1000s of the same entities that all require the exact same update code.”
“I’ve yet to see the case I can run the exact same 100 instructions on every game object”
That’s my experience too, and not an argument I’m making for the record.
I also don’t disagree that data driven systems with unbounded flexibility can have the characteristics that you mention. My career has had me exclusively working on proprietary code bases and I’ve seen that a lot.
Gameplay code, and AAA animation in particular are notoriously leafy and hard to put in a nice tidy box (You should see the tomb raider anim graphs and state machines – complex because of the rich behavior set they encapsulate)
And yet, many of the entities will have parts of their logic chain that are uncannily similar to each other in behavior. Environment queries, path finding, steering, cloth/deforming mesh, attached secondary animatables, breakables, target acquisition, ballistics/weapon systems, effects chains etc.
These are the areas ripe for breaking into components because they represent a smallish set of behaviors that is easily encapsulated with a small data driven configuration state vector and straight-line code.
I definitely agree it’d be a fools errand to try to componentize things that behave very differently to everything else. Boy, have I seen that mistake too.
Option D, if it doesn’t fit don’t force it 🙂
The turrets are codependent and there is a cascading effect which you want to resolve in one go. So have just one fat system. Which does the computation based on both turrets. It can compute the Targeting and Animation components in the same go. If you have multiple such tanks you still do iterative work just a “deeper” less decoupled computation. But some problems are not shallow so trying to solve them with shallow computations is what is IMHO the main problem here.