Bug: Snipers becoming "nearsighted"

[planetfall]

I'm putting this here instead of Coder's Corner, since I think it might be a bug with the engine rather than a scripting error on my part -- I've scoured my scripts looking for something that might this effect, but to no avail.

The situation as it stands:

There are some cores (enemy drones) that are supposed to be shootable by snipers. At first, they work as expected. After running the map for a while, however, they develop a strange interaction. Snipers become oblivious to the drones when they first enter the sniper's range, but as they get closer the sniper eventually recognizes them and fires. After experimenting a bit with building snipers at different times, it appears that this is an issue per-sniper. New snipers have no problem targeting the drones even if the sniper right next to it can't. Also, it seems that it gets worse the longer the map goes on: eventually it gets to the point that the sniper can't attack the drone until it gets within a few cells of the sniper. Once that happens, the sniper "wakes up" and shoots at the core until either it is destroyed or it exits the sniper's normal range. If it exits the range it has to get within the much smaller "notice range" before the sniper can shoot it again.

What's more, saving and reloading actually resets the problem so I can't just send a save file.

So my thought process with debugging this is as follows:

- Is it a bug in CONST_SNIPERIGNORELOS?
No. Setting this to false does not correct the problem.

- Is it because setting CONST_SNIPERTARGET is getting set constantly?
The drones search for a state in an off-map control core that tells them whether they should be sniper-targetable. Perhaps setting this rapidly is updating the snipers too fast? Changing this so that the state is only changed when the control is toggled fixes nothing. Neither does removing the drones' ability to change their snipertarget status altogether.

- Does it do the same thing with runners?
Yes, it seems to. This includes runners spawned from regular runner nests as well as by scripts. I also removed everything from the map that could potentially mess with the runners, and it still continues. So, I guess this is proof that it is an engine error and not a script error?

- What if I remove all sniper targetable cores?
Done, and still it continues.

- Does it persist after saving?
Well, no. But if I reload when the sniper targetable cores are all gone, it also doesn't re-appear after saving, reloading, and letting it run a while. Even if there are 800 bazillion runners everywhere.

It seems as though a map with both runners and snipeable cores messes with things. Of course, there were both runners and snipertarget cores on Farbor... Let me replay that map and see.

It doesn't seem to mess up (but considering it's a rush map, it may just not have time for the bug effect to build up.) However, since I've looked at the Farbor scripts, I know that the ships don't actually get destroyed and instead teleport back to the launch pads and reset to the building state. Whereas in my map the drones are constantly being spawned and destroyed. That might be important...

[knucracker]

What I need is basically a minimal (or close to it) way to reproduce the problem.  I don't mind letting a mission run...  Do you have a simple map that will demonstrate the problem after some time?

What I will be on the lookout for is making sure that the snipers just aren't running low on ammo, or aren't within their re-arm period.  After that, I'll look at the speed of the targets and the angle of approach to make sure the snipers are able to get a lock.  After that, I'll look at the number of units to see if some other target might be distracting the sniper.  And lastly if none of that sheds any light, I'll look at units that are destroyed to see if they are leaving a 'ghost' or anything like that.

I'm skeptical... as I always am about pretty much everything.  The reason for skepticism is that this bug has only been reported now.  That doesn't mean it isn't real, just that I have go looking for something that takes a special setup to create or that isn't at first obvious.

[Flabort]

I have to say I've seen it before.
In his most recent series map, and the unreleased 'boss' map he's making, the bug where snipers don't always shoot can be seen.

In fact, when I first saw snipers shooting at those drones, I thought THAT was the bug. Because out of 5 overlapping sniper ranges, I only saw one shooting at those things, so I thought that sniper must be bugged.

[planetfall]

OK, here you go... it'll take some time even on 4x speed. In the real level it takes much less time (maybe because there are many more cores being spawned elsewhere? idk)

(This is worth noting, even if not on topic - while resizing a copy of the problem map to make the test map, clipped enemies that created units other than power zones left their special drops behind, off the map edge.)

[Relli]

I'm skeptical... as I always am about pretty much everything.  The reason for skepticism is that this bug has only been reported now.  That doesn't mean it isn't real, just that I have go looking for something that takes a special setup to create or that isn't at first obvious.

But hasn't this happened before? I had it happen to me on the Arc Eternal mission with the gliders, where you first get Berthas. Chanson, I believe. It was reported as a known bug.

[knucracker]

How long (in game time) are we talking about?
I'm at 8 hours in the drone test but don't yet see a problem.  I'll let it run for longer, just thought it wouldn't hurt to ask what time you had before you saw a problem.

Also, when I see a problem will it be that the drones fly out further over the snipers being shot at or something like that?  What I'm actually watching is each sniper to see if each is firing as fast as possible.  I'm looking for a case where a sniper sits there doing nothing when it has plenty of ammo, its barrel is recovered from recoil and there is a target in range.

[kwinse]

I made a simple test where 1 drone would move slowly along a corridor towards 1 waiting sniper (read: enough time for the sniper to rearm before the next got in range), and the controlling core would trace where the drone died. After the first couple shots where the sniper's barrel got into position, the drone always died in the same place, no matter how long I left the game on.

So if the sniper bug is real, it's not evident by this test case, or I didn't give it enough time. I did find I couldn't scroll the trace log, but I haven't done much investigating of that.

[knucracker]

I'm at 18 hours now on the drone test (in game time).  I'm not seeing the issue....
So, I must not be seeing what I'm supposed to be looking at.  Maybe the posted test mission didn't actually show the problem by accident?

[planetfall]

How long (in game time) are we talking about?
I'm at 8 hours in the drone test but don't yet see a problem.  I'll let it run for longer, just thought it wouldn't hurt to ask what time you had before you saw a problem.

Also, when I see a problem will it be that the drones fly out further over the snipers being shot at or something like that?  What I'm actually watching is each sniper to see if each is firing as fast as possible.  I'm looking for a case where a sniper sits there doing nothing when it has plenty of ammo, its barrel is recovered from recoil and there is a target in range.

IDK, about 30-40 mins in.

It may also have something to do with computer power... as I understand it you have a rather powerful computer, whereas I'm running on a 2006 iMac (ugh...)

However, I tested the full map on a friend's computer which I estimate is about 10-20 times as fast as mine, and the problem was pretty apparent. Let me try something else, because there is one major difference (aside from all the other evil going on) between the full map and the test map...

Yup. I made the shields produced by the drones not beamable, and the problem didn't occur. Then I added some beamable core spawners to the map, and the problem appeared faster. (and this time I have video: )

I tried this because at first I thought that it was some kind of "running out of UIDs" problem, that the ids of the drones had somehow passed max int and the snipers weren't targeting negative UIDs or something to that effect. But then I threw out that idea when I saw that it was just decreasing "lock on range" and not ignoring them outright.

[knucracker]

What do you mean by "beamable core spawner".  Do you mean that things are being moved with some CRPL calls (like setting the position using SetUnitAttribute) or something to do with Beams?

[planetfall]

Meaning it periodically creates some cores that fly at the player units and are destroyed by beams. (I'm just reusing old scripts.) No idea why that messes with sniper targeting, but it does.

[Relli]

I did find I couldn't scroll the trace log, but I haven't done much investigating of that.

This one I can actually answer. Usually the trace log is added to or reset every frame, and whenever it is, it forces it down to the bottom of the list. If you pause, you should find that you can scroll it perfectly. Unless of course the core is set to Operate While Paused.

[kwinse]

And that's why I was hesitant to report it, thanks.

[knucracker]

Ok, I found the issue.  What a subtle little thing.  Technical description follows, so grab some coffee.

Runners and CRPLTowers use a type of spatial hashing as they move about.  This makes it more efficient for things like Snipers to find targets that are in range.  Without this, a Sniper might have to look at every Runner or CRPLTower on a map and do a distance calculation.

As Runners and CT's move about the map they add and remove themselves to 'cells'.  Naturally when a runner or CT dies, it removes itself from the current cell it occupies.  Well, that was true 99.99% of the time for CT's.  There was an edge case (literally an edge case) where a CT could cross the border from one cell to another, get killed, then remove itself (or attempt to) from the new cell not the old one.  All of that could happen in one game frame.  It had to do with the order of death (the sniper doing the killing after the CT had updated itself) and a state check I do to prevent work in a script if the hosting unit is already dead.

So, the reason this was so hard to spot is because it required killing a CT at just the right spots.  The slower the CT was moving the harder to recreate.  Also, since any given game is deterministic, things that perturb the mission (like adding or removing other units) would seem to affect how long it would take to reproduce.  

Now, the reason the Snipers just sit there is because a killed CT would leave a 'ghost' in the spatial hashing data structure (in the form of a null).  Snipers look for non empty cells when doing a target search and null is not empty.  Of course snipers could check for null, but the target search is a time sensitive area so extra checks to handle a case that was never supposed to  happen wasn't added.  Later on when it came time to actually fire, a null check was done.  That's why a sniper will just sit there and point at an 'empty' space.  It is pointing at the ghost of a dead CT but the firing algorithm won't shoot because to it there appears to be nothing there (I see dead people....)

Loading from a saved mission fixes this problem because the spatial hashing data structure doesn't persist anything.  It gets populated by units at run time.  So the ghosts head to the light, so to speak, after a game load.  After that, your mission is no longer haunted and snipers go back to seeing things that are actually there.

I'll put up at least a beta build later today after I check over some other things.

[Flabort]

And this is why Virgil deserves praise, even tough nuts like this bug don't stop him for long.
From the bug itself, and the context of the error, I don't think it was easily guessable.