Optimizing scripts for save/load time and file size.

[planetfall]

I've been going through scripts and optimizing them by NotPersisting variables, because the maps are slow to save and load. (I keep saying "I've been doing X" in this forum with nothing to show for it... hopefully that can change soon.) I then ran into an interesting thought: adding commands to the scripts increases the size of the final save twofold, since the map saves both a compiied and uncompiled version of each script. So NotPersisting an int may actually slow down save/load, if the script isn't on enough cores, because the extra characters in the script that also have to be saved/loaded. Whereas something like a giant list or string can save space if one prevents it from being saved on even one unnecessary core. Though then I realized that since variable names are strings, even a humble int takes up more space than anticipated.

My practical questions are:

1. approximately how many cores must have a script where an int is NotPersisted before the space saved from not storing the variable outweighs the space taken by the NotPersist line? Probably depends on the length of the variable name, so let's say... four characters.

2. considering a situation like this:

Code Select Expand


:awake
    "Some  text" ->textConstant
    "textConstant" NotPersist


The relevant script is on only one core. How long must the text and variable name be for this to be worthwhile? (Yes, for my particular case it does have to be stored in a variable.)

3. how efficient are compiled opcodes stored? That is to say, when a script is compiled to a list of opcodes, what data type is something like a SetCreeper, and how much does its presence increase file size compared to an int/float/string constant?

4. will the amount of save/load time my optimizations eventually save ever compare to the amount of time spent creating them, debugging them, and creating this post? (well, probably not... but now I'm curious about the topic in an 'academic' sort of way. Or maybe that's just the sunk cost fallacy talking.)

[GoodMorning]

Considering opcodes, they likely need be no more than 1 byte apiece (If we have 256 or fewer commands, but who's counting?). There will also be a line number in there, which might be gained by lookup in the source. This allows the trace to give targeted 'empty stack' warnings. So, I would guess that the opcodes are at worst cheaper to store than a variable, as CW is 32 bit for the most part. If we knew the maximum length of scripts, we could have a better guess...

Text is relatively large, and since all variables can be accessed or created through SetScriptVar, and as refread and refwrite exist... I'd be guessing that Virgil has a hashtable structure or a linked list. Therefore, a NotPersist is likely to be better on save/load for any variable. Even if opcodes take twice the space, a two-character var name exceeds it.

For the source, 8 characters of NotPersist means that three Cores are outweighing it just on variable name length (Although the save structure might be implemented as a table of varnames, with Core/value pairs in lists. This means that your text constants must be longer than the varname plus NotPersist to gain. It depends on what data structure Virgil chose.)

So, based on largely magical thinking (estimations) and guesswork (heuristic techniques):

1. 3+, depending.
2. 18+, depending! and basing it off that var name
3. Probably 'very'. Try adding a short additional opcode (add) and a long one (SetCreeperFlowRateOnDigitalis), and compare save sizes.
4. Likely (nearly certainly) not. However, there are two potential cures: save the map immediately after a script recompile, when only supplied vars are stored; or realise that the time cost is far greater to you than to us, and decide from there.

Two notes: removing a long set of debug opcodes can help, and this is starting to sound like CCore &c.

[planetfall]

Good to know, even if just speculation. It's best to get in the habit of using NotPersist when possible (I previously only used it for local copies of lists that are stored permanently in a manager.)

And if by CCore you mean Celestial Core, you'd be correct.

[GoodMorning]

I have tried to avoid it, for determinism: to discourage (Bad thing happens) -> Reload and hope for a different random number. For small scripts there's not a lot of difference, and the large saves normally shrink as Cores are destroyed.

Still, for big scripts it's a good idea. That's one reason that I used the asynchronous model for my addons to your tentacle script. Storing the entire descendant /ancestor list required every Core in a chain to store every other Core's UID. Storing it only in parent/child meant that for a five-core chain, I stored 8 UIDs instead of 20 (a 2(N-1), pass damage each frame or N(N-1), kill immediately choice).

I have high hopes for the Celestial maps. If we could get CNs to drop via CRPL, you wouldn't even need to disable alternate control mode.

Out of mild interest, have you ever felt that it would be nice to be able to discover the UID in-editor? Yes, it's trivial to write a script for it, but...

[planetfall]

Not sure what you mean about determinism. None of the variables I've NotPersisted are based on random values and I'm fairly sure the status of the RNG persists in saves (if it doesn't, then vanilla random elements like spore towers aren't deterministic either.)

[GoodMorning]

That's a tricky one. I think the RNG is partially seeded with the gamestate.

Don't worry about it, it was an idle thought.

[Grayzzur]

Also realize that the scripts are not stored in the save games in plain ASCII text. I've seen samples where a 16k savegame contains 24k in script text. They are likely zipped, and text compression tends to be in the 80-90% range. If you have large custom images, those will bloat your save game size as well.

[GoodMorning]

This may be helpful also, though I hadn't encountered it before in CRPL: Lists are reference typed, and so are global.

	Core 1	Core 2
1st	Create and populate list.
2nd		Get list with GetScriptVar, append new item.
3rd	Trace the list. New item appears.

This may be useful for saving on GetCoresWithVar calls, as well as simplifying save/load. It also means that NotPersist and the list reloading code likely outweighs the reference storage on additional Cores. Of course, if you have different lists, rather than a global one with "copies", you have a different problem, which this will not help with.

This assumes, of course, that it remains a reference type across save/load.

(I realise that this thread was likely abandoned, but this might help someone who searches on something similar)