Link Time Optimizations

For discussion of ports to POSIX based systems, especially using GNUStep.

Moderators: winston, another_commander, Getafix

another_commander
Quite Grand Sub-Admiral
Quite Grand Sub-Admiral
Posts: 7134
Joined: Wed Feb 28, 2007 7:54 am

Link Time Optimizations

Post by another_commander »

Now that we have an upgraded build system for Windows, I am in the process of enabling LTO (link time optimizations) to the executables included with the installers. LTO offers a general improvement in performance of about 5-20% on average and is basically a ticket for better performance without any code changes. The tradeoff is noticeably longer link times and more complicated debugging, which is why I am thinking of enabling it only on installer builds. This way developers can continue using standard non-LTO builds which compile quickly and can be debugged without problem, while CI installers produced by github or final release binaries for which compile time is not a critical factor can get the optimization benefits for free.

But before I can enable this I need to be sure that the Linux builds can work with LTO too. Given that Linux builds have been using recent GCC versions, I would expect no issues but we have to be sure. So, any Linux user willing to assist, could you please add the -flto flag in your ADDITIONAL_CLFAGS, ADDITIONAL_OBJCFLAGS and ADDITIONAL_LDFLAGS in GNUmakefile (you may need to add the ADDITIONAL_LDLFAGS line manually) and try to build the game binary normally? You should be able to get an executable that works just like the original but be prepared to wait for a few minutes during the linking phase.

NOTE: Do not mix LTO with non-LTO object files, make sure you execute a make clean before starting the build just to be sure.

If all good we can go ahead, otherwise this will be a Windows feature only. Thanks in advance for anyone willing to provide feedback.
User avatar
Lone_Wolf
---- E L I T E ----
---- E L I T E ----
Posts: 777
Joined: Wed Aug 08, 2007 10:59 pm
Location: Netherlands

Re: Link Time Optimizations

Post by Lone_Wolf »

GCC is being a bit silly again

Code: Select all

src/Core/NSUserDefaults+Override.m: In function ‘-[NSUserDefaults(Override) writeDictionary:toFile:]’:
src/Core/NSUserDefaults+Override.m:40:17: error: format not a string literal and no format arguments [-Werror=format-security]
   40 |                 NSLog(@"Defaults database filename is empty when writing");
      |                 ^
cc1obj: some warnings being treated as errors
Want me to file an issue ?
OS : Arch Linux 64-bit - rolling release

From: The Netherlands, Europe

OXPs : My user page (needs updating)

Retired, occasionally active
another_commander
Quite Grand Sub-Admiral
Quite Grand Sub-Admiral
Posts: 7134
Joined: Wed Feb 28, 2007 7:54 am

Re: Link Time Optimizations

Post by another_commander »

No don't worry, just fixed it in master.
Commander_X
---- E L I T E ----
---- E L I T E ----
Posts: 736
Joined: Sat Aug 09, 2014 4:16 pm

Re: Link Time Optimizations

Post by Commander_X »

I gave it a try, the binary works[*].

Funny enough, it seems to be ignoring ESPEAK_DATA_PATH environment variable, so unless I'm doing a

Code: Select all

cd <whatever>/oolite.app; ./oolite
. it would bail complaining it cannot find "Resources/espeak-data/phonetab".
This happens both with -flto and without -flto flags, now -- seems to be a new thing?
User avatar
Lone_Wolf
---- E L I T E ----
---- E L I T E ----
Posts: 777
Joined: Wed Aug 08, 2007 10:59 pm
Location: Netherlands

Re: Link Time Optimizations

Post by Lone_Wolf »

Either the modern build enables LTO or I've been building oolite with LTO without realising it for sometime already.

EDIT1: no problems running it
EDIT2 : the log is without adding -flto to GNUmakefile

Code: Select all

make -f GNUmakefile debug=no strip=yes
make[1]: Entering directory '/home/panoramix/Documents/Aur/pkgbuilds/oolite-git/src/oolite-git'
This is gnustep-make 2.9.3. Type 'make print-gnustep-make-help' for help.
Running in gnustep-make version 2 strict mode.
Compiling modern build
/usr/share/GNUstep/Makefiles/objc.make:30: objc.make is deprecated.  Please use tool.make instead
cd .; \
/usr/share/GNUstep/Makefiles/mkinstalldirs ./obj.spk
Compiling modern build
/usr/share/GNUstep/Makefiles/objc.make:30: objc.make is deprecated.  Please use tool.make instead
Making all for objc_program oolite...
Compiling modern build
/usr/share/GNUstep/Makefiles/objc.make:30: objc.make is deprecated.  Please use tool.make instead
cd .; \
/usr/share/GNUstep/Makefiles/mkinstalldirs ./obj.spk/oolite.obj/
Compiling modern build
/usr/share/GNUstep/Makefiles/objc.make:30: objc.make is deprecated.  Please use tool.make instead
gcc src/Core/legacy_random.c -c \
      -MMD -MP -DGNUSTEP -DGNUSTEP_BASE_LIBRARY=1 -DGNU_RUNTIME=1 -DGNUSTEP_BASE_LIBRARY=1 -pthread -pthread -fPIC -Wall -DGSWARN -DGSDIAGNOSE -Wno-import -march=x86-64 -mtune=generic -O2 -pipe -fno-plt -fexceptions -Wp,-D_FORTIFY_SOURCE=3 -Wformat -Werror=format-security -fstack-clash-protection -fcf-protection -fno-omit-frame-pointer -mno-omit-leaf-frame-pointer -g -ffile-prefix-map=/build/gnustep-make/src=/usr/src/debug/gnustep-make -flto=auto -Wall -DLINUX -DNEED_STRLCPY `sdl-config --cflags` `nspr-config --cflags` -DOOLITE_MODERN_BUILD=1 -DOO_OXP_VERIFIER_ENABLED=1 -DOO_LOCALIZATION_TOOLS=1 -DDEBUG_GRAPHVIZ=1 -Ideps/Linux-deps/x86_64/mozilla/include -Isrc/SDL -Isrc/Core -Isrc/BSDCompat -Isrc/Core/Scripting -Isrc/Core/Materials -Isrc/Core/Entities -Isrc/Core/OXPVerifier -Isrc/Core/Debug -Isrc/Core/Tables -Isrc/Core/MiniZip -Ideps/Linux-deps/include -I. -I/home/panoramix/GNUstep/Library/Headers -I/usr/include \
       -o obj.spk/oolite.obj/legacy_random.c.o
gcc src/BSDCompat/strlcpy.c -c \
      -MMD -MP -DGNUSTEP -DGNUSTEP_BASE_LIBRARY=1 -DGNU_RUNTIME=1 -DGNUSTEP_BASE_LIBRARY=1 -pthread -pthread -fPIC -Wall -DGSWARN -DGSDIAGNOSE -Wno-import -march=x86-64 -mtune=generic -O2 -pipe -fno-plt -fexceptions -Wp,-D_FORTIFY_SOURCE=3 -Wformat -Werror=format-security -fstack-clash-protection -fcf-protection -fno-omit-frame-pointer -mno-omit-leaf-frame-pointer -g -ffile-prefix-map=/build/gnustep-make/src=/usr/src/debug/gnustep-make -flto=auto -Wall -DLINUX -DNEED_STRLCPY `sdl-config --cflags` `nspr-config --cflags` -DOOLITE_MODERN_BUILD=1 -DOO_OXP_VERIFIER_ENABLED=1 -DOO_LOCALIZATION_TOOLS=1 -DDEBUG_GRAPHVIZ=1 -Ideps/Linux-deps/x86_64/mozilla/include -Isrc/SDL -Isrc/Core -Isrc/BSDCompat -Isrc/Core/Scripting -Isrc/Core/Materials -Isrc/Core/Entities -Isrc/Core/OXPVerifier -Isrc/Core/Debug -Isrc/Core/Tables -Isrc/Core/MiniZip -Ideps/Linux-deps/include -I. -I/home/panoramix/GNUstep/Library/Headers -I/usr/include \
       -o obj.spk/oolite.obj/strlcpy.c.o
full log at http://0x0.st/KCJv.txt
Last edited by Lone_Wolf on Fri Dec 12, 2025 10:11 pm, edited 1 time in total.
OS : Arch Linux 64-bit - rolling release

From: The Netherlands, Europe

OXPs : My user page (needs updating)

Retired, occasionally active
Commander_X
---- E L I T E ----
---- E L I T E ----
Posts: 736
Joined: Sat Aug 09, 2014 4:16 pm

Re: Link Time Optimizations

Post by Commander_X »

Commander_X wrote: Fri Dec 12, 2025 8:46 pm
[...]
Funny enough, it seems to be ignoring ESPEAK_DATA_PATH environment variable
[...]
This seems to be caused by the line 783 as github shows it, in Core/Universe.m:

Code: Select all

espeak_Initialize(AUDIO_OUTPUT_PLAYBACK, 100, "Resources", 0);
Replacing "Resources" with NULL (as I found it in a July this year version of the file), fixes the issue.
another_commander
Quite Grand Sub-Admiral
Quite Grand Sub-Admiral
Posts: 7134
Joined: Wed Feb 28, 2007 7:54 am

Re: Link Time Optimizations

Post by another_commander »

Lone_Wolf wrote: Fri Dec 12, 2025 8:56 pm
Either the modern build enables LTO or I've been building oolite with LTO without realising it for sometime already.
Yup, you are already using LTO. Good.
another_commander
Quite Grand Sub-Admiral
Quite Grand Sub-Admiral
Posts: 7134
Joined: Wed Feb 28, 2007 7:54 am

Re: Link Time Optimizations

Post by another_commander »

Commander_X wrote: Fri Dec 12, 2025 9:17 pm
Commander_X wrote: Fri Dec 12, 2025 8:46 pm
[...]
Funny enough, it seems to be ignoring ESPEAK_DATA_PATH environment variable
[...]
This seems to be caused by the line 783 as github shows it, in Core/Universe.m:

Code: Select all

espeak_Initialize(AUDIO_OUTPUT_PLAYBACK, 100, "Resources", 0);
Replacing "Resources" with NULL (as I found it in a July this year version of the file), fixes the issue.
Can you please try this code and see if it works for you?

Code: Select all

int result = espeak_Initialize(AUDIO_OUTPUT_PLAYBACK, 100, "Resources", 0);
if (result < 0)  espeak_Initialize(AUDIO_OUTPUT_PLAYBACK, 100, NULL, 0); // re-attempt init with legacy path parameter
Commander_X
---- E L I T E ----
---- E L I T E ----
Posts: 736
Joined: Sat Aug 09, 2014 4:16 pm

Re: Link Time Optimizations

Post by Commander_X »

another_commander wrote: Sat Dec 13, 2025 11:33 am
[...]
Can you please try this code and see if it works for you?

Code: Select all

int result = espeak_Initialize(AUDIO_OUTPUT_PLAYBACK, 100, "Resources", 0);
if (result < 0)  espeak_Initialize(AUDIO_OUTPUT_PLAYBACK, 100, NULL, 0); // re-attempt init with legacy path parameter
Just did, it seems to go out on the first line (doesn't get to the if line).
User avatar
mcarans
---- E L I T E ----
---- E L I T E ----
Posts: 656
Joined: Sun Jun 20, 2010 6:00 pm

Re: Link Time Optimizations

Post by mcarans »

another_commander wrote: Fri Dec 12, 2025 9:22 am
Now that we have an upgraded build system for Windows, I am in the process of enabling LTO (link time optimizations) to the executables included with the installers. LTO offers a general improvement in performance of about 5-20% on average and is basically a ticket for better performance without any code changes. The tradeoff is noticeably longer link times and more complicated debugging, which is why I am thinking of enabling it only on installer builds. This way developers can continue using standard non-LTO builds which compile quickly and can be debugged without problem, while CI installers produced by github or final release binaries for which compile time is not a critical factor can get the optimization benefits for free.

But before I can enable this I need to be sure that the Linux builds can work with LTO too. Given that Linux builds have been using recent GCC versions, I would expect no issues but we have to be sure. So, any Linux user willing to assist, could you please add the -flto flag in your ADDITIONAL_CLFAGS, ADDITIONAL_OBJCFLAGS and ADDITIONAL_LDFLAGS in GNUmakefile (you may need to add the ADDITIONAL_LDLFAGS line manually) and try to build the game binary normally? You should be able to get an executable that works just like the original but be prepared to wait for a few minutes during the linking phase.

NOTE: Do not mix LTO with non-LTO object files, make sure you execute a make clean before starting the build just to be sure.

If all good we can go ahead, otherwise this will be a Windows feature only. Thanks in advance for anyone willing to provide feedback.
Nice optimisation - it works fine on the modern Linux Clang build.
another_commander
Quite Grand Sub-Admiral
Quite Grand Sub-Admiral
Posts: 7134
Joined: Wed Feb 28, 2007 7:54 am

Re: Link Time Optimizations

Post by another_commander »

@Commander_X: OK, how about this one then:

Code: Select all

if (!SDL_getenv("ESPEAK_DATA_PATH"))  espeak_Initialize(AUDIO_OUTPUT_PLAYBACK, 100, "Resources", 0);
else  espeak_Initialize(AUDIO_OUTPUT_PLAYBACK, 100, NULL, 0);
User avatar
mcarans
---- E L I T E ----
---- E L I T E ----
Posts: 656
Joined: Sun Jun 20, 2010 6:00 pm

Re: Link Time Optimizations

Post by mcarans »

another_commander wrote: Sat Dec 13, 2025 6:12 pm
@Commander_X: OK, how about this one then:

Code: Select all

if (!SDL_getenv("ESPEAK_DATA_PATH"))  espeak_Initialize(AUDIO_OUTPUT_PLAYBACK, 100, "Resources", 0);
else  espeak_Initialize(AUDIO_OUTPUT_PLAYBACK, 100, NULL, 0);
For the new Linux AppImage to work (see viewtopic.php?p=303449#p303449), that would be:

Code: Select all

if (!SDL_getenv("ESPEAK_DATA_PATH"))
{
    espeak_Initialize(AUDIO_OUTPUT_PLAYBACK, 100, [[ResourceManager builtInPath] UTF8String], 0);
}
else
{
    espeak_Initialize(AUDIO_OUTPUT_PLAYBACK, 100, NULL, 0);
}
Commander_X
---- E L I T E ----
---- E L I T E ----
Posts: 736
Joined: Sat Aug 09, 2014 4:16 pm

Re: Link Time Optimizations

Post by Commander_X »

Considering for both suggestions I'm targeting the else branch, I tried mcarans non-hardcoding solution, and it works.
Hope it does, too, for environments where env variable is not set :)
User avatar
mcarans
---- E L I T E ----
---- E L I T E ----
Posts: 656
Joined: Sun Jun 20, 2010 6:00 pm

Re: Link Time Optimizations

Post by mcarans »

Commander_X wrote: Sat Dec 13, 2025 7:21 pm
Considering for both suggestions I'm targeting the else branch, I tried mcarans non-hardcoding solution, and it works.
Hope it does, too, for environments where env variable is not set :)
I have tested the modern_linux branch with both the ESPEAK_DATA_PATH and -flto changes and it works fine. I have checked in those changes to the branch.
another_commander
Quite Grand Sub-Admiral
Quite Grand Sub-Admiral
Posts: 7134
Joined: Wed Feb 28, 2007 7:54 am

Re: Link Time Optimizations

Post by another_commander »

All right, link time optimizations are now enabled for the deployment and test release installers for Linux and Windows*.

The snapshot installers have also been slightly re-configured. They now ship executables with debug information enabled and LTO disabled, so that debugging, if needed, can become easier. This makes the snapshot variant somewhat more useful, because until now the only difference between a test release and a snapshot was the watermark on the top right of the screen. If you want to concentrate on testing, you can use snapshot. If you want to just play with the best performance possible use deployment or test release.



* The modern build is required for LTO. Currently github builds Windows with the legacy environment and Linux with the modern configuration. As a result, only the Linux installers from github have LTO enabled, but you can still generate LTO builds for Windows if you build the installers yourself. This is expected to change soon, when we transition the github CI workflow to use the modern build configuration also for Windows.
Post Reply