Exclude png.h and pngconf.h by default on Linux?

[Lone_Wolf]

Code: Select all

$ ls -l /usr/include/png*
lrwxrwxrwx 1 root root 18  3 dec 21:49 /usr/include/pngconf.h -> libpng16/pngconf.h
lrwxrwxrwx 1 root root 14  3 dec 21:49 /usr/include/png.h -> libpng16/png.h
lrwxrwxrwx 1 root root 21  3 dec 21:49 /usr/include/pnglibconf.h -> libpng16/pnglibconf.h
$ 

The files in /usr/include/libpng16 are the ones that are provided in libpng 1.6.50 by upstream.

oolite sourcecode however has the png.h file in 3 git submodules and 3 folders
oolite-linux-dependencies/include/png.h
oolite-windows-dependencies/x86/include/png.h
oolite-windows-dependencies/x86_64/include/png.h

and 3 folders :
deps/Linux-deps/include/png.h
deps/Windows-deps/x86/include/png.h
deps/Windows-deps/x86_64/include/png.h

All of them look the same and are for libpng 14.

To me the error messages suggest that something in oolite sourcecode is hardcoded to use libpng 14 regardless of linker flags.
I can try removing all occurrences of png.h & pngconf.h in oolite sourcecode before building, but even if that works it feels like papering over a flaw instead of solving the flaw.

[MrFlibble]

Code: Select all

$ ls -l /usr/include/png*
lrwxrwxrwx 1 root root 18  3 dec 21:49 /usr/include/pngconf.h -> libpng16/pngconf.h
lrwxrwxrwx 1 root root 14  3 dec 21:49 /usr/include/png.h -> libpng16/png.h
lrwxrwxrwx 1 root root 21  3 dec 21:49 /usr/include/pnglibconf.h -> libpng16/pnglibconf.h
$ 

The files in /usr/include/libpng16 are the ones that are provided in libpng 1.6.50 by upstream.

oolite sourcecode however has the png.h file in 3 git submodules and 3 folders
oolite-linux-dependencies/include/png.h
oolite-windows-dependencies/x86/include/png.h
oolite-windows-dependencies/x86_64/include/png.h

and 3 folders :
deps/Linux-deps/include/png.h
deps/Windows-deps/x86/include/png.h
deps/Windows-deps/x86_64/include/png.h

All of them look the same and are for libpng 14.

To me the error messages suggest that something in oolite sourcecode is hardcoded to use libpng 14 regardless of linker flags.
I can try removing all occurrences of png.h & pngconf.h in oolite sourcecode before building, but even if that works it feels like papering over a flaw instead of solving the flaw.

I've not had time of late to get immersed in the code, but my fork doesn't use those includes on Linux, rather the system ones.

[mcarans]

Code: Select all

$ ls -l /usr/include/png*
lrwxrwxrwx 1 root root 18  3 dec 21:49 /usr/include/pngconf.h -> libpng16/pngconf.h
lrwxrwxrwx 1 root root 14  3 dec 21:49 /usr/include/png.h -> libpng16/png.h
lrwxrwxrwx 1 root root 21  3 dec 21:49 /usr/include/pnglibconf.h -> libpng16/pnglibconf.h
$ 

The files in /usr/include/libpng16 are the ones that are provided in libpng 1.6.50 by upstream.

oolite sourcecode however has the png.h file in 3 git submodules and 3 folders
oolite-linux-dependencies/include/png.h
oolite-windows-dependencies/x86/include/png.h
oolite-windows-dependencies/x86_64/include/png.h

and 3 folders :
deps/Linux-deps/include/png.h
deps/Windows-deps/x86/include/png.h
deps/Windows-deps/x86_64/include/png.h

All of them look the same and are for libpng 14.

To me the error messages suggest that something in oolite sourcecode is hardcoded to use libpng 14 regardless of linker flags.
I can try removing all occurrences of png.h & pngconf.h in oolite sourcecode before building, but even if that works it feels like papering over a flaw instead of solving the flaw.

The problem is that there is the system png.h (libpng16/png.h) and the obsolete Oolite one oolite-linux-dependencies/include/png.h and the latter is being found because oolite-linux-dependencies/include comes first in the includes. As MrFlibble mentioned, the oolite-linux-dependencies/include folder is obsolete on current Linux distros including yours by the error you're seeing. This folder should be deleted and I think that's what MrFlibble has done in his fork which will hopefully be merged into trunk in the near future.

[Lone_Wolf]

and the latter is being found because oolite-linux-dependencies/include comes first in the includes.

Code: Select all

ADDITIONAL_INCLUDE_DIRS      = -I$(LIBJS_INC_DIR) -Isrc/SDL -Isrc/Core -Isrc/BSDCompat -Isrc/Core/Scripting -Isrc/Core/Materials -Isrc/Core/Entities -Isrc/Core/OXPVerifier -Isrc/Core/Debug -Isrc/Core/Tables -Isrc/Core/MiniZip -Ideps/Linux-deps/include

In GNUmakefile is that line, but that refers to another folder . any idea where oolite-linux-dependencies/include is added ?

ADDED

Not making the git submodule oolite-linux-dependencies available does solve the 'PNG loading warning' message but doesn't affect the graphical issue (colored blocks with no visible text in startmenu).

Going to try removing more parts.

[Lone_Wolf]

I have found a workaround that allows usage of libpng 1.6.x on archlinux.
I have to remove png.h & pngconf.h in oolite-linux-dependencies/include/ AND in deps/Linux-deps/include/ folders, so in 2 places.

Now everything starts, the logs just show the familiar grey scale warning .

Code: Select all

17:58:00.839 [texture.load.png.warning]: ----- A PNG loading warning occurred for /usr/share/oolite-git/Resources/Textures/trumblekit.png: iCCP: profile 'ICC Profile': 'RGB ': RGB color space not permitted on grayscale PNG.

Has someone tested if the github pre-release linux installer version also can work with libpng 1.6.x ?

[MrFlibble]

Has someone tested if the github pre-release linux installer version also can work with libpng 1.6.x ?

It should. Mine does. FWIW it's also using espeak-ng.

Wish I could find a few hours in a row to refresh my memory of how, but if you look at the files between master, and my flibblefork2 of oolite and oolite-linux-dependencies (mostly gone!), you should be able to spot how I did it.

I've hacked up the workflows/build-all.yml to not bother building the windows variant (I'm not playing with that), and to simply use doxygen from apt install rather than building from scratch. There's also a scattering of trailing white-space cleanup wherever I touched a file.

I've had a go at updating the build instructions in README.md

Please note the issues. The keyboard one is just weird, and may only affect xfce4.

The scripting I did to clean up png files in OXPs is here.

[Lone_Wolf]

To check what your fork changes I'll have to figure out what happens in those 36 commits it's ahead of master.
The issues look like they could be oolite master bugs and not unique to your fork.


I looked at the releases page of oolite github and noticed 3 builds for linux :

OoliteInstall-1.91.0.7712-251017-85ce217-linux-dev-x86_64.run
OoliteInstall-1.91.0.7712-251017-85ce217-linux-test-x86_64.run
OoliteInstall-1.91.0.7712-251017-85ce217-linux-x86_64.run

Do those correspond to the targets pkg-posix-nightly, pkg-posix-test, pkg-posix (possibly not in that order) in Makefile ?

If so they all use installers/posix/make_installer.sh which builds the deps-* version of those targets.

[MrFlibble]

To check what your fork changes I'll have to figure out what happens in those 36 commits it's ahead of master.
The issues look like they could be oolite master bugs and not unique to your fork.


I looked at the releases page of oolite github and noticed 3 builds for linux :

OoliteInstall-1.91.0.7712-251017-85ce217-linux-dev-x86_64.run
OoliteInstall-1.91.0.7712-251017-85ce217-linux-test-x86_64.run
OoliteInstall-1.91.0.7712-251017-85ce217-linux-x86_64.run

Do those correspond to the targets pkg-posix-nightly, pkg-posix-test, pkg-posix (possibly not in that order) in Makefile ?

If so they all use installers/posix/make_installer.sh which builds the deps-* version of those targets.

Dev is with a visible version watermark and debug facility. Test is with debug facility and no visible watermark. the other one is no debug.

I tend to just use test.

[Lone_Wolf]

MrFlibble, I have a hard time following the commits in your fork and feel you tried to do to much.

If I were to tackle this issue, I'd follow something like this and only change what's absolutely necessary

Setup atleast 2 test systems with different distros
I'd use my own bare-metal workstation running archlinux and a VM running debian stable .

Arch Linux package versions are typically the newest stable versions, while debian stable is often used as a baseline to determine minimum required versions ((ubuntu does to many non-standard things to be used as baseline))
If something builds and runs on both it will likely run fine on the vast majority of distros.


Main goal : use external system provided libraries for linux instead of oolite provided libs were possible
Secondary goal : update the libs to current versions

test procedure
- release target runtime
- test build of other targets
- runtime test of posix-pkg

Task 1
1. switch from espeak to espeak-ng system library
test
create PR

Task 2
1. switch from libpng 1.4 to 1.6
test
create PR

Task 3
use sdl12-compat (runs on sdl2 )
test
create PR

One by one tackle all remaining external deps

[MrFlibble]

I appreciate your focus.

MrFlibble, I have a hard time following the commits in your fork and feel you tried to do to much.

I did have a brief skim through the file diffs between my fork and master yesterday.. Much easier to see what's happening than looking at the plethora of commits I did in order to test build on github. There's a lot of do, try, undo, redo, which are not easily purged from the commit history.

If I were to tackle this issue, I'd follow something like this and only change what's absolutely necessary

Setup atleast 2 test systems with different distros
I'd use my own bare-metal workstation running archlinux and a VM running debian stable .

Arch Linux package versions are typically the newest stable versions, while debian stable is often used as a baseline to determine minimum required versions ((ubuntu does to many non-standard things to be used as baseline))
If something builds and runs on both it will likely run fine on the vast majority of distros.

Using the latest stuff is great, but not if you want the resulting binaries to ALSO work on older 'current stable' distros using their system libraries. I might develop on arch/gentoo/debian latest, but would still use "oldest current" to build for most compatibility. I do test stuff on other distro's in VMs.

In fact, these days I'm lazy, and mostly stick to LinuxMint for desktop and dev, with debian, arch and alpine for leaner server-type setups.

Main goal : use external system provided libraries for linux instead of oolite provided libs were possible
Secondary goal : update the libs to current versions

Main goal.. YES. That's what I set out to do.

Secondary goal... NO. Not exactly. The resulting installers on Github releases should aim to be as compatible with 'oldest current' distros as possible. If I build with "ubuntu latest", it won't run on older still supported distros... even IF libraries get bundled. That said, I see no harm in testing that it'll build with newest libraries on newest distros, and perhaps even build deb packages using all ubuntu and deb 'workers' currently available on github. A single deb using older minimal library versions might be more convenient.

Perhaps secondary goal might be better defined as a corollary of the first.. Where system libraries can be available, even if it means user must install them (or apt pull them in), do not have stale versions of those libraries in the Oolite source tree., thus whatever the build host, the latest libraries might get used.

test procedure
- release target runtime
- test build of other targets
- runtime test of posix-pkg

Task 1
1. switch from espeak to espeak-ng system library
test
create PR

Task 2
1. switch from libpng 1.4 to 1.6
test
create PR

Task 3
use sdl12-compat (runs on sdl2 )
test
create PR

One by one tackle all remaining external deps

I did all that, without a PR in between, as it was in iterative process covering all those bases with a lot of trial and error.

A lot of poking around went on over quite a while. I would have done more PR's along the way but wanted to get to a sane and stable point which would build functionally for as many current systems as possible using the oldest available github Ubuntu 'worker' for max compatibility and least effort. I have ideas to do chroot/proot builds.

I figured too many flaky PR's would be "just noise". I suppose I could re-do it all, in those stages with PR's between, but don't have time right now.

Perhaps another approach (at this point) would be for me to do a "summary commit", with a final summary list of changes for the PR. If I choose to do that, I'll merge master changes first and re-test.

Either way.. Ignore the commits when looking at my fork vs. master.. Look at the file diffs!!!

Re: workflows/build-all.yml It can be mostly ignored when looking at what I've done to help clean up the library mess. I've chopped out all the windows stuff to speed-up Linux tests while I work, and I'll likely put those back to normal ASAP, however the changes to the doxygen build phase, and some of the other cleanups to the Linux build process there may prove worthwhile.

[phkb]

To test, I read the linked article, apt install pngcrush, unzip my hacked up version of DTT_Warlance, then use this script to select errant files in the tree, and remove only the troublesome profile.

Code: Select all

find . -name "*.png" -type f 2>/dev/null |  while read fn ; do
  pngcrush -n -q "$fn" 2>&1 | grep -q "iCCP: Not recognizing known sRGB profile that has been edited" && {
    echo -n "Fixing: $fn :"
    pngcrush -ow -rem iCCP "$fn" >/dev/null 2>&1 && echo Good || {
      echo Fail ; exit 1
    }
  }
done

Here's a version I did for Windows:
Step 1: Put a copy of pngcrush.exe in C:\Oolite\AddOns
Step 2: Create a file called "FIndBadPNGFile.bat" and copy the following code into it:

Code: Select all

@echo off
for /r C:\Oolite\AddOns\ %%f in (*.png) do (
    c:\Oolite\AddOns\pngcrush.exe -n -q "%%f" > chkfile.txt 2>&1
    find /i /c "iCCP: known incorrect sRGB profile" chkfile.txt && ( echo "%%f" >> BadFiles.txt )
)
del chkfile.txt

Step 3: From a command line, do the following commands:

Code: Select all

cd c:\Oolite\AddOns
FindBadPNGFiles.bat

At the end of the process, you'll have a file called "BadFiles.txt" which will list all the PNG files that have an incorrect sRGB profile. When I open these files in GIMP 3.0.8, I get the following dialog:

I just accept the defaults and click "Convert".
Then, from the "File" menu I click "Overwrite {filname.png}"
That seems to fix the issue. Although any suggestions on a better way to use GIMP here would be appreciated.

[MrFlibble]

Here's a version I did for Windows:

Nice! Did you notice the next script on my post, which unzips, fixes, and re-zips all the OXZs in a directory?

[phkb]

Nice! Did you notice the next script on my post, which unzips, fixes, and re-zips all the OXZs in a directory?

Yes, I saw that. Most of my OXPs are in OXP format though, so I don’t need to zip them. I also want to manually adjust readme files, version numbers etc, so I didn’t want a fully automatic solution. I just wanted to know what I was up against.