Oolite Bulletins

A question for all our Linux users out there: Does any one know if the keyboard scan codes are consistent across all versions of Linux? Or does each implementation roll its own?

phkb wrote: ↑Sat Sep 11, 2021 11:13 am

A question for all our Linux users out there: Does any one know if the keyboard scan codes are consistent across all versions of Linux? Or does each implementation roll its own?

Scan codes (or make/break codes) typically come from the keyboard hardware. That means they are dependent on the keyboard layout. There seems to be three big standards (ISO, ANSI and JIS) which have subtypes (full keyboard, compact keyboard etc). https://en.wikipedia.org/wiki/Keyboard_ ... al_layouts

I would really be surprised to see if they varied over operating systems, but of course I never tested them all...

phkb wrote: ↑Sat Sep 11, 2021 11:13 am

[...] Does any one know if the keyboard scan codes are consistent across all versions of Linux? [...]

For Oolite I think the best answer should be found in SDL's documentation (which should deal with most of the game related input-output for both Linux and Windows).
Similarly with Windows, Linux also has keyboard driver layouts that can be changed, and most of the applications honor the locale environment settings as well.

hiran wrote: ↑Sat Sep 11, 2021 5:31 pm

I would really be surprised to see if they varied over operating systems, but of course I never tested them all...

Be surprised. For example:
On Windows, the "A" key on the keyboard has a scancode of 30. (Specificiation is here: scancode.doc).
On a Mac, the "A" key scancode is 0. (Specification is here: MacintoshToolboxEssentials.pdf).
On Linux Mint, the "A" key scan code is 38. (Specification is... somewhere... maybe...)

Commander_X wrote: ↑Sat Sep 11, 2021 6:05 pm

For Oolite I think the best answer should be found in SDL's documentation

And from what I've read of it, the documentation mostly states how you can get the scancode. Which doesn't help me if I have the scancode already, but want to know what character would actually be produced by pressing said key.

Full disclosure: this project has already become much bigger than I anticipated. I originally (and naively, apparently) thought I could ignore the nuts and bolts of SDL, and stay mostly in the game code and logic. But, in order to allow for all keys of any keyboard to be utilised, with any combination of shift/ctrl/alt, things got a lot more complicated. So now I'm messing about in the SDL code layer, and crossing my fingers and hoping that all variants of Linux have the same specification for keycode mapping. Otherwise, my current plan (and my only plan) to make this work is pretty much dead. There is a snippet of a scancode result set on the ArchLinux "Keyboard Input" page, which at least matches what I've found in Linux Mint, so maybe it will all be OK. It would just be nice to really confirm it.

phkb wrote: ↑Sat Sep 11, 2021 8:24 pm

Full disclosure: this project has already become much bigger than I anticipated.

<sniggers> The question is are you enjoying it?

Cody wrote: ↑Sat Sep 11, 2021 8:40 pm

The question is are you enjoying it?

It’s been a challenge! Some parts have been more enjoyable than others. The frustration is that I had hoped this would be finished by now, as I’d like to move on to other aspects of the game that need attention.

BTW, my apologies to hiran and Commander_X if my previous post came across as aggressive, as that was definitely not my intention. The frustration in trying to get a cross-platform, reliable solution to the issues I’ve uncovered might have led to some poor communication on my part.

I blame the penguinistas! I believe Getafix had something like ten flavours of Linux on his testing machine.

phkb wrote: ↑Sat Sep 11, 2021 8:24 pm

hiran wrote: ↑Sat Sep 11, 2021 5:31 pm

I would really be surprised to see if they varied over operating systems, but of course I never tested them all...

Be surprised. For example:
On Windows, the "A" key on the keyboard has a scancode of 30. (Specificiation is here: scancode.doc).
On a Mac, the "A" key scancode is 0. (Specification is here: MacintoshToolboxEssentials.pdf).
On Linux Mint, the "A" key scan code is 38. (Specification is... somewhere... maybe...)

Whoa... I stand corrected. Thank you for that eye opener. Looking at my preferred language (as it is 'platform independent') again: https://docs.oracle.com/en/java/javase/ ... Event.html
* Not all keyboards or systems are capable of generating all virtual key codes. No attempt is made in Java to generate these keys artificially.
* Virtual key codes do not identify a physical key: they depend on the platform and keyboard layout. For example, the key that generates VK_Q when using a U.S. keyboard layout will generate VK_A when using a French keyboard layout.


This is exactly what you perceive so I guess SDL is in the same situation and there seems no simple solution.

phkb wrote: ↑Sat Sep 11, 2021 8:46 pm

BTW, my apologies to hiran and Commander_X if my previous post came across as aggressive, as that was definitely not my intention. The frustration in trying to get a cross-platform, reliable solution to the issues I’ve uncovered might have led to some poor communication on my part.

Absolutely no offense taken from my side. I had made some statements, you were able to prove me wrong.
Am happy to have learned from your feedback.

Would it help your mission if I created a graphical file displaying my german full-sized keyboard with all the keycodes that are shown by xev following https://askubuntu.com/questions/245333/ ... ntu#344680 ?

Also it seems there is a function where you can pass a keyboard event (or a made up keyboard event since you know the codes involved) and get the character for it: https://www.linuxhowtos.org/manpages/3/ ... String.htm
The XLookupString function translates a key event to a KeySym and a string. The KeySym is obtained by using the standard interpretation of the Shift, Lock, group, and numlock modifiers as defined in the X Protocol specification. If the KeySym has been rebound (see XRebindKeysym), the bound string will be stored in the buffer. Otherwise, the KeySym is mapped, if possible, to an ISO Latin-1 character or (if the Control modifier is on) to an ASCII control character, and that character is stored in the buffer. XLookupString returns the number of characters that are stored in the buffer.

...and here may be something similar for Windows: https://stackoverflow.com/a/5826175/4222206
The function below, GetCharFromKey(Key key) will do the trick. It uses a series of win32 calls to decode the key pressed:
* get the virtual key from WPF key
* get the scan code from the virtual key
* get your unicode character

phkb wrote: ↑Sat Sep 11, 2021 8:24 pm

Full disclosure: this project has already become much bigger than I anticipated. I originally (and naively, apparently) thought I could ignore the nuts and bolts of SDL, and stay mostly in the game code and logic. But, in order to allow for all keys of any keyboard to be utilised, with any combination of shift/ctrl/alt, things got a lot more complicated. So now I'm messing about in the SDL code layer, and crossing my fingers and hoping that all variants of Linux have the same specification for keycode mapping. Otherwise, my current plan (and my only plan) to make this work is pretty much dead. There is a snippet of a scancode result set on the ArchLinux "Keyboard Input" page, which at least matches what I've found in Linux Mint, so maybe it will all be OK. It would just be nice to really confirm it.

FWIW, your latest commit on the new_keyconfig branch seems to work really well for me (provided that I manually add the key "keyboard-code" = ita; for my Italian keyboard to my .GNUstepDefaults), with only a minor bug where the Shift modifier gets set to Yes if I happen to press shift in the screen for entering the key to use. You have done a great job so far and it would be great to have a cross-platform solution implemented, but at least on Windows, that latest version is almost fully functional as far as I can tell.

another_commander wrote: ↑Sat Sep 11, 2021 9:40 pm

at least on Windows, that latest version is almost fully functional as far as I can tell.

Relief! That's great news!

another_commander wrote: ↑Sat Sep 11, 2021 9:40 pm

with only a minor bug where the Shift modifier gets set to Yes if I happen to press shift in the screen for entering the key to use

Actually, that's not strictly a bug (yet). The logic is, if you used shift to enter the key, Oolite will need that recorded with the key in order to match it up because I'm being exclusive in the matching routine now. The last commit was really a prelim one - the Mac build isn't fully complete in that one (it's quite possibly broken), and there's a bug in the Linux mapping doc (which I have fixed locally). I really just pushed it up so I wouldn't lose anything in case of a system failure!

another_commander wrote: ↑Sat Sep 11, 2021 9:40 pm

provided that I manually add the key "keyboard-code" = ita; for my Italian keyboard to my .GNUstepDefaults

If I can get all this to come together, I'll be adding a selector to the top of the keyboard config screen where you can select your keyboard layout from all the available options, so you shouldn't need to enter this manually.

hiran wrote: ↑Sat Sep 11, 2021 9:10 pm

Also it seems there is a function where you can pass a keyboard event (or a made up keyboard event since you know the codes involved) and get the character for it

hiran wrote: ↑Sat Sep 11, 2021 9:10 pm

..and here may be something similar for Windows

I'm really, really trying to avoid having to go another layer down, to Win32 and the XWindows equivalent! There's enough OS-specific code in the Oolite code-base as it is!

hiran wrote: ↑Sat Sep 11, 2021 9:10 pm

Would it help your mission if I created a graphical file displaying my german full-sized keyboard with all the keycodes that are shown by xev

It would certainly confirm if there are any differences between Mint (that I'm using for testing) and Ubuntu, but as the former is based off the latter, I'm not expecting too many surprises there.

phkb wrote: ↑Sat Sep 11, 2021 8:46 pm

[...] BTW, my apologies to hiran and Commander_X if my previous post came across as aggressive [...]

If that was aggressive, then I don't know what my current stress over the BPlanets attempt should be called
Regardless, hope/seems the comments might've helped -- Mac is using a different framework than Windows/Linux deals with (original NextStep approach as opposed to GNUStep), and for this endeavor, Cocoa vs SDL plus internationalization is very likely the real challenge.

Well, I think the most obvious bugs are sorted now. For Windows users with a 1.91 trunk install, you can download a small update package here: (Download link removed, pending trunk release package).

Here's a breakdown of the contents:
1. oolite.exe - a 64bit version of Oolite 1.91
2. descriptions.plist
3. keyconfig2.plist
4. missiontext.plist
5. verifyOXP.plist
6. keymappings_linux.plist
7. keymappings_mac.plist
8. keymappings_windows.plist

All the plist files go in the oolite.app/Resources/Config folder, and the exe goes in the oolite.app folder.

I've confirmed the code compiles on Linux and Mac, but I can't produce a package to share for those OS's at this point. If you have the ability to build the game yourself, pull the "new_keyconfig" branch and give it a whirl.

Here's a summary of what's in this release:
1. Keyboard layout control via the Keyboard Config screen. At the moment, I've included a range of keyboard layouts, 16 in all so far. Hopefully I've managed to map everything on each layout correctly. The goal in this mapping exercise was consistency, rather than anything related to usability, in order to make our documentation as uniform as possible. So, I was trying to make sure that the "\" key to change the space compass target either mapped to a single keypress (as on the US keyboard) or a modified keypress (as on the German keyboard, which I think requires an "Alt" modifier). In both cases, we could describe pressing the "\" key and it should logically work the same for both layouts.

2. Key mapping files. For each keyboard layout, an equivalent entry is required in the three keymapping_*.plist files. The reason for 3, instead of just one, is to limit the size and complexity of these files, and to keep the key entries the same. So, if you look up the "gbr" key in the keyconfig2.plist file, there should be a "gbr" key in each of the mapping files.

The content of these mapping files only covers keys that produce a character, as that was the key (pun intended!) piece of information that was lacking when some modifier keys are pressed. So, special function keys, return, tab

It's almost time to merge this PR into the main branch. When we do, it will constitute a massive change in the way keyboard input is processed, impacting on every system and every screen. And while I'm confident the majority of all use-cases have been covered, there remains a healthy possibility that I've missed something, somewhere. There are a lot of edge cases. Did you know, for instance, on the F6 screen, that holding the Ctrl key while moving the cursor with the arrow keys makes the cursor move faster? Neither did I, until recently!

So, this update really does need extensive testing, especially with non-US keyboards. I can simulate some keyboard layouts, but it's not quite the same as having a real one in front of you. Also, while I have a virtual machine for Mac and Linux, again, having a real machine to do testing on will clarify if we're close to the mark or not.

In short, we need your help. We need people to test this release hard, and expect some things to be broken. If you're used to running the latest nightly for your main game, you might find the journey a little rougher than you're used to. With a little perseverance, we should be able to identify and resolve any major issues quickly, and hopefully get back to a stable release in a short amount of time.

Quick overview: What we've done is added the ability to configure your keyboard layout from within the game itself. At the same time, we've stopped using the old keyconfig.plist format in favour of a new one, keyconfig2.plist. This allows us to handle international keyboards with much greater flexibility than ever before. Also, in doing this change, we've opened up the possibilities for customisation even more than before: now you can have multiple keyboard customisations, and switch between on-the-fly. So, there are a lot of up-sides to this change.

But it's also worth pointing out that this update represents a kind of breaking change for any current keyconfig layouts or OXP's. It's breaking in the sense that those existing keyconfig.plist files won't do anything in the new version, not that the game will break. If you have a custom keyconfig.plist file you want to keep using, you'll have two options: (1) migrate all keyboard layout to the new format. There is a sample OXP at this location (https://github.com/OoliteProject/oolite ... onfig2.oxp) that demonstrates what's involved in creating your own custom layout; or (2) simply remap your desired keyboard layout using the new in-game mapping utility. The benefit of doing option (1) is that you can easily switch back to the default layout at any time. The benefit of doing option (2) is that you don't have to edit a single plist file, and you can do it all in the game, even while you're playing.

I'll post here again when I've merged the PR into trunk, and when a package is available to download and test.

BTW: If any of the dev team have any reservations about this PR, or the way it's handling things, please let me know!

Huge thumbs up. Check github please, I think I found one last case of Ctrl conflict.