Autopilot for Oolite

[MrFlibble]

I just had a closer look at Jeff's solution. It uses xdotool to fire off keyboard input into Oolite.
xdotool works with X.org display server. However since a couple of years Ubuntu migrated towards the Wayland display server - and xdotool cannot deal with that.
Now there might be something like kdotool, dotool but they all require some extra effort.

I've been watching this thread with interest. Jeff's autopilot is quite bonkers. I've done similar in the past (bash, imagemagick, xdotool etc.) for automation, and know how fragile it can be.

I'm on Mint XFCE, so still use X.org. I do however have a couple of Wayland DE's set up as alternative logins on my 'lesser laptop'.

A direct effect of the improved security between client applications is that Wayland is not very 'snoop tool friendly'.

I take it Oolite is actually still not Wayland compatible, so it will use Xwayland. However, if Oolite has grown "Wayland-wings", is there an RDP server for Wayland? If so, then it might be possible to mirror the Wayland screen to a client in an 'in memory VNC Xserver', and do all the magic in that.

Terminals on Sway/Weston have two environment variables for clients, the old DISPLAY, and WAYLAND_DISPLAY. The clients if compatible, will consume the latter first, falling back on the former under X. It looks like Xwayland clients are all spawned by default in DISPLAY :0

Code: Select all

xdotool search \.\*

...returns a list of current windows that are being supported by Xwayland, in my test case, three x-terminals, xclock and glxgears. So there's hope for doing this without changing desktop.

Another way.. Xnest. Keep all the autopilot guff in an X server while using a Wayland desktop.

Code: Select all

# Install the bits to do this. openbox is an optional lightweight DE.
apt install xnest openbox
# Spawn an X server as display :12
Xnest :12

#Now open another xterm.
DISPLAY=:12 xterm
# that terminal should appear without window handles in the Xnest.

#Now in the terminal in the Xnest:
# stop 'modern apps' spawning on Wayland, outside the Xnest
unset WAYLAND_DISPLAY
# Now launch a WM or DE, other sessions ar available. I found my xfce4-session worked fine.
openbox

If you used openbox in the above, a right-click on the desktop will get you a menu where you can invoke more xterms.

[hiran]

Hmmm. Somehow the amount of tools required does not really make me happy.

I checked using Java, and it is possible to send events into the window system. That means keyboard and mouse.
Maybe I could find a way to also simulate a joystick. But there seems one big disadvantage:

It is to generate events only. There is no way in Java to find out things about the desktop - such as which windows are opened, which window is focused (important to know whether Oolite is focused and will receive the keyboard event) or which window is visible where (important to know whether/which mouse events are routed to Oolite).
Without such capability one would not even be able to tell if Oolite is still showing the splash screen or the menu, or if the ship has left the station.

While the alert status would give away if the ship is docked or in flight, we still would not know about keyboard focus and window positions. Could the debug console deliver such information?

[MrFlibble]

Hmmm. Somehow the amount of tools required does not really make me happy.

I checked using Java, and it is possible to send events into the window system. That means keyboard and mouse.
Maybe I could find a way to also simulate a joystick. But there seems one big disadvantage:

It is to generate events only. There is no way in Java to find out things about the desktop - such as which windows are opened, which window is focused (important to know whether Oolite is focused and will receive the keyboard event) or which window is visible where (important to know whether/which mouse events are routed to Oolite).
Without such capability one would not even be able to tell if Oolite is still showing the splash screen or the menu, or if the ship has left the station.

While the alert status would give away if the ship is docked or in flight, we still would not know about keyboard focus and window positions. Could the debug console deliver such information?

xdotool and other window peeking tools under X can force the window maximised, focused, and push keyboard events. Ascertaining the state can be done with screen scraping and image matching via e.g. imagemagick. Some assumptions have to be made about the user not faffing with things while it's all happening. The latter can be leveraged to verify full-screen and 'known state'.

A great deal of 'training' needs to be done. Not changing HUD etc. is essential.

Not sure if the window state can be popped with DC.

[hiran]

xdotool and other window peeking tools under X can force the window maximised, focused, and push keyboard events. Ascertaining the state can be done with screen scraping and image matching via e.g. imagemagick. Some assumptions have to be made about the user not faffing with things while it's all happening. The latter can be leveraged to verify full-screen and 'known state'.

A great deal of 'training' needs to be done. Not changing HUD etc. is essential.

Not sure if the window state can be popped with DC.

xdotool et al requires X. Recent versions of Ubuntu use Wayland (at least that's what I read although I cannot see evidence in my process list).
The even bigger problem for me is I's likely need native tools (like xdotools) for different environments, making this big effort - both for creation as well as maintenance. If window information could be gained through the debug console (and obviously Oolite must know whether it is on the screen and where) it would reduce complexity a lot. The big 'if only'...

[MrFlibble]

xdotool and other window peeking tools under X can force the window maximised, focused, and push keyboard events. Ascertaining the state can be done with screen scraping and image matching via e.g. imagemagick. Some assumptions have to be made about the user not faffing with things while it's all happening. The latter can be leveraged to verify full-screen and 'known state'.

A great deal of 'training' needs to be done. Not changing HUD etc. is essential.

Not sure if the window state can be popped with DC.

xdotool et al requires X. Recent versions of Ubuntu use Wayland (at least that's what I read although I cannot see evidence in my process list).
The even bigger problem for me is I's likely need native tools (like xdotools) for different environments, making this big effort - both for creation as well as maintenance. If window information could be gained through the debug console (and obviously Oolite must know whether it is on the screen and where) it would reduce complexity a lot. The big 'if only'...

This is an intellectual exercise best left in the Linux realm. Trying to do this as a production tool, and/or on other platforms would probably be madness.

So.. for fun:-

As long as Oolite requires X, the old tools should work. You know I checked with a Wayland session, Oolite gets launched in Xwayland, so xdotool can see/tweak it.

Under Linux & X, to set, and therefore know the state of a window is simple enough with xdotool, either to 100% window, or by sending shift-F12. In any case, screen-shots can be compared with a known state by masking the variable areas, and subtracting now from reference image, counting the diffs, and aiming at zero. The transparency of most HUDs and MFDs would make that tricky, which steers in the direction of opaque HUDs.

This is ugly, but was quick. If Oolite is running, this makes it fullscreen:

Code: Select all

#!/bin/bash

q(){ echo "$1" ; exit ; }

res=1920x1080
Oo=$(xdotool search --class 'Oolite') # Get window id once.
[ "x$Oo" = "x" ] && q "Couldn't get window id for Oolite. We could simply run it at this point and test again"

testfull(){
  xwininfo -id $Oo | grep -q "geometry ${res}+0+0" || return -1
}

makefull(){
  testfull && return 0
  echo "Making fullscreen"
  xdotool key -window $Oo shift+F12
  sleep 0.5 # Wait for resize. Maybe use --sync?
  testfull || return -1
}

makefull && echo "FULLSCREEN" || q "Failed"

We can get alert state from DC/MQTT already!

Being able to get window state would of course be handy for this if attempting cross-platform, though I'm not sure what else that could be good for.

The ability to push input actions via DC/MQTT could be very handy in more than one use case. If we could push keyboard/joystick events, that'd open the game up to oddball devices that don't present as HIDs but could be rapidly conjured with e.g. ESP 8266 and Tasmota motion sensor joystick.. random buttons and switches.

[Wildeblood]

Just returning from linux nerd land to the original topic for a while, what would one expect an Autopilot OXP to actually do? In game, from a player's perspective? Hiran? Anyone?

In what circumstances would the autopilot take over piloting? Having assumed control of the ship, what else would it do, apart from set PS.speed to cruise or maximum speed? In what circumstances would the autopilot disengage?

[phkb]

In what circumstances would the autopilot take over piloting?

The obvious target would be sunskimming. Oh, wait! There's an OXP for that: AutoSkim.oxz (Full disclosure: This OXP hasn't been fully tested, which is why I haven't published it to the DM).

It could be useful as a template for other actions, though.

[hiran]

I'd like to enhance barrell rolls to not just evade enemy fire but followup on a target - be it the station or something else. Or accept user input to bias the barrel roll towards the player's input.

I'd go for targeting support during dogfight until the player wishes to turn the ship. Landing sequence on planets. Maybe add some features that are scattered around other OXPs.