Exploring X11 windows and devices and emulating mouse movements and keystrokes

The X Window System or X11 comes from 1984 and is still widely used (despite we have some other options like Wayland).
This protocol and set of libraries and interfaces gives us GUI (graphical user interface), manages our displays, windows, keyboards and mice.

Since Relational pipes
v0.18 we can interact with this wonderful technology through the relpipe-in-x11 and relpipe-out-x11 tools.
In this example we will show how to acquire information about the input devices, screens and windows or capture and emit X11 events (key presses and mouse movements).

Featured Content Ads

add advertising here

COGNAC Metaverse

Explore World's first metaverse that is as real as our Real world.

https://cognac.knowasiak.com

Listing windows

In X11 almost everything is a window. Especially classic applications like XCalc are composed from many windows.
We list all windows of the current display this way:

relpipe-in-x11 --list-windows true | relpipe-out-tabular | less -RSi
relpipe-in-x11 --list-windows true | relpipe-out-recfile | less

and realize that there are so many windows (hundreds on a common desktop) with so much metadata (long names, titles etc.).

%rec: x11_window
%type: id int
%type: root int
%type: parent int
%type: level int
%type: process int
%type: x int
%type: y int
%type: width int
%type: height int

id: 1254
root: 1254
parent: 0
level: 0
process: -1
name: 
res_class: 
res_name: 
x: 0
y: 0
width: 3840
height: 2160

…

id: 125829135
root: 1254
parent: 155189297
level: 3
process: 10046
name: Exploring X11 windows and devices and emulating mouse movements and keystrokes – Relational pipes - Falkon
res_class: Falkon
res_name: Falkon Browser
x: 0
y: 0
width: 3840
height: 2044

…

Windows are identified by their id and organized hierarchically – have parent attribute that contains ID of their ancestor in the tree.
There is also the process attribute that contains the PID (process ID) of the running program that created this window.
However we should be aware that this information is voluntarily provided by the X11 clients, not managed by the X11 server, and thus might be missing (quite often) or even misleading (rarely).

In order to not get lost in so many windows, we can run a separate X11 server and put only some programs/windows on its display:

Featured Content Ads

add advertising here

COGNAC Metaverse

Explore World's first metaverse that is as real as our Real world.

https://cognac.knowasiak.com

# Run a nested X11 server: 
Xephyr :8 -screen 640x480

# Run an application on its display: 
DISPLAY=:8 xcalc

# List the windows: 
DISPLAY=:8 relpipe-in-x11 --list-windows true | relpipe-out-tabular

This classic calculator

x11_window: ╭─── │ id (integer) ├─── │          681 │ │      2097174 │ │      2097175 │ │      2097180 │ │      2097181 │ │      2097182 │ │      2097183 │ │      2097184 │ │      2097185 │ │      2097186 │ │      2097187 │ │      2097188 │ │      2097189 │ │      2097190 │ │      2097191 │ │      2097192 │ │      2097193 │ │      2097194 │ │      2097195 │ │      2097196 │ │      2097197 │ │      2097198 │ │      2097199 │ │      2097200 │ │      2097201 │ │      2097202 │ │      2097203 │ │      2097204 │ │      2097205 │ │      2097206 │ │      2097207 │ │      2097208 │ │      2097209 │ │      2097210 │ │      2097211 │ │      2097212 │ │      2097213 │ │      2097214 │ │      2097215 │ │      2097216 │ │      2097217 │ │      2097218 │ │      2097219 │ │      2097220 │ │      2097221 │ │      2097222 │ │      2097223 │ │      2097224 │ │      2097225 │ │      2097226 │ │      2097227 │ │      2097228 │ ╰─── Record count: 52

Filtering windows

Another way to deal This kind of SQL expressions

#!/bi

get_x11_child_windows() --relation 'window' --parameter "

format_result()

get_x11_child_windows

When we run this script The root window (screen)

Thanks to Relational

with the abundance of windows is SQL, especially the recursive CTE (common table expressions).
allows us to work with tree structures and filter all descendants of given parent window. n/bash relpipe-tr-sql "$SQL" >$1" cat; "Calculator" | format_result on a desktop with many windows and running XCalc, we will get a listing of the only 51 windows – similar to the listing above.
is not included in this case, but we can also add it using another UNION ALL. pipes modular design we can use also other transformation tools and languages – e.g. Scheme (relpipe-tr-scheme), AWK (relpipe-tr-awk) or XPath (relpipe-tr-xpath). class="apPluginContainer bsaProContainer-9 bsa-default bsa-pro-col-1">

Featured Content Ads

add advertising here

Your AI Companion only on new Bing.

Join the new internet where you get what you ask, Fast and precise.

https://www.bing.com

Embedding and composing windows

Once we know the ID of a particular window, we can have some fun and run another application inside it:

/usr/lib/xscreensaver/euphoria --regular -window-id 148897953
mpv --wid=148897935 "film.mkv"

So we can e.g. watch a film or enjoy a screen saver while calculating:

And we can setup everything automatically whithout typing the IDs by hand:

read_nullbyte()  local IFS=; for v in "$@"; do export "$v"; read -r -d '' "$v"; done 

./the-x11-script-listed-above.sh 
	| relpipe-tr-xpath 
		--relation '.*' 
			--where 'level>=5' 
			--output-attribute 'wid' integer 'id' 
	| relpipe-out-nullbyte | while read_nullbyte wid; do
		sleep 1;
		/usr/lib/xscreensaver/euphoria --regular -window-id "$wid" &
	 done

and experience pure euphoria:

This X11 feature can be used also in much more useful way for composing GUI from several separate programs that might be even written in different programming languages.

Listing input devices

In order to list our keyboards, mice and similar devices we will invoke:

relpipe-in-x11 | relpipe-out-tabular

or we can explicitly say --list-input-devices true; in both cases we will get a listing similar to this one:

x11_input_device:
 ╭──────────────┬───────────────────────────────────────────────────────────┬───────────────╮
 │ id (integer) │ name                                             (string) │ type (string) │
 ├──────────────┼───────────────────────────────────────────────────────────┼───────────────┤
 │            2 │ Virtual core pointer                                      │               │
 │            3 │ Virtual core keyboard                                     │               │
 │            4 │ Virtual core XTEST pointer                                │               │
 │            5 │ Virtual core XTEST keyboard                               │               │
 │            6 │ Power Button                                              │ keyboard      │
 │            7 │ Video Bus                                                 │ keyboard      │
 │            8 │ Power Button                                              │ keyboard      │
 │           10 │ Logitech USB Trackball                                    │ mouse         │
 │           11 │ ZSA Technology Labs Inc ErgoDox EZ Shine                  │ keyboard      │
 │           12 │ ZSA Technology Labs Inc ErgoDox EZ Shine                  │ mouse         │
 │           13 │ ZSA Technology Labs Inc ErgoDox EZ Shine System Control   │ keyboard      │
 │           14 │ ZSA Technology Labs Inc ErgoDox EZ Shine Consumer Control │ mouse         │
 │           15 │ ZSA Technology Labs Inc ErgoDox EZ Shine Keyboard         │ keyboard      │
 │           16 │ AT Translated Set 2 keyboard                              │ keyboard      │
 │           17 │ ZSA Technology Labs Inc ErgoDox EZ Shine Consumer Control │ keyboard      │
 │            9 │ 3Dconnexion 3Dconnexion Universal Receiver Mouse          │ mouse         │
 ╰──────────────┴───────────────────────────────────────────────────────────┴───────────────╯
Record count: 16

Device IDs can be used to identify the source of input events and their filtering.

Capturing input events

Once we connect to an X11 server (through the relpipe-in-x11 X11 client in our case)
we can listen to the input events and monitor the key codes and mouse movements.

relpipe-in-x11 --list-input-events true | relpipe-out-csv

To see the events immediatelly, it is important to use an output filter that does no buffering (e.g. relpipe-out-csv or relpipe-out-gui).

"device","type","state","key","button","x","y"
"11","key","released","36","-1","1405","1416"
"11","key","pressed","38","-1","1405","1416"
"11","key","released","38","-1","1405","1416"
"11","key","pressed","43","-1","1405","1416"
"11","key","released","43","-1","1405","1416"
"11","key","pressed","32","-1","1405","1416"
"11","key","released","32","-1","1405","1416"
"11","key","pressed","44","-1","1405","1416"
"11","key","released","44","-1","1405","1416"
"9","motion",,"-1","-1","1405","1416"
"9","motion",,"-1","-1","1405","1417"
"9","motion",,"-1","-1","1406","1418"
"9","motion",,"-1","-1","1406","1419"
"9","button","pressed","-1","1","1406","1420"
"9","button","released","-1","1","1406","1420"
"9","motion",,"-1","-1","1406","1420"
"9","motion",,"-1","-1","1405","1420"

In order to capture events from a different X11 server, we set the DISPLAY environment variable.
This way, we can also capture events remotely over SSH.

ssh example.com DISPLAY=:0 relpipe-in-x11 --list-input-events true | relpipe-out-csv

We can log the events to a file or forward them to another X11 server through relpipe-out-x11.
Or we can multiplex the events and forward them to several X11 servers e.g. to run simultaneous tests of different versions or configurations or certain GUI application.

Simulating input events

With the relpipe-out-x11 client we can connect to an X11 server and emit input events like they come from a keyboard or mouse
(actually they would come from a virtual/simulated one).
This command reads relational data with same structure as produced by its counterpart relpipe-in-x11.
We can use data captured earlier or create some new one using any tool like
relpipe-in-cli or other relpipe-in-* optionally transformed through a relpipe-tr-* filter.
We may also translate other signals like MIDI events to X11 ones (relpipe-in-jack).

For example, the following command moves the cursor to the coordinates 100,200 and then to 640,480.
Such events might be produced also in a shell loop and passed through the pipe to a single relpipe-out-x11 process
(i.e. without connecting and disconnecting the X11 server on each event).

relpipe-in-cli 
	--relation "x11_input_event"      
		--attribute "type" string     
		--attribute "x"    integer    
		--attribute "y"    integer    
		--records 
			"motion" "100" "200" 
			"motion" "640" "480" 
	| relpipe-out-x11

The relpipe-out-x11 tool has two options:
--dry-run true that suppresses the events and just tests validity of the input; and
--debug true that prints the events in a text form (XML) on STDERR (so we can see that something comes through).

relpipe-in-x11 --list-input-events true | relpipe-out-x11 --dry-run true --debug true

Using SSH, DDC, relpipe-in-x11 and relpipe-out-x11,
we can build a software replacement of a KVM switch and control several computers from one seat (keyboard, video, mouse).

Knowasiak