- Joined
- Jan 16, 2009
- Messages
- 716
tldr: The goal of this post is to find an accurate and robust get_mouse_screen_position() for the community.
Big thanks to @Unryze for helping me find stuff in the 1.26 game.dll!
So for a few patches now, there is a way to get the mouse world coordinates. However there is no way to get the mouse screen coordinates directly.
This shouldn’t be an issue at all since all the data required to transform world coordinates to screen coordinates should be here.
The world camera’s view and projection matrices are needed for that.
Using the camera’s eye, target, field of view, near_z and far_z and the screen’s aspect ratio the 2 transforms are computed (view transform, projection transform)
However, one might notice that the camera’s target is not in the center of the screen but slightly
higher. In fact it’s centered on (0.355/0.6) * screen_height.
This is known because by digging in the 1.26 game.dll code, it can be found that the world frame starts vertically at 0.13 and ends at 0.58 (in UI coordinates). For reforged they have kept this centering but expanded the frame in some way.
Still, while using this to correct the aspect ratio to
For instance when the mouse is at the 0.13 ui height, the projection doesn’t output -1 for the y-axis. In the same way, when the mouse is at the 0.8 ui width, the projection doesn’t output 1 for the x-axis.
Now if one digs more into the 1.26 game.dll code, the projection matrix used by warcraft 3 is quite different.
Instead of having
Now if this projection matrix is used, the projection output starts to get really close to what is expected. However it doesn't match quite perfectly (by 1% pretty much).
Moreover, as soon as the fov or the world frame’s aspect ratio is modified, the deviation starts to increase to the point of this not being usable robustly for a get_mouse_screen_position(). Even using the mouse scroll zoom in fully seems to skew the results.
At this point I believe I am missing some step that was either added in reforged or that was not found in the 1.26 game.dll or maybe something else entirely.
So please share any ideas or findings you might have on this matter so we can finally have an accurate and robust get_mouse_screen_position().
Here is my pseudocode to get an okayish get_mouse_screen_position() with the default camera (fov 70 and non altered world frame).
Big thanks to @Unryze for helping me find stuff in the 1.26 game.dll!
So for a few patches now, there is a way to get the mouse world coordinates. However there is no way to get the mouse screen coordinates directly.
This shouldn’t be an issue at all since all the data required to transform world coordinates to screen coordinates should be here.
The world camera’s view and projection matrices are needed for that.
Using the camera’s eye, target, field of view, near_z and far_z and the screen’s aspect ratio the 2 transforms are computed (view transform, projection transform)
However, one might notice that the camera’s target is not in the center of the screen but slightly
higher. In fact it’s centered on (0.355/0.6) * screen_height.
This is known because by digging in the 1.26 game.dll code, it can be found that the world frame starts vertically at 0.13 and ends at 0.58 (in UI coordinates). For reforged they have kept this centering but expanded the frame in some way.
Still, while using this to correct the aspect ratio to
0.8/(0.58 - 0.13), the values outputted by the projection transform are not what is expected. For instance when the mouse is at the 0.13 ui height, the projection doesn’t output -1 for the y-axis. In the same way, when the mouse is at the 0.8 ui width, the projection doesn’t output 1 for the x-axis.
Now if one digs more into the 1.26 game.dll code, the projection matrix used by warcraft 3 is quite different.
Instead of having
1 / tan(0.5 * field_of_view_y) for the [2][2] field, they actually have 1 / tan(0.5 * field_of_view_y/square_root(aspect_ratio * aspect_ratio + 1)).Now if this projection matrix is used, the projection output starts to get really close to what is expected. However it doesn't match quite perfectly (by 1% pretty much).
Moreover, as soon as the fov or the world frame’s aspect ratio is modified, the deviation starts to increase to the point of this not being usable robustly for a get_mouse_screen_position(). Even using the mouse scroll zoom in fully seems to skew the results.
At this point I believe I am missing some step that was either added in reforged or that was not found in the 1.26 game.dll or maybe something else entirely.
So please share any ideas or findings you might have on this matter so we can finally have an accurate and robust get_mouse_screen_position().
Here is my pseudocode to get an okayish get_mouse_screen_position() with the default camera (fov 70 and non altered world frame).
Code:
position // The mouse position.
// View transform
world_up = (0,0,1)
eye = get_camera_eye()
target = get_camera_target()
forward = get_normalized(eye-at); // in RH coord system, z goes backward, hence 'eye-at' instead of 'at-eye'.
right = get_normalized(get_cross_product(world_up, forward));
up = get_cross_product(forward, right);
view_11 = right.x
view_12 = up.x
view_13 = forward.x
view_21 = right.y
view_22 = up.y
view_23 = forward.y
view_31 = right.z
view_32 = up.z
view_33 = forward.z
view_41 = -get_dot_product(right, eye)
view_42 = -get_dot_product(up, eye)
view_43 = -get_dot_product(forward, eye)
view_position = (position.x * view_11 + position.y * view_21 + position.z * view_31 + view_41,
position.x * view_12 + position.y * view_22 + position.z * view_32 + view_42,
position.x * view_13 + position.y * view_23 + position.z * view_33 + view_43)
// Projection transform
fov = get_camera_fov()
near = get_camera_near()
far = get_camera_far()
world_frame_top = 0.58
world_frame_bottom = 0.13
world_frame_left = 0.
world_frame_right = 0.8
world_frame_aspect_ratio = (world_frame_right - world_frame_left) / (world_frame_top - world_frame_bottom)
projection_22 = 1.0 / tan(0.5 * fov / square_root(world_frame_aspect_ratio * world_frame_aspect_ratio + 1.0))
projection_11 = projection_22 / world_frame_aspect_ratio
projection_33 = (near + far) / (far - near)
projection_43 = (-2. * near * far) / (far - near) // This was also found in the 1.26 dll.
projection_34 = -1
w = view_position.z * projection_34 // homogeneous coordinates allow for the projection to actually be doable with a matrix transform.
inverse_w = 1.0 / w
projection_xy = (inverse_w * (view_position.x * projection_11),
inverse_w * (view_position.y * projection_22))
ui_position = (world_frame_left + (world_frame_right - world_frame_left) * (0.5 + 0.5 * projection_xy.x),
world_frame_bottom + (world_frame_top - world_frame_bottom) * (0.5 + 0.5 * projection_xy.y))
print("ui_position = "+to_string(ui_position))
