the main program uses a dialog box as its main window. it defines a custom message, wm_mousehook which will be used between the main program and the hook dll. when the main program receives this message, wparam contains the handle of the window that the mouse cursor is on. of course, this is an arbitrary arrangement. i decide to send the handle in wparam for the sake of simplicity. you can choose your own method of communication between the main program and the hook dll.
.if hookflag==false
invoke installhook,hdlg
.if eax!=null
mov hookflag,true
invoke setdlgitemtext,hdlg,idc_hook,addr unhooktext
.endif
the program maintains a flag, hookflag, to monitor the state of the hook. it's false if the hook is not installed and true if the hook is installed.
when the user presses hook button, the program checks if the hook is already installed. if it is not, it call installhook function in the hook dll to install it. note that we pass the handle of the main dialog as the parameter of the function so the hook dll can send the wm_mousehook messages to the right window i.e. our own.
when the program is loaded, the hook dll is loaded too. actually, dlls are loaded immediately after the program is in memory. the dll entrypoint function is called before the first instruction in the main program is execute even. so when the main program executes the dll(s) is/are initialized. we put the following code in the dll entrypoint function of the hook dll:
.if reason==dll_process_attach
push hinst
pop hinstance
.endif
the code just saves the instance handle of the hook dll itself to a global variable named hinstance for use within the installhook function. since the dll entrypoint function is called before other functions in the dll are called , hinstance is always valid. we put hinstance in .data section so that this value is kept on per-process basis. since when the mouse cursor hovers over a window, the hook dll is mapped into the process. imagine that there is already a dll that occupies the intended load address of the hook dll, the hook dll would be remapped to another address. the value of hinstance will be updated to those of the new load address. when the user presses unhook button and then hook button, setwindowshookex will be called again. however, this time, it will use the new load address as the instance handle which will be wrong because in the example process, the hook dll's load address hasn't been changed. the hook will be a local one where you can hook only the mouse events that occur in your own window. hardly desirable.
installhook proc hwnd:dword
push hwnd
pop hwnd
invoke setwindowshookex,wh_mouse,addr mouseproc,hinstance,null
mov hhook,eax
ret
installhook endp
the installhook function itself is very simple. it saves the window handle passed as its parameter to a global variable named hwnd for future use. it then calls setwindowshookex to install a mouse hook. the return value of setwindowshookex is stored in a global variable named hhook for use with unhookwindowshookex.
after setwindowshookex is called, the mouse hook is functional. whenever a mouse event occurs in the system, mouseproc ( your hook procedure) is called.
mouseproc proc ncode:dword,wparam:dword,lparam:dword
invoke callnexthookex,hhook,ncode,wparam,lparam
mov edx,lparam
assume edx:ptr mousehookstruct
invoke windowfrompoint,[edx].pt.x,[edx].pt.y
invoke postmessage,hwnd,wm_mousehook,eax,0
assume edx:nothing
xor eax,eax
ret
