Ok, I found the solution, in case anyone else runs into this. Windows only sends mouse wheel events to the control that has focus, but Panel is not a focusable control by default. There doesn't seem to be a way to easily change this, so I added the following in my Form's ctor to force the panel to focus when the mouse hovers over it
drawingPanel.MouseHover += (sender, args) => drawingPanel.Focus();