I found the bug, and fixed it! However, the basic problem is that the kqueue functions that Xee uses to monitor changes in the filesystem (even when it does things like delete an image itself) don't work on SMB shares. So behaviour will always be a bit broken on SMB, but at least browsing works right now.
(Turns out the trigger for the bug is simply viewing enough images on an SMB share. Just opening one and holding down the skip-to-next button will trigger it after a while.)