When Camtasia Studio or SnagIt grabs a solid block of color instead of an image which looks fine on the screen, the problem is usually due to graphics hardware acceleration being used to display the image. The newer media players from Microsoft and RealNetworks are good about trying to use any graphics hardware acceleration that is available on the system.
For example, some systems have graphics hardware that can perform a "hardware overlay" of video on the screen, which bypasses the normal Windows display memory that Camtasia Studio and SnagIt capture. Most video capture programs that capture video from a video camera use "hardware overlay" by default for their video preview. Note that this kind of problem affects all screen capture programs, and even the image put in the clipboard by the Print Screen key.
The best solution is to disable hardware acceleration in the application that is playing the video, or disable hardware acceleration system wide. Because this effects all applications on the system, it may cause performance problems for some applications and others may not run at all.