Depends on your version of windows?
For me (on 2000) I:
load up windows explorer, goto tools->folder options
and then click the file type TAB.
That will list ALL known file associations.
Scroll to the avi/mpeg or what ever file you want.
Highlight it and click the ADVANCED button.
That will bring up the CHANGE ICON window, and you can browse icons on your local drive or open .exe files and get access to icons etc etc.
Might be easier on XP but not been on that OS for a loooong time
