I return with mixed results and theories.
qyot27_foxyhd_720p29.97.mp4
Played with audio only, similar symptoms to a bad mux (like the broken MP4Box builds I mentioned). I then extracted and remuxed the streams and it's still the same. I'm quite lost on this one. The only things that raise any suspicion is that it's CRF15, so there's a slight possiblity it was bitrate related (although it was a simple source so I somehow doubt that).
Then I noticed the max consecutive B-frames was set at 16; this may be the problem even though in normal sources it's unlikely you will get so many consecutive B-frames, but I would have expected to see a few frames up until a large sequence of B-frames and then just see the player lag rather than not display any video at all. Also it's a long shot, but it could even be a problem that MKV has caused - also whether it was encoded in Virtualdubmod (in other words VfW) or not is relevant (though I suspect you know better than to use VfW).
qyot27-killingbeta-hd_720p59.94.mp4
Slightly better results. Plays fine up until 11 seconds where it stutters and loses the audio - I wondered why since an ASP encode at 720p shouldn't really trouble a player that's been dealing with 1080i H.264 at 20mbps, but the bitrate is the reason why. At this part in the video, the bitrate skyrockets. Overall I think I found a peak of 35mbps, and 11 seconds in is roughly where it reaches 30mbps, which is hardly a surprise that it failed (yet it could even be a fragged HD, so don't discount that possibility).
I pondered it for a while, but then remembered that HD-DVD and Bluray players do not support MPEG-4 ASP, and neither does this chipset AFAIK, therefore I would assume that ASP decoding is done in software and if that's the case then I would assume the ASP decoding is limited because naturally if they have hardware MPEG-2/WMV/H.264 decoding, they would want to keep the onboard CPU low power and cheap. Also for what it's worth, ASP officially supports a maximum of 720x576 @ 30fps with a max bitrate of 8mbps, so it's not like I'm not getting something I should, so I'm not so dissapointed with that.
Since no one wanted to volunteer, I grabbed some AMVs for testing. First (in no particular order) is:
A Winter's Solstice - anubisx00.avi
Pass. Perfect sync and no judder due to framerates. Looks pretty awesome.
Bladebeat Graffiti - Kisanzi - VNS.avi
Pass. Perfect sync and no judder due to framerates. Looks pretty awesome. Would have benefitted from H.264
[BasharOfTheAges[Genkaiten.mp4
Pass. Perfect sync and no judder due to framerates. Looks pretty awesome.
[Decoy]-Bleach_Technique_Beat-_-Curve-_-Chinese.mp4
Fail. Displayed symptoms of a bad mux. Audio played with no video. I then extracted the raw H.264 and AAC, then muxed it back to MP4 using the MP4Box I linked to earlier. Now plays perfectly and get's same comments as previous.
[Koop] Skittles (1280x720).mp4
Pass. There were a few occasions where I thought I saw lag, but that might be framerate problems with my hardware (I have a 25fps PAL TV, but it can also do 29.97 fps NTSC. If a file is 23.976fps, it may judder in either mode due to duplicate frames and such).
Edit- Max bitrate is OVER NINE THOUSAAAAAAAANDkbps (actually 9800, should be no problem) and framerate is 24fps, so it's likely it's just duplicate frame judder (24>29.97 or 25>24)
I also tested Nero's 1080p H.264 trailer with little success. At first the player refused due to it having an "unsupported Nero Digital profile", which was easily solved by extracting the raw streams and putting them into MP4 by yourself. It plays awesome for about 30 seconds but as it gets complex it slows down and the audio cuts out.
When I say bad MP4Box builds, at the moment I can only assume. I find it strange that the player will not play them, but equally strange that software players do. Is it the player that needs more MP4 support, or are the files incorrect but software splitters are more forgiving?