Since MythTV (and Linux in general) seems to like ffmpeg for video encoding/decoding, I figured I’d use that. You can get a binary version for Windows and read the documentation.

The actual command line I use to transcode is:
ffmpeg -i $in_file -vcodec xvid -qscale 5 -acodec copy $out_file

That means to use $in_file as input (a VOB file in my case), use the Xvid codec for the video, set the “quality” to 5, copy the audio straight from the original and save as $out_file. The quality in this case is just simplification of lots of other settings that are available. 1 is perfect and 31 is the worst. 5 results in files that are about 500MB per hour with MPEG artifacts that are visible when I’m sat at y desk but not when I sit on my bed six feet away which is where I normally watch video from. It may be worth transcoding a short clip with a few different settings to see which your happy with.

I made the whole process semi-automatic by writing a CLI PHP script that checks for VOB files in a specifc folder and transcodes the ones it finds. That way I can have the transcoding going on in the background while I rip the DVDs (and then leave it running it overnight to finish). I could make it available to anyone who wants it, but a batch files doing the same thing would probably be more useful for people…

There is one last caveat. I originally encoded the movies with MP3 audio and then half way though decided I want to keep the 5.1 audio (which the above method does). However the version of ffmpeg I used at first had a problem such that AVIs with AC3 audio played back with no sound. If you have a similar problem make sure you have the latest version of ffmpeg you can get.