One of the users asked me to write a small encoding guide to use high
resolution mpeg2, so i wrote him a quick one with the pack of needed
tools. I guess you already have most of these, but I have finetuned using

various collected information and custom quant matrices my settings for

high resolution mpeg2, and the quality of the encodes is at least comparable

to h.264 at the same bitrate .). I will paste you the guide here, but I

guess the only relevant things/information for you are the custom matrix

from the archive and the quenc settings I use. It is actually finetuned for

720p content, but SMS can't play that thanks to the texture size limit.

For maximum size possible (1024*576) there are little artifacts in corners

in 720p because we are using too much vram. It really does give excellent

quality.

- Download THIS archive. It contains all the tools you need. I assume you

  have proper codecs for playing everything on you computer installed;
- Install avisynth from tools folder;
- Unpack quenc somewhere where you like;
- Copy sagit.xcm to the folder where your quenc is (or somewhere where you
  can find it later when you will be setting up quenc);
- Copy the avisynth sample script somewhere and open it in you favourite
  text editor. You will see two commands, change the first one so that it
  points to the file you want to encode, and the second one to the
  resolution you want to use. With the resolution settings you see there,

  you will get CORRUPTION. You can't have bigger horizontal width then 1024

  because of maximum ps2 texture size limit. So my recommendation is probably

  960x540 or a little bit more, try a few tests to get picture without any

  corruption;
- After you have edit this file, fire up quenc program, select the avisynth

  script, and set everything exactly as it is in Quenc_Settings.jpg. This

  settings are actually optimal to encode to 720p mpeg2, so you can probably

  lower the bitrate a little bit. You have to select the Custom quant matrix

  sagit.xcm which was in the archive. There is also utility qmatrix in the

  archive which is a editor of custom matrices, with plenty of examples.

  However the matrix I use is especially good for high resolutions. So if

  you have set up everything correctly you can start your encode. In the way

  I have it set up, it is slow, but the quality at the bitrate you  get is

  comparable if not almost equal to h.264. Try playing with some small sample

  files and you will see what you can get;

Peter
 

Thanks to all authors of the used programs, doom9.org forums and
doom9.org forum users.