Compiling x264 on Linux

[Qyot27]

While I know many of you that use Linux around here probably don't blink at compiling your own builds, I feel that it could be of use to have a thread detailing how to do so for those that want to, but don't have much (or any) experience with compiling programs. Heck, I'm just about clueless myself, although I have managed to do so when I need to.

One of the biggest benefits, at least to me, of using x264 under Linux is because the sole 64-bit capable machine I have access to only has a 32-bit version of Windows on it. For people in this kind of situation, Linux is a good way to use the full 64-bit power of the computer while not needing to buy another OS (unless you go with Red Hat, et al). And the difference in encoding time between 32- and 64-bit versions of x264 can be pretty dramatic.

The best approach, in my opinion, would be to set this up by distribution (and version, if you feel the need, and the process differs so greatly). For instance:

Ubuntu 8.10 [Intrepid Ibex]:
0. This process uses GCC 4.3.2 to do the compiling. I don't know how this might change under a different version of GCC. Just be aware of this fact. Also be aware that this may require that you have the Universe and/or Multiverse repositories open.

1. Install the necessary dependencies (you can do this in Synaptic if you want, but I'm going to use the Terminal commands for ease of doing all this in one go). You only need to do this once.

Code: Select all

sudo apt-get update
sudo apt-get install build-essential git-core checkinstall yasm texi2html

2. Download, compile, and install zlib. zlib is required for building GPAC, which provides the MP4 output support in x264.

Code: Select all

wget http://www.zlib.net/zlib-1.2.3.tar.gz
tar -zxf zlib-1.2.3.tar.gz
cd zlib-1.2.3
./configure
make
sudo make install

3. Download, compile, and install GPAC 0.4.5. Once again, you'll only need to do this once (although feasibly when GPAC updates it might be a good idea; the reason I don't suggest this for zlib is because, well, the main site's source tarball hasn't been updated since 2005, and I don't know how easily this works using a newer version).

Code: Select all

wget http://superb-east.dl.sourceforge.net/sourceforge/gpac/gpac-0.4.5.tar.gz
wget http://voxel.dl.sourceforge.net/sourceforge/gpac/gpac_extra_libs-0.4.5.tar.gz
tar -zxf gpac-0.4.5.tar.gz
tar -zxf gpac_extra_libs-0.4.5.tar.gz
cd gpac_extra_libs
cp -r * ../gpac/extra_lib
cd ../gpac
chmod +x configure
./configure --disable-opengl --use-js=no --use-ft=no --use-jpeg=no --use-png=no --use-faad=no --use-mad=no --use-xvid=no --use-ffmpeg=no --use-ogg=no --use-vorbis=no --use-theora=no --use-openjpeg=no
make lib
make apps
sudo make install-lib
sudo make install
sudo cp bin/gcc/libgpac.so /usr/lib

(The string of disables I used with the configure step is a result of my paranoia about getting GPAC to build. The only ones indicated on the help forum over at Sourceforge as needing to be disabled were opengl, jpeg, and png. However, since what I put down above worked, then I don't mind. If I get confirmation on how the others don't affect the build process, I'll amend that step. But until then, I'd just rather not take the chance)

4. Download, compile, and install x264. If you want to use some of the common patches* (see links below), download the diff files and copy them to the x264 directory (the one git creates). Then replace file.diff in line #4 to the actual name of the patch and repeat for as many of the patches as necessary.

Code: Select all

cd ~/
git clone git://git.videolan.org/x264.git
cd x264
patch -p1 <file.diff
./configure --enable-shared --extra-cflags="-march=pentium3"
make
sudo checkinstall --fstrans=no --install=yes --pkgname=x264 --pkgversion "1:0.svn-$(grep X264_VERSION config.h | cut -d' ' -f 4)-0.0ubuntu1" --default
sudo ldconfig

When using ./configure, be sure to set the --extra-cflags option to the proper CPU type you have. I use Pentium 3 on mine because I have a Celeron 2; on another computer I use -march=athlon64, since it has an Orleans processor. You can check the list of proper CPU flags here:
http://gcc.gnu.org/onlinedocs/gcc-4.3.2 ... 64-Options

*Patches:
x264_win_zone_parse_fix_05.diff:
http://skystrife.com/x264/

x264_hrd_pulldown.10_interlace.diff:
http://forum.doom9.org/showpost.php?p=1 ... count=1726

Keep up with patches here:
http://forum.doom9.org/showthread.php?t=130364


Now then, as far as using x264 is concerned, I'm going to assume you already know how to use x264 from the command line. If necessary, fire up MeGUI or Zarx264gui under Windows, make them expose the command line options, copy those down, and then bring that text file over to your Linux install.

What I'm going to explain instead is how to feed x264 your video so that you don't need to bother with trying to get raw video encoded correctly (which can be a real bitch - see my second post for a quick fix of sorts).

Linux builds of x264 don't support AVI or AviSynth input, because of fairly obvious reasons (lack of VFW and subsequently, lack of a *working* AviSynth 3.0). This presents a challenge, because otherwise you would need to output your video in raw format and then give this to x264 - however, doing that eliminates both the framerate and resolution information from the file, and requires you to specify this in x264's options. Quite frankly, this can be annoying. But there is a solution. Let's go into more detail below.

You'll want to make sure that mplayer is installed first. The best option for this is to go to Applications->Add/Remove Programs, and find SMPlayer (which I prefer because of the GUI; if you want standard mplayer, you can get it from Synaptic or something). Install it.

The video should be in a lossless format that mplayer can open. This could be any number of things; personally, I use YV12-mode HuffYUV (ffvhuff, if you wanna be technical about it). Uncompressed YV12 would be fine as well.

Now that that's done, we're all set. What's going to happen is that we'll make a named pipe using mkfifo, then have mplayer open the video, and pipe it over to x264.

Code: Select all

mkfifo input.y4m
x264 input.y4m <options> | mplayer -nosound -vo yuv4mpeg:file=input.y4m inputfile

In reality, this would probably look something like this:

Code: Select all

mkfifo input.y4m
x264 input.y4m --crf 18.0 --ref 16 --mixed-refs --no-fast-pskip --bframes 16 --b-adapt 2 --b-pyramid --weightb --direct auto --deblock 1:1 --subme 9 --trellis 2 --partitions all  --8x8dct --scenecut 100 --threads auto --thread-input --sar 1:1 --aud --progress --no-dct-decimate --no-psnr --no-ssim --output "dawn.mp4" | mplayer -nosound -vo yuv4mpeg:file=input.y4m dawn.avi

Most of this information is cobbled together from various other sources (primarily the Doom9 forums and Ubuntu forums) I found online or asked about directly. If you want to customize your builds with special flags then you can check with ./configure --help on a per-program basis.

[Qyot27]

For making the proper raw files to give to x264, you can process the files using mencoder. The easiest way is to edit mencoder.conf to always be set to a certain type of compression.

x264 only takes I420 format video. So, you'd make mencoder.conf look like this, more or less:

Code: Select all

ass=no
subcp=enca:ru:cp1251
fontconfig=no
nosound=yes
autoexpand=no
of=rawvideo=yes
ovc=raw=yes
rawopts="vf format:I420"
ffourcc=I420
force-avi-aspect=1/1

This will allow you to use this simple command-line to convert to the right format:

Code: Select all

mencoder inputfile -o outputfile

This can be used under either a Windows version (requires Wine) or the version that comes with SMPlayer. Using Wine, you'll want to put mencoder.exe and the mplayer subfolder that holds mencoder.conf in Wine's windows folder. You'll also need to make sure to call wine mencoder when you use the Terminal rather than simply mencoder. For the native version, you'll stick mencoder.conf in ~/.mplayer and go on your merry way.

[dragontamer5788]

Woah woah woah... someone other than me uses Linux? o.O

Do you use Cinelerra for editing too? :-p

------------

Anyway, I run Debian, which is what Ubuntu is apparently based on. I get all of my multimedia programs from the apt repository at debian-multimedia. Just add:

Code: Select all

deb http://www.debian-multimedia.org lenny main
deb-src http://www.debian-multimedia.org lenny main

Then of course...

Code: Select all

aptitude update
aptitude install x264

Which installs x264, no sweat . I'm unsure if this will work on Ubuntu (those are debian packages afterall...) but Ubuntu apparently has decent compatibility with Debian. Anyway, I just get mencoder, cinelerra, x264 and so on from there. Unless you want the latest bleeding edge builds... I don't think it really matters to go about compiling the development branch... unless... there is something that I'm missing? The version of x264 in the debian-multimedia repository is 0.65.x. Is there anything that I'm missing by sticking with this slightly older version?

[Qyot27]

Nah, I could never get into Cinelerra. I still do my editing in Premiere under Windows. Once Saya (or Lumiera) gets released I'll take a look at those, though.

Well, the bleeding edge stuff is basically the reason I posted this, as those accustomed to using MeGUI know that it's the current revisions that get added to the update chain. Unfortunately, having builds that new (and really, some repositories aren't exactly *new* considering they could be holding a build several months old - and with x264's fairly rapid development pace, it means things get outdated quickly) on Linux means compiling it yourself. The 0.65 part of the version number doesn't tell me much because I pay attention to the revision number, which is the final part - currently at 0.67.1129. 'r1129' tells me more than '0.67' does.

Anyway, are the x264 builds in the Debian repos enabled for MP4 output? There's also other minor issues such as optimizing for one's processor and including relevant patches (for Blu-ray authoring compatibility, as an example) that can be addressed with doing the compiles yourself. On Windows, others supplying up-to-date builds is a pretty well-oiled machine and has been for a couple years now. But that doesn't exist so much on other platforms because it's assumed you can compile them yourself if you want them. It's just that sometimes it's difficult to bring all the relevant info on doing so together so that those starting out can get a handle on it.

[dragontamer5788]

MP4 is a good reason... but more seriously... they've already went up a minor version? Wow, you're right, x264 is pretty fast. Generally, I prefer to avoid the hassle of compiling, even if it isn't fully up-to-date. Also, I haven't bothered to learn x264 yet... and just do all my processing from MEncoder. (which uses x264 library, so I can get h.264 encodes but it is probably even less up-to-date). More of a "yeah, I know the work in compiling stuff... and it ain't pretty "

Looking over a few things however...

When using ./configure, be sure to set the --extra-cflags option to the proper CPU type you have. I use Pentium 3 on mine because I have a Celeron 2; on another computer I use -march=athlon64, since it has an Orleans processor. You can check the list of proper CPU flags here:

Erm... you have a ~10 year old computer? I somehow doubt that. It is sometimes difficult to figure out the true core of a Celeron... which can be anything from a Pentium 2 up to a Core 2 Duo! Aside from the cache, they behave like whatever core they are, except they have smaller cache. So run...

Code: Select all

cat /proc/cpuinfo

and you'll get the output of your CPUinfo. From there, you can select the real architecture of your machine (or at least... the one hardwired into your CPU)

[Qyot27]

Erm... you have a ~10 year old computer? I somehow doubt that. It is sometimes difficult to figure out the true core of a Celeron... which can be anything from a Pentium 2 up to a Core 2 Duo! Aside from the cache, they behave like whatever core they are, except they have smaller cache. So run...

Code: Select all

cat /proc/cpuinfo

and you'll get the output of your CPUinfo. From there, you can select the real architecture of your machine (or at least... the one hardwired into your CPU)

Yes, my computer is that old. It's an eMachines T1110 - 256 MB RAM, 1 GHz Celeron 2/Pentium 3 Coppermine (which I was aware of not only from CPU-Z, but also from the fact it's what mencoder and mplayer spit out at me), 100 MHz FSB, i810e chipset. It's from 2001, actually. I'm sorely in need of an update, I know. And yes, it's also my editing setup, and has been since mid-2003. I have it set up as a dual-boot between XP Home SP3 and Ubuntu 8.10. The only reason for XP Home is because that's what the restore disc the computer came with uses. I bought XP Pro SP2 (since slipstreamed to include SP3) last year just before the retail deadline went into effect, but save for very short spans of time in a virtual machine, I'm saving that for when I get a better setup.

Heck, my family even still has the NEC computer we bought about 14 years ago...runs Windows 95 on a Pentium-166 (or maybe it was a -133). And it still works fine, although the monitor is a tad screwy.

[trythil]

patch -p1 <file.diff

Doing that doesn't take advantage of git's versioning capabilities, which is something that you'd want to do if you plan to upgrade to newer versions of the software tools. Instead, do:

Code: Select all

git am file.diff

git-am applies patches generated by git format-patch (and any other patch mechanism that generates similarly-formatted patches). The advantage of doing it this way is that you append to your repository's history, which can make it easier to merge patches later on. It also allows you to back out patches that don't apply cleanly, and all the other fun stuff that you get with version control.

Re: mplayer: you can also use raw YV12 output (-of rawvideo -ovc raw -nosound), which can be handy if yuv4mpeg messes up on you.

Re: cflags: I don't think explicitly setting architecture makes much difference to x264. The critical bits of x264 are already hand-written assembler, and the configuration script will determine the correct platform to use automatically.

[Qyot27]

patch -p1 <file.diff

Doing that doesn't take advantage of git's versioning capabilities, which is something that you'd want to do if you plan to upgrade to newer versions of the software tools. Instead, do:

Code: Select all

git am file.diff

git-am applies patches generated by git format-patch (and any other patch mechanism that generates similarly-formatted patches). The advantage of doing it this way is that you append to your repository's history, which can make it easier to merge patches later on. It also allows you to back out patches that don't apply cleanly, and all the other fun stuff that you get with version control.

Well, I just found the reference to using patch in one of Dark Shikari's posts that's buried in the Current Patches thread, and just went with that. Didn't know about git's internal patching capabilities. When you say 'append to your repository's history', that refers to what exactly? When I go to update x264 I actually completely remove it first, then completely remove the source directory and download everything from git again, since I can't see as many issues arising that way. Then again I tend to be paranoid on some issues and not on others that are seemingly just as important.

Or newer software tools in reference to git and gcc, rather than x264?

Re: mplayer: you can also use raw YV12 output (-of rawvideo -ovc raw -nosound), which can be handy if yuv4mpeg messes up on you.

Those options are actually covered in the mencoder.conf in the second post. -vf format=I420 was also specifically required in the tests I ran, unless that's what it automatically defaults to in the case of YV12 (I can only remember it kept telling me I need to declare -vf format). -vf format=YV12 causes color errors.

Re: cflags: I don't think explicitly setting architecture makes much difference to x264. The critical bits of x264 are already hand-written assembler, and the configuration script will determine the correct platform to use automatically.

True, but the cflags make sure that everything (or nearly everything?) else is optimized also.

In my tests, I used just --crf 18 and --progress (dunno if --progress really affects speed, though). Running a non-cflags-enabled build and one with them enabled netted an increase on average from 0.5 to 1.0 frame per second, which considering my computer's age is pretty significant (the actual numbers were something like 3.65->4.02fps or 3.65->4.53fps). I noticed similar little jumps on the Athlon64 machine I tested on using -march=athlon64, although there it seemed like less of a difference (2-3 frames, but only from say, 12-14fps to 15-17fps, respectively). Normally I don't stick to using just --crf 18, though, and any sort of percentage boost can help when dealing with average fps render values in the 0.2-0.3 range or lower.

At this point I can't remember if I was testing 720p or 480p, as I had done both a few times.

[trythil]

Well, I just found the reference to using patch in one of Dark Shikari's posts that's buried in the Current Patches thread, and just went with that. Didn't know about git's internal patching capabilities. When you say 'append to your repository's history', that refers to what exactly? When I go to update x264 I actually completely remove it first, then completely remove the source directory and download everything from git again, since I can't see as many issues arising that way. Then again I tend to be paranoid on some issues and not on others that are seemingly just as important.

git's a version control system. It's used to manage (among many other projects) the Linux kernel codebase. (Can you imagine deleting and re-fetching all that every time you wanted to update?)
git-pull is designed to bring your local repository up-to-date, and merge any commits that you haven't pushed upstream. It is very rarely necessary to delete and re-fetch.

When you commit changes (i.e. apply patches), that change is recorded. A series of changes is a git history.

A good git tutorial is here: http://www.kernel.org/pub/software/scm/ ... orial.html

Those options are actually covered in the mencoder.conf in the second post. -vf format=I420 was also specifically required in the tests I ran, unless that's what it automatically defaults to in the case of YV12 (I can only remember it kept telling me I need to declare -vf format). -vf format=YV12 causes color errors.

If it's color errors like swapped U and V channels, you can throw swapuv in the filter chain. That usually does it for me, anyway.

Re: cflags: I don't think explicitly setting architecture makes much difference to x264. The critical bits of x264 are already hand-written assembler, and the configuration script will determine the correct platform to use automatically.

True, but the cflags make sure that everything (or nearly everything?) else is optimized also.[/quote]

How did you isolate the cflags changes from every other factor? (Are you absolutely sure it was compiler optimizations making the difference?)

While you might get a small performance boost out of tweaking compiler flags, overly-aggressive optimization can cause also subtle bugs in program behavior. The default compiler flags are best unless you're sure you know what you're doing.

[trythil]

overly-aggressive optimization can cause also subtle bugs in program behavior

Actually, that probably isn't such a big deal anymore.