onGrass

Animation/VFX Pipeline OS: Linux vs Windows

This is one of the hottest topics I've been asked to comment on lately. Here's my take on it.

A few words before I begin. I'm not a Windows pipeline expert. However I'm an avid Windows user at home (Adobe Lightroom, Photoshop, Premiere, etc), and I have quite a few years of past experience in using it in small to medium size (5..30 Maya seats) studios. Nowadays I'm way out of the loop in terms of modern state of its corporate side.

So, be prepared: I can easily mention a Windows problem that has been resolved 10 or 15 years ago. If you notice any of that in the paragraphs below, please comment and I will keep this post updated.

For the record, in the past, I used to code in DOS and Windows using C/C++, and between 1988 and 2004 DOS/Windows was pretty much the only platform I used for work. I started playing with Linux in 1995, but it was outside any production environment. Then in 2004 I ported my working C++ project from MSVC to gcc under Fedora Core 4 and since 2006 I have never returned to Windows for any kind of software development, except for a few minor Maya plugin ports (gcc to Visual Studio). I don't hate or love either of the OSes. I tend to dislike Apple products, mostly because they are overpriced and overhyped, but that has nothing to do with my today's topic.

Now let's start from the end. In my opinion, building a Windows-based animation pipeline for a medium-to-large animation studio/project (50+ Maya seats) is a bad idea in general. Starting at roughly ~200 seats this idea stops being bad and quickly becomes idiotic. Here's why I think so.

0. Some size estimates. Render farm size and total number of machines on the production floor. In my experience with non-GPU renderers a typical number of farm nodes is around 2..4 times the number of workstations. So if we are talking 100 Maya seats, it's roughly 300..500 hosts in total. Infestation of modern GPU-based renderers like Redshift can cut this number by half or more, but we are not quite there yet when it comes to studios of that size. Also, if your studio has reached the 50-seat mark, you should be prepared for scaling up by at least twice.

1. The OS image. For a large studio you have to have a standard but customizable image, built by your IT department for all your pipeline requirements. You need more than one version, each for a different group of hosts (headless farm nodes, workstation for 3D artists, comp artists, texturing, etc can be based on slightly different hardware, e.g. RAM, GPU, etc). Custom built kernels are not very common these days, but at least they are an option. We also need tools to quickly update OS images on dozens to hundreds hosts, ideally within hours. Same is true about graphic driver updates, core libraries, GNU utilities, etc. This layer is usually handled by the IT department and it has to be clearly separate from the application software layer. I assume all of that can probably be handled on Windows too, well except custom built kernels or libraries. But the ability to patch any piece of the critical OS infrastructure with an Open Source OS is pretty amazing.

2. Application software, including commercial DCCs, plugins, as well as in-house software and scripts have to be maintained by the pipeline (a.k.a. Production Technology or Technical Directors) department. In Linux pipelines, unlike in Windows' ones, application software traditionally gets installed on a single network share, available on all machines. That eliminates the need to handle complicated administration, replication and deployment tools and allows pipeline TDs to work on a different (from the IT) abstraction layer, and be in full control over all aspects of the application software layer. Deployment of new versions, features and bugfixes happens a few times per day. Pipeline people should be able to do that kind of work without any complications coming from systems administration tools. As far as I understand, shared software installation is either impossible or quite complicated in Windows.

3. Don't get me wrong: Windows Registry is a pretty neat idea. For a single computer it's a big step towards setup simpification and unification. What can be better suited for that than a central database of all possible configuration parameters of the machine and its software? Now let's remember what I said in the previous paragraph about application software installed on a network share. How do you marry a local database with the idea of application software being installed somewhere else? Any global change in configuraion would requite updating hundreds of registries. I know it can be automated, but I also know how problematic database merging can be. A good old 50-year-old Unix trick eliminates the need for this. Instead of the database, we just use per-application configuration files, also shared, whose locations are specified by environment variables. User-configurable parameters are also supported, but they can be stored in user home directories. At least that's how Maya does it.

4. Now let's talk about environment variables. I know they also exist in Windows. My Windows desktop at home has like 50 of them, but I almost never change their values. In Linux they are everything. We routinely use (set and get) hundreds of them in Linux pipelines. Among other important things, they are used to keep contexts in shells (terminals). Artists use different shells for different contexts. Contexts include projects, shots or assets they are working on, plus a lot more. Contexts dictate everything, including software flavors and versions to be used on each project, as there is usually more than one, each with its unique requirements. Contexts can even point to different pipeline databases or tracking/reporting systems. Centrally installed software when being started from a shell, in fact is rarely being executed directly, Instead, there is a practice of using shell scripts called launchers or wrappers. Depending on shell contexts they can select software versions, load necessary plugins, renderers, scripts and tools, relevant to the set of current contexts. Only after that they would spawn the instance of the software they "wrap". While it's possible to use a similar approach in Windows, it would lose half of its functionality without centrally installed software and config files.

5. Symbolic links. Just like environment variables - they are everywhere in Linux. While technically they exist in Windows, Microsoft made all attempts to render them useless. First, because you cannot really link anything on a network file system (unless it's an NFS share and you control symlinks using a Linux box), and second, because there is no easy way to tell the difference between a regular file/directory and a symlink. One simple example: software versioning and depolyment. You can switch the entire studio to a different repository, software branch/version, or just a specific script/plugin/executable/library by just flipping a symlink. That alone gives Linux file systems a huge advantage. Hard links are also being used quite often.

6. Since symlinks are almost a requirements, you definitely want your high-performance server to be an NFS one. Then if your Windows workstations and farm machines are connected to it via Samba, we got a new bottleneck right here.

7. Large scale productions are all about automation. Ideally, all tasks that do not require creative decisions should be automated, and repetetive tasks are the first candidates for elimination. Linux was designed for that. Cron jobs, login scripts, shells, Python and other interpreters, command files etc. And it's not only scripting friendly, it's also development and debugging friendly. What I mean here is endless open-source scripting/debugging/profiling tools coming with each Linux distribution, plus the Linux execution model itself (the contents of the /proc directory) lets you peek into the state of working processes to see their command line arguments, open files, environment variables, trace system calls, memory allocation and so on.

8. Money. Windows is not free. Visual Studio is not free either. GNU/Linux is. It might not be the most important topic, but it's also not the least one either. You can only afford this many Visual Studio licenses. They cannot be available on all machines on the floor. In Linux you can compile/debug/profile on any available machine, including farm nodes without even thinking of it.

What are Linux disadvantages? There are a few:

1. You need a decent size team of Linux-savvy IT, PT and R&D people. But you would need a few smart people in those departments even if you have 50 Windows seats anyways, right?
2. Your atrists need to fight their fear of terminals and command lines. It's doable - they are smart people, trust me!
3. There are a few Windows/MacOS apps you cannot (or don't want to) run in Linux, e.g. Adobe Photoshop, Substance Painter, editing suites etc. However those are usually limited to a few departments, like art, story, texturing and editorial, so you cannot avoid having a few Windows/MacOS machines on the floor. It's fine as long as they are being used a bit "outside" of the pipeline, like tools for specific tasks that can read/write some intermediate files that can be accessed and controlled from Linux. Artists in those departments usually have Linux plus Windows/MacOS boxes. 
onGrass

Versioning in Animation Pipelines

Here we are - only six years since the last time I posted in this blog :)

Anyways, today's topic is: "Asset versioning in CG animation pipelines". What I want to explain below is one minor, yet a quite critical part of it. Apparently, not everyone understands it, including some established production houses.

So here we go:

Why treating assets as source code in terms of versioning is plain wrong, at least in Maya-based animation pipelines.

First, let's look into how we version source code. Version Control Systems (VCSs) like git, Subversion, Perforce, CVS, etc would do that for us. No matter how different those systems are, they pretty much follow the same paradigm: there is only one version of every source file you want to use when compiling your project. You never need or want to use two or more for the same build process. It would actually be a big problem if you could compile a source file with a different version of a header file. To compile and deliver the project we make a disk snapshot of the source file tree, with one version of each file being present for the compiler to use. All previous versions are kept in the system (just not in that disk snapshot) too, to preserve the history and for cases when we want to roll back some changes, since they were wrong, or if we want to branch the code from some point in the past.

Let's formulate a few important principles of this paradigm here:
1. The goal of any software project is to compile and deliver a set of binary files that can be used as a whole.
2. To compile (deliver) a project you need to put all its files into a single directory structure, where each file is represented by one version, considered appropriate for the build.
3. Compilation (delivery) of a project olny takes minutes or hours. The whole projects can be built overnight. Compared to development time it's literally nothing.
4. At any moment we should be able to restore the state of the entire project as it was at any given point of time in the past, if we need to, and compile it with the same result, no matter how many times we do that or when we do that.

The same paradigm sounds pretty natural for all projects that consist of source code objects or their alikes. For example, game engines, like Unreal Engine use a similar approach for all elements of the game: source files, levels, assets, textures, etc. You do need only one version of each to compile a game, and you should be able to re-compile it as it was a month ago, if you need to.

The truth is: most animation projects are quite different from what I've described above, and here's why:

1. The goal of an animation project is to deliver a set of animated image sequences, usually split into episodes, sequences, shots. Shots are usually the units of development and delivery.
2. Delivery and approval of individual shots is usually spread in time over months or even years. Multiple shots are being worked on concurrently by different shot artists. A shot, once its final render is approved, is rarely touched again, although that happens too. Rendering of each shot might take a few days. Re-rending the whole project is not feasible at all, since it would take weeks if not months.
3. Different shots might use different versions of assets. As long as they visually look the same, it's not a problem at all. In fact it's even a requirement: the same semantic entity (a character, a prop, a set) has to be represented by different assets in different shots, for example, if the character uses different clothing, or the prop gets dirty or broken.
4. At any moment we should be able to restore the state of any shot as it was at any given point of time in the past, if we need to, and render it with the same result as if it was done back then.

So the only item that stays unchanged between the two paradigms is #4. Besides, there are a few new requirements that do not exist in software development:

5. Some files incorporate live links to other files using file paths (referencing in Maya).
6. Same assets are usually referenced by many other files, e.g. a character asset used in many shots this character is in.

Those might sound similar to the concept of header files, yet they are critically different, due to # 2 and #3.

To resume, you can have multiple versions of the same asset used in multiple scenarios the same time and its completely normal. Even more than that: using just one (correct/latest/approved) version of each asset for all scenarios would cause a lot of troubles: publishing a new version of an asset and making it the only available (correct/latest/approved) would implicitly change the state of all files that refer to it by its path while they are still in development, or (even worse) are done: there will be no way to go back to the previous state of an individual shot.

So if your asset repository directory structure looks like this:
asset_name/
       asset_name.ma                 <- latest/approved
       versions/
            asset_name_001.ma    <- previous versions
            asset_name_002.ma
            asset_name_003.ma


then you are in trouble. Instead you should always use:

asset_name/
       version_001/

            asset_name.ma
  version_002/
            asset_name.ma
  version_003/
            asset_name.ma

Your next question is probably: "How would shot artists using all those versions know when a new version of an asset is published? What if it fixes some critical bugs?". My answer is: "we should give artists proper tools to make an informed decision and update to the version they need, but only if they want to". It could be a dialog that would warn the artists about availability of newer versions of assets, along with descriptive commit notes. Yes, we are losing the ability to auto-update multiple scenes by publishing a new fix, but we are also reducing a lot of surprises. Remember our #4? When a shot artist opens a scene that was working a month ago and finds all characters and props slightly changed, because they were all "fixed", it's not a good surprise, trust me. They should be able to explicitly choose between the "old" and the "new" states instead, for every asset in the scene.
onGrass

Color Science

Okay guys, did you miss me? So did I :)

I've been busy doing stuff. Among the stuff I've been busy with was something called the Color Science. Today my rant is totally about that, by far the most confusing and intimidating part of computer graphics.

After days of banging my head over those wonderful 3x3 matrices, DCI P3, XYZ, X'Y'Z', xyz (they are all different btw) color spaces, white points, primaries and you name what, I was almost ready to kill myself. I have to warn the world: there is a huge conspiracy by color scientists all over the place. They seem to be nice and sharing, writing detailed books and creating helpful online resources like http://www.brucelindbloom.com/index.html?Eqn_RGB_XYZ_Matrix.html. This is all bullshit. In fact, they are evil creatures secretly invading our industry from Mars, Aldebaran or even further away. I also think they all share the same Aldebaranic DNA strands. At least I cannot find any other reason why all books, web sites and whatnot use column-major matrices without even mentioning that. All those days when I have tried inverting and multiplying these matrices using Imath::Matrix44 (which obviously did not result in proper results without transposing them first), they must have been high-fiving each others' tentacles in joy.

Rise people of CG! Don't let them kill you! Always transpose your color matrices!
onGrass

Autodesk again

This time I will rant not about their software but about their customer support. It's not been that great all those years, but it looks like it's only getting worse. A week ago a co-worker of mine submitted a support request on their portal. He attached a short python script to illustrate his problem. After a few days a support guy came up with a response. He said he could not open the .py file. You got it right. A support person of a billion dollar vendor cannot open a file attached to a support ticket created in their system. How's that? I immediately tried it and downloaded the script without any problems.

Seriously, I don't know how much the company I work for pays Autodesk a year, but I believe it's a 5-digit number. I really think they should've payed us instead for this kind of support.
onGrass

Some free stuff to share

Hi everyone,

Today I'm not going to complain about mental ray. Instead, I will share some C++ code I wrote in my free time as a part of a bigger project, not announced yet. Not even decided whether it will ever be. So here is the tarball:

http://www.alexsegal.net/ftp/imgutils.tar

It contains source code of two utilities and their Makefiles for Linux.

1. linearize - performs color transformation into linear space for textures. It can read an sRGB or a Rec.709 texture and save it as a linear image.
2. exr4nuke - post-processes rendered Open EXR images to optimize them for comping in Nuke. It does two things: re-compresses the image as zip individual scanline and crops dataWindow according to the bounding box of all non-empty pixels.

To compile them, you have to have the following libraries installed on your Linux system:
- OpenEXR
- OpenImageIO
- boost

For Windows you'll have to re-create the Makefiles (or Visual Studio projects) from scratch.

Enjoy!
onGrass

sRGB, Rec.709 to linear and back

I did some color conversion these days. Just to keep it somewhere I will post the formulas here. It was simple with sRGB - everything is on Wikipedia, but for some reason Rec.709 standard does no have the reverse formula in the doc describing the standard (http://www.itu.int/rec/R-REC-BT.709-5-200204-I/en) so I had to derive it myself.

Conversion to linear:

if f > thresh:
    f = pow((f+a) / (1+a), c)
else:
    f = f / b

From Rec.709: use the following values:
thresh = 0.0801
a = 0.099
b = 4.5
c = 2.2222

From sRGB: use the following values:
thresh = 0.04045
a = 0.055
b = 12.92
c = 2.4
       
Conversion from linear:

if f > thresh:
    f = (1+a) * pow(f, 1.0/c) - a
else:
    f = f * b

To Rec.709:
thresh = 0.018
a = 0.099
b = 4.5
c = 2.2222

To sRGB:
thresh = 0.0031308
a = 0.055
b = 12.92
c = 2.4
onGrass

Announcement

Just wanted to let everyone know: I had to reject a livejournal user who sent me a request to join the community. I had two reasons to think it was a bot:

1. His/her live journal is empty
2. The user name is unreadable (at least for me).

So if you are human but your account matches two of these criteria, and you want to join the community, please send me a private livejournal message or leave a comment to this entry before attempting to join.
onGrass

Subsurface in reflections

There is officially no known way to render character reflections in mirrors. More precisely, there is no method to render subsurface scattering in reflected/refracted rays, well, unless you use your own sss shader. The misss* subsurface shader that comes with mental ray will only substitute missing SSS with lambert, but it means the reflections will not look like skin.
It would be funny, if we didn't have a feature-length project ahead with lots (I mean LOTS) of character looking into mirrors.

UPD: The issue seems to be more complicated. It looks like stock skin shaders in maya/xsi/max do render sss in reflections. I will need to do more investigation on this.
onGrass

Stereo

A nice new feature in mental ray version 3.8: the ability to render two images for left and right eyes during the same render session. As usual, mental images were good at little tiny things that make the entire thing useless. New "stereo" flag in camera declaration is missing a way to specify the convergence (or zero parallax plane) distance. For some strange reason the cameras are always converged at the focal length point.

It looks like they at last managed to confuse themselves by using incorrect terminology. The documentation refers to cameras converged at a "focal distance", but there is no such thing neither in real world nor digital photography. There are two things instead: "focal length" which is a lens characteristic, directly affecting its angle of view, and "focus distance" which defines the distance of objects the lens will project to film keeping them in perfect focus.

I would still mind having both the focus distance and the convergence plane to be controlled by the same parameter, but that at least would make some sense. But the focal length has nothing to do with eye convergence, that's why the entire thing is just useless.