Inside Krakatoa and MAXScript

On The Usefulness Of The Web

2011-12-20T10:30:00.000-08:00

Sorry for the long silence, but I had to redirect most of my Krakatoa-related musings to the official Thinkblog on our company website. It only makes sense since that place gets a lot more traffic.

As you probably remember from my previous post, I am now employed as Product Specialist by Thinkbox Software. As result, I can now enjoy doing full-time what I used to do half of my time at Frantic Films and Prime Focus - co-develop Krakatoa and other products like Frost and XMesh, write documentation and tutorials, record videos, demo at expos and even travel the world! All from my new home in Vancouver!

In the last couple of months, I had to use the Internet and the Powers Of Google to teach myself a lot of things like MEL scripting, Python and basic Maya and Houdini skills. I must say it went pretty well and I solved most problems I had to deal with. I didn't even have to post questions, most of the FAQs were already posted and answered by others long ago. Oh joy!

But here comes the funny bit. The other day I had to do some development which involved the recreation of a certain 3ds Max modifier as a script. In 3ds Max. I wasn't exactly sure where to start and I felt lazy. Checking the 3ds Max SDK felt like too much work. Having the routine of checking the Web for my answers about the other applications, I decided to google it to see how others might have solved this.

And then I found it. A thread on CGTalk with the exact question I had. And several answers. And code. And more code, going deeper and deeper. I read the answers - they were clear, the examples worked. Just what I needed.

So what's so funny about this, you might ask? Well, guess who wrote that code...

I had absolutely no recollection of solving the problem before, but reading that thread clearly showed I must be getting old. It was my code, my explanations, and they read like something somebody else had written. I was in awe. This was the first time I felt like Young Me knew more than Old Me knows...

The Internet appears to be a good extension of our brains. In addition to Google and Wikipedia that provide access to other people's knowledge, it turns out it is a good way to store our own knowledge just in case we start forgetting stuff... ;)

End User Excitement 2011

2011-07-05T02:30:00.000-07:00

If you have missed the news, I have a new job!

As of June 20th, after 7 years, 6 months and one week as Technical Director, I am officially employed by Thinkbox Software.

"New" might not be exactly correct, because I have been spending over 50% of my time at Frantic Films/Prime Focus co-developing Krakatoa & Co. in the last few years. For all users who felt uneasy by the fact that my name was missing from the Thinkbox' About page, breathe freely now! As Product Specialist located in Vancouver, I will be involved in the development of documentation, tutorials, customer demos, user interface and tools, as well as participate in feature planning.

I spent the last week in Holland where I participated with two one-hour talks in the End User Event 2011. I was lucky to get the opening slot on the first day, ensuring the 90-seats hall #1 was full for my "Krakatoa and Frost" demo. It was meant as a quick introduction to the power of the two products used together, including MagmaFlow control over Radius, Color and Orientation channels; Procedural particle placement using ray intersection and nearest point queries; Fast collision detection using Krakatoa operators in Particle Flow, increasing particle count using Frost and PRT Volume; Dynamic filling of closed volumes using PFlow and Krakatoa Collision; Creation of rivets using Krakatoa, Frost and MAXScript to sample control maps on complex structures; Cloud modeling and rendering using Sphere Gizmos, Frost, PRT Volume and MagmaFlow, including a quick preview of some Krakatoa 2.0 goodness. I intend to post some new tutorials based on these demos on the Frost Tutorials page later this month. (EDIT: First tutorial is now available here)

Image by Master Zap

I heard some people mentioning that I was talking too fast, but this was necessary in order to fit 6 practical examples created pretty much from scratch in front of live audience within 60 minutes. Each time I finished a topic I looked at my watch and was amazed to see exactly the minutes I hoped to see - 10, 20, 30, 40, 50... done! I had executed these demos dozens of times at home, but never timed the complete presentation to make sure it fits in the time slot. Turned out my internal clock was pretty precise, nearly as precise as the Dutch railways that also impressed me a lot ;)

My second talk was an overview of the software development at Frantic Films and Prime Focus leading to the intellectual property now owned by Thinkbox Software, in the context of the visual effects projects that made the tools necessary. In this light, I tried to explain how applications like Deadline, Krakatoa, Frost and Flood came to be.

Despite my jet lag I managed to listen to a lot of presentations by others. My general impression was that all studios, big and small, have to fight the exactly same problems when it comes to pipeline and artist friendly tools, and there is a lot of wheel reinventing going on. I loved the Planet 51 pipeline presentation which reminded me so much of the Prime Focus pipeline it was spooky! Great minds (read: Laszlo Sebo and Gonzalo Rueda) think alike! I was also positively shocked to see Neil Hazzard spend time demoing the script I wrote for Nitrous/Quicksilver NPR mode customization.

Both my talks gave me a short opportunity to show some internal W.I.P. development for Krakatoa 2.0. My impression was that these upcoming features will make a lot of people happy, but I won`t discuss them here yet in hopes to make the upcoming Siggraph 2011 in Vancouver even more exciting. You know me, I like teasing as much as my new old boss Chris Bond does ;)

Stay tuned, there's just a month left until Siggraph!

Full Speed Ahead

2011-04-15T17:37:00.000-07:00

I realize I haven't posted in a while. Not even during the pre-release frenzy surrounding Frost, Krakatoa's BFF. Having spent the last 4 days in Las Vegas showing off Deadline, Krakatoa and Frost to anyone who came to see them, I thought it was time to write something about the future.

I am sure those of you who remember my "How Fast Is Fast" blog about Krakatoa 1.6 wouldn't expect yet another similar jump in performance from the next version. Well, you would be very very wrong, and I have the numbers to prove it. In fact, the upcoming version of Krakatoa might provide a bigger speed-up over 1.6 than 1.6 had over 1.5!

Creating Partitions and loading PRT file sequences from disk are among the most typical workflows in the Krakatoa pipeline, and the time to load the particles has traditionally been about half the time of the rendering process. Several factors affect the speed of loading - the speed of the hard drive or network connection serving the files, the speed of the CPU reading the data and decompressing the stream into memory, and the number of additional operations performed on the particles while loading, like MagmaFlow and Material evaluation, deformations and culling. In the past, the latter operations were gradually updated to support multi-threading, but the initial loading has remained limited to two threads, and typically saturated only one core.

Not anymore. I had the pleasure to benchmark an early alpha build of what might become Krakatoa 2.0 on a variety of machines with a multitude of storage solutions. To test the pure loading speed, I created a Box with dimensions 100x100x100, converted to a PRT Volume and partitioned it to disk as 100 partitions, each one with 1 million particles. Note that this introduces some overhead to the loading process - loading 10 partitions with 10 million each, or one partition with all 100 million would be somewhat faster, but I wanted to produce a more realistic case which is nearly the worst case of partitioning since Krakatoa currently limits the max. number of partitions to 100. Also, it would be able to create enough threads for any number of cores. I kept the default channels layout - Position, Velocity, Normals, Color, Density and ID - to simulate a typical case even though I did not need some of these like Velocity or ID for the actual rendering. I then created a single PRT Loader from all 100 partitions, created a default Spotlight and rendered.

My first test used the slightly outdated hardware tasked to perform the Thinkbox demos at the NAB show in Las Vegas - it was a dual Intel Core Duo, in other words four physical cores, no Hyperthreading. The interesting thing about this machine though was that it contained one 7200 RPM harddrive, two striped 10000 RPM drives, one SSD drive and a Fusion-io card, all connected to the same hardware. This gave me the ability to find out how the storage medium affects the new software.

For comparison, I used the current 1.6.1 build. Loading the 100 million particles with it took 57.3 seconds and the total rendering time was 2 minutes 46 seconds. It did not matter what drive I loaded the particles from because the speed was fully determined by the performance of the one core reading the ZIP stream from the PRT file.
Loading the particles from the 7200 RPM drive using the new build cut the loading time down to 38.2 seconds and the total rendering time to 2 minutes 29 seconds. This is not a very impressive speed-up, but it reached the physical limitations of the hard drive, while loading the CPUs as much as the I/O bottleneck allowed. Having 4 cores, 4 threads were created to read 4 PRT streams at once, but the drive could not keep up with the demand.
Loading the particles from the two 10000 RPM drives brought the 4 CPUs to about 80% saturation before the bandwidth of the hard disks became the bottleneck again. The time to load the 100MP went down to impressive 16.1 seconds, but it was obvious that there was more to be expected from the solid state drives. And indeed, running the exactly same tests from the SSD drive gave me 11.2 seconds for loading and 1 minute 49.9 seconds total render time, while saturating all 4 cores completely! Trying the same with the even faster Fusion-io card produced the same loading and rendering time, clearly proving I had reached the CPU bottleneck.

Thankfully, Fusion-io was well represented at NAB and I got the chance to run the benchmark on an 8 core machine to test the waters. My gut feeling told me I should expect about half the loading time with twice as many cores so I wasn't exactly surprised when the faster system loaded the 100 million particles in 6 seconds and finished rendering in only 52 seconds! (Un)fortunately, all 8 cores were once again at 100%, making the result CPU-bound instead of I/O bound, leaving me wanting to test on a 16 or 32 core machine to see what a Funsion-io card can really do for Krakatoa. My gut feeling tells me again we could expect loading times of 3 seconds or less for 100 million particles on such a system, but until I actually get to try one, I can live with the results pretty well. Supposedly, a good SSD drive could keep up with an 8 core system to produce the 6 seconds loading time, too, so you don't have to spend the equivalent of a new car to get that performance...

There are several other areas that have seen some speed up in the new version - on my home i7 quadcore machine, the sorting for both lighting and drawing of 100 MP went down from 12 seconds to about 8, and the drawing was reduced from 5 to 4 seconds. That machine is not very good for testing the loading improvements due to a slow hard drive though, so if you could imagine a modern computer with a lot of cores and fast SSD drives, Krakatoa will literally fly on it later this year!

Obviously, this is just the tip of the iceberg when it comes to what will be new in the next version of Krakatoa. Wait for Siggraph and be very, very excited - I know I am...

Servicing the Pack

2010-09-25T00:29:00.000-07:00

The Krakatoa 1.6.0.43178 build released on August 31st is no more. It has been replaced by a 1.6.0.43376 build released on September 24th. If you are already running 1.6.0, you should do yourself a favor and update now. Go, I will be waiting!

This is the first time we replace a build without incrementing the version number. We call it a Service Pack 1, but you can read it as "The build we should have released to start with" :) It fixes several bugs related to rendering FumeFX, adds the Icon Size spinners to all PRT objects that were missing one in the original build, and because there have been more than three weeks since the release, it also adds some new features here and there - Yours Truly cannot sit around without touching up the user interface...

Here is the list of changes:

Bug Fixes

PRT FumeFX with FumeFX 2.0 was causing a crash when switching to Post Cache mode. This has been fixed.
PRT FumeFX was showing only a portion of the simulation due to incorrect handling of Adaptive bounds. This has been fixed.
Thinking Particles was causing particle evaluation to hang at render time if there were two or more TP objects in the scene. Fixed.
Deformation Modifiers were inverting the Velocity Channel in PRT Loaders. This has been fixed.
Fixed the MAXScript call for saving all rollouts to presets which was causing a System Exception.
Fixed a crash when trying to load a Save Channels Preset.
Turning off the Viewp.Spacing in PRT Volume causes the render-time Spacing value to be used for the viewport, too, but changing the spinner wasn't updating the viewport cache and required a manual update. This has been fixed.

Feature Improvements

The Thinking Particles Partitioning code was tweaked to allow an Integer Helper containing "*RandomSeed*" in its name to be incremented just like a regular "RandomSeed" property. This can be used to adjust the Random Seed of MatterWaves by connecting an Integer Helper node to the RandomSeed property which is otherwise not exposed to MAXScript.

MagmaFlow

Undo/Redo wasn't restoring the Input Nodes' value correctly. This has been fixed. Please note that if the track is keyframed, performing Undo/Redo can affect the complete curve as the animation state at the time of Undo creation is not stored and there is no way to store/restore the complete controller. This fix also improves the Macro Recorder playback.
The ">Interactive Mode (SLOW!)" option in the Krakatoa Channels Modifier wasn't handling Geometry objects used in Surface Operators. This has been fixed.
Added conversion buttons to the SurfDataValue Operator to quickly turn the output value from Integer to Float, from Float to Vector, extract X,Y and Z components of Vector or transform Vector From World to Object space or into View Space.
Added an explicit INSERT/BRANCH mode displayed in the title bar of the MagmaFlow Editor and controlled by the Insert key in the Numeric Keypad. When set to Branch and a single node is selected that has an output socket connection, creating a new operator will not insert into the existing connection but branch into a new output connection. Once the node is branched, the mode will revert to INSERT automatically, since it makes more sense most of the time. In previous versions, this was done with the SHIFT key, but it could collide with some keyboard shortcuts that use the SHIFT key as a modifier key.
Fixed the mouse right-click hit testing when the MagmaFlow Editor is docked as Extended Viewport.
Added support for Schematic Flow as Extended Viewport, including correct right-click handling.

User Interface

PRT Volume, PRT FumeFX and PRT Source now expose an Icon Size spinner to the UI. The MacroScripts now use the XY size to define the icon size of PRT Volume and PRT FumeFX.
Added support for saving and loading the Channels To Save list.
Added all rollouts to the right-click menu of the Float/Dock icons with a check mark for visibility. Selecting will toggle visibility, holding SHIFT will navigate to the rollout.
Added right-click handling to all "Back To Main Controls..." buttons in addition to the left-clicking.
Added the keyboard shortcut names to the right-click menu of the Float/Dock icons.
Changed the navigation buttons throughout the UI to unhide hidden rollouts before navigating to them.
Reworked all MacroScripts to disable themselves and avoid any crashes if the Krakatoa path is removed from the plugin.ini.
The Rollout Preset files were not restoring the visibility of the "Shader Parameters" rollout correctly - even if saved as hidden, it would be displayed because the update of "Main Controls" rollout's Shading Mode list was forcing it to appear. This has been fixed by processing the "Main Controls" before "Shader Parameters".
Optimized the "Load From History" dialog to use the History Cache instead of scanning the files from disk. With 8281 History files on disk, the old code was taking 47 seconds to update the first time the dialog was opened, and 10 seconds in consecutive attempts. With the new code, the time to open the dialog went down to 0.078 seconds!

Just the first 4 points of the Bug Fixes list should be a good reason to update if you are using FumeFX 2.0, Thinking Particles or modifiers on PRT Loaders. In fact, I had to postpone a video tutorial recording session I had planned because it totally depends on a modifier on a PRT Loader and would behave quite strangely in the original 1.6.0 release... Stay tuned! And if you haven't watched the other YouTube videos, go here, here, here and here.

If you find more bugs hiding in the newest build, please let us know. We will chase them and hit them with a big hammer!

I Am A Cheater, Too!

2010-09-24T01:00:00.000-07:00

Thanks to this CG Society article, I discovered the following "Making Of Tron" video on YouTube:

Part 1 Part 2 Part 3 Part 4 Part 5 Part 6 Part 7 Part 8 Part 9

The video is beyond awesome, but the most striking part of it, even more than the behind the scenes footage and image material, was THIS SENTENCE by Steven Lisberger:

"The Motion Picture Academy refused to nominate Tron for special effects because they said we cheated when we used computers..."

While I point at Star Wars as the main influence for me to end up in the visual effects industry, Tron has a very special place in my heart, too. Unlike Star Wars, Tron was not shown in communist Bulgaria's movie theaters back in the early 80s. But the magazine for international journalism "Parallels" ("Paraleli" in Bulgarian) had a large article with images about the making of the "first computer generated film" which I used for a school project about "an art form of my choice" (film is considered the Seventh Art Form, so I went with Science Fiction Movies as my theme).

For 6 years, I could only dream about seeing the actual film. My dream came true thanks to "Perestroika" on February 14th 1988, a Sunday. It was 3 days before my birthday and it happened to be a weekend. I was in my first year of army service and my girlfriend came to visit me so I got a two days leave. We went to a so-called "Cinema Video Club" - these were coffee shops with video projector that were showing private (read: illegally copied) VHS records of western movies that could not be seen or obtained through any other channels. Copyright law did not extend beyond the Iron Curtain, and thankfully the idea that western movies were "capitalist propaganda designed to brainwash the youth" was not so popular anymore, so we could finally see films we had missed originally. And they were showing Tron that day!

Of course, I was completely blown away by what I saw then. A year later, during my second year of military service, I was in a computer center and had the opportunity to play with simple computer graphics. This led to getting a PC home with me for another two years after I finished service as I continued working as consultant to one department of the military academy. Then, in the first year as student of architecture, I presented a design project that was made and printed on a computer. Guess what the reaction of my professors was? "This is not fair to your peers, you cheated by using a computer".

As I have mentioned in previous blogs, I never intended to turn computer graphics into profession, mainly because it was impossible in the current political climate. But for some people who were living in a different environment, Tron turned out to be the actual inspiration for working in the visual effects industry.

There still is an original Tron coin-op game in one of our offices. This is because the company previously known as Frantic Films was started by Chris Bond who was one of those people who were inspired by Tron and actually went into the visual effects industry because of that inspiration. With Tron Legacy less than three months away, it is both a wonderful and curious feeling to look back at where things started and realize how far we, the industry and the world have come...

Shipping Time

2010-09-01T00:30:00.000-07:00

Krakatoa v1.6.0 is here! Where? There!

Surprisingly, we actually hit the shipping deadline this time. Not that it would have mattered one way or another since all users with support contract and even anyone who asked for it had access to the Beta builds for months. In the past, we always went by the saying "It will ship when it is ready". Being a niche product developed by a small team, we always had that luxury large publicly traded companies do not have - we'd rather delay a release than ship with known bugs. That's not to say there are no unknown bugs in there - if you find any, please tell us!

As mentioned in previous blogs, Krakatoa is now heavily multi-threaded and thus a whole lot faster. It integrates better with FumeFX incl. v2.0, supports the shipping version of Thinking Particles 4, provides new MagmaFlow operators, produces some Render Elements, generates much better shadows and fixes a lot of shortcoming of version 1.5.1. The UI is faster and more streamlined, while the number of additional tools has gone up. Look out for the Krakatoa menu in the Main Menu bar of 3ds Max - it exposes most tools that required customized toolbar icons in the past.

The new version even comes with a new license dialog written specifically to replace the totally unusable FlexLM one that has plagued humanity for years. Yes, the number one support problem of Krakatoa was caused by a piece of code that wasn't written by us. We really hope entering the license server or file will be much easier now...

On the curious side, possibly the smallest change with the largest impact on usability (or at least my sanity as a user) was the addition of a separator between the Copy and Clear menu items of the Log Window's Edit menu. You have no idea how many times I have cleared the window instead of copying the selection in previous versions!

We committed some "crimes", too. A day before release, a user asked for better PRT object icons to easier distinguish between PRT Loaders, PRT Volumes and PRT FumeFX objects. The request came literally in the last minute and no developers in their right mind would start changing that amount of code between Release Candidate 2 and Release Build. Well, fearless team that we are, we designed new icons, wrote a script to export their meshes into C++ compatible vertex and face lists and submitted the new code AN HOUR before the final build was produced! The impact risk was low, the positive effect on usability was significant. We had discussed that change before, but there were so many more important features and fixes, we never found the time. When everything else was done, it felt like we could afford to sneak that in and make some people happy...

Now that we have the final build out there, I really hope to be able to produce some new videos - my first and last Krakatoa-related videos were recorded using v1.0.0 back in 2007!

You can read more about the features of Krakatoa v1.6.0 here and here.

Changes and Dependencies

2010-07-20T23:13:00.000-07:00

A couple of days ago somebody sent me a link to an automatic "Siggraph Paper Title Generator" which produced quite convincing scientific-sounding output. The title of this post could easily be mistaken for something programming-related, but it is about something else completely.

I haven't posted since April because the last couple of months were quite crazy. In short, my family just finished relocating through half Canada - from Winnipeg, Manitoba (the city closest to the geometric center of North America) to Vancouver, British Columbia (at the very edge of the continent). This is a quite positive change - no more winter temperatures lower than on the South Pole, no more mosquitos in the summer (considered the "national bird" in Manitoba ;o)) And last but not least, finally some hills, mountains and curves on the roads to make the driver in me happy.

The move itself went ok without any incidents, all our stuff arrived a week later than expected but absolutely preserved (especially the PC and the TV - sigh of relief), and the first thing after ensuring electric power was still on was calling the cable/internet provider to order the Most Important Thing after water and bread. This was on June 28th and I was told that the earliest possible appointment date would be July 11th. TWO WEEKS without Web, Email and Skype? Oh my...

The very next day I replaced my trusty dumb phone with a smart one just to get some form of online access. This gave us at least some email, web services and the ability to chat via text messaging with our parents in Europe, plus the added bonus of GPS turn-by-turn navigation. While this was one of the best purchases considering the situation, it wasn't enough and I finally realized how dependent (read: Addicted!) we have become to the internet.

Two days after arrival I was chatting with a colleague and discovered that another company otherwise known for its wireless services and cute animal commercials had started to provide optical cable in our area. In fact, they even had a brochure sent to the "current occupant" of our apartment, but my selective reading brain totally skipped over the first two letters of the title (which said "TV"!) that also promised phone and internet. So I immediately called the company and asked if we can even get their services, since I knew that some large buildings downtown had preferred partnership with the former cable TV monopolist I already had an appointment with. The answer I received left me nearly speechless. "Looking at our data here, it appears that your building has been pre-wired for out optical cable services, and TV and internet are already pre-paid by the building management for a year!". Oh, and they could come 5 days earlier than the competition (YAY!)

Too good to be true?

The lady on the phone promised to call me half an hour later to confirm my credit check and the appointment for the 6th of July. She never did, so the next day I called again. The person at the other end of the line went through the whole identification process and confirmed the 6th of July appointment. So on the 6th of July I took an hour off from work and went home to be around during the installation... except that nobody showed up. Another phone call later I learned the amazing truth: Computers Hate Me!

I was told that a software bug in the internal system of the company is preventing anyone in the call center from scheduling that appointment (three people so far had attempted to enter it and all got the error which I was told "nobody had seen before"). Being a TD and fighting the good fight with bugs every day, I don't really mind bugs, but I do mind when people don't tell me about them (Funnily enough, the name of the company implies they would "tell us" when something like that happens ;o)). So I was told that the top-dogs of the software team were looking into the problem and would call me as soon as it was fixed. I was even encouraged to call again the next day to check. Of course I did and had my fifth conversation with a fifth person about the whole issue. "Please believe us that we are working on fixing the bug". In all this time, I never received a call back from anyone. Probably because nobody cared personally, or probably because my bill was already paid for a year anyway, so an important stimulus was missing... ;o)

While getting home from work via a slightly different route that day, I realized that the head quarters of the company I was dealing with was literally the NEXT BUILDING to ours! So the next morning I went into the lobby and asked to talk to someone responsible for customer relations. Instead, I got connected with the technical team responsible for that "bug" and was told the bug was fixed and they could finally schedule an appointment. (Until this day I don't know whether this was just good timing or there was never a real bug). Of course, this was on July 9th so the earliest possible opening was for July 16th - over two weeks after my first call...

Being as addicted as I turned out to be, I could not wait 5 more days without internet. So we scheduled the 16th but also left the initial appointment with the other company on the 11th and on that glorious Sunday finally got some cable to scratch that online itch. Four days later we switched to the optical cable with HD, free PVR, 15 Mbps wireless access from all devices and finally felt home... This blog is just a reminder how much our lives have changed in the last decade and how dependent we have become... Now let's hope SkyNet won't take over soon ;o)

Back on topic (this is a Krakatoa/MAXScript blog after all) - the next shocker is that the year of the 20th birthday of 3D Studio will be the first in 11 years with me missing Siggraph. I wish everyone going a great time and while I am sad we won't be able to show you first hand what we have done to Krakatoa 1.6, I promise we will be very active online once it is released. I am spending a lot of time polishing both the documentation and the UI, and a lot of features are still being tweaked for maximum flexibility.

I hope the next blog will be a lot more interesting.

How Fast is Fast?

2010-04-18T14:13:00.000-07:00

In pretty much every review or comment about Krakatoa, I see something along the lines of "Krakatoa renders really fast". Don't get me wrong, I like reading that, but let's be fair - speed is relative (ask Albert E.) and people don't say this because Krakatoa is actually fast, but because most mesh-based particle rendering solutions for 3ds Max are slower in comparison.

All this good press makes my job harder as I try to push for speed improvements in the development of future Krakatoa versions (you might not know this, but my middle name is "Speeeed!"). We are in the business of selling software to users, and if a single copy of Krakatoa can get the job on time, why buy more? There was even an anecdotal case where an important customer of Krakatoa wanted to buy additional licenses, but before the transaction could be finalized, they told us they got their project finished with the few seats they already had because "Krakatoa renders so fast". Obviously, while a fast product is good for the image of the product, it could be quite detrimental to sales - thankfully, the client still bought those additional licenses so no harm done, but it made us thinking...

What would a company do if its existence depended solely on software sales? Things like limiting the Evaluation version to do a lot less (most of the particle management features of Krakatoa like partitioning, converting particle files between formats, deforming and modifying, importing back into Particle Flow etc. are basically free) or making Krakatoa deliberately slower come to mind. Thankfully, the main reason for Krakatoa's existence is the internal need of our visual effects production for a fast particle renderer. And watching Krakatoa in real-world production proves my point - Krakatoa can be damn slow and we have to do something about it!

So how fast is fast? Some people might answer "real time on the GPU would be good enough". Unfortunately, this is not the right answer. There have been some examples of real time CUDA-accelerated particle rendering in nVidia demos, but this is not where we are going and the reason is the data amounts we deal with. The main objective of Krakatoa is the "fast rendering of vast amounts of particles". In this objective, the second part has higher importance than the first one. With the arrival of 64 bit computing and the increases of installed RAM, our workstations can typically fit around 700 MP (Million Particles, or MegaPoints) in 16 GB of RAM, and we often go there or near. A typical graphics card with 1GB of RAM can handle less than 1/16th of that amount, so we don't care about that approach just yet, although it would be great for fast tests with a fraction of the particle count.

Another thing to keep in mind is that various areas of Krakatoa have different impact on speed - some are as fast as we could make them, some could be made faster, some can become bottlenecks depending on the settings. For example, the loading of particles which also includes the evaluation of materials, maps, deformation modifiers, MagmaFlows and particle culling was sped up in v1.5.0 via multi-threading and is now several times faster than in 1.1.x. The sorting portion has been multi-threaded for years and is probably as fast as it could be. The drawing in the Lighting and the Final Passes of Particle Rendering has remained single-threaded since the first version of Krakatoa and has been measured at around 2 MP/second. Given the increasing number of cores in today's machines, this is an area that could improve a lot! The evaluation and processing of Matte Objects was also sped up in 1.5.0 by simply switching from a raytracer to a rasterizer, but it could also benefit from multi-threading the depth map generation.

Adding a new light adds another sorting pass (already fast) and another drawing pass (not as fast as it could be). The moment you check the Motion Blur option, you ask Krakatoa to draw the Final Pass several times. If the number of samples is 8, Krakatoa has to do 8 times the drawing work and typically scales linearly (it actually takes 8 times longer to draw the particles if you ignore the loading time or render with PCache and LCache enabled). But if you have 8 cores in your workstation and Krakatoa would use them all to speed up the drawing, it would mean that 8 passes motion blur would "cost" as much as one pass right now. Wouldn't that be great?

You betcha! So that's what the next version of Krakatoa will do (and more). And the more particles you throw at it, the better it will scale. On my workstation, the pure drawing speed of 100 million particles comes close to 17.5 MP/second! The generation of a Matte depth map from 100 million polygons, something that is also performed before each motion blur pass, went down from 32 seconds to 4 (8x4=32, you know?). And we are not even half-done yet. Add to that the loading speed up with PRT Volumes already reported in a previous blog and you will see how this new build is shaping up as the fastest Krakatoa you have ever seen. In the same blog I mentioned it might ship as v1.6.0 - that suspicion turned out true and we already started updating the documentation for this upcoming 1.6.0 build which should be expected sometime before Siggraph.

I am quite excited about these improvements which we haven't even passed to the Beta testers yet as they happened in the last 10 days. All this wouldn't have been possible without our relentless production team which not only pushes Krakatoa to places it has never been to yet, but also constantly bitches about how slow it is! Love you guys! :)

So don't panic, we won't make Krakatoa slower just to sell more network render licenses and Deadline seats to run them, or disable more features so you cannot play with it for free. We understand that a faster Krakatoa is easier to love, and a Krakatoa you love is one you would if not buy yourself, then at least suggest to your employer (the guy with the wallet ;))...

Stay tuned!

Back To Basics - Krakatoa Best Practices For Fast Iterations.

2010-03-21T10:56:00.000-07:00

This morning I received a PM from a CGTalk member with approximately the following content:

...These days I am studying Krakatoa by myself, it is interesting. Now I can generate particles from FumeFX data. To render Particle Flow, I use 800000 particles, but it is very slow to update PF, takes almost 10 minutes or even more, is it normal? Or did I miss some steps.. My PC is a Dell T5500 workstation, 64bit OS. When PF and Krakatoa work together, how can it be made faster? I can't image using millions of particles...my pc would update PF all day long, haha!

For the more advanced users of Krakatoa out there, the following answers might look like deja-vu from the Online Help's Introductory Tutorials and the Studio Daily Introductory Videos posted back in 2007. But I feel it is something that cannot be repeated often enough, and given that nobody ever reads any Help files (I know I don't), a blog post on a site hosted by Google might be more discoverable...

Here is my answer, slightly expanded.

Well, we cannot speed up PFlow, so we generally try to work around these issues. Here are some things one could do to speed up work.

1. Save the particles to PRT sequence once, then render as often as you want.
PFlow is slow at updating to a specific frame because it is history-dependent and has to calculate all preceding frames. So if you have 200 frames of animation and want to render frame 190, you have to wait for all 190 frames to update. But if you have already updated frame 190, getting frame 191 should be relatively fast, and this is true for the saving process too. So you set Krakatoa to save PRTs to disk and let it run (overnight? ). Once it is done, you have traded off time for disk space. Now you have Gigabytes of particle data on disk, but you can render each frame in about a second without any preroll! The Krakatoa PRT Loader also lets you tweak the timing/speed of the playback, cull particles, deform the particles if they don't match your vision, bend the particles to follow a spline path, deform by a skinned mesh to create particles moved by a character animation, create and modify particle data channels using MagmaFlow, duplicate the particles by cloning the PRT Loader multiple times and so on. It is a whole new particle data workflow you have there...

2. Partitioning.
You can expand the above approach with some partitioning (saving out several smaller PRT sequences with varying random seeds to produce a denser final result). So you could for example reduce the particle count to something you can live with time-wise, for example 50K, 100K or 500K particles, and partition only one PRT sequence to check out if you like the motion. You could also use the ability of the Iterative Mode of Krakatoa to render a frame at a fraction of the resolution while preserving the density to get a representative frame from few particles that gives an idea what the same frame with 10 times more particles could look like.

If it looks good after it has finished saving, you can let the computer run multiple additional partitions with the same settings to produce more and more particles using the same FumeFX sim (and/or PFlow setup). This way you don't have to wait for the whole quantity to be processed at once and make more iterations until you like the general motion, then let a copy of Max running in the background save 10 or 20 or 50 sequences of the same setup with different random seeds to combine into one dense sequence with millions of particles within the PRT Loader.

3. Use all your CPUs/Cores.
PFlow is single-threaded, but your machine might have more CPUs sitting idle. You can open several copies of 3ds Max + Krakatoa on your workstation (thankfully, both the 3ds Max and the Krakatoa licenses support that), load the same scene in each one of them and let each one partition a sub-range of the total range of PRT sequences. For example, if you have 4 cores, you can open 4 copies of 3ds Max and let the first one partition from 1 to 3, the next one from 4 to 6, the third one from 7 to 9 and the last one from 10 to 12. After about the same time it would take a single copy of Max to create 3 partitions you will have 12 and all cores will be used. I have done this with 8 cores and it works great, assuming memory is not an issue. Saving particles through Krakatoa does not load particle data into memory, only 3ds Max, FumeFX and PFlow will use memory, so chances are this won't be an issue.

4. Render the FumeFX directly.
Krakatoa lets you render every voxel of a FumeFX simulation as a particle. This does not give you the same look as driving PFlow particles through the simulation grid, but for some effects it can be a lot faster.
The drawbacks: The particles don't have velocity so you cannot apply motion blur and you need a very fine-resolution grid to create enough particles, which of course increases the FumeFX simulation time.

Once you have simulated the FumeFX and you like the look, you can just check the ">FumeFX" button in the Krakatoa Main Controls rollout to enable it as source and render away. But you can also use some techniques to increase the particle density. Save the simulation to a single PRT sequence, load in a PRT Loader, add a High-Frequency Noise Modifier to the stack as outlined in this tutorial on our site, check the ">PRT Loader Modifiers" option in the Partitioning Rollout and partition the resulting particle system multiple times to increase the density by jittering the particles around their original positions. The result might look a bit noisy though, but it is a possibility.

Finally, you could just render the FumeFX as voxels, which generally requires less particles to produce the same density. Usually you would need to set the Krakatoa Voxel size to the same value as the FumeFX grid size.

These are some of the typical approaches to speed up the iterative process of creating good looking particles and rendering them in Krakatoa. It does not apply solely to FumeFX and Particle Flow - you can combine this knowledge in other situations regardless of the original source of the particles.

Please add your comments below with your own Best Practices. I will migrate this to the FAQ someday.

May your cores be always busy!

And The Answer Is...

2010-02-21T21:06:00.000-08:00

42, of course!

When I started this blog, I promised insight into Krakatoa, MAXScript, Life, The Universe and Everything.
Well, there is no better time for that than this year. 4 days ago I turned 42, and for anyone who holds the Hitch Hiker's Guide To The Galaxy dear to his heart, this age sounds quite cool.

Except it probably isn't.

You see, the other day my parents told me they read or saw an interview with James Cameron and he mentioned he was still feeling like an 8 years old. Oh my, I told them, so I am older than The King Of Pandora!? I have always felt like 16, except when I was actually 16 when I felt like 20. Crazy eh?

Anyway, I don't think I will have more answers about Life, the Universe etc., but I can surely talk about the few things that I know about. For example we are beta-testing Krakatoa v1.5.2, so I might mention some things about it.

Probably the coolest thing that happened to this version (quite by accident) was the speed up that it provides to PRT Volume particle generation. We were working on something completely different (which won't be in 1.5.2, but I guess you will all get sooner than later) and discovered that our memory management wasn't working as expected. Normally, when loading particles from most sources like PFlow, TP, PRT sequences, Mesh Vertices etc., Krakatoa not only gets the particles, but it gets the actual particle count long before it has loaded a single particle. All these sources are able to tell Krakatoa "I am going to give you N particles" and Krakatoa can collect these numbers and reserve memory for all of them before it starts loading their data. (This is the case in the 64 bit build, in 32 bit half of the time it might not be able to allocate enough to fit them all in... Switch to 64 bit if you haven't yet!)

There are a couple exceptions to this - CSV files do not provide even a hint about the actual particle count of the incoming particles - the only way to find out would be to actually read them all, and since reading text files is slow enough, we just do it once and allocate memory as we go. Currently, the same applies to PRT Volumes. The PRT Volume could theoretically estimate the final count based on the mesh, voxel size etc., but it is a bit complicated with all the Shell options and testing against the mesh volume, so right now it does not report any expected count to Krakatoa.

The result of loading PRT Volumes in v1.5.0 and 1.5.1 was exponential slowdown as the particle count went up. We realized that this was due to many (and slow) memory allocations and decided to allow the user to allocate enough memory for a given amount of particles via some manual controls in these cases where Krakatoa does not know the final count. The result was quite mind-blowing. Creating 50 million particles with a PRT Volume went from 11+ minutes down to 17.5 seconds which is 38 times faster. When I reported this to our boss, he asked jokingly "Why not 40x?". So I tested 100MP and that was 92 times faster, so I guess everybody should be happy now. After closer inspection we realized that the memory allocation logic in the previous versions had some flaws, so we are working on fixing them in hopes to be able to get the same speedup without manual memory allocation controls.

Another quite positive development that has been on my Wishlist for a while is the addition of a Particle Input Stream. This allows anyone with basic MAXScript knowledge to open the render-time particle stream of a PRT Loader or PRT Volume and read particle data from it. For those of us with advanced MAXScript knowlegde, it meant the introduction of the Krakatoa Particle Data Viewer utility which can be used to peek into the particle data of a scene object. It lets you select a range of data to display, filter that data by any combination of search criteria, even filter by the value of the Selection channel which could be set by a KCM, thus allowing for filtering by arbitrary MagmaFlow logic! Finally, selecting one or more rows in the editor will highlight the corresponding particles in the viewport!

In MagmaFlow, we added Quaternion operators, the missing and quite important Natural Logarithm operator and fixed some bugs in the handling of nested BlackOps. Flows will now save a material library with any maps used by TextureMap Input nodes and will load them back when loading the flow. TextureMap Inputs can now show a preview of the selected map inside the MagmaFlow command panel. Connections to scene object will be restored when loading the flow if a scene object with the same name exists. Object references using the $ path syntax inside Script Input nodes are now supported for interactive updates. We have some more quite sexy operators in the works, but they might have to wait for the next major release, so I will stop here.
Related to KCMs, we are introducing two new MacroScripts - one for saving a modifier stack to disk, one for loading it from disk. This lets you create a whole bunch of related (or not so related) KCMs on an object, save them to disk together with any other modifiers found on the stack like deformers, Krakatoa Skin Wrap etc, then select one or more objects in the same or in a completely different scene and apply the same modifiers to them.

For the few people using the Presets and History dialog (according to our surveys, most users are confused by it, but I am unsure how to improve it, so I am open to suggestions!), we added two vertical icon strips for faster browsing of records by image. You can now just scroll the image strips and load presets or browse thumbnails, saved preview images or settings much more intuitively.

If you read my blogs, you probably remember my rant about how people don't use the MacroScripts with those colorful incons we provided? Well, we not only replaced the icons with slightly better ones, we also added a Krakatoa menu to the Main Menu bar of 3ds Max (which can be turned off in Preferences) so you can now access all tools and settings from there without having to customize a toolbar.

Initially, we wanted v1.5.2 to be a pure bug fix release, so we squashed a whole lot of bugs - you can read their obituaries here. At this point, we are in the same situation as with 1.5.0 which could have been called 2.0.0 without any problem - 1.5.2 could easily be called 1.6.0 since it brings quite a few new features too.

As mentioned, Krakatoa 1.5.2 is currently in Beta. If you have a commercial license and you want to test it but have no access to the Beta forum, let us know. If you don't have a license, you will have to wait a few more weeks...

A Krakatoa Year In Retrospection

2010-01-03T10:57:00.000-08:00

Here we are, finally in 2010 - we had 3ds Max 2010 for almost a year now so I am quite used to the number already. The past 2009 was a Good Year for Krakatoa customers (at least I hope so) due to the release of v1.5 and all that came with it.
Our company worked under the name Frantic Films VFX on a movie hopefully nobody saw, on another one under the name Prime Focus VFX that many people saw, on a third one that nearly every teenage girl saw and on the movie everybody seems to have seen at least three times since it just made a billion today. And all four of them used Krakatoa in their production. There was a fifth movie that had tons of Krakatoa in it, but it is still not released so I cannot talk about it, and after Avatar it is difficult to be excited about other movies anyway :)

So the beginning of the year found us developing Dragonball:Evolution and G.I.Joe at the same time, the former handled mostly in Winnipeg with animation done in Vancouver, and the latter done mostly in L.A. with animation coming from Vancouver and some effects done in Winnipeg. From RnD point of view, the two movies had some parallels - in DB:E, we had to grow objects using particles, in G.I.Joe we had to destroy them. While the actual implementation of the effects in production ended up quite different, the research process showed that the two tasks can be seen as complementary.
For example, some early tests called for the building of an exo-skeleton from particles drawn from the environments (dust, rocks etc.). Now if you try to use PFlow's Find Target to land particles on a moving surface, it is not impossible, but quite hard to control correctly. Whereas Thinking Particles uses an approach where the user control over a moving particle is gradually stripped away, forcing the particle to go to an exact location, PFlow just uses forces to try to get there even if the point is constantly escaping. If you look at the history of special effects in movies, shooting the action in reverse and playing it backwards has been one of the earliest and most amazing tricks of cinema. With the ability of the Krakatoa PRT Loader to easily control the flow of time using the Playback Graph parameter, simulating the EMISSION of particles from a surface together with complex forces and playing back the resulting PRT sequence to produce the build up of particles on the surface is quite easy. Try it out some time!

The production of the Nanomite effects for G.I.Joe involved two companies (Prime Focus VFX and Digital Domain) using vastly different rendering approaches to create the same look. In the end, particle (point) rendering and voxel rendering looked quite similar - at that point in time, production was locked to a build of Krakatoa that did not have Voxel Rendering yet, and I guess we wouldn't have used it even if it worked already. Assuming that a cloud of metal-eating miniature robots would produce some reflections, we added Environment Reflection Mapping support to Krakatoa, but ended up not using it for the movie. So the end users of the 1.5 commercial release benefited from this development. For a while, there was also support for Anisotropic Specular Highlights in the beta builds, but it just wasn't done right and we decided to pull it out of the shipping product. It might return someday in a better shape. If you haven't seen the "Particles In The Zoo" video by Matthias Müller, you should go watch it now - he used the Environment Reflections ability of Krakatoa to great effect, especially on the "scales" objects.

Then we had the real blockbusters - Twilight:New Moon and of course AVATAR. (I think Blogger.com should add support for the Papyrus font so we can write the name as JC intended) ;)

For the New Moon movie, Prime Focus employed Krakatoa for the "apparition effect", as well as for the foam on the wave that hits Bella. Both effects can be seen in the official trailer (at 1:10 and 1:03 respectively). Not being a teenage girl, I haven't seen the movie yet, but will probably rent it on Blu-Ray when it comes out. In the look development / RnD phase, I also tried to use Krakatoa's new Voxel Rendering for the "diamond skin" effect, but we ended up using an alternative approach based on V-Ray. The Krakatoa effect looked quite promising though and I am sure the knowledge gathered from those tests will end up somewhere else.

Strangely enough, my involvement with Avatar was mostly unrelated to Krakatoa - like with G.I.Joe, I worked on pipeline tools to speed up the production workflow. For G.I.Joe, we used a prototype of an assembly system where all assets were separated and combined only at render time, without using an actual MAX file to hold the scene. The scene was assembled on the fly for editing or rendering, then the changes were saved back to new versions of the asset files in their original locations on the network, and any change to an asset would propagate automatically throughout the sequence. No XRefs involved!
In the case of Avatar, James Cameron insisted on absolute continuity of the animation sequences shown on the 3D screens in the Bio Lab and the Ops center. So we ended up developing a database application (called SAGI = Screen Art Graphic Interface) that would keep track of shot lengths and what is seen on which screen. The 2D artists working on the screen graphics could use a User Interface to this database to enter their latest versions and request a 3D rendering of all screens affected by their entry. The SAGI application would write a control file and "drop" it into a folder monitored by the other part of the system called ASAR (short for Automatic Screen Art Rendering). ASAR was written in MAXScript and was running on a Deadline Slave as a never ending MAXScript job, checking the drop folder for SAGI request files periodically. When it would find a file, it would process it by loading the 3D assets, applying the right timing of the right textures for the Left and Right eye and submitting all necessary passes as new Deadline jobs. This made a human error in the assembly and rendering phase impossible and allowed 2D artists to trigger 3D rendering jobs without any knowledge of 3ds Max or Deadline. But most importantly, it allowed us to change the length of shots and preserve the visual continuity of the screen contents between shots without much human intervention. (You can read the official Press Release here).
That does not mean that Krakatoa wasn't involved in the making of the movie. It helped give the "holotable" terrain its distinctive LIDAR look and was seen on the "false color images" screens showing the energy flow at the Tree Of Souls. (seen below before and after)

Apropos LIDAR - about a year before I joined the company, some shots for X-Men 2 also involved a "holotable" graphics display. It is amazing how these kinds of jobs come around again and again (along with crystal growth and particle disintegration).

Another curious fact about X-Men 2 and Avatar - the former happens to be the movie with the highest Tomatometer rating Frantic/Prime Focus has ever worked on. The latter appears to be the one with the highest financial success. Reading Box Office Mojo daily makes me feel great, especially after all the online negativity before its release.

Outside of the world of VFX, the past year was marked by some of the best concerts in my life, led by the Winnipeg performance of Leonard Cohen, followed not-so-closely by AC/DC and KISS and farther behind by Metallica. (the above sentence should give you an idea of the approximate range of my musical taste). The year also included the worst concert I have been to, unfortunately by an artist I generally love, thus the disappointment was even bigger, but I won't discuss this further. It also saw my AI favorite losing the title despite performances like this, this and this (yes, I am a Glambert!) but producing an amazing album and getting to do this.

Just before the end of the year, me and my wife got to see Chaplin's movie "City Lights" with music peformed live by the Winnipeg Symphony Orchestra. Despite being in black and white and having an almost square aspect ratio, it was possibly the second best movie experience I had this year, right behind Avatar (which was in color, wide screen and 3D, of course)...

So on to a new year with lots of changes for me on the horizon and hopefully with more amazing music and a 2.0 version of Krakatoa on the market!

Happy New Year everyone!

"Grainy, like a Krakatoa render..."

2009-11-21T18:28:00.000-08:00

As you could imagine, I spend a little time googling "krakatoa particles" and "krakatoa render" about once a week to find out what people are doing with it and what they are talking about on various forums.
A couple of weeks ago I discovered the following (rather old) thread on the SideEffects forums related to a Houdini rendering which was described, as the title of this blog says, as "pretty grainy, like a Krakatoa render".
I guess this is a good cause for a new blog.

I must say the actual animation shown in the thread wasn't that grainy, but I am more concerned about the public image of Krakatoa. I suspect the assumption that Krakatoa has a particular look is caused by the huge amount of animations on YouTube that simply have the wrong settings.

Of course, in some cases Krakatoa is being used to produce sand or pieces of solid objects flying around. In such cases making each particle distinguishable as a dot can be desirable. In fact, the Nanomites in G.I.Joe were also rather grainy, but it was the look that was requested.

When using Krakatoa to create effects like Ink in Water or Wispy Smoke though, the rule of thumb is - if you can see a particle as a particle, your Density is too high! The main idea behind Krakatoa (even before it was called Krakatoa) was to take several hundred million particles and draw them together into the image buffer with very low density per particle to accumulate a SMOOTH final result - in the movie "Stay" where our particle rendering was first used (inspired by Doc. Baily's Spore rendering), the effect looked like glowing plasma. In the movie Cursed, we actually used FLOOD to drive millions of particles through a simulation and rendered them in the same renderer to get the Wispy Smoke seen in this PDF.

The recent "Ink" animation created by weareflink for the CCTV shows a pretty good use of Krakatoa simulating ink in water without being able to distinguish single particles. Also, the Vilnius SPA animation by DekoLT is a great example of high density but enough particles to create solid-looking clouds.

Another factor that can cause a lot of grain in the rendering esp. of solid-looking clouds is the Light Pass Density. As you probably know, Krakatoa lets you decouple the density of particles as seen by the lights from the density used by the camera to draw the particles. High density in the Lighting Pass with very high particle counts can produce self-shadowing of particles at the very surface of the cloud because the outer-most layer of particles would "eat up" all the light and the very next particle layer below them would appear very dark as opposed to the very bright lit ones. This can produce not only grain but even very undesired moire effects.

Reducing the Light Density a bit and letting more light penetrate the volume (also resulting in a sweet sub-surface scattering effect) usually solves this problem.

So please, if you are rendering in Krakatoa, make sure you crank up the particle count AND lower the Density until you get a smooth result, then play with the balance of the Lighting Pass vs. Final Pass Density to get the correct amount of light penetration and pixel coverage...

EDIT: You can find some illustrations on this new documentation page.

Have a smooth rendering! ;)

Krakatoa 1.5 - Confusing Changes For The Better

2009-10-25T22:27:00.000-07:00

Once again, while most of this is already covered in the online documentation, I feel that spelling it out for the few people reading my blog might be a Very Good Idea. I will probably have to update the FAQ or just link to this Blog or something like that.

First, the default lights handling.
When designing Krakatoa 1.0.0, we discovered that particle rendering with default scene lights never looked good. This was mainly because the default mode for default lights in Max is a "headlight" right behind the camera, which does not produce very good looking shadows. The alternative mode is two lights which works even worse with volumetric rendering, and requires one more light sorting/attenuation map generation pass...
So we made the decision back then to render particles as self-illuminated if no actual light node was detected in the scene. As result, one could just create some particles, hit render and get an idea where the particles were. On top of that, it worked great with Additive Mode where lighting was usually not desired (although Additive Mode + Lighting was somewhat supported).

When the Krakatoa version which ended up being released as 1.5.0 (it was initially developed as 1.2.0 and nearly shipped as 2.0.0 due to the amount of features added, but that's another story) added support for an Emission channel and an Absorption channel in addition to the Color (Scattering) channel, we had to revise this design decision. In short, having a per-particle Emission channel meant that it would be a Very Bad Idea to render particles as fully self-illuminated when no lights are found in the scene. At the same time, rendering Default Lighting was still as unusable as it was two years earlier - we looked into it again and finally decided to bite the bullet and render particles as not illuminated if there are no explicit lights in the scene.

What does this mean? In short, if you open a Krakatoa 1.1.x scene without scene lights in Krakatoa 1.5.x or just start a new Max scene, add some particles and hit render, you get... nothing. Or so it seems. Looking at the Alpha channel, the particles are there (and the log / progress dialog show that they are actually being loaded and processed). It is just that self-illumination is not applied implicitly, no scene lighting is applied either and you end up with black particles on black background (changing the background color to a brighter color shows that, too).

As you can imagine, this is one of the main problems new users of 1.5.x encounter and report on the forums and in support emails. The possible solutions are

Check >Override Emission and >Use checkbuttons in the Global Render Values rollout (or alternatively check >Use Emission and >Override Emission in the Main Controls rollout). The default Emission Override color is set to white, so your particles will render as white by default.
You can also add a Map to the Color Override in the Global Render Values if you want more interesting results.
If you want to emulate somewhat the Krakatoa 1.1.x behavior where each particle renders the Color as Self-Illumination, you could also add a Global Channels Override KCM and set it to Color Input>Emission Output, then check ">Use Emission" without enabling the >Emission Override option - this will copy the Color of the particle into its Emission channel. PRT Loaders and PRT Volumes will render in their actual color.
Alternatively, adding a Vertex Color Map to the Emission Override slot will render the Color channel and put it into the Emission channel, but this approach is generally slower compared to using a Global KCM.
You could of course also create a Light in the scene to illuminate the particles, but this will cause longer render times due to the illumination pass when rendering in Particle Mode.

While I admit that this seems like a step backwards, the ability to specify Emission per particle means more flexibility (as we will see later). Flipping two checkboxes or creating a light shouldn't be a big price to pay...

The second large change made to the main controls of Krakatoa was the replacement of the big >USE LIGHTING button with a much smaller >Ignore Scene Lights button. We wanted the lighting mode to be the default state of the UI and the underlying renderer, not a special mode one would have to activate. In light of (pun intended) the above discussion about emission and default / scene lights, this was the only logical way to go. On top of that, the >Ignore Scene Lights option does not switch the renderer into a different mode, it simply assumes any existing scene lights are actually turned off. (you could do that with the Light Lister, but it would be a PITA). At a certain point in the development, that option was nearly removed from the system, but it was deemed necessary to quickly produce Self-Illuminated particles using the Emission channel without taking scene lights into account.

And here we come to the third major change - Additve Rendering in Krakatoa v1.1.x used to be a distinct rendering state as opposed to Volumetric Rendering. But Krakatoa v1.5.0 implemented the full volumetric shading equation with Scattering (Color), Absorption and Emission terms, making it possible to control how much color is scattered into the eye, how much is absorbed from each color component as light passes through the particles and how much light is emitted by the particles. Additive rendering is simply fully Emissive rendering lacking Scattering and Absorption, which means that now we can define PER PARTICLE whether it should be rendered volumetrically or additively (or somewhere in between).
You can find more info here.

Knowing that our users will still want to be able to render particles additively without much clicking around the UI (Krakatoa has been doing additive rendering long before it even had that name, since around 2004), we decided to add a special option called >Force Additive Mode. What this option does is this: it copies the Color channel into the Emission channel, while setting both Color and Absorption to black. Thus, particles do not absorb light, do not scatter any light into the eye, they just emit light and accumulate into the pixels as desired. To illustrate what is going on, the >Use Emission and >Use Absorption options are getting grayed out in that mode. In this case, scene lights are also ignored completely even if >Ignore Scene Lights is not checked - particles with black Color and Absorption would produce no lighting effects at all, while wasting time sorting for attenuation map drawing that would not be used anyway...

As you can see, we wanted to make Krakatoa a lot more flexible by providing per-particle channels for the main shading values, but this required some changes to be made to both the default behavior of the renderer and to the controls in its User Interface.

Understanding the processes going on inside the renderer and the logic behind these changes should help you master the new version and lose your old habits built in the less advanced environment of Krakatoa 1.1.x.

I have the feeling I will continue this post in the near future with some more notes on feature and UI changes, so stay tuned!...

Krakatoa Icons Are There To Make Your Life Easier

2009-10-23T22:01:00.000-07:00

Over two years ago Krakatoa 1.0.0 shipped to customers, packaged with some useful MacroScripts (even including pictures of volcanoes). To this day, I shake my head in disbelief when I see that nobody is using them.

As usual, some back story.

Krakatoa was implemented as a .DLR, a Render Plugin like most other 3rd party renderers. Normally, a Max-compliant renderer provides one or more tabs with one or more rollouts inside the Render Dialog. But designing User Interfaces using a resource editor in Visual Studio is a task nobody really enjoys. It is so last millennium, and a real PITA. Before Krakatoa, another renderer (or rather, bridge to a renderer) was under development by the same team - Amaretto. In both cases, the decision was made to split the UI and the core code and implement as many controls as possible using MAXScript. This had several positive implications:

The UI could be developed A LOT faster.
Fixes to UI bugs could be made without even restarting Max.
Fixes to UI bugs could be distributed to customers by simply replacing an .MS file without recompiling the DLR, often the same HOUR the bug was reported. In fact, some fixes during Beta were even distributed by telling the user what line to edit or remark (Do It Yourself fixing!)
Customers could replace portions of the code or modify the UI if desired, as well as read the UI scripts and learn how to access all internal properties of the renderer. All scripted components ship unprotected.
Last but not least, the UI design and implementation could be taken over by someone outside of the RnD team, someone who uses the software in production and knows a bit of MAXScript... That's how I got involved.

This design decision for Amaretto carried over to Krakatoa. In fact, the original UI of Krakatoa written by Mark Wiebe, the "father" and "godfather" of the volumetric particle renderer, was an edited version of the Amaretto UI script file. Krakatoa was already in early Beta when I tried to use that UI and immediately felt the desire to modify it.

One negative side effect of all this was the inability to display the UI inside the Renderer tab of the Render Scene dialog. It is simply not possible to add scripted rollouts to that tab. So the solution was to add a big fat button to that rollout which would then open the scripted UI of the renderer. This was, of course, a slightly inconvenient and quite untypical workflow.

At this point in time, Max 6 had come and gone and the Production/Draft slots of the renderer were replaced by Production only and the ability to save/load render presets. We started discussing ways to keep Krakatoa assigned as the renderer while being able to switch to Scanline, mental ray, VRay, Brazil, you name it. We did not want to lose the current settings of Krakatoa, but the fact that Krakatoa required cooperation with another renderer to produce the final image (including casting shadows from particles onto geometry) meant that we had to make it easy for the user to swap back and forth.

To my surprise, the Render Presets turned out to work great with Krakatoa despite the fact Krakatoa doesn't even use Parameter Blocks like any other Max plugin (our developers found Parameter Blocks clunky and not very stable when changing their structure between versions, so both Amaretto and Krakatoa use a custom string-based storage for all non-animatable / non-object-related properties). So saving and loading a Render Preset with Krakatoa restored everything perfectly.

As result, I decided to create two MacroScripts that would:

Assign Krakatoa as the current renderer if it isn't the current renderer yet when the "Toggle GUI On/Off" is checked, while storing the last renderer's settings in a Render Preset before assigning Krakatoa.
Toggle the Krakatoa UI on and off when the main icon is checked/unchecked.
Load the last renderer's settings from that preset if the "Remove Krakatoa" button is pressed, while saving the current Krakatoa settings to another Render Preset
Load back Krakatoa via that Render Preset if the Toggle button was checked again.
Provide a prompt to let the user reset Krakatoa to factory defaults instead of loading a previous Render Preset if desired.

These MacroScripts can be found along with a dozen other useful buttons in the Krakatoa category of the Customize User Interface. They are the MAIN BUTTONS to access Krakatoa, much faster and more powerful than the Open Krakatoa GUI and Assign Renderer controls in the Render Scene Dialog.

The fact nobody seems to use them makes me sad. It even makes me write Blogs, as you can see here...

So I beg you, if you are a Krakatoa user, spend 2 minutes to create a Krakatoa toolbar and drag the most vital icons to it. Once you start using them, you will be

Orders of magnitude more productive
A Happier Person

The available icons are documented here, in the Fine Manual Nobody Reads:
http://software.primefocusworld.com/software/support/krakatoa/macroscripts.php

Some of the other icons provide one-click creation of PRT Loaders, one-click creation of PRT Volumes from any number of selected geometry objects, one-click assignment of KCMs to any number of selected supported objects, as well as buttons to access the Krakatoa Log Window, the Shadows On Geometry utility, and the ONLY way to access the Krakatoa Schematic Flow tool introduced in v1.5.1.

Obviously, you can even assign these MacroScripts to keyboard shortcuts if you want to be even faster!

This information is already in the Online Documentation, but I hope a Blog will be more effective in spreading the word. To quote a sign that hangs in our office, "Only YOU Can Prevent Render Madness!" Make your life easier. Use the shortcuts!

Lost Gems in MAXScript - Forcing Global Scope Access

2009-09-27T20:55:00.000-07:00

A thread on CGTalk just made me realize that there is a hidden gem in the MAXScript syntax barely anyone knows about, let alone uses. Since it is in part my fault that people don't suspect it is there and because the next update of the MAXScript documentation is quite a few months away, I feel it is a good idea to mention it here.

If you search for "Global" in the help, the corresponding topic will only have rank 22, and the page that links to it is rank 8, but I don't expect anyone to actually read it. I promise I will reorganize these topics next time around, at least the ones related to Scope of Variables, because this is the biggest source of errors and misunderstandings in the programming practice.

Some background info first:

MAXScript has a couple of peculiarities which make coding more "relaxed" (in that the rules are more relaxed for non-programmers), but as a side effect cause a lot of problems in certain situations:

*First of all, variables do not need to be declared before being used. You can use a variable name even if it was never declared or defined, and it will have the value of 'undefined'.

*Second, variables do not require an explicit scope declaration. If a variable is used in the top level context (Listener, or a script without any parentheses), it will be automatically assumed global. If a variable is used in a context below the global (inside parentheses), it is assumed local. This is unless the developer explicitly declared it as either type. For many years, I skipped the explicit scope specification because I knew what I was doing. In the last two years or so, I started forcing myself to declare each variable as explicitly global or local to make it clearer to others - if you read the source of the Krakatoa GUI, you will notice.

Now the trouble from these behaviors is this: If you have a script calling a function or using a variable which was supposed to be defined in the global scope by some other script file, if that other script happened not to be evaluated yet, your call would cause the function to create an undefined local variable instead at evaluation time. Even if the script defining the actual function would get evaluated later, the function call would still be looking at the local variable and seeing undefined, causing an error at run time like "Call needs function or class, got undefined".

Everybody with a little MAXScript knowledge knows that the solution is to pre-declare the global variables in both scripts. This way, when the script calling the function is evaluated, the call will not create a new local variable if the function does not exist yet, but will check the local and global scopes for that name, find it in the global scope and set the pointer at that memory address. When the actual function definition is evaluated, it will also find the existing global variable and will replace the undefined value with the actual function, making the first script operational and avoiding the error.

So here is the little hidden gem most people don't know about - the double-colon :: syntax.

In the topic "Specifying Global Variables As Global Using ::" linked from "MAXScript Language Improvements in 3ds Max 8", it is demonstrated that you can prefix any variable with :: and this will force it to look ONLY in the Global Scope, completely ignoring any local scopes!

In other words, whenever you intend to call a global function or access a global variable but you are not sure whether it is already defined at the moment of script evaluation, instead of pre-declaring the variable as global in the script to ensure it is "visible" to the caller, you can simply prefix it with ::!

You can see some examples I posted in this CGTalk thread:
http://forums.cgsociety.org/showthread.php?f=98&t=810878

You might say: But Bobo, Global Variables are EVIL! Why would you use them?
The answer is that they have their positive role when defining libraries of functions, for example a single struct definition with many functions to be accessible by many other scripts. Such libraries are normally stored in an .MS file saved in the Stdplugs\Stdscripts folder or subfolders thereof. Since that folder is the first location to be auto-loaded by MAXScript at startup, these structs and functions would be visible to any other scripts launched later.

Unfortunately, if two scripts are placed in the same folder and the one defines global "library" functions the other script should access, ensuring the loading order of these scripts become tricky and usually involves fileIn() or include() calls from another script, making things quite complicated (this is a whole other topic to write about). Thus, using :: makes the problem go away and lets you call a function or access a struct you know exists or WILL exist in global scope regardless of the script files loading order!

I believe the "Specifying Global Variables As Global Using ::" topic should be merged with the "Scope of Variables" topic or at least extensively linked from all relevant topics. I apologize that this is not the case yet and hope you will find this little known feature useful!

Happy coding!

A Quarter Century Of Geekdom

2009-09-15T22:11:00.000-07:00

I've got news for you - I am a geek. Chances are if you are reading this, you are in the same boat. And I just realized I am an OLD geek. That's right, it is September 2009, which means that pretty much exactly 25 years ago (+11 days) I touched a computer keyboard for the first time in my life!

The fact I still remember the exact date (September 4th 1984) is just one of the many evidences of geekiness, but since it marks the beginning of my life behind the screens, it could be presented as evidence "A". Of course, there were some other things in my early life that prepared the soil - the fact that I was very into reading (with the occasional attempts at writing) Science Fiction where computers used to live in the 70s (until the birth of the first Apple later that decade) and that I was crazy about Star Wars. I was so into it I actually wrote a rather long poem depicting everything happening in The Empire Strikes Back when I was 15 - since I know at least one Bulgarian is reading this blog, here is the link ).

In those early days behind the Iron Curtain, nobody expected a political change in our lifetime. In fact we were brainwashed to assume the status quo could never change - the Soviet Union had existed for almost 70 years and the Bulgarian anthem contained a line about how "Moscow is with us in peace and war". A nuclear holocaust appeared more probable than a political revolution. Thus the mere idea of working in the area of Hollywood visual effects at any time in the future was quite out there, and of course it never came to my mind. But the desire to do something creative including computers and futuristic space ships was quite natural and I was really very surprised when despite my best attempts to do something else with my life, I ended up doing just that... In a way, I believe that I am living the dream that I never had.

On that fateful September day of 1984, a schoolmate of mine told me about a computer club. Like everything in those days membership was completely free. All I had to do was show up at 8 in the morning and register. My friend also gave me a 4 pages introduction to computers and BASIC programming and after swallowing the whole info before going to bed, I couldn't get any sleep because my brain was trying to combine the few graphics-related commands it knew into something resembling a Zaxxon-like diagonal scrolling game. When the course started in the morning, our teacher told us about an expo with free access to computers that was going on in a big concert hall in Sofia (the expo was called TNTM, short for "Technical and Scientific Creativity of the Youth" in Bulgarian). I went to that hall and spent some of the most exciting days of my life there - I felt something big was happening in my life, but I had no idea how big.

Here is something else interesting to consider. In those days Bulgaria had a population of about 8 million, so the chances of being born Bulgarian were pretty small statistically speaking. Every country in the communist block had an industrial specialty in order to avoid duplicated efforts in multiple countries. For example the USSR was producing most of the cars and planes, while Bulgaria specialized in building forklifts and... computers. In the early 80s, a clone of the Apple 2 was "created" in Bulgaria, a couple of years later the building of PC clones started and most of them were exported. But it also meant that hardware and software were available and the education of specialists in these areas was part of the state policy. (When the Perestroika happened, around 1988 almost half of the computer viruses in the world were Bulgarian, a sad result of having too many specialists with nothing to do). What I am trying to say is that no matter what I did or where I went, computers were around me and appeared to be stalking me... Was I meant to be doing what I am doing? I think so.

So that's how it all started for me. A quarter of a century later, I still get the same excitement when I sit behind a computer, with the small difference I have a slightly better idea what I am doing.

Food for thought: Apple II used a CPU running somewhere between 1 and 2 MHz. My next computer, the Sinclair ZX Spectrum, used a 3.5 MHz CPU. 25 years later I am working on 8 cores with 2.66 GHz each. That is a really nice curve, Mr. Moore.

Remember The Good Old Days?

2009-09-14T21:12:00.000-07:00

At least once a week we (the Krakatoa developers) google the web to see if anything new has been posted related to our favorite piece of software. On good days, we find a new creation by Matthias Müller. (If you have been living under a rock, go check out his particle work on YouTube - http://www.youtube.com/user/MatthiasmVideos )
Some times I find comments related to such videos that are quite amusing. For one, people still consider one million particles a big deal. On a modern machine (I just bought one yesterday, but about that later), one million particles require about 0.6 seconds for lighting with one light and about as long to render, textured with a couple of procedural maps. Including the PFlow processing and getting the particles into memory, it still takes less than 2 seconds. The new Voxel mode is a bit slower, in the same benchmark it took slightly more than 3 seconds. Around the time Krakatoa was first released, we considered 1MP (MegaPoints!) a typical amount for quick testing. With the new Intel i7 CPUs, we are seeing ourselves forced to move to 10MP just to get rid of statistical errors.
But people still assume particle rendering requires renderfarms working on frames for days. If I may quote one comment, "It's amazing what you can do with brute force computation these days". The fact is hard disk and memory bandwidth still don't allow us to saturate 8 cores at 100% - one can get about 75% in Voxel Mode, and in Particle Mode Krakatoa is still single-threaded, drawing at nearly 2MP/second. Still, we have no plans to make it realtime, CUDA or not. The problem has never been the drawing performance, but the time it takes to get all the gigabytes of data into memory, and graphics memory is today where RAM was a couple of years ago - my card has "just" 1 GB...

So I got a new top of the line gaming machine the other day. 4 cores i7 920, shows up as 8 cores in Task Manager. My office workstation has Dual QuadCore Xeons and costs a fortune (I assume), but this new gaming box for under $2K totally kicks its behind! I had to start writing a public benchmark that could be used to compare the performance of all systems out there running Krakatoa to find out how the two compare. Before buying it, I studied all the hardware site reports about how Tri-channel memory does not bring any advantage in typical applications. Well, it is quite possible that it brings some to particle rendering - if you have not heard yet, Krakatoa is quite memory intensive! While people were telling stories about how 64 bit would not bring performance improvements or would actually cause slowdowns, Krakatoa was twice as fast in 64 bit since day one. I would love to compare my results to a similar system with dual-channel memory to see whether the CPUs have anything to do with the performance differences. Hopefully once the benchmark script is released, we will be able to provide the answer.

Apropos scripting... While reading through the Krakatoa-related talkbacks on one of the video sites, I found a comment that made me thinking. It said "Remember when Bobo was writing Maxscripts back in the day?"
I kind of remember those days. Yesterday was one of them. The big difference is now I write them as a job. MAXScript actually manages to put bread (and chocolate, and ketchup) on my table. VR Boboland really has one or two new scripts a year (although DIMaster, NextPlease and LightTouch are quite cool ones and a lot more polished than most of my earlier works). And I still try to go to CGTalk and The Area and help people out, but the days of me writing eveybody's tools for free are probably over. Still, I had not thought about this until reading that post.

Those days were good, but writing the Krakatoa GUI is quite an adventure, too...

Here we go...with a slight delay

2009-09-12T21:24:00.000-07:00

After 14 years online using various newsgroups, forums and VR Boboland to communicate in cyberspace, I felt it was time to finally start a blog. I have been contemplating this step for years now, but strangely enough the final push came from the movie "Julie&Julia", a quite funny comedy nobody should be allowed to see before dinner. The movie not only featured a quote from The Hitch Hiker's Guide To The Galaxy and showed that anyone (not only rats) can cook, but demonstrated how easy it is to start a blog and get hooked on it.

Some weeks ago, just before Siggraph, I tried Twitter but quickly discovered that a character limit is not something I enjoy - I quite like writing long posts. As a matter of fact, I have been using cell phones for over 10 years now but have managed to send less than 10 text messages during this period. But just give me a large keyboard and unlimited space and watch me going!

So here we go - I intend to share my thoughts on some of my favorite subjects including MAXScript, 3ds Max in Visual Effects production, Krakatoa development as well as Life, The Universe, And Everything.