Tuesday, December 20, 2011

On The Usefulness Of The Web

Sorry for the long silence, but I had to redirect most of my Krakatoa-related musings to the official Thinkblog on our company website. It only makes sense since that place gets a lot more traffic.

As you probably remember from my previous post, I am now employed as Product Specialist by Thinkbox Software. As result, I can now enjoy doing full-time what I used to do half of my time at Frantic Films and Prime Focus - co-develop Krakatoa and other products like Frost and XMesh, write documentation and tutorials, record videos, demo at expos and even travel the world! All from my new home in Vancouver!

In the last couple of months, I had to use the Internet and the Powers Of Google to teach myself a lot of things like MEL scripting, Python and basic Maya and Houdini skills. I must say it went pretty well and I solved most problems I had to deal with. I didn't even have to post questions, most of the FAQs were already posted and answered by others long ago. Oh joy!

But here comes the funny bit. The other day I had to do some development which involved the recreation of a certain 3ds Max modifier as a script. In 3ds Max. I wasn't exactly sure where to start and I felt lazy.  Checking the 3ds Max SDK felt like too much work. Having the routine of checking the Web for my answers about the other applications, I decided to google it to see how others might have solved this.

And then I found it. A thread on CGTalk with the exact question I had. And several answers. And code. And more code, going deeper and deeper. I read the answers - they were clear, the examples worked. Just what I needed.

So what's so funny about this, you might ask? Well, guess who wrote that code...

I had absolutely no recollection of solving the problem before, but reading that thread clearly showed I must be getting old. It was my code, my explanations, and they read like something somebody else had written. I was in awe. This was the first time I felt like Young Me knew more than Old Me knows...

The Internet appears to be a good extension of our brains. In addition to Google and Wikipedia that provide access to other people's knowledge, it turns out it is a good way to store our own knowledge just in case we start forgetting stuff... ;)

Tuesday, July 5, 2011

End User Excitement 2011

If you have missed the news, I have a new job!

As of June 20th, after 7 years, 6 months and one week as Technical Director, I am officially employed by Thinkbox Software.

"New" might not be exactly correct, because I have been spending over 50% of my time at Frantic Films/Prime Focus co-developing Krakatoa & Co. in the last few years. For all users who felt uneasy by the fact that my name was missing from the Thinkbox' About page, breathe freely now! As Product Specialist located in Vancouver, I will be involved in the development of documentation, tutorials, customer demos, user interface and tools, as well as participate in feature planning.

I spent the last week in Holland where I participated with two one-hour talks in the End User Event 2011. I was lucky to get the opening slot on the first day, ensuring the 90-seats hall #1 was full for my "Krakatoa and Frost" demo. It was meant as a quick introduction to the power of the two products used together, including MagmaFlow control over Radius, Color and Orientation channels; Procedural particle placement using ray intersection and nearest point queries; Fast collision detection using Krakatoa operators in Particle Flow, increasing particle count using Frost and PRT Volume; Dynamic filling of closed volumes using PFlow and Krakatoa Collision; Creation of rivets using Krakatoa, Frost and MAXScript to sample control maps on complex structures; Cloud modeling and rendering using Sphere Gizmos, Frost, PRT Volume and MagmaFlow, including a quick preview of some Krakatoa 2.0 goodness. I intend to post some new tutorials based on these demos on the Frost Tutorials page later this month. (EDIT: First tutorial is now available here)

Image by Master Zap
I heard some people mentioning that I was talking too fast, but this was necessary in order to fit 6 practical examples created pretty much from scratch in front of live audience within 60 minutes. Each time I finished a topic I looked at my watch and was amazed to see exactly the minutes I hoped to see - 10, 20, 30, 40, 50... done! I had executed these demos dozens of times at home, but never timed the complete presentation to make sure it fits in the time slot. Turned out my internal clock was pretty precise, nearly as precise as the Dutch railways that also impressed me a lot ;)

My second talk was an overview of the software development at Frantic Films and Prime Focus leading to the intellectual property now owned by Thinkbox Software, in the context of the visual effects projects that made the tools necessary. In this light, I tried to explain how applications like Deadline, Krakatoa, Frost and Flood came to be.

Despite my jet lag I managed to listen to a lot of presentations by others. My general impression was that all studios, big and small, have to fight the exactly same problems when it comes to pipeline and artist friendly tools, and there is a lot of wheel reinventing going on. I loved the Planet 51 pipeline presentation which reminded me so much of the Prime Focus pipeline it was spooky! Great minds (read: Laszlo Sebo and Gonzalo Rueda) think alike! I was also positively shocked to see Neil Hazzard spend time demoing the script I wrote for Nitrous/Quicksilver NPR mode customization.

Both my talks gave me a short opportunity to show some internal W.I.P. development for Krakatoa 2.0. My impression was that these upcoming features will make a lot of people happy, but I won`t discuss them here yet in hopes to make the upcoming Siggraph 2011 in Vancouver even more exciting. You know me, I like teasing as much as my new old boss Chris Bond does ;)

Stay tuned, there's just a month left until Siggraph!

Friday, April 15, 2011

Full Speed Ahead

I realize I haven't posted in a while. Not even during the pre-release frenzy surrounding Frost, Krakatoa's BFF. Having spent the last 4 days in Las Vegas showing off Deadline, Krakatoa and Frost to anyone who came to see them, I thought it was time to write something about the future.

I am sure those of you who remember my "How Fast Is Fast" blog about Krakatoa 1.6 wouldn't expect yet another similar jump in performance from the next version. Well, you would be very very wrong, and I have the numbers to prove it. In fact, the upcoming version of Krakatoa might provide a bigger speed-up over 1.6 than 1.6 had over 1.5!

Creating Partitions and loading PRT file sequences from disk are among the most typical workflows in the Krakatoa pipeline, and the time to load the particles has traditionally been about half the time of the rendering process. Several factors affect the speed of loading - the speed of the hard drive or network connection serving the files, the speed of the CPU reading the data and decompressing the stream into memory, and the number of additional operations performed on the particles while loading, like MagmaFlow and Material evaluation, deformations and culling. In the past, the latter operations were gradually updated to support multi-threading, but the initial loading has remained limited to two threads, and typically saturated only one core.

Not anymore. I had the pleasure to benchmark an early alpha build of what might become Krakatoa 2.0 on a variety of machines with a multitude of storage solutions. To test the pure loading speed, I created a Box with dimensions 100x100x100, converted to a PRT Volume and partitioned it to disk as 100 partitions, each one with 1 million particles. Note that this introduces some overhead to the loading process - loading 10 partitions with 10 million each, or one partition with all 100 million would be somewhat faster, but I wanted to produce a more realistic case which is nearly the worst case of partitioning since Krakatoa currently limits the max. number of partitions to 100. Also, it would be able to create enough threads for any number of cores. I kept the default channels layout - Position, Velocity, Normals, Color, Density and ID - to simulate a typical case even though I did not need some of these like Velocity or ID for the actual rendering. I then created a single PRT Loader from all 100 partitions, created a default Spotlight and rendered.

My first test used the slightly outdated hardware tasked to perform the Thinkbox demos at the NAB show in Las Vegas - it was a dual Intel Core Duo, in other words four physical cores, no Hyperthreading. The interesting thing about this machine though was that it contained one 7200 RPM harddrive, two striped 10000 RPM drives, one SSD drive and a Fusion-io card, all connected to the same hardware. This gave me the ability to find out how the storage medium affects the new software.

For comparison, I used the current 1.6.1 build. Loading the 100 million particles with it took 57.3 seconds and the total rendering time was 2 minutes 46 seconds. It did not matter what drive I loaded the particles from because the speed was fully determined by the performance of the one core reading the ZIP stream from the PRT file.
Loading the particles from the 7200 RPM drive using the new build cut the loading time down to 38.2 seconds and the total rendering time to 2 minutes 29 seconds. This is not a very impressive speed-up, but it reached the physical limitations of the hard drive, while loading the CPUs as much as the I/O bottleneck allowed. Having 4 cores, 4 threads were created to read 4 PRT streams at once, but the drive could not keep up with the demand.
Loading the particles from the two 10000 RPM drives brought the 4 CPUs to about 80% saturation before the bandwidth of the hard disks became the bottleneck again. The time to load the 100MP went down to impressive 16.1 seconds, but it was obvious that there was more to be expected from the solid state drives. And indeed, running the exactly same tests from the SSD drive gave me 11.2 seconds for loading and 1 minute 49.9 seconds total render time, while saturating all 4 cores completely! Trying the same with the even faster Fusion-io card produced the same loading and rendering time, clearly proving I had reached the CPU bottleneck.

Thankfully, Fusion-io was well represented at NAB and I got the chance to run the benchmark on an 8 core machine to test the waters. My gut feeling told me I should expect about half the loading time with twice as many cores so I wasn't exactly surprised when the faster system loaded the 100 million particles in 6 seconds and finished rendering in only 52 seconds! (Un)fortunately, all 8 cores were once again at 100%, making the result CPU-bound instead of I/O bound, leaving me wanting to test on a 16 or 32 core machine to see what a Funsion-io card can really do for Krakatoa. My gut feeling tells me again we could expect loading times of 3 seconds or less for 100 million particles on such a system, but until I actually get to try one, I can live with the results pretty well. Supposedly, a good SSD drive could keep up with an 8 core system to produce the 6 seconds loading time, too, so you don't have to spend the equivalent of a new car to get that performance...

There are several other areas that have seen some speed up in the new version - on my home i7 quadcore machine, the sorting for both lighting and drawing of 100 MP went down from 12 seconds to about 8, and the drawing was reduced from 5 to 4 seconds. That machine is not very good for testing the loading improvements due to a slow hard drive though, so if you could imagine a modern computer with a lot of cores and fast SSD drives, Krakatoa will literally fly on it later this year!

Obviously, this is just the tip of the iceberg when it comes to what will be new in the next version of Krakatoa. Wait for Siggraph and be very, very excited - I know I am...

Saturday, September 25, 2010

Servicing the Pack

The Krakatoa build released on August 31st is no more. It has been replaced by a build released on September 24th. If you are already running 1.6.0, you should do yourself a favor and update now. Go, I will be waiting!

This is the first time we replace a build without incrementing the version number. We call it a Service Pack 1, but you can read it as "The build we should have released to start with" :) It fixes several bugs related to rendering FumeFX, adds the Icon Size spinners to all PRT objects that were missing one in the original build, and because there have been more than three weeks since the release, it also adds some new features here and there - Yours Truly cannot sit around without touching up the user interface...

Here is the list of changes:

Bug Fixes
  • PRT FumeFX with FumeFX 2.0 was causing a crash when switching to Post Cache mode. This has been fixed.
  • PRT FumeFX was showing only a portion of the simulation due to incorrect handling of Adaptive bounds. This has been fixed.
  • Thinking Particles was causing particle evaluation to hang at render time if there were two or more TP objects in the scene. Fixed.
  • Deformation Modifiers were inverting the Velocity Channel in PRT Loaders. This has been fixed.
  • Fixed the MAXScript call for saving all rollouts to presets which was causing a System Exception.
  • Fixed a crash when trying to load a Save Channels Preset.
  • Turning off the Viewp.Spacing in PRT Volume causes the render-time Spacing value to be used for the viewport, too, but changing the spinner wasn't updating the viewport cache and required a manual update. This has been fixed.

Feature Improvements
  • The Thinking Particles Partitioning code was tweaked to allow an Integer Helper containing "*RandomSeed*" in its name to be incremented just like a regular "RandomSeed" property. This can be used to adjust the Random Seed of MatterWaves by connecting an Integer Helper node to the RandomSeed property which is otherwise not exposed to MAXScript.
  • Undo/Redo wasn't restoring the Input Nodes' value correctly. This has been fixed. Please note that if the track is keyframed, performing Undo/Redo can affect the complete curve as the animation state at the time of Undo creation is not stored and there is no way to store/restore the complete controller. This fix also improves the Macro Recorder playback.
  • The ">Interactive Mode (SLOW!)" option in the Krakatoa Channels Modifier wasn't handling Geometry objects used in Surface Operators. This has been fixed.
  • Added conversion buttons to the SurfDataValue Operator to quickly turn the output value from Integer to Float, from Float to Vector, extract X,Y and Z components of Vector or transform Vector From World to Object space or into View Space.
  • Added an explicit INSERT/BRANCH mode displayed in the title bar of the MagmaFlow Editor and controlled by the Insert key in the Numeric Keypad. When set to Branch and a single node is selected that has an output socket connection, creating a new operator will not insert into the existing connection but branch into a new output connection. Once the node is branched, the mode will revert to INSERT automatically, since it makes more sense most of the time. In previous versions, this was done with the SHIFT key, but it could collide with some keyboard shortcuts that use the SHIFT key as a modifier key.
  • Fixed the mouse right-click hit testing when the MagmaFlow Editor is docked as Extended Viewport.
  • Added support for Schematic Flow as Extended Viewport, including correct right-click handling.

User Interface
  • PRT Volume, PRT FumeFX and PRT Source now expose an Icon Size spinner to the UI. The MacroScripts now use the XY size to define the icon size of PRT Volume and PRT FumeFX.
  • Added support for saving and loading the Channels To Save list.
  • Added all rollouts to the right-click menu of the Float/Dock icons with a check mark for visibility. Selecting will toggle visibility, holding SHIFT will navigate to the rollout.
  • Added right-click handling to all "Back To Main Controls..." buttons in addition to the left-clicking.
  • Added the keyboard shortcut names to the right-click menu of the Float/Dock icons.
  • Changed the navigation buttons throughout the UI to unhide hidden rollouts before navigating to them.
  • Reworked all MacroScripts to disable themselves and avoid any crashes if the Krakatoa path is removed from the plugin.ini.
  • The Rollout Preset files were not restoring the visibility of the "Shader Parameters" rollout correctly - even if saved as hidden, it would be displayed because the update of "Main Controls" rollout's Shading Mode list was forcing it to appear. This has been fixed by processing the "Main Controls" before "Shader Parameters".
  • Optimized the "Load From History" dialog to use the History Cache instead of scanning the files from disk. With 8281 History files on disk, the old code was taking 47 seconds to update the first time the dialog was opened, and 10 seconds in consecutive attempts. With the new code, the time to open the dialog went down to 0.078 seconds!

Just the first 4 points of the Bug Fixes list should be a good reason to update if you are using FumeFX 2.0, Thinking Particles or modifiers on PRT Loaders. In fact, I had to postpone a video tutorial recording session I had planned because it totally depends on a modifier on a PRT Loader and would behave quite strangely in the original 1.6.0 release... Stay tuned! And if you haven't watched the other YouTube videos, go here, here, here and here.

If you find more bugs hiding in the newest build, please let us know. We will chase them and hit them with a big hammer!

Friday, September 24, 2010

I Am A Cheater, Too!

Thanks to this CG Society article, I discovered the following "Making Of Tron" video on YouTube:

Part 1 Part 2 Part 3 Part 4 Part 5 Part 6 Part 7 Part 8 Part 9

The video is beyond awesome, but the most striking part of it, even more than the behind the scenes footage and image material, was THIS SENTENCE by Steven Lisberger:

"The Motion Picture Academy refused to nominate Tron for special effects because they said we cheated when we used computers..."

While I point at Star Wars as the main influence for me to end up in the visual effects industry, Tron has a very special place in my heart, too. Unlike Star Wars, Tron was not shown in communist Bulgaria's movie theaters back in the early 80s. But the magazine for international journalism "Parallels" ("Paraleli" in Bulgarian) had a large article with images about the making of the "first computer generated film" which I used for a school project about "an art form of my choice" (film is considered the Seventh Art Form, so I went with Science Fiction Movies as my theme).

For 6 years, I could only dream about seeing the actual film. My dream came true thanks to "Perestroika" on February 14th 1988, a Sunday. It was 3 days before my birthday and it happened to be a weekend. I was in my first year of army service and my girlfriend came to visit me so I got a two days leave. We went to a so-called "Cinema Video Club" - these were coffee shops with video projector that were showing private (read: illegally copied) VHS records of western movies that could not be seen or obtained through any other channels. Copyright law did not extend beyond the Iron Curtain, and thankfully the idea that western movies were "capitalist propaganda designed to brainwash the youth" was not so popular anymore, so we could finally see films we had missed originally. And they were showing Tron that day!

Of course, I was completely blown away by what I saw then. A year later, during my second year of military service, I was in a computer center and had the opportunity to play with simple computer graphics. This led to getting a PC home with me for another two years after I finished service as I continued working as consultant to one department of the military academy. Then, in the first year as student of architecture, I presented a design project that was made and printed on a computer. Guess what the reaction of my professors was? "This is not fair to your peers, you cheated by using a computer".

As I have mentioned in previous blogs, I never intended to turn computer graphics into profession, mainly because it was impossible in the current political climate. But for some people who were living in a different environment, Tron turned out to be the actual inspiration for working in the visual effects industry.

There still is an original Tron coin-op game in one of our offices. This is because the company previously known as Frantic Films was started by Chris Bond who was one of those people who were inspired by Tron and actually went into the visual effects industry because of that inspiration. With Tron Legacy less than three months away, it is both a wonderful and curious feeling to look back at where things started and realize how far we, the industry and the world have come...

Wednesday, September 1, 2010

Shipping Time

Krakatoa v1.6.0 is here! Where? There!

Surprisingly, we actually hit the shipping deadline this time. Not that it would have mattered one way or another since all users with support contract and even anyone who asked for it had access to the Beta builds for months. In the past, we always went by the saying "It will ship when it is ready". Being a niche product developed by a small team, we always had that luxury large publicly traded companies do not have - we'd rather delay a release than ship with known bugs. That's not to say there are no unknown bugs in there - if you find any, please tell us!

As mentioned in previous blogs, Krakatoa is now heavily multi-threaded and thus a whole lot faster. It integrates better with FumeFX incl. v2.0, supports the shipping version of Thinking Particles 4, provides new MagmaFlow operators, produces some Render Elements, generates much better shadows and fixes a lot of shortcoming of version 1.5.1. The UI is faster and more streamlined, while the number of additional tools has gone up. Look out for the Krakatoa menu in the Main Menu bar of 3ds Max - it exposes most tools that required customized toolbar icons in the past.

The new version even comes with a new license dialog written specifically to replace the totally unusable FlexLM one that has plagued humanity for years. Yes, the number one support problem of Krakatoa was caused by a piece of code that wasn't written by us. We really hope entering the license server or file will be much easier now...

On the curious side, possibly the smallest change with the largest impact on usability (or at least my sanity as a user) was the addition of a separator between the Copy and Clear menu items of the Log Window's Edit menu. You have no idea how many times I have cleared the window instead of copying the selection in previous versions!

We committed some "crimes", too. A day before release, a user asked for better PRT object icons to easier distinguish between PRT Loaders, PRT Volumes and PRT FumeFX objects. The request came literally in the last minute and no developers in their right mind would start changing that amount of code between Release Candidate 2 and Release Build. Well, fearless team that we are, we designed new icons, wrote a script to export their meshes into C++ compatible vertex and face lists and submitted the new code AN HOUR before the final build was produced! The impact risk was low, the positive effect on usability was significant. We had discussed that change before, but there were so many more important features and fixes, we never found the time. When everything else was done, it felt like we could afford to sneak that in and make some people happy...

Now that we have the final build out there, I really hope to be able to produce some new videos - my first and last Krakatoa-related videos were recorded using v1.0.0 back in 2007!

You can read more about the features of Krakatoa v1.6.0 here and here.

Tuesday, July 20, 2010

Changes and Dependencies

A couple of days ago somebody sent me a link to an automatic "Siggraph Paper Title Generator" which produced quite convincing scientific-sounding output. The title of this post could easily be mistaken for something programming-related, but it is about something else completely.

I haven't posted since April because the last couple of months were quite crazy. In short, my family just finished relocating through half Canada - from Winnipeg, Manitoba (the city closest to the geometric center of North America) to Vancouver, British Columbia (at the very edge of the continent). This is a quite positive change - no more winter temperatures lower than on the South Pole, no more mosquitos in the summer (considered the "national bird" in Manitoba ;o)) And last but not least, finally some hills, mountains and curves on the roads to make the driver in me happy.

The move itself went ok without any incidents, all our stuff arrived a week later than expected but absolutely preserved (especially the PC and the TV - sigh of relief), and the first thing after ensuring electric power was still on was calling the cable/internet provider to order the Most Important Thing after water and bread. This was on June 28th and I was told that the earliest possible appointment date would be July 11th. TWO WEEKS without Web, Email and Skype? Oh my...

The very next day I replaced my trusty dumb phone with a smart one just to get some form of online access. This gave us at least some email, web services and the ability to chat via text messaging with our parents in Europe, plus the added bonus of GPS turn-by-turn navigation. While this was one of the best purchases considering the situation, it wasn't enough and I finally realized how dependent (read: Addicted!) we have become to the internet.

Two days after arrival I was chatting with a colleague and discovered that another company otherwise known for its wireless services and cute animal commercials had started to provide optical cable in our area. In fact, they even had a brochure sent to the "current occupant" of our apartment, but my selective reading brain totally skipped over the first two letters of the title (which said "TV"!) that also promised phone and internet. So I immediately called the company and asked if we can even get their services, since I knew that some large buildings downtown had preferred partnership with the former cable TV monopolist I already had an appointment with. The answer I received left me nearly speechless. "Looking at our data here, it appears that your building has been pre-wired for out optical cable services, and TV and internet are already pre-paid by the building management for a year!". Oh, and they could come 5 days earlier than the competition (YAY!)

Too good to be true?

The lady on the phone promised to call me half an hour later to confirm my credit check and the appointment for the 6th of July. She never did, so the next day I called again. The person at the other end of the line went through the whole identification process and confirmed the 6th of July appointment. So on the 6th of July I took an hour off from work and went home to be around during the installation... except that nobody showed up. Another phone call later I learned the amazing truth: Computers Hate Me!

I was told that a software bug in the internal system of the company is preventing anyone in the call center from scheduling that appointment (three people so far had attempted to enter it and all got the error which I was told "nobody had seen before"). Being a TD and fighting the good fight with bugs every day, I don't really mind bugs, but I do mind when people don't tell me about them (Funnily enough, the name of the company implies they would "tell us" when something like that happens ;o)). So I was told that the top-dogs of the software team were looking into the problem and would call me as soon as it was fixed. I was even encouraged to call again the next day to check. Of course I did and had my fifth conversation with a fifth person about the whole issue. "Please believe us that we are working on fixing the bug". In all this time, I never received a call back from anyone. Probably because nobody cared personally, or probably because my bill was already paid for a year anyway, so an important stimulus was missing... ;o)

While getting home from work via a slightly different route that day, I realized that the head quarters of the company I was dealing with was literally the NEXT BUILDING to ours! So the next morning I went into the lobby and asked to talk to someone responsible for customer relations. Instead, I got connected with the technical team responsible for that "bug" and was told the bug was fixed and they could finally schedule an appointment. (Until this day I don't know whether this was just good timing or there was never a real bug). Of course, this was on July 9th so the earliest possible opening was for July 16th - over two weeks after my first call...

Being as addicted as I turned out to be, I could not wait 5 more days without internet. So we scheduled the 16th but also left the initial appointment with the other company on the 11th and on that glorious Sunday finally got some cable to scratch that online itch. Four days later we switched to the optical cable with HD, free PVR, 15 Mbps wireless access from all devices and finally felt home... This blog is just a reminder how much our lives have changed in the last decade and how dependent we have become... Now let's hope SkyNet won't take over soon ;o)

Back on topic (this is a Krakatoa/MAXScript blog after all) - the next shocker is that the year of the 20th birthday of 3D Studio will be the first in 11 years with me missing Siggraph. I wish everyone going a great time and while I am sad we won't be able to show you first hand what we have done to Krakatoa 1.6, I promise we will be very active online once it is released. I am spending a lot of time polishing both the documentation and the UI, and a lot of features are still being tweaked for maximum flexibility.

I hope the next blog will be a lot more interesting.