10NES

I've spent the last day gutting and redesigning and rebuilding the UI for my emulator, and I think I finally struck on a UI scheme that will work out the way I want.

Since this is the (at least) 10th rewrite of such, and the solution is now in Visual Studio 2010, and the rendering done with Direct3D 10, I've decided to name it 10NES, which happens to be the name of the fabulous piece of technology which made the NES's power button blink.

My new UI 'technology' has evolved from all my efforts to integrate the emulator into WPF.

Since this projects raison d'etre is allowing users to easily define and apply advanced post processing effects to existing NES games, it really necessitates a very rich UI. From the beginning, I'd used WPF to create my UI pieces, using the model-view-viewmodel approach, as I really see no reason to ever do anything else. The quest really became how to mix WPF with 'snazzy game graphics', in a way which doesn't absolutely crush performance.

The original version of my emulator simply put the output into WPF's WriteableBitmap and threw it on the screen. The performance wasn't awful - the emulator has to render the output in software, of course - but I was at the mercy of WPFs composition engine. When the mouse was moving over any UI pieces, the emulator would choke and stutter hard. Adding any sort of animations to WPF really places a heavy demand on the CPU, I've found that WPF can hit as hard as Crysis depending on the type of machine it's running on. Still - I had all the UI I wanted, and I didnt have to cobble it together out of triangles by hand.

So I started nosing around for another way to draw the ouput, and at the same time was trying to get a version running under linux to be able to run it on my netbook. So OpenGL made good sense. I whipped up a WPF control to host OpenGL (via the Tao Framework) and placed it in there. The main problem here, is that WPF and OpenGL both need low level access to the video card, and they can't both own the same pixels. The upshot is that I can't overlay any WPF content over the OpenGL.

Direct3D would have the same problem, but the WPF team was kind enough to introduce the D3DImage class, which can take a direct3d surface, and composite it inside WPF like any other bitmap. Sounds fantastic, but the rub here is, Direct3D9 doesn't share its junk with other applications on the video card. The surface has to be copied out into regular system ram, and passed over, and reuploaded as a new texture. This would get up to 3 frames per second if I was lucky, and keep the CPU pegged at 100% the whole time.

Microsoft extended this behavior, and created Direct3D9Ex, which does share its junk with other applications on the video card - however this is only possible with the new driver model introduced with Vista (WDDM), and only with Direct3D 10 capable hardware. I was still on Windows XP, and gave up pretty quick trying to make it work. Also, D3D9 really is a hot wet mess of bullcrap.

Around this time, however, I decided to play with the XNA framework, and used the same sort of tricks I used to host OpenGL inside a WPF window to host a XNA 'game' inside the window. This was clunky, XNA is heavily tied to this Game class, and this Game class is a Form. So every time it would start up, the form would display quickly, my application would steal away its draw context, and kill it. There was also frustration trying to stop the Game class from drawing its frames at its own fixed times (ie; call game.Tick() myself), which lead to lots of choppy output with the game object calling Draw out of sync with my emulator presenting something to draw. Also, the XNA framework would only expose a Direct3D9 surface, not a Direct3D9Ex, so even if I upgraded to Vista I'd never be able to overlay content with any sort of performance.

There was a side jaunt into using SDL around this time - it turned out to be the fastest option on my netbook and its smelly hippy linux OS, but I didn't spend much time on it.

So my main focus fell back on to OpenGL, it offered the best cross platform support, and an API I've long been familiar with. It's verbose, and all that verbosity can add up when each verb has to be marshalled across COM boundaries, but it was OK. About this time I started playing with fragment shaders to index and color the pixels from my emulator, saving a whole bunch of operations on the CPU side.

Then summer happened and I stopped working on it.

After summer happened, I decided to upgrade my home machine to Windows 7, because it kicks serious ass, and is the first Microsoft OS to actually perform *better* than the last one, on the same hardware. I got sick of my ancient radeon x800, and invested 40 bucks to upgrade to a newer Direct3D 10.1 capable card.

I went back to work on my emulator again, but there was a problem. OpenGL was broken now! I couldn't create a framebuffer on my home machine, I couldn't create a fragment shader on my work machine. Whether it was broken at the Tao bindings, the OS, or the driver layer I didn't bother to figure out. I could now embed Direct3D content into the application, so I went to try that.

Hosting Direct3D9Ex inside WPF performed as well as I expected it would.. That is, as well as WPF's WriteableBitmap did. I could overlay WPF content, the compositor would take over and make the emulator stutter, but only when I was trying to use stuff. Having a multi-core CPU really puts an end to it, but honestly, thats a lot of hardware to throw at such a simple app.

So I decided to mess around with Direct3D 10, and started falling in love with the API almost immediately. It's so spartan and free of nonsense, you send your shaders to the GPU, then you stream in your resources and geometry, and the video card does all the work. While WPF's D3DImage cannot take a D3D10 surface directly (why not?), you can render a D3D10 surface in D3D9, then host that in WPF. I never actually tried that, though, because at that point I was sick of competing with WPF for drawing time.

Around this point, I started really messing with pixel shaders, and moved all of the drawing logic out of my emulator and onto the GPU. I originally did this as an expiriment to see if I could use the GPU to accelerate my emulator - and it worked, more than halving the CPU load. But there's a cool side effect.

Now the GPU knows everything about the nintendo, and can layer in whatever effects it wants in a game-context aware manner without costing the CPU even one measly clock cycle.

This immediately became what I wanted to do (real water and real clouds in 1943, enhanced lighting effects to really bring home the psychological horror of war in Contra, and so on), so I realized that all the other expirimentation and nonsense had to end, I actually have a goal now, and something I want to release instead of just dork around on when I'm bored.

So I thought about what I actually wanted from WPF. I wanted the ease of creating rich UI components, the data binding, some of the automation features. I didn't want the compositor, however.

So I struck upon the idea of loading my XAML out of thin air without any host container, and somehow sending it to Direct3D 10 - going completely the other way. This way I'm in control of when and how the UI is drawn, even in full screen exclusive mode, and going fullscreen in any of my previous schemes was a major PITA, as to do it properly required tearing down whatever my renderer was, destroying the WPF window, then recreating the renderer with exclusive access to the video hardware.

So to implement this, I created a sublass of Direct3D10's Texture2D class (well, SlimDX's class that exposes it), which held onto a WPF visual. When I want to update the visual, I Measure() and Arrange() it to the size of the texture, render the visual to a RenderTargetBitmap, and use the CopyPixels function to send the pixels in question over to the D3D Texture, which can then be composited.

It's a simple scheme, and works well, and performance is really good, but the rub here is interactivity. My WPF controls dont actually exist on the screen, only a picture of them does. So I need to translate mouse and keyboard events back to it. This part gets tricky, and so far involves:

1) finding out which texel on screen was clicked, by finding the intersection of the ray created by the point between the mouse click and the cameras location, and the geometry, then finally the texel on the intersected face in that geometry. Ok I'm cheating so far since everything is 2D and mapped into screen space, but eventually I'll need this. This isn't a big deal, I can copy and paste this out of any old "how to make k-rad games at home" book.

2) Taking the texture coordinates from step 1, mapping them to 'WPF visual' space, then doing a HitTest against the visual to find the child element affected by the event.

3) Using the RaiseEvent() method on that element, to raise the appropriate event.

2 and 3 seem simple, but they're a pain in the ass, and are involving a lot of crawling up the visual tree to find something I can actually interact properly with. For instance, if the hittest hits a button, it won't necessarily return a button. It will return the ButtonChrome decorator which is over the button. So sending ButtonBase.Click to this won't work.

Crawling up the tree to find the parent of type ButtonBase and calling Click on that will work, however, it won't fire any Command bound to that ButtonBase, so I check to see if there is a command bound to it, and if there is and it's CanExecute returns true, I Execute it. Ditto with Clicking on a checkbox, it doesn't do what I want it to do, so I manually set IsChecked - which properly updates its bindings.

So basically at this point I'm writing a whole lot of code looking for specific UI elements, and calling specific methods on them to get the functionality I want. I have a feeling I could make all this trouble go away if there was a way I could make my own virtual MouseDevice, because I think the fact that I have to attach the default physical mouse to the events is my main problem - WPF's input framework is smart enough to know that the mouse really isn't over that control. Unfortunately, I can't just sublass MouseDevice.

But, for all the hassle, it is working better, I'm in complete control of what to draw, and when, and I can interact with the UI with the emulator running, and not have it miss a beat. In the end there are really only a handful of base types I need to be able to interact with, so it wont be that bad.

It will however, be heavily tied to the windows platform, so I had two other ideas:

1) I'm not sure if it's possible, but if I could do the same thing, but use Silverlight 3 in the background, instead of WPF, then the moonlight project would give it longer legs.

2) I could create my own state manager and classes for UI primitives (buttons, checkboxes, etc), and basically use WPF during the build phase to create a set of sprite sheets for these UI primitives, by rendering each control in each state. Then WPF itself just becomes part of the content creation pipeline, and doesn't exist in the final product (only the pretty visuals). This would be very cool, as I could reuse the same UI on any platform I chose, and create a native linux application that doesn't look like a bag of smashed up frog anuses. I think this (or something like it) will be the way I want to go, eventually.

So now that I've outlined what I've done and what I want to do, I can refer to this post every time I stray off on one of my tangents (like this entire post).

10NES needs to be released at midnight of 2010, or else everything is lost and I will need a new clever name! So I got work to do.