Debug PlayStation3

Since I did a similar post for the Xbox 360 XDK a while back and since this is intended to be mostly a developer-oriented blog, I thought it would be in keeping with that spirit to let people have a general idea of what to expect from a debug PS3 – which recently arrived.

Photo album: All of the screenshots linked to below (and more) can be viewed separately in this photo album here.

Profiling/debugging

There are some limitations as to what you can do with regards to profiling – even on a Debugging Station. Code instrumentation through gcov is entirely possible, but this was not very appealing to me anymore given that we have been able to do this more or less even on Retail with Themaister’s net-stdio implementation. Gprof seems to have been made unavailable as Sony started moving more and more away from the open source GNU toolchain they originally based their development environment on. What’s left is a proprietary profiler that will not work on Debugging Stations but requires a Reference Tool instead – it has 256MB extra main RAM, bumping up the main system RAM in total to 512MB (plus 256MB RAM for the RSX) – which, together with the unit’s Communication Processor, gives it the extra horsepower it needs to do real-time profiling on-the-fly amongst some other things like realtime video capturing and graphical analysis through GPAD – which I’ll touch on in a moment.

LibGCM HUD


A very nice feature which seems to be only available on debug PS3s is the ability to run code with an RSX profiling tool called ‘GCM HUD’. As the name would imply, this is a Heads-Up Display overlaid on top of the application you’re running that neatly provides you with a point-and-click interface giving you access to features such as as:

  • Fragment/Vertex program debugger (with the ability to set breakpoints and step through the code line by line from the console itself – with no PC having to be involved or connected to the PS3.
  • RSX Performance Counters (telling you how effectively you’re utilizing the RSX).
  • RSX Command Buffer log (showing you the workload of the RSX in real-time)

The whole interface is mouse-driven – so to interact with it, you have to hook up an USB mouse to be able to control it. This is not where the usefulness of this tool ends, however – GPAD (Graphics Performance Analyzer and Debugger) allows you to interface your PC with this HUD and dump all the performance data to your PC. However, again, some options – such as live video capturing – can only be done with the Reference Tool.

GPAD on a Windows PC interfacing with a debug PS3 running GCM HUD

Samples

A brief rundown of the samples that struck my eye –

Deferred shading

Deferred shading lends itself well to the architecture of the PS3 where you have a comparatively humble GPU (RSX) and up to 6 SPEs each running at 3.2GHz – the idea is to essentially subdivide the result of a shading algorithm into multiple parts that can be spread across different render targets/CPUs only to combine them at the end into one composite whole. Using this approach, the RSX can simply offload a lot of the vertex and fragment computations that are done in shaders to the SPEs which in turn crunch through the calculations (the original shader algorithm having been subdivided into parts for each SPE to chew through) only for the RSX to combine all these separate parts and render the picture.

Commercial game developers like DICE have started using this approach for games like Battlefield 3 to achieve graphical results and performance which normally would have been unattainable if all that was available to them was the RSX alone. A link to a slideshow presentation is available here.

Deferred shading sample pics

Yes, the FPS count starts to drop heavily once you start adding extra spot lights.

PSGL samples

At one point, Sony was asking developers whether they would be interested in having PSGL conform to the OpenGL ES 2.0 specs (link here). This has unfortunately never happened however, as developers seem to have mostly preferred to go with libGCM as their main graphics API of choice on PS3. This has meant that the development environment has started becoming more libGCM-centric over the years with PSGL eventually becoming a second-class citizen – in fact, new features like 3D stereo mode is not even possible unless you are using libGCM directly.

However, this has not deterred them from still making available a couple of nice samples that illustrate what PSGL is capable of in conjunction with Cg shaders (which is mostly what we use with the homebrew emulators up until now). Some nice examples show Cg being applied to render hair, skin mapping, normal mapping, parallax mapping, and sophisticated water effects (the latter ones definitely being a cut above our own water.cg shader – made by Themaister).

Below you can see some screenshots illustrating nice tech demos using Cg and SPUs in tandem –

Cg being used for hair specular highlights

Cg being used for water ripple effects

Conclusion

What the debug PS3 will allow us to do is to finally start getting rid of bugs – memory leaks that were simply impossible to flush out with a retail box because of the inherent limitations of printf debugging. There are quite a few memory leaks that can now be tracked down in FBANext PS3 through crash dumps and live TTY logging.

Most ambitious of all, it will finally allow us to start writing code for SPUs – which I didn’t even want to attempt doing before because of the pain that was the retail environment. It remains to be seen whether we can really offload much if anything in the emulators to the SPUs – however, I have seen many creative uses for PPU-to-SPU offloading in some samples already, including function/virtual function call offloading to specific SPEs amongst a host of other things I didn’t even consider before.

Expect to see some real solid progress on MAMENext PS3 very soon, and for the emulators to improve immeasurably overall. Still, there are some inherent limitations even to the Debugging Station – the proprietary profiler will only allow me to do static analysis right now with a Debugging Station – if we wanted real-time profiling, we would have to be in possession of a Reference Toolkit, which is simply above anybody’s budget to be honest. This means that it might be impossible for any homebrew dev to ever do a serious attempt at porting a demanding emulator such as PCSX2 or Dolphin even with a debugging station because libgcov will only carry them so far – definitely forget it altogether on the retail units and the limited development environment they have available to them (whether it be the PS3 SDK or PSL1GHT).

The debugger that comes with it however is a seriously powerful tool and can carry us a long ways since it shows us live disassemblies of the code – meaning we can at least set a breakpoint somewhere, look at the ASM that the ‘shitty’ PS3 compiler turned our C/C++ code into, and then rewrite the code ad nauseam until the code generation starts to make more sense of our code and translate it to faster ASM code.

Clarification on psx-scene

Update (Halsafar): Original post http://pastebin.com/gc0036Qg

While I don’t wish for this site to start devolving into scene drama, I felt a post was necessary on why it is that I’ve recently decided to part ways with psx-scene and what led me to taking the drastic decision I felt was necessary in order to make a statement. This affects all other homebrew developers in the grand scheme of things – and I acted accordingly as the situation demanded it.

New TOS waiving your rights

After the acquisition by QJ.Net, they quietly injected into the already existing Terms of Services a new law that expressly authorizes PSX-scene to effectively assume ownership of any and all material you make available to the public on their community site. I will let this quote do the talking so that there can be no debate about this –

Psx-scene Terms of Service

“User Provided Content, License. You are solely responsible for all content or materials that you post, submit to, or transmit through the Service. By submitting materials or content to PSX-SCENE.COM, you grant PSX-SCENE.COM a license to copy, use, display and create derivative works of the material or content submitted for any purpose, including, without limitation, the promotion and marketing of the Service and the operation of the PSX-SCENE.COM system. By submitting materials or content, you automatically agree (or, to the extent you do not own all rights to such materials or content, you represent and warrant that the owner of the content or materials has expressly agreed) that without any particular time limit, and without the payment of any fees, PSX-SCENE.COM and anyone it permits may reproduce, display, distribute and create new works of authorship based on and including the content or materials. You may not submit content or materials trademarked or copyrighted by anyone other than yourself.”

Now, obviously, this is disingenuous on its face. Stepping aside for a moment the fact that they can not simply waive your rights like this – the fact remains that there are licences to obey when you as a porter decide to port an emulator to another system. Those licences are very clear on ownership rights and your ability or permission to be able to buy or resell them. For psx-scene to put themselves into the enviable position where they believe they can ‘reproduce, display, distribute and create new works of authorship’ speaks to the audacity and clear disrespect of these new site owners who happen to view developers as a bunch of schmucks to be hoodwinked and cajoled into servitude.

‘Successful business != Conning people into waiving their rights’

As a response to that, I took down all emulator threads on their forum that I have personally worked on ( seeing as the previous owner on there granted me Moderator status with no strings attached). Also, I complied with the wishes of many former members who wanted all their content to be removed because they have similar misgrievances about this new ‘clause’ they added to their terms and they don’t like this site profiteering or benefiting in any way from it. Now, obviously, that resulted in my ban – it couldn’t have gone any other way – you know once you throw down the gauntlet like this, what the consequences of that are going to be. I personally requested for them to cast the first stone and ban me – I had no more desire staying on there and feel no need to associate myself with that site anymore.

Now the sad thing is – this kind of profiteering has become fashionable now over the past few years – when you mistake being a ‘good entrepreneur’ with being a ‘con artist’ and a ‘profiteer’ and, worst of all, you don’t even care about any of the amoral connotations that brings along with it – then obviously you’re going to have friction between developers who are doing this mainly out of a passion and then these kinds of cynical businessmen following in the grand footsteps of PT Barnum and his ilk.

SNES9xNext – Android Release

With VBANext rolling on in the background we took a small side step and got SNES9xNext repository going. To help mark this release I prepared a port of SNES9xNext for Android. This is particularily interesting since all current SNES ports on Android are of SNES9x 1.43. The SNES9x 1.43 branch is known for its raw speed so it is a chose candidate for running on slower CPUs. Since SNES9x 1.5X the team has gone towards improving audio by adding blargg’s audio core among other things. In turn the SNES9x 1.5X branch is quite a bit slower. Putting arguments aside which is better the plan here is to streamline being able to quickly deploy each version. Currently SNES9x 1.43 runs full speed with frameskip=0 on all games on most native Android 1.6 or higher devices. While SNES9x 1.52 requires a newer tablet basically. On my HTC Incredible S which has a 1Ghz ARM-7 CPU I get ~55fps on most games with frameskip = 0 and full speed with frameksip = 1.


Download SNES9xNext 1.52 Android

Please post feedback in the forums!

SD3

Shadowrun

Enjoy!
Halsafar

Homebrew Cafe Forum opened

Homebrew Cafe Forums

The Homebrew Cafe Forum has just been opened. Hop on and register – moderation will be limited to pretty much three guys at the start, but we’ll see where it goes.

It was becoming necessary to have a forum eventually – as people have been meaning to get into contact with me through some means other than unofficial scene forums. This forum will serve a variety of purposes – there is a PS3 emulators section where users can post questions, suggestions, bug reports, requests and whatnot to do with the PS3 emulators. There is also a PC section dedicated to Themaister’s SSNES (and upcoming PC frontend ports). There are also other boards of a more slightly technical bent where porters/devs can get into contact with me or others – or simply idle tech talk if that’s what people prefer.

Android NDK r6 Released

This is a day late but none the less important to the mobile efforts.  Google released the sixth major version of its native development kit (NDK).  For those who are unaware this allows you to run C/C++ on Android devices and communicate to the Dalvik via the Java Native Interface (JNI).

This release brings us one very cool change for future tablet owners.  You can now build for x86 architecture!  Truly exciting for the next generation of tablets running on Intel chips. (http://gizmodo.com/5551822/intel-oak-trail-is-official-tablet-processors-with-windows-android-and-meego-support).

For developers this release brings a critical new piece of software for the debugging process.  Currently debugging the Java side of things is 100% solved and very easy to use.  However if you need to debug things in native land you are completely sol unless your phone manufacturer left gdb support available on it (or you can root your phone).  The NDK does come with ndk-gdb which is gdb as we all know it.  However some phones like mine (HTC Incredible S) are locked down in some nasty ways that as a side effect broke gdb.  So really all I have to go on is a stack trace which contains all hex addresses and no symbols, only which library the crash occured in.  There are ways to take the stack trace and a decompiled version of your library start to map things out.  This proved cumbersome at best.  Finally Google has released ndk-stack.  This analyzes the stack trace and spits out some meaningful information.

ndk-stack converts this classic stack dump:

I/DEBUG   ( 1272): *** *** *** *** *** *** *** *** *** *** *** *** *** *** *** ***
I/DEBUG   ( 1272): Build fingerprint: 'sasktel_canada_wwe/htc_vivo/vivo/vivo:2.2.1/FRG83D/339842:user/release-keys'
I/DEBUG   ( 1272): pid: 17249, tid: 17273  >>> ca.halsafar.foo <<<
I/DEBUG   ( 1272): signal 11 (SIGSEGV), fault addr 00000000
I/DEBUG   ( 1272):  r0 00000000  r1 80d93b8c  r2 80d94b8c  r3 00008928
I/DEBUG   ( 1272):  r4 80d7d2b0  r5 000168d8  r6 80d94b8c  r7 80d93ab0
I/DEBUG   ( 1272):  r8 00008928  r9 44224f40  10 80d7d378  fp 49e94ed8
I/DEBUG   ( 1272):  ip 00000000  sp 49e94b28  lr 80d2a488  pc 80d516dc  cpsr a0000010
I/DEBUG   ( 1272):  d0  6843646565707361  d1  2f76654464696f6e
I/DEBUG   ( 1272):  d2  65747461626d6167  d3  2e2e2f2f696e6a65
I/DEBUG   ( 1272):  d4  626d616762696c2f  d5  6372732f65747461
I/DEBUG   ( 1272):  d6  2e79726f6d656d2f  d7  293935343a707063
I/DEBUG   ( 1272):  d8  4397000041f00000  d9  43e0000000000000
I/DEBUG   ( 1272):  d10 41dfffffffc00000  d11 c3e0000000000000
I/DEBUG   ( 1272):  d12 0000000000000000  d13 0000000000000000
I/DEBUG   ( 1272):  d14 0000000000000000  d15 0000000000000000
I/DEBUG   ( 1272):  d16 3fe40fd000000000  d17 3e23b592a81fa000
I/DEBUG   ( 1272):  d18 3fe0000000000000  d19 3fe00000013b592b
I/DEBUG   ( 1272):  d20 003f003f003f003f  d21 003f003f003f003f
I/DEBUG   ( 1272):  d22 0006000300070002  d23 0006000300070002
I/DEBUG   ( 1272):  d24 0101010101010101  d25 0000000000000000
I/DEBUG   ( 1272):  d26 0000000000000000  d27 0000000000000000
I/DEBUG   ( 1272):  d28 0000000000000000  d29 0000000000000000
I/DEBUG   ( 1272):  d30 001f001f001f001f  d31 001f001f001f001f
I/DEBUG   ( 1272):  scr 20000012
I/DEBUG   ( 1272):
I/DEBUG   ( 1272):          #00  pc 000516dc  /data/data/ca.halsafar.foo/lib/libfoo.so
I/DEBUG   ( 1272):          #01  pc 000171f4  /system/lib/libdvm.so
I/DEBUG   ( 1272):          #02  pc 00045a78  /system/lib/libdvm.so
I/DEBUG   ( 1272):          #03  pc 0004b25a  /system/lib/libdvm.so
I/DEBUG   ( 1272):          #04  pc 0001c160  /system/lib/libdvm.so
I/DEBUG   ( 1272):          #05  pc 00022ed4  /system/lib/libdvm.so
I/DEBUG   ( 1272):          #06  pc 00021d70  /system/lib/libdvm.so
I/DEBUG   ( 1272):          #07  pc 0005d860  /system/lib/libdvm.so
I/DEBUG   ( 1272):          #08  pc 0005da90  /system/lib/libdvm.so
I/DEBUG   ( 1272):          #09  pc 000507cc  /system/lib/libdvm.so
I/DEBUG   ( 1272):          #10  pc 00011014  /system/lib/libc.so
I/DEBUG   ( 1272):          #11  pc 00010b00  /system/lib/libc.so
I/DEBUG   ( 1272):
I/DEBUG   ( 1272): code around pc:
I/DEBUG   ( 1272): 80d516bc a0600002 b3a00000 e28550d8 e7840005
I/DEBUG   ( 1272): 80d516cc e28710dc e1a02006 e1a03008 e1a0000c
I/DEBUG   ( 1272): 80d516dc e59cc000 e1a0e00f e59cf00c e59f1054
I/DEBUG   ( 1272): 80d516ec e1a03080 e59f0050 e3a02b5a e282c0c8
I/DEBUG   ( 1272): 80d516fc e784300c e08f2000 e08f1001 e3a00003
I/DEBUG   ( 1272):
I/DEBUG   ( 1272): code around lr:
I/DEBUG   ( 1272): 80d2a468 e1a01082 ebffffb9 e1a06000 e5940000
I/DEBUG   ( 1272): 80d2a478 e3a01a05 e2813030 e7901003 eb0009e3
I/DEBUG   ( 1272): 80d2a488 e3560000 a04060c6 e5850000 e1a00006
I/DEBUG   ( 1272): 80d2a498 e8bd8070 e92d4010 e1a04000 e5900000
I/DEBUG   ( 1272): 80d2a4a8 e3500000 0a000002 eb0020df e5940000
I/DEBUG   ( 1272):
I/DEBUG   ( 1272): stack:
I/DEBUG   ( 1272):     49e94ae8  80d93ab0
I/DEBUG   ( 1272):     49e94aec  80d7d378  /data/data/ca.halsafar.foo/lib/libfoo.so
I/DEBUG   ( 1272):     49e94af0  44224f40
I/DEBUG   ( 1272):     49e94af4  80d7d378  /data/data/ca.halsafar.foo/lib/libfoo.so
I/DEBUG   ( 1272):     49e94af8  49e94ed8
I/DEBUG   ( 1272):     49e94afc  80d33d78  /data/data/ca.halsafar.foo/lib/libfoo.so
I/DEBUG   ( 1272):     49e94b00  0026a148  [heap]
I/DEBUG   ( 1272):     49e94b04  80d93b84
I/DEBUG   ( 1272):     49e94b08  80db9088
I/DEBUG   ( 1272):     49e94b0c  80d93b84
I/DEBUG   ( 1272):     49e94b10  ffffffff
I/DEBUG   ( 1272):     49e94b14  80d93ab0
I/DEBUG   ( 1272):     49e94b18  80d7d2b0  /data/data/ca.halsafar.foo/lib/libfoo.so
I/DEBUG   ( 1272):     49e94b1c  000168d8  [heap]
I/DEBUG   ( 1272):     49e94b20  df002777
I/DEBUG   ( 1272):     49e94b24  e3a070ad
I/DEBUG   ( 1272): #00 49e94b28  80d93b84
I/DEBUG   ( 1272):     49e94b2c  00000000
I/DEBUG   ( 1272):     49e94b30  00300298  [heap]
I/DEBUG   ( 1272):     49e94b34  49e94b70
I/DEBUG   ( 1272):     49e94b38  00000000
I/DEBUG   ( 1272):     49e94b3c  479790a0  /dev/ashmem/mspace/dalvik-heap/2 (deleted)
I/DEBUG   ( 1272):     49e94b40  44224f48
I/DEBUG   ( 1272):     49e94b44  49e94b50
I/DEBUG   ( 1272):     49e94b48  44224f2c
I/DEBUG   ( 1272):     49e94b4c  80a171f8  /system/lib/libdvm.so
I/DEBUG   ( 1272): #01 49e94b50  0027e3f0  [heap]
I/DEBUG   ( 1272):     49e94b54  41aaecec  /dev/ashmem/dalvik-LinearAlloc (deleted)
I/DEBUG   ( 1272):     49e94b58  479790a0  /dev/ashmem/mspace/dalvik-heap/2 (deleted)
I/DEBUG   ( 1272):     49e94b5c  44224f40
I/DEBUG   ( 1272):     49e94b60  80a173c0  /system/lib/libdvm.so
I/DEBUG   ( 1272):     49e94b64  0027e3f0  [heap]
I/DEBUG   ( 1272):     49e94b68  49e94b70
I/DEBUG   ( 1272):     49e94b6c  80a45a7b  /system/lib/libdvm.so

Into this:

Stack frame #00  pc 000516dc  /data/data/ca.halsafar.foo/lib/libfoo.so: Routine step in /home/halsafar/androidDev/foo/jni//Application.cpp:469
Stack frame #01  pc 000171f4  /system/lib/libdvm.so
Stack frame #02  pc 00045a78  /system/lib/libdvm.so
Stack frame #03  pc 0004b25a  /system/lib/libdvm.so
Stack frame #04  pc 0001c160  /system/lib/libdvm.so
Stack frame #05  pc 00022ed4  /system/lib/libdvm.so
Stack frame #06  pc 00021d70  /system/lib/libdvm.so
Stack frame #07  pc 0005d860  /system/lib/libdvm.so
Stack frame #08  pc 0005da90  /system/lib/libdvm.so
Stack frame #09  pc 000507cc  /system/lib/libdvm.so
Stack frame #10  pc 00011014  /system/lib/libc.so
Stack frame #11  pc 00010b00  /system/lib/libc.so

I have not posted any information on the mobile side so far. Consider this a first step into getting into the habit of posting. The plan is to bring all the Android ports inline with the the PS3 ones.

Halsafar