User talk:Neo

Hi Neo,

We (geyser and Alloc) are positively impressed by your recent contributions to the "OBD" knowledge database. Wow.

What kind of activity (professional, Oni-related) has led you to such expertise? What tools do you use to analyse Oni?

OUP could use some expert OBD knowledge right now: we'd like to produce a fully functional release Soon(TM).

Would you consider private dialogue with us (mail, ICQ)? (about TXMP and recurrent data types)

geyser 03:09, 6 March 2007 (CET)

Hi guys,

What's so much to be impressed, I just added some matrices here and there (mostly) ! :)

I've been digging through Oni binary files for some time now (I think it was the summer of 2002 when I opened a dat file in a hex editor out of curiosity and when I saw all this TXMP, PNTA, IDXA, M3GM etc. stuff I thought that maybe I can get something out of this binary format). While I'm not a game developer I have a bit of a passion for 3D stuff so I learned a bit of Open GL and a bit more of Direct 3D so I'm quite familiar with transforms, quaternions, textures etc.

As for used tools: much like anyone else I think. A hex editor, Excel for some things, SQL Server (database) for some others, Visual Studio (while it's file hex editor is quite dull it has a cool memory view in debug mode (you can set the number of displayed columns and the data type to display (integer/float, decimal/hexadecimal, 1/2/4 bytes). Obviously since this was in debug mode I had a program that loaded the DAT file into memory and where I could select a "file" (for various reasons I happened to call them "resources"). In addition I sometimes used a disasembler (home made, don't ask what it is) to stare at some things inside the executable :). Last but not least I have a "viewer" for a bunch of things from Oni (texures, 3D models, animations, environment). Good to test things like what I added to OBAN and OBOA file formats.

As for mail/messenger (MSN or Yahoo) I guess it's OK but I don't feel like posting my mail address in here, it's a public place... any ideas ?

Neo

OK, we won't ask you about your home-made disassembler ^^, but the viewer definitely sounds nice.

Viewers (and exporters/importers) for things other than TXMP is what OUP currently lacks.

Needless to say, we'd like to see those (is it C/C++?) and with any luck integrate them with OUP... ^^

As for contact, our coordinates are available (just click our sigs). My YIM alias is serguei_mechkov.

geyser 02:53, 7 March 2007 (CET)

I only have a few more doubts about the TXMP format. Will experiment a little tomorrow night.

geyser 02:53, 7 March 2007 (CET)

Your homemade tools remind me of someone from the previous Oni Central Forum... Have we met? ^^

geyser 02:53, 7 March 2007 (CET)

Nope, we had not met before. As for viewer I intend to make it available but not right now, I kept doing experiments with it for the last 2 weeks or so and the whole source looks like after a war. The current one is written in C#, I also have one that is written in C++ but it has not been updated in a while so it lacks/gets wrong some things. I'll add you to my YIM list and you can ask for more details there, I'm not familiar with this wiki stuff to write too much... :) Neo

---

I've tried to run your Onibrowser several times, but I've failed. The test system is a laptop with WinXp + SP2, .NET 2.0 and XNA 1.0. When I doubleclick on the exe the harddisk makes only some noise for some seconds. That's it. Nothing else happens. No error message, no window. Nothing. Any idea what's wrong?

Can you add some code to the vtuneapi.dll that draws the trigger volumes (volume color: red, opacity: 10%)?

Ssg 12:40, 22 December 2007 (CET) 1. Hmm, that's unusual. Things to check: - make sure you run it from the local disk and not from a network share - make sure you have the correct xna version installed. The latest OniBrowser uses this [Framework 1.0 Refresh]. Previously there was another 1.0 version (without Refresh in title) but that was removed from download. - update DirectX

If nothing works take a look in the event viewer, maybe there is something about failing to start (go to Control Panel, Administrative Tools, Event Viewer, Application Log).

2. I suppose it is possible but I'm not sure what it takes. Besides, I already did this for OniBrowser so I'll try to post a new version of OniBrowser one of these days.

Neo

The DirectX update was helpful. Now it works. Thank you.

Ssg 14:45, 23 December 2007 (CET)

2 again. Here's an updated OniBrowser that shows trigger volumes, flags and sounds. You need to upgrade to XNA 2.0.

Neo

Does not work. I've installed XNA 2.0 but nothing happened. Again no error message and no window. The strange thing is that the "old" version of OniBrowser doesn't work with XNA 2.0 too. It needs XNA 1.0. So any ideas what's wrong with the current OniBrowser version?

Ssg 15:50, 3 January 2008 (CET)

Oops... I forgot a file. Please redownload

Neo

Doesn't work either. Same result as above. The "old" version has a XPtheme.dll. Maybe that's the problem? (I've copied it to the folder with the current version but no success.)

Ssg 17:36, 3 January 2008 (CET)

I'm out of ideas. The last zip missed Oni.Files.dll so check if you got it now. XPTheme.dll is old, not needed anymore. Try to take a look in the event viewer for an error about Oni.Browser.exe.

Neo

The event viewer gaves me an error message: "Couldn't find Microsoft.VC80.CRT". I've googled a bit and it looks like a general problem for Vuisal C/C++ programs. So please search the net for "VC80.CRT" to solve this problem. (Microsoft MSDN help: http://msdn2.microsoft.com/en-us/library/ms235342(vs.80).aspx )

I've installed the "Microsoft Visual C++ 2005 Redistributable Package (x86)" (http://www.microsoft.com/downloads/details.aspx?FamilyId=32BC1BEE-A3F9-4C13-9C99-220B62A191EE&displaylang=en ) but it didn't help.

Ssg 22:00, 3 January 2008 (CET)

Stupid me, I tend to forget that XNA is actually written in C++ and it needs that. Now the only problem is that you need the SP1 C++ redist. You were close :) .

Neo

Hooray. It works. Thank you. And yes, I was really close. It was a fifty-fifty chance and I chose the wrong. As usual. (Grrrr... Murphy... just you wait!)

I would like to take screenshots as in Oni (screenshot+number.bmp). Do you think you can add such a feature? And can you make your program run in real fullscreen?

Ssg 02:27, 4 January 2008 (CET)

Is there a way to switch the visibility of the flags/sounds/tvs on/off in general and separate?

Ssg 02:35, 4 January 2008 (CET)

oh, a new version - gonna check it out

Kuchumovn 11:31, 4 January 2008 (CET)

• screenshots - I think I can add such a thing but isn't pressing Print Screen key enough?
• fullscreen - ha ha... old versions did that, I think I can restore that functionality easily.
• flag/sounds/tvs visibility - not yet, wanting to do it this weekend

Neo

• screenshots: I would like to create an overview of all TVs. It's not very handy to do that with the Print Screen key. That's why I've asked for a screenshot-shortcut.

Ssg 18:22, 4 January 2008 (CET)

Updated OniBrowser

• fullscreen - F11 key
• screenshot - F12 key
• toggle flags visibility - F5
• toggle sounds visibility - F6
• toggle trigger volumes visibility - F7

Neo

Coooooooool... thank you very much.

Ssg 20:49, 5 January 2008 (CET)

"screenshots: I would like to create an overview of all TVs"

What's the point of creating an overview if we'll edit them?

I mean, TVs are about the first thing we'll hack Very Soon.

geyser 23:23, 5 January 2008 (CET)

Maybe there're people out there who like to create scripts for the original game?

Btw: Very Soon (TM)? Is that the same soon as this? :p

Ssg 02:23, 6 January 2008 (CET)

@ VerySoon(TM): I suppose you haven't tried the Edition?

The COMING SOON thing is more like an inside joke now.

I don't know why I didn't remove the index.php page ^_^

The original TVs are quite unusable for custom logic.

There are partial workarounds, but it's still a pain.

The future belongs to custom-matched TVs and scripts.

But it's OK if you want to do the overview, of course.

Maybe it will help people get familiar with TV setup.

And then maybe they'll start modding BINA, who knows?

geyser 21:27, 6 January 2008 (CET)

---

hi, Neo

while porting your TRAM viewing code i noticed that there is some kind of jitter in model animations (for example look at barabas' rocket animation (level 0) - feet are some kind of jittering)

also you use LERP method in your code

i haven't found it in the internet, but i found SLERP - i guess it is the same

so, do you have any ideas why do animations jitter?

or am i just paranoiac?

Kuchumovn 11:02, 9 January 2008 (CET)

jitter: no idea, I don't see anything wrong with that animation

quaternion:

• lerp = linear interpolation in a 4D space
• slerp = linear interpolation on a sphere

They are not the same mathematically but I didn't notice "visual" differences between using one or the other. Oni uses lerp (slerp needs some trigonometric functions so it will slow down things probably) so I did the same.

Neo

thanks

Kuchumovn 13:06, 9 January 2008 (CET)

about the animations jitter: youtube

i don't really care, just maybe you have encountered something like this

Kuchumovn 22:06, 13 January 2008 (CET)

Hmm... it's like I'm looking at that video ten times and the tenth time I think I'm seeing some jittering when he is landing. As for possible reasons, I don't know. The animation itself can have problems, there maybe a bug in the code (though I have checked it and it looks correct) or all the computations needed to get the final transform matrices combined with floating point (im)precision generate errors.

Neo

seems that these are calfs and arms who cause the jitter

actually i haven't found a 3D math library in java, so i have grabbed different sources from the web - maybe there are some errors that cause the jitter

thanks for your feedback

Kuchumovn 00:37, 15 January 2008 (CET)

Hmm... I looked at your math/animation code and I seems correct. What is the name of the animation from your video anyway?

Neo

the animation in the video is "Muro Combat Crouch Forward" from level 13 - it looks correct in OniBrowser

a lot of animations jitter (heavy kicks and slow punches)

yesterday i've found out that if i disable parent transformation for arms and calfs, jitter doesn't appear

i'm gonna test it today in more details

Kuchumovn 13:39, 15 January 2008 (CET)

i have also just saw that there is a field "FPS"

i placed it into the realTimeFrame formula, and now, i think, the resulting animation is quite appropriate, especially if you are playing a game

so i think we are done - let it be an openGL_driver/video_card issue for now (so that it won't cause nightmares)

Kuchumovn 17:45, 15 January 2008 (CET)

Neo, geyser mentioned that OniSplit can export TXMP out as image files. Can you tell me the command to do so, I would like to try it.

Thanks EdT

Yes, it can though currently it's a bit more complicated than it needs to be. You need first to export a .dat file to .oni files and then export images from those .oni files.

• onisplit -export:TXMP* destination_directory_for_oni_files source_dat_file
• onisplit -extract:png directory_with_oni_files/TXMP* destination_directory_for_image_files

Neo

New OniSplit version that make extracting easier: OniSplit v0.8.8

onisplit -extract:png destination_directory_for_image_files source_dat_file

TGA and DDS are also supported using -extract:tga and -extract:dds.

Neo

Neo, I got an error with the link to the latest OniSplit.

EdT

Oops, I ate a -. Link fixed.

Neo

Got it. I now get error message with OniSplit: (I'm on Mac OSX 10.4.11 using Mono project)

 Reading file /Oni/GameDataFolder/level0_Final.dat
 Exporting to /Oni/TXMPfiles

 System.TypeInitializationException: An exception was thrown by the type initializer for System.Drawing.GDIPlus --->
 System.DllNotFoundException: /Library/Frameworks/Mono.framework/Versions/1.2.5.1/lib/libgdiplus.dylib
 at (wrapper managed-to-native) System.Drawing.GDIPlus:GdiplusStartup (ulong&,System.Drawing.GdiplusStartupInput&,System.Drawing.GdiplusStartupOutput&)
 at System.Drawing.GDIPlus..cctor () [0x00000] --- End of inner exception stack trace ---

 at <0x00000> <unknown method>
 at System.Drawing.Bitmap..ctor (Int32 width, Int32 height, PixelFormat format) [0x00000] 
 at System.Drawing.Bitmap..ctor (Int32 width, Int32 height, PixelFormat format) [0x00000] 
 at (wrapper remoting-invoke-with-check) System.Drawing.Bitmap:.ctor (int,int,System.Drawing.Imaging.PixelFormat)
 at Oni.Files.Imaging.PngFile.Write (System.String filePath) [0x00000] 
 at Oni.Files.Imaging.ImageExporter.Export (Oni.Files.FileManager fileManager, System.String sourceFilePath, System.String outputDirPath, System.String prefix) [0x00000] 
 at Oni.Files.Program.ExportTextures (System.String[] args) [0x00000] 
 at Oni.Files.Program.Main (System.String[] args) [0x00000] 

 Unhandled Exception: System.TypeInitializationException: An exception was thrown by the type initializer for 
 System.Drawing.GDIPlus ---> System.DllNotFoundException: /Library/Frameworks/Mono.framework/Versions/1.2.5.1/lib/libgdiplus.dylib
 at (wrapper managed-to-native) System.Drawing.GDIPlus:GdiplusStartup (ulong&,System.Drawing.GdiplusStartupInput&,System.Drawing.GdiplusStartupOutput&)
 at System.Drawing.GDIPlus..cctor () [0x00000] --- End of inner exception stack trace ---

 at <0x00000> <unknown method>
 at System.Drawing.Bitmap..ctor (Int32 width, Int32 height, PixelFormat format) [0x00000] 
 at System.Drawing.Bitmap..ctor (Int32 width, Int32 height, PixelFormat format) [0x00000] 
 at (wrapper remoting-invoke-with-check) System.Drawing.Bitmap:.ctor (int,int,System.Drawing.Imaging.PixelFormat)
 at Oni.Files.Imaging.PngFile.Write (System.String filePath) [0x00000] 
 at Oni.Files.Imaging.ImageExporter.Export (Oni.Files.FileManager fileManager, System.String sourceFilePath, System.String outputDirPath, System.String prefix) [0x00000] 
 at Oni.Files.Program.ExportTextures (System.String[] args) [0x00000] 
 at Oni.Files.Program.Main (System.String[] args) [0x00000]

EdT

Umm... that's unexpected. I'm not sure what is wrong and if I can fix it. Can you use TGA files (-extract:tga)?

Neo

I tried on another machine Intel Mac OSX 10.5.1 I can extract .png and .tga but image is scrambled, here is one example of TXMPoni_kanji_1.png

EdT

He he... it looks nice :). I forgot that Mac .dat files are different. Here's a fixed version: OniSplit v0.8.9

Neo

Success! I can extract and view .tga versions of the TXMP, but somehow .png causes the terminal to hang. It say exporting, but nothing happens. no error message. I waited several minutes but still nothing happens. (tga extraction is very quick) Very strange.

I was able to add the command to an application I'm making for the 7th Anniversary Edition.

Also, do you have plans to allow importing of TXMP?

I think that would be very useful, especially for Mac users. Since the current programs such as OUP are not Mac compatible. If it could be something like importing TXMPBOSS1talking.tga and then converting it to TXMPBOSS1talking.oni. Then it would be a simple matter of recompiling level0 into the .dat, .raw and .sep files.

Finally, can you tell me the list of commands in OniSplit?

Thanks so much.

EdT

• PNG issue: It seems that Mono's graphics library has some problems on OS X because I can export png files from Mac .dat files using Mono on Windows without problems. I'll see what can I do about it though I don't feel like writing my own png compression code :).
• Importing TXMP: Yes, I'm working on it.
• For a list of commands use the -help option:

OniSplit -help

There are 2 additional options for -extract that are not listed. They are intended to extract M3GM and other 3D stuff to Wavefront OBJ (-extract:obj) and Collada (-extract:dae) file formats. They are not complete and they may have bugs.

Neo

Don't worry about the .png issue, Mac users can extract as .tga and edit the files. I'm looking forward to when we can import TXMP.

Thanks for your great work.

EdT

I tried out the extract obj on a Mac and got this:

mono onisplit.exe -extract:obj ../../OBJfiles ../../GameDataFolder/level0_Final.dat Reading file /Oni/GameDataFolder/level0_Final.dat Exporting to /Oni/OBJfiles Illegal instruction

Also, can you please keep me posted whenever you update OniSplit, then I can test it on the Mac for you.

EdT

New OniSplit version: OniSplit v0.8.10

I did not look into that "illegal instruction" thing yet but I added a texture import option:

onisplit -create:txmp destination_directory [options] source_image_file

where options can be

• -nouwrap - set the "U wrapping disabled" flag
• -nowwrap - set the "V wrapping disabled" flag
• -format:bgr32, -format:bgra32, -format:bgr555, -format:bgra5551, -format:bgra4444, -format:dxt1 - the texture format to generate
• -genmipmaps - generate mipmaps if they are not already present in the source file and if the source file is not dxt1 compressed

Example:

onisplit -create:txmp imported -format:dxt1 -genmipmaps tga\concrete.tga

Notes:

• I haven't seen any TXMP that use format bgra32 but it should work with Oni. Did not tested it yet.
• DXT1 compression is kind of lousy (lousy not lossy :)).
• source_image_file can be dds or tga. "Color indexed" and "black and white" tga files are not supported currently. Other image file formats (png, jpg, bmp, tif) may work depending on Mono/.NET support for them.
• onisplit will automatically prepend TXMP to the output file name if the source file name does not start with TXMP already.

Neo

I will test it out as soon as I can and will report back to you. Thanks!

EdT

I forgot to ask you about that "illegal instruction" error. Can you post the full error mesage here?

Neo

First test with OniSplit .8.10, I get this error message

 mono OniSplit.exe -create:txmp  ../../ModTXMP ../../ModTXMP/TXMPBOSS1nametag.tga

 System.IndexOutOfRangeException: Array index is out of range.
 at Oni.Files.Imaging.Color.WriteBgra32 (Color color, System.Byte[] data, Int32 index) [0x00000] 
 at Oni.Files.Imaging.Surface.SetPixel (Int32 x, Int32 y, Color color) [0x00000] 
 at Oni.Files.Imaging.TgaReader.LoadRleTrueColor (Oni.Files.Imaging.TgaHeader header, System.IO.BinaryReader reader) [0x00000] 
 at Oni.Files.Imaging.TgaReader.Read (System.String filePath) [0x00000] 
 at Oni.Files.Imaging.TextureImporter.LoadImage (System.String filePath) [0x00000] 
 at Oni.Files.Imaging.TextureImporter.Import (System.String filePath, System.String outputDirPath) [0x00000] 
 at Oni.Files.Program.CreateTexture (System.String[] args) [0x00000] 
 at Oni.Files.Program.Main (System.String[] args) [0x00000]

Even when I add the options such as -format:dxt1 I still get the same message.

Now for the .obj:

 mono onisplit.exe -extract:obj ../../OBJfiles ../../GameDataFolder/level0_Final.dat

 Reading file /Oni/GameDataFolder/level0_Final.dat
 Exporting 26 instances to /Oni/OBJfiles
 Illegal instruction

The terminal just ends with the words "Illegal instruction"

EdT

TGA fixed: OniSplit v0.8.11 Stupid me, I did y-- instead of y++ :).

Neo

It sort of works. The TXMP comes out reversed horizontally and vertically. It seems that the new image has to be the same size as original, otherwise I get a gray TXMP.

Also, if possible, can you include a command that will convert a single TMPX .oni file into a .tga? Currently, I have to create new TXMP.oni file then put it in the level0 folder, then recompile level0 and then either play the game or again extract TXMP to see the result.

Thanks

EdT

• As far as I know it can convert a single .oni file to .tga. Did you tried?
• Reversed both horizontally and vertically? I think it's only vertically. I'll fix that.
• Gray TXMP: I'm not sure what's happening. One thing to check is if the width and height of image are a power of 2. Non power of 2 texture dimensions may result in problems either in OniSplit or Oni.

Neo

Maybe I responded too quickly the last time. :-)

You're right, I tried this command and it works: mono onisplit.exe -extract:tga ../../ModTXMP/test ../../ModTXMP/TXMPBOSS2face.oni

Sorry, it was only flipped vertically when viewed in the game.

Finally, I tried again with 256x256 to replace 64x64 and this time it worked. Not sure why I got the gray TXMP last time.

Being able to create TXMP in OniSplit is great! Thank you!

EdT

I think I fixed the flipping: OniSplit v0.8.12

Neo

It works. See this screenshot: http://oni.bungie.org/community/forum/attachment.php?item=209

Thanks,

EdT

Another question, is it possible to create TXMPs from a directory of image files? Like a batch process of the image files, or should I work on a script to do the batch processing?

EdT

Yes, you can specify multiple input files on the command line:

 onisplit -create:txmp output_dir image_dir\ex1.tga image_dir\ex2.tga

or you can use wildcards:

 onisplit -create:txmp output_dir image_dir\*.tga

Neo

Awesome, the wildcard version is what I needed!.

Many thanks.

EdT

I found a bug when extracting TXMP with Alpha channels. See the following screenshots:

I did not modify the files at all, I first extracted them as .tga files, then created new .oni files from the extracted .tgas.

Here are the extracted face and chestpack files:

http://edt.oni2.net/OniSplit/TXMPFKS_face.tga

http://edt.oni2.net/OniSplit/TXMPFKS_chestpack.tga

I hope you can fix this bug.

Thanks,

EdT

That's not a bug, alpha channel is exported correctly. What happens there is that those TXMPs have an environment map and currenly OniSplit has no option to add a environment map when creating the TXMP. The only solution I see is to add an option like "-env:env_txmp_name" to allow you to specify one. Of course the problem remains how do you know when you need an env map and how that option works with something like batch processing so opinions are welcome.

Neo

Thank you for explaining the situation to me.

Please add the option for the environment map.

I can think of a couple of ways to handle when an env map is needed. If Onisplit when first extracting the TXMP can flag the file having an env map, perhaps making the name TXMP7... then when creating the TXMP, the user or program can run the command with the env map option.

The other way is trial and error, if the TXMP comes out wrong, like above, then the user needs to use the env map option.

EdT

New OniSplit version: OniSplit v0.8.13.

Envmap problem: for now I added an "-envmap" option through which the name of the env map texture can be specified:

onisplit -create:txmp dest_dir -genmipmaps -envmap:envksface TXMPIteration%2FKS_face.tga

I'll try to figure out a better way to do it. Storing texture options in a tga file seems possible but I'm not sure if editing tools preserve that information.

The envmap name can have the TXMP prefix but it is not required, it is added automatically if missing.

Export/Import for Wavefront OBJ files:

• Named M3GMs and (unfortunately unnamed) M3GM contained in ONWC files can be exported to obj file format:

 onisplit -extract:obj dest_dir M3GMnotfound.oni
 onisplit -extract:obj dest_dir ONWCw1_tap.oni

Or for greedy :) people:

 onisplit -extract:obj dest_dir level0_Final.dat

• Importing obj files:

 onisplit -create:m3gm dest_dir TCTF_Shotgun.obj

For importing obj files the obj file must only contain triangles so when exporting a model from a 3D design tool it needs to be triangulated. If the obj file uses more than one texture then only one will be taken into consideration. If the obj file does not contain a texture or someone wants to use a different one then the texture name to be used can be specified with -tex option:

onisplit -create:m3gm dest_dir -tex:TCTF_Shotgun TCTF_Shotgun.obj

Just like in -envmap option case the texture name can start with TXMP but it is not required.

Note that in both -envmap and -tex cases a .oni file is not created for the specified texture. The .oni file for that texture must be create separatly.

Neo

I have a question, how and where would I find the name of the env map that is used for a TXMP? For example, you stated "-envmap:envksface TXMPIteration%2FKS_face.tga"

Finally, could I post on Oni Central Forum, whenever, a new OniSplit is released along with the information you provide?

Thank you.

EdT

Hmm... you have a good question :). One way that comes to mind is a hidden option of OniSplit:

 onisplit -dump TXMPIteration001%2FKS_face.oni

This will print something like:

TXMPIteration001/KS_face:TXMPInstance {
       TextureName:"Iteration001/KS_face"
       Flags:4609
       Width:128
       Height:128
       Format:0
       Animation:null
       EnvMap:"TXMPenvksface"
       DataOffset:32
}

As for posting on Oni Central Forum I have nothing against it. Though I should do it myself probably but I don't have an account in there and I have yet to decide if I'll create one.

Neo

---

The easiest way to find the envmap texture is to open the .oni file in binary.

At the end of the "dat" part (before the "raw" bitmap data) will be the names.

For a TXMP without an envmap, there will be only the name of the TXMP itself.

For an envmapped TXMP, there will be an extra name - that of the envmap TXMP.

geyser 22:35, 4 February 2008 (CET)

This is interesting, I did the -dump as you stated and this is what I got:

TXMPIteration001/KS_face:TXMPInstance { TextureName:"Iteration001/KS_face" Flags:4097 Width:128 Height:128 Format:8 Animation:null EnvMap:null DataOffset:32 }

In the Mac version, I don't have a EnvMap. I also looked at the file with a hex editor and there was no envmap name.

EdT