User:Geyser/CharacterTutorialMixamoToUE4

I (geyser) will temporarily use this page for a sorta tutorial describing how I am making a UE4-compatible Konoko using the head from Oni's "HD Konoko" mod by Leucha&SeverED and the body of a bikini-clad Lara Croft found on CadNav.

The important points when importing a character for use with UE4_Mannequin_Skeleton (without any custom settings) are as follows.

• For locomotion, UE4 will be using blueprints designed for the Mannequin, so unless we tweak the lateral speed and height tracks in those blueprints (or implement some advanced locomotion scheme), the imported character needs to have its lower body proportions more or less consistent with the Mannequin; the actual size of the character can be adjusted separately after the import.
• Characters that have an inconsistent lower body size will appear to either sink or hover ingame, and the kinematics of the feet will not match the ground speed during locomotion; practically, we need to make sure that the height of the pelvis to the ground (or rather some typical length of the legs, measured from the origin of the pelvis to the sole of the feet) is roughly the same as for the Mannequin.
• The rest pose of the character can be anything (for example, a T-Pose consistent with Mixamo), not necessarily the A-Pose used by the mannequin itself; that is because - at least for characters using the UE4_Mannequin_Skeleton - UE4 does not animate in relation to the rest pose, but in terms of absolute rotations with respect to the root.
• Because the engine is using absolute rotations, built-in animations (and IK) will look wrong on the imported character, unless the internal layout of the bones is very close to that of the Mannequin; for example, the clavicle bones start at the front of the sternum, not next to the spine, and hand bones are closer to the back of the hand than to the palm; the spine layout is important too.
• The foot_l and foot_r bones must be pointing straight down towards the heel/sole, not going at an angle towards the toe joint like in the case of Mixamo;

The workflow is roughly as follows:

1. get the model from CadNav and fix its symmetry issues if any;
2. auto-rig the model in Mixamo, fix new symmetry issues if any;
3. constrain the limbs into a "straight" axis-aligned T-Pose;
4. adjust the body proportions to match the (HD) Konoko from Oni;
5. match UE4's A-Pose and set it as a new reference;
6. remove Lara's head, touch-up the neck, attach Konoko's head;
7. adapt the bone orientations for use with the UE4 rig;
8. do a final touch-up of symmetry issues (optional);
9. simplify the geometry of the underwear; add clothes.

The required assets are:

• the HD Konoko (e.g., in the COLLADA format as exported by OniSplit);
• the Lara mesh from CadNav: https://www.cadnav.com/3d-models/model-39309.html
• Mixamo's X-bot in the T-Pose (can be downloaded from the Mixamo site);
• UE4's Mannequin in the A-Pose (can be exported from UE4 in FBX form).

The required tools are:

• Blender
• Mixamo's auto-rigger (online)
• a homemade tool to automatically fix symmetry issues

The workflow is still not ideal (the homemade tools are not yet fully-featured). So this may be updated later, when and if a quicker/cleaner workflow is developed.

Get the model from CadNav and symmetrize it

Getting the model from CadNav is obviously the easy part: there's this bikini-clad Lara Croft model, which has the right amount of detail for our needs, and a fittingly "toonish" texture style. We need the "naked" body for the possibility to add "torn" clothes or have clothing schemes with exposed arms, belly, lower legs etc.

https://www.cadnav.com/3d-models/model-39309.html

Unless you have 3DS Max, you will be downloading the OBJ mesh, which is textured but not rigged (i.e., there is no skeleton and no weight maps assigned to different body parts - only a static geometry). Examining it in, e.g., Blender, we see that the mesh is not entirely symmetric (even if you leave out the hair). There are corresponding quads that are not triangulated the same way on the left and right (mostly the three rings, the bra straps and the eyeballs, but also a few spots on the skin). More importantly there is some inconsistent topology in the crotch and breast areas, and some gaps between the bra and the breasts, so the mesh will have visible holes if looked at in close up from some angles.

The topology can only be fixed carefully by hand (e.g., in Blender), typically through dissolving two or more triangles into a quad or N-gon, then triangulating that polygon differently, or "poking" it and dissolving back into a satisfactory triangulation. The areas where the topology is completely different or incomplete (such as under the bra) require an even more creative approach. The case of the eyeballs is a bit special because they're obviously made of quads, so you can just quadrangulate them and leave them as quads, or not bother with them at all (since we won't be needing them in the end anyway).

Once the topology is symmetric (except for the hair), you can apply an algorithm that detects vertex pairs and makes sure that their locations are perfectly symmetric along X (that way, Mixamo will have a fair chance to produce a symmetric rig). Blender has some algorithms for making a mesh symmetric. You can use them, or make a custom tool of your own, e.g., with the Autodesk FBX SDK (that's what I did).

Auto-rig the model in Mixamo and symmetrize it again

Once you have a symmetric mesh, you can export it to FBX (OBJ also works) and upload to Mixamo. Mixamo will ask you to place critical joints manually (elbows, wrists, knees and "crotch") on a front view of the character. It is not very precise, but do your best to place them somewhere sensible.

Mixamo allows you to download the rigged character either in its original pose or in a T-Pose that is closer to the reference used by Mixamo's animations. Since Lara is in a sort of T-Pose to begin with, and since we wanted to assign her mesh to a skeleton without necessarily deforming it into a new pose at the same time, we can just download the "original pose".

Upon examining the "original pose" downloaded from Mixamo, we see that the skeleton is not entirely symmetric - there are small discrepancies for the Z coordinates of most joints and the "roll" values of the bones. The weight maps are not entirely symmetric either. So (if you want to be thorough about it) you are back to carefully "symmetrizing" the skeleton (it is only a few values, so it can be done by hand) and the weight maps (with a custom automatic tool, e.g., using the FBX SDK). Some joints will also need to be relocated because Mixamo gets them wrong (e.g., the fingers and the shoulders), and that demands some patience and creativity, though not as much as fixing the geometry.

As a final touch, since we will be adapting the character to the UE4 rig, we should rotate the feet inwards a bit, and make sure that the LeftFoot and RightFoot bones point straight down. We also want to put the model into an "Empty" scaled by 0.01 (to accommodate UE4's units, i.e., centimeters). If we do that, we notice that Lara is 7.52m tall, so we scale her down 4 times, getting a more realistic 188 cm.

After you do all the above, you should have something like this: http://geyser.oni2.net/UE4/LaraTutorial/LaraAfterStep2.blend (the dummy leaf bones added to the toes, fingertips and head top are optional - if you have those, then the actual leaf bones will roundtrip better through FBX; otherwise their length and roll value may change arbitrarily - possibly a bug of Blender)

Constrain the rotation to the straight T-Pose

The Konoko from Oni (and, to a lesser extent, Mixamo's own X-bot) come with a very straightened-up T-Pose, where all the bones are aligned with world axes (so, very much unlike Lara, whose T-Pose is a bit casual, with non-vertical legs, curled fingers etc). To adjust Lara's size and proportions to Konoko's, we will need to put Lara into a similar "straight" T-Pose first. In Blender, this is done by adding "copy rotation" constraints to a skeleton's bones.

Set up some Empty objects that have the appropriate axis-aligned orientations for the appropriate subgroups of the skeleton: one for the spine (Y up, X left), one for the legs (Y down, X right), one for the left arm (Y left, X back), one for the right arm (Y right, X forward), one for the left thumb (Y left, X up) and one for the right thumb (Y right, X down). It should look like this: http://geyser.oni2.net/UE4/LaraTutorial/LaraStep3axes.blend

Then go into Pose mode for Lara's armature, select a bone, then in the "Properties" editor panel choose "Bone constraints". Click "Add Bone Constraint", select "Transform/Copy Rotation" from the dropdown, then in the "Target" field pick the Empty with the appropriate orientation. To speed up for the remaining bones of the same orientation group, you can select all the remaining bones of that group while keeping the first bone active (this can be done with the Ctrl+LMB lasso tool), then go to Pose/Constraints and click "Copy constraints to selected bones". After you're done straightening Lara, you should have something like this: http://geyser.oni2.net/UE4/LaraTutorial/LaraStep3straight.blend

You may have noticed that the spine of Lara's casual pose was actually mostly straight to begin with: it just leans backwards a bit and then bends forwards again starting at the neck. So mostly we are just straightening the leg chains, the shoulder-arm chains, and the fingers.

Match the size and proportions to Konoko

Now we import HD Konoko, for example from HERE and use her as a template for Lara's proportions. The first thing to do is "Apply rotation" for Konoko's pelvis (so that we can move her around intuitively) and then shift it upwards by about 0.89, so that the soles of Lara's feet (in the straightened T-Pose) coincide with the inside of Konoko's boots. Conveniently, the non-reflective parts of Konoko's outfit appear transparent in Material mode, so Lara's feet inside Konoko's boots are clearly visible. You can shift Konoko forward by 0.05 or so as well.

Now we rotate Konoko's arms by 90° (getting her into the same T-Pose as Lara), and set the scale of Lara's armature to about 0.86, so that her arms are vertically aligned with Konoko's. This is what you get after you do this basic superposition, adjust Lara's scale to 0.86 and apply it: http://geyser.oni2.net/UE4/LaraTutorial/LaraStep4match1.blend

We notice that Lara's body cannot be fit under Konoko's armor just by moving the two characters around, but it can be managed with some extra rotations of the spine segments. The different layout of the skeleton is also clearly visible: Konoko's spine goes through the center of the neck cylinder, whereas Lara's is further towards the back, where the actual spinal column would be. These observation may become important later if we want to fine-tune the character's rest pose, and more specifically the neutral shape of the spine that works best for all of Oni's animations (movement, combat, etc). For now, though, we are mainly concerned with the overall length of the arms and the relative length of their segments. For Konoko's armor to work on the new character, we clearly need to move Lara's shoulders outwards, make the shoulder bones longer, and bring the elbow closer to the hand (i.e., make the upper arm longer and the forearm shorter).

Since we will be lengthening and shortening some bones by changing their scale, we need to make sure that the scale of the hand and fingers is not affected by these operations, so we select the hand bones and uncheck "Inherit scale", first (same for the forearms and arms if we want to manipulate the length of all the segments individually). The most important changes to the pose are:

• shoulder bones moved outward by 0.23, then scaled lengthwise by 1.25;
• upper arm bones scaled lengthwise by 1.375; lower arm bones by 0.8;
• on second thought, scale the shoulder bones by 1.25 along all axes, to add some muscle;
• and scale the upper arm bones by 1.125 along X and Z, too, otherwise the arms look thin.

After all these arm adjustments are done, you should get something like this: http://geyser.oni2.net/UE4/LaraTutorial/LaraStep4match2.blend

Apart from Lara's shoulders being a bit too far back and her fingers being a bit short compared to Konoko's, this looks satisfactory.

Matching the UE4 mannequin's A-pose

We now want to put Lara in the exact same pose as the UE4 mannequin, i.e., with the exact same global orientations of all the bones (except maybe the clavicles and the spine). For that we will need a version of the UE4 mannequin where the bone axes have been redefined by analogy with the Mixamo rig, so that they look sensible when manipulated in Blender. Here is an FBX I made: http://geyser.oni2.net/UE4/LaraTutorial/MannequinToMixamo.fbx

The first thing to do is remove the rotation constraints of the "straight T-Pose" that we used in the previous step. You do that by selecting Lara's armature, then selecting all bones, and doing "Pose/Constraints/Clear pose constraints". After you do that, hide the HD Konoko, and import the mannequin's FBX, you should have this: http://geyser.oni2.net/UE4/LaraTutorial/LaraStep5initial.blend (I renamed the mannequin's armature to "_root" for convenience in the next step)

Now, select Lara's armature, pick any bone, and create a "Copy rotation" constraint for it, with the corresponding bone of the mannequin's armature as the Target. The procedure is the same as for the straightened T-Pose we did earlier, except that the rotation is taken from the bone of an armature rather than from a top-level "Axes*" object. So first we pick "_root" as the target, and then look up the correct bone under the "Bone:" field that appears when you select "_root" as "Target:". Autocompletion works, so start typing the name of the bone and it will narrow down the list. Do not constrain the spine and the shoulder/clavicle bones, because they look fine as they are, and their default layout in their respective rigs is too different. Once you are done constraining Lara's pose, you should have something like this: http://geyser.oni2.net/UE4/LaraTutorial/LaraStep5match1.blend

Double-check the constraints (since this is manual work, it is easy to accidentally assign the wrong target bone), and examine Lara's pose with the armatures hidden - is it really the same as the mannequin's A-Pose? Upon close examination, you will see that the final segments of the fingers look weird: the mannequin's mesh has straight fingers, but in our matched pose they are not. That's because the mannequin's rig is imperfect and, specifically, the final segments of the fingers are not aligned with the actual fingertips of the mesh. Our concern, though, is that the mesh of the A-Posed Lara should look the same as the A-posed mannequin, so we should straighten up her fingers for the reference pose. We do that by constraining the fingertip bones not to their exact counterparts, but to the previous segments. This is what it looks like when corrected (with the armatures and mannequin mesh hidden): http://geyser.oni2.net/UE4/LaraTutorial/LaraStep5match2.blend

Finally, we need to apply the constrained A-Pose as a new rest pose. Although intuitively it is a single operation, it is done in several steps. First, select Lara's mesh, and in the toolbox pick the Modifiers panel (with a wrench icon), then click Apply (this will deform Lara's mesh according to the rotations, scales and translations we set up for the armature, and the armature will no longer affect the mesh). Next, select the armature, go to Pose mode, select all of the bones, and do "Pose/Apply/Apply pose as rest pose". This will set the default transforms of the bones to the current posed ones, so that constraints will no longer be needed, and the information about the original shape of the armature will be lost. Last of all, you select Lara's mesh again, pick the Modifiers panel, and click "Add Modifier", pick Armature, and select Lara's armature (which was disconnected when we applied the modifier to the mesh). As for the constraints, we can now select Lara's armature, go to Pose mode, and do "Pose/Constraints/Clear pose constraints". Here is how it looks after all the above is done: http://geyser.oni2.net/UE4/LaraTutorial/LaraStep5newrest.blend

Replacing the head

Lara is now nearly ready for UE4, and it is finally time to remove Lara's head and attach Konoko's. The easiest way to do this is to disconnect Lara's head along an edge loop at the top of the neck, and then delete everything that's not connected to the body. After you do that, and unhide Konoko's head, you get this: http://geyser.oni2.net/UE4/LaraTutorial/LaraStep6severed1.blend (pun half intended)

We see that Lara's neck is uncannily thin and short - that's because Lara's character was originally much taller and less muscular. The best way to fix this is by scaling the neck bone, without letting the scale affect the head bone. As a reference, the cylinder/cone that makes up the original Konoko's neck is about 8.5574cm across and 8.8701cm front-to-back at the thickest point (base of the neck), whereas for the current A-posed Lara we have roughly 6.962cm across. A scale of 1.6 lengthwise (Y) and sidewise (X) for Lara's neck seems to work well (never mind if the bra strap starts looking weird, we are planning to fix it later anyway): http://geyser.oni2.net/UE4/LaraTutorial/LaraStep6severed2.blend

Before we work on the mesh further, we need to apply that extra 1.6 scale to the neck, with another round of "apply modifier to mesh"/"apply transforms to armature's bones"/"add the armature modifier back to the mesh" like we did above when setting the new rest pose. After you've applied the neck scale, you should have this: http://geyser.oni2.net/UE4/LaraTutorial/LaraStep6severed3.blend

The vertices at the top of the neck tube now need to be brought closer to Konoko's head. We don't need to connect the two geometries exactly, instead we just want to make the tube nice and smooth, and not too wide, so that it enters the head mesh from below, without too much clipping. We also want to manipulate the vertices in pairs, so as not to break the symmetry. After you're done with that, it should look like this: http://geyser.oni2.net/UE4/LaraTutorial/LaraStep6stitched.blend

Finally we merge the two meshes (Lara's body and Konoko's head). There is a preliminary task that needs to be taken care of first: namely, the UV map for Konoko's head (originally "head_texcoord") needs to be renamed to the same (default) name as used by Lara ("UVMap"). Only then will the materials of the two meshes be merged properly. Once the UV map of the head mesh is renamed, select the head mesh, then Shift-click the body, and do Ctrl+J to "join". Delete the rest of Konoko's hierarchy since we don't need it any more, and you should get this: http://geyser.oni2.net/UE4/LaraTutorial/LaraStep6united.blend

At this point the head is not yet weighted to the skeleton. So we complete the process by selecting the head mesh (select any part of it, then grow the selection) and assigning it to the vertex group mixamorig:Head. The resulting model, with the rest of Konoko deleted, is here: http://geyser.oni2.net/UE4/LaraTutorial/LaraStep6weighted.blend

Reorienting the bone axes according to UE4's conventions

The model so far can have some imperfections that we may want to fix (leftover symmetry issues, mesh layout that was messed up a bit when we scaled the bones, too much detail for the underwear, chest size, weight mapping of the breasts) but for now we just want to export this model so that it can be used in UE4. It is already in the exact same A-pose as the UE4 mannequin, but the bones' orientations do not agree with UE4's conventions. (That's perhaps the least intuitive part: the bones of UE4 skeleton do not have Y as their main axis, whereas Blender's bones are always along Y.)

For reference, here is an FBX with the mannequin's bones as they are supposed to be: http://geyser.oni2.net/UE4/LaraTutorial/SK_Mannequin.FBX

Unless you have an automated way for rotating bones exactly by 90° in exactly the right direction depending on the bone, this needs to be done manually. There are at least two ways to do this in Blender.

FBX roundtrip and reparenting

If you export the character to FBX and then reimport with different axis settings under the armature tab, you will end up with a skeleton with bones pointing, e.g., along the +X axis of the original bones. If you import enough of such reoriented skeletons, you can assemble the correct UE4-ready skeleton from them by joining the skeletons, reparenting bones around, and deleting unneeded bones as you go. That's a rather tedious and confusing process, but it gets the job done.

The default armature settings for FBX import are primary axis +Y and secondary axis +X. To match UE4, the skeleton subgroups need to be reimported as follows.

Spine: primary axis -Z, secondary axis +Y

Left arm (except hand and fingers): primary axis +X, secondary +Y

Right arm (except hand and fingers): primary axis -X, secondary +Y

Left leg (except toe), right toe, right hand and fingers: primary axis -Z, secondary axis -Y

Right leg (except toe), left toe, left hand and fingers: primary axis Z, secondary axis Y

In-place bone rotation

Blender's implementation of bones is not the most intuitive, but it turns out there is a way to rotate bones in Edit mode. This is allowed by selecting "Normal" as the rotation mode (as opposed to "Gimbal", "Global" and "Local") and setting "Individual origins" as "Pivot center for rotation/scaling". Only sideways rotations of the bone can be done that way, though; rotation around the bone's own axis must be done by incrementing the bone's "roll" value. Also, before you rotate a bone, you must disconnect any child bones, otherwise the rotation of the bone will affect them.

The rotations that need to be applied to the bones (in Edit mode, disconnected, in "Normal" rotation mode, with "Individual origins" as pivot center) are as follows:

Spine: -90° around X, then set roll to -90°

Left arm (except hand and fingers): -90° around Z, then add 180° to roll

Right arm (except hand and fingers): -90° around Z.

Left leg (except toe), right toe, right hand and fingers: -90 around X, then add 90° to roll

Right leg (except toe), left toe, left hand and fingers: 90° around X, then add 90° to roll

Finalization

Whichever way you chose to rotate the bones around, you should have now something like this: http://geyser.oni2.net/UE4/LaraTutorial/LaraStep7ortho.blend

After renaming all the bones in consistence with UE4's rig it looks like this: http://geyser.oni2.net/UE4/LaraTutorial/LaraStep7renamed.blend

After adding the extra IK bones (a couple of root bones, and some copies of the hand and foot bones), parenting them properly, and removing the leaf bones, it looks like this: http://geyser.oni2.net/UE4/LaraTutorial/LaraStep7tree.blend

Finally we just need to check the transformations of all the object (zero translations/rotations and unit scales for everything, except the top-level empty which must have scale 0.01 for the unit change). It looks like everything is consistent with the SK_Mannequin from UE4's FBX, so we can try and use her in UE4.

The exported FBX is here: http://geyser.oni2.net/UE4/LaraTutorial/LaraStep7export.fbx The same FBX with the textures extracted rather than embedded is here: http://geyser.oni2.net/UE4/LaraTutorial/LaraStep7export.zip