Time is flying by, I am not sure what happened over the last few months but Infinite-Realities has been crazy busy!

IR’s mobile Capture team just got back from a Face Capture Shoot in India working on a brand new super cool Bollywood film, I was over in LA for a few weeks working on a top secret dynamic capture system, working as a consultant, plus some recent work with a British actor for a very cool Develpment studio in Europe, with bits and pieces for other well known VFX and Games studios as well. All under NDA : (

In amongst all that I have just found time to upgrade my current Capture system from 16 mixed Cameras to 42 dedicated Canon 550 DSLR’s. These Cameras are incredible for “Volumetric Photogrammetry”.

Finding spare time in amongst contract work is hard but I have been able to setup some 360 Full Body tests (coming soon) as well as a dedicated 360 Degree Volumetric Head Capture system. As far as I know there is nothing quite like it in the world (at the moment) and the results are just a joy to work with.

Here is a glimpse at some very early RAW test results, it’s a homage to EA. I like to call my system “FACE Backwards™”

…..I kid! I kid! It’s not called anything. It’s a custom capture system that allows you to record multiple static images, at a single moment in time, synchronized. No special software or trigger hardware.

You can capture a face mid expression, with very minimal motion blur. Meaning you can get your actor to express extreme emotion in any facial pose and record that data, as 3D geometry.
A full FACS session, up to about 90 expressions can be captured between 30 minutes to 1 hour.

Some early results, no colour processing:

(High Details, 5.5 Million Polygons. 360 Degrees)

(Plenty of Pore details, it even Captures Eyelashes, 5.5 Million Polygons. 180 Degrees)

(Mid Level Capture, 750,000 Polygons. 180 Degrees)

(High Details, 5.5 Million Polygons. 360 Degrees)

These are RAW, high resolution Captures.

Last week I got to try out this Capture system with a very talented British Actor, he was able to turn on and off emotions in a split second. Amazing skillset. Great for FACS Capture, as well as custom emotions. Rage, Frustration, Smiling, Anger, Sadness etc

Using 10 DSLR’s recording Synchronized footage, the idea is to process exported frames in Agisoft with the help from Alexey Pasumansky. Alexey is developing a script to batch process the data, to then stream back in Softimage with colour information to track with.

The only draw back with using DSLR’s is the awful built in compression! it really affects 3D Stereo Extraction but I have some other Cameras on order to overcome this issue. For now the DSLR’s are pretty good. It means anyone can build a 4D Capture array for under £5k, unlike most commercial systems that can vary from £50k to £150k.

Click here to view the embedded video.

4D Mesh Data Processed in Agisoft Photoscan

Click here to view the embedded video.

Alexey kindly  processed some of the data in Agisoft Photoscan Pro, we wanted to test the pipeline in Agisoft to prove the 4D system could work and I think the results are pretty good. Unlike most commercial 4D Systems because of the multi camera array and the way the data is processed you get very little stretching or seaming, it’s far more “Volumetric”

8GB’s of data streamed in Softimage using Thiago Costa’s OBJ Sequencer with some extra ICE and Softimage trickery (best viewed at 720p)

Examples of me, trying to act tough ala Matrix style (performance is inspired by the research of George Borshukov for the Matrix films in 2003)

AGisoft “Frame-00124″ Test:

Synchronized Video Streams:

wmode=transparent[/youtube]

Despite the bonus of no calibration what so ever, being able to process old 3D captures taken with my DI3D kit as well as some older stereo photographs, I think the results are pretty incredible. Using up to 16 fixed synchronized DSLR’s, I am now able to capture a slice of time in 3D from multiple angles. Currently stitching the full body data using a motorized turntable but I hope to upgrade to more Cameras soon for 180-360 capture. It also includes a built in Face capture system into the Camera array, instead of being separate.

Infinite-Realities can now offer a custom built mobile 360 degree Full Body/Face Capture service to rival that of any Cyberware or Mephisto system but with 100′s of Megapixels of Color data to utilize, without having to use lasers or expensive structured light systems that are prone to error, subject discomfort and movement.

These are the very first prototype captures, processed rigidly into a water tight 360 Mesh, using off the shelf Cameras and some super cheap but very powerful Software. I hope to improve the output over the coming weeks.

360 Closed Mesh

Individual Slices:

You can choose from a variety of Cameras to texture with, adding upto 100′s of Megapixels of combined data. Most front shots are suitable to just use roughly 36MP’s for the body and about 20-24MP’s for the face.

Using some freely available data from the Artec-Group website, I wanted show others how to use Scanner-Killer using a virtual environment to learn how to calibrate and scan. Using a virtual scene gives you allot of freedom to test and play with Scanner-Killer without having to buy expensive equipment. It also shows you the limitations and benefits of Stereo Photogrammetry.

I used  3DSMax to simulate a 16 Camera Array, well 18 if you include the Ceiling Pod (in the past, xyz-rgb have set-up a similar system IRL using 24 Cameras!) Using this virtual scene it will allow us to evaluate the costs of actually doing this in the studio and simulate whether it will work or not. If it does (which it did!) it means you can do a full body capture in under 1 second with about 10 minutes to process the data. Compare this to the actual Artec Scan from Artec-Group, which according to their website took about 20 minutes to capture and 3.5 hours to process! The benefits are clear. Artec MH scanners are also costly, £10K+ compared to a 16 Camera Single Shot System for under £8K.

For processing the data I highly recommend the SK Core Standalone $299 license or even better the Pro License (with 4D Batch and MP Options)

Virtual Scene

Here is an example of how the cameras are arranged in 3DSMax, 16 around the Subject and 2 above for a total of 18, 9 “Pods”

We split them into top half and bottom half for increased resolution both in Mesh density and texture resolution, yielding on average 18MP per capture slice, resulting in a very big texture file when everything is merged.

This is enough to cover a full 360 of your subject matter to be scanned in one single capture. Of course using 3DSMax we have to render each frame out one at a time. But in reality it would be a press of a button. Each Pod is calibrated using a virtual checker board. In the scene there are 4 Virtual Lights (in yellow) and 4 Virtual Noise Projectors (in green) to mimic Digital Projectors. We project a Noise pattern onto the Subject.

.

Rendered Captures Processed with SK

Each Camera was rendered and processed one by one with the SK software. If you have SK Pro you can do this in bulk with one button, as well as have the ability to export out 3 sets of Disparity maps which are useful for other processing techniques later on (although this option might be included in SK Core?)

Each capture frame and an example Edge Cull map from the generated Disparity maps.

Each Capture slice was loaded into ZBrush, then using the 3rd Disparity Map we are able to clip the edges of unwanted and stretched artefacts using Masking. Before this was a manual process and very time consuming but this technique saves the hassle!

All we have to do now is align them all! No easy task but possible. Using a 3rd Party application like Mesh Lab these capture slices can be Rigidly aligned very easily.

Rigid Registration

For this example I used Artec-Group software to quickly align all 17 Captures into one Closed Model. The next stage is some cutting and fusing the data.

Comparison between the original Artec Scan and the synthetic SK Scan

Even just using Rigid Registration the results are pretty good, there are some anomalies due to Calibration inconsistencies but these tests prove that a single shot capture system comprising of 16+ Cameras (8 Pods) is a worthy endeavour. Reducing capture times down to under 1 second from 20 minutes! Dense detailed Meshes + 36MP full colour = Pretty, pretty good!

There is no doubting how superb the original Artec-Group scan is, the hardware is incredible but the capture times, processing times and low colour resolution are a slight drawback. I still own an Artec MH and cannot recommend them highly enough.

The SK scan lacks some detail and there are some fusing issues but with Hao Li’s Non-Rigid Registration all these problems should be addressed.

So in short, these are tests you can do now! in pretty much any 3rd Party Software like Modo, Softimage, Maya, 3DSMax, Lightwave etc combined with an SK License your all set to learn the work-flow and get scanning in Real Life!

*UPDATE*

Working with Hao Li, we plan to post some Non-Rigid Registration tests later this week. Updates to follow.

This was a custom developed script by Steven Caron, it cost a few $$$’s to develop but I am happy to share it if it helps others.

If you use it, please do quote the source and developer and please share your results! always interested to see cool stuff(s)! You can download the .rar above that has the actual Python script in it, or you copy and paste below directly into the Script Editor.

We offer no support but this should work fine in Softimage 2012.

import win32com.client
from win32com.client import constants

xsi = win32com.client.Dispatch("XSI.Application").Application
xsiPrint = xsi.LogMessage

class SubCurve:
    """
    class to hold subcurve points in
    """
    def __init__(self):
        self.points = []
    def AddPoint(self, x, y, z, w = 1.0):
        self.points.append( x )
        self.points.append( y )
        self.points.append( z )
        self.points.append( w )
    def MakeUniformKnots(self):
        self.knots = [0.0]*(len(self.points)/4+2)
        knotVal = 1.0
        for i in range(3,len(self.knots)):
            self.knots[i] = knotVal
            if i < len(self.knots)-3:
                knotVal += 1.0
    def RotatePoints(self):
        rotMatrix = XSIMath.CreateMatrix3(1.0, 0.0, 0.0, 0.0, 0.0, -1.0, 0.0, 1.0, 0.0) # 90 degree rotation
        for i in range(0,len(self.points),4):
            vec = XSIMath.CreateVector3( self.points[i], self.points[i+1], self.points[i+2] )
            vec.MulByMatrix3InPlace(rotMatrix)
            self.points[i] = vec.X
            self.points[i+1] = vec.Y
            self.points[i+2] = vec.Z

class CurveList:
    """
    basic curve list class
    """
    def __init__(self):
        self.name = "ASECurve_1"
        self.subCurves = []
        self.position = XSIMath.CreateVector3()
    def AddSubCurve(self, curve):
        self.subCurves.append( curve )
    def RotatePosition(self):
        rotMatrix = XSIMath.CreateMatrix3(1.0, 0.0, 0.0, 0.0, 0.0, -1.0, 0.0, 1.0, 0.0) # 90 degree rotation
        self.position.MulByMatrix3InPlace(rotMatrix)

# no undos, takes up a lot of memory, also keep logging quiet
undoPref = xsi.GetValue("preferences.General.undo")
logMessagePref = xsi.GetValue("preferences.scripting.cmdlog")
xsi.SetValue("preferences.General.undo", 0, "")
xsi.SetValue("preferences.scripting.cmdlog", False, "")

# all curves objects to be made
ASECurves = []

# get file from user input
filebrowser = XSIUIToolkit.FileBrowser
filebrowser.DialogTitle = "Select an .ase file..."
filebrowser.InitialDirectory = xsi.ActiveProject2.Path
filebrowser.Filter = "ASE (*.ASE)|*.ASE|All Files (*.*)|*.*||"
filebrowser.ShowOpen()

if filebrowser.FilePathName:
    file = open( filebrowser.FilePathName, "r" )

    progress = XSIUIToolkit.ProgressBar
    # parse the file once to get the line count
    progress.Maximum = len(file.readlines())
    file.close()
    progress.Caption = "Parsing %s..." % (filebrowser.FileName)

    # actually do the work this time
    file = open( filebrowser.FilePathName, "r" )

    curveName = ""
    currentCurve = None
    currentSubCurve = None

    progress.Visible = True
    for line in file:
        progress.Increment(1)
        if "*SHAPEOBJECT" in line:
            currentCurve = CurveList()
        if "*NODE_NAME" in line:
            curveName = line.replace("*NODE_NAME ","")
            curveName = curveName.strip()
            curveName = curveName.strip('"')
            curveName = curveName.replace("-","_")
            currentCurve.name = curveName
            continue
        if "*TM_POS" in line:
            position = line.split()
            currentCurve.position.X = float(position[1])
            currentCurve.position.Y = float(position[2])
            currentCurve.position.Z = float(position[3])
        if "*SHAPE_LINE" in line:
            if "COUNT" in line:
                continue
            currentSubCurve = SubCurve()
            continue
        if "*SHAPE_VERTEX_KNOT" in line or "*SHAPE_VERTEX_INTERP" in line:
            currentVert = line.split()
            currentSubCurve.AddPoint( float(currentVert[2]), float(currentVert[3]), float(currentVert[4]) )
            continue
        if "}" in line:
            if currentCurve is not None and currentSubCurve is not None:
                currentCurve.AddSubCurve( currentSubCurve )
                currentSubCurve = None
                continue
            elif currentCurve is not None and len(currentCurve.subCurves) > 0:
                ASECurves.append( currentCurve )
                currentCurve = None
            continue
        if progress.CancelPressed == True:
            break

    file.close()
    if progress.CancelPressed == False:
        # no create the curves in softimage
        progress.Value = 0
        progress.Maximum = len(ASECurves)
        for i in range(len(ASECurves)):
            curve = ASECurves[i]
            progress.Caption = "Creating curve %s..." % (curve.name)
            hairGroup = xsi.ActiveSceneRoot.AddNull( curve.name )
            curve.RotatePosition()
            xform = hairGroup.Kinematics.Local.Transform
            xform.SetTranslation( curve.position )
            hairGroup.Kinematics.Local.Transform = xform
            siCurve = hairGroup.AddNurbsCurveList2()
            siCurve.Name = curve.name
            for j in range(len(curve.subCurves)):
                subCurve = curve.subCurves[j]
                progress.Maximum = len(curve.subCurves)
                subCurve.MakeUniformKnots()
                subCurve.RotatePoints()
                siCurve.ActivePrimitive.Geometry.AddCurve( subCurve.points, subCurve.knots, False, 3, 0,
constants.siSINurbs )
                progress.Value = j
            progress.Value = i

    progress.Visible = False

xsi.SetValue("preferences.General.undo", undoPref, "")
xsi.SetValue("preferences.scripting.cmdlog", logMessagePref, "")

The next steps are going to be fantastic (I hope) Dual Scattering is what is needed.

Click here to view the embedded video.

The character is based from capture data acquired with a DI3D Capture kit back in 2009, I finally got time to re-build her in Softimage using a more advanced sculpt. The characters likeness is from the very beautiful figure model, Joceline Brooke-Hamilton.

Proof of concept renders:

These are just quick proof of concept renders, just to make sure the pipeline worked.

Rough render settings for Hair. Quick test

Rough render settings for Hair. Quick test

Beauty Render, still needs higher render settings but better.

I was super excited to be able to get hair in the same scene as the character and to be able to have scattering, GI bounce and soft area shadows all together and render lightning fast. Arnold is incredible.

The hair you see here is ultra basic. In the female renders, it’s just one colour using the basic Hair shader built into Arnold. Just yestarday I found out how to add mutant hair colours and more variations along with some cool ICE nodes from Marcel Reinhard (thank you) and I am also working on some other tricks from the help of Anto Matkovic and Borja (another thank you(s)

I kindly got permission from Marcos to be able to show these as all Arnold users are still under NDA so I can’t say much about what is in store for the future.

The hair model was built by the very talented Dani Garcia (aka Woody)
Importer Script by the legendary Steven Caron -  I will be sharing the script later this weekend in another post, for anyone interested in using it.

Not easy but possible.

Just for fun

So I wanted to share some of the HDR shots we took on Tuesday. I used HDRShop to combine a bunch of bracketed exposures into one HDR image. Split into Probe HDR’s and Latitude Longitude HDR’s. They aren’t perfect as we used a garden mirror ball (which has dents and smudges on it) but still cool looking! and it is possible to use the HDR’s to render with. I recommend HDRSHop to unwrap the Probes into Longitude-Latitude Spherical Map Textures. Apply the map to an inverted sphere and hey presto!!

HDR Downloads (you can render with these as Probe Images)

CLICK IMAGES TO DOWNLOAD (20MB’s each)

Here is an early demonstration of using the Normal Maps to deform and displace a low resolution scan. C4D is very good at this.  Due to limited customer support from my usual capture system, I turned to Breeze Software to capture and Scanner-Killer to process the 3D Captures.

I’m not sure if it is possible in Max, Maya or Softimage to use a World Space Normal Map to deform a mesh directly?? not just at render time as is how Normal Maps are usually used. If anyone has any work-flows on this I would love to hear about it.

XYZ=RGB

The multi array LED Dome, is a Custom design and build by Merry Hodgkinson. First Prototype created for capturing Globally Illuminated Subjects, along with World Space Normal Map acquisition.

Click here to view the embedded video.

I designed a way to process the different lighting conditions acquired by multi directional lighting that doesn’t require any coding. You can simple input the data into Photoshop or Premier, or After Effects and even stream the data to convert from colour to World Space Normal Maps. Merry further designed a technique to take this data to displace geometry using the Normal Maps in Maxon’s Cinema 4D.

Due to low level noise and blurring that can occur during Passive or Active 3D Extraction, displacing the 3D Mesh extractions using World Space Normal Map information can add much finer details, like pores and wrinkles not possible with normal capture or without any kind of mesoscopic coding.

We used the very robust DI3D system from Dimension Imaging to capture.
.

Thanks to Merry for the demonstration and the hard work he put into to wiring it up. A fantastic novel design! and a perfect match for a Stereo Photogrammetry Capture system. The other benefit is the fact the texture information is uniformly lit from every angle when in GI mode. The results are ideally suited for texture application on Computer Game Characters or for Digital Double Replication. Perfect for next generation *Replicants* !!

The light dome is similar in concept to the work done at USC ICT Graphics Lab.

We believe this new, lightweight more robust system is the first of it’s kind in the UK and could eventually be offered as an affordable, practical service to the Computer Games and Visual Effects Industry in Europe.

Colour (three source – r,b,g) Photometric Stereo, based on the research by Carlos Hernandez and George Vogiatzis.

The lights have yet to be calibrated and this is a rough set-up to prove the system can work, using off the shelf hardware and software. Next step is to apply to geometry and stream.

Further References:

Carlos Hernandez and George Vogiatzis,

High-detail 3D capture of facial performance
Martin Klaudiny Adrian Hilton James Edge
Centre for Vision, Speech and Signal Processing
University of Surrey,