Note: I've tried to make sure this website can be used with JS disabled, but some of the articles use MathJax to display math formulas. Those formulas will sadly be displayed as LaTeX code if your browser has JavaScript disabled.
Sometimes life gets a bit weird, and you go down a strange rabbit hole to extract medical data of your own teeth and put it into Blender.
Some time ago, I went to the dentist to get a crown replaced. I expected the classical dental crown process: they'd fill your mouth with dental alginate to get a mold, then they'd use that mold to make a plaster copy of your teeth, which they'd manually sculpt the crown around.
But we live in the future now! What used to be a sticky and nauseating process (alginate blocking your throat for a while wasn't fun...), has now become fully digital. In this computerized age, the dentist puts an electric toothbrush-shaped 3D scanner into your mouth and moves it around until the machine says it has seen everything there is to see. From that, a 3D model of your teeth is produced, and a crown is digitally modelled to fit the gap in the scan.
The 3D model is then 3D-printed, sanded, and installed on the patient's mouth. It's... quite neat, actually!
Of course, as soon as I realized my mouth was being laser-scanned I asked the dentist if I could have that data. As one does. For research, you know. Dentist didn't know, but was amused and said she'd find out.
After getting in contact with the company that 3d prints crowns for them, she was told she had to do the scan in a different way so that the software would allow her to share it with me. She thankfully decided to try, re-scan my teeth, and see if she could share the data. And it worked! Well, partially, at least.
I could now access the scan, but only through a mobile application the scanner company provides. And that app doesn't have any way to export that data. So close, yet so far!
The android app the 3D scanner company provides
The data was right there, but I couldn't really use it. And, well, that's the kind of zeroth world problem that really gets my hacker juices flowing!
By the way, I'll be censoring the name of the 3D scanner company. I wouldn't want a fun diversion to turn into a headache.
I decided to attack the Android app and try to get it to spill its secrets.
Initially I tried to intercept the network communications of the Android app, MITM-ing it using a proxy on my computer. Unfortunately the app thwarted my attempts and I got nowhere. Networking is clearly not my forte. I could feel my university friends mocking me for my puny h4kz0r skillz.
Then, having dinner with some of those friends, they wondered whether I had tried re-running the app with the phone in airplane mode. If the model loaded without a network connection, then surely it must be storing that data locally on the phone. And indeed it did! The app had a locally cached copy of my scan and it worked fine with no network connection. I just needed a way to get that data out of my phone. Unfortunately, my phone is not rooted, so that seemed like a dead end.
However, I guessed the Android emulator would probably be rooted, so that seemed like a promising avenue. I opened Android Studio on my computer, installed the app on the Android emulator, opened my scan in the app, and then located the temporary files it produced and copied them to my computer. The data was now mine! All mine!
After all this I got 2 files from the Android emulator:
cf2efe94-39bb-4f36-a154-2996835428a6.dcm71139844-aada-4e8f-bc51-2e023e191bab.dcmLooking on the internet, it seems like the dcm extension is typically used
for the DICOM medical image file format.
That'd make sense for this, I guessed.
Opening them on a text editor, however, they both looked like this:
<HPS version="1.1">
<Packed_geometry>
<Schema>CE</Schema>
<Binary_data>
<CE version="1.0">
<Vertices vertex_count="138395"
base64_encoded_bytes="1660740"
check_value="3872808768">
5GfuFjsWc3QuYlRVBVpvW4pGn4aweamL+jS4nfVeHxEkQA3vUCEkIo...
Ok, so it's an XML file with nodes that contain base64-encoded binary data. But from what I could read, DICOM isn't supposed to be XML 🤔
After digging through all this, my theory is the binary contents of the Vertices tag are in fact DICOM, but it's wrapped in this XML-based HPS container. Oh well, the XML part is easy enough to parse, and the rest should just be DICOM. No biggie, right? (ominous foreshadowing...)
Also, half-way through the file there was a section that looked like this:
<TextureImage Version="2"
Width="2048"
Height="2048"
TextureName="ColorTexture"
BytesPerPixel="3"
Base64EncodedBytes="340834"
RefTextureCoordId="0"
Id="0"
TextureCoordSet="0">
/9j/4AAQSkZJRgABAQEAeAB4AAD/2wBDAAIBAQIBAQICAgICAgICAwUDAwMDAwYEBAMFBwYHBwcGBwc...
A texture! Could this be the texture for the albedo color? 2048x2048 seems too big for it to be a thumbnail.
I then decoded that base64 chunk, and saw it began with �JFIF��x�x��
Heh, JFIF, I know that one. That's the magic number for JPEG files!
Dumped the decoded contents to a file, and I indeed got a JPEG image that looks like this.
Teeth! Pretty sure that's a UV-mapped albedo texture. Nice!
The colors look weird, though. Did someone store BGR data as RGB? Let's try a channel swap.
Yup! That looks much better.
Ok, now for the geometry!
That Vertices node looks like base64-encoded data. First order of
business, base64-decode it. Unfortunately, I couldn't make heads or tails of
that data. Nothing matched what I expected the DICOM format to contain, I
couldn't find areas of the data that looked like reasonable floating point
numbers... I was stumped, and gave up quickly. I would later find out this was
a wise decision.
Ok, so I can't make sense of that data. But the Android app can read it, even when in airplane mode. So clearly, that app has everything it needs to read and understand that file.
Time to decompile the app!
So I grabbed the APK file and ran it through an online APK decompiler. When I opened the decompiled output, I saw this.
Huh! Xamarin and Mono! Interesting. That means this is a .NET application wrapped in a thin Java layer to make it run on Android. They likely did this so they could have a single .NET codebase for iOS and Android, and then they added a small wrapper for each platform. Not a bad way to develop mobile apps, to be honest.
Anyway, let's find their custom .NET code and see what we can do with it.
Hmmm... assemblies... that sounds like a very .NET thing indeed. Is this a zip file or something? How do I unpack this?
Some quick internet sleuthing reveals pyxamstore, a python script that can parse these blobs and extract their contents. So I ran the blob through that script... et voilà! We have dlls.
Ok, we're back in business!
Now that we have those .NET dlls, we can try to decompile them. A nice thing about Java and .NET is that because they compile down to relatively high-level bytecode, you generally get pretty decent decompilations out of it.
And also, there's some very nice tools like JetBrains dotPeek, which is a free .NET decompiler that I had never used before, but was pleasantly surprised by.
So I opened my unpacked assemblies with dotPeek, and started trying to find the code that handled the scanned geometry.
I quickly found a type named FacetModel, and knew this was exactly what I wanted.
However, I also found this one:
Ooooh, so the Dcm part is encrypted. That explains why I couldn't make heads or tails of it...
At this point I thought "Well, data is encrypted, I'm screwed. Better ask the scanner company for help".
So I sent them a message asking for help decrypting the DCM files, by perhaps giving me access to some tool that could extract DCM data to an STL file? According to their own documentation they do have tooling to export 3D scans to STL, so...
They answered the next day and said that no, "it's not possible" for me to extract the data from the DCM file, and the company that prints the crowns would have to do it by exporting the original scan with the official scan management software.
Well, that wasn't very encouraging. I didn't think I'd be able to convince multiple people to spend a bunch of time for me on this...
But at this point I remembered it works in airplane mode. And that means everything needed to decrypt the data should be in the app.
Going through the disassembled output, it looked like the app was getting its decryption key from the app manifest? But I couldn't see an easy way to extract that data...
So then I thought: These assemblies don't know they're not on a phone anymore. The .NET bytecode is the same for all platforms. That's the whole "code once, run anywhere" philosophy. So I could load them as normal .NET assemblies and... just call those functions!
So I started digging through the API in those DLLs, and tried invoking a bunch
of functions that looked promising, (functions like
HpsAndDcmDocument.FromFile(string filePath), or
DcmModelHandler(XPathNavigator xmlNav)) but I kept running into the same
failure message:
After several hours of failures, though, one of my attempts finally worked. I ended up with a small C# app that loaded a couple of the Mono assemblies from the Android app and tricked them into getting all the scan data out:
Assembly dcmHandlerAssembly = Assembly.LoadFrom("COMPANYNAME.Mobile.DcmHandler.dll");
Type dcmHandlerType = dcmHandlerAssembly.GetType("COMPANYNAME.Mobile.DcmHandler.DcmHandler");
object dcmHandler = Activator.CreateInstance(dcmHandlerType);
MethodInfo loadModelFromStream = dcmHandlerType.GetMethod("LoadModelFromStream");
// Load FacetModel type, create one
Assembly dataStructuresAssembly = Assembly.LoadFrom("COMPANYNAME.Mobile.DataStructures.dll");
Type FacetModelType = dataStructuresAssembly.GetType("COMPANYNAME.Mobile.DataStructures.ThreeD.FacetModel");
object facetModel = Activator.CreateInstance(FacetModelType, new object[] { 1, 1 });
// Load Dcm file
System.IO.Stream file = System.IO.File.OpenRead(filename);
Task task = (Task)loadModelFromStream.Invoke(dcmHandler, new object[] { facetModel, file });
task.Wait();
and with this I got a FacetModel with a bunch of useful info!
After that, it was just a matter of dumping the data to an OBJ file that I could import into Blender.
First I got the vertices out, and that looked quite nice.
After some more massaging I had faces
And with some more fiddling I had UVs too
With this I had everything! I loaded the other DCM file and got the upper jaw from it. Put it all together, added a coated material in Blender for a nicer "wet" look, and... done!
So now I had my teeth in three dee, and I could finally rest!
And that's it! This was a fun and interesting rabbit hole to go down into.
Hope you enjoyed the writeup too!