Wednesday, January 22, 2014

Parsing C2 histfiles

Once again, a quick filler post about parsing a random creatures file format.

This time we will be investigating the Creatures 2 "History file" format.

 This one is also already documented online, but being able to parse it can allow us to extract some interesting data.


I guess this post needed at least one picture



Where?

The History files are found in your C2 game directory, inside the adequately named "History" folder.

In there you will find :
  • The GameLog file we've already learnt to parse.
  • A series of "Photo_XXXX.s16" files. These are your C2 Norns photo albums, where XXXX is each Norns moniker.These are standard .s16 files you can browse with one of the many available tools.We'll learn to parse and manipulate them in an upcoming article.
  • A series of "cr_XXXX" files, these are the History Files we are interested in this time.
  • Optionally a set of folders, one by alternative world you played using the C2 world switcher.They contain all of the preceding items for each world.
For each Norn that took an active part in your game, you can find a "cr_XXXX" file in the History folder where XXXX is once again the Norn's moniker.

What?

Here's the information you can get out of this file, along with the format information :

Type        Meaning     Explanation
4 bytes     Moniker     The Norn's Moniker    
CString     Name        The Norn's name if any
4 bytes     MumMoniker  The Norn's Mother Moniker
CString     MumName     The Norn's Mother's name if any
4 bytes     DadMoniker  The Norn's Father Moniker (\x00\x00\x00\x00 if none, but always present)
CString     DadName     The Norn's Father's name if any
CString     Birthday    A litteral representation of the Norns birth date ( ex :22:59 Jan 10 2014 )
CString     Birthplace  The Norn's birthplace
CString     OwnerName   Owner info as set in the owner kit.
CString     OwnerURL    Owner info as set in the owner kit.
CString     OwnerNotes  Owner info as set in the owner kit.
CString     OwnerEMail  Owner info as set in the owner kit.
1 LONG      State       The Norn's status : 0=alive, 1=dead, 2=exported
1 LONG      Gender      The Norn's gender: 1=male, 2=female
1 LONG      Age         The Norn's age ? Seems to be 0 for all my Norns
CString     Epitaph     The Norn's epitaph if any
1 LONG      GravePhotoIndex An index in the Norns photo album for finding the Grave picture ( 4294967295 by default)    
1 LONG      TimeOfDeath     Time of death, expressed as an Unix timestamp
1 LONG      TimeOfBirth     Time of birth,expressed as an Unix timestamp
1 LONG      TimeOfAdolescence Time of reaching adolescence,expressed as an Unix timestamp    
1 LONG      DeathRegistered    Self explanatory, 0=no,1=yes
1 LONG      Genus         The Creatures kind ( yes, grendels and ettins also have histfiles!) 1=norn, 2=grendel, 3=ettin
1 LONG      LifeStage     The 0-6 Norn's life stage, as always:
0=Baby,1=Child,2=Adolescent,3=Youth,4=Adult,5=Old,6=Senile

256 bytes     ChemsAtDeath A 256 bytes array, containing the 256 chemical levels inside the Norn at death
9 LONGs        Padding     Unused, Must be zeroes



Potential uses:

Well, that looks like a one stop file for a lot of interesting information !

What could we do with all that ?
  • Obviously this looks like an appropriate place for quickly extracting a Norns family details such as mother and father's monikers and names
  • Scanning the whole fileset will quickly help you identifying which Norns are still alive, dead or have been exported.This is useful if you doubt about still having a copy of a given Norn.
  • You could finally edit that misspelled epitaph on your favorite Norn's grave.Yes, this does work! The history files are the ones containing the actual ingame grave information.And it works for changing pictures too.Nothing is set in stone anymore ;)
  • You could perform Norn autopsy by printing out and studying the chemicals present in it's body at death. This would have an advantage over existing tools such as cred32 in at least 3 ways : Works with C2, Has a stable and fixed record of the chemicals that won't change, can be performed long after death, even after a Norn's body has disappeared from the world.We will be writing a custom tool to do just that in an upcoming article, meanwhile if you're into Norn autopsies, I suggest you take a look at this great "Discover Albia" post.
  • Not directly related, but by browsing the photoalbum files, you can finally recover all of the pictures ever taken of all your deceased Norns.This is good to have as once they're dead, the game only lets you see one last picture forever.

Additional "Cstring" info:

Right before we get to  the actual demo, just a word on the "Cstring" type used by some of these entries.

Contrary to what one might think, what the CDN calls "Cstrings" are not the C language '\0' terminated strings.But rather strings prefixed with length information.
They take two forms:
If the string is shorter than 256 characters, it is read as
<"1 byte" length> <"length bytes" actual string>

If you read out the length byte and find out it is "255", then you'll have to discard that byte, and read an additional "Word" ( 2 bytes).
That means the string is longer than 256 bytes and that "Word" gives you the string length, as in :
<"1 '225'byte"><"1 Word" length><"length bytes" actual string>

The script:

As usual here is a small python script illustrating  how to parse the history file format.
( this one has to be run from your creatures directory )
Oh, and by the way, may the purists pardon my highly unpythonic use of the language, but I'm trying to keep things simple and readable for everybody there. I know this code could be made much shorter and reusable.
The complete file is available at Parsehistfiles_example.py.

import os
import struct


def readbyte( readfromfile ):
    return struct.unpack("B",readfromfile.read(1))[0]

def readLong( readfromfile ):
    return struct.unpack("L",readfromfile.read(struct.calcsize("L")))[0]

def readCstring (readfromfile):
    """
    CStrings are strings which have length data linked with them. The format
    is flexible and they are read as follows:

    READ BYTE Length
    If Length=255 then READ WORD Length
    READ Char[Length] into String
    """
    strlength = struct.unpack("B", readfromfile.read(1))[0]
    if strlength == 0xff:
        strlength = struct.unpack("H", readfromfile.read(2))[0]
    return readfromfile.read(strlength)

files=os.listdir(".")

Norns={}
for histfile in files:
    if histfile[0:2]=="cr":
        Norns[histfile[3:]]={}
        fic=open(histfile,"rb")

        Moniker=fic.read(4)
      
        Name=readCstring(fic)
        Norns[Moniker]["Name"]=Name

        MumMoniker=fic.read(4)
        Norns[Moniker]["MumMoniker"]=MumMoniker

        MomName=readCstring(fic)
        Norns[Moniker]["MomName"]=MomName

        DadMoniker=fic.read(4)
        Norns[Moniker]["DadMoniker"]=DadMoniker

        DadName=readCstring(fic)
        Norns[Moniker]["DadName"]=DadName

        BirthDate=readCstring(fic)
        Norns[Moniker]["BirthDate"]=BirthDate

        BirthPlace=readCstring(fic)
        Norns[Moniker]["BirthPlace"]=BirthPlace

        OwnerName=readCstring(fic)
        Norns[Moniker]["OwnerName"]=OwnerName

        OwnerURL=readCstring(fic)
        Norns[Moniker]["OwnerURL"]=OwnerURL
      
        OwnerNotes=readCstring(fic)
        Norns[Moniker]["OwnerNotes"]=OwnerNotes

        OwnerMail=readCstring(fic)
        Norns[Moniker]["OwnerMail"]=OwnerMail

        State=readLong(fic)
        State=["alive","dead","exported"][State] #Isn't python awesome ?
        Norns[Moniker]["State"]=State

        Gender=readLong(fic)
        Gender=[None,"male","female"][Gender]
        Norns[Moniker]["Gender"]=Gender

        Age=readLong(fic)
        Norns[Moniker]["Age"]=Age

        Epitaph=readCstring(fic)
        Norns[Moniker]["Epitaph"]=Epitaph

        GravePhotoIndex=readLong(fic)
        Norns[Moniker]["GravePhotoIndex"]=GravePhotoIndex

        TimeOfDeath=readLong(fic)
        Norns[Moniker]["TimeOfDeath"]=TimeOfDeath

        TimeOfBirth=readLong(fic)
        Norns[Moniker]["TimeOfBirth"]=TimeOfBirth

        TimeOfAdolescence=readLong(fic) # Yay! Kisspoping time !
        Norns[Moniker]["TimeOfAdolescence"]=TimeOfAdolescence

        DeathRegistered=readLong(fic)
        DeathRegistered=["no","yes"][DeathRegistered]
        Norns[Moniker]["DeathRegistered"]=DeathRegistered

        Genus=readLong(fic)
        Genus=[None,"Norn","Grendel","Ettin"][Genus]
        Norns[Moniker]["Genus"]=Genus

        Lifestage=readLong(fic)
        Lifestage=["Baby","Child","Adolescent","Youth","Adult","Old","Senile"][Lifestage]
        Norns[Moniker]["Lifestage"]=Lifestage

        ChemicalsAtDeath=fic.read(256)
        Norns[Moniker]["ChemicalsAtDeath"]=ChemicalsAtDeath

print "Available monikers:"
for norn in Norns:
    print norn,":",Norns[norn]["State"]

who = raw_input("Enter a moniker: ")

for key in Norns[who]:
    if key != "ChemicalsAtDeath":
         print key,":",Norns[who][key]
    elif Norns[who]["State"]=="dead":
        print "Chemicals at death:"
        for chem in Norns[who]["ChemicalsAtDeath"]: print ord(chem),
    else:
        print "The creature is not dead!"
        

If we run the script we get :

Available monikers:
1KTQ : alive
4YJH : alive
5KPD : exported
2QRW : alive
8KEI : alive
4KOV : alive
4XSW : alive
0FSF : alive
4VTI : alive
2MYK : dead
3LON : alive
4BJO : alive
Enter a moniker: 2MYK
DeathRegistered : yes
State : dead
OwnerURL :
TimeOfBirth : 1388689204
BirthDate : 20:00 Jan 02 2014
DadMoniker : 4YJH

MomName :
OwnerName :
MumMoniker : test
Epitaph : Drowned a 4th time in a row while unattended.
TimeOfDeath : 1389628745
Name : <pas de nom>
Gender : female
Age : 0
DadName :
Genus : Norn
OwnerNotes :
BirthPlace : The birthplace
OwnerMail :
Lifestage : Child
TimeOfAdolescence : 486834176
GravePhotoIndex : 4294967295
Chemicals at death:
0 0 116 250 0 155 139 86 20 0 19 5 0 0 0 25 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 248 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 1 0 0 0 0 0 0 0 0 0 0 0 0 0 0 161 84 68 1 192 0 0 75 216 188 1 0 34 0 0 0 0 0 0 0 0 0 8 0 0 0 0 0 0 0 6 1 254 0 0 0 0 0 0 0 0 0 0 0 0 137 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 13 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 91 98 0 104 91 0 0 0 0 0 0 0 0 0 0 0



After another upcoming post on reversing and parsing the C2 alchemicals.str file, we will wrap all this up by designing an usable C2 autopsy tool that will match meaningful chemicals names to this garbage :)


No comments:

Post a Comment