How to Download and Mount Expansion File Resources in Marmalade C++

Problem:

You have a marmalade C++ game in excess of the 50mb limit imposed by an android store.  How do you split up your game, download the required assets from an external server, and make it all work as if nothing happened?

This is what I was faced with about a month ago when we released Catch the Monkey for android.  Our android phone version is 78mb.  Google Play limits an APK to 50mb.  Our Kindle Fire and Nook version is 270mb.  Both those stores limit the APK to 50mb.

While Google Play will host an expansion file and automatically (you hope) deliver it with your application download it may not.  The amazon and nook stores do not host expansion files for you, so you must host them yourself.

Solution:

This article documents how I split up my game assets and used HTTP download from my or google’s server of an OBB file.  I then mount the OBB file as a zipmount, and pull resources from there.  This has no performance impact on the game.

I wrote in a previous article (Google Expansion Files, LVL Checks, JNI) how to deal with Google Play to get the URL of your expansion file for manual downloading, so while that code is in this source code, I will skip over it and focus on the generic downloading obb issues.

 

Overview of the Solution

1. Check for the existence of your obb file.  It could be there because google downloaded it for you, or this is not the first time the user ran your game.
2. Mount the obb file as a zipmount drive
3. Check the validity of the zipmount.  It could be a partial file from a failed download or an old file.
4. If everything is OK, load your resources from the zipmount instead of your typical data folder, start the game.
5. If not, start the asynchronous download and wait to receive it bit by bit.
6. Save the file in pieces to the specially designated folder
7. Upon completion perform your validity check, and if it fails, start over at #5

Things to Know First

1. Google hosts an OBB file, which is simply an uncompressed zip file.  You can use winzip to create a zip file by setting an option to turn off compression, then rename the file to .obb.
2. Marmalade s3eVFSMount() allows you to mount an uncompressed zip file (obb) as a drive, so you have full visibility/access to the files inside.  Since it isn’t compressed it has no impact on resource manager loading from it.
3. You will have to handle HTTP redirects for downloads.  I have seen Google do up to 9 layers of redirect before the actual download URL was determined.  If hosting from your own server, you will want to handle this as well so you can move the files at some later time
4. In general an android device should store your expansion file at “raw:///mnt/sdcard/android/obb/com.company.appname”.  Google Play should automagically place your file there.  But every android device is slightly different:
   • A fresh Samsung Galaxy Tab 2 does not contain an /android/obb folder, so you have to make it.  You then have to make your own folder under that folder.
   • The Nook uses “raw:///mnt/media/<YourAppName>”
   • The Kindle Fire uses “raw:///mnt/sdcard/<YourAppName>”
   • As with everything Android, never assume.  You have to test on each device.
5. On some devices, the user can see and delete files in your folder.  This is not true on the more locked down devices like Nook and Kindle Fire.

The Source Code

I encapsulated all of the logic/code for handling this mess into a single screen (which I call Scenes in my game engine) called SceneLoadResources.  This screen is plug and play for me between projects, but it won’t be for you as it inherits from classes you don’t have.  However, everything you need to know to make your own screen is in that single cpp file.

Source: SceneLoadResources.zip

Step 1: Check for your obb file

SceneLoadResources::SceneLoadResources() : SceneUIWindow("black", RM.centeringIndentX, 0, false)
{	

	downloadedFile = false;
	obbPath = "raw:///mnt/sdcard/android/obb"; //samsung galaxy doesn't have an obb path by default, it could be made by google play but i'm not sure
	basePath = "raw:///mnt/sdcard/android/obb/com.company.appname";	
	assetFile = basePath + "/main.1000003.com.company.appname.obb";	

	string localFileName = assetFile;

	status = kNone;
	//the simulator doesn't accept the raw:// file names, the Nook requires it, so we have to chuck that part of the name before we start working
	if (s3eDeviceGetInt(S3E_DEVICE_OS) == S3E_OS_ID_WINDOWS)    
	{
		basePath = basePath.substr(6);
		assetFile = assetFile.substr(6);
		localFileName = localFileName.substr(6);
	}

In the screen constructor I setup the paths I’ll be working with. Notice the simulator doesn’t accept RAW paths, therefore I strip those off the front of each path and then it will work using the standard data-ram folder. Of course, you have to create the mnt and subsequent folders for this to work.

Also notice the use of global variables. This is because some of the work is done through asynch callbacks which do not have access to state in your class, so I had to separate values between my class and the global scope.

SceneLoadResources::Init() is how I setup my screen elements, you can safely ignore it as it has nothing to do with the issue at hand.

OnActive() is an event triggered whenever a scene has focus. This is where I put all the logic to this solution:

void SceneLoadResources::OnActive()
{
        assetFileLocal = false;
	assetFileValid = false;

	//check to see if we have our asset file locally
	if (s3eFileCheckExists(assetFile.c_str()) == S3E_TRUE)
		assetFileLocal = true;
	else
		assetFileLocal = false;

	//we have a file, so validate it
	if (assetFileLocal)
	{
		ValidateAssetFile();
	}

If the file is local and valid, the game can load as usual. If one of those are not true, then we have work to do.

Step 2 & 3: Check the validity of the file and zipmount

void SceneLoadResources::ValidateAssetFile()
{
...
	//first I need to mount the zip file as a drive
	//this only works on android, which means it cannot be executed in the simulator
	//I account for including/excluding the zipmount:// in the path of my resource files in the RM class
	if (s3eDeviceGetInt(S3E_DEVICE_OS) == S3E_OS_ID_ANDROID)    
		error = s3eVFSMount(assetFile.c_str(), "zipmount://", 0);		

	if (error != 0)
		s3eDebugOutputString("Error: zipmount failed on local file.  Assuming it is invalid");

I think the method is pretty well documented within the source. I’ll point out two key elements:
I first attempt to mount the zip file. If this fails for any reason, I consider the file invalid and proceed to downloading a new fresh file.

assetFileValid = true;
for (int i=0; i<(int)RM.assets.size(); i++)
{
   //these files came with the APK, so they are not in the zipmount, so i don't want to check for them
   if (RM.assets[i] == "UI.group" 
 || RM.assets[i] == "title.group" 
 || RM.assets[i] == "fonts.group"
 || RM.assets[i] == "loading.group")
	continue;

  //i have to add the .bin because resource manager doesn't require it, so it isn't part of my strings
  string filename = RM.GroupName(RM.assets[i]) + ".bin"; 
  if (!s3eFileCheckExists(filename.c_str()))
  {

I keep all of my asset files in a string vector inside my ResourceManager (called RM in the code). If the zipmount worked, I then check for each .group file in the zipmount. RM.GroupName is smart enough to know what files come from the data folder and what ones should come from the zipmount, so it will put “zipmount://” at the beginning. Since we are doing file checks, we have to add .bin to each resource file name.
If every file my resource manager expects is in the zipmount, then we know we have a good file, and can proceed to starting the game.

Step 4 Start up the game

Update() is called once per frame, prior to rendering. The first thing I do in here is check to see if we have a local file and it is valid (could have always been there, or we just finished downloading it). If so, we fire up the game:

s3eDeviceBacklightOn();
if (assetFileLocal && assetFileValid)
{
   SM->ChangeScene(new SceneTitle());

You will notice I turn the backlight on. This is because the download of the file on nook/kindle takes over 5 minutes. This is longer than most people’s device sleep timeout, so by calling this every frame I force the app to stay alive while downloading, otherwise it would fall asleep and the user would get grumpy because they would have to keep tapping the screen to keep the download alive. This allows the user to leave the device and walk away, and when they hear the title music, they know the game has started.

Step 5&6 Download Asynchronously through HTTP

I use a global state variable “status” to manage state for the update method. Once I have a URL I assign status to kStarting. This kicks off the http request using the CIwHTTP object and will call the callback function StartHttpFile().

if (status == kStarting)
{			

	//now that we know we have a folder, we will start the HTTP get process	
	SAFE_DELETE(theHttpObject);
	theHttpObject = new CIwHTTP;
	if (theHttpObject->Get(remoteFileName.c_str(), StartHttpFile, NULL) == S3E_RESULT_SUCCESS)
	{
		status = kGetting; //this means we kicked off the start, but it is async
		recieveFileSize = 0;				
		recievedSoFar = 0;
		progress = 0;
		timeSinceLastByte = frameCount;				
	}

Once the HTTP object has received some kind of response, our callback is fired and we can see what is happening:

static int32 StartHttpFile(void*, void*)
{
	int responseCode = theHttpObject->GetResponseCode();

	if (responseCode == 302)
	{
		//it's a redirect, google play redirects to 
		string newLocation;
		theHttpObject->GetHeader("Location", newLocation);
		string message = "Redirecting to " + newLocation;
		s3eDebugOutputString(message.c_str());

		SAFE_DELETE(theHttpObject);
		theHttpObject = new CIwHTTP;
		theHttpObject->Get(newLocation.c_str(), StartHttpFile, NULL);
		return 0;

	}

If we get a 302 response, it is a server redirect, so I delete the current request and start a new one with the new URL. As many times as the server redirects, this will keep following.

Assuming we got the response we were looking for, we start up the file receiving process by creating the local file handle. Notice that the buffer size we allocate is either the total file size or the maximum buffer size, whichever is smaller:

if (!recieveFileSize || recieveFileSize > maxBufferSize)
            bufferSize = maxBufferSize;
		else
			bufferSize = recieveFileSize;

        s3eFree(dataBuffer);
		dataBuffer = (char*)s3eMalloc(bufferSize);

We then stream the file down to us in chunks. I tested many different buffer sizes, and I found 32k to be the most stable and best performance on an android device. I flush every cycle to eliminate caching, as my file is 220mb and I don’t want to potentially run out of memory.

static void StreamHttpFile()
{
//read the data, write it to the local file
int actualBytes = theHttpObject->ReadData(dataBuffer, bufferSize);
s3eFileWrite(dataBuffer, actualBytes, 1, localFile);
s3eFileFlush(localFile);
recievedSoFar += actualBytes;
progress = (float)recievedSoFar / (float)recieveFileSize; //update progress value for progress bar

if (theHttpObject->GetStatus() == S3E_RESULT_ERROR)
s3eDebugOutputString("ERROR");
}

Because of the state handling, the update method will keep calling StreamHttpFile() every frame (30 frames per second). This just keeps happening till we get an error or the entire file is received.

if (status == kDownloading)
{
  if (recievedSoFar < recieveFileSize)
  {			
	int previousBytes = recievedSoFar;

	//slow it down so it isn't 30 requests per second, which looks like a DOS attack
	//this keeps it to 10 per second which is nicer on the server
	if (frameCount % 3 == 0)
        	StreamHttpFile();

I had to slow down the requests to the server, 30 per second is pretty extreme PER DEVICE, if 5 people are downloading that is 150 per second. Not impossible, but I found better performance and fuller buffers by spacing it out to 10 per second.

Also, I noticed during testing that the server would sometimes stop sending data. The received bytes would be 0 for a while, and then would continue again. This could be due to load or something. I set an abort after 30 seconds of not receiving any data. This seemed to be an appropriate timeout for my server and google, with no false positives.

Step 7 Validate the Download

if (recievedSoFar == recieveFileSize)
{
	s3eFileClose(localFile);
	status = kOK;
	assetFileLocal = true;
	downloadedFile = true;
	ValidateAssetFile();

	if (assetFileValid == false)
        {
		status = kError; //downloaded file is corrupt or wrong or something, so start all over
	}

Update keeps looping and downloading from the stream. It stops once we receive all the bytes we expect. At that point we close up the file and fire off our validation routine.

If the download failed, we display a message to the user and allow them to click a retry button. The retry button resets the state to the beginning and the whole process starts over again.

And that is how you can download expansion resource files to get past APK file limits in stores.

I may have skipped over something important, if so, post a question in the comments and I’ll do my best to answer it.