Tuesday, September 10, 2013

How to build a better Android JNI library using NDK

I've been using Android for a few years now and just recently had to build a fairly substantial library in C++.  I've always hated the typical structure of the JNI layer, at least the one I learned originally.  The new library that needed integrating was completely devoid of any JNI structure.  Given the nature of the project, I didn't want to hard code the paths.

I looked at a variety of jni code generators.  There weren't many restrictions, it did have to build on windows, mac, and pc.  After looking at them, I decided my project wasn't that big so it seemed overkill to put in a code generator that would need to be maintained.

In the end, I chose a simple method that gets the job done with minimal frustration.  The only real requirement is that you must specify the calling paramters and return code.  Once you have done a few, you get used to it and the rest are easy.  A code generator would eliminate that need.  (Checkout the bottom of this page for some possible generators: jni generator info.)

Typically, it looks something like this:

Java code:

public class FooUtil {
    try {
         System.loadLibrary("awesomeFooLib");
         native_init = true;
     } catch (UnsatisfiedLinkerError e) {
     }

     private native int fooMeN(int blah);

     public int fooMe(int blah) {
           if (native_init) {
                return fooMe(blah);
           }
           return -1;
     }
}

The C layer looks like:
JNIEXPORT jint JNICALL JNI_OnLoad(JavaVM *vm, void *reserved) {
    return JNI_VERSION_1_6;
}
JNIEXPORT jint JNICALL Java_com_mtterra_android_utils_fooMeN(
        JNIEnv *env, jobject self, jint blah)
{
     // do something interesting
     return 0;
}

The problem with the above is all the hardcoded paths and the general fuglyness of the whole thing.  You can use javah -jni <classfile> to generate a header file with the long method names.  As much as I would like it to read better, JNI coding doesn't lend itself well for that.  However, it can be better.  Note: if you are using JNI you should make sure to read JNI Tips.

The new method is to register your JNI functions when the library is loaded instead of hard coding the paths.  This allows for much better readability and somewhat better portability.  The Java class portion stays the same, only the JNI changes.  I used C calling convention above, using C++ (below) just requires wrapping the code with extern "C" and calling functions off of the env pointer is simpler.

#ifdef __cplusplus
extern "C" {
#endif

JNIEXPORT jint JNICALL JNI_OnLoad(JavaVM *vm, void *reserved) {
     JNIEnv *env;
     jint ok = vm->GetEnv((void **)& env, JNI_VERSION_1_6);

     // Add a function to register all of the jni calls
     if (ok == 0) {
         installJNIFunctions(env);
     }

     return JNI_VERSION_1_6;
}

   // Add your normal-looking functions
static int JNICALL fooMe(int blah) {
    return 42;
}

    // Define all the functions here that need to be registered
static JNINativeMethod fooLibraryMethods[] = {
    // Calling parameters are pretty easy, the end of this post has more info.
    // The first paramter is the name that your java defines as the native call.
    // Then the method signature then the local function to be called.
    { (char * const) "fooMeN", "(I)I", (void *)&fooMe) }
};


void installJNIFunctions(JNIEnv *env) {
     // This part defines the linkage to your app.  It is a hard coded path but it is also
     // very easy to change if you move the code to another app or put it in a library at some
     // point.
     // In your .h file you would add
     // #define FOO_LIBRARY "com/mtterra/android/utils"
     // substituting your package name obviously.
    jclass fooLibCls = env->FindClass(FOO_LIBRARY);

    // Make sure the class was found
    if (fooLibCls == NULL) {
        env->fatalError("Unable to find " FOO_LIBRARY);
        return;
    }

    // get the number of methods available
    int noofMethods= sizeof(fooLibraryMethods) / sizeof(fooLibraryMethods[0]);

    // finally, register the methods
    env->RegisterNatives(fooLibCls, fooLibraryMethods, noof Methods);

     //done....reap the rewards of unmangle names
}

#ifdef __cplusplus
} // extern "C"
#endif

The hardest part is writing the method signatures.  In the above example, I is standard for int.  So (I)I is an int argument to the method with a return type of int.  Looking at a more interesting example:

long foo(String s, int i, String t);  This would translate to a method signature of  (Ljava/lang/String;ILjava/lang/String)J

To see the full list of java types that can be used and more examples see this.  As far as the ndk build tools, there aren't any needed changes to compile the code.

Using the above method makes the code very readable and keeps from complicating the build environment.  Of course, if you have hundreds of jni calls then maybe a wrapper or jni code generator would be a better way to go.  In general, I have lots of libraries but there are few interfacing points where jni is involved.  JNI calls are rather expensive so cutting down the number of calls is a good thing.

---------------------------------------------------------------------------------------------------

QuickLogger Fitness - super quick fitness tracking
Android app on Google Play


Tuesday, March 27, 2012

Google IO registration day, now!

Ready....set....wait..... more spinner.....more spinner..  sigh

Yeah!  Finally got in on my mobile device.  See ya all there!



QuickLogger Fitness - super quick fitness tracking
Android app on Google Play

Friday, November 18, 2011

Building Android ICS (ice cream sandwich) with Suse 12.1

The main information for setup and install is best found here: http://source.android.com/source/initializing.html.

That said, there are always build issues since everyone has a slightly different environment.  I am currently running Suse 11.4 that has been upgraded over time from 11.1.  This is all run under a VM in a laptop set for 3/4 GB of RAM available to it on a  3yr old core 2 processor.  Needless to say, it takes a while to build.

There have been several issues that have come up that others may run into.
First, GCC 4.5.0 that comes in 11.3, didn't work.  It came up with an internal compiler error in caller-save.c.  I heard there were issues with 4.6 and that the best compiler for it is probably 4.4.3.  I decided to upgrade to 4.5.1 which comes in 11.4 Suse distro.  This fixed the problem.

3/4GB of RAM is a bit low for doing this build.  I bumped it to 2GB and it didn't have any issues.  I have approximately 20GB of free disk space which seems to be enough.  It takes around 6 or so to do the build.  I used 2G cache size for the build.

Also, make needs to be version 3.81.  Apparently, 3.82 has an bug in it that causes some issues.

The next big issue is with OpenJDK.  There is at least one error:

cts/apps/CtsVerifier/src/com/android/cts/verifier/PassFailButtons.java:191: onCreateDialog(int,android.os.Bundle) in android.app.Activity cannot implement onCreateDialog(int,android.os.Bundle) in com.android.cts.verifier.PassFailButtons.PassFailActivity; attempting to assign weaker access privileges; was public
    private static <T extends android.app.Activity & PassFailActivity>
                    ^
1 error
make: *** [out/target/common/obj/APPS/CtsVerifier_intermediates/classes-full-debug.jar] Error 41

Digging a bit showed that OpenJDK was the problem.  I went and downloaded the Sun JDK which is what is  suggested.  Download info is part of the top installation link.  ICS is built with Java 1.6.  Previously, 1.5 was the required java compiler.  So setting the location of the JDK is important depending on which type of build you want.

Some additional packages are required.  I also did this with a Suse 12.1 64 bit clean install which require even more packages with 4.6.2 as the compiler vs 4.5.1.  The full list is outline below.

bison - used by 'lunch' command
glibc-devel libraries (will see missing gnu-stupbs-32.h)
flex
mesa libraries
zlib devel (32 bit)
ncurses devel (32 bit)
gperf
perl-Switch module
libX11-devel 32 bit

I had the gcc-4.6.2 64 bit version installed.  You'll need to install gcc & gcc-c++ 32bit as well.

Make sure to load the envsetup.sh before the next step: (source build/envsetup.sh).

After installing, setup the path correctly.  For bash, this is:
'export PATH=$PATH;/usr/java/jdk1.6.0_29/bin'
which isn't good enough, had to set the real flag:
'export ANDROID_JAVA_HOME=/usr/java/jdk1.6.0_29'.


In order to get an older revision (the current is 1.7 which won't work), you must signup for an oracle account.

Before you do a make, open development/tools/emulator/opngl/host/renderer/Android.mk.  Double check that it has LOCAL_LDLIBS += -lX11 added.  This fixes a build error that results in 'undefined reference to XInitThreads'.  (Courtesy of Diego Stamigni)

WIth the build 4.0.3 of ICS, there were some build problems using 4.6.2.

- Issue with external/oprofile/libpp/format_output.h, line 94. Change:
mutable count & counts;  to just 'count & counts;'  With latest ics version this has likely been changed.

- ptrdiff_t does not name a type
Add #include <cstddef> in file external/gtest/include/gtest/internal/gtest-param-util.h

- ParamName set but not used
Comment out the line in frameworks/compile/slang/slang_rs_export_foreach.cpp or you can modified the make with -Werror.

Now do a make -j4 and you should be all set.

NOTE: Using Suse 11.4 with gcc 4.5.1, I could build full-magura but there were a number of problems with full-eng.  With Suse 12.1 and gcc 4.6.2, full-eng built without problems.

To run the full-eng emulator, export ANDRIOD_PRODUCT_OUT=~/<android_root>/out/target/product/generic then run the emulator located in out/host/linux-x86/bin/emulator.

Good luck!









Tuesday, December 14, 2010

Skipping entry in package table 0 because it is not complex

I saw this message for a while before investigating it. A couple of google search led me to nowhere. Basically, there were some questions on it but no answers. Given that there are probably a few ways to get into this, its probably better to figure out how to track it down.

So the message looks like the following:
Skipping entry 0x7f090003 in package table 0 because it is not complex!

A great message if there ever was one. Anyway, the first thing to do is look at the generated output. In eclipse, its under <project>/gen/<pkgname>/R.java. The R.java file looks like the following:

/* AUTO-GENERATED FILE. DO NOT MODIFY.
*
* This class was automatically generated by the
* aapt tool from the resource data it found. It
* should not be modified by hand.
*/

package fi.eye.android;

public final class R {
public static final class anim {
public static final int boxanimation=0x7f040000;
public static final int fade_in=0x7f040001;
public static final int fade_out=0x7f040002;
public static final int home_grid_fade=0x7f040003;
public static final int slide_left=0x7f040004;
public static final int slide_right=0x7f040005;

continues for a long time......

So its easy to just search for the offending value to determine what value to look for. In my case, it was a style attribute. Some items require a style with a one or more specific items defined. So in the your res/values/styles.xml file you might have something like:

<style name="Theme.myTheme" parent="android:style/Theme.White">
   <item name="android:....">value</item>
</style>

The 'value' is the important bit. some item names refer to more complicated values, such as android:textViewStyle. So you can either define another style with the set of common values or replace it with a specific value such as <item name="android:textSize">14sp</item>.

After a reload it should be good to go.

Thursday, November 4, 2010

Android ScrollView workaround fun

There are lots of little things that can be rather...headache worthy. ScrollView is a great little widget that with one simple entry, you get scrolling. However, depending on what you put in it, can lead to some interesting results. For instance, don't stick things like Lists in it. In my case, it was a simple EditText box that caused some issues.

Upon launch, EditText grabbed the focus causing the virtual keyboard to come up. This was due to having no available focuses, it chose my EditText box. Digging into it, the obvious solution that has many posts android:focusable="false" and the android:focusableInTouchMode="false". Of course, that doesn't help if you really want to focus on it.

The end result was to set these to false and upon click, change the modes. All to get around the issue with focus firing on this text box. Something like:

public OnTouchListener mNotesTouchListener = new OnTouchListener() {

public boolean onTouch(View arg0, MotionEvent arg1) {
notes.setFocusableInTouchMode(true);
notes.setFocusable(true);
return false;
}
};

And the good thing is none of it affects using the trackball and DPad. Granted this seems like a minor issue. At least it can be worked around.

Sunday, July 4, 2010

ADB reporting unknown target device

A quick note for developers updating to Froyo 2.2. After upgrading my Nexus One to version 2.2 (FRF91), the Android Device Chooser that pops up when you try to debug your application indicates the device is an unknown target rather than version 2.2.

The Android SDK and AVD manager will indicate all has been updated (assuming you did that). The solution is to make sure you upgraded Eclipse with the latest version of the adb/ddms plugin. In Eclipse, go to Help->Check For Updates should take care of the issue.

An easy thing to overlook and my first check on Google didn't pull up anything. So maybe this will help someone else.

UPDATE: One caveat, only one usb port actually works for this. Nothing worked while trying to get the other USB ports working. I'll have to dig up another usb cable to try. There have been reports of some dodgy cables.

Cheers

Tuesday, May 11, 2010

Porting of cURL to Android OS using NDK

In continuing my journey into Android territory, I decided it would be useful to understand the NDK development kit.  Given I want to transfer some files and possibly do a couple of other projects requiring low level work, I selected the cURL kit to port.  I've used cURL for a number of projects and it is my hope it runs well under Android.  The project, curltest, was run under Android 2.1 but it should work for other versions as well.  Given that most Android phones will be getting upgrades rolled out over the next couple of months to version 2.1, seems like a reasonable target.  I will be added HTTPS capability as soon as I port openssl, for now, this is strictly unsecured transfers.

I did see some examples of partial porting and even in the curl library source there is an Android.mk which describes how to set it up although not in enough detail.  Both were helpful but didn't actually get me to the holy land of a usable test application. (http://www.mail-archive.com/curl-library@cool.haxx.se/msg03433.html was helpful but didn't work for me.)

I found it worth the exercise of downloading the full android sources and building them.  Look at the build process, the bionic library, and well as what's in external.  The external directory is where you would add repositories to do things like run configure to generate config.h and Makefiles.

You won't actually need Linux since I have included the config file here but in case you want to follow way it was created, I've included it.

Summary of the steps to take for this port:
  - Lots of searches and view the current state of curl porting
  - Setup source repository (I used Linux, you could try windows using cygwin)
  - Create curl_config.h under the full android source tree
  - Untar curltest.tar.gz under $NDKROOT or Create NDK JNI App on Windows
  - Cross-link or add to eclipse project to build Java front-end
  - Unpack cURL into $NDKROOT/apps/curltest/jni/curl
  - Copy curltest/jni/curlbuild.h to curltest/jni/curl/include/curl
  - Copy curltest/jni/curl_config.h to curltest/jni/curl/lib
  - Build and test

Setup source repository on Linux

It's one thing to describe doing X, quite another to actually do it.  Anyway, I ended up creating my own android repository to start the process.  While I was using Windows 7 for my NDK development, I switched to Linux to create the curl_config.h file needed in the build.  What this means it that you run a couple of scripts.  First up, repo.  Created a directory and cd to it.  Download the repo script 'wget https://android.git.kernel.org/repo'.   Move repo script to somewhere in your path and issue 'repo init -u git://android.git.kernel.org/platform/manifest.git'  then after run 'repo sync'.  These two should get you all the goodies.  I had some stalls where I had to restart the sync process but after a long while it completed successfully.

Create curl_config.h under the full android source tree

Next, I created a script to call the configure command to generated the curl_config.h.  CD to  $ANDROID_ROOT/external.  Untar curl libraries, I renamed my from curl-7.20.1 to just curl.  cURL has an Android.mk inside of it.  In the file were some instructions for generating the curl_config.h.  I found them helpful as a start but ultimately I had to fiddle with a number of things to get this to actually work.  Here is the script I used:

#!/bin/sh

ANDROID_ROOT="$HOME/android_src" &amp;&amp; \
TOOLCHAIN_VER="4.4.0"  \
PLATFORM_VER="5" \
CROSS_COMPILE=arm-eabi- \
PATH=$ANDROID_ROOT/prebuilt/linux-x86/toolchain/arm-eabi-$TOOLCHAIN_VER/bin:$PATH  &amp;&amp; \
CPPFLAGS="-I $ANDROID_ROOT/system/core/include -I$ANDROID_ROOT/bionic/libc/include -I$ANDROID_ROOT/ndk/build/platforms/android-5/arch-arm/usr/include -I$ANDROID_ROOT/bionic/libc/kernel/arch-arm -L $ANDROID_ROOT/prebuilt/linux-x86/toolchain/arm-eabi-$TOOLCHAIN_VER/lib/gcc/arm-eabi/$TOOLCHAIN_VER/interwork -L$ANDROID_ROOT/ndk/build/platforms/android-$PLATFORM_VER/arch-arm/usr/lib  -L$ANDROID_ROOT/out/target/product/generic/system/lib " \
CFLAGS="-fno-exceptions -Wno-multichar -mthumb -mthumb-interwork -nostdlib -lc -ldl -lm "  \
./configure CC=arm-eabi-gcc --host=arm-linux --disable-tftp --disable-sspi --disable-ipv6 --disable-ldaps --disable-ldap --disable-telnet --disable-pop3 --disable-ftp --without-ssl --disable-imap --disable-smtp --disable-pop3 --disable-rtsp --disable-ares --without-ca-bundle --disable-warnings --disable-manual --without-nss --enable-shared --without-zlib --without-random

# openssl/zlib version
#./configure CC=arm-eabi-gcc --host=arm-linux --disable-tftp --disable-sspi --disable-ipv6 --disable-ldaps --disable-ldap --disable-telnet --disable-pop3 --disable-ftp --with-ssl=$ANDROID_ROOT/external/openssl --disable-imap --disable-smtp --disable-pop3 --disable-rtsp --disable-ares --without-ca-bundle --disable-warnings --disable-manual --without-nss --enable-shared --with-zlib=$ANDROID_ROOT/external/zlib --without-random


Run the script in the top curl directory (where configure resides) and it should go through many tests before ultimately creating the lib/curl_config.h file.

Create NDK JNI App on Windows

Untar curltest.tar.gz under $NDKROOT.  This has the Java side application already created.  Or you can simply create a new Application under Eclipse which will generated the src &amp; res directories that will need to be modified.  I'm not going into App generation except that you will target $NDKROOT/apps/curltest/project as your directory tree.

Create library name under $NDKROOT/apps/, I used curltest.
CD to curltest and create the Application.mk file.
APP_PROJECT_PATH := $(call my-dir)/project
APP_MODULES := curltest libcurl

Once that is done, cd to project and create your standard Java AndroidManifest.xml file.  If you created an App from Eclipse you'll need to modify this file.  Make sure to give it INTERNET permissions.  The rest depends on what's in your Java code.

<?xml version="1.0" encoding="utf-8"?>
<manifest xmlns:android="http://schemas.android.com/apk/res/android"
      android:versionCode="1"
      android:versionName="1.0" package="com.mtterra.curltest">
    <application android:icon="@drawable/icon" android:label="@string/app_name">
        <activity android:name=".curltest"
                  android:label="@string/app_name">
            <intent-filter>
                <action android:name="android.intent.action.MAIN" />
                <category android:name="android.intent.category.LAUNCHER" />
            </intent-filter>
        </activity>

    </application>
    <uses-sdk android:minSdkVersion="5"/>
    <uses-permission android:name="android.permission.INTERNET"/>

</manifest>




Then cd to jni and create the Android.mk file.  This is where the one from cURL would go.  However, I didn't use it as anything more than a reference.

Now for the actual JNI interfacing.  There are lots of complaints here because everything needs to be specified up front and you need different packages for different hardware.  Except for some dynamic registration mechanism, I'm not sure you can get away with writing to hardware AND getting independence from it.  Actually, I am sure, you can't.

Here is my curltest.c interface:

typedef struct pageInfo_t {
char *data;
int len;
} pageInfo_t;

static size_t
HTTPData(void *buffer, size_t size, size_t nmemb, void *userData) {
int len = size * nmemb;
pageInfo_t *page = (pageInfo_t *)userData;


if (buffer &amp;&amp; page-&gt;data &amp;&amp; (page-&gt;len + len &lt; (16 * 1024)) ) { memcpy(&amp;page-&gt;data[page-&gt;len], buffer, len);
page-&gt;len += len;
}
return len;
}

//Interface function that will receive web page from Java
jstring
Java_com_mtterra_curltest_curltest_JNIGetWebpage( JNIEnv* env,
jobject entryObject,
jstring webpageJStr )
{
pageInfo_t page;
CURL *curl;
CURLcode res;
char *buffer;

const jbyte *webpage;
webpage = (*env)-&gt;GetStringUTFChars(env, webpageJStr, NULL);
if (webpage == NULL) {
return NULL; /* OutOfMemoryError already thrown */
}

page.data = (char *)malloc(16 * 1024);
page.len = 0;
if (page.data)
memset(page.data, 32, 16 * 1024);

buffer = (char *)malloc(1024);

curl = curl_easy_init();
if(curl) {
curl_easy_setopt(curl, CURLOPT_URL, webpage);
curl_easy_setopt(curl, CURLOPT_WRITEFUNCTION, HTTPData);
curl_easy_setopt(curl, CURLOPT_WRITEDATA, &amp;page);

res = curl_easy_perform(curl);
/* always cleanup */
curl_easy_cleanup(curl);
(*env)-&gt;ReleaseStringUTFChars(env, webpageJStr, webpage);
if(res == 0) {
if (buffer) {
page.data[page.len &lt; 256 ? page.len : 256] = '\0'; sprintf(buffer, "pagedata(%d): %s. done.\n", page.len, page.data); return (*env)-&gt;NewStringUTF(env, buffer);
}
}
sprintf(buffer, "Result %d", res);
return (*env)-&gt;NewStringUTF(env, buffer);
} else {
return (*env)-&gt;NewStringUTF(env, "Unable to init cURL");
}
}




Finally, it's time to setup Eclipse to build.  There are some assumptions here, like having a functioning Eclipse build environment.  In order to get your environment over to Eclipse, you can use File-&gt;Import then select 'Existing Projects into Workspace' function.

Unpack cURL into $NDKROOT/apps/curltest/jni/curl.  Move the curl_config.h file created above to $NDKROOT/apps/curltest/jni/curl/lib. 

Finally, we create the Android.mk file in curltest/jni and build it.  The Android.mk use is as follows:

LOCAL_PATH:= $(call my-dir)

CFLAGS := -Wpointer-arith -Wwrite-strings -Wunused -Winline \
 -Wnested-externs -Wmissing-declarations -Wmissing-prototypes -Wno-long-long \
 -Wfloat-equal -Wno-multichar -Wsign-compare -Wno-format-nonliteral \
 -Wendif-labels -Wstrict-prototypes -Wdeclaration-after-statement \
 -Wno-system-headers -DHAVE_CONFIG_H

include $(CLEAR_VARS)
include $(LOCAL_PATH)/curl/lib/Makefile.inc


LOCAL_SRC_FILES := $(addprefix curl/lib/,$(CSOURCES))
LOCAL_CFLAGS += $(CFLAGS)
LOCAL_C_INCLUDES += $(LOCAL_PATH)/curl/include/

LOCAL_COPY_HEADERS_TO := libcurl
LOCAL_COPY_HEADERS := $(addprefix curl/include/curl/,$(HHEADERS))

LOCAL_MODULE:= libcurl

include $(BUILD_STATIC_LIBRARY)

# Build shared library now
# curltest

include $(CLEAR_VARS)

LOCAL_MODULE := curltest
LOCAL_SRC_FILES := curltest.c
LOCAL_STATIC_LIBRARIES := libcurl
LOCAL_C_INCLUDES += $(LOCAL_PATH)/curl/include
include $(BUILD_SHARED_LIBRARY)

Going back to your $NDKROOT you should be able to build the library with 'make APP=curltest -B V=1'.  The -B means rebuilt it all and V=1 makes it verbose. 

After this, you should be able to build the application.  When I get back to Eclipse I'll hit F5 to refresh the files as a habit although likely not necessary.  Another interesting thing is that some people have indicated that you need to do an 'adb push' to the emulator so the library is there.  I haven't found this is the case.  Basically, if it doesn't find your library the interface name or some other part is broken and you need to find the issue.

The finished result looks something like:



Here is the NDK project files used.  However, you'll have to add the curl sources to this.

The Software Rogue - http://thesoftwarerogue.blogspot.com 


QuickLogger Fitness - super quick fitness tracking
Android app on Google Play