Tuesday, May 11, 2010

Porting of cURL to Android OS using NDK

In continuing my journey into Android territory, I decided it would be useful to understand the NDK development kit.  Given I want to transfer some files and possibly do a couple of other projects requiring low level work, I selected the cURL kit to port.  I've used cURL for a number of projects and it is my hope it runs well under Android.  The project, curltest, was run under Android 2.1 but it should work for other versions as well.  Given that most Android phones will be getting upgrades rolled out over the next couple of months to version 2.1, seems like a reasonable target.  I will be added HTTPS capability as soon as I port openssl, for now, this is strictly unsecured transfers.

I did see some examples of partial porting and even in the curl library source there is an Android.mk which describes how to set it up although not in enough detail.  Both were helpful but didn't actually get me to the holy land of a usable test application. (http://www.mail-archive.com/curl-library@cool.haxx.se/msg03433.html was helpful but didn't work for me.)

I found it worth the exercise of downloading the full android sources and building them.  Look at the build process, the bionic library, and well as what's in external.  The external directory is where you would add repositories to do things like run configure to generate config.h and Makefiles.

You won't actually need Linux since I have included the config file here but in case you want to follow way it was created, I've included it.

Summary of the steps to take for this port:
  - Lots of searches and view the current state of curl porting
  - Setup source repository (I used Linux, you could try windows using cygwin)
  - Create curl_config.h under the full android source tree
  - Untar curltest.tar.gz under $NDKROOT or Create NDK JNI App on Windows
  - Cross-link or add to eclipse project to build Java front-end
  - Unpack cURL into $NDKROOT/apps/curltest/jni/curl
  - Copy curltest/jni/curlbuild.h to curltest/jni/curl/include/curl
  - Copy curltest/jni/curl_config.h to curltest/jni/curl/lib
  - Build and test

Setup source repository on Linux

It's one thing to describe doing X, quite another to actually do it.  Anyway, I ended up creating my own android repository to start the process.  While I was using Windows 7 for my NDK development, I switched to Linux to create the curl_config.h file needed in the build.  What this means it that you run a couple of scripts.  First up, repo.  Created a directory and cd to it.  Download the repo script 'wget https://android.git.kernel.org/repo'.   Move repo script to somewhere in your path and issue 'repo init -u git://android.git.kernel.org/platform/manifest.git'  then after run 'repo sync'.  These two should get you all the goodies.  I had some stalls where I had to restart the sync process but after a long while it completed successfully.

Create curl_config.h under the full android source tree

Next, I created a script to call the configure command to generated the curl_config.h.  CD to  $ANDROID_ROOT/external.  Untar curl libraries, I renamed my from curl-7.20.1 to just curl.  cURL has an Android.mk inside of it.  In the file were some instructions for generating the curl_config.h.  I found them helpful as a start but ultimately I had to fiddle with a number of things to get this to actually work.  Here is the script I used:

#!/bin/sh

ANDROID_ROOT="$HOME/android_src" && \
TOOLCHAIN_VER="4.4.0"  \
PLATFORM_VER="5" \
CROSS_COMPILE=arm-eabi- \
PATH=$ANDROID_ROOT/prebuilt/linux-x86/toolchain/arm-eabi-$TOOLCHAIN_VER/bin:$PATH  && \
CPPFLAGS="-I $ANDROID_ROOT/system/core/include -I$ANDROID_ROOT/bionic/libc/include -I$ANDROID_ROOT/ndk/build/platforms/android-5/arch-arm/usr/include -I$ANDROID_ROOT/bionic/libc/kernel/arch-arm -L $ANDROID_ROOT/prebuilt/linux-x86/toolchain/arm-eabi-$TOOLCHAIN_VER/lib/gcc/arm-eabi/$TOOLCHAIN_VER/interwork -L$ANDROID_ROOT/ndk/build/platforms/android-$PLATFORM_VER/arch-arm/usr/lib  -L$ANDROID_ROOT/out/target/product/generic/system/lib " \
CFLAGS="-fno-exceptions -Wno-multichar -mthumb -mthumb-interwork -nostdlib -lc -ldl -lm "  \
./configure CC=arm-eabi-gcc --host=arm-linux --disable-tftp --disable-sspi --disable-ipv6 --disable-ldaps --disable-ldap --disable-telnet --disable-pop3 --disable-ftp --without-ssl --disable-imap --disable-smtp --disable-pop3 --disable-rtsp --disable-ares --without-ca-bundle --disable-warnings --disable-manual --without-nss --enable-shared --without-zlib --without-random

# openssl/zlib version
#./configure CC=arm-eabi-gcc --host=arm-linux --disable-tftp --disable-sspi --disable-ipv6 --disable-ldaps --disable-ldap --disable-telnet --disable-pop3 --disable-ftp --with-ssl=$ANDROID_ROOT/external/openssl --disable-imap --disable-smtp --disable-pop3 --disable-rtsp --disable-ares --without-ca-bundle --disable-warnings --disable-manual --without-nss --enable-shared --with-zlib=$ANDROID_ROOT/external/zlib --without-random


Run the script in the top curl directory (where configure resides) and it should go through many tests before ultimately creating the lib/curl_config.h file.

Create NDK JNI App on Windows

Untar curltest.tar.gz under $NDKROOT.  This has the Java side application already created.  Or you can simply create a new Application under Eclipse which will generated the src & res directories that will need to be modified.  I'm not going into App generation except that you will target $NDKROOT/apps/curltest/project as your directory tree.

Create library name under $NDKROOT/apps/, I used curltest.
CD to curltest and create the Application.mk file.
APP_PROJECT_PATH := $(call my-dir)/project
APP_MODULES := curltest libcurl

Once that is done, cd to project and create your standard Java AndroidManifest.xml file.  If you created an App from Eclipse you'll need to modify this file.  Make sure to give it INTERNET permissions.  The rest depends on what's in your Java code.

<?xml version="1.0" encoding="utf-8"?>
<manifest xmlns:android="http://schemas.android.com/apk/res/android"
      android:versionCode="1"
      android:versionName="1.0" package="com.mtterra.curltest">
    <application android:icon="@drawable/icon" android:label="@string/app_name">
        <activity android:name=".curltest"
                  android:label="@string/app_name">
            <intent-filter>
                <action android:name="android.intent.action.MAIN" />
                <category android:name="android.intent.category.LAUNCHER" />
            </intent-filter>
        </activity>

    </application>
    <uses-sdk android:minSdkVersion="5"/>
    <uses-permission android:name="android.permission.INTERNET"/>

</manifest>




Then cd to jni and create the Android.mk file.  This is where the one from cURL would go.  However, I didn't use it as anything more than a reference.

Now for the actual JNI interfacing.  There are lots of complaints here because everything needs to be specified up front and you need different packages for different hardware.  Except for some dynamic registration mechanism, I'm not sure you can get away with writing to hardware AND getting independence from it.  Actually, I am sure, you can't.

Here is my curltest.c interface:

typedef struct pageInfo_t {
char *data;
int len;
} pageInfo_t;

static size_t
HTTPData(void *buffer, size_t size, size_t nmemb, void *userData) {
int len = size * nmemb;
pageInfo_t *page = (pageInfo_t *)userData;


if (buffer &amp;&amp; page-&gt;data &amp;&amp; (page-&gt;len + len &lt; (16 * 1024)) ) { memcpy(&amp;page-&gt;data[page-&gt;len], buffer, len);
page-&gt;len += len;
}
return len;
}

//Interface function that will receive web page from Java
jstring
Java_com_mtterra_curltest_curltest_JNIGetWebpage( JNIEnv* env,
jobject entryObject,
jstring webpageJStr )
{
pageInfo_t page;
CURL *curl;
CURLcode res;
char *buffer;

const jbyte *webpage;
webpage = (*env)-&gt;GetStringUTFChars(env, webpageJStr, NULL);
if (webpage == NULL) {
return NULL; /* OutOfMemoryError already thrown */
}

page.data = (char *)malloc(16 * 1024);
page.len = 0;
if (page.data)
memset(page.data, 32, 16 * 1024);

buffer = (char *)malloc(1024);

curl = curl_easy_init();
if(curl) {
curl_easy_setopt(curl, CURLOPT_URL, webpage);
curl_easy_setopt(curl, CURLOPT_WRITEFUNCTION, HTTPData);
curl_easy_setopt(curl, CURLOPT_WRITEDATA, &amp;page);

res = curl_easy_perform(curl);
/* always cleanup */
curl_easy_cleanup(curl);
(*env)-&gt;ReleaseStringUTFChars(env, webpageJStr, webpage);
if(res == 0) {
if (buffer) {
page.data[page.len &lt; 256 ? page.len : 256] = '\0'; sprintf(buffer, "pagedata(%d): %s. done.\n", page.len, page.data); return (*env)-&gt;NewStringUTF(env, buffer);
}
}
sprintf(buffer, "Result %d", res);
return (*env)-&gt;NewStringUTF(env, buffer);
} else {
return (*env)-&gt;NewStringUTF(env, "Unable to init cURL");
}
}




Finally, it's time to setup Eclipse to build.  There are some assumptions here, like having a functioning Eclipse build environment.  In order to get your environment over to Eclipse, you can use File-&gt;Import then select 'Existing Projects into Workspace' function.

Unpack cURL into $NDKROOT/apps/curltest/jni/curl.  Move the curl_config.h file created above to $NDKROOT/apps/curltest/jni/curl/lib. 

Finally, we create the Android.mk file in curltest/jni and build it.  The Android.mk use is as follows:

LOCAL_PATH:= $(call my-dir)

CFLAGS := -Wpointer-arith -Wwrite-strings -Wunused -Winline \
 -Wnested-externs -Wmissing-declarations -Wmissing-prototypes -Wno-long-long \
 -Wfloat-equal -Wno-multichar -Wsign-compare -Wno-format-nonliteral \
 -Wendif-labels -Wstrict-prototypes -Wdeclaration-after-statement \
 -Wno-system-headers -DHAVE_CONFIG_H

include $(CLEAR_VARS)
include $(LOCAL_PATH)/curl/lib/Makefile.inc


LOCAL_SRC_FILES := $(addprefix curl/lib/,$(CSOURCES))
LOCAL_CFLAGS += $(CFLAGS)
LOCAL_C_INCLUDES += $(LOCAL_PATH)/curl/include/

LOCAL_COPY_HEADERS_TO := libcurl
LOCAL_COPY_HEADERS := $(addprefix curl/include/curl/,$(HHEADERS))

LOCAL_MODULE:= libcurl

include $(BUILD_STATIC_LIBRARY)

# Build shared library now
# curltest

include $(CLEAR_VARS)

LOCAL_MODULE := curltest
LOCAL_SRC_FILES := curltest.c
LOCAL_STATIC_LIBRARIES := libcurl
LOCAL_C_INCLUDES += $(LOCAL_PATH)/curl/include
include $(BUILD_SHARED_LIBRARY)

Going back to your $NDKROOT you should be able to build the library with 'make APP=curltest -B V=1'.  The -B means rebuilt it all and V=1 makes it verbose. 

After this, you should be able to build the application.  When I get back to Eclipse I'll hit F5 to refresh the files as a habit although likely not necessary.  Another interesting thing is that some people have indicated that you need to do an 'adb push' to the emulator so the library is there.  I haven't found this is the case.  Basically, if it doesn't find your library the interface name or some other part is broken and you need to find the issue.

The finished result looks something like:



Here is the NDK project files used.  However, you'll have to add the curl sources to this.

The Software Rogue - http://thesoftwarerogue.blogspot.com 


QuickLogger Fitness - super quick fitness tracking
Android app on Google Play

10 comments:

  1. thanks for the guide,
    typo at repo init -u git://android.git/kernel.org/platform/manifest.git' should be
    android.git.kernel.org

    got it working after some problems in libcrypto,
    but now I need ssl support in curl, but can't get curl configure to find openssl, I always get
    "SSL support: no" from curl configure.
    Dis u get curl working with ssl?

    ReplyDelete
  2. Excellent and thanks for the correction. I haven't added SSL yet. It follows the same process. You'll have to add an ssl library to your project and change appropriate settings on the curl side. I should hopefully get around to doing it soon.

    Cheers

    ReplyDelete
  3. I'm a newbie and I'm really grateful for your tutorial!
    However most features of curl are turned off/disabled, it compiles and works in emulator!

    ReplyDelete
  4. I was able to get curl to build with openSSL support. This is how I did it, you mileage may vary:

    followed the guide up until the configure script.
    modified the ./configure call to include the flag:
    --with-ssl=$ANDROID_ROOT/external/openssl

    copied the source into a similar directory tree as this guide says.

    modified the Android.mk file to have a new variable (under LOCAL_PATH):

    OPENSSL_INCLUDE_DIR := {my android root directory}/external/openssl/include


    modified the LOCAL_C_INCLUDES:

    LOCAL_C_INCLUDES += $(LOCAL_PATH)/include/ \
    $(OPENSSL_INCLUDE_DIR)

    (i also removed the curl executable build section)

    cd to $NDK_ROOT and did:
    make APP={app name}

    hope this helps someone

    ReplyDelete
    Replies
    1. hi natez0r,

      would you please support me with this version support ssl, I don't have enough experiences to do that, just send to me the new files should I replace & add.

      Delete
  5. Please somebody port libssl. I've been checking both cURL and wget, but none support ssl connections so far.
    I wanted to used wget or cURL as download manager for android since I couldn't find one that would actually work. I would need ssl to pull login page to rapidshare and save login cookie.

    Holding my fingers crossed

    ReplyDelete
  6. Hi,

    thanks for your great effort but I have a problem with any https request I do, always get response result 1 it means unsupported protocol, do i have to change anything in curl_config.h to make ssl support

    thanks

    ReplyDelete
  7. Excellent work!
    I have add some extra work to gen curl_config.h in order to compile under CyanogenMod7 (including openssl):
    **************
    #!/bin/sh

    TOOLCHAIN_VER="4.4.0"
    PLATFORM_VER="5"
    #CROSS_COMPILE=arm-eabi-
    #this variables should be already on path because of cyano
    #PATH=$ANDROID_BUILD_TOP/prebuilt/linux-x86/toolchain/arm-eabi-$TOOLCHAIN_VER/bin:$PATH
    CPPFLAGS="-I $ANDROID_BUILD_TOP/system/core/include -I$ANDROID_BUILD_TOP/bionic/libc/include -I$ANDROID_BUILD_TOP/ndk/build/platforms/android-5/arch-arm/usr/include -I$ANDROID_BUILD_TOP/bionic/libc/kernel/arch-arm -L $ANDROID_BUILD_TOP/prebuilt/linux-x86/toolchain/arm-eabi-$TOOLCHAIN_VER/lib/gcc/arm-eabi/$TOOLCHAIN_VER/interwork -L$ANDROID_BUILD_TOP/ndk/build/platforms/android-$PLATFORM_VER/arch-arm/usr/lib -L$ANDROID_BUILD_TOP/out/target/product/generic/system/lib -I$ANDROID_BUILD_TOP/prebuilt/ndk/android-ndk-r4/platforms/android-5/arch-arm/usr/include -L$ANDROID_BUILD_TOP/prebuilt/ndk/android-ndk-r4/platforms/android-5/arch-arm/usr/lib -L$ANDROID_BUILD_TOP/out/target/product/olympus/system/lib " \
    CFLAGS="-fno-exceptions -Wno-multichar -mthumb -mthumb-interwork -nostdlib -lc -ldl -lm " \
    ./configure CC=arm-eabi-gcc --host=arm-linux --disable-tftp --disable-sspi --disable-ipv6 --disable-ldaps --disable-ldap --disable-telnet --disable-pop3 --disable-ftp --disable-imap --disable-smtp --disable-pop3 --disable-rtsp --disable-ares --without-ca-bundle --disable-warnings --disable-manual --without-nss --enable-shared --without-zlib --without-random --with-ssl=$ANDROID_BUILD_TOP/external/openssl

    **************

    also to Android.mk:
    LOCAL_SYSTEM_SHARED_LIBRARIES := libc libssl libcrypto
    LOCAL_C_INCLUDES += external/openssl/include


    **************

    now my goal is to compile libcurl dynamically... any success?
    it will be really useful to have the library :D

    thanks!

    ReplyDelete
  8. I know this is an old post, but it ranks high on searches for "curl openssl android"

    The section "Setup source repository on Linux" can be achieve by following the instructions here:
    http://source.android.com/source/downloading.html

    ReplyDelete
  9. Updated script file I used FYI (I used standalone toolchain).......

    #!/bin/sh

    ANDROID_ROOT="$HOME/toolchain" \
    TOOLCHAIN_VER="4.6" \
    PLATFORM_VER="9" \
    CROSS_COMPILE=arm-eabi- \
    CPPFLAGS="-I$ANDROID_ROOT/sysroot/usr/include" \
    LDFLAGS="-L$ANDROID_ROOT/sysroot/usr/lib" \
    LIBS="-lc -ldl -lm" \
    CFLAGS="-fno-exceptions -Wno-multichar -mthumb -mthumb-interwork -nostdlib" \
    ./configure CC=arm-linux-androideabi-gcc --host=arm-linux --disable-tftp --disable-sspi --disable-ipv6 --disable-ldaps --disable-ldap --disable-telnet --disable-pop3 --disable-ftp --without-ssl --disable-imap --disable-smtp --disable-pop3 --disable-rtsp --disable-ares --without-ca-bundle --disable-warnings --disable-manual --without-nss --enable-shared --without-zlib --without-random CPPFLAGS="-I$ANDROID_ROOT/platforms/android-14/arch-arm/usr/include"

    ReplyDelete