Quantcast
Channel: Apache Timeline
Viewing all articles
Browse latest Browse all 5648

distributed cache + native lib

$
0
0
Hey,
I am trying to add a native lib file from HDFS to distributed cache so it can be loaded inside a UDF. This seems pretty standard and has been around for years, so I’m guessing I am overlooking something. Here’s my basic approach, please let me know if you spot the problem.

1. Put library in hdfs, located at ‘hdfs://localhost:8020/tmp/libcpp.dylib
2. Loaded into distributed cache in pig script:

set mapred.create.symlink yes;
set mapred.cache.files 'hdfs://localhost:8020/tmp/libcpp_jni_operators.dylib#libcpp_jni_operators.dylib';

Inside UDF:
System.loadLibrary("libcpp_jni_operators.dylib”);

The UDF fails due to library not found on java.library.path. (And inspecting that path at runtime, the library is indeed not present).

Thanks,
Adam

Viewing all articles
Browse latest Browse all 5648

Trending Articles