Introduction

Hadoop is a de-facto parallel computing platform for many industrial companies. However, it is non-trivial to write MapReduce programs on Hadoop for complex data analysis tasks. Therefore, high-level structure query languages like Pig can help data scientists better perform analysis tasks without distracting to details of MapReduce programs on Hadoop. One problem with Pig is that, it is lack of capability to deal with complex conditions such as “if”, “while” and “switch”, which are widely supported by other general-purpose programming languages. To tackle the issue, one solution is to embed Pig into a general-purpose programming language.

There are several great references for introductory materials about this topic. For example, [here], [here], [here] and [here]. In this post, we do not want to discuss the basic idea of embedding Pig into Python but demonstrate some important practices.

One important fact that needs to pay attention is that “Python” here is essentially Jython, a JVM-based Python implementation which uses the same syntax with Python. However, the standard libraries used in Python may or may not be available in Jython.

Pig User-Defined Functions

Within Pig, user-defined functions (UDF) are usually used for performing complicated tasks which are not supported well enough from Pig native functions. One example is to draw a Gaussian random variable from each row of the data source, centered by some field names.

Many languages can be used to write Pig UDFs. One easy choice is Jython. For general information about Jython UDFs, please see here.

When functions are included in the main script of the embedding code, it is by default treated as Pig UDFs. For instance,

def abc:
    print 'Hello World'
if __name__ == '__main__':
    from org.apache.pig.scripting import Pig
    P = Pig.compileFromFile(...)
    Q = P.bind(...)
    results = Q.run()

Here, function “abc” will be treated as a Pig UDF not a normal Jython function.

Python User-Defined Functions

As normal functions defined in the script will be treated as Pig UDFs, how can we write functions without putting too much program into the main script. One way is to put those functions into a separate “.py” file and used as a module. Therefore, we can use those functions by calling modules and corresponding functions. For instance,

import ComplexTask
if __name__ == '__main__':
    tasks = ComplexTask.ParseTasks(...)
    from org.apache.pig.scripting import Pig
    P = Pig.compileFromFile(...)
    Q = P.bind(...)
    results = Q.run()

Here, “ComplexTask” is another Python script “ComplexTask.py” in the same directory. In such way, we can put different functionalities into different modules.

Summary

Once we can use Python to control the workflow of Pig scripts, it would be even better that we can start and configure different workflows from different configuration files. Unfortunately, we do not have such tools so far. Thus, we probably need to read and parse XML configuration files by hand.

One easy way to install GCC higher version (which supports C++ 11 standard) in alternative directory (meaning without root privilege) is as follows:

• Download the latest GCC from http://gcc.gnu.org/
• Untar the package
• If you don’t have prerequisites for GCC (http://gcc.gnu.org/install/prerequisites.html), you need to the following steps:
  • Go to the source directory.
  • Do “contrib/download_prerequisites”.
• Create a new directory called “gcc-build” peer to the source directory.
• In “gcc-build”, perform “../gcc-XXX/configure –prefix=YYY”
• Do “make” and “make install”
• Add “YYY/lib” or “YYY/lib64” into your LD_LIBRARY_PATH and LD_RUN_PATH

In this post, I woud like to explore how random number would be generated by using newly added features from C++ 0x in GCC.

Rather than explaining the details of C++ 0x, I just post code here:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69

#include <random>
#include <iostream>
#include <string>
#include <stdio.h>
#include <time.h>
 
using namespace std;
 
int main()
{
  int n;
  double p, lambda, shape, mu, sigma;
  mt19937 eng(time(NULL));
 
  uniform_int_distribution<int> uniform_int(3,7);
  for(int i=0; i < 10; i++)
    cout << "[Uniform INT distribution]:" << uniform_int(eng) << endl;
  cout << endl;
 
  uniform_real_distribution<double> uniform_real(0.0,1.0);
  for(int i=0; i < 10; i++)
    cout << "[Uniform REAL distribution]:" << uniform_real(eng) << endl;
  cout << endl;
 
  n = 5;
  p = 0.3;
  binomial_distribution<int> binomial(n, p);
  for(int i=0; i < 10; i++)
    cout << "[Binomial distribution]:" << binomial(eng) << endl;
  cout << endl;
 
  lambda = 4.0;
  exponential_distribution<double> exponential(lambda);
  for(int i=0; i < 10; i++)
    cout << "[Exponential distribution]:" << exponential(eng) << endl;
  cout << endl;
 
  shape = 3.0;
  gamma_distribution<double> gamma(shape);
  for(int i=0; i < 10; i++)
    cout << "[Gamma distribution]:" << gamma(eng) << endl;
  cout << endl;
 
  p = 0.5;
  geometric_distribution<int> geometric(p);
  for(int i=0; i < 10; i++)
    cout << "[Geometric distribution]:" << geometric(eng) << endl;
  cout << endl;
 
  mu = 3.0; sigma = 4.0;
  normal_distribution<double> normal(mu, sigma);
  for(int i=0; i < 10; i++)
    cout << "[Gaussian distribution]:" << normal(eng) << endl;
  cout << endl;
 
  lambda = 7.0;
  poisson_distribution<int> poisson(lambda);
  for(int i=0; i < 10; i++)
    cout << "[Poission distribution]:" << poisson(eng) << endl;
  cout << endl;
 
  p = 0.6;
  bernoulli_distribution bernoulli(p);
  for(int i=0; i < 10; i++)
    cout << "[Bernoulli distribution]:" << bernoulli(eng) << endl;
  cout << endl;
 
  return (0);
}

Note, the code is very clean in the sense that you don’t need any extra libraries at all.

Please compile with:

g++ -std=gnu++0x

The GCC I used is GCC 4.6.1

Several references:
1. http://www.johndcook.com/test_TR1_random.html (sort of out-dated)
2. http://www.johndcook.com/cpp_TR1_random.html (sort of out-dated)
3. http://gcc.gnu.org/onlinedocs/libstdc++/manual/status.html#status.iso.200x (current status of C++ 0x in GCC)
4. http://gcc.gnu.org/onlinedocs/libstdc++/manual/bk01pt01ch03s02.html ( a list of header files of C++ in GCC)