What is Apache Sqoop?

Apache Sqoop is a tool to bulk import/export data into Hadoop ecosystem (HDFS, HBase or Hive).

• works with number of databases and commercial data-warehouses.
• available as command line tool, can be used in java with passing appropriate arguments.
• graduated from the Incubator & became top-level-project in ASF.

Current version of Sqoop does a map-only job with all the transformation happen in map task. ( Sqoop2 will possibly have reduce task as well)

Fig : Sqoop 2 Architecture diagram taken from Cloudera.com

Example : 

alok@ubuntu:~/apache/sqoop-1.4.2$ bin/sqoop import --connect jdbc:mysql://<hostname>:3306/<dbname> --username <user> -P --driver com.mysql.jdbc.Driver --table <tablename> --hbase-table <hbase-tablename> --column-family <hbase-columnFamily> --hbase-create-table

or use it like this in your java programs - 

ArrayList<String> args = new ArrayList<String>();

args.add("--connect");

args.add("jdbc:mysql://<hostname>:3306/<dbname>");

args.add("--username");

args.add("<user>");

args.add("--driver");

args.add("com.mysql.jdbc.Driver");

args.add("--table");

args.add("<tablename>");

args.add("--hbase-table");

args.add("<hbase-tablename>");

args.add("--column-family");

args.add("<hbase-colFamilyName>");

args.add("--hbase-create-table");

args.add("--num-mappers");

args.add("2");

int ret = Sqoop.runTool(args.toArray(new String[args.size()]));

• Sqoop can write data directly to HDFS or HBase or Hive.
• It can also export data back to RDBMS tables from Hadoop.
• Sqoop integrates with Oozie, allowing you to schedule and automate import and export tasks.

How to : working with HBase Coprocessor

HBase Coprocessor : It allows user code to get executed at each region(for a table) in region server. Clients only get the final responses from every region. HBase provides AggregateProtocol to support common aggregation (sum,avg,min,max,std) functionality.

Coprocessor framework is divided into : Endpoint : It allows you to write your own pluggable class which extends BaseEndpointCoprocessor and can have any number of methods which you want to be executed at table region server. Method executes much faster at regionserver and minimizes the network load as only results get transmitted to the client. Client need to do the final reduction on results returned by each region server.

Example : Below example illustrates just call to HBase coprocessor, A separate 'GroupByAggregationProtocol' interface extending 'CoprocessorProtocol' with methods required and Actual implementing class which implements 'GroupByAggregationProtocol' and extends 'BaseEndpointCoprocessor' must be created and deployed in each regionserver.

Map<byte[], Map<String, List<Long>>> resultFromCoprocessor = table
        .coprocessorExec(GroupByAggregationProtocol.class,
        <start-RowKey>,  // byte array or can be null
        <end-Rowkey>,   // byte array or can be null
        new Batch.Call<GroupByAggregationProtocol,  Map<String, List<Long>>>() {
             @Override
             public Map<String, List<Long>> call(GroupByAggregationProtocol aggregation)  throws IOException {
                return aggregation.getGroupBySum(filterList, scan);
             }
});

for (Map.Entry<byte[], Map<String, List<Long>>> entry : resultFromCoprocessor
 .entrySet()) {
 Map<String, List<Long>> en = entry.getValue();
 // Iterate through results from each regionserver   ......
       }    
}

Endpoint Coprocessors can be assumed as stored procedure in RDBMS.
Observers : It provides a hook to override few default methods of HBase when a event occurs.
It can be at three sub-levels
a) RegionObserver : handles/override Get, Put, Delete, Scan, and so on. It can be of type pre or post (eg : preGet, postDelete etc.)
b) MasterObserver : handles table creation, deletion and alter events. eg : preCreateTable or postCreateTable.
c) WALObserver : handles write-ahead log creation events.
eg : preWALWrite or postWALWrite .

Observer Coprocessors can be assumed as triggers in RDBMS.

What is IaaS, Paas & SaaS?

These are three main Cloud Computing Stack : Infrastructure as a Service, Platform as a Service and Software as a Service.

• SaaS applications are designed for end-users, accessible over the web.
• PaaS are set of tools and services to help developers design, develop, build & deploy application quickly.
• IaaS serves the need of storage, hardware, servers and networking components.

These cloud computing stacks can provide : 

• any Service on-demand.
• any Platform on-demand.
• large Infrastructure on-demand.

Elasticity of cloud computing brings scalability & accessibility to the applications. 

Examples : Amazon AWS (EC2), Google Cloud (Gmail), Microsoft Azure(Sky Drive) etc.

Java RMI : Remote Method Invocation

Remote Method Invocation (RMI) : It allows an object running in one Java virtual machine (say a client machine)  to invoke methods on an object running in another Java virtual machine (a Server machine).

• Server accepts tasks from clients, runs the tasks, and returns any results. The server code consists of an interface and a class. The interface defines the methods that can be invoked from the client. 
• RMI interface extends the interface java.rmi.Remote, and each method declares java.rmi.RemoteException in its throws clause. 
• Server register its remote objects with RMI's simple naming facility, the RMI registry.
• Client program obtains a stub for the registry on the server's host, looks up the remote object's stub by name in the registry, and then invokes method on the remote object using the stub.
• A Serializable object can be passed to-and-fro Client-Server.
• Source files can be compiled like :
  
  javac -d destDir RMIInterface.java RMIInterfaceImpl.java Client.java

RMI stub : In simple, its a proxy or surrogate which helps in managing invocation of remote objects.

Agile Scrum Methodology

What is Agile : Able to move Quickly or Easily 

In Software Industry or Product Development, it says team should divide their task in Iteration and rapidly start working once design (most IMP. spend as much time you have to make it better) is ready.

The Role Of Scrum

Scrum has three fundamental roles: Product Owner, Scrum Master, and Team Member.

• Product Owner is responsible for communicating the vision of the product and creates a prioritized wish list called a product backlog. 
• Scrum Master acts as a liaison between the Product Owner and the team and keeps the team focused on its goal. S/He meets team each day in Daily Scrum to assess its progress.
• Team Members are responsible for determining how tasks will be accomplished. They can select any work of their choice which they commit to finish.

More Scrum Terminologies 

1. Sprint Planning
2. Sprint Goal
3. Sprint Backlog
4. Sprint Burndown
5. Sprint Review

Agile Scrum Benefits

1. Shorter Delivery Cycles
2. Customer Involvement via feedback 
3. Self Organizing Team
4. Adaptable to Change

Scrum Cycle repeats at end of every Sprint with newer Goal and prioritized tasks.

Things to remember : In Map Reduce

Q 1. What is IdentityMapper?
A - An empty Mapper which directly writes key/value to the output.
         Mapper<K,V,K,V>
Q 2. What is InverseMapper?
A - A Mapper which swaps the <Key,Value> to <Value,Key>.
         Mapper<K,V,V,K>
Q 3. What is IdentityReducer?
A - It performs no reduction, directly writes key/value to the output.
         Reducer<K,V,K,V>
Q 4. What is Partitioner?
A - It runs after completion of Map Jobs. A custom Partitioner can be implemented to decide which key/value should go to which Reducer.
In Map-Reduce model, unique key 'K' with all its Iterable<V> should go to same Reducer.
Q 5. What are the uses of Combiner?
A - It helps in performing local aggregation on Map jobs output to reduce the ammount of data sent to any Reducer.
Q 6. Where Map outputs are stored?
A - Intermediate or Grouped Map output are stored in Sequence File(can be gzipped) on HDFS cluster.
Q 7. How to set number of mapper & reducer?
A - JobConf class object is used to set number of mapper and reducer.
JobConf is present in package org.apache.hadoop.mapred and extends org.apache.hadoop.conf.Configuration
public void setNumMapTasks(int n);// sets number of mapper Job
public void setNumReduceTasks(int n);// sets number of reducer Job
Q 8. What is ChainMapper?
A - It allows to use multiple Mapper class in single Map task.
Output of one mapper is passed to another mapper and so on.
Each Mappper get executed in chain.
Q 9. What is RegexMapper?
A - A Mapper that extracts text matching a regular expression.

Things to remember : In Core JAVA

Q 1. Can you tell, which Algorithm is used by HashMap/HashTable?
A - HashMap internally uses bucket to store key-value pair. When a key is passed to HashMap, it is not used as 'key' as it is! It gets converted to another HashKey using HashCode(). When same HashKey is generated for multiple key(s) (ie: Collision in HashMap/HashTable), It(another key-value pair) get stored in same bucket as next item( Each bucket is a Linked List, It can contain multiple key-value pair).
HashMap can take a 'initial Capacity' & 'load Factor' in its constructor. 
initial Capacity : number of bucket get created at the time of initialization. 
load Factor : number of buckets get increased when Items cross this load factor.
HashTable is a synchronized version of HashMap. But HashMap gives performance bonus as object is not accessed by multiple Threads. 
Q 2. Name some way of Inter Process Communication(IPC)?
A - These are :

1. Socket
2. Message Queue
3. Pipe
4. Signal
5. File
6. Remote Method Invocation (RMI)
7. Shared Memory
8. SOAP, REST, Thrift, XML, JSON

Q 3. What is Mutual Exclusion?

A - Mutual Exclusion in OS (Mutex) is a collection of techniques/algorithms for sharing resources so that concurrent uses do not conflict and cause unwanted interactions. One of the most commonly used techniques for mutual exclusion is the semaphore.
Q 4. What are Abstraction and Encapsulation?
A - Abstraction : Hiding away unimportant details of an object, focuses on outside view.
      Encapsulation : It is defined as the process of wrapping up the data members and member functions together into a single unit.