What is : Hadoop Sequence File?

Hadoop Sequence File : These are flat files consisting of binary Key-Value Pair. It can store any key-value pair as byte arrays.

3 Types : 

UnCompressed

Record-Compressed

Block-Compressed.

Need for Sequential File : Hadoop is meant for processing BigData. It has 64 MB default block size on any cluster. Which mean any file with size lesser than 64 MB will eventually occupy 64 MB physical space on disk storage.

      In practical, Applications deal with files with fewer KB. So, It is advantageous to keep number of such small file in sequential Key-Value pair, which allows programmers to run similar logic on each file found in a block with help of Map-Reduce Jobs.

    -org.apache.hadoop.io.SequenceFile Class provides Read, Write methods. It also grants provision for 'Sorting' of SequenceFile Keys.

Thank YOU

How to : choose between DOM, SAX or XMLStreamWriter

What is XML : I call it a "language for Internet", It help applications to communicate  seamlessly. At the same time it's in human readable format too. Any XML file typically contains 'Elements' and 'Attributes', which are also called XML 'Node'. 
(Element, CData, Comment, Attribute, Entity, Text are few examples of Node Type).
There are number of API(s) available to work with XML, and each has its positive and negative aspects.

W3C DOM : 

good for - random read and  write with XML nodes.
not suitable for - larger memory footprint, performance.

SAX :

good for : faster read access, it's lightweight.
not suitable for - writing/creating XML nodes.

XMLStreamWriter :

good for - streaming out XML while building it, useful in web services handling larger files.
not suitable for - random read or write, It's sequential, one-way, cursor like implementation. 


All what you need is to choose your API closest to your need. XMLStreamWriter is a good for all purpose. Most effective for mobile devices.

Thank YOU

What is SEG_Y? Headers and Traces.

SEG_Y is open standard file format for storing geophysical ( eg: seismic ) data. These are stored on magnetic tapes and usually of several Gigs in size.

• Headers :

contains optional SEG_Y tape label.
next 3200 bytes contains EBCDIC headers.
next 400 bytes contains Binary headers.

• Traces :

Traces contains Trace Header and Trace Data.
first 240 bytes contain Trace header.
next 4004 bytes contain Trace data.

SharePoint 2010 Products configuration wizard Errors & Fix

1. Exception – Failed to create the configuration database. An exception of type System.Security.Cryptography.CryptographicException was thrown. Additional exception information: The data is invalid.
Resolution – This has two steps
Step 1: Make sure that the “Network Service” account has full access to the “14” directory under %commonprogramfiles\Microsoft Shared\Web Server Extensions.
Step 2: Delete the registry key located under “SOFTWARE\Microsoft\Shared Tools\Web Server Extensions\14.0\Secure\FarmAdmin” Registry key and then run the SharePoint 2010 Products Configuration Wizard.
It is likely that this registry key is required to be cleared each time you run the wizard after an unsuccessful attempt :)


2. Exception - Failed to register SharePoint Services. An exception of type System.Runtime.InteropServices.COMException was thrown. Additional exception information: Could not access the Search service configuration database.
I followed these steps and the configuration finished successfully.
1. On the Start menu, click Run. In the Open box, type regedit and then click OK.
2. In the Registry Editor, navigate to the following subkey, and then delete it:
HKEY_LOCAL_MACHINE\SOFTWARE\Microsoft\Shared Tools\Web Server Extensions\12.0\WSS\Services\Microsoft.SharePoint.Search.Administration.SPSearchService3.
 Run the SharePoint Products and Technologies Configuration Wizard again.

How to : working with HBase Delete API

• org.apache.hadoop.hbase.client.Delete

     HBase provide Delete to perform delete on a column(s), Column-Family(s) or entire Row, when Delete object is instantiated with a rowkey. 
     Delete accepts a Long Timestamp as parameter with Column-Family and a qualifier, which deletes all versions having smaller time-stamps. Delete creates a tombstone for any column or its version been deleted, HBase does the final deletion later when it goes for major compaction. 
IMPORTANT : If you try to 'put' data with the same timestamp which has been deleted recently, you'll not see it until HBase does its compaction. Though you'll not get any error or exception while doing  a 'put' but the same time you'll not see any result with 'scan' or 'get' until compaction happen. 
   If you don't provide a timestamp, default is current system time in milliseconds. 
Currently Update is not supported in HBase tables. A 'Delete' with 'put' is required to achieve this. If Update is on a column having multiple versions then timestamp plays critical role in maintaining the version order. Design your HBase schema accordingly :)

     To delete multiple rows or bulk delete, use 
public void delete(List<Delete> deletes)
            throws IOException method which is under HTable class.

Hadoop, HBase in Amazon Cloud (AWS)

Hadoop & HBase can be configured in Amazon Cloud to take the advantage of Distributed Computing where one can quickly start new instances as per requirement (or) load to an application at any time.


Elastic Compute Cloud  (EC2) : It provides easy access and configurable instances required to scale application easily.
Elastic Map Reduce (EMR) : It provides support for Hadoop to run Map-Reduce jobs on top of EC2 and S3 data storage.
Simple Storage Service (S3) : A persistent data storage service in Amazon Cloud with high availability.


Out of many running Amazon Machine Image (AMI), one acts as Namenode and rest as DataNode for Hadoop. NameNode or MasterNode contains HBase HMaster running on it which uses 'ssh', public-private-key to communicate to other SlaveNode. JobTracker talks to the NameNode and gets location of Data and it tells TaskTracker to perform the actual processing of Data on several DataNode/SlaveNode. HBase keep same data on many DataNode( equal to replication value defined HBase configuration) for faster access. When a table size/hit increases rapidly, it divides it into two to handle the bottleneck for any DataNode. 


Locally configured Hadoop, HBase can be converted to Amazon AMI and deployed to Cloud directly with any number of same instance. 

How to : working with HBase Filter API

• org.apache.hadoop.hbase.filter

        HBase provides Filter to perform searches for RowKey, Column Family, Column & Column Values. It could be anything from Binary comparator, Column Value, Prefix Filter, Regex to Timestamp filters etc.
A complete list is available at HBase Filter API documentation page.


You can create a FilterList which may contain number of FilterList as its child. Each Filter in a FilterList evaluates on either FilterList.Operator.MUST_PASS_ONE or FilterList.Operator.MUST_PASS_ALL .


Once Filter/FilterList is created  (eg: yourFilterList.addFilter(yourFilter); ), You should add it to Scan object (eg: scan.setFilter(yourFilterList); ). that's it. At this point you are ready to call getScanner method with HTable object instance. It returns you a iterable ResultScanner which contains number of rows matching your Filter criteria.