I have come across requirements where in I am supposed to generate the output in nested Json format.Below is a sample code which helps to do the same.The input to this code is a csv file which contains 3 columns . company name department employee name Example: google,jessica,sales google,sita,technology We… Read more »
In an earlier post of mine I had given an example of how to use VTD Xml.We could run into requirements where in we would like to use VTD xml in a multi threaded application.This post provides a working example for integrating VTD xml in a multi threaded application. The… Read more »
There would be occasions where in we would also need to know the name of the file that we are processing. We can use wholetextfile() which returns a PairRDD of filename and filecontents. wholetextfile() put whole contents of the file in to a single record, which means if we are… Read more »
There would be instances where in we are given a huge xml which contains smaller xmls and we need to extract the same for further processing.We may not be able to parse such Xmls using TextInputFormat, since it considers every line as a record, but in the xml below, our… Read more »
In SPARK 2, datasets do not have api like leftouterjoin() or rightouterjoin() similar to that of RDD.So if we have to join two datasets, then we need write specialized code which would help us in achieving the outer joins. To the join API we need to pass the join type… Read more »
Below is an example of partitioning the data based on custom logic. For writing a custom partitioner we should extend the Partitioner class , and implement the getPartition() method.For this example I have a input file which contains data in the format of <Continent,Country>. I would like to re-partition the… Read more »
VTD-XML is a good alternative to Simple API for XML (SAX) and Document Object Model (DOM), as it does not force you to trade processing performance for usability. The Java-based, non-validating VTD – XML parser is faster than DOM and better than SAX.Unlike other XML processing technologies, VTD-XML is designed… Read more »
There would be instances where in we do not have direct access to Database server running in an external network, i.e we cannot open connection to the database directly from our local machine .Usually to access these databases we will have to login to another server(Bastion host) thru putty to… Read more »
By default SPARK reads text files with newline(‘\n’) character as the Record delimiter.But there could be instances where in record delimiter is some other character, for eg: CTRL+A (‘\001’) or a Pipe(“|”) character. So how can we read such files? We can set the textinputformat.record.delimiter parameter in the Configuration object… Read more »
Given a squared sized grid of size in which each cell has a lowercase letter. Denote the character in the the row and in the th column as G[i][j]. You can perform one operation as many times as you like: Swap two column adjacent characters in the same row G[i][j]… Read more »