Author Archives: baahu

SPARK : How to generate Nested Json using Dataset

I have come across requirements where in I am supposed to generate the output in nested Json format.Below is a sample code which helps to do the same.The input to this code is a csv file which contains 3 columns . company name department employee name Example: google,jessica,sales google,sita,technology We… Read more »

VTD Xml Example

baahu   January 23, 2017   2 Comments on VTD Xml Example

VTD-XML is a good alternative to Simple API for XML (SAX) and Document Object Model (DOM), as it does not force you to trade processing performance for usability. The Java-based, non-validating VTD – XML parser is faster than DOM and better than SAX.Unlike other XML processing technologies, VTD-XML is designed… Read more »

SPARK: Java code to Read files with Custom Record Delimiter

By default SPARK reads text files with newline(‘\n’) character as the Record delimiter.But there could be instances where in record delimiter is some other character, for eg: CTRL+A (‘\001’) or a Pipe(“|”) character. So how can we read such files? We can set the textinputformat.record.delimiter parameter in the Configuration object… Read more »