Below is a simple python code to list files under a specific directory in s3 bucket. You should have boto3 installed.
Below is a simple code to get master dns of a EMR cluster.
Below is a simple python based Kafka producer which reads data from twitter and puts data into kafka topicYou will have to register with twitter to get tweets streamed into this app.After registration you will have your own access_tokens,access_token_secret,consumer_key,consumer_secret . Install tweepy and twitter libraries using below command pip install… Read more »
Below is a simple Spark Streaming application which reads data from kafka topic and prints the content. I have installed kafka in my laptop. auto commit of offset is disabled since I am committing the offset to kafka using commitAsync API. Every batch of data will contain 1 second worth… Read more »
I have come across requirements where in I am supposed to generate the output in nested Json format.Below is a sample code which helps to do the same.The input to this code is a csv file which contains 3 columns . company name department employee name Example: google,jessica,sales google,sita,technology We… Read more »
There would be occasions where in we would also need to know the name of the file that we are processing. We can use wholetextfile() which returns a PairRDD of filename and filecontents. wholetextfile() put whole contents of the file in to a single record, which means if we are… Read more »
There would be instances where in we are given a huge xml which contains smaller xmls and we need to extract the same for further processing.We may not be able to parse such Xmls using TextInputFormat, since it considers every line as a record, but in the xml below, our… Read more »
VTD-XML is a good alternative to Simple API for XML (SAX) and Document Object Model (DOM), as it does not force you to trade processing performance for usability. The Java-based, non-validating VTD – XML parser is faster than DOM and better than SAX.Unlike other XML processing technologies, VTD-XML is designed… Read more »
There would be instances where in we do not have direct access to Database server running in an external network, i.e we cannot open connection to the database directly from our local machine .Usually to access these databases we will have to login to another server(Bastion host) thru putty to… Read more »