1. DILV -Data Integrity and Lifecycle Validator
Savio Fernandes & IT Manager
General Mills India Pvt. Ltd.
Logo of your organization

2. Abstract
Data is everywhere-In today’s world of data explosion, big data applications and their implementations are growing dramatically.
Data is changing- abilities to innovate and change the course in a short span is desired. It directly impacts not just the bottom line by saving costs but also changes the top line revenue generation.
Today data brings challenges of 4V’s (Volume, Variety, Veracity and Velocity). General Mills has put in considerable investment in ensuring that it uses right set of data at right point of time to deliver “Right Insight”.
Testing team was assigned the role of ensuring quality in “Big Data Test”. Test very high volume of data with variety and veracity was a big challenge because manual testing would only test 5-7% of entire data set and by attempting to increase the coverage, impacts the time to market.
DILV addresses these challenges ensuring data is tested with 100% accuracy which builds customer confidence in the data quality.

3. Challenges Faced before DILV
Test very high volume of data with variety and veracity was a big challenge because manual testing would test 5-7% of entire data set and by attempting to increase the coverage impacts the time to market.
Validation across conventional and unconventional sources (Flat file, data marts, oracle, sql server and Hadoop
Complexity in connecting various data sources and bringing onto common platform.
Generate exception report for users to understand and troubleshoot data issues
Existing Tools were concentrated on data mining & analysis with no specific test support
Specific Test tools had DW testing ability connecting at a time to only one data source.

4. Concept of DILV The concept behind the tool was
To get the source and target on a common platform where each data validations could be applied seamlessly irrespective of its data complexity.
Provide user friendly option to the testers to define their complex test scenario across varied platform as well as provide option for expert users to perform their task using SQL Query by hiding all the complexity in the background.
Provide facility for users to define test suite to run the test unattended or schedule jobs to achieve data DevOps.
To provide offline capability for processing huge data set.
Generate user friendly reports which could be easily understood by the testers.

5. The concept behind the tool was
Concept of DILV
The concept behind the tool was
To get the source and target on a common platform where each data validations could be applied seamlessly irrespective of its data complexity.
Provide user friendly option to the testers to define their complex test scenario across varied platform as well as provide option for expert users to perform their task using SQL Query by hiding all the complexity in the background.
Provide facility for users to define test suite to run the test unattended or schedule jobs to achieve data DevOps.
To provide offline capability for processing huge data set.
Generate user friendly reports which could be easily understood by the testers.

6. Impact of DILV
Validation across conventional and unconventional sources (Flat file, data marts, oracle, sql server and Hadoop)
100% accuracy on the selected very large sample size
Integration with DevOps
Drives 60% of savings in testing timelines
ROI more than 150%

7. References & Appendix
This Concept is an innovation to facilitate Data Testing.
This tool is in-house developed, tested and implemented.
There is no external reference for this tool.

8. Author Biography
Multi-skilled IT Professional with over 20+ years of versatile experience across diverse organizational domain. Very capable with a proven ability to Lead large size software development projects as well as provide IT Services that will improve the efficiency and performance of a company. Extensive experience of working at different levels in the organization and performing various roles from Software Engineer, Technical Architect, Project Manager, Operations Manager to Manager of People.

9. Logo of your organization
Thank You!!!
Logo of your organization