Programming Questions? Ask a Programmer for Answers ASAP
My client is giving us access to a large database in
My client is giving us access to a large database in oracle and we have to audit the quality of the data. They have requested us to use java so that we are done with the audit they can have they team continue to audit as business rules change. Looking for a Java library that would make it easier to do the basic testing. IE * valid range of numbers dates etc for each file. * Does the unique key in the transaction file exist in the master table etc * Name field in multiple tables are the same Business rules check the sequence of transactions in the transaction file to make sure they conform to a limited list of valid sequences
Thanks for using JustAnswer.
This is Leela and I will help you with the question today.
I understand that you want to analyze DATA from big data base on some business rules.
If you can provide more details on the VOLUME of data, like number of records, number of tables, I can help you with details on analysis.
About 1000 tables with 16000 fields and total volume is about 200 terabytes.
Wow. That is really a huge data. You would need to use HADOOP or any other Big Data system in order to LOAD and ANALYZE the data.
Will you be doing a GENERAL analysis or looking for specific insights in the data?
I agree with your thoughts. Can you please elaborate a bit more on what do you mean by VERIFYING data? Is it looking for missing column values in a row or anything else?
We have java libraries to connect to databases, we can easily use them to run SQL queries and find some insights, but that is going to take lot of time and you would need HIGH END servers to run the program.
When I said Hadoop system I mean MapReduce which is a java program or you can use Spark also to analyse the data stored in Hadoop.
First I want to help you with the Java library question.. please share more details on the data validation part.
If needed we can have a quick call for a quick discussion, please let me know.
Thanks for sharing the details, I would check and get back to you in sometime.
I have analyzed your needs, as you are are looking for a java library, you can use Hibernate Validator in order to validate the data as per the specific constraints you have.
Please check and let me know if this helps.
The Hibernate.org does look good for most of what I need to do.
However the most complex of the test involves looking at several transaction tables that have a type of transaction. There are certain valid transaction series and others that I am not sure what they really mean. How can I get a list of all valid transactions sequences and also the ones that are not valid. with a software tool?
Glad you see value in hibernate, it is a great offering for java user to interact with databases.
With regards ***** ***** question about VALID TRANSACTION.. A transaction is just a ROW in a database table, whether a row is VALID or not is defined by some BUSINESS RULES and not a GENERAL logic. So i am not very sure if there can be any rule which can automatically find if a RULE is VALID or not.
Mostly you can use Hibernate Validator and define your business rules and try to validate them.
Hope this helps.
By business logic I meant the way you tried to define a VALID transaction, as per my understanding there is no automatic way, but would require some programming or at least some definition of business rules.
To help further, we can use additional services to discuss more over a phone call as you may not want to provide sensitive details in chstchat. Please let me so that we can work accordingly.