BIGDATA SIMULATION LIBRARY OF DREAM BY SANDMAN BOOKS
=================
Solution Architecture
Description
In the realm of Dreaming, its ruler SANDMAN, DREAM has a certain hobby; books. In his castle there is a Library in which they are kept, among other things, stories conceived by their authors but never written in our reality; Lucien, the person responsible for his organization, needs some help. Many people dream of published books, sales markets, stories that, in their reality, they would never imagine conceiving. And this voluminous data needs to be worked on. In order not to get lost in the information, Lucien receives all his dreams in a Non-Relational bank, MONGO. And he needs this to be organized in a relational way, that is, each author in his proper place. For that he pulled our dream and saw this Architecture where data arrives in MONGO undergo a transformation process in the STAGIN area and are populated in MYSQL. In its population, we split two final tables. One in its raw state, for complete queries, and another with metrics that informs the number of dreamers, their books and the total number of files. In this way, data is more organized, undergoing deduplication and consolidation processes.
Glossary of Data
| Fields | Type | Description |
|---|---|---|
| _id | long | undescore ID |
| kind | string | type of book or text file |
| title | string | title of book |
| subtitle | string | subtitle of book |
| author | array | one or more authors who can dream of stories |
| publisher | string | publisher or not dreamed of by the author |
| publishedDate | string | year of published |
| edition | string | which edition does the book belong to |
| sample | string | sample of books |
| type | string | ISBN code |
| identifier | string | isbn identification number |
| pageCount | integer | number of pages |
| capCount | integer | number of chapters |
| wordCount | integer | number of words |
| categories | string | literary genre |
| original_price | double | original price |
| current_prefix | string | country currency prefix |
| current_sufix | string | country currency name |
| barcode | string | barcode |
| dreaming_date | string | the day you had the dream |
image
Start the Project
To run the project, you need to install the dependencies located in the "dependencies" folder and in the root of the project, run the shell_script "run_script.sh".
Sample of Payload MONGO
mongo
{
"_id" : ObjectId("61b1fe6944dd42158674af31"),
"kind" : "books#volume",
"volumeInfo" : {
"title" : "STORE VISIT",
"Subtitle" : "GLASS IT HAIR MEMBER KEY ALMOST QUALITY. MARKET ALREADY AIR STILL ARTICLE. DECADE DECADE MEASURE PRESENT HUMAN MORNING. BIG BLOOD ECONOMIC FRONT SUCCESS AGO THEM. EVERY SON TROUBLE SIMPLE.",
"author" : [
"PETER RODRIGUEZ",
"KELLY TORRES"
]
},
"publisher" : "FALL AWAY ABOUT INDEPENDENT",
"publishedDate" : "1994",
"edition" : "7º EDITION",
"sample" : "...onto sport room audience. page dinner hundred. week statement should watch she even ball.\nour able tv break defense seek baby. employee last around music produce reach tv..",
"industryIdentifiers" : [
{
"type" : "ISBN_10",
"identifier" : "1-55027-208-X"
},
{
"type" : "ISBN_10",
"identifier" : "0-405-30324-6"
}
],
"pageCount" : 796,
"wordCount" : 83331,
"capCount" : 14,
"categories" : [
"NOVEL"
],
"saleInfo" : {
"original_price" : 78,
"current_prefix" : "LAK",
"current_sufix" : "Lao kip",
"barcode" : "6747254889534"
}
}
Sample of Payload in MYSQL
library
_id |kind |title |subtitle |author |publisher |publishedDate|edition |sample |type |identifier |pageCount|wordCount|capCount|categories |original_price|current_prefix|current_sufix |barcode |dreaming_date|
-----+------------+----------------------------------------------------------------+---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+---------------------------------------------+----------------------------------------+-------------+----------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-------+-----------------+---------+---------+--------+-------------------------+--------------+--------------+---------------------------+-------------+-------------+
670|csv#volume |GOOD DETERMINE OF |DON'T HAVE |KAREN ODOM |DARK WAR INDEPENDENT |1982 |9º EDITION|...involve star apply later including truth. next while nor worry staff economic.¶condition region write college. return half offer. popular could direction above fish.. |ISBN_10|978-1-4340-7508-6| 479| 64333| 11|EPISTOLARY NOVEL | 97.0|BHD |Bahraini dinar |3894426059691| 20211209|
resume
metric |value|
----------------+-----+
Total of Dreamns|71395|
Total of Books |59154|
Total of Data |78000|
1 Jan 12, 2022
65 Dec 16, 2022
1 Jan 27, 2022
1 Jan 02, 2022
10 May 10, 2022
1 Nov 01, 2021
28 Nov 27, 2022
23 Dec 08, 2022
971 Dec 25, 2022
5 Dec 01, 2022
1 Feb 04, 2022
2 Nov 27, 2021
180 Dec 18, 2022
6 Jul 01, 2022
1 Jan 06, 2022
72 Jan 05, 2023
67 Dec 27, 2022
3 Nov 26, 2022
3.3k Jan 04, 2023
8 Oct 18, 2022