Technical Manual (TM)

 Technical Manual (TM)Geocode Data ReaderTeam #10Anamarie HaSoftware Architect / UML ModelerAshi RankaFeasibility Analyst / PrototyperDamini CousikImplementer / Quality Focal PointKarthik Kenkere SreedharamurthySoftware Architect / ImplementerPawandeep GillRequirements Engineer / PrototyperSriramkumar ThamizharasanImplementer / Project ManagerSuchetha I BhatProject Manager / Operational Concept EngineerVidhubala SelvarajLife Cycle Planner / Tester11/15/2020Table of Contents TOC \h \u \z \n Technical Manual (TM)IntroductionAWS ComponentsData Storage and ManagementGeocode_databasesuggestionTablecountry_geocode_tablerecovery_tableactivity_tableLambda FunctionsAPI ManagementUser AuthenticationSimple Email Service (SES)Frontend DevelopmentSource Version ControlIntegration, Build, and DeploymentDomain ManagementIDEIntroductionThis document provides the technical specifications of the development of the GEOCODE Data Reader project developed by Team 10 of CSCI 577a Software Engineering course at USC Viterbi School of Engineering. The below sections will provide a detailed breakdown of the technical components starting from Data Storage, Lambda Functions, APIs management, Authentication module, frontend module etc.AWS ComponentsPrimary Account holder: Matt BemisAccount Login ID: wbemis@usc.eduAccount Password: geocodeThatworks!Account access level: Root userRegion: US East (N. Virginia) us-east-1Once logged in, use the search bar to search for various services to select and use accordingly.The following table lists all the components of AWS used in the system and the purpose.ponent NamePrimary Usage1.Amazon DynamoDBData Storage2.AWS LambdaBackend lambda functions3.Amazon API GatewayAPIs for frontend backend communication4.Amazon CognitoUser authentication5.AWS AmplifyContinuous integration and Deployment6.AWS CodecommitCentral git-based source version control7.AWS Simple Email Service (SES)Sending and Receiving emails8.AWS Cloud9Online AWS IDEData Storage and ManagementThe application uses Amazon DynamoDB (NoSQL DB) to store and manage data as tables. The below table lists all the tables used for the application and the reason for usage.No.Table NamePurpose1.Geocode_databaseTable to store every institution and corresponding 31 details in the GEOCODE database2.suggestionTableTable to store the suggestions made by a user (add/edit)3.country_geocode_tableTable storing the last GEOCODE assigned to a country4.recovery_tableTable storing the deleted institutions for 30 days5.activity_tableLogs all the transactions done on the application for logging and analytics purposesGeocode_databasePartition key: Institution_Name_lowerFaster search and allows case insensitive search on the field of Institution Name.Primary Sort Key: PESC CodeAllows to uniquely identify each record. Helps with institutions with the same name and different PESC CodeOther fields:Institution Code from Data Source , Status , Data Source Date , Data Source File Link , Other Code Type2 , Other Code Type3 , Other Code Type1 , Institution Name , Institution Category , Status Notes , Name of Institution3 (Local Language) , WWW , Other Code Name3 , Postal Address 2 , State/Province (Local Language) , PESC Code , Other Code Name1 , Other Code Name2 , Postal Address 1 , Phone , City , Name of Institution2 (Local Language) , Data Source , Name of Institution1 (Local Language) , Establishment , Email , Postal Code , LastUpdated , State/Province , Status Reason , Country Code , State/Province CodesuggestionTablePartition Key: suggestionIDUniversally unique identifier (UUID) to uniquely identify every suggestion.Other fields:Institution Code from Data Source , Status , Data Source Date , Data Source File Link , Other Code Type2 , Other Code Type3 , Other Code Type1 , Institution Name , Institution Category , Status Notes , Name of Institution3 (Local Language) , WWW , Other Code Name3 , Postal Address 2 , State/Province (Local Language) , PESC Code , Other Code Name1 , Other Code Name2 , Postal Address 1 , Phone , City , Name of Institution2 (Local Language) , Data Source , Name of Institution1 (Local Language) , Establishment , Email , Postal Code , LastUpdated , State/Province , Status Reason , Country Code , State/Province Code , ApprovalStatus , Approver , Suggested Email , Suggested Usernamecountry_geocode_tablePartition Key: Country CodeUnique ISO Code to to identify each country and get/put last assigned GEOCODEOther fields:PESC Coderecovery_tablePartition Key: Institution NameTo identify the institution by name and move it to Geocode_database if recovered or remain in recovery_table for 30 days till automatic deletion.Other fields:Institution Code from Data Source , Status , Data Source Date , Data Source File Link , Other Code Type2 , Other Code Type3 , Other Code Type1 , Institution Name , Institution Category , Status Notes , Name of Institution3 (Local Language) , WWW , Other Code Name3 , Postal Address 2 , State/Province (Local Language) , PESC Code , Other Code Name1 , Other Code Name2 , Postal Address 1 , Phone , City , Name of Institution2 (Local Language) , Data Source , Name of Institution1 (Local Language) , Establishment , Email , Postal Code , LastUpdated , State/Province , Status Reason , Country Code , State/Province Code , ExpirationTime , Timestampactivity_tablePartition Key: transaction_idUniversally unique identifier (UUID) to uniquely identify every transaction to be stored for 90 days.Other fields: email id , ExpirationTime , Timestamp , transaction_type , usernameLambda FunctionsThe application uses AWS Lambda functions to interact with the backend storage tables and serves request events sent from frontend using API calls.The below table indicates all the lambda functions used by the application, its purpose and the language used.No.Lambda Function NamePurposeLanguage1.GEOCODE_PutdataPuts a new record into either Geocode_database or suggestionTableNodeJS2.GEOCODE_Recovery_get_all_dataGets all rows in recovery_tableNodeJS3.GEOCODE_getDataTakes input of required search fields and filters and returns a list of JSON objects.NodeJS4.GEOCODE_putalldataTakes CSV string of file and parses it to perform bulk upload to Geocode_databasePython 3.85.GEOCODE_suggest_getalldataGets all rows in suggestionTableNodeJS6.NEW_PUTALL_DATAUploads a CSV file of records to Geocode_database. Function to refresh DB. Has to be run for every 5000 rows at a time.Python 3.87.country_countGets the count of records for a given ISO Country CodeNodeJS8.deleteSuggestionDeletes a row (suggestion) in suggestionTable given transaction_id. Used for reject suggestion functionality.NodeJS9.delete_from_geocodePuts record in recovery_table. Adds a TTL field for current date plus 30 days. Deletes record from Geocode_database.NodeJS10.get_all_activityGets all records in activity_table and returns it as list of dictionary objects.NodeJS11.get_country_geocodeGets the last assigned PESC Code for a given Country Code from country_geocode_tableNodeJS12.log_transactionPuts a record into activity_tableNodeJS13.markSuggestionApprovedUpdates a suggestion ApprovalStatus to true and updates the Approver field with the name of the admin that approved.NodeJS14.populate_country_geocodePopulates country_geocode_table with Country Code with the last assigned to PESC Code. To populate the table for the first time.NodeJS15.put_country_geocodeUpdates the latest PESC Code assigned to Country Code in country_geocode_table.NodeJS16.restore_institutionRestores institution in the recovery_tablel to Geocode_database. Removes the TTL field and puts in Geocode_database.NodeJS17.sendEmailToAdminSends an email to RECEIVER from SENDER (geocodeinfo@) using aws-ses SDK with Name, Phone, Email ID and Description from the contact us form.NodeJSLibraries used in Lambda functions:boto3 - Python library to interact with DynamoDB used for bulk upload functionalityaws-sdk - NodeJS library to interact with all AWS services. In the application, used to interact with DynamoDB for search, get, put, update, delete functionalities. Also, Simple Email Service functionality is included in the above library.API ManagementThe application uses Amazon API Gateway to manage POST request API calls from the frontend and triggers the appropriate Lambda function it is attached to.The below table lists all the APIs used by the application and the lambda function it is attached to.No.API NameAttached Lambda Function1.putDataAPIGEOCODE_Putdata2.getRecoveryBinDataAPIGEOCODE_Recovery_get_all_data3.getDataAPIGEOCODE_getData4.putAllDataAPIGEOCODE_putalldata5.suggestGetAllDataAPIGEOCODE_suggest_getalldata6.getInstitutionCountAPIcountry_count7.deleteSuggestionAPIdeleteSuggestion8.moveToRecoveryAPIdelete_from_geocode9.get_all_activityAPIget_all_activity10.get_country_geocodeget_country_geocode11.logTransactionAPIlog_transaction12.markSuggestionApprovedAPImarkSuggestionApproved13.put_country_geocodeput_country_geocode14.restoreInstitutionAPIrestore_institution15.sendEmailsendEmailToAdminUser AuthenticationThe application uses Amazon Cognito to store and manage user data of the application.Pool Name: GEOCODE Data Reader User PoolPool ID: us-east-1_kq8IYjRf9Client ID: kuqhuq37lh92f8dnthbuhlpboGroups: AdministratorsAll registered users need to be confirmed using Confirmation Email Link sent to the email of the user. All registered users have general user privileges. When manually added to the Administrators group, permissions are managed in frontend Javascript for administrator pages and functionalities.Steps to manually add registered and confirmed user to Administrators group:Login to AWS dashboard using the login credentials in AWS Components.Search Cognito and navigate to the Cognito dashboard.Select Manage User PoolsSelect GEOCODE Data Reader User PoolSelect the Users and Groups subsection under General settings section in the left-side nav bar.Search for user based on email, name etc. select username field to get all details of user.Select Add to Group.Select dropdown and select “Administrators” and click Add to group.Verify that Administrators tag is added to Groups field in user field.Steps to view all users in the Administrators group:Follow steps 1 to 4 in previous instruction set.After selecting Users and Groups subsection, click on Groups tab.Select the group name Administrators.Scroll and view second card with table of all users in Administrators group.Steps to delete an user:Follow steps 1 to 6 in first instruction set.Click Disable user to disable the user account.Click the Delete user button in red to delete user from user pool.Simple Email Service (SES)The contact form input is sent to a lambda function that sends an email to the inbox maintained by the client.Email ID: geocodeinfo@Password: cryformeG1!Instructions to add an email to SES and verify:Login to AWS dashboard using the login credentials in AWS Components.Search Amazon Simple Email Service and navigate to the SES dashboard.Under the Identity management section of the left-side nav bar, choose Email AddressesSelect Verify a new email address and enter the email ID to be verified.An email will be sent to the email address for verification.Copy the link given in the email body and paste it into a new tab in the browser (or) Click on verification link given in the email body to verify email address.Verify that the email address has been verified in the verification status column in SES dashboard.Instructions to change the mailbox email address:Navigate to AWS Lambda from the AWS dashboard by searching for the same.Select the Lambda function sendEmailToAdmin.Change RECEIVER (Line 16) and SENDER (Line 17) to email address.Frontend DevelopmentThe frontend is developed using HTML5, CSS/CSS3 and Javascript. The development is done on AWS Cloud9 Online IDE.The below table shows all the HTML files and the intended purpose.No.PagePurpose1.activity_log.htmlHTML page with paginated table of all records in activity_table. Redirected from piechart in admin.html2.admin.htmlAdmin home page. Landing page for admin. Contains pie chart of usage statistics. Pie chart redirects to activity_log.html3.adminAddInstitution.htmlAdmin add institution form page.4.bulkUpload.htmlBulk Upload page. Contains button to download template CSV file and file upload functionality.5.contactus.htmlContact form page for the registered user with fields Name, Email, Phone and Description6.forgotPassword.htmlPassword reset page for user that forgets password. Redirected from login.html on click of Forgot Password button.7.guest.htmlGuest search page containing search bar and filters and option to redirect to guestContact.html. Paginated table records with search results.8.guestContact.htmlContact form page for the guest user with fields Name, Email, Phone and Description9.index.htmlLanding page of the application. Contains world map for visualization, login button on navbar and option for guest users to search.10.login.htmlLogin Page redirected from index.html on click of Login or redirect from Sign up and forgot Password reset.11.pendingTasks.htmlContains all records in activity_table in paginated form. On load page, API to get all records in activity_table called and records table populated.12.recovery.htmlContains all records in recovery_table in paginated form. On load page, API to get all records in recovery_table called and records table populated. Button to populate modal with institution details to be recovered.13.register.htmlContains the Sign up functionality form with fields Name, Email ID, Password, Confirmation password to register a new user using Cognito. Popup to alert user to verification email.14.user_add_institution.htmlContains form for suggestion addition functionality for registered user. Popup to alert to successful addition of suggestion.15.userViewInstitution.htmlContains the user search functionality with search bar and filters. Paginated table to represent search result. Button to make suggest edit modal form visible and appropriate fields populated. Popup to alert to successful addition of suggestion.16.viewInstitution.htmlContains the admin search functionality with search bar and filters. Paginated table to represent search result. Buttons to make edit modal/delete modal forms visible and appropriate fields populated. Delete modal needs password confirmation.The below table shows all the Javascript files and the intended purpose.No.Javascript FilePurpose1.activity.jsOn load of activity_log.html gets all records in activity_table and populates records table2.add_suggestion.jsCreates UUID for every suggestion and gets all the details in the form in user suggest edit (in userViewInstitution) or user_add_instituion and inserts into suggestionTable. Also creates a record with a UUID in activity_table with “suggest”3.add_uni.jsTakes the input fields from the adminAddInstitution form and pushes it to Geocode_database. Also creates a log with an UUID with “add”.4.admin_edit_delete.jsFills the edit modal form and the delete modal with the institution details of the button clicked on viewInstitution (admin search page). Edit function updates record in Geocode_database with form_data fields in edit modal form. Set Inactive updates the record in the delete modal with Status set to “Inactive”. delete_uni function calls the API that deletes the institution from Geocode_database and moves it to recovery_table. Both set_inactive and delete_uni takes admin password and verifies identity of admin for double verification.5.admin_suggestion.jsApprove_suggestion function fills the suggestion details modal form in the pending tasks page based on the details of the institution of the button clicked (Highlights changed details from Geocode_database record in red). Approve function updates the institution details or provides a new PESC Code based on the last PESC Code given to country (set_PCode function) and adds the institution to Geocode_database. Reject calls the API that deletes the suggestion identified by suggestionID from suggestionTable. Both the operations are logged in activity_table with UUID as “approve” and “reject”.6.amazon-cognito-identity.min.jsAmazon Cognito Identity SDK file for JS provided by AWS. For the Cognito header functions and variables.7.aws-sdk-2.771.0.min.jsAWS SDK file for Cognito helper functions built from AWS SDK builder for JS.8.bulk_upload.jstemplate_CSV function creates a template with the first row as the columns to be entered. Generates a download of the same when Download Template button is clicked in bulkUpload.html file. send_bulk_file function reads the CSV into a string and calls API to put all rows in CSV file into Geocode_database. Also creates a record in activity_table with UUID and “upload”.9.cognito_helper_functions.jsContains the functions for user management from the frontend JS functionality. Register function registers a new UNCONFIRMED user using Cognito. A verification email is sent to the email ID that is used to register with a verification link. signIN function logs in a registered user and based on the group the user belongs to redirects to either userViewInstitution.html(USER) or admin.html(ADMIN). signOut function logs the o=user and redirects to index.html. forgotPassword function calls the function to send a verification code to the email that is registered and redirects to forgotPassword.html to enter code and new password. confirmPassword resets the password with new password and uses verification code to verify the action and redirects to login.html.10.config.jsContains the configuration of UserPoolID and ClientID and region of the Cognito User Pool for the application to facilitate all cognito authentication functionalities.11.contact_us.jssubmitToAPI function gets all the fields inserted in contact form and verifies the fields entered and calls the API that calls the sendEmailtoAdmin Lambda function to send an email to the inbox geocodeinfo@12.dashboard.jsCalls API to get all records in activity_table and count of each type of activity. Populates Amcharts Pie chart with the count of activity and redirect on click to activity_log.html with detailed activity log.13.export.jsContains search functionality and conversion of search result to CSV or JSON format. JSONToCSVConverter function converts to UTF-8 BOM format to support local language fields. JSONConverter function creates a JSON file of the same search result. Both the functions create a log in the activity_table with “export”.14.index_graph.jsCreates the world map in index.html page and on click of a country calls API to get count of records in Geocode_database with particular ISO Country Code. Populates tooltip with Name of country and count.15.main.jsContains helper functions for frontend design elements such as password visibility control, side navigation bar and toggling filter menu.16.recover.jsContains frontend JS functions necessary for the recovery bin feature. Fill_recover function fills the modal form with details of the institution based on the button clicked in the records table in recovery.html. Recover function calls the API to restore the particular record from recovery_table to Geocode_database.17.search_uni.jsContains the frontend JS functions for admin, user and guest search functions. Admin_search_uni function gets the fields filled in viewInstitution, userViewInstitution and guest files and calls the API to search based on filters. Populates records table in all the pages in appropriate format. Admin_search_uni adds 2 additional buttons to facilitate edit and delete functionalities. User_search_uni function adds 1 addition button to allow suggestion of edit functionality. Guest function has no other additional fields in the search functionality. All records are put into Datatable format to allow pagination and in-table search.18.suggest_edit.jsSuggest function fills in details of the institution clicked in the userViewInstitution file into modal form to allow suggest edit functionality.css/main.css file contains all design CSS style elements for elements across all the HTML pages.Source Version ControlThe source code is stored and the version control is performed using AWS Codecommit.HTTPS Link: Link: ssh://git-codecommit.us-east-1.v1/repos/geocodeDataReaderDeployment branch: masterDevelopment branch: feature-mainPageInstruction to clone Codecommit repository:(HTTPS) git clone geocodeDataReader(SSH) git clone ssh://git-codecommit.us-east-1.v1/repos/geocodeDataReader geocodeDataReaderFurther instructions to connect to codecommit repository can be found here: , Build, and DeploymentAWS Component used: AWS AmplifySince the deployed branch is the master branch of the repository, any change committed and pushed to the master branch of the repository, will trigger the integration and build process of AWS Amplify. The four stages are Provision, Build, Deploy and Verify.Instructions to check provision and build process:Login to AWS dashboard using the login credentials in AWS Components.Search AWS Amplify on the search bar in the dashboard.Select geocodeDataReader application in the dashboard.Verify the process of build and deployment in the application console.Select each stage of deployment to see details of each stage.Verification of application on various screens can be done in the fourth stage Verify.Domain ManagementThe deployed application is hosted to a custom domain setup using Route 53. The domain is currently: to check domain management:Follow steps 1 through 3 in the previous instruction set.Select the domain management sub section under App settings in the left side nav bar.Check status of the Custom domainTo delete, select actions and click delete.IDEAWS IDE Cloud 9 used for modification of frontend code.Steps to access code and edit in Cloud9:Login to AWS dashboard using the login credentials in AWS Components.Search Cloud9 on the search bar in the dashboard.View environment geocode-webapp-development and click on Open IDE.You will be redirected to Cloud9 instance of the repository with bash terminal access. ................
................

In order to avoid copyright disputes, this page is only a partial summary.

Google Online Preview   Download