University of Edinburgh



School and Programme Quality SystemTechnical ArchitectureVersion 1.207/02/2017Contents TOC \o "1-2" \h \z \u 1Project plan PAGEREF _Toc474230261 \h 41.1HSS014 CAHSS School and School and Programme Quality System PAGEREF _Toc474230262 \h 42Service design PAGEREF _Toc474230263 \h 62.1Service description PAGEREF _Toc474230264 \h 62.2Resilience measures PAGEREF _Toc474230265 \h 82.3Disaster recovery category PAGEREF _Toc474230266 \h 82.4Backup policy PAGEREF _Toc474230267 \h 92.5Security issues PAGEREF _Toc474230268 \h 92.6Authentication and authorisation PAGEREF _Toc474230269 \h 92.7External access PAGEREF _Toc474230270 \h 92.8Interfaces and dependencies PAGEREF _Toc474230271 \h 92.9Exceptions and other issues PAGEREF _Toc474230272 \h 93Service specification PAGEREF _Toc474230273 \h 103.1URLs, certificates and channels PAGEREF _Toc474230274 \h 103.2Servers PAGEREF _Toc474230275 \h 133.3Users, roles and groups PAGEREF _Toc474230276 \h 143.4Data sources PAGEREF _Toc474230277 \h 153.5Firewall configuration PAGEREF _Toc474230278 \h 153.6Scheduled tasks PAGEREF _Toc474230279 \h 153.7Software licences PAGEREF _Toc474230280 \h 154Service operation PAGEREF _Toc474230281 \h 164.1Support contacts PAGEREF _Toc474230282 \h 164.2Startup and shutdown steps PAGEREF _Toc474230283 \h 164.3Log files PAGEREF _Toc474230284 \h 164.4Configuration files PAGEREF _Toc474230285 \h 164.5Patching PAGEREF _Toc474230286 \h 165Common procedures PAGEREF _Toc474230287 \h 185.1Start an individual Gunicorn manually PAGEREF _Toc474230288 \h 186Disaster recovery plan PAGEREF _Toc474230289 \h 19Version controlDateVersionAuthorSectionsAmendments10/11/20161.0Riky HarrisAllInitial draft31/01/20171.1Riky HarrisVariousUpdated URLs and certs07/02/20171.2Riky HarrisVariousUpdates from team feedbackProject planHSS014 CAHSS School and School and Programme Quality SystemStakeholdersRoleUnitNameTechnical Architect Development TechnologyRiky HarrisPeer ReviewerDevelopment TechnologyProject ManagerProject ServicesBen ArmstrongProduction RepresentativeProduction ManagementHeather LarnachITI RepresentativeIT InfrastructureKey deliverablesDeliverableBusiness benefitUse existing Python platform to host the applicationsAdds consistency and best practice to application hostingUse Puppet for application tier configurationEnsures consistent base for deployment by BambooTechnical commitmentsCommitmentY/NJustification (if not)Will the project conduct a load test?NWill the project conduct a DR test?NBeing delivered as part of INF123 Will a service restart be tested?YSummary of technical changesThe HSS014 project is for the initial development and deployment of the School and Programme Quality System. This consists of a UI and a microservice, and both use the shared Python hosting platform.Estimated costsThere are no additional costs for the hosting of the service, as the existing shared Python platform and existing database infrastructure are used.Service designService descriptionThe School and Programme Quality System is built from a Python and Django front-end UI application and a back-end Python and Django microservice, and is used by HSS staff to access the annual report writing processes for quality monitoring at a Programme level and quality reporting at a School level. This could later be expanded out to be used by all colleges.Both applications are active-active hosted on the shared Python platform (see the Python Infrastructure TAD for more details). This follows the standard deployment for Python applications by being configured with Puppet and built, tested and deployed by Bamboo from Git. There is a TRN tier as well as DEV, TEST and LIVE, which is hosted as separate applications on the TEST infrastructure.There is no dedicated database for the application, but the required data is accessed by the microservice directly from the STAR* databases via a specific functional account.Key technologiesTechnologyVersionNew or existingPython3.4 (via SCL)ExistingDjangoPer applicationExistingGunicornPer applicationExistingApache2.4ExistingCentOS7ExistingPuppet clientExistingTechnical diagramsNB: This diagram refers to the similarly architected Estates Operational Reports service.Resilience measuresThe applications are hosted on two application servers in an active-active configuration.Disaster recovery categoryApplicationCategorySchool and Programme Quality System3Backup policyComponentVariance from backup policyOperating System Standard backupDatabaseStandard backupFile systemStandard backupSecurity issuesNone.Authentication and authorisationThe UI application is using Cosign for authentication, with Central Auth to authorise the user as a member of staff.The microservice uses token-based OAuth for authorisation of consumers and publishers of the service, including the UI.External accessNameContact detailsAccess methodDescription of needN/AInterfaces and dependenciesIn order for the application to operate, it must be able to access the STAR* database.Exceptions and other issuesNone.Service specificationURLs, certificates and channelsDevelopmentApplicationURLSchool and Programme Quality System UIdev.spqs.euclid.ed.ac.ukSchool and Programme Quality System microservicedev.api.euclid.ed.ac.uk TestApplicationURLSchool and Programme Quality System UItest.spqs.euclid.ed.ac.ukSchool and Programme Quality System microservicetest.api.euclid.ed.ac.uk TrainApplicationURLSchool and Programme Quality System UItrn.spqs.euclid.ed.ac.ukSchool and Programme Quality System microservicetrn.api.euclid.ed.ac.uk LiveApplicationURLSchool and Programme Quality System UIspqs.euclid.ed.ac.ukSchool and Programme Quality System microserviceapi.euclid.ed.ac.uk CertificatesCertificate CNCAServerLocationspqs.euclid.ed.ac.ukQuoVadispython-kb1-devLoad Balancersspqs.euclid.ed.ac.ukQuoVadispython-at1-devtest.spqs.euclid.ed.ac.ukQuoVadispython-kb1-testtest.spqs.euclid.ed.ac.ukQuoVadispython-at1-testtrn.spqs.euclid.ed.ac.ukQuoVadispython-kb1-testtrn.spqs.euclid.ed.ac.ukQuoVadispython-at1-testspqs.euclid.ed.ac.ukQuoVadispython-kb1-livespqs.euclid.ed.ac.ukQuoVadispython-at1-liveCertificate CNCAServerLocationapi.euclid.ed.ac.uk QuoVadispython-kb1-devLoad Balancersapi.euclid.ed.ac.uk QuoVadispython-at1-devtest.api.euclid.ed.ac.uk QuoVadispython-kb1-testtest.api.euclid.ed.ac.uk QuoVadispython-at1-testtrn.api.euclid.ed.ac.uk QuoVadispython-kb1-testtrn.api.euclid.ed.ac.uk QuoVadispython-at1-testapi.euclid.ed.ac.uk QuoVadispython-kb1-liveapi.euclid.ed.ac.uk QuoVadispython-at1-liveCertificate CNCAServerLocationspqs.euclid.ed.ac.ukEdUnipython-kb1-devOn server, deployed via Puppetspqs.euclid.ed.ac.ukEdUnipython-at1-devtest.spqs.euclid.ed.ac.ukEdUnipython-kb1-testtest.spqs.euclid.ed.ac.ukEdUnipython-at1-testtrn.spqs.euclid.ed.ac.ukEdUnipython-kb1-testtrn.spqs.euclid.ed.ac.ukEdUnipython-at1-testspqs.euclid.ed.ac.ukEdUnipython-kb1-livespqs.euclid.ed.ac.ukEdUnipython-at1-liveMyED channelsChannel nameTypeDescriptionSchool and Programme Quality SystemN/APuppet configurationThe Puppet code for the School and Programme Quality System services sit in the spqs and spqsapi profiles.ProfileDocumentationspqs serversDevelopmentTestLiveServerspython-kb1-dev.ispython-at1-dev.ispython-kb1-test.ispython-at1-test.ispython-kb1-live.ispython-at1-live.isPhysical / VirtualVirtualVirtualVirtualShared / DedicatedSharedSharedSharedCPU cores222Memory6GB6GB6GBOSCentOS 7CentOS 7CentOS 7Software and versionsPython 3.4Apache 2.4Python 3.4Apache 2.4Python 3.4Apache 2.4DependenciesSMTP, CosignSMTP, CosignSMTP, CosignFile systemsServer namesVolumeSizePurposepython-*/50GBRootpython-*/apps50GBApplication homespython-*/srv150GBWeb contentUsers, roles and groupsLinuxUsernameHome directoryDescriptionisapps /home/isappsIS Apps userspqs/home/python/spqsUI application owner accountspqsapi/home/python/spqsapiMicroservice application owner accountGroupMembersDescriptionisappsisappsIS Apps groupsystemd-journalisappsAllows access to the systemd journalpythonspqsspqsapiGroup for application functional accountsOracleInstanceUsernameRolesDescriptionSTAR*spqsAPPLICATIONOwner accountInstanceOPS$ usernameRolesDescriptionSTAR*N/AInstanceDatabase roleDescriptionSTAR*APPLICATIONSchema owner roleInstanceSchemaTablespaceSTAR*SPQSSPQS_DATA, SPQS_INDEXInstanceDB link nameOwnerSource instanceSource userSTAR*N/ANB: As Django needs many workarounds to connect to a qualified object, a single user (the schema owner) is the only user required.Data sourcesPythonData source namespqs_{dev|test|live}UsernamespqsDatabaseSTAR*Firewall configurationCentral firewallSourceDestinationPortProtocol129.215.180.xpython-*22, 80, 443SSH, HTTP, HTTPSpython-kb1-dev.ispython-at1-dev.issitsdb-devkb.issitsdb-devat.is1832Oracle Netpython-kb1-test.ispython-at1-test.issitsdb-testkb.issitsdb-testat.is1832Oracle Netpython-kb1-live.ispython-at1-live.issitsdb-livekb.issitsdb-liveat.is1832Oracle NetThe access from 129.215.180.x is for diagnostic purposes only, firewall configuration is not required for services proxied by the load balancers.Scheduled tasksNone.Software licencesNone.Service operationSupport contactsNone.Startup and shutdown stepsThe Gunicorn for each application is set by Puppet to start on server boot using a Puppet-created systemd unit file. Bamboo also manages the Gunicorn as part of the deployment process.A puppet run should be performed manually to start the service by the isapps user (this will enforce the “running” state of all Gunicorn processes):sudo /opt/puppetlabs/bin/puppet agent –tvTo stop the Gunicorn server, there is no shutdown script, instead kill the PID of the parent Gunicorn process as the function account which owns it (). Note that Puppet will try to start Gunicorn when it runs next.Log filesThe Apache log files can be found on the application servers at: /var/log/httpd/Configuration filesThe Python platform is completely automated, with infrastructure (on top of the ITI-provided base build) managed by Puppet, and the applications deployed by Bamboo from Git. Configuration should be checked and altered in Puppet as changes made directly to the filesystem will be overwritten.PatchingRegular patching of the Operating System will ensure that the host OS, web server and Python versions are constantly up to date and receive security fixes provided by CentOS and the EPEL and SCL projects.Patching of the pip-installed Python components must be made by altering the versions in the requirements.txt file in the application code, committing this to Git (following the IS Applications code workflow standards), then rebuild and deploy via Bamboo.Patching of the application code must be made by altering the code, committing this to Git (following the IS Applications code workflow standards), then rebuild and deploy the application via mon proceduresStart an individual Gunicorn manuallyIn order to start the two Gunicorns for the service manually, run each of these commands. As spqs:/bin/bash -c "source /etc/profile ; source /opt/rh/rh-python34/enable ; source /apps/venvs/spqs/bin/activate ; PYTHONPATH=/srv/spqs /apps/venvs/spqs/bin/gunicorn --bind 127.0.0.1:8082 -k tornado --threads 4 --workers 4 -n spqs -p /apps/pid/spqs.pid config.wsgi"As spqsapi:/bin/bash -c "source /etc/profile ; source /opt/rh/rh-python34/enable ; source /apps/venvs/spqsapi/bin/activate ; PYTHONPATH=/srv/spqsapi /apps/venvs/spqsapi/bin/gunicorn --bind 127.0.0.1:8083 -k tornado --threads 4 --workers 4 -n spqsapi -p /apps/pid/spqsapi.pid config.wsgi"Disaster recovery planThe applications are dual-sited, they will remain available after the loss of one site without any intervention, however, this is dependent on the database being available.For details on the DR plan for the EUCLID database platform, please see its infrastructure TAD. ................
................

In order to avoid copyright disputes, this page is only a partial summary.

Google Online Preview   Download