Overview - DoD Procurement Toolbox
IntegratedAcquisitionEnvironmentEntity Management Extracts Policy – v1.0April 30, 2013Change LogVersion #Date of ChangeSectionDescription of ChangeChanged By1.04/30/2013AllInitial DraftAdam StansiferTable of Contents TOC \o "2-3" \h \z \t "Heading 1,1,Appendix Heading,1" 1Overview PAGEREF _Toc355252734 \h 42Requirements PAGEREF _Toc355252735 \h 42.1Interface Requirements PAGEREF _Toc355252736 \h 42.2Interface File Formats PAGEREF _Toc355252737 \h 42.3Interface Details PAGEREF _Toc355252738 \h 53Naming Conventions PAGEREF _Toc355252739 \h 53.1File Naming Conventions PAGEREF _Toc355252740 \h 53.2SFTP and Web File Directory PAGEREF _Toc355252741 \h 74Data Format PAGEREF _Toc355252742 \h 84.1Delimited PAGEREF _Toc355252743 \h 84.2New Line Separators PAGEREF _Toc355252744 \h 84.3Header PAGEREF _Toc355252745 \h 94.4Footer PAGEREF _Toc355252746 \h 95Rules and Policies PAGEREF _Toc355252747 \h 95.1File posting Date / Time PAGEREF _Toc355252748 \h 95.2Processing Rules PAGEREF _Toc355252749 \h 105.3Data Persistance Policy PAGEREF _Toc355252750 \h 105.4Interface Deletion Policy PAGEREF _Toc355252751 \h 105.5Refresh Policy PAGEREF _Toc355252752 \h 106Security PAGEREF _Toc355252753 \h 107Interface Agreement PAGEREF _Toc355252754 \h 117.1Policies regarding the dissemination of SAM data PAGEREF _Toc355252755 \h 118Extract Grain & Inclusion Criteria PAGEREF _Toc355252756 \h 118.1Extract Grain: / (Uniqueness criteria) PAGEREF _Toc355252757 \h 118.2Inclusion Criteria PAGEREF _Toc355252758 \h 129Attribute List PAGEREF _Toc355252759 \h 12OverviewThe purpose of this document is to describe the Entity Management Extracts that allow SAM to share, where appropriate, the entity profiles within the federal government for many functions, including, procurement, contracts, payments, size determination, and validation. The interface provides data pertaining to all vendor organizations to authorized and authenticated users via Secure File Transfer Protocol (SFTP) or the Extracts section of the SAM website. The interface includes daily downloads of database updates/new records as well as a monthly refresh of the entire SAM Entity Management database. This interface is automatically scheduled for generation by SAM; the resultant extracts are then made available for automatic or manual retrieval by the SAM consumer.RequirementsInterface RequirementsEach list of data elements will be constructed as a CSV (i.e. plain text) document. These documents will be made available to SAM users over sFTP, and HTTP (i.e. via the SAM website). The following requirements need to be met for Entity Management Extract interfaces:SSH software is required to connect via SSH/SFTPExtract information will be provided based on the sensitivity level of the data contained in that extract. Roles based access will be used to assign permissions to read data at each of these various sensitivities. The four SAM extracts include:Public – This data can be downloaded by any authenticated user (i.e. anyone who has registered for a SAM account). This file will contain data that is otherwise viewable to public users who browse the SAM website.FOUO – This data will require permission via a role assignment. This file contains data included in the Public file as well as all data categorized as “For Official Use Only.” It excludes Sensitive and System Only information.Sensitive – This data will require permission via a role assignment. This file contains data included in the Public and FOUO files, as well as data categorized as sensitive information about entities SAM.In addition, the legacy MPIN extract will be carried forward and will be referred to as the SAM MPIN extract. MPIN – This data will require permission via a role assignment. This file contains a small subset of data (4 elements) included in the Public files (CAGE Code, DUNS, DUNS+4), as well as data included in the Sensitive files (MPIN).Interface File FormatsExtract data will be provided in two zip files:Data File (.DAT) – This file will contain the actual extract data. It will be formatted as described in this document as well as the extract mapping document. See the file naming conventions section to understand the naming of this main extract file.Metadata File (.MDAT) – This file will contain all column header information for the given file. This way the user can be assured that the most up to date header metadata information is available at any given time. See the file naming conventions section to understand the naming of this Metadata extract file.Please refer to the SAM Master Extract Mapping spreadsheet for the list of data elements.The Entity Management Extracts interface provides data pertaining to all entity organizations to authorized and authenticated users via Web Service, Secure FTP or the Extracts portion of the SAM website.All extracts must be generated and compressed as .ZIP files due to large file sizes.Interface DetailsTable 2.1: Interface DetailsParameterDetailsInterface IDINT047, INT048, INT049Trigger EventScheduled monthly and dailyImplementationAutomated generation; manual retrievalProtocolsSFTP, HTTP (Website)Data FormatCSVFrequencyExtracts are generated from SAM data daily and monthly.Extracts may be accessed daily/monthly (via sFTP or HTTP); this will not, however, generate an ad-hoc report or refresh the data extract more frequently than once per day.Authentication MethodUsername and password is required for all, but public, extracts, regardless of format or method of delivery.Request ParametersN/AResponse ParametersN/AException ConditionsIf the generated extracts cannot be stored on the filesystem due to lack of storage capacity, a notification should be sent to the administrator with details about the extract and the error condition.Naming ConventionsNew refresh files are available on the first Sunday of each month. Daily update files are produced Tuesday through Saturday and are available on the server by 8:00 AM Eastern Time.File Naming ConventionsAs outlined below, the SAM extract files will consist of two ZIP files, each with one file inside. The first zip file contains the extract data. The second zip file will contain metadata, such as column headers, to be used with the data file. The file naming convention for SAM Entity Management Extracts interface will be as follows:The File that contains extract data will be named as: SequenceValues1SAM_2[Sensitivity] (select from {PUBLIC, FOUO, SENSITIVE, SYSTEMONLY, MPIN})3Underscore (_)4[Refresh] (select from {Monthly, Daily})5Underscore (_)6YYYYMMDD (date of receipt)7.DATFile that contains the metadata data will be named as: SequenceValues1SAM_2[Sensitivity] (select from {PUBLIC, FOUO, SENSITIVE, SYSTEMONLY, MPIN})3Underscore (_)4select from {MONTHLY, DAILY}5Underscore (_)6YYYYMMDD (date of receipt)7.MDATThe metadata (MDAT) file will contain only the column headers separated by pipe (|) for the respective data elements in the (DAT) file.This Format of the ZIP file containing the extract data will be as follows:SequenceValues1SAM_2[Sensitivity] (select from {PUBLIC, FOUO, SENSITIVE, SYSTEMONLY, MPIN})3Underscore (_)4select from {MONTHLY, DAILY}5Underscore (_)6YYYYMMDD (date of receipt)7Underscore (_)8DAT9.ZIPThis Format of the ZIP file containing the metadata data will be as follows:SequenceValues1SAM_2[Sensitivity] (select from {PUBLIC, FOUO, SENSITIVE, SYSTEMONLY, MPIN})3Underscore (_)4select from {MONTHLY, DAILY}5Underscore (_)6YYYYMMDD (date of receipt)7Underscore (_)8MDAT9.ZIPFor example:SAM_PUBLIC_MONTHLY_20110102_DAT.ZIP – This is the monthly refresh of the Public extract for February 2011 (made available in this example on February 1st). This zip file will contain SAM_PUBLIC_MONTHLY_20110102.DATSAM_PUBLIC_MONTHLY_20110102_MDAT.ZIP – This is the metadata file for the February 2011 refresh of the Public extract (made available in this example on February 1st). This zip file will contain SAM_PUBLIC_MONTHLY_20110102.MDATSAM_SYSTEMONLY_DAILY_20110102_DAT.ZIP – This is the daily update of the System Only extract, made available on February 10, 2011. This zip file will contain SAM_SENSITIVE_DAILY_20110102.DATSAM_SENSITIVE_DAILY_20110102_MDAT.ZIP – This is the daily metadata update of the Sensitive extract, made available on February 10, 2011. This zip file will contain SAM_SENSITIVE_DAILY_20110102.MDATSFTP and Web File Directory The directory for storing SAM interface data will be organized as follows:Current folder: in/current/entitymgmt/extractsProcessed folder: in/processed/entitymgmt/extractsArchived folder: in/archived/year/entitymgmt/extractsError folder: in/error/entitymgmtError log folder: logs/interface/error/These directories will be visible to only those users who have been granted access to them.Folder Structure below reflects the structure of both legacy and SAM (to be) extracts.The data will be available on secure ftp server sftp..Data FormatDelimitedThe plain text extracts will be provided via SFTP and HTTP (website). The data delimiters used will be as follows:Pipe (|) is used to separate single valued data elementsTilde (~) is used to separate multiple occurrences of a data element. For a concatenated string it will be the main delimiter to separate multiple concatenated strings.Caret (^) is used to separate values in a multi-valued strings.For example in the concatenated string below:BL^30000000^15000000^30000000^15000000~LS^42^5000000^0^0~PG^1^0^0^02 ~’s separate 3 strings. BL^30000000^15000000^30000000^15000000, LS^42^5000000^0^0 and PG^1^0^0^0Where as ^’s separate each of the elements within a particular concatenated string.All Delimiters (|, ~, and ^) will be removed from data and replaced with an empty string.New Line SeparatorsLine feed will be used as the new line separator.HeaderHeader, the first record in the data file will contain following elements:PositionValues1BOF5[Sensitivity] (select from {PUBLIC, FOUO, SENSITIVE, SYSTEMONLY, MPIN})16Start Date (YYYYMMDD). [00000000 for Monthly extracts]25End Date (YYYYMMDD) [For Monthly Extracts this is the last date of data included in the monthly extracts.34Number of records (7 digits): Eg: 102 records will be 000010242File Sequence Value (7 digits): Eg 631 will be 0000631For example, a header could appear as follows:Daily File: BOF SENSITIVE 20121210 201212100000102 0000631Monthly File: BOF SENSITIVE 00000000 20121203 0684123 0000631FooterFooter, the last record in the data file will contain following elements:PositionValues1EOF5[Sensitivity] (select from {PUBLIC, FOUO, SENSITIVE, SYSTEMONLY, MPIN})16Start Date (YYYYMMDD). [00000000 for Monthly extracts]25End Date (YYYYMMDD)34Number of records (7 digits): Eg: 102 records will be 000010242File Sequence Value (7 digits): Eg 631 will be 0000631For example, a footer file could appear as follows: Daily File: EOF SENSITIVE 20121210 201212100000102 0000631Monthly File: EOF SENSITIVE 00000000 20121203 0684123 0000631Rules and PoliciesFile posting Date / TimeSAM daily extract are a reflection of all the changes made in SAM on the previous day. SAM Daily extracts will be posted every day weekday, from Tuesday to Saturday between 12:00 AM and 8:00 AM EST. They will include data that has changed since the previous execution of daily extract.SAM Monthly extracts will be a full volume extract that will extract all the data that matches the data extraction criteria. SAM Monthly extracts will be posted on the 1st Sunday of every month between 12:01 AM and 11:59 PM.Under certain scenarios like data corruption, database downtime, application upgrades there might be a delay in posting the daily / monthly extracts.SAM data quality team will make every effort to post clean data, however, in case of data corruption / file damage of extracted data following process will be followed: For a Daily file: If data corruption is identified the same day, the corrupt daily file will be replaced with the new file. If the data corruption is identified in the following days, all of the previous posted files until the point of corruption will be considered corrupt and will be removed and replaced by a single range file. The range file shall contain all the changes from the point of data corruption. The start date in file header / footer in this case will be the date the first data corruption was identified, and end date will be execution date minus one day. The range file will be in the same format as the Daily files.For a Monthly File: The corrupt monthly file will be removed and the clean monthly extract file will be posted. This extract will be based on data as it stands at the point of execution. (Not necessarily the 1st Sunday of a month in this case) Processing RulesFor the daily extracts, SAM will make data available to authorized and authenticated users via Secure FTP or the Extracts portion of the SAM website. For the monthly extracts, users will be able to retrieve a “snapshot” of all active entity records. This service allows users who have been previously authorized and issued a user name and password to retrieve information about vendors who are registered in the CCR database.Data Persistance PolicyInterface data will persist for a period of 30 days, after which the data will be deleted and will no longer be available either on the web site or SFTP folder. This means that a monthly file will be kept up until the next monthly file is available, at which time it will be removed. There will be 30 days’ worth of daily files posted to the site at all times.Interface Deletion PolicyAs the data in this interface is derived from the Entity Management records, deletion rules follow entity record deletion rules.Refresh PolicyThe CSV extracts will be generated monthly and daily. The daily files will offer a record for all completed transactions taking place within Entity Management since the last daily file generation and are available for 30 days. The refresh files will contain a record for every active profile and profiles expired less than six months at the time of generation.SecuritySSH software is required to connect via SSH/SFTP to the SAM Extract server and use software appropriate commands and syntax. Each user will be assigned a Username and Password allowing for access to the extract files on the SAM Server. Each Username will be assigned permission to access these extracts on an as needed basis for the type of data wanted. In addition, IP-based authentication will be used for SFTP file transfer.Interface AgreementInterface agreements will be established with organizations whose systems are consumers of SAM extract data.Policies regarding the dissemination of SAM dataAll SAM extract data must be handled as stated in the Non-Disclosure Agreement associated with the approval for access to such extracts. Data is to be handled by the individual (data receiver) who has prior approval and is listed on the user profile.All extracts, with the exception of the Public extract, must be downloaded only to a secured government network that has the Authority to Operate as a government system. This data must never reside within a commercial or unsecured environment.The transfer of data must follow the security guidelines and methodologies set forth in Interconnection Security Agreement (ISA). Extract Grain & Inclusion CriteriaExtract Grain: / (Uniqueness criteria)The data extracted will be unique at the DODAAC, DUNS and DUNS_plus4 combination. Inclusion CriteriaOnly following records will be included in the daily / monthly extracts.Monthly Full Volume ExtractMonthlyDailyActive RecordsAll Active records are includedOnly records that changed since last execution are included.Expired RecordsOnly 6 months of expired records.Only records that expired since last run are included.De-activated recordsNot includedOnly records that have been deactivated since last execution are included.Changed DataAll active records and only 6 months of expired records.Only records changed since previous execution are included.SAM Extract CodeActive, 'A', Expired, 'E' send complete record. 1 = Delete send DUNS, DUNS +4, DODAAC, CAGE Code 2 = New send complete record 3 = Update send complete record 4 = Expired send DUNS, DUNS +4, DODAAC, CAGE CodeAttribute List<Attribute list from SAM Master Extract Mapping document> ................
................
In order to avoid copyright disputes, this page is only a partial summary.
To fulfill the demand for quickly locating and searching documents.
It is intelligent file search solution for home and business.
Related searches
- engineering toolbox unit converter
- osha flu season toolbox talk
- flu toolbox talk
- situational awareness toolbox talk
- toolbox talk flu prevention
- toolbox inventory pdf
- matlab deep learning toolbox pdf
- matlab deep learning toolbox download
- deep learning toolbox matlab
- matlab toolbox download
- deep learning toolbox matlab download
- matlab neural network toolbox pdf