Section 1 – Licensed Data General Info - Morningstar
Licensed Data – Best Practices Manual
Licensed Data General Info
1 Morningstar Contacts
Morningstar provides a multi-tier customer support structure, which ensures we are accessible and understand your business needs. In addition to your Relationship Manager, a dedicated Client Manager is assigned to your account and will work together with you throughout the term of your contract. Your Client Manager will be available to provide product support during local business hours, Monday through Friday, excluding nationally recognized and market holidays. Should you require 24x7 technical support, please discuss these needs with your Client Manager. If at any time your service needs are not being met, please communicate your concerns to your Relationship Manager.
Contact information for your assigned Client Manager will be sent to you directly. As your primary point of contact, please reach out to your Client Manager with any questions regarding your package deliveries or about other Morningstar solutions for your business.
Periodically, Morningstar will send out communications related to the latest developments or changes that are being made to the various areas of the business, data or products. These communications will originate from one of two e-mail addresses:
Morningstar Client Services - clientservice@
Used for Morningstar Data Release Notes, a monthly electronic newsletter.
Licensed Data – licenseddata@
Used for announcements related to your Licensed Data Feeds including production announcements and notices about upcoming changes.
To ensure receipt of valuable information, the above addresses and the addresses of your assigned point of contact should be added to the safe senders list if a spam filter is in place.
2 Client Contacts
Your Client Manager will be in contact with you on a regular basis to ensure your needs are being met. Please make all efforts to communicate changes to your contact information to us on a regular basis.
It is important for Morningstar to maintain up-to-date contact information to ensure that you receive important communications impacting your data feed or the availability of other Morningstar products and services.
3 Morningstar Holidays
Your Client Manager and Sales Director will not be available on Morningstar recognized holidays: Files do not post on these holidays. A yearly calendar of specific dates is available.
❑ New Year’s Day
❑ Martin Luther King Jr. Day
❑ Presidents’ Day
❑ Good Friday
❑ Memorial Day
❑ Independence Day
❑ Labor Day
❑ Thanksgiving and the following Friday
❑ Christmas Day
The Data Feed
1 DataWarehouse (OnDemand) Defined
DataWarehouse is our most comprehensive data package. It includes portfolio, performance and operational data as of the latest month-ended and is available for the open-end, load waived, money market, exchange traded fund, closed-end, VA/L Underlying Fund, VA/L subaccount, separate account, hedge fund, CIT, IGSA, and UIT universes. All global countries, Morningstar category and index files are also available through this package.
This package is fulfilled via an OnDemand delivery mechanism. This means that files can be made available to your Morningstar dedicated FTP site on a daily, or monthly basis.
2 Files
The following section discusses the delivery timing and method, as well as a general overview of the package contents and file naming conventions.
1 Content
An initial delivery of files for the entire universe(s) per package will be delivered to you in the form of the monthly package. Even if you have chosen to receive a daily package, keep in mind that you will also need to process the monthly package to refresh your entire database each month.
For clients who choose monthly package deliveries, each XML file will contain one full universe in one wrap file. Clients will receive the monthly package after the database completes month-end production.
For clients who choose daily package deliveries, OnDemand delivers delta files which contain complete data for only those funds that have had database changes since the last file delivery. The delta file will not contain just the nodes that have changed. It will contain all nodes and the information that changed will not be flagged in any way.
If you are receiving daily deliveries, it is highly recommended that you also make it a point to download the monthly file as soon as it is made available, prior to downloading any new daily files. The files must be processed in chronological order to ensure that no data updates are missed.
2 Delivery Timing and Method
Deliveries can be scheduled on a daily or monthly cycle. For clients choosing a daily, ongoing access is granted to the monthly package as well. For clients choosing a monthly delivery cycle, it is recommended that a process also be set-up to accommodate bringing down one or more of our daily deliveries. However rare, there may be instances where we will have a need to provide you with a daily delta that contains the full universe or very necessary data corrections.
The first daily file of the month will be available the day after the monthly file is available. Except for the US holidays listed previously, daily files will continue to be delivered every business day up until the last business day of the month.
Each month, clients will receive the entire universe (monthly file) on the next scheduled delivery* after the database signs off on month-end production. No deliveries will take place during database production days or on the US holidays listed above (applicable to files for all universes and countries). Database production days will last between two and six business days.
*Delivery timeframe for monthly files will vary by universe and country and will depend on the timeframe specified in the contract. Please work with your Client Manager to determine your delivery timeframe.
Files will be delivered by 11:00pm CST on the business day and frequency (daily or monthly) specified in your contract. Files are delivered via an Internet File Transfer Protocol (FTP) account at a private, designated Morningstar FTP site. Files may be delivered on multiple Morningstar FTP sites.
All files will remain on the FTP site for ten calendar days. Files older than ten calendar days will be deleted automatically.
3 Daylight Savings Time
With regards to data feed production – we do not make any special modifications for daylight savings time. The system time changes automatically, so we post files at the same time, whether it’s standard time or daylight savings time.
With regards to our FTP server – the server is based on a product called Serv-U from Solarwinds. It records timestamps in UTC format when files are transferred. This format allows the operating systems or FTP client software to make their own adjustments on how time is displayed. The IETF specification for SFTP transfers is that the file date is set in UTC time after the file has been received by the SSH_FXP_SETSTAT command. Serv-U conforms to this specification. The way your software displays UTC time will determine what time is displayed. As expected this is handled automatically by our FTP software and modern operating systems.
If the software you are using is capable of issuing the MDTM command, then you are also able to use this command to modify timestamps if you find something is not as expected. Further information can be found here:
4 File Format & Compression
There will be one wrapped XML file per package per universe and four text files. The XML file for each delivery will be packaged in a .zip or .gz archive format. The default compression format that Morningstar will typically use is .zip. However, files with sizes that are nearing around 400MB zipped (for XML) or around 1GB zipped (for flat files) are candidates to be switched to the .gz archive format. The gz format becomes the next logical format to use as it utilizes a higher compression ratio and can compress files with sizes greater than 4GB, unlike .zip. Today, files for the VA universe and some of the Available-For-Sale (AFS) or Domicile (DMCLE) universes are packaged using the .gz archive format.
For .gz archived files we utilize the default compression level, -6 (biased towards high compression at the expense of speed).
5 Naming Conventions
The XML file naming convention consists of the following:
PackageName_Universe_Country_CycleId_yyyyMMdd.xml.zip
PackageName_Universe_Country_CycleId_yyyyMMdd.xml.gz.
Examples: DataWarehouse35_FO_USA_D_20110101.xml.zip
DataWarehouse35_FE_USA_M_20110101.xml.zip
DataWarehouse35_VA_USA_M_20110101.xml.gz
Where:
Universe will be represented by one of the following values:
❑ FO for open-end funds
❑ FC for closed-end funds
❑ FE for exchange traded funds
❑ FH for hedge funds
❑ FM for money markets
❑ FV for Variable Annuity/Life underlying funds
❑ SG for Group Retirement Plan Annuity (GRPA)
❑ SA for separate accounts/CIT’s
❑ CT for CITs
❑ UT for Unit Investment Trust’s
❑ VA for Variable Annuity/Life subaccounts
❑ CA for categories
❑ XI for indices
❑ LW for Load Waived
❑ AFS for funds packaged by “Available For Sale” regions
❑ DMCLE for funds packaged by “Domicile” regions
And
CycleId will be represented by one of the following values:
❑ D for daily
❑ M for monthly
The .txt file naming conventions are similar to the XML file names as described above. However, each .txt file will have a different file name ending, to distinguish the function of the file:
❑ UpdateIds: PackageName_Universe_Country_CycleId_yyyyMMdd_updateIds.txt
o Contains the Investment Vehicle IDs of the delta investments in the XML file.
❑ Delete: PackageName_Universe_Country_CycleId_yyyyMMdd_delete.txt
o Contains the Investment Vehicle IDs of the deleted, merged, or liquidated investments.
❑ Trigger : PackageName_Universe_Country_CycleId_yyyyMMdd_trigger.txt
o Empty file to indicate completion of all files. Do not begin downloading if this file is not present.
❑ Under Review: PackageName_Universe_Country_CycleId_yyyyMMdd_underReview.txt
o Contains the Investment Vehicle IDs of the under review funds. These are active funds that are placed under review because they do not pass all of Morningstar’s business QA rules.
Please note that the file naming convention includes a date stamp. The date stamp represents the date when Morningstar began producing the file. As a result, new files will not overwrite old files posted to the FTP site.
6 UpdateIds Files
The UpdateIds text (.txt) file will only include a list of the Investment Vehicle IDs for those funds where one or more datapoints have changed. Regardless of what has changed in the fund, all data for the fund will be included in the XML file. All delta files (delivered on a daily or weekly cycle) include only the data for those investments whose Investment Vehicle IDs appear in the UpdateIds file. All XML files delivered on a monthly cycle include the data for all funds, regardless of whether they had changes or not. It is recommended that all funds in all XML files be processed to ensure that no changes are missed.
The UpdateIds file can be used to gauge the need to process a delta file, based on whether any of your respective funds are contained in the list (and therefore in the delta file). The file can also be used simply as a way to check your production process and determine whether it captured updates for your funds.
7 Deletion Files
The deletion file will include the Investment Vehicle IDs of those investments that have been deleted, merged or liquidated. Note that these investments will no longer be included in the XML file.
Two possible courses of action to handle the deletion files are:
❑ Flag the investments as unavailable, or
❑ Delete the investments
Each client can treat these Investment Vehicles as they deem appropriate for their use. However, it is Morningstar’s recommendation that any investments whose ID’s are included in the deletion file simply be flagged as unavailable. The logic for this recommendation is that the information for these vehicles will not be included in future DataWarehouse files.
8 Under Review Files
The UnderReview file will include a list of Investment Vehicle IDs for those investments that have been placed "under review" by Morningstar's Data team for data quality concerns we have been unable to resolve in time for the DataWarehouse file production. Investment Vehicle IDs of under review investments that have previously appeared in the file may appear on an ongoing basis as changes to their respective data occurs. The data for these funds will not be included in the XML file, but the Investment Vehicle ID is included in the under review file to make you aware that while the fund is under review, it continues to receive data updates to once again make it available to you in the future. Once all data has been verified, the full updated record will be included in the DataWarehouse file, so you will not miss any corrections should they be made.
Two possible courses of action to handle the UnderReview files are:
❑ Delete the investments, or
❑ Flag the investments as unavailable
Each client can treat these Investment Vehicles as they deem appropriate for their use. However, it is Morningstar’s recommendation that any investments whose ID’s are included in the under-review file simply be deleted or flagged as unavailable. The logic for this recommendation is that while these are unavailable today, they will become available at a later period. While these funds are under review, all data which is currently being displayed could be what is under review, so the best option is to ensure that no incorrect data is being displayed by deleting the investment. As stated earlier, once the investment has been fully tested and verified, it will be returned to the feed and a full fresh set of data will be made available.
9 Trigger Files
The trigger file is an empty .txt file that will be posted once the XML file and the UpdateIds, UnderReview, and Deletion files have completed posting to the FTP site. You should not begin downloading any files in the folder if this file is not present.
3 DataWarehouse Dictionary
1 Dictionary
A data dictionary has been created for the DataWarehouse package to clarify the datapoints included in the feed. The DataWarehouse dictionary lists all the data elements contained in your data package and is a useful document for data related questions. Simply search for the data element and it will provide you with the standard/business description name, definition, XML path, notes that capture business rules or exceptions, a typecodes indicator, and a listing of the universes to which the datapoint applies.
If an element contains typecodes, this will be noted next to the element. A hyperlink will provide you with the reference table that shows all possible typecode values for that element.
The DataWarehouse dictionary will also be made available during implementation and upon request.
2 Annual Returns in Year-End Data Feed Delivery – Special Note on XML Path
The Data Warehouse package contains calendar annual returns for the previous 10 years in each delivery. These are provided as AnnualReturnNAVY1, AnnualReturnNAVY2, AnnualReturnNAVY3… AnnualReturnNAVY10 (please note: market price and gross annual returns are also available). The annual return Y1 element represents the annual returns of the prior year in each delivery. For example, if the current year is 2013, the annual return Y1 element would provide the annual return for 2012 and the annual return Y10 element would provide the annual return for 2003.
For data feeds delivered with year-end data (that is, data as of December 31st delivered in early January), there will be an additional annual return Y0 data element provided AnnualReturnNAVY0) which represents the current year annual return. Continuing the example above, if the current year is 2013 and the data feed contains data as of year-end, the annual return Y0 element will be present and it would provide the annual return for 2013, the annual return Y1 element will provide the annual return for 2012, and so on.
This note on the XML path for annual returns in year-end data feed deliveries is also provided in the Data Warehouse dictionary.
4 Schema files
An XML schema defines the legal building blocks of an XML document. Attributes of an XML schema are featured below:
❑ Defines elements and attributes that can appear in a document
❑ Identifies child elements and the order and number of child elements
❑ Characterizes whether an element is empty or can include text data types for elements and attributes
❑ Defines default and fixed values for elements and attributes
Validations are generated for each DataWarehouse package delivered to ensure that the XML files match the structure and format of the schema. Morningstar only delivers DataWarehouse packages after they pass our validation procedures without generating errors. The schema for the DataWarehouse package will be made available during implementation and upon request.
The schema for the DataWarehouse package is the same across all US and global universes mentioned in this document. This does not include the 529 or Equities universes.
When prepping your processes to pull in our DataWarehouse files, there is one very important item to note: Always code to the schema file. Please pay particular attention to typecodes and elements that can be repeated. If you are looking for a particular set of data and multiple instances of the element are allowed or there are typecodes associated to that element consider whether you will pull in all instances of the data or if you need to put filters around the datapoint,
Please also note that coding to the schema elements should be done using relative path locations and not via hard-coded paths as this will require additional programing when new data elements are added to a file. The most common language for this type of query against and xml is XPath. If relative paths are used, additions to the schema of new data points will only require new programing to be done if you wish to consume this new data.
If you need assistance in determining the best way to get the data you need, please reach out to your Client Manager.
FTP Site
1 General FTP Information
Data files can be downloaded at: .
Where:
Frequency is one of the following:
❑ Daily
❑ Monthly
And:
Package Name is either:
❑ DataWarehouse** (for the latest version)
❑ Test (for Test files)
The FTP site can only be accessed by utilizing the login information provided by Morningstar. Each client login is given read-only rights to the FTP site and those read-only rights apply to the terminus folder noted in each FTP path.
For example, if you receive access to Morningstar’s DataWarehouse feed for Closed-end funds, the FTP paths permissioned will be:
However, only the FC folder will be visible\accessible so this should be accounted for when clients are testing their FTP by using the full FTP folder path rather than trying to step through each folder.
All clients are asked to check their ability to access the FTP site by testing with the login information immediately after it is provided. Should you receive any error messages related to accessibility, check for internet connectivity issues and the presence of a firewall. If these exist, work with your internal Network team to resolve.
Please ensure that you are applying these in your process:
❑ Firewall rules at any client site should be opened to allow access to ftp.
❑ External clients should update their FTP application to use DNS name ftp. instead of the IP address.
❑ The IP address of ftp. should be resolved via DNS query.
❑ Clients should also update their firewall rules to allow FTP access to 216.228.237.25 and 216.228.224.25 for active mode or PASV mode with TCP ports 20, 21, 22 (sftp), 990 (implicit ftps), and 24210-24220 (PASV).
If after troubleshooting with your internal Network team it is determined that there are no internet connectivity issues, there is no firewall present and the above FTP best practices have been applied, you are encouraged to work with your Client Manager to further troubleshoot.
For security purposes, FTP login information (username and password) will be provided over the phone and may be updated on a quarterly basis, with 60 days’ advance notice.
If your internal requirements are to use sftp or SSL (Secure Sockets Layer) technology, you can achieve this using SFTP client on ftp.. ftp. supports the SFTP connection as long as you are using a SFTP client.
Notification of data feed file deliveries will not be e-mailed. As mentioned previously, a trigger file will post with each package once the files are ready to download.
2 Downloading from the FTP site
1 Manual
To access the FTP site and download a file, open an instance of Internet Explorer (or in Windows Explorer – make the address bar visible by going to the View menu. Under Toolbars, ensure that the “Address Bar” option is selected). Copy and paste the link provided, or simply type in (substituting the appropriate Frequency and PackageName, as noted above) in the address bar.
The following window should pop-up:
[pic]
Using the login information provided by Morningstar, enter the user name and password.
If you receive an error message indicating that the login is incorrect, please check that you have entered the information correctly. If so, contact your Morningstar Client Manager to confirm your login information and to help you troubleshoot the issue.
Once successfully logged in the following screen appears.
[pic]
A list of folders representing your licensed universes will be shown. In the above example, the available folders contain the Category (CA), open-end (FO), and Index (XI) universes.
While you may have only licensed for Daily deliveries, please note that the Monthly cycle folder and data is also made available to you. The Monthly files should be a part of your download process once a month.
Upon opening the FO folder, the following files will appear:
[pic]
The files can then be dragged into the desired storage area or the option to “Save As” can be selected when you right click on the file.
Please note that a trigger file will be posted with all OnDemand packages. This file serves as an indicator that all files have finished posting. If the trigger file is not present, do not download any of the files.
2 Automated
Morningstar created our OnDemand packages with automation in mind. For that reason, the packages are posted along with a trigger file. The trigger file serves as a notification that the package is ready to be downloaded. No e-mail reminders will be sent to notify of files posted.
An automated process should be set up to retrieve your data packages. A simple shell script can do the trick as it can be written to contain a command to begin the downloading process once the trigger file has been located.
Given that the files will be posted on a set business day, the shell script should be placed on a schedule to run automatically at various intervals either starting on the stated delivery business day or shortly thereafter. As noted previously, files will be delivered by 11:00pm CST on the business day and frequency specified in your contract.
Tools that can be used to schedule the shell script to run automatically are a basic schedule task wizard. The Microsoft Windows Operating System (O/S) by default includes the Scheduled Task Wizard, or if working in Linux or any other Unix-like O/S, Crontab is another commonly used task scheduler.
Keep in mind that there are pros and cons to using one scheduler over another. For example, there may be a scheduler that does not differentiate between weekends and weekdays while another is more flexible in that it can handle excluding weekends. While we only discuss the above two examples of schedulers that can be used to the same end, the internet may offer more suggestions for additional tools and advice on how to compensate for limitations, or, perhaps your company may already have a preferred task scheduling tool. Whatever the case may be, choose an automated retrieval method that is appropriate for your technical environment.
3 Some FTP Access Issues and Resolutions
The most common reasons and resolutions to why a client is unable to access the FTP site:
❑ The FTP site address is incorrect
o The FTP site address is (substituting the appropriate Frequency and PackageName as noted in the FTP Site section)
o Check the spelling of the FTP site to ensure that it has been typed or copied and pasted correctly.
❑ The login is incorrect
o Ensure that Caps Lock is not on, as logins are case-sensitive. Check the spelling of the username when you enter it into the login screen. Carefully enter the password. If you still cannot login to your FTP folder, you can verify your login information with your Client Manager.
❑ A firewall that does not allow access to FTP sites.
o You are advised to look into whether a firewall will restrict access to Morningstar’s FTP site. If so, arrangements should be made with your Network Administrator to gain access to the FTP site.
*For additional assistance troubleshooting FTP site access issues, please contact your company’s Network Administrator or your Morningstar Client Manager with the following details:
1. Date and time
2. Name of customer reporting issue
3. Summary of the issue
4. FTP Login name used
5. Folder path or files affected
6. Was this during upload or download?
Storage of XML Files
For future use, the XML files can be stored, per your archiving preference, for later use, during the term of your contract. Please keep in mind, however, that processing of files should occur in the order of their release dates. If files are processed out of order, an update may be missed, leaving your database out of synch with the latest data available. For that reason, if you need to reprocess a past file, all files occurring after it will need to be processed in sequence up to the most recent file.
Also keep in mind that the XML files are used as a method for transferring data over the internet using a common language. In the case of the XML files that Morningstar is providing to you, it is a way for Morningstar to ensure that we can help you easily and continually synch your database with the latest data available in our databases. Since the purpose of XML is to transfer and synch databases, Morningstar’s XML files should not be used to take the place of a production database. XML files are slow to work with in this capacity, as there is no indexing in the files. The files should be used strictly to synch your databases and then be stored for future reference, as necessary and as allowed by your contract.
Parsing and Storing the Data
Parsing the data from the XML files that you have received from Morningstar is an important and necessary step in getting the data stored in your databases.
The key to parsing an XML file is to consider using a parser that will utilize the smallest memory footprint. Also consider a parser that does not commit the entire XML tree to memory before it can begin processing. These are important points to take into account since the size of some XML files may reach the 3-4 gigabyte (GB) range or higher, depending on the universe and the amount of data available. Not all files will be this large, but regardless of file size, it is important to limit the amount of disk space consumed.
Next, because Morningstar’s XML files use UTF-8 encoding, it is important to utilize a parser that has the ability to correctly handle Unicode.
Finally, an XML library should always be used to open and parse Morningstar’s XML files. The XML files should not be opened using Internet Explorer or any XML editing software. These applications will copy the entire file into memory (e.g. DOM), which can be several times the size of the file on the disk (e.g. a 1GB file might become 8GB in memory and may take a few hours to load, depending on your hard drive data access speed and processor speed).
The use of unidirectional, event driven push or pull parsers are tools that, if available, can help you work through the process of getting data ready to load into your database.
A broadly supported example of a unidirectional event driven push parser is a SAX Parser. Though popular, this type of parser has been noted to have its drawbacks with regards to ease of use and the amount of coding needed to implement. However, the benefit to using this type of parser is that it can scan through and parse gigabytes worth of XML data without hitting resource limits. The following are some examples of XML libraries that can be used to implement the SAX parser:
❑ JAXP (Java API for XML Processing)
❑ LibXML
❑ MSXML
❑ Xerces
❑ Crimson XML
❑ Fusion XML SAX Parser.
The pull parser has been noted as a high performing alternative to other parsers such as DOM or SAX implemented parsers. StAX and XPP are the more notable of the pull parsers. The benefits to this type of parser are its ease of use and low memory consumption.
1 Database Set-up
If you are beginning this process from scratch and do not yet have a database in place, Morningstar recommends using a two database set-up to work with the XML files and data.
The two database set-up will include one database for staging the data while the other database is used for production. Following this method allows the production systems to be unaffected by the process of loading the daily or monthly files. Since the monthly file will include the entire universe of data, it will be a heavy processing load on the production database, so the aim is to minimize this impact by pushing that load to the staging database. Once all of the data is moved into the staging database from the XML file, a process to load the new data to the production database using an internal mechanism (e.g. replication) would need to be run.
2 Warnings
1 Hard Coding and Lookup Tables
Morningstar strongly recommends against hard coding and the use of lookup tables with our data. The use of hardcoded datapoints and lookup tables can cause errors when modifications to data sets are made. The assumption cannot be made that there are datapoints that will always remain static. The slightest change in naming conventions or even additions of one datapoint option (such as adding a new fund category) can greatly impact your processing. We especially strongly recommend against hard coding or placing the following sets of datapoints in lookup tables:
❑ Indexes
❑ Legal Types
❑ Categories
❑ Exchange lists
2 Restrictions and Filters
Finally, when implementing restrictions and filters in your process, take into account that stringent filters and restrictions will cause unnecessary flags that could delay your processing. If necessary, schedule a meeting with your Client Manager to review.
Schema Changes
The changing financial environment often requires us to adjust or enhance our data feeds. These changes may include adding additional data points, as well as altering file formats. All data feed file adjustments will be outlined in our Release Notes, which are discussed further in the section titled “Communications from Morningstar.”
To help our clients manage file updates we limit new XML releases to a quarterly schedule. The flexibility of XML allows us to make updates more frequently. We will make our best efforts not to alter the existing file structure as we expand our package solutions. In turn, our clients’ processes will not require changes because the XML paths will remain the same for current elements.
Adjusting data point formats is not considered a structural change and can occur during the standard quarterly enhancements. Morningstar also reserves the right to add selected data off cycle. Files may be adjusted prior to the quarterly updates to meet the growing demands of the marketplace or capture methodology enhancements and regulatory changes. Also note that the addition of new data points is also not considered a structural change as you should be coding to relative locations within the files and not hardcoding them.
All clients will be notified one month in advance of all quarterly changes and test files will not be available for these enhancements since they will not affect existing data paths. However, three months of testing will be provided for structural alterations to XML files. Test files will be posted to a designated folder on the FTP site. After three months only the new versions will be available in the standard FTP location.
1 Test Files
Test files will be provided whenever there is a structural change to an XML schema. The timeline and changes will be outlined in Release Notes. During this time period, we will have parallel deliveries for the current Monthly production files and the test files. Test files will be delivered to a designated FTP site and will be delivered after the regularly scheduled Monthly production files.
Clients are strongly encouraged to begin development work as soon as possible, at the beginning of the allotted 90 day window. All available test files should be used to check that ongoing files will not cause your systems any errors. Should development or structure change questions arise, ensuring that these are addressed during the testing period will facilitate the use of live files going forward. As always, feel free to contact your Client Manager for assistance.
The designated FTP site for test files is the Monthly Test folder. Clients will be provided with access to the Monthly Test folder on the Morningstar FTP.
The location of the Monthly Test folder is:
Monthly test files will be placed in this folder instead of the “live” FTP locations. Additionally, this location can be used for one-off file postings.
Contractual Requirements
1 Licensed Datapoints
The DataWarehouse package will include all datapoints that Morningstar offers for licensing. When the client downloads the package, no datapoints are restricted within the package. However, clients may only pull those datapoints for which they are licensed. The contract (generally the Contract Exhibit) will contain a list of those datapoints that are covered under the licensing agreement.
2 Copyright, Logo and Disclaimer Information
The following disclaimer will need to appear in every instance where data licensed from Morningstar is used or displayed along with the date to which the data pertains. The disclaimer may be included in the end note of a print publication or a printed report generated by a software product, the initial login, or splash screen of a software product or every discrete page of a Web site.
©[Insert year] Morningstar, Inc. All Rights Reserved. The information contained herein: (1) is proprietary to Morningstar and/or its content providers; (2) may not be copied or distributed; and (3) is not warranted to be accurate, complete, or timely. Neither Morningstar nor its content providers are responsible for any damages or losses arising from any use of this information. Past performance is no guarantee of future results.
Please refer to your contract for other specific notices and disclaimers pertaining to the data contained in your feed.
Communications from Morningstar
1 Data Release Notes
Morningstar e-mails an informative summary of current data initiatives, enhancements, or changes known as “Morningstar Data Release Notes” (Release Notes) every month. Morningstar will provide additional communication to any client affected by major changes, but the Release Notes will ensure that you have access to the most timely and pertinent information available regarding our data initiatives. Please contact your Client Manager if you have any questions regarding Release Notes or the distribution list.
Frequently Asked Questions (FAQ’s)
Q. In what formats is the DataWarehouse package available?
A. The package is only available in a wrapped XML format.
Q. What is XML?
A. Extensible Markup Language (XML) is a markup language for documents containing structured information. A markup language is a mechanism to identify structures in a document.
The XML specification defines a standard way to add markup to documents. XML provides a facility to define tags and the structural relationships between them. In the case of the XML files that Morningstar is providing to you, it is a way for Morningstar to ensure that we can help you to easily and continually synch your database with the latest data available in our databases.
Q. What is a schema?
A. The XML Schema Definition Language is an XML language for describing and constraining the content of XML documents. It is also known as an XSD. We can define the XML document in terms of constraints on the elements and attributes that can appear and the relationships and types of data for each, among other things.
Q. What is the typical size of an XML package?
A. There is no exact answer for this question as the size of each XML package will vary depending on the number of investments included in the package and the amount of datapoint coverage that we have for each investment. Our packages currently range anywhere from 1KB to 4GB. Please keep in mind that as we expand datapoint offerings and data is populated in the database, package sizes will fluctuate.
Q. How do I know if I’m mapping to the correct datapoints?
A. Your Morningstar Client Manager is available to help with mapping questions. Additionally, a DataWarehouse dictionary will be made available to you and should assist you in your mapping endeavor. The dictionary will contain descriptions of all datapoints in the package as well as field names and paths, for your convenience.
Q. How are the files delivered?
A. Files are delivered via an Internet File Transfer Protocol (FTP) account at a private Morningstar FTP site. You will be provided with a username and password in order to access each FTP site.
Q. Why can’t Morningstar push files to clients? Why do we have to go to the Morningstar FTP site to download files?
A. The most efficient way to distribute the files to all clients, both for clients and Morningstar, is to have Morningstar host the FTP site onto which the files will be dropped. This way, Morningstar has full control over the FTP site and the server. Should issues arise with the server or the FTP site, Morningstar can quickly troubleshoot the issues and ensure that your packages are delivered on a timely basis.
Q. Why can’t I access the FTP site?
A. The most common reasons why the FTP site is not accessible are: an incorrect FTP site address, an incorrect login, or a firewall.
❑ The FTP site address is (substituting the appropriate Frequency and PackageName as noted in the FTP Site section)
❑ Login information can be obtained by contacting your Client Manager.
❑ All clients are advised to look into whether a firewall will restrict access to Morningstar’s FTP site. If so, arrangements should be made with your Network Administrator to gain access to the FTP site.
For additional assistance troubleshooting FTP site access issues, please contact your Network Administrator or your Morningstar Client Manager.
Q. Why are my data packages no longer on the FTP site?
A. The FTP site automatically purges a file 10 days after it is posted in preparation for the next round of file deliveries. Please access the files soon after posting in order to ensure that an update is not missed. We will typically not repost files to production locations, as this would affect all clients who monitor the FTP site via an automated process. Please contact your Relationship Manager if you are in need of a file that is no longer available on the FTP. If the file is still available in Morningstar’s archives, the Relationship Manager may be able to post the file to a Test location on the FTP, but files are never guaranteed to be archived for any period of time.
Q. Why is the data in my package different than , Direct, Principia, Advisor Workstation, etc.?
A. As our data teams continually strive to collect the most up-to-date and accurate data, the timing difference of production processes accounts for differences in investment and data availability. Additionally, each product has a different market focus so while all products source data from the same central database, it is possible for products to have different display logics for different data points. Data feed files are meant to capture raw data sets for clients to aggregate and display according to their specifications or needs. It is for this reason that there may be differences in the displays of the data between feeds and a software product. Finally, keep in mind that feed products are a snapshot of our database in time, while our software products are consistently connected to our central database and updating throughout the day.
Q. When are packages delivered?
A. Your package delivery schedule will be specified in the contract. Please check your contract for your delivery timeline or contact your Morningstar Client Manager.
Q. Who do we contact if there is a question regarding the package or package delivery?
A. Your Client Manager should be your initial point of contact.
Q. How often are there enhancements to the packages?
A. Timely and accurate data is at the core of our business and we understand the importance of continuity within our feeds, but changes occur due to technology, product, and methodology enhancements. In order to facilitate a more streamlined process, we will only make changes to our packages, as necessary, on a quarterly basis.
Q. How are we informed of updates to the packages?
A. An e-mail containing the Release Notes will be sent on a monthly basis. This notification is to provide you with an organized and detailed description of upcoming changes and enhancements to your current package(s).
Q. Is it important for me to download and process all daily or weekly delta XML files? What should I do if I miss a file?
A. Yes, all daily or weekly delta files should be downloaded and processed in chronological order to ensure that you have the most updated and complete data set for each fund. If a file is missed, then, if available, you should process that file before processing any additional files. If the file is unavailable, then you will miss any updates in that file and will have to wait for the monthly file to receive the entire universe of data for all funds and start the month over with a fresh and complete data set.
Q. Why is today’s daily delta file empty? Is this correct or do I need to re-download the file?
A. There will be occasions when a daily delta file will be empty simply because no funds in that particular universe received any data changes during the production cycle. If you received no errors during your download process then having received an empty file is fine.
If you feel that the empty file is an error, take a look at the updateIds file and see if there are any Investment Vehicle IDs listed in the file. If that file is empty as well, there is no doubt that you have received the correct file.
Q. Why is today’s delta file so much larger than other days?
A. Our data department strives to collect the most updated data as it is made public. As such, there are time periods when we experience a spike in the level of data submissions for the different datapoint types, and, as a result, you see a spike in the size of the delta files.
Operational data updates are heaviest during the prospectus and annual report filing seasons (around March, May and September). Portfolio data updates occur every month-end and you will generally see an increase at each quarter-end. Finally, performance data is heaviest at the end of the month when pricing and total net asset data is submitted.
For planning purposes, please ensure that your processes can handle a full monthly file as there may be instances when the volume of deltas is high enough for all investments in the universe to be included in a delta file.
Q. If I subscribe to a monthly package, do I need to process the daily files?
A. It is not required that a monthly subscriber process the daily files. However, it is our recommendation that your development team set up a process to pull down daily files. This will allow you the option to pull down a daily file when you deem it necessary or if we notify you that an important updates or new and necessary information needs to be provided to you.
Q. If I subscribe to a daily package, do I need to process the monthly file?
A. Yes, the monthly file is also an important file to clients receiving daily packages. After the end of every month, our data team recalculates Morningstar Ratings and proprietary statistics utilizing the latest end of month pricing data. Trailing return dates also roll forward to the latest month-end. As a result, all funds are updated with new data and the monthly file incorporates all of these updates for your convenience.
Q. Why are there no daily delta files during the first {two, three…six } business days of the month?
A. Depending on the universe of data, the first days (anywhere from two through six) business days of the month are reserved for Morningstar’s database production. In this time, data is being collected, tested for accuracy, corrected, recalculated and packaged to be made available to you. For this reason, no daily files are made available since we want to ensure that you receive complete and accurate data. The first file of the month will incorporate all data for all funds, so any changes that occurred since the last daily file will be captured in this file.
Q. What is an Investment Vehicle ID?
A. The Investment Vehicle ID is Morningstar’s standard identifier. Internally we refer to it as a Secid. It is a unique 10 digit alpha-numeric identifier that is assigned to every investment vehicle when it is added to our database. You will see this ID referenced in the XML files within a node called “InvestmentVehicle _Id.” Since this identifier is unique and stays with the investment regardless of name, Ticker. CUSIP or other identifier changes, we recommend that you utilize our Investment Vehicle ID as a reference to map to ongoing data updates from Morningstar. This will ensure that should a Ticker change, for example, that continuity of data updates is preserved.
Q. Why am I seeing ^M in the files that I downloaded?
A. In Windows, the file uses Carriage (CR) and LF (Line feed) for the end of the line. The common practice is for FTP protocol to remove the CR. Linux uses only LF, so CR becomes an extra “invalid” character which is displayed as “^M” in Linux editor.
Q. How many concurrent connections can we have to the FTP?
A. Each client (login) is limited to up to 10 concurrent connections, at any time, as additional connections could be rejected, refused or forced to be terminated unexpectedly.
Q. What happens to data feeds when the stock markets are closed for a non-scheduled trading holiday?
A. On rare occasions, the stock markets close unexpectedly (for instance, due to the death of a former President, severe weather systems, or other unforeseen events). In general, most data feeds will continue to deliver on these days, as planned. There may be some exceptions, depending on both the nature of how the markets react to the situation, and on the content of the data within the given file. Any feeds containing daily performance data would likely be low in coverage. Files may also be smaller in size and delivered later than usual.
Q. Why would an Id appear in multiple deliveries of the underreview or delete files? Why would data for “deleted” investments continue to get updates?
A. An Id will appear in multiple deliveries of the underreview or delete files if any of the direct or linked data for that investment is updated. Our .txt files utilize the same delta logic – if any portion of the data that ties back to that investment Id, our “delta” process will flag that investment for inclusion in the next delivery. However, the “status” of the investment will determine in which file that Id shows up. If the “status” of the investment points to an active investment in our database, the id and it’s data gets included in the XML and updateId’s files. If the “status” of the investment points to an active investment, but it’s data is under review within our database, the investment’s Id gets included in the underreview file. If the “status” of the investment is obsolete (which includes liquidated and merged funds), the investment’s Id is included in the delete.txt file.
................
................
In order to avoid copyright disputes, this page is only a partial summary.
To fulfill the demand for quickly locating and searching documents.
It is intelligent file search solution for home and business.
Related download
- section 1 licensed data general info morningstar
- financial planner s approach to investment selections for
- dear client morningstar log in
- morningstar s risk adjusted ratings
- chapter 19 predicting mutual fund performance
- oversight policy for deferred compensation
- af outline northwestern university
- mutual fund governance and fund performance
Related searches
- the nature of science section 1 answers
- 14th amendment section 1 summary
- 14th amendment section 1 meaning
- article 1 section 1 constitution
- chapter 15 section 1 pages 532 537
- section 1 5 salary
- section 1 reinforcments
- article ii section 1 of the constitution
- section 1 chapter 2 science
- 14th amendment section 1 explained
- 14th amendment section 1 text
- economics section 1 assessment answers