OEM Ready Testing



OEM Ready TestingThere were three issues identified during testing:[CUST-1431] OEM Ready Test Issue: UAC Manifest Failure on ATUpdatePBA.exe and ATUpdateFDE.exe [CUST-1432] OEM Ready Test Issue: Verify that the application checks for the OS version in the approved way (ATUpdatePBA.exe and ATUpdateFDE.exe) [CUST-1435] OEM Ready Test Issue: Verify that the application passes specific AppVerifier checks (ATUpdatePBA.exe) The first two issues are straight forward to correct. We should check all of our executables for these two issues. The third issue will need some support from engineering.OEM Ready ProcessDevelopment guidance?Plan AheadBefore you buildDetermine your application’s eligibility:Read the OEM Ready program documentation to understand the application’s eligibility and what test cases applications must pass: . New Update Available Now!If your software is already Certified for Windows Vista, you are already considered OEM Ready.Development first stepsBuild & Pre-Test?3 weeks priorTest your application:After you build your application, confirm it will pass the OEM Ready Program Test Cases.? Download the OEM Ready software qualification test suite: . New Update Available Now!Test suite training materials:Manual: : Send test suite feedback to: oafeedbk@Technical support for the OEM Ready program: mdasw@OEM Ready Program Frequently Asked Questions?OEM Ready Testing Issue Resolution Process:Questions should initially be directed to the technical support email address: mdasw@.? If you encounter an issue during the OEM Ready testing process, it must be resolved prior to making a test submission, either by correcting the issue or by receiving an OEM Ready Resolution ID number.? You must complete an OEM Ready Testing Issue Resolution Request Form to receive an OEM Ready Resolution ID number.? The form is designed to help gather all of the information that the OEM Ready team may need to help you resolve a testing issue in the shortest time possible.? Please complete one form per unique testing issue.OEM Ready Testing Issue Resolution Form: final steps?Submit Test Results1 week priorOEM Ready Submission Process OverviewAfter completion of testing applications for the OEM Ready Program, you can submit the test logs to Windows Quality Online Services (Winqual) for processing. To submit results for qualification consideration, please ensure that you follow these steps.Obtain a code-signing ID.This ID is required for submitting your product to the OEM Ready Program through Windows Quality Online Services. This ensures your organization is authentic and to secure transmission of applications and test results to Microsoft.Establish a Winqual accountA Windows Quality Online Services (Winqual) account is necessary to submit products to the OEM Ready Program.Submit Test Results Log on at: Select from WINDOWS LOGO PROGRAMS --> SOFTWARE --> OEM Ready Edit an existing submission, or click CREATE NEW to start a new OEM Ready submission Upload your qualification report and log files package Answer the questions about your application Sign any required legal agreementsCheck Submission Status.After the completion of the review, the status of the submission ID will change to indicate whether the application has passed or failed testing.? If your test submission fails, you can submit new logs to Winqual for retest after reported failures or omissions are corrected.OEM Ready Program Frequently Asked QuestionsProcess / Business Questions?What is the OEM Application Fundamentals Signature Program? The OEM Ready Program is the new name for the OEM Application Fundamentals Signature Program.? The proper name is OEM Ready and the signature portion of the program has been eliminated so we changed the name.? ?Is it technically possible for OEM to create an image to be shipped to customers while the application is in the process of completing the OEM Ready process? No.? All applications need to be submitted to the OEM Ready program before the final image is built by the OEM.? Exceptions to this rule will be handled by the individual OEMs.?Do applications need to be signed as part of the OEM Ready process? Signing is not required for the OEM Ready program.? Applications simply need to be submitted once successful testing is completed.? However, if the application includes drivers they will need to be signed through the Windows Logo Program for Hardware.?Do applications that are not preloaded (not installed) on OEM systems need to pass OEM Ready? The OEM Ready program is aimed at preinstalled and pre-loaded applications.?Are driver developers included in the OEM Ready program? Driver vendors are included in the OEM Ready program today.?Does Microsoft make sure all the applications embedded within the OS pass all the OEM Ready tests? Applications shipped within Windows are excluded from the program.?How does an ISV/OEM? get started with the OEM Ready Program? Go to either: Innovate On Windows Vista?or OEM Ready Program?How does an ISV/OEM get technical support for the program and/or tests? Send email to mdasw@?Is there an associated fee to get technical support? No – There are no fees associated with the support alias.?Where can I get the OEM Ready test suite? Go to:? OEM Ready Program?How do I provide Feedback on the OEM Ready test suite? Feedback on the test suite can be emailed to oafeedbk@Submission Process?What is the submission process? After completion of testing applications for the OEM Ready Program, you can submit the test logs to Windows Quality Online Services (Winqual) for processing. To submit results for qualification consideration, please ensure that you follow these steps.Obtain a code-signing ID.This ID is required for submitting your product to the OEM Ready Program through Windows Quality Online Services. This ensures your organization is authentic and to secure transmission of applications and test results to Microsoft. Establish a Winqual accountA Windows Quality Online Services (Winqual) account is necessary to submit products to the OEM Ready Program. Submit Test Results Log on at: Select from WINDOWS LOGO PROGRAMS > SOFTWARE > OEM Ready Edit an existing submission, or click CREATE NEW to start a new OEM Ready submission Upload your qualification report and log files package Answer the questions about your application Sign any required legal agreementsCheck Submission Status.After the completion of the review, the status of the submission ID will change to indicate whether the application has passed or failed testing.? If your test submission fails, you can submit new logs to Winqual for retest after reported failures or omissions are corrected.?What is the turnaround time on a submission? The goal of the program is to turn around application submissions within seven days.?After finishing pre-test, will it be ISV or OEM to submit to Microsoft for certification? The owner/publisher of the tested application makes the submission through the OEM Ready submission process.?If one product has passed Windows Vista Software Logo Program, is it necessary to also pass the OEM Ready tests? No.?If the version has passed Windows Vista Software Certification process, they automatically are included in the OEM Ready program.?Do applications that are part of a hardware logo certified package need to complete OEM Ready testing and go through the submission process? Yes.?When can I begin submitting applications to the OEM Ready Program? The program is open for submissions today.?Is a Windows Quality Online Services (Winqual) account needed to submit for the OEM Ready Program? Yes – please see for more details.?Which Verisign Certificate is required for a Winqual account that will work for OEM Ready submissions? The Organizational Certificate ($99 USD) will work for OEM Ready submissions.?Some system recovery software is considered to be critical system software by OEMs and should not be uninstallable by the end user. Is it possible to obtain a waiver for the Uninstall test? Yes.? These type of critical applications will be considered for waivers.? Please submit your request through the submission process once it is published.?What about all the applets that are included in Vista OS such as Calculator, Paint, games, etc.? Do we need to concern ourselves with these? No.? Applications included within Windows are excluded.General?Do both 32- and 64-bit versions of an application need to be tested? No. You can choose one. 32-bit is preferred.?What about locales and different Vista SKUs? One locale and Vista SKU will be sufficient. Vista Ultimate should be used.?If the application setup is accessible from the desktop, is the application considered a candidate for certification? Yes. That means OS version check and AppVerifier tests apply. If the application is already installed when it leaves the factory, OEM Ready tests would not apply to the setup.?The OEM Ready tests require a clean OS. What about interaction with other apps? Today the OEM Ready program does not include any application inter-operational scenarios. There might be interactions that cause tests to fail, but those will be handled on a case-by-case basis.?Are there specific hardware requirements? No, not at this point. Any machine that runs Vista Ultimate.?Are utilities for drivers candidates for the OEM Ready test? Yes, utilities are tested like any other application.?Do executables need to be provided with the submission? No.?Can fonts be installed in the %windir%\fonts directory? The current test case states: "If the application installs drivers and/or Windows redistributive components, they may be installed to existing Windows directories." Fonts are allowed to be installed.?Is an application that is added to the Media Center menu considered a plug-in? Most Media Center applications are stand-alone (not a plug-in) and require OEM Ready testing. Any exceptions would require a waiver.?If an executable is only used during OOBE, does it need to comply with OEM Ready? No, only applications and services that run during normal user desktop experience need to be tested.?How should a product (suite) which bundles several different applications together be handled; for example, Microsoft Office? You will need to submit the application suite (Office) and test the individual applications. This means that TC2, TC4 and TC6 apply to the suite, whereas each individual application should be tested against the remaining four test cases.?How can we get a waiver for requireAdministrator? The waiver process is described on the OEM Ready MSDN page.Test Case #1 – UAC?Do services have to be manifest? Yes. But since the split token only applies to interactive processes, "asInvoker" is the recommended setting.?If the manifest file is encrypted, would it still pass? Is an exemption needed? Encrypted manifests will usually fail the test. Applications with encrypted manifest files should request a waiver. These are handled on a case-by-case basis.?Is a waiver needed for all applications that are not manifest with asInvoker but with requireAdministrator or highestAvailable? No. You only need waivers for the main executable. Any child executable that has requireAdministrator does not require a waiver.?Can a file have an external manifest? Yes. However if an internal manifest is already present (for example for Side by Side), the external manifest will be ignored.?Do Command line applications also need a trustInfo section with requestedExecutionlevel (Test Case #1)? Yes. All applications need it.Test Case #2 – Driver?What are kernel-mode drivers? In short, a Kernel-mode driver is one which interacts with the kernel. A KMDF driver consists of a DriverEntry function that identifies the driver as based on KMDF, a set of callback functions that KMDF calls so that the driver can respond to events that affect its device, and other driver-specific utility functions. Nearly every KMDF driver must have the following:A DriverEntry function, which represents the driver's primary entry point. An EvtDriverDeviceAdd callback, which is called when the Plug and Play manager enumerates one of the driver's devices (not required for drivers that support non-Plug and Play devices). One or more EvtIo* callbacks, which handle specific types of I/O requests from a particular queue.You can visit and if you want to know more on this. For the OEM Ready program, you need to have a .cat file for a kernel-mode driver. You need to choose the catalog file in the listbox and select all the kernel-mode drivers by the checkbox. However, you should be aware if you are using such a driver in your application. If you do not have a kernel-mode driver, this test is not required.?Microsoft should rewrite the code for Embedded Boot Driver Test 2.3 to work with Vista Russian. For the current release, only English is supported.?Exactly what kind of signature is this and how does the ISV obtain it? E.g. Can they simply sign the executable themselves? If not, what kind of testing do they have to go though? Driver signatures are NOT the same that we use for regular executables (from CAs like VeriSign). The ISVs need to go through the driver signing process through Microsoft. Here are few resources that should be helpful.Driver Signing Requirements for Windows: About the Windows Logo Program: Installation and Driver Signing: Case #3 – Version Check?Test Case #3 also uses AppVerifier. Does it apply to managed code? Yes it does. The install monitoring in test 6 also applies to managed code.?Does an application have to pass all three conditions for an OS version check? (EULA, message, NT Eventlog) Only if it checks the OS version and fails to run.Test Case #4 – Uninstall?What is the reason for all of the keys for Uninstall? The keys are needed so that end-users can uninstall the application from Programs and Features.?Some installer products may not populate VersionMajor and VersionMinor and only Uninstall string. What's the guidance on that? The six entries in the registry must exist.?Some installer products populate MajorVersion and MinorVersion instead of VersionMajor and VersionMinor. Is that acceptable? Yes.?Can we have an application that an end-user is not allowed to uninstall? There are some applications that are deemed necessary for the stability of the machine and should not be uninstalled. A waiver is required for these on a case-by-case basis.?What if an application creates two entries in the Uninstall key. One has ARPSYSTEMCOMPONENT set to 1 and misses the Uninstall entry. The other entry has all six required entries. Is this OK? If the one that is picked up by ARP (Add or Remove Programs) has all the information, it would pass this test case.Test Case #5 – AppVerifier?Do the AppVerifier tests also apply to Uninstall? No. Not to setup.exe either.?Does a warning in AppVerifier log mean you pass or fail a test? It depends on the test case. A break in the debugger would be considered a fail.?It is mentioned that TC5 does not apply to managed code. Is a waiver needed for this? No. It can be marked as N/A.?It is mentioned that TC5 does not apply to fully managed code. How do we know if the application is fully managed? If you can load it in FxCop, it is managed code.?Is there a way to restore the state of the registry to before WinDBG–I was run? You can export the hive "HKLM\Software\Microsoft\Windows NT\CurrentVersion\AeDebug" and "HKLM\Software\Wow6432Node\Microsoft\Windows NT\CurrentVersion\AeDebug" to a reg file before running, or save it from another machine and then import it. There is no WinDBG flag or solution.?Application does not break in the debugger, but Application Verifier gives an error. Do we need to fix it? Yes.?Application Verifier does not work correctly for Tamper Resistant Software. Can we get a waiver for this? The ISV would need to test the software without the Tamper Resistance enabled and be able to submit AppVerif logs.?When running AppVerifier, we prefer not to break into the debugger. This way we can run through all our automated tests without the break interruptions. Is this possible? In AppVerif you can configure the Stop options to not hit a BreakPoint when something happens. Breaks are only written to the log file.?Does AppVerif have an event mechanism one can subscribe to for log errors? The automation layer for AppVerif can drive AppVerifier but has no event functionality.?On shutdown of an application, AppVerifier complains about not properly freeing TLS: 0350. However the application is going away and the memory is reclaimed anyway. Paging in memory for the sake of cleaning up nicely will slow down the process of shutting down, when it will get reclaimed by the system anyway. AVRF ignores outstanding resources at shutdown because it is a common and recommended behavior to not free them. AVRF only complains when a DLL is dynamically unloaded and it has outstanding resources.?AppVerif breaks in debugger on a 0013 Stop. How do we debug from this? Generally the first thing you want to do is !analyze –v. From there the help from AppVerif in the debugger is self-explanatory. For !analyze –v to work, the debugger needs to be able to find the pdb files.Test Case #6 – Install to correct folders?The requirement doesn't mention %programdata% or %localappdata%. TC6 is about where binaries should be installed. The correct location of data files is not addressed in OEM Ready spec.?Can the OEM install software to a different location than %ProgramFiles%? TC6 applies to binaries. Exe, DLLs, etc. should go to ProgramFiles or %AppData%. Documents can be installed in %public% or %user%.?If an application creates a log file in C:\ is that allowed? Yes. There is nothing in the OEM Ready program that prohibits this.?Is MSI a requirement for OEM Ready? No, MSI is not a requirement. A clean, reversible, transaction-based installation is required.?AppVerif gives warnings in TC6 for hard-coded paths. Is that a problem? Although hard-coding paths is not recommended practice, it will not fail the OEM Ready test.?Some Installer packages make some temporary folders or files and leave them after installing. Imaging tool (System State Analyzer) lists these temporary files. Can those be ignored? Yes. TC6 applies to binaries that an end-user will execute under normal circumstances (start menu, programs, desktop, etc).?There is no relationship between the AppVerifier FilePath log and the System State analyzer. Correct, the AppVerifier FilePath test is misleading and should be removed.Test Case #7 – Restart Manager Aware?What is an idle state? There are applications that don't have an idle state (e.g. just a wizard that runs through). TC7 would not apply to these.?On some applications, several .exe files need to communicate with each other for the shutdown process. There is no way to send shutdown message to several .exe files simultaneously. How we can test this kind of application for Restart Manager Awareness testing? The RMLogoTest allows for up to 25 processes to be tested simultaneously.?My Protected Mode IE broker is an ActiveX .exe file. The RMLogoTest tool reports the application as failing. RMLogoTest only works on "discoverable applications". Please see the program documentation for browser plugins.Test Run Results:Log File: TimeSeverityData2008-10-21T20:56:08.1058325ZInformationINFO: Executing test case 67eed312-c60b-4f75-939d-ac6d31f531402008-10-21T20:56:08.3554325ZInformationINFO: Checking the manifests of executables...2008-10-21T20:56:08.3710325ZInformationINFO: Total number of executables: 22008-10-21T20:56:08.4178325ZInformationINFO: Command line: C:\Tools\sigcheck.exe -m "C:\Program Files\Fingerprint Sensor\ATUpdatePBA.exe"2008-10-21T20:56:08.9482325ZInformationINFO: SigCheck output: c:\program files\fingerprint sensor\ATUpdatePBA.exe: Verified: Signed Signing date: 12:33 PM 10/3/2008 Publisher: AuthenTec, Inc. Description: Secure update application Product: ATUpdate Version: 8.0.0 File version: 8.0.0 Manifest: <assembly xmlns="urn:schemas-microsoft-com:asm.v1" manifestVersion="1.0"> </assembly> 2008-10-21T20:56:08.9482325ZInformationINFO: Embedded manifest exists: <assembly xmlns="urn:schemas-microsoft-com:asm.v1" manifestVersion="1.0"> </assembly>2008-10-21T20:56:09.4006325ZWarningEXCEPTION: <trustInfo> cannot be found in the manifest../n at Microsoft.LogoTest.mon.ManifestChecker.VerifyManifest(String manifest)2008-10-21T20:56:09.4006325ZWarningWARNING: Intermediate result: C:\Program Files\Fingerprint Sensor\ATUpdatePBA.exe FAILED the test. It does not have manifest that is compliant with UAC guidelines.2008-10-21T20:56:09.4162325ZInformationINFO: Command line: C:\Tools\sigcheck.exe -m "C:\Program Files\Fingerprint Sensor\AtService.exe"2008-10-21T20:56:09.6970325ZInformationINFO: SigCheck output: c:\program files\fingerprint sensor\AtService.exe: Verified: Signed Signing date: 12:33 PM 10/3/2008 Publisher: AuthenTec, Inc. Description: AFSS Service Product: AuthenTec Fingerprint Security System (AFSS) Version: 8.0.200.33 File version: 8.0.200.33 Manifest: <assembly xmlns="urn:schemas-microsoft-com:asm.v1" manifestVersion="1.0"> </assembly> 2008-10-21T20:56:09.6970325ZInformationINFO: Embedded manifest exists: <assembly xmlns="urn:schemas-microsoft-com:asm.v1" manifestVersion="1.0"> </assembly>2008-10-21T20:56:09.6970325ZWarningEXCEPTION: <trustInfo> cannot be found in the manifest../n at Microsoft.LogoTest.mon.ManifestChecker.VerifyManifest(String manifest)2008-10-21T20:56:09.6970325ZWarningWARNING: Intermediate result: C:\Program Files\Fingerprint Sensor\AtService.exe FAILED the test. It does not have manifest that is compliant with UAC guidelines.2008-10-21T20:56:09.6970325ZErrorASSERT: The test failed because one or more of the executable did not comply with the UAC guidelines2008-10-21T20:56:09.7438325ZInformationINFO: Test result: FailedLog File: TimeSeverityData2008-10-21T21:00:11.3566325ZInformationINFO: Executing test case ce1c166d-3f8d-47f4-8a85-64cf58012bd12008-10-21T21:00:11.3722325ZInformationINFO: Verifying that all kernel-mode drivers are signed...2008-10-21T21:00:11.3722325ZInformationINFO: Number of driver files installed by the application: 02008-10-21T21:00:11.3722325ZInformationINFO: Number of catalog files installed outside catroot folder: 02008-10-21T21:00:11.3878325ZInformationINFO: Test result: PassedLog File: TimeSeverityData2008-10-21T21:10:43.1848260ZInformationINFO: Executing test case df013890-ec0a-4de7-82d7-ab21d159064e2008-10-21T21:10:43.5124260ZInformationINFO: Verifying that the applicaction's boot start drivers have embedded signatures2008-10-21T21:10:43.5280260ZInformationINFO: Number of drivers installed by the application: 02008-10-21T21:10:43.6060260ZInformationINFO: Test result: Passed ................
................

In order to avoid copyright disputes, this page is only a partial summary.

Google Online Preview   Download