Report on the Pilot of the Netflix Classification Tool



Report on the Pilot of the Netflix Classification ToolAugust 2018DisclaimerThe material in this report is of a general nature and should not be regarded as legal advice or relied on for assistance in any particular circumstance or emergency situation. In any important matter, you should seek appropriate independent professional advice in relation to your own circumstances. The Commonwealth accepts no responsibility or liability for any damage, loss or expense incurred as a result of the reliance on information contained in this report.This report has been prepared for consultation purposes only and does not indicate the Commonwealth’s commitment to a particular course of action. Additionally, any third party views or recommendations included in this report do not reflect the views of the Commonwealth, or indicate its commitment to a particular course of action.Copyright? Commonwealth of Australia 2018 The material in this report is licensed under a Creative Commons Attribution—4.0 International licence, with the exception of:the Commonwealth Coat of Armsthis Department’s logoany third party materialany material protected by a trademark, andany images and/or photographs.More information on this CC BY licence is set out as follows:Creative Commons website—Attribution 4.0 international (CC by 4.0)—licenses/by/4.0.Enquiries about this licence and any use of this discussion paper can be sent to: copyright@.au.Third party copyrightThe Department has made all reasonable efforts to clearly identify material where the copyright is owned by a third party. Permission may need to be obtained from third parties to re-use their material.AttributionThe CC BY licence is a standard form licence agreement that allows you to copy and redistribute the material in any medium or format, as well as remix, transform, and build upon the material, on the condition that you provide a link to the licence, you indicate if changes were made, and you attribute the material as follows:Licensed from the Commonwealth of Australia under a Creative Commons Attribution 4.0 International licence.Enquiries about the use of any material in this publication can be sent to: copyright@.au.Using the Commonwealth Coat of ArmsGuidelines for using the Commonwealth Coat of Arms are available from the Department of Prime Minister and Cabinet website at .au/government/its-honour.Contents TOC \h \z \t "Heading 2,1,Heading 3,2,Heading 4,3" Executive summary PAGEREF _Toc20923013 \h 41.Background PAGEREF _Toc20923014 \h 41.1Classification of film content in Australia PAGEREF _Toc20923015 \h 41.2The Netflix Classification Tool PAGEREF _Toc20923016 \h 51.3Pilot of the Netflix Tool PAGEREF _Toc20923017 \h 51.4Key Performance Measures PAGEREF _Toc20923018 \h 52.Structure of the pilot PAGEREF _Toc20923019 \h 62.1Governance PAGEREF _Toc20923020 \h 62.2Audit program PAGEREF _Toc20923021 \h 72.2.1Phase One: Diagnostic phase—December 2016 to October 2017 PAGEREF _Toc20923022 \h 72.2.2Phase two: Evaluation phase—November 2017 to May 2018 PAGEREF _Toc20923023 \h 72.2.3Definition of ‘broadly consistent’ decisions PAGEREF _Toc20923024 \h 83.Results of the auditing program PAGEREF _Toc20923025 \h 93.1Phase One: Diagnostic Phase (December 2016 to October 2017) PAGEREF _Toc20923026 \h 93.2Phase Two: Evaluation Phase (December 2017 to May 2018) PAGEREF _Toc20923027 \h 93.3Results for ratings only PAGEREF _Toc20923028 \h 93.4Results for ‘Broadly Consistent’ criteria PAGEREF _Toc20923029 \h 103.5Changes to decisions of the Tool PAGEREF _Toc20923030 \h 103.6Performance against Key Performance Measures (KPM) PAGEREF _Toc20923031 \h 11KPM 1: Accuracy in producing classification decisions (ratings and consumer advice) that are broadly consistent with Australian community standards and classification decisions made by the Classification Board PAGEREF _Toc20923032 \h 11KPM 2: Ability to refuse classification to relevant material PAGEREF _Toc20923033 \h 12KPM 3: Ability to display classifications (ratings and consumer advice) produced by the Tool on the Australian Netflix web browser interface PAGEREF _Toc20923034 \h 12KPM 4: Ability to provide classifications (ratings and consumer advice) generated by the Tool in agreed format for display on the National Classification Database PAGEREF _Toc20923035 \h 12KPM 5: Ability to update the Australian Netflix Interface with new classification decisions as a result of revocations made by the Classification Board in a timely manner PAGEREF _Toc20923036 \h 12KPM 6: Satisfaction amongst Australian Netflix users (complaints management) PAGEREF _Toc20923037 \h 12KPM 7: Ability to adapt to changes in the National Classification Scheme PAGEREF _Toc20923038 \h 13KPM 8: Netflix’s ability to make adjustments to the Tool to improve its performance PAGEREF _Toc20923039 \h 13KPM 9: Tool must not produce a classification and consumer advice for films already classified by the Classification Board/Classification Review Board/approved tool/public exhibition films and/or upon advice from the Branch. PAGEREF _Toc20923040 \h 13KPM 10: Ability to advise the Director/Department of all Tool decisions PAGEREF _Toc20923041 \h 134.Conclusion and recommendations PAGEREF _Toc20923042 \h 14Appendix A: evaluation phase results PAGEREF _Toc20923043 \h 15Tables / images TOC \h \z \t "Table/figure heading,4,Table heading,4" Table 1: Audit program results for Ratings only during the Evaluation Phase PAGEREF _Toc20922991 \h 9Table 2: ‘Broadly consistent’ Tool decisions PAGEREF _Toc20922992 \h 10Executive summaryIn December 2016, the Minister for Communications and the Arts, Senator the Hon Mitch Fifield (the Minister) launched a pilot of the Netflix Classification Tool (the Tool) to produce Australian classification ratings and consumer advice (classification decisions) for films and television series available online in Australia via the Netflix streaming service. The pilot was administered by the Department of Communications and the Arts (the Department) in consultation with key stakeholders—Netflix International B.V (Netflix) and the Classification Board (the Board).The pilot period was from December 2016 to May 2018. It consisted of two phases, a diagnostic phase and an evaluation phase where classification decisions of the Tool were reviewed by staff assessors and the Board. A number of key performance measures were developed to assess the effectiveness of the Tool to generate classification decisions that are ‘broadly consistent’ with Australian community standards and classification decisions of the Board.The evaluation found that the Tool generated the same rating, or one rating higher than the Board (where it was a borderline decision only) in 96% of instances. Taking into account the Tool classification decisions that were lower than the Board’s classification decision, and recognising the different application of consumer advice by the Board, the Tool generated classification decisions that were ‘broadly consistent’ in 93% of instances.Where classification decisions of the Tool were not broadly consistent with those classification decisions that the Board would have determined, the Tool classification decision was amended on the National Classification Database.Although there were differences in the consumer advice generated by the Tool and consumer advice applied by the Board, which resulted in the Board changing some of the Tool’s classification decisions on the National Classification Database, overall the Tool is capable of producing consumer advice that informs consumers about the content in a film.1.Background1.1Classification of film content in AustraliaUnder the Classification (Publications, Films and Computer Games) Act 1995 (the Act), films, computer games and certain publications must be classified before they can be ‘published’ (which includes sold or distributed) in Australia.Owing to convergence and the digitisation of entertainment media, there has been a dramatic increase in service providers and the volume of entertainment media content available in Australia (and globally) which has created challenges for the Board in classifying the volume of material. One means of addressing the volume of content is the use of automated tools to make classification decisions. These tools are programmed to analyse data about the content found in a film or game and to generate a classification decision—that is, a rating and consumer advice.The Act was amended on 11?September 2014, to provide the Minister responsible for classification, with the power to approve classification tools for the purposes of classifying publications, films and/or computer games. Classification decisions produced from approved classification tools are taken to be classification decisions of the Board.In 2016, following a successful pilot, an automated tool for mobile and online games operated through the International Age Rating Coalition (IARC) was approved by the Minister for ongoing use in Australia. The IARC Tool uses a questionnaire completed by games’ developers to gather pertinent information about game content and its impact level and from this, it generates the appropriate classification rating and consumer advice for each participating flix—which had been submitting its original content for classification by the Board since it launched in Australia in 2015—developed its own automated tool to classify its film content (the Tool).1.2The Netflix Classification ToolOn 11 October 2016, the Minister approved a pilot of the Tool that was designed and developed by Netflix. The pilot commenced in December 2016. Approval for the ongoing use of the Tool would be subject to a satisfactory evaluation of the Tool. To administer the pilot, the Department developed a Memorandum of Understanding (MOU) with Netflix, with the pilot to be administered by the Classification Branch (the Branch) of the Department.The Tool has been designed to produce classification decisions that are in accordance with the Act, the National Classification Code (the Code) and the Guidelines for the Classification of Films (Film Guidelines).1.3Pilot of the Netflix ToolThe pilot operated from 4 December 2016 to 30 May 2018. The pilot assessed the overall performance of the Tool in generating classification decisions that are ‘broadly consistent’ with Australian community standards and with classification decisions made by the Board. The pilot considered a range of performance measures, agreed between the Department and Netflix, which are outlined below.1.4Key Performance Measures The MOU and the Approval Instrument identify the Key Performance Measures of the Tool. They are:Accuracy in producing classification decisions (ratings and consumer advice) that are ‘broadly consistent’ with Australian community standards and classification decisions made by the Classification Board;Ability to refuse classification to relevant material;Ability to display classifications (ratings and consumer advice) produced by the Tool on the Australian Netflix web browser interface;Ability to provide classifications (ratings and consumer advice) generated by the Tool in agreed format for display on the National Classification Database (NCD);Ability to update the Australian Netflix Interface with new classification decisions as a result of revocations made by the Classification Board in a timely manner;Satisfaction amongst Australian Netflix users (complaints management); Ability to adapt to changes in the National Classification Scheme;Netflix’s ability to make adjustments to the Tool to improve its performance;The Tool must not produce a classification decision for films: already classified by the Classification Board / Classification Review Board / approved tool; public exhibition films; and/or upon advice from the Branch; andAbility to advise the Director of the Board/Department of all Tool decisions.2.Structure of the pilotThe pilot comprised the following broad components:The Tool was integrated with the classification database and the Netflix Australian user interface, to enable the display of Australian classification decisions generated by the Tool.To assess the performance and reliability of the Tool’s classification decisions, in terms of being ‘broadly consistent with Australian community standards and classification decisions made by the Classification Board’ and to identify areas for improvement, it was agreed that a sample of the Tool’s classification decisions would be assessed. This activity was completed by trained Departmental assessors who viewed a film and made an independent recommendation to the Board about an appropriate classification rating and consumer advice. The Board then made a determination on a classification rating and consumer advice for the film. This process is referred to in this report singularly as an audit and collectively as the audit program.The audit program was conducted in two phases:In the first phase of the pilot, results of all audits were communicated to Netflix and used to improve the Tool. In addition, an agreed list of consumer advice was provided to Netflix to optimise consistency of consumer advice generated by the Tool to that applied by the flix made a series of adjustments to the Tool during the first phase and the Tool was determined jointly by Netflix and the Branch to be ready for a final evaluation (the second phase of the pilot).In the second phase of the pilot, results from audits conducted on a random sample of film titles were analysed to assess whether the Tool produced classification ratings and consumer advice that were ‘broadly consistent’ with the classification decisions of the Board and Australian community standards.The pilot was evaluated by the Department against all of the agreed Key Performance Measures and the results presented in this report, along with recommendations regarding the ongoing use of the Tool in Australia.2.1GovernanceThroughout the pilot, the Department met regularly (via teleconference) with the Manager, Global Public Policy, Netflix and with the Director of the Board. The Department was responsible for designing the methodology of the pilot and the evaluation criteria of the Tool. This included liaising with Netflix about issues and optimisation of the Tool, statistical analysis, IT and database integration, reporting of auditing program results and day-to-day administration of the pilot. Netflix, having developed the Tool, was responsible for recalibrating it based on feedback from the Department when issues were identified during the auditing program. The Board was responsible for deciding whether classification decisions of the Tool aligned with classification decisions of the Board and Australian community standards.2.2Audit programTo investigate and assess the performance of the Tool, an audit program consisting of two phases was established:Phase One—diagnostic phase (December 2016—October 2017)Phase Two—evaluation phase (November 2017 to May 2018)Trained Departmental assessors were responsible for assessing the films and providing a recommendation to the Board who then decided if the Tool’s classification decision aligned with the classification decision of the Board for each title. The Board was provided with detailed reports and the opportunity to view the film which it used to inform its decision about whether classification decisions of the Tool aligned with classification decisions of the Board.2.2.1Phase One: Diagnostic phase—December 2016 to October 2017In this phase, both targeted and random audits were conducted to identify issues affecting the alignment of the Tool’s classification decisions with classification decisions of the Board. Feedback was given to Netflix to enable refinements to be made to the Tool. A small number of other titles were targeted for auditing owing to high media profile and/or complaints or enquiries. During this phase, 119 audits were flix continuously refined the Tool throughout this period in response to issues identified. To allow for this, and to chart the progress of the Tool towards being ‘broadly consistent’ with classification decisions of the Board and Australian community standards, titles to be audited were selected from batches of titles that had been classified by the Tool after the most recent recalibration of the Tool. Following completion of the 119 random and targeted audits, and after a series of refinements were made by Netflix to improve the Tool, the Department was sufficiently satisfied with the performance of the Tool to bring the process of diagnosis and adjustment to a close.2.2.2Phase two: Evaluation phase—November 2017 to May 2018This phase focussed on assessing the Tool’s ‘ability to make decisions that are broadly consistent with Australian community standards and classification decisions made by the Classification Board’. To establish a base population number for calculating a robust sample, the Department estimated the number of decisions that the Tool would make during this evaluation period. At the beginning of the pilot, Netflix projected that the Tool would generate 1,400 decisions in 12 months. The estimated population (total number of decisions by the Tool) during the evaluation period would therefore be approximately 350 decisions. Using 350 as a base, the Department calculated that a sample of 117 random audits would be sufficiently robust to test the reliability of the Tool with a confidence interval of 90% and a margin for error of 5%. However, during the period 8?November 2017–31 December 2017, a higher number of classification decisions (473) were made by the Tool. In order to preserve the statistical validity for a larger number of classification decisions than previously estimated, the Department completed 127 audits in the evaluation phase.2.2.3Definition of ‘broadly consistent’ decisions The first of the Key Performance Measures, relating to the Tool’s ‘ability to make decisions that are broadly consistent with Australian community standards and classification decisions made by the Classification Board’ was given particular consideration because it establishes the standard for evaluating individual classification decisions of the Tool. It is crucial to the assessment of the Tool’s reliability, that it produces consistent classification decisions across the range of classification categories, which are consistent with decisions of the Board and Australian community standards.The set of criteria for ‘broadly consistent’ classification decisions by the Tool was developed to ensure that consumers receive relevant information to inform their viewing choices and to effectively protect consumers from inappropriate, unsolicited or harmful content.The Board is required to provide classification decisions that align with Australian community standards. They are trained and—as evidenced by the low number of complaints from the public and positive feedback received during regular community consultation—have a proven track record in this regard.The making of classification decisions requires close attention to detail and a degree of judgement from Board members in application of the Act, the National Classification Code and classification guidelines. On occasion, the Board may have a split decision, rather than a unanimous decision, about what constitutes the most appropriate classification for a film. Such differences of opinion are reflective of the differences in the broader Australian community. This means that some entertainment media may ‘tip-over’ into a higher classification category, or conversely, may be mitigated into a lower classification category. These types of decisions are known as ‘borderline’ classifications, as it is possible to give equal validity to both arguments—one for a higher classification and the other for a lower classification. Further, the Classification Review Board can make a different classification decision to the Board in the context of a review of a classification decision of the Board.The formulation of consumer advice by the Board is determined by the most impactful of the classifiable elements and is governed by Board policy. An automated Tool will produce consumer advice that it has been programmed to generate. It is crucial that consumer advice provides useful information about the content of a film, so that it informs and assists consumers’ choices.As part of the evaluation criteria, it was decided that if the Tool generated a rating that was one rating higher than a rating decision of the Board, and that rating could be considered to be a ‘borderline’ decision, the Tool’s rating would be evaluated as being a classification decision that is ‘broadly consistent’ with the Board and Australian community standards. However, a rating one or more categories lower than that which would have been given by the Board would not be deemed to be ‘broadly consistent’ because such a rating would not provide adequate protection to consumers.Therefore, the definition of ‘broadly consistent’ agreed between the Branch and Netflix for the purposes of Phase Two of the auditing program (evaluation) was as follows:The rating is the same as the rating of the Classification Board; orThe rating is one level higher than the rating of the Classification Board, where the rating could be considered ‘borderline’; ANDThe classification decision contains an advisory from the agreed consumer advice list provided to Netflix and the content is present in the film.3.Results of the auditing program3.1Phase One: Diagnostic Phase (December 2016 to October 2017)Phase One focussed on critically assessing and identifying areas for improvement of the Tool. As such, the ‘broadly consistent’ criteria were not used in this Phase.Of the 119 titles audited in this phase, 90 (76%) were targeted and 29 (24%) were randomly selected. Of the 90 targeted titles, 82 were targeted to assess refinements made to the Tool’s logic and test improvements to the Tool. In addition, three titles were targeted owing to media interest (13 Reasons Why: Season 1, To the Bone, Killer Kids: Season 1); one title was targeted at the request of Netflix (Marvel’s Iron Fist); one title (Big Mouth) was targeted owing to it being the subject of a complaint; and three titles were targeted at the Board’s request (Schitt’s Creek Season 1, 2 and 3).During Phase One, audit outcomes were used to identify trends where the Tool over or under-assessed the impact and/or frequency of classifiable elements as compared with classification decisions of the Board. Over the course of Phase One, each iteration of the Tool showed an improvement in the alignment of Tool classification decisions with those of the Board.3.2Phase Two: Evaluation Phase (December 2017 to May 2018)During the evaluation phase, 127 random audits were conducted by Departmental assessors, drawn from 473 classification decisions made by the Tool between 8 November 2017 and 31?December 2017. Results of the evaluation phase are based on the 127 random audits. Appendix A provides a list of the 127 titles audited during this phase. During this period, an additional six (6) titles were also targeted for auditing at the Board’s request (Ali Wong: Hard Knock Wife; 13 Reasons Why: Season 2; The Honeymoon Stand Up: Special Collection; Sarah Silverman: A Speck of Dust; Rory Scovel Tries Stand-Up For The First Time; and Rodney Carrington: Here Comes The Truth). In order to maintain the integrity of a random data sample for the evaluation, results of these six additional audits have not been included in the evaluation analysis. 3.3Results for ratings onlyDuring the evaluation phase, the Tool generated the same rating or one rating higher than the Board in 96% of instances (122/127). Of these, 117 ratings generated by the Tool were the same rating as the Board and five ratings generated by the Tool were one rating higher than that of the Board. The other five Tool ratings were found to be lower than that of the Board. No Tool rating was more than one classification rating higher or lower than the Board. Table 1 provides a breakdown of results.Table 1: Audit program results for Ratings only during the Evaluation PhaseResult of assessments for rating onlyNumber %Same Rating as the Classification Board 11792One Rating Higher than the Classification Board 54One Rating Lower than the Classification Board 54Total1271003.4Results for ‘Broadly Consistent’ criteriaThe performance of the Tool against the agreed definition of ‘broadly consistent’ during the evaluation phase of the audit program can be seen in Table 2. Of the decisions audited, 93% (118/127) were evaluated to be ‘broadly consistent’ Table 2: ‘Broadly consistent’ Tool decisionsBroadly consistent Tool DecisionsNumber %Tool decision is Broadly Consistent11893Tool decision is not Broadly Consistent (lower rating and different CA*) 54Tool decision is not Broadly consistent (same rating but different CA)43Total127100* CA—Consumer AdviceThe Tool generated 118 classification decisions that were ‘broadly consistent’:Of these, 113 Tool classification decisions had the same rating as the Board. Five Tool classification decisions were one rating level higher than the Board’s classification decision but were considered ‘broadly consistent’, as the decision was evaluated to be ‘borderline’ and contained consumer advice about content present in the film.Nine decisions were found to be not ‘broadly consistent’:Five Tool classification decisions were found to be not ‘broadly consistent’ owing to being one rating lower than a rating of the Board (Alias Grace; A Holiday Engagement; Daughters of Mother India; Maya Memsaab and Happy Together).Four Tool classification decisions were found to have the same rating as the Board, however, they were not ‘broadly consistent’ because of differences in consumer advice that was applied by the Board (Holiday Baggage; Everybody Loves Somebody; YZ; and Once Upon a Holiday). 3.5Changes to decisions of the ToolThe Department worked with Netflix to produce an agreed list of consumer advice that could be generated by the Tool. This made the consumer advice of the Tool substantially more informative even though the list did not encompass the full list of consumer advice that could be applied by the Board. However, the Tool is not able to produce exactly the same consumer advice as that determined by the Board.In the vast majority of cases, the Board changed the consumer advice generated by the Tool because it was either overly comprehensive, partially incomplete, or differently worded. For example, for the film, 13 Reasons Why: Season 2, the Tool generated the following unpunctuated consumer advice of ‘Strong Coarse Language Strong Drug Use Strong Sex Scenes Strong Themes Strong Violence Strong Sexual Violence’ whereas the Board applied the punctuated consumer advice of ‘Strong themes, sexual violence, violence, drug use and coarse language’. The Board did not find the need to include ‘sex scenes’ in its consumer advice as the scenes were not at a strong impact level. It is the Department’s view that it is unrealistic to expect the Tool to exactly replicate the consumer advice of the Board, as it does not formulate consumer advice in the same way. Overall, the Tool was able to generate consumer advice in line with the agreed list of consumer advice that would enable consumers to make an informed choice about viewing the film.For classification decisions of the Tool that were different to those that would have been made by the Board, even in instances where the classification decision was broadly consistent, changes were made to the National Classification Database to reflect the Board’s position. Out of the 252 titles audited during the pilot (119 in the diagnostic phase, 127 in the evaluation phase and six additional Board directed audits), changes were made to 147 classification decisions of the Tool. Importantly, these changes do not go to the quality of the classification decisions, but rather to ensuring consistency with Board practices during conduct of the pilot.For example, at times, the Board made changes when the Tool used the modifier ‘strong’ repeatedly in the consumer advice (i.e. before each element) to only using the modifier once in the consumer advice. For example, for the film Interrogation, the Tool generated consumer advice of ‘Strong themes Strong violence Strong blood and gore Strong coarse language’ whereas the Board applied consumer advice of ‘Strong violence, themes and coarse language. In this instance, the additional advice of ‘blood and gore’ the Tool generated to explain the type of violence in the film is useful to consumers, as it provides further information about the content in the film; whereas, the Board does not include advice for ‘blood and gore’ except where that is the sole and overwhelming type of violence in the film. Based on research about the comparative use of ratings and consumer advice, it is unlikely that such consumer advice generated by the Tool, when viewed alongside the correct rating, would impair a consumer’s ability to choose Netflix content appropriately for themselves or their children.Of the 147 classification decisions changed by the Board:104 Tool classification decisions were changed because the Board determined different consumer advice (the rating was the same as the Board); 43 classification decisions were changed because the Board determined a different rating (and therefore, different consumer advice). Of the 43 classification decisions changed, 33 occurred during the diagnostic phase of the pilot, as titles were targeted to improve the Tool, with the logic of the Tool subsequently being altered. As a result of these improvements, during the evaluation phase, the Board only changed 10 classification decisions because they determined a different rating.3.6Performance against Key Performance Measures (KPM)KPM 1: Accuracy in producing classification decisions (ratings and consumer advice) that are broadly consistent with Australian community standards and classification decisions made by the Classification BoardMeasure: Measurement of Netflix Tool decisions against evaluation criteria.Based on the results of the evaluation phase, it is the opinion of the Branch that the Tool has demonstrated its ability to generate classification decisions (ratings and consumer advice) that are ‘broadly consistent’ with Australian community standards and classification decisions made by the Board. In particular, the evaluation found that the Tool generated a classification decision that was the same, or one rating higher, in 96% of instances (refer Table 1). Given five decisions generated by the Tool were one rating lower than a rating that the Board determined, and the differences around the application of consumer advice by the Board and the Tool (four decisions), a ‘broadly consistent’ decision was generated in 93% of instances (refer Table 2).KPM 2: Ability to refuse classification to relevant materialMeasure: Ensuring the Tool does not classify any material the Classification Board would Refuse Classification.During the evaluation phase, no content was classified by the Tool that would warrant being refused flix has control over the content it chooses to stream for its Australian audiences. Netflix has advised that content that would likely be refused classification is not part of their suite of programs. As such, the risk of content that would be refused classification appearing on Netflix in Australia is unlikely. KPM 3: Ability to display classifications (ratings and consumer advice) produced by the Tool on the Australian Netflix web browser interfaceMeasure: Ensuring Tool decisions are being published on the Australian Netflix interface in an accurate (agreed format) and timely fashion.Classifications (ratings and consumer advice) are being displayed on the Australian Netflix web browser interface. The Department worked with Netflix during the pilot to improve the display of consumer advice. The Branch is aware that the classification decision is not displayed on all interfaces, for example, mobile phones, and the Branch will continue to work with Netflix on this issue. KPM 4: Ability to provide classifications (ratings and consumer advice) generated by the Tool in agreed format for display on the National Classification DatabaseMeasure: Ensuring Tool decisions are being provided for publication on the National Classification Database in an accurate and timely flix tool classification decisions are being published on the National Classification Database accurately and within acceptable timeframes. KPM 5: Ability to update the Australian Netflix Interface with new classification decisions as a result of revocations made by the Classification Board in a timely mannerMeasure: Ensuring Netflix are updating their interface as a result of revocations in a responsive flix are able to update their interface if a classification decision needs to be changed. KPM 6: Satisfaction amongst Australian Netflix users (complaints management)Measure: Monitoring any consumer complaints received by the Department and/or Board and/or Netflix about Netflix content, audit results of content audited as a result of complaints or media attention.Whilst there was some media and public interest about classification decisions on particular titles, there have only been two consumer complaints received by the Department about the Tool’s decisions during the pilot. The Department will continue to monitor any consumer complaints.In accordance with obligations under the MOU, Netflix published the following email address on its customer support website: australia-classifications@. Netflix did not receive any complaints which warranted further action under Section 8.2 of the MOU. Netflix will continue to monitor any consumer complaints and advise the Department. KPM 7: Ability to adapt to changes in the National Classification SchemeMeasure: Ensuring any changes in the scheme that occur are mirrored in the Netflix Tool in a timely fashion; ensuring the Tool reflects current Australian community standards.Although there have not been any changes to the National Classification Scheme during the evaluation period, Netflix has demonstrated its ability to make amendments to the Tool in a timely manner, including adoption of an approved consumer advice list and accommodating changes to that list during the diagnostic phase.KPM 8: Netflix’s ability to make adjustments to the Tool to improve its performanceMeasure: Netflix’s responsiveness to feedback about performance of the ToolThroughout both the diagnostic and evaluation phases of the pilot, Netflix has responded promptly and effectively to feedback about the performance of the Tool.KPM 9: Tool must not produce a classification and consumer advice for films already classified by the Classification Board/Classification Review Board/approved tool/public exhibition films and/or upon advice from the Branch.Measure: Number of films classified by the Tool which had been previously classified.Of the 2473 classification decisions generated by the Tool across the whole pilot period, 51 classification decisions (2.1%) were for films that had previously been classified by the Classification Board. By the end of the pilot, only one percent of classification decisions were for films that had been classified previously by the Board. Netflix is continuing to improve internal processes to ensure titles that have been previously classified by the Classification Board are not classified again by the Tool. KPM 10: Ability to advise the Director/Department of all Tool decisionsMeasure: Ensuring Tool decisions are being provided to the Department via the API in an accurate and timely fashion.All Tool classification decisions are entered directly into the Department’s internal classification database COBRA via the Application Programming Interface (API). The classification decisions are then automatically uploaded to the National Classification Database, which is searchable by the public. Reports can be generated for all Tool classification decisions. This system has worked exceptionally well throughout the pilot and any issues found, either by the Department, the Board or by Netflix, have been resolved in a timely fashion by Netflix or the Department’s IT area.4.Conclusion and recommendationsBased on the pilot findings, the Department has reached the following conclusions:The Tool is able to produce ratings that are the same, or one rating level higher, than a rating made by the Board in 96% of instances.The Tool is able to generate classification decisions that are ‘broadly consistent’ in 93% of instances. The Department recognises an automated Tool and the Board apply consumer advice differently. This highlights the need for an overall review of the application of consumer advice by the Board and automated tools. The Department makes the following recommendations:The Department recommends that the Minister approve the Netflix Tool for ongoing use in Australia subject to ongoing monitoring. The Department recommends reviewing the application of consumer advice by automated tools and the Board. Appendix A: evaluation phase resultsOf 127 Tool classification decisions audited, 118 (93%) were found to be broadly consistent with the decisions of the Classification Board and Australian community standards.Film titleTool ratingBoard ratingBroadly consistentVoyeur MA 15+MA 15+YesBruno and Boots: This Can't Be Happening at Macdonald Hall PGPGYesJohn & Jane MMYesPrateeksha PGPGYesBigfoot GGYesCooked: Season 1 PGPGYesOriented MMYesAlien Mummies PGPGYesLady Dynamite: Season 2 MA 15+MA 15+YesFloating! MMYesKaptn Oskar MA 15+MA 15+YesUnbreakable Kimmy Schmidt: Season 3 MMYesKevin Hart Presents Lil Rel: Relevent MA 15+MA 15+YesZumbo's Just Desserts: Season 1 PGPGYesLove: Season 2 MA 15+MA 15+YesBarbra: The Music ... The Mem'ries ... The Magic!MMYesI Am the Ambassador: Season 1, 2 PGPGYesSonic Boom: Season 1PGPGYesBeyond Bollywood MMYesThe State of Marriage PGPGYesConcrete Football MMYesTodd Barry: Spicy Honey MMYesJoan Didion: The Center Will Not Hold MMYesFlaked: Season 2 MA 15+MA 15+YesThe Testimony MMYesJago: A Life Underwater PGPGYesThe Drowning MMYesDate with Love PGPGYes9 Seconds—Eternal Time: Season 1 PGPGYesAustralien Skies PGPGYesBig Bang MMYesSandook MMYesKrish Trish and Baltiboy: Best Friends Forever PGPGYesThe Day I Met El Chapo: Series 1 MMYesLoreena McKennitt: Nights From The AlhambraGGYesAlmost Holy MMYesMadre MA 15+MA 15+YesFate/Apocrypha: Part 1 MA 15+MA 15+YesStretch Armstrong & the Flex Fighters: Season 1 PGPGYesJe suis Charlie MMYesLouis C.K.: Live at the Comedy Store MA 15+MA 15+YesFalling Flowers MMYesWinning PGPGYesDiscovering Bigfoot MMYesThe Killer MA 15+MA 15+YesSteak (R)evolutionGGYesTesting the Menu: Season 1 PGPGYesBrain Games: Season 4 PGPGYesWilliams MMYesInvasion 1897 MA 15+MA 15+YesEk Jaan Hain Hum MMYesThe Lost Brother MA 15+MA 15+YesEl Camino ChristmasMMYesA Stand Up Guy MA 15+MA 15+YesMasaan MMYesHello, My Twenties!: Season 1 MMYesMaking the American Man MMYesFinding Traction PGPGYesThe Toys That Made Us: Season 1 MMYesOffice MA 15+MA 15+YesYes We Can!: Season 1 PGPGYesJudd Apatow: The Return MMYesBikes vs Cars MMYesEnter the Battlefield PGPGYesMost Valuable Players PGPGYesTrailer Park Boys: Season 11 MA 15+MA 15+YesWild Australia with Ray Mears: Season 1 GGYesContinuum: Season 4 MA 15+MA 15+YesThe Ranch: Part 3 MMYesKrish Trish and Baltiboy PGPGYesCouple of Days MMYesA Decent Arrangement MMYes30 Days in September MMYesCraig Ferguson: Tickle FightMA 15+MA 15+YesSense8: Season 2 MA 15+MA 15+YesWelcome Mr. President MMYesDeath of a Gentleman MMYesThe Overnighters MMYesJim Gaffigan: King Baby MMYesA Matter of Faith PGPGYesDarkweb MA 15+MA 15+YesBright MA 15+MA 15+YesWho the F**k Is That Guy? MMYesMy Happy Family MMYesOctober KissGGYesJedi Junior High GGYesLoktak Lairembee MMYesThe Long, Long Holiday: Season 1PGPGYesA Noble Intention MA 15+MA 15+YesLa Mante: Saison 1 MA 15+MA 15+YesLlama Llama: Season 1 GGYesA&E: When Patients Attack MA 15+MA 15+YesThe Naked Truth: Season 1 MA 15+MA 15+YesAfter School: Lucky or Not: Season 2 MMYesSecurity MA 15+MA 15+YesDeadly Paradise MMYesChapo: el escape del siglo MA 15+MA 15+YesTeach Us All MMYesRoad to Yesterday PGPGYesUnder the Sun PGPGYesInterrogation MA 15+MA 15+YesSix Feet High MMYesAnimism MMYesAbzurdah MA 15+MA 15+YesUnder an Arctic Sky MMYesConflict MMYesThe Alps Murders MMYesJudah Friedlander: America Is the Greatest Country in the United States MMYesVeronica MA 15+MA 15+YesThe Women Who Kill Lions MMYesHow to Win the US Presidency PGPGYesFloyd Norman: An Animated Life PGPGYesBright Night MA 15+MA 15+YesRaiders!: The Story of the Greatest Fan Film Ever Made MPGYesLa casa de papel: Part 1 R 18+MA 15+YesMughal-E-Azam MPGYesAmok MA 15+MYesAisa Yeh Jahaan MPGYesOnce Upon A HolidayGGNoEverybody Loves Somebody MMNoYZ PGPGNoHoliday Baggage PGPGNoAlias Grace MMA 15+NoA Holiday EngagementGPGNoDaughters of Mother India MMA 15+NoHappy Together MMA 15+NoMaya Memsaab MMA 15+No ................
................

In order to avoid copyright disputes, this page is only a partial summary.

Google Online Preview   Download