PERFORMANCE EVALUATION

While Section 8 (Operation) dealt primarily with the parts and services we provide in the Aerospace community, in this section the standard is back to dealing with the SYSTEM, and so when Monitoring and Measurement is discussed, it is not monitoring and measurement of the product, but rather of the system. It is important to note this before continuing the evaluation of the standard, and it is key to realizing that this section applies to the performance of the system, rather than a product or service.

Monitoring and Measurement follows the same PDCA-type cycle as many of the other systems do within the standard. You first identify what has to be measured (Section 9.1), and then check on its performance through tools such as Internal Auditing (Section 9.2) and Management Review (Section 9.3).

For many years in the Aerospace and Defense industry,the person largely held responsible for performance evaluation of the QMS, and the Management Review,was the designated Management Representative. While ISO 9001:2015 removed the Management Representative, AS9100 D left it in. The reasoning for removal from the ISO standard was the fact that much of those responsibilities were not perceived by organizations as top management’s responsibility when there was someone “representing” them. AS9100 left it in because in the aerospace world this mindset never really took hold, management always owned the responsibility.

What is it?

• The organization will decide what needs to be measured, how it should be measured, and how often it should be reported
• Documented information on performance is required, after all, you have to prove you did it
• Customer perception needs to be evaluated as part of the organization’s scheme of measurement, but it is up to the organization to determine how it will collect and review the information (Notice the freedom created by the phrase “retain appropriate documented information.” Many companies have failed by gathering data that they never used, only keep what is useful.)
• The organization uses the documented information to evaluate its QMS in relation to:
  • Conformity to the expectations for its products and services
  • Satisfaction of the customer
  • Performance of its QMS in relationship to its own needs
  • The effectiveness in relationship to risks and opportunities identified in Planning(Section 6.1)
  • Performance evaluation
  • External provider performance
  • Need for improvement and gaps in performance objectives

It is worth noting that the AS9100 standard calls out some very specific monitoring targets:

• On-time delivery
• Customer satisfaction
• Customer complaints
• Corrective Action requests

These items must also be backed by an action plan IF there are gaps.

Objective

• Measure where you are so you know how far you have to go, and do not waste resources if you are already there

How do I do it?

1. Determine what needs to be measured and confirm that the measurement is consistent with:

• The Quality Objectives set
• The product-specific requirements
• The process-specific requirements
• Any other Key Performance Indicators (KPIs) for interested parties identified earlier in Section 4

2. Determine what, to whom, when, and where you will report the findings. This should align with the ownership for the processes and the roles and responsibilities identified earlier in Section 5 (Leadership)
3. Add the relevant KPIs into Management Review and set the time to review at appropriate intervals based on the methods chosen above
4. Ensure that there is an alignment between the customer’s perception and your own evaluation. In the case of any discrepancy,prepare to explain the difference
5. Monitor and analyze the results and IF there are gaps, generate action plans to close them

Tools and techniques to achieve it

• Customer Scorecards
• Balanced Scorecards
• Customer surveys
• Operational plans generated as part of annual planning, in the case of gaps

Documents you can use to prove it

• Management Review meeting minutes
• KPI dashboards
• Processes for customer perception solicitation
• Sales surveys and evaluations
• Corrective Action logs linking the activities done to the monitoring of those issues

Questions to ensure you know it

• How closely does our perception of our performance match with the customer’s perception of how we are performing?
• How often are we polling our customer base for an opinion of our performance?
• Does our Management Review contain all of the KPIs we have identified in our process descriptions and Quality Objectives set for our organization?
• How valuable is the data we are mining regarding process performance, and are our KPIs still relevant to our level of maturity?
• Can I walk someone through the steps we are taking and then back up the plan to explain to them how we are getting back to good if we are not meeting target?

How can you fail at it?

• You haven’t included all the key indicators for the standard, such as on-time delivery and Corrective Action requests
• Your KPIs do not match customer perceptions, and management doesn’t ask for an explanation
• You don’t solicit customer feedback because your principle is “no news is good news”

An internal audit is driven by top management’s needs and expectations first, then it is driven by the conformity with applicable requirements (AS9100, customer, legal, and regulatory). Its results area key input in Management Review (Section 9.3) to determine improvement actions and needed resources.

While there has been a lot of information published around the subject of internal auditing (both positive and negative), it is an important part of making certain what your organization assumes is going on is in fact happening. This discussion is outside the scope of this book, but ISO 19011 can provide additional information. (ISO 19011 sets forth guidelines for management systems auditing.) However, it is well known that there are industry specific “Norms” on how audits are conducted, and so you would be wise to consult your organizations own processes to see what they use. In addition, customer specific requirements may add other auditing requirements,such as product audits, fresh eyes audits, layered process audits, and special process audits.

What is it?

• Internal audits must be performed at planned intervals, however, that interval is not defined
• The internal audit shall check the organization against the international standard requirements and its own requirements to ensure effective implementation
• The audit program at a facility must be planned, established, implemented, and maintained
• Audits are scheduled and prioritized based on:
  • Importance of the processes
  • Identification of risk
  • Changes to the organization
  • Results of previous audits
• Audit scope and criteria must be defined for each audit. The audit is then carried out by impartial auditors who report to management to initiate Corrective Action
• Retain documented information on the audit and on actions taken

Objective

• Check your own system to know how you are performing so that someone does not have to check it for you…because they will

How do I do it?

1. Using ISO 19011, familiarize your organization with the “guidelines for auditing”
2. Identify all of your processes on your Process and Interaction map
3. Highlight those processes that are extremely important and audit those more often(this ties into the “importance” of process discussion)
4. Audit each process at least once over an appropriate period of time for your organization. Select auditors and ensure they are not auditing their own area of work
5. On completion of the audit conduct a formal close-out meeting with the management team
6. Add the audit results into Management Review along with completion of the audit
7. Follow up Corrective Actions to closure and adjust the audit frequency for the next cycle using the information gained from this cycle

Tools and techniques to achieve it

• Process approach style auditing using a variety of tools, but they all revolve around process
• Turtle Diagrams
• SIPOC Diagrams
• Checklists - if used appropriately

Documents you can use to prove it

• Internal audit schedule
• Internal audit reports
• Management Review meeting minutes
• Corrective Action reports
• Auditor competence records

Questions to ensure you know it

• How do we know which auditors can audit which processes and ensure their independence?
• How do we assign criticality to make sure some processes that are key get more attention than others?
• How do we ensure each process is covered for each cycle?
• How are audit findings and reports communicated and closed by management?

How can you fail at it?

• You don’t audit the entire system during a cycle
• You don’t highlight high-risk processes for added surveillance
• You use auditors that have authority over a process they are auditing
• You don’t place the Corrective Actions into Management Review
• You don’t generate findings of significance

Management Review is the “information superhighway”of the QMS. The QMS, with its inputs and outputs, is a repository for information to come in and information and resources to flow out. Each process within the QMS should have its Key Performance Indicators (KPIs) reviewed throughout the cycle as part of Management Review.

This is the venue for those items deemed important for an organization to get the attention required to remove roadblocks and move them forward. This is also the place where the data and the graphs come together to tell the story of what is going on in the organization, and a good auditor will combine a variety of information and paint a picture of the organization.

What is it?

Management Review has to be at planned intervals, the timing of which is left to the discretion of the organization (the standard does not specify exactly how long the interval should be).

Management Review has a defined set of inputs, including:

• Status of actions from previous reviews
• Changes in external and internal issues
• Information on the performance of the QMS. This includes trends in:
  • Customer satisfaction and interested parties
  • How completely objectives are being met
  • Process and product conformity, including measurement result trends
  • Nonconformity and Corrective Actions
  • Audit results
  • On-time delivery
• Adequacy of resources
• Assessment of progress in affecting risk and opportunities
• Opportunities for improvement

Management Review also has required outputs,including:

• Opportunities for improvement
• Needs for changes in the QMS
• Resource needs
• Risks identified (Unique to AS9100)
• Documented information showing results of Management Review meetings

Objective

• Get the people with the resources in front of the data so that they can make a decision and allocate those resources

How do I do it?

1. Gather the required inputs listed above
2. Identify which processes provide these inputs
3. Review what documented information and KPIs have already been established for these items
4. Set a schedule for the Management Review meetings
5. Review the documented information and KPIs with the intention of generating the outputs required by the standard
6. Rank the inputs and decide on priorities for action
7. Establish actions into the system and follow a PDCA process

Tools and techniques to achieve it

• Brainstorming
• Critical thinking
• Priority analysis
• SIPOC or Turtle methodology looking at Management Review as a process, see example in figure 9.2

Documents you can use to prove it

• Management Review meeting minutes

Questions to ensure you know it

• Can we align all of the inputs with existing documented information?
• How often do we meet to go over the required information?
• Can we combine this meeting with another meeting to ensure we get the people in the room that we need to generate activity and interest?
• What are the last five actions that came out of Management Review?
• Are the actions being driven by the data or an agenda?
• Are the actions value added and driving us toward a competitive advantage?

How can you fail at it?

• You don’t fulfill all the inputs and outputs
• Top management is not engaged
• The team does not understand what the data says
• The action items do not match the data and are non-value adds

Wrapping up Section 9 of the Standard

As you consider this section, understand that the standard is not looking for reactive performance evaluation and reactive review by management. Too many organizations in the past have only taken reactive positions even regarding these important reviews. The idea is to become proactive and encourage the use of concepts such as using leading measures to inform lagging measures. In addition,Management Reviews should include information on the performance and effectiveness of the Quality Management System, including trends in several areas. Waiting to take action after a trend has been going in the wrong direction for a year is not what the standard intended.