Previous chapters explained the subsequent steps from planning to implementation. This chapter discusses the CHECK stage. It explains why common indicators of performance (KPIs) as well as criteria jointly decided upon, must be used for the evaluation of the impacts and results of actions. As previously discussed, monitoring helps in the first place to steer the project or action plan during its implementation and adjust the actions if needed. What is more, the use of common indicators also helps to make better comparisons of the performance of specific actions or solutions in different cities. Best practices and “success stories” of satisfactory and effective actions and solutions, are the benchmark for cities that contemplate these actions or solutions for their own jurisdictions, while “failure stories” teach what should be done in a different way. In this way, comparable indicators help to build a repository of validated, proven solutions, which supports cities in selecting the most appropriate types of solutions that meet the city’s expectations in the best way.
This chapter will focus on:
- The processes of setting up baselines that adequately reflect the current situation;
- Collecting information on KPIs chosen earlier;
- Interpretation of monitoring information;
- The role of monitoring information for improvement and benchmarks in support of decision making (including investments) and selection of appropriate targets.
No improvement without measurements
When executing programmes, projects and actions plans, regular systematic collection and analysis of information are essential to track the progress in implementation against pre-set targets and objectives. It aims to answer the question “Did we deliver?”
Monitoring provides information on where an action plan, project or programme is at any given moment, or how it develops over time, relative to chosen targets and expected outcomes. Besides, monitoring sheds light on the efficiency of the planned or executed actions, and the use of resources this requires from the city administration and the stakeholders. For a city administration, setting relevant targets when the project is planned can be a real challenge that would benefit from a benchmark database of best practices. When actions are actually being implemented in a local context, there is a need to follow the progress of the actions by comparing how the main related KPIs evolve against the targets, and to act by (re-)adjusting the action plans to meet the targets, or, in case the chosen targets prove to be unrealistic, by adjusting the targets themselves. Thus, monitoring of the value of the concerned KPIs is an essential step for further improvement. It is impossible to manage and improve what is not measured. Besides, information coming from monitoring is vital for the reporting and communication of progress with different sectors of the city administration and with the stakeholders involved in the action plan or project.
While monitoring provides records of activities and results, and signals problems to be remedied along the way, it is descriptive and, as such, may not be able to explain why a particular problem has arisen, or why a particular outcome was achieved or not. Not monitoring, but evaluation deals with questions of cause and effect, although monitoring data might be used for answering these questions and prepare the evaluation. Evaluation is assessing or estimating the value, worth or impact of an activity or intervention, and is typically done on a periodic basis.
Monitoring is an essential part of the management of complex projects, action plans or programmes, especially when there are permanent improvement strategies in place. However, there is a critical need for defining what specific indexes best describe the achievement, targets, impacts, and so forth. A common challenge regarding measuring and monitoring of KPIs is that KPIs depend on data availability. The task of monitoring KPI achievements is often given to a public sector practitioner or a consultant. If some of the data are more difficult to acquire than others, the monitoring might become incomplete and some parts of the VISION would thus be neglected
To avoid this problem, it is recommended to ensure that all possible end users of the KPI measuring and monitoring take part in the final selection and design of the KPI set. Another way to tackle this challenge when it occurs during implementation, could be to partner with a research institution or a company that can assist in the collection of data, or that can assist in re-designing the indicators to identify more feasible ways to measure the same goal. It is important to have the availability of data in mind when selecting and evaluating the indicator set for each project, action plan or programme.
Benchmarking entails collecting sets of KPI values for a wider group of projects and programmes. This is very useful for comparing the impacts of different options before implementation, as well as promoting success stories and best practices through benchmarks after implementation, a key element for replication and scaling-up. Furthermore, the use of a large benchmarking database helps in selecting appropriate KPIs for setting relevant targets and exploring applicable solutions for new projects and action plans. And lastly, evaluation of the performance of chosen solutions afterwards and entry of this information into a benchmarking database, helps to save and disclose this information for refined use in the future.
One of the main barriers to overcome in smart and sustainable strategies and programs for a city or community, is related to the unfamiliarity with novel approaches and to the perception of risks, when it is contemplated to use innovative strategies and policies instead of business as usual. Thus, in order to support decision making as well as better motivate financial investments, benchmarks of success stories and best practices based on relevant and standardized KPIs, are not only particularly helpful in demonstrating potential benefits and assets for cities and communities, but also in indicating the volume of accompanying investments. Identifying similar challenges and validated potential solutions among a long list of success stories, will (re-)assure any decision maker and investor. This replaces the perception of risk with one of confidence and trust, thus contributing to engagement in ambitious but realistic action plans and projects. Such a benchmark of best practices will not only guide advisors and consultants, but also raise levels of expertise among cities’ decision makers and project leaders, and among engaged stakeholders.
In addition, accessing and analysing benchmark information can contribute to developing ambitious yet realistic expectations of foreseen targets and ways to meet them, through its reports on existing, inspirational cases. By choosing success stories that are on the leading edge of smart, sustainable and energy-efficient urban development, such an analysis can identify best practices that help to improve their own specific situation and context.
Collecting KPIs from projects/programs implemented in each city or community in a systematic and consistent manner will make it possible to feed progressively a common database that will serve as a benchmark, where any type of city/community should be “represented”. With such a major tool, decisions made at city or community level to engage in smart and sustainable development, will be easier, while support for the implementation in terms of finance and stakeholder engagement, will be more consistent and collective.
However, an important prerequisite for such use of KPI-based information is that best practices shall be collected and benchmarked using comparable indicators. If this is not done, the comparison and evaluation of which solution would best meet a city’s expectations, will not be possible or relevant. For this reason, common indicators of performances (KPIs), as well as already decided, mandatory criteria, should be used for the evaluation of the impacts and results of actions.
Criteria are different from indicators in that they are a set of key requirements that are mandatory to fulfil for the said project. These requirements are either fulfilled or not, and the further planning and implementation process depends on this. Next to KPIs, these criteria are also very important for knowing whether a project will be able to meet or has met its intended purpose(s). The BREEAM communities and the ZenN questions in the “Taxonomy of Near-zero Energy Renovation Options and their Influence on Architectural and Cultural Heritage” provide examples of such criteria (BREEAM, 2017; ZenN, 2017). Each project can use these sources to select criteria that might be relevant for their project and the consensus among stakeholders. It must be noted that deciding upon which criteria are relevant, is highly dependent on the national context, as many of these criteria are often integrated into national planning regulations. These criteria can be found by looking at the planning and building acts of each country. One example is the Norwegian criteria that public transport hubs must be within a specific distance from dwellings, and that the positioning of private houses must be so that there is sunlight in the building until at least 3 p.m. in the summer months. In addition, it is possible in many countries to add criteria to the legal agreement between property owners and private developers, which, depending on the national rules on legal agreements on properties, come in addition to these national regulations. KPIs will add decision support service, or ‘climate service’ tools on top of these criteria.
Standards can support monitoring and benchmarking significantly. Using and referring to standardised KPIs helps in bridging “different languages” and in creating a common understanding between stakeholders, especially between authorities and technicians. In this way, it contributes to an efficient management of the measurement of the progress of the project towards planned targets, and thus to the selection of appropriate corrective actions/adjustments to be implemented during the ACT stage (see next chapter). ISO 3712x series of standards are fully consistent with and complementary to ISO 37101 Quality Management Systems Approach. What is more, they contribute directly to quantifying progress for UN SDGs purposes.
The “United for Smart Sustainable Cities” (U4SSC) tool is another useful KPIs related initiative, a UN one, coordinated by ITU, UNECE and UN-Habitat, to achieve Sustainable Development Goal 11: "Make cities and human settlements inclusive, safe, resilient and sustainable". The U4SSC list of KPIs is consistent with the ISO 3712X series. Recently, the Global Standardisation Development Organisations (SDOs) –ISO, IEC and ITU-T – started to join their forces in a single Task Force - Joint Smart Cities Task Force (JSCTF) -, to work together on a common complementary set of standards