Though the leadership oftentimes agrees with the need for a monitoring system or a method for evaluating the efficacy of a project, program, or organization performance overall, in practice designing and implementing a new monitoring system has several challenges, such as:
• stakeholder’s buy-in,
• technology shortfalls,
• lack of experience,
• limited financial and staff resources,
• gaps in technical knowledge with regard to logical frameworks design and definition, performance indicators definition, the retrieval, collection, preparation and interpretation of data,
• inefficient monitoring and evaluation practices,
Question from: Jaya
Accountability is one of the main goals in M&E, both in government and nonprofit organizations. M&E for accountability commonly focuses on upward accountability to a government or funding agency. It is often an obligation to demonstrate that the contracted work has been conducted in compliance with the agreed standards or to report on results and plans. Downward accountability involves making accounts and plans transparent to the primary stakeholders, i.e. clients.
With M&E software platforms such as Indicata, it is possible to demonstrate whether the contracted work has been accomplished in accordance with the agreed plans or not. It is very important to have tools which will help report results vs plans and find gaps or inconsistency for further fixes and performance improvements.
Question from: Somar Rezk
The first and most important thing is to provide 'smart' data entry templates and train the staff for effective data collection procedures and rules. Digital data gathering with validation in place will help omit most of the errors. Next should come data cleaning exercises. Data Quality Review (DQR) and Data Quality Audit (DQA) tools allow to evaluate data quality across projects and/or programs. Synergy Indicata provides tools to review and measure the data quality in projects.
Question from: Moseed AL-Arasi, Nilaramba
Learning is the process through which the information generated from M&E is reflected upon and intentionally used to continuously improve a project’s ability to achieve results. Documenting targets, results, changes and challenges can serve as input from past learning into the new cycle of a project or program. There are several ways/tools that can help you visually represent your goals, outcomes, outputs, activities, related indicators and document the learning that can be saved as a knowledge base. Synergy Indicata provides all the necessary functionality for that.
Question from: Anita Ubah
It can actually do both, depending on what the organization wants to focus on, but in the end, the goal is to find the bottleneck and improve processes and performance.
Question from: Andy Chi Tembon
When operating in poor and remote locations during Covid-19, there may be the need to pause non-critical M&E for some projects while maintaining essential or time-sensitive functions. The importance of phones, internet access and personal protective equipment (if in-person data collection is necessary) is essential. Data collection shifts to smartphones and tablets.
Synergy Indicata encompasses a data collection platform that allows offline data collection, conducting surveys offline and synchronizing it with the server when there is the Internet.
Question from: Kombo Choga
Follow your passion, focus on where your interests are, for example having a degree in International Development or in Social Development of Communities would help as well.
These universities offer master's, PhD degrees and different certificates either directly in evaluation or with available concentrations in evaluation.
Question from: Ruth Amamo
Outcome evaluation can be done based on both its outputs and independently through outcome level indicators. Outcome harvesting has proven to be a useful approach in a variety of locales and for diverse interventions. It is used when the focus is more on outcomes rather than on outputs or activities, and is well-suited for evaluation in dynamic, uncertain situations, for example when the outcomes were not specific or measurable at the time of planning the intervention.
Question from: Anastasia Kushleyko
It is possible to use remote tools such as surveys/interviews and other data collection methods and where possible rely on indicators that can automatically gather data in real-time, e.g. IoT data collection nowadays is quite modern. Please watch our second webinar to learn more.
Question from: Suman Gupta
It is very important to understand the importance and need of an M&E culture in the organization. Thus, to start the M&E transformation the following actions should be performed:
1. Engage the management as their ownership in this initiative is crucial.
2. Prepare surface-level capabilities (skills and knowledge) within the organization, if there is no any, then hire SMEs (subject matter experts).
3. Be ready for changes in deep-level capabilities such as mindsets - people are resistant to changes, so management should find ways to have a smooth transition, by involving employees and gaining their ownership, or making it clear that M&E is not for rebuke but for a better way of working, for finding gaps, and for making programs as effective as possible.
4. Produce tangible 'artefacts' - such as a visual representation of a system to be demonstrated during monthly/quarterly all-staff meetings, or semiannual M&E reports/dashboards to serve as employees' homepage.
5. After having all these in place - it will need automation, thus technology is playing an important role for real-time data collection, ease of key findings, harmonisation and automation of business processes.
Question from: Liliana Carral, Jaya
Probably they see M&E for rebuke or for blaming accountable people, or they have never been shown the advantages of M&E. Please check the relevant question raised by Liliana Carral and respective answer.
Question from: Maazua
We would not say that other M&E practitioners are wrong. They would document the data, however, they would use only the data they need, so the other data will become an asset only when they realize what and why is needed to be analyzed. Data gathering and cleaning requires effort, so it is a matter of resources and cost - can we afford to gather and ensure the quality of data that we do not need but may need in the future? Also what if the collected data will become not actual anymore?
Question from: Aster
We recommend preparing a robust and solid M&E plan including a result framework and indicators that will allow measuring the success of the given result framework. You need to make sure that the result framework is focused on your organization's global strategic plan and this must be aligned with your management’s vision. The whole M&E plan must be feasible, implementable, in line with your corporate plans, and result-oriented. Then a very good presentation of this will help you become part of corporate plans.
Question from: Shorooq Shabillo
This is a very global question and we think that the starting point can be the active collaboration with the local government or communities. In these processes, M&E specialists and consultants need to play the key role, to convince and prove the real need of M&E science in spheres like education, health, child protection, etc. As soon as the governing bodies agree on the importance of M&E, you can initiate the next step, which is the design of an M&E Information System, and then translate all this knowledge into technology. All these should happen at a contract level, so accountability will be an obligation. At a contract level, the goals for implanting organization should be stated so the progress will be possible to track.
We are proud to highlight that through Synergy Indicata, complete monitoring and evaluation software, which streamlines complex M&E processes, we have provided solutions to several governments and international organizations, such Save the Children, UNICEF, Educate Above All, etc. For more information about our projects please check here.
Question from: Hanomy Hajeb
Based on the experience that we have with our clients, first of all, you need to adapt your Result Framework and Indicators to correspond to the current situation based on the COVID-19 pandemic. You might need to cut costs from non-priority objectives (during the period of the COVID-19 pandemic) and instead mobilize your resources on hiring local partners, acquiring tablets or mobile applications for data collection, etc.
In a data capturing process, you should involve a local partner to minimize the number of on-ground operations, then there might be a need for data validation as well (again by another local partner).
Developing/purchasing a mobile application for field visit surveys can be useful. Another option is online surveys with a layer of random data quality checks.
Question from: T. Senthoran
The target local people should be registered providing specific contact information. For data verification, random calls will be made to ensure that the data collector didn’t cheat and to avoid any planned responders.
Question from: Eniola Saheed Adedayo
As representatives from Synergy, we would like to specify Synergy Indicata, which is an M&E software system for storing both quantitative and qualitative data. One of its advantages is its sophisticated and powerful reporting tools (analytics). Indicata’s robust reporting provides users more power and flexibility in the creation of ad-hoc reports, and their export to Word, Excel and PDF. Reports by any combination of groups, columns, rows and quantitative data fields can be created by non-technical staff on-the-fly in a highly user-friendly and logical manner with the Report Builder.
Indicata provides Data Entry form builder where any type of quantitative fields can be stored for further surveys/data collection.
Question from: Mara Tissera Luna (Argentina, Hungary)
The nature of most emergency responses demands a more relaxed and sympathetic M&E approach. Immediate response comes first, and any M&E / data collection is often secondary. That said, emergency response activities have to be planned, executed, and monitored – the extent of which is guided by parameters of the response.
During emergency, only primary needs should be collected, or only basic and most important indicators should be considered (e.g., number of people injured, or number of people saved, etc.).
Also, data collectors should come up with rapid methods for data collection, for example, if during earthquake people should be registered then it might happen by taking person’s fingerprint - just for identification, but then in post factum detailed level information can be collected and monitored.
Question from: Ahmad Ramzi
You need to tailor your program strategy plans, result framework, and indicators so that they will reflect the current situations and will be feasible and practice for this period of time, e.g. define new simple (minimum level of disaggregate, no need for baseline, etc.) indicators to measure the results. Afterwards via composite indicators, you can combine data of these simple COVID-related indicators with the regular ones. Please also check the answer of T. Senthoran's question above.
Question from: Samantha Woodcroft
You need to design corporate-level result framework (e.g. including results/objectives relevant for partners’ performance as well). Then you should define your key performance indicators not only at an output level but also at an impact level. For example, to measure partners’ performance the following simple KPIs can be defined and attached to the corresponding result level:
- Revenue through partners
- Number of partners
- Number of deal registrations
- Number of people trained by an implementing partner
Based on the sector, you may have other indicators, e.g. for education - “How many out of school children have been reached out”, or for health - “How many children have been vaccinated”.
Please check the answer to the relevant question raised by Liliana Carral and Jaya.
Question from: Saba Khan
We already have experience with implementing AI solutions within M&E systems to give our users smart predictions.
As a very simple example: if the system noticed the correlation between two indicators: “Car accidents mortality rate” and “Number of traffic violations”, then AI can predict increasing the ticket cost for traffic violations as an effective solution to the problem, so by establishing a liaison between indicators and analyzing them AI can identify and close the gaps.
Question from: Dexter Hati
As a relevant example for you, we will mention the system developed through Synergy Indicata “The European Union Facility for Refugees in Turkey”.
The Facility Monitoring Platform (EU FMP) is an automated online monitoring and evaluation system which was gathering mainly data for Syrian refugees. The main objective of the system is to support the Government of Turkey in formulating a comprehensive strategy, effectively managing implementation, and promoting the accountable and transparent use of resources. This kind of systems can dramatically change the precise identification of beneficiaries and their further progress. The first step should be the proper identification (via ID, fingerprint, etc.) of beneficiaries and then their involvement in the right program/project. Pre- and post-evaluation should be there for assessing results accurately. These will help you measure the impact of the programs and/or projects.
Question from: Deniz Menteşeoğlu
This was addressed during the webinar by Emilya Khachatryan – please check the recording:
Cheap mobile devices and reliable internet services are enabling organizations to collect larger volumes of more reliable data.
Nowadays most of big data is collected from IoT devices such as sensors, cameras, smart parking solution, GPS tracking devices that are embedded in the water and power infrastructure, for example.
Data of “Electricity consumption rate” or “Basic Water Supply” indicators can easily be conducted through these devices, which will bring to big data. M&E is nicer and more accurate whenever data is big, i.e. better reports and analysis can be conducted, but in this case, M&E software becomes a must.
Potential problems include:
- M&E practitioners are inexperienced with large datasets and complex data models.
- Data cleanup and aggregation can be difficult.
Question from: Anush Shakhsuvaryan
Both, of course, are important, and we can’t really say which one more as they are closely intertwined. Accountability is necessary but having M&E on top of it you will have both necessity and sufficiency. M&E can be a useful tool during project execution if the parameters allow it to be, e.g., frequency of data collection. M&E can ensure better accountability both towards donors, as well as clients/beneficiaries.
Question from: Charles Downs
Please check the answer to the question by Mara Tissera Luna posted above.
Question from: Gulnara Kokonbaeva
Face to face meetings are probably the best way to collect feedback from beneficiaries as they can also facilitate identification, validation and deduplication. That said, face to face meetings are not always possible, due to the lack of funding, limited access to rural communities, conflict or a pandemic. In case of sufficient access to the Internet, beneficiary feedback can be collected online, however as we know this is mostly not possible, and in most cases the only alternative is using phone calls, SMS or IVR, or by using local focal points.
Recommended read: Data Collection in Times of Physical Distancing
Question from: Mubarak Nour
First of all, a good M&E system should be feasible and implementable. There should be SMEs who will be a dedicated staff in M&E-related activities. These team should define and design M&E work plan with corresponding budget, M&E frameworks/Logical Framework, smart KPIs, etc.
Recommended read: 12 Key Components of M&E System
Question from: bbanguna
COVID-19 has had a negative impact on data collection and might have emphasized certain biases.
Check your bias, gender data experts warn
COVID-19: Guidance for labour statistics data collection
Question from: Dalita Artin Avanesian
The primary function of the Project Management Office is to define and maintain standards for project management within the organization. There are several types of PMOs - Supportive, Controlling and Directive. Centralized and corporate M&E system approach is dependent on the type of the organization and it should be aligned with the management and PMO strategies and objectives.
If the question is about a project-level M&E, monitoring and control (please check PMBOK 6 edition) is one of the main phases (out of 5 phases - initiation, planning, execution, monitoring and control, closing) of project life cycle which should happen throughout the entire length of a project, and it should be conducted according to the standards set by PMO.
Question from: Hamza Abdalla
Having or not having a centralized M&E unit depends on the structure of the organization and/or sector, its objectives, as well as on the M&E practice and culture.
While there are benefits to centralizing M&E, not every organization is able to do so. Reasons may lie in the non-compatibility of programs, which can require various, sometimes incompatible M&E approaches, frameworks, subject matter expertise, etc.
Question from: Hamza Abdalla
Project knowledge management – cost, schedule, scope, resources, stakeholder, and other knowledge management areas are fulfilled in 5 phases - initiation, planning, execution, monitoring and control, closing correspondingly. M&E or Monitoring and Control (check PMBOK 6th edition) is an important part of any project and it should be implemented throughout the entire duration of a project.
Question from: Fikru Habte
First of all, you need to identify stakeholders and other things that are creating those challenges. Then, you will need to gain the ownership and interest of those identified stakeholders, trying to find some incentive and motivation for each. You will need to demonstrate to them that good M&E is not for finding guilty people or for rebuke. You should make clear to everybody that good M&E is for a better way of working, for finding gaps, for making programs as effective as possible.
Please also check the answer to the relevant question raised by Liliana Carral and Jaya.
Question from: Elisa Riquelme
Our recommendation would be to establish and have a separate independent department.
Question from: Lydia Mukaye
Please check the answer to the question raised by Liliana Carral and Jaya.
Question from: Ahmad Ullah Qazi
For humanitarian assistance in an emergency, one of the most important activities is the identification of victims, then further tracking of corresponding beneficiaries through management and information systems. During emergency people can’t provide an ID card (passport, UNHCR passport), thus rapid biometric identification (fingering, iris scan, etc.) should be used. Further, if the beneficiary gives consent, details should be entered in a management system for detailed monitoring. Note, personal data is sensitive, thus all governmental, GDPR rules should be considered during this activity, i.e. in case beneficiary is requesting to delete their personal data from the system, that should be done immediately. Another method is data anonymization/encryption that should be carried out in Management and Information Systems.
Question from: Salauddin Ahmed
This may vary. The trending skills now may include: data collection automation (through IoT and smart devices), calculation of results (Impact, Outcome, Output, etc.) and aggregation automation through a management and information system (M&E systems), AI integration for projections.
Question from: Mohammedhal