FAQ about Monitoring and Evaluation

Self-Service Analytica
Frequently Asked Questions
about Monitoring and Evaluation

Explore answers to the most commonly asked questions about monitoring and evaluation

Your questions answered

We have put together some of the common questions we received during the inaugural Synergy M&E Webinar Series. The webinars looked at key challenges to creating a culture of monitoring and evaluation in organizations as well as the trends and important innovations at the intersection of M&E and information technology. We hope these answers will give you more information about monitoring and evaluation in government and nonprofit organizations.

We continuously work on creating new content for the section, so please check back regularly for the latest updates.

Webinar 1: Current State and Key Challenges

What challenges do you face while working with organizations that are new to M&E?

Though the leadership oftentimes agrees with the need for a monitoring system or a method for evaluating the efficacy of a project, program, or organization performance overall, in practice designing and implementing a new monitoring system has several challenges, such as:
• stakeholder’s buy-in,
• technology shortfalls,
• lack of experience,
• limited financial and staff resources,
• gaps in technical knowledge with regard to logical frameworks design and definition, performance indicators definition, the retrieval, collection, preparation and interpretation of data,
• inefficient monitoring and evaluation practices,
• timelines.

Question from: Jaya

Accountability is the main objective of M&E in nonprofit organizations. How can we link M&E and accountability? Is there any process or procedure?

Accountability is one of the main goals in M&E, both in government and nonprofit organizations. M&E for accountability commonly focuses on upward accountability to a government or funding agency. It is often an obligation to demonstrate that the contracted work has been conducted in compliance with the agreed standards or to report on results and plans. Downward accountability involves making accounts and plans transparent to the primary stakeholders, i.e. clients.
With M&E software platforms such as Indicata, it is possible to demonstrate whether the contracted work has been accomplished in accordance with the agreed plans or not. It is very important to have tools which will help report results vs plans and find gaps or inconsistency for further fixes and performance improvements.

Question from: Somar Rezk

What methods should be used to ensure data quality in a project?

The first and most important thing is to provide 'smart' data entry templates and train the staff for effective data collection procedures and rules. Digital data gathering with validation in place will help omit most of the errors. Next should come data cleaning exercises. Data Quality Review (DQR) and Data Quality Audit (DQA) tools allow to evaluate data quality across projects and/or programs. Synergy Indicata provides tools to review and measure the data quality in projects.

Question from: Moseed AL-Arasi, Nilaramba

What tools can be used to drive and document learning in an organization?

Learning is the process through which the information generated from M&E is reflected upon and intentionally used to continuously improve a project’s ability to achieve results. Documenting targets, results, changes and challenges can serve as input from past learning into the new cycle of a project or program. There are several ways/tools that can help you visually represent your goals, outcomes, outputs, activities, related indicators and document the learning that can be saved as a knowledge base. Synergy Indicata provides all the necessary functionality for that.

Question from: Anita Ubah

I think it is important for people to understand that M&E is used to identify and mitigate issues that may impact a project and not to evaluate the performance of individuals in a given project/organization.

It can actually do both, depending on what the organization wants to focus on, but in the end, the goal is to find the bottleneck and improve processes and performance.

Question from: Andy Chi Tembon

The current COVID-19 pandemic's impact on business continuity has underscored the dire need to bridge the digital divide, especially for organizations working with resource-poor and remotely located communities. What experiences and guidance do you have regarding cost-effective technologies and methods for on-going monitoring and reporting of programming?

When operating in poor and remote locations during Covid-19, there may be the need to pause non-critical M&E for some projects while maintaining essential or time-sensitive functions. The importance of phones, internet access and personal protective equipment (if in-person data collection is necessary) is essential. Data collection shifts to smartphones and tablets.

Synergy Indicata encompasses a data collection platform that allows offline data collection, conducting surveys offline and synchronizing it with the server when there is the Internet.

Question from: Kombo Choga

What master's degree would you recommend to someone who has been practicing M&E for a while and is very passionate about it?

Follow your passion, focus on where your interests are, for example having a degree in International Development or in Social Development of Communities would help as well.
These universities offer master's, PhD degrees and different certificates either directly in evaluation or with available concentrations in evaluation.

Question from: Ruth Amamo

Do you have any experience in practicing outcome harvesting in the non-governmental sector? Would you recommend this method?

Outcome evaluation can be done based on both its outputs and independently through outcome level indicators. Outcome harvesting has proven to be a useful approach in a variety of locales and for diverse interventions. It is used when the focus is more on outcomes rather than on outputs or activities, and is well-suited for evaluation in dynamic, uncertain situations, for example when the outcomes were not specific or measurable at the time of planning the intervention.

Question from: Anastasia Kushleyko

As global mobility is limited due to COVID-19, how do you conduct distance monitoring and how do you verify the information/data?

It is possible to use remote tools such as surveys/interviews and other data collection methods and where possible rely on indicators that can automatically gather data in real-time, e.g. IoT data collection nowadays is quite modern. Please watch our second webinar to learn more.

Question from: Suman Gupta

How can we establish an M&E culture in organizations?

It is very important to understand the importance and need of an M&E culture in the organization. Thus, to start the M&E transformation the following actions should be performed:
1. Engage the management as their ownership in this initiative is crucial.
2. Prepare surface-level capabilities (skills and knowledge) within the organization, if there is no any, then hire SMEs (subject matter experts).
3. Be ready for changes in deep-level capabilities such as mindsets - people are resistant to changes, so management should find ways to have a smooth transition, by involving employees and gaining their ownership, or making it clear that M&E is not for rebuke but for a better way of working, for finding gaps, and for making programs as effective as possible.
4. Produce tangible 'artefacts' - such as a visual representation of a system to be demonstrated during monthly/quarterly all-staff meetings, or semiannual M&E reports/dashboards to serve as employees' homepage.
5. After having all these in place - it will need automation, thus technology is playing an important role for real-time data collection, ease of key findings, harmonisation and automation of business processes.

Question from: Liliana Carral, Jaya

Why do some organizations see M&E as a donor requirement but not as integral part of the implementation?

Probably they see M&E for rebuke or for blaming accountable people, or they have never been shown the advantages of M&E. Please check the relevant question raised by Liliana Carral and respective answer.

Question from: Maazua

Some M&E practitioners say that it is important to collect much data to use in the future, the panelists say we only have to collect the data we need. Are the M&E practitioners mentioned above wrong?

We would not say that other M&E practitioners are wrong. They would document the data, however, they would use only the data they need, so the other data will become an asset only when they realize what and why is needed to be analyzed. Data gathering and cleaning requires effort, so it is a matter of resources and cost - can we afford to gather and ensure the quality of data that we do not need but may need in the future? Also what if the collected data will become not actual anymore?

Question from: Aster

How can you convince the senior management to involve you in the planning level as an M&E officer?

We recommend preparing a robust and solid M&E plan including a result framework and indicators that will allow measuring the success of the given result framework. You need to make sure that the result framework is focused on your organization's global strategic plan and this must be aligned with your management’s vision. The whole M&E plan must be feasible, implementable, in line with your corporate plans, and result-oriented. Then a very good presentation of this will help you become part of corporate plans.

Question from: Shorooq Shabillo

How can we help the local government and community conduct M&E for humanitarian and development organizations using an information system? How to make organizations accountable for the communities they work for? Are there other successful stories like charity.org?

This is a very global question and we think that the starting point can be the active collaboration with the local government or communities. In these processes, M&E specialists and consultants need to play the key role, to convince and prove the real need of M&E science in spheres like education, health, child protection, etc. As soon as the governing bodies agree on the importance of M&E, you can initiate the next step, which is the design of an M&E Information System, and then translate all this knowledge into technology. All these should happen at a contract level, so accountability will be an obligation. At a contract level, the goals for implanting organization should be stated so the progress will be possible to track.

We are proud to highlight that through Synergy Indicata, complete monitoring and evaluation software, which streamlines complex M&E processes, we have provided solutions to several governments and international organizations, such Save the Children, UNICEF, Educate Above All, etc. For more information about our projects please check here.

Question from: Hanomy Hajeb

Because of the present COVID-19 pandemic we can't go to the field and do the baseline survey we have planned. What can be the alternative strategies to do the baseline with minimum field visits or without field visits at all?

Based on the experience that we have with our clients, first of all, you need to adapt your Result Framework and Indicators to correspond to the current situation based on the COVID-19 pandemic. You might need to cut costs from non-priority objectives (during the period of the COVID-19 pandemic) and instead mobilize your resources on hiring local partners, acquiring tablets or mobile applications for data collection, etc.
In a data capturing process, you should involve a local partner to minimize the number of on-ground operations, then there might be a need for data validation as well (again by another local partner).
Developing/purchasing a mobile application for field visit surveys can be useful. Another option is online surveys with a layer of random data quality checks.

Question from: T. Senthoran

How do M&E experts ensure that local people get involved in data collection?

The target local people should be registered providing specific contact information. For data verification, random calls will be made to ensure that the data collector didn’t cheat and to avoid any planned responders.

Question from: Eniola Saheed Adedayo

I'm a socio-cultural anthropologist who deals mainly with collecting and analyzing qualitative data with social research methods. It would be really helpful if the speakers could speak more about their favorite software for the analysis of quantitative data.

As representatives from Synergy, we would like to specify Synergy Indicata, which is an M&E software system for storing both quantitative and qualitative data. One of its advantages is its sophisticated and powerful reporting tools (analytics). Indicata’s robust reporting provides users more power and flexibility in the creation of ad-hoc reports, and their export to Word, Excel and PDF. Reports by any combination of groups, columns, rows and quantitative data fields can be created by non-technical staff on-the-fly in a highly user-friendly and logical manner with the Report Builder.

Indicata provides Data Entry form builder where any type of quantitative fields can be stored for further surveys/data collection.

Question from: Mara Tissera Luna (Argentina, Hungary)

How can I address data collection during an emergency? While my program did not necessarily target a relief situation, an emergency came up.

The nature of most emergency responses demands a more relaxed and sympathetic M&E approach. Immediate response comes first, and any M&E / data collection is often secondary. That said, emergency response activities have to be planned, executed, and monitored – the extent of which is guided by parameters of the response.

During emergency, only primary needs should be collected, or only basic and most important indicators should be considered (e.g., number of people injured, or number of people saved, etc.).

Also, data collectors should come up with rapid methods for data collection, for example, if during earthquake people should be registered then it might happen by taking person’s fingerprint - just for identification, but then in post factum detailed level information can be collected and monitored.

Question from: Ahmad Ramzi

Do you have any suggestions for conducting a program evaluation that is inclusive of changes due to the COVID-19 pandemic?

You need to tailor your program strategy plans, result framework, and indicators so that they will reflect the current situations and will be feasible and practice for this period of time, e.g. define new simple (minimum level of disaggregate, no need for baseline, etc.) indicators to measure the results. Afterwards via composite indicators, you can combine data of these simple COVID-related indicators with the regular ones. Please also check the answer of T. Senthoran's question above.

Question from: Samantha Woodcroft

How can an effective M&E system be developed in organizations other than project implementation? How can the best indicators be set for their organizational M&E only?

You need to design corporate-level result framework (e.g. including results/objectives relevant for partners’ performance as well). Then you should define your key performance indicators not only at an output level but also at an impact level. For example, to measure partners’ performance the following simple KPIs can be defined and attached to the corresponding result level:

  • Revenue through partners
  • Number of partners
  • Number of deal registrations
  • Number of people trained by an implementing partner

Based on the sector, you may have other indicators, e.g. for education - “How many out of school children have been reached out”, or for health - “How many children have been vaccinated”.

Please check the answer to the relevant question raised by Liliana Carral and Jaya.

Question from: Saba Khan

We are all aware that data is a very important aspect of M&E and it anchors decision making. How best can big data analytics be utilized in M&E to improve decision making?

We already have experience with implementing AI solutions within M&E systems to give our users smart predictions.

As a very simple example: if the system noticed the correlation between two indicators: “Car accidents mortality rate” and “Number of traffic violations”, then AI can predict increasing the ticket cost for traffic violations as an effective solution to the problem, so by establishing a liaison between indicators and analyzing them AI can identify and close the gaps.

Question from: Dexter Hati

What are the ways of monitoring and evaluating results of projects on social cohesion among refugees and host communities?

As a relevant example for you, we will mention the system developed through Synergy Indicata “The European Union Facility for Refugees in Turkey”.

The Facility Monitoring Platform (EU FMP) is an automated online monitoring and evaluation system which was gathering mainly data for Syrian refugees. The main objective of the system is to support the Government of Turkey in formulating a comprehensive strategy, effectively managing implementation, and promoting the accountable and transparent use of resources. This kind of systems can dramatically change the precise identification of beneficiaries and their further progress. The first step should be the proper identification (via ID, fingerprint, etc.) of beneficiaries and then their involvement in the right program/project. Pre- and post-evaluation should be there for assessing results accurately. These will help you measure the impact of the programs and/or projects.

Question from: Deniz Menteşeoğlu

Ho do the panelists see the application of big data in M&E?

This was addressed during the webinar by Emilya Khachatryan – please check the recording:

Cheap mobile devices and reliable internet services are enabling organizations to collect larger volumes of more reliable data.

Nowadays most of big data is collected from IoT devices such as sensors, cameras, smart parking solution, GPS tracking devices that are embedded in the water and power infrastructure, for example.

Data of “Electricity consumption rate” or “Basic Water Supply” indicators can easily be conducted through these devices, which will bring to big data. M&E is nicer and more accurate whenever data is big, i.e. better reports and analysis can be conducted, but in this case, M&E software becomes a must.

Potential problems include:

  1. M&E practitioners are inexperienced with large datasets and complex data models.
  2. Data cleanup and aggregation can be difficult.

Question from: Anush Shakhsuvaryan

The panelists started by saying that the key issue for M&E was to be useful for ongoing project activities and management - that is a different priority than accountability. Do you think one is more important?

Both, of course, are important, and we can’t really say which one more as they are closely intertwined. Accountability is necessary but having M&E on top of it you will have both necessity and sufficiency. M&E can be a useful tool during project execution if the parameters allow it to be, e.g., frequency of data collection. M&E can ensure better accountability both towards donors, as well as clients/beneficiaries.

Question from: Charles Downs

What tool is the best to analyze qualitative data?

Please check the answer to the question by Mara Tissera Luna posted above.
Question from: Gulnara Kokonbaeva

What is the best way you can collect feedback from beneficiaries especially during the COVID-19 pandemic? Is there any structured or recommended tool which can be used in a fragile context like Somalia?

Face to face meetings are probably the best way to collect feedback from beneficiaries as they can also facilitate identification, validation and deduplication. That said, face to face meetings are not always possible, due to the lack of funding, limited access to rural communities, conflict or a pandemic. In case of sufficient access to the Internet, beneficiary feedback can be collected online, however as we know this is mostly not possible, and in most cases the only alternative is using phone calls, SMS or IVR, or by using local focal points.

Recommended read: Data Collection in Times of Physical Distancing

Question from: Mubarak Nour

What are the properties of a good M&E system?

First of all, a good M&E system should be feasible and implementable. There should be SMEs who will be a dedicated staff in M&E-related activities. These team should define and design M&E work plan with corresponding budget, M&E frameworks/Logical Framework, smart KPIs, etc.

Question from: bbanguna

Have you come across biases in data gathered during COVID-19?

COVID-19 has had a negative impact on data collection and might have emphasized certain biases.

Recommended reads:

Check your bias, gender data experts warn

COVID-19: Guidance for labour statistics data collection

Question from: Dalita Artin Avanesian

How can you promote monitoring while you have a PMO, i.e. most of the PMO job is about following up and monitoring?

The primary function of the Project Management Office is to define and maintain standards for project management within the organization. There are several types of PMOs - Supportive, Controlling and Directive. Centralized and corporate M&E system approach is dependent on the type of the organization and it should be aligned with the management and PMO strategies and objectives.

If the question is about a project-level M&E, monitoring and control (please check PMBOK 6 edition) is one of the main phases (out of 5 phases - initiation, planning, execution, monitoring and control, closing) of project life cycle which should happen throughout the entire length of a project, and it should be conducted according to the standards set by PMO.

Question from: Hamza Abdalla

Is it better to have a centralized M&E unit or specific and decentralized units especially in the public health sector? People in this industry don’t think that M&E should be a centralized unit.

Having or not having a centralized M&E unit depends on the structure of the organization and/or sector, its objectives, as well as on the M&E practice and culture.

While there are benefits to centralizing M&E, not every organization is able to do so. Reasons may lie in the non-compatibility of programs, which can require various, sometimes incompatible M&E approaches, frameworks, subject matter expertise, etc.

Question from: Hamza Abdalla

Knowledge management is often linked with M&E, and where do planning and implementation of projects stand there?

Project knowledge management – cost, schedule, scope, resources, stakeholder, and other knowledge management areas are fulfilled in 5 phases - initiation, planning, execution, monitoring and control, closing correspondingly. M&E or Monitoring and Control (check PMBOK 6th edition) is an important part of any project and it should be implemented throughout the entire duration of a project.

Question from: Fikru Habte

How can you overcome the challenges of M&E knowledge and capacities within local organizations?

First of all, you need to identify stakeholders and other things that are creating those challenges. Then, you will need to gain the ownership and interest of those identified stakeholders, trying to find some incentive and motivation for each. You will need to demonstrate to them that good M&E is not for finding guilty people or for rebuke. You should make clear to everybody that good M&E is for a better way of working, for finding gaps, for making programs as effective as possible.

Please also check the answer to the relevant question raised by Liliana Carral and Jaya.

Question from: Elisa Riquelme

Where would you suggest that the M&E team seat in an organization?

Our recommendation would be to establish and have a separate independent department.

Question from: Lydia Mukaye

How can an M&E officer work more proactively to create an appetite for M&E's work within the organization?

Please check the answer to the question raised by Liliana Carral and Jaya.

Question from: Ahmad Ullah Qazi

What methods can we use to monitor humanitarian assistance in emergency?

For humanitarian assistance in an emergency, one of the most important activities is the identification of victims, then further tracking of corresponding beneficiaries through management and information systems. During emergency people can’t provide an ID card (passport, UNHCR passport), thus rapid biometric identification (fingering, iris scan, etc.) should be used. Further, if the beneficiary gives consent, details should be entered in a management system for detailed monitoring. Note, personal data is sensitive, thus all governmental, GDPR rules should be considered during this activity, i.e. in case beneficiary is requesting to delete their personal data from the system, that should be done immediately. Another method is data anonymization/encryption that should be carried out in Management and Information Systems.

Question from: Salauddin Ahmed

Which M&E skills may be considered trending?

This may vary. The trending skills now may include: data collection automation (through IoT and smart devices), calculation of results (Impact, Outcome, Output, etc.) and aggregation automation through a management and information system (M&E systems), AI integration for projections.

Question from: Mohammedhal

Webinar 2: Technologies and Innovations

Are organizations ready to cope with the change in terms of the availability of a budget and trained staff?

It depends on the organization and type of the project/program. However, we see a trend towards the increase of M&E budgets and relevant staff training (excluding the COVID-19 period).

Question from: Kelly-Ange Irakoze

How will 5G technology affect the way we do M&E?

We expect that 5G technology will improve the Internet access for a large number of people, therefore enhancing data collection and M&E data reporting on-the-go via mobile phones, tablets and other portable devices.

Question from: Pasteur R.

How can we compromise between addressing the increased demand for real-time data and the need to provide robust and technical sound indicators?

The use of technology allows M&E specialists to rapidly translate and structure real-time data into indicators and create a results framework/theory of change/any other type of frameworks to organize, systemize and analyze the given data.

Question from: Ashraf

How can M&E make an impact on the project implementation?

As you plan your project objectives, logistics and resources, it is best to start designing and incorporating your M&E simultaneously with the input and feedback from all stakeholders and team members involved in the project. It is important to link your M&E plan with the project’s strategic plan in an integral manner. This ensures an effective allocation of resources, establishment of roles and responsibilities, an understanding of the political and administrative structures of the community where the project will take place, etc. Likewise, planning the M&E activities early on also helps identify your baseline values to measure the success and failure of the interventions during the project implementation and your impact during project implementation and closing.

Question from: Maazua

How can we ensure that everyone in the organization is made aware of the M&E frameworks in place so that they are involved in the process to ensure their impact on the overall objectives is felt?

You should communicate the importance of M&E in your organization to the stakeholders, as well as avoid common mistakes in M&E, which reduce the effectiveness of M&E data collection and reporting. Make sure the M&E data is communicated back to the stakeholders who report the data so that they are aware of its importance in the big picture.

An M&E framework and its progress can be part of every employee’s homepage.

Question from: Nelly Saiti

How does M&E contribute to ensuring that qualitative data is quantifiable and what technologies would you recommend?

A common method that evaluators use to analyze qualitative data is triangulation, which involves taking data, finding themes, coding them, and then comparing or triangulating the data from different data sources and different data collection methods. Here automation or AI technologies can be quite helpful.

Question from: Nelly Saiti

How is the development or program effectiveness measured at global/HQ level of an organization working in different countries and with different donors? What M&E systems do you use and how do you standardize them?

A global/HQ level Result Framework can be maintained and individual Project or Program level framework results can be mapped to the global result framework results. This practice can be done via Indicators and Composite Indicators (through the bottom-up aggregation) at the global level or through the mapping of result levels to maintain a cohesive and connected M&E structure. This way you will be able to see how each project/program is contributing to the global corporate level.

Question from: Retta G.

Since outcomes are typically more difficult to monitor and evaluate and data is often not available and primary data collection is required, what tools can be used to manage and collect the outcome data?

Outcome data can be collected through:

  • Surveys
  • Interviews
  • Focus and other small groups
  • Observations
  • Records and documents

Question from: Hamda Hersi

The topic of remote monitoring is very critical for many areas. What impact will this have on the technology?

With the use of technology, remote monitoring is improved on all fronts, as it is possible to automate the following key concepts which remote monitoring hinges on:

  • Include M&E as part of ongoing practice,
  • Standardize and simplify,
  • Open opportunities for replication; recognize good practice,
  • Achieve participation from users,
  • Focus on demand-side requirements; aim for quality engagement in M&E processes and obtain multiple perspectives,
  • Seek out lessons for improved practice,
  • Budget and plan for adequate time and resources – including training where necessary,
  • Include high-level (long-term) goals.

Question from: Matthias Leibbrand

How can M&E help avoid data overload and still be effective in measuring impact?

A properly designed M&E provides a structured and systemized option of sorting, organizing and maintaining data via indicators and M&E frameworks, which allows M&E specialists to sort and maintain essential data, without getting overwhelmed by data. In M&E software solutions, the bulk core (sometimes even unstructured) data is segregated in different databases while only essential and aggregated data is kept in the main system database.

Question from: Maazua

Online data collection or remote data collection is cost-effective. How important are observation and face-to-face interviews to monitor the effectiveness of the intervention?

Yes, observations and interviews are also important tools for data collection, especially for outcome results. However, the data collected via those methods also need to be digitized and systemized to maintain data completeness and allow M&E specialists to analyze it effectively.

Question from: Anu Bhowmik

Which is more useful to consider in M&E - output or outcome impact?

All components are required to structure the results framework of a program/project. It is important for M&E practitioners to design the framework in a way that covers the main goal of the program/project but for ease of implementation and measuring its impact this goal should be broken down into smaller goals (outcomes), then to be broken into feasible and tangible outputs.

Question from: Yaghouba Kane

In terms of technology, how can we have a successful MIS system (a successful automation of M&E system) that can help track routine data (output data), capture data coming from the field, aggregate it to measure indicators (with MIS in built business logics) and generate reports or dashboard in real-time for the program team and management?

For any technology project, there are three paths – open source, custom development and proprietary product. Each approach has its own pros and cons and the decision depends on many factors (organization’s vision, budget, timeline, maintainability, etc.).

Based on our experience, organizations that have chosen custom development usually carry a higher risk of failure. Also, these organizations tend to switch to open source or proprietary product once they realize that their in-house product either is not capable to scale or requires high maintenance in comparison with other similar products.

Please read this whitepaper to learn more about open-source vs. propriety product.

Please also check the answer to the relevant question raised by Liliana Carral and Jaya.

Question from: Retta G.

How can we keep data secure if we introduce an M&E technology in an organization?

For sensitive data protection, M&E software providers must develop a data encryption policy based on which they should be able to store and protect data with different types of encryption algorithms. Synergy has developed a GDPR- compliant policy. For example, in Synergy Indicata a user can define which data is sensitive, and based on that the system will automatically create a database table with a column-level encryption.

Also, proper M&E software solutions have multi-factor authentication and multi-layer data encryptions, which limit unauthorized accesses and cyber-attacks. In addition, most of the popular cloud providers cover all known cyber-attacks from an infrastructure perspective.

Question from: Badrun Nahar

Do you think media monitoring could help address the M&E challenge in the current pandemic situation?

Yes, any type of remote and automated data collection that is feasible and useful addresses ME challenges during the current coronavirus pandemic.

Question from: Badrun Nahar

How often primary data collection is recommended for outcome level indicators and what tools and systems are available and affordable?

Usually primary data collection is recommended for output level indicators while for outcome level indicators surveys are recommended tools.

Synergy Indicata is a powerful M&E data entry (for both qualitative and quantitative data) and analytics tool.

Question from: Sinshaw

How do you measure the sincerity of the responses? When talking to people, we are trained to see if they have doubts about what they are saying. How to capture this in remote data collection?

First of all, the questionnaire template should be tailored to address the challenges of remote surveys, i.e. to minimize the questions whose responses are dependent on person's sincerity/emotions (to have structured answers, e.g. Yes, No or to provide predetermined options).

Statistics shows that if in about 10% cases a respondent is changing their answer, there is a possibility of insincerity or non-confidence. In these cases, double verification questions can come for help.

Question from: Adiza Lamien Ouando

Is it necessary to collect huge data to measure outcome? I think we can use LQAS Sampling techniques to measure the outcome.

This totally depends on the outcome description, and outcome result can be measured through surveys, interviews, focus groups or observations, without collecting big data.

The LQAS random sampling methodology is mostly for quality control, which provides less information but often requires substantially smaller sample sizes. As for huge data - if it is collected correctly and accurately, it is always possible to avoid mess and manage it through technology, i.e. automate, aggregate, slice and dice, and filter big data.

Question from: Anu Bhowmik

How does the technological advancement affect the M&E activities?

Some technological advancements like NLG (Natural-Language Generation), AI (Artificial Intelligence), ML (Machine Learning) and IoT (Internet of Things) can shift the efforts and focus of M&E practitioners towards value-add activities and reduce their routine burden, specifically on data collection and reporting activities.

Question from: Mubarak Nour

Please recommend digital survey tools/platforms?

The recommended standalone platforms for conducting surveys include SurveyCTO, KoBoToolbox and SurveyMonkey. They are robust platforms with the ability to create complex data collections forms and integration with data analysis platforms.

Also, M&E solutions like Synergy Indicata has the built-in survey module. This works seamlessly with other M&E-related functionalities while data visualization and analysis can be done within the solution itself.

Question from: Sinshaw

Does changing to Industrial 4.0 affect M&E much?

Absolutely! This means that cutting-edge technologies like Predictive Analytics, ML (Machine Learning) and IoT (Internet of things) will be engaged heavily to help organizations deliver their programs in the most effective and efficient manner. The vision for M&E might be to have machines fully/mostly cover the monitoring component and assist M&E professionals/program directors in driving and leading mission-critical programs.

Question from: Khin Zaw

How about RCTs type if evaluation designs and donors expect that? Often budgets constrain us to conduct such powerful surveys.

A randomized controlled trial (RCT) is an experimental form of impact evaluation in which the population receiving the program (direct beneficiaries) is chosen at random from the eligible population, and a control group is also chosen at random from the same eligible population. In this case surveys and data collections are more cost-effective, thus donors should agree with this approach if the need of this evaluation is justified in the scope of the project and all templates, processes are designed/planned correctly.

Question from: Sinshaw

Is it recommended to use LQAS to track outcome indicators and what merging of anticipated change in indicator progress is acceptable in that regard?

Lot Quality Assurance Sampling (LQAS) is a random sampling method used for quality control in industrial production. Compared to similar sampling techniques like stratified and cluster sampling, LQAS provides less information but often requires substantially smaller sample sizes. It uses small sample sizes to classify health or administrative geographical areas, to inform if these areas have achieved a pre-determined target for a given indicator (measuring outcome indicators).

Conducting population-based surveys using LQAS for both baseline and ongoing M&E assessments for outcome level results can be very useful. LQAS is a tool for managers to help them improve the overall program performance.

Is Re indicator progress acceptable or not? If this was anticipated, benchmarks, targets should be set up correspondingly. Keep in mind that M&E should always be feasible.

Recommended read: Lot Quality Assurance Sampling Measures Performance

Question from: Sinshaw

Webinar 3: Synergy Indicata M&E Software

#1 Does the system also support more qualitative evidence, e.g. outcome mapping, MSC, outcome harvesting, etc.?

The system support qualitative data collection and mapping through qualitative indicators. For more details on how data collection and data analysis are performed, see Question #8. 

Another advantage that the system has is the possibility of mapping the results with each other. This wat, the MEAL officers can define which outcome is contributing to which result. The screenshots below showcase how this is done.

Question from: Anna Pierobon
Company: Civil Rights Defenders
Position: PME Officer
Country: Sweden

#2 Can the data in the project logframe be synergized with the data in the programme logfrme?

Synergy of the project and program logframe data can be achieved through Project to Program level result mapping. This means that each project level result (Goal, Outcome, Output, etc.) can be mapped with the program level results. For more details, check Question #1.

Question from: Anna Pierobon
Company: Civil Rights Defenders
Position: PME Officer
Country: Sweden

#3 Can these figures be synced on a corporate website? so that they are automatically updated on the webpage too

All analytical pages in the system have the share with public functionality, i.e., each report, graph, list can be published via a URL, and this URL can be embedded in corporate websites. Kindly check the screenshots below on this is done in the system.

Question from: Anna Pierobon
Company: Civil Rights Defenders
Position: PME Officer
Country: Sweden

#4 Is the programme director the only person that can edit this?

We used the Programme Director role as an example. The system administrator can create as many users as required, and each user can have a role with a predefined set of permissions assigned to them. The screenshots below shows the Indicata User Management screens.

For more details on user management and permissions assignment functionalities, check Question # 17.

Question from: Anna Pierobon
Company: Civil Rights Defenders
Position: PME Officer
Country: Sweden

#5 Is there a module for Strategy ?

Synergy Indicata supports Corporate Strategy result frameworks as it is illustrated in the screenshot below. This can be easily customized / configured if a need to design specific Strategy execution arises.

Question from: ayousief
Company: Qatar Foundation
Position: Manager, Performance & Evaluation
Country: Qatar

#6 Can you please demonstrate how you map the project indicator with the program indicator?

Project and program indicator mapping is carried out through composite indicators, i.e., the program can use composite indicators and be based on project indicators (to be considered as core/source indicators). The MEAL user can create program indicator with a specific data point (called an expression type data point in Synergy Indicata) that will be based on a project Indicator. For more details on indicator mapping, see the screenshot below.

Also, check Question # 7 for more information.

Question from: Bidur Bastola
Company: CARE
Position: DMEL&KM Coordinator
Country: Nepal

#7 And how we can see progress in program ITT entering progress in project ITT?

In the project indicator IPTT (Indicator Progress Tracking Table) screen, the data collector user enters figures for actual data points.

The IPTT screen for program (composite) indicators are automatically designed. This means that the program indicator data points (fields) are based on the project (core/source) indicators. As such, the values in them are read-only and are automatically calculated/aggregated. The screenshots below show the IPTTs of the project and program indicators.

Project (Core/Source) Indicator IPTT:

Composite Indicator’s Data Point reference to source project indicator:

Program Composite Auto-Calculated values in IPTT

Question from: Bidur Bastola
Company: CARE
Position: DMEL&KM Coordinator
Country: Nepal

#8 For IPTT, is it including the tools for both qualitative and quantitative data to measure the result of output and outcome level according to indicators?

The IPTT supports input of both qualitative and quantitative data. The screenshots below demonstrate the IPTTs of two sample qualitative indicators. The first bunch of screenshots shows the qualitative indicator named “Women's participation in decision-making at project/local level”. The data points of this indicator are text areas for entering narrative description. The second bunch of the screenshots show the IPTT of a value-based indicator named: “Level of Satisfaction”. In this case, the data point fields are drop-down lists with predefined items.

  1. Quantitate Indicator with narrative data collection:

     
  2. Quantitate Indicator with value-based list:

Question from: Chanleakhena - Plan Cambodia
Company: Plan International
Position: M&E Officer
Country: Cambodia

#9 Also in analytics, can we compare planned vs actual progress?

Users can create analytical reports where the first component shows the actual value, while the second component shows the planned target values. Another component that report can have is the progress measure that shows the achievement percentage. 

In general, any numeric value (e.g., planned resource LOE vs actual LOE, planned budget vs actual budge, etc.) in the system can be brought to the analytics, and a comparison can be drawn between them in the table reports, charts, etc. Kindly check the screenshots below for different types of analytical reports that allow comparing data.

Question from: Mohammed Abdelaziz
Company: QSIT
Position: Industry Manager
Country: Egypt

#10 If customers have their own GIS server, with their own maps on it. if so, can u connect to ArcGIS and use map services from there?

Indicata has a built-in interoperability capability that has been developed to ensure the openness of the technology platform and to support data exchange with external systems. We provide two methods of integration, and both approaches adhere to the internationally recognized standards of interoperability, confidentiality, and open standards in the information exchange service. The two methods of integration are as follows:

  1. Point-to-point integration – direct integration with third-party systems via a well-structured API. It is the simplest topology that directly connects two nodes through a common link.

  2. Integration via Enterprise Service Bus (ESB) – we have developed an Integration Gateway called Synergy Integration Gateway (SIG) to receive and send messages among integration participant systems. Synergy Integration Gateway (SIG) is a real-time data integration middleware for applications that require instantaneous data across all applications and/or integration parties. It allows developers to set up client listeners to receive messages from other applications and publish them to one or more message queues.

Question from: Mohammed Abdelaziz
Company: QSIT
Position: Industry Manager
Country: Egypt

#11 What does the Log frame Tab include?

 LogFrame is an out-of-box control in Synergy Indicata. For more details on this, kindly check Question #18.

Question from: Qazi, Ahmad Ullah
Company: ILO
Position: Monitoring, Evaluation and Communications
Country: Pakistan

#12 Question from: Qazi, Ahmad Ullah Company: ILO Position: Monitoring, Evaluation and Communications Country: Pakistan?

Please, check Question #8. 

Question from: Stephen J. Ajalu
Company: World Bank
Position: Senior Urban Development Specialist
Country: Uganda

#13 Does it have any mechanism to clean and validate the data?

Indicata has several levels of validation checks/data accuracy validations. These validations include, but are to limited to forcing to fill in the required fields, not allowing to fill in 0 in the indicator denominator, or not allowing actual values to be less than the baseline values, etc. 

To minimize the possibility of a human error in data entry and, thereby, improve data quality and completeness, Synergy Indicata provides the following built-in types of data validation:

Field-level validation: At this level, the system ensures that the data format and range comply with the format and range specified in the data entry form/field description. For example, if the data entry field is set to collect a numeric data (i.e., age), then users cannot type any letters or special characters. Or, if the field is expected to accept a range of values (i.e., year of birth), the system may accept values between 1910 and 2021.

Form/screen level validation: These types of validations are enforced when there is a need to compare one or more data elements on the same form/screen and ensure proper quality of data entry. For example, in capturing a project’s start and end dates, the system performs screen level validation and ensures that the start date is earlier than the end date.

Server-side validation/logical validation: These types of validations are conducted across entities that are not downloaded to the client side but need to be compared and validated before they are saved in the database. For example, the project activity budget cannot exceed the total project budget.

Server-side validation/data element as part of a relational database: With these types of validations, the system conducts data type validation and data integrity validations at the level of Relational Database Management System (RBDMS).

Question from: T.Senthoran
Company: Save the Children International
Position: MEAL Coordinator
Country: Sri Lanka

#14 Is there any possible way to connect to remote db as a source of data colllection, I mean by that connecting to oak database (Postgres or mysql)?

The Indicata architecture is based on micro-services, and each micro-service has its own database. All micro-services have end-points and communicate with each other through APIs. This means that for those services, it is not essential whether they communicate with an internal or external party. From security perspective, we do not recommend direct connection to third-party databases (although it is possible) but recommend having data transfer through integration.

The screenshot below illustrates how easily integration with external systems can be implemented.

For more details, please, check Question # 10.

Question from: Haithm Garallah
Company: Moore Yemen
Position: Data (R&D), Supervisor
Country: Yemen

#15 This is no doubt a comprehensive results aggregation and analysis system. However, this will require dedicated HR to manage this system or the M&E person to be trained heavily in this system or outsourcing of M&E system to the developer agency, which will make the project M&E officer Jobless.

One of the outstanding features of our systems is that multiple processes in it are automated and digitalized. However, the ultimate success of the system depends hugely on the dedicated personnel involved in the process of feeding the system with reliable and accurate data. This means that for the M&E related processes to be efficient, designated staff must persistently and consistently collect the indicator data and enter them into the system. As a final step, the data entered must be analyzed to reveal trends, and this will also require human involvement.

There is also need for system administrator for maintaining system and configuring it according to organization changes and current needs.

Question from: Arif, Tahmid
Company: ILO
Position: Programme Officer (M&E)
Country: Bangladesh

#16 Are the only two possible login options ‘Program Director’ and ‘Admin’? Or are there other login options for M&E Officers or M&E Managers?

During the webinar, we used the Admin and Program Director users as examples. The system administrator can create as many users as it is required, and each user can have its role with a set of predefined permissions assigned to them. For more details on the user management functionality of Synergy Indicata, please, check Questions # 4 and # 17.

Question from: Ashelyn Pindell
Company: Center for Civilians in Conflict
Position: Monitoring Evaluation and Learning Officer
Country: USA

#17 How are data visibility and security related issues catered for in the system?

This is carried out through our very powerful User Management. Each Indicata user has a unique and secure sign-in with password validation associated with a set of administrator-defined user permissions. Therefore, according to the user credentials entered, the system will enable the user to view, edit, delete only the system resources to which the user has access.

Indicata provides differentiated access to information based on a combined model of OrBAC (Organisation-based access control), ABAC (Attribute-Based Access Control), and RBAC (Role-Based Access Control).

The OrBAC approach is based on the organizational structure of the Client and the permissions are granted based on the user’s position in that structure

With the ABAC, the authorized users are granted access rights based on different attributes predefined within the system. For example, a user can be granted edit/delete permissions for all the schools in a specific location, while for the schools located in other regions his/her access rights can be limited to read-only.

The RBAC approach allows granular management of user permissions and authorizations on data items to ensure strict security and access control. Indicata gives the possibility to provide more detailed permissions for each role for each state on the level of each field in the system, thus being a role-based workflow-driven system. This enables the administrators or users who have access permissions to configure and control the extent to which the participants are involved in the process. This is done by assigning each participant with the appropriate permissions for the process states according to the role that they play in it.

Indicata does not impose any predefined roles or set any limitations to the variety of possible roles or user groups. Rather, the Indicata model is entirely configurable to reflect any kind of staff structure.  

User permissions are defined at the field level through the User Management module, meaning that system information will not be visible by any user without the appropriate permissions. Security permissions can be applied to restrict what module, section, data field, or other information is available to which user/role. Administrators can set up security individual profiles for user and group access. 

System administrations define roles and apply permissions to each role. Role permissions can apply to all users or only a subset of them. In addition to the role-based permissions (when the user inherits the permissions from the assigned role), users may be additionally assigned individual permissions (custom). Users can be associated with one or multiple roles that entitle the user to access all resources protected by those roles.  

Access/role permission control management is configured to be exceptionally user-friendly – ticking or un-ticking a check-box in the 

User Management module is all that is required – and can also be used to restrict roles to certain functions, assign one or more roles to registered users, restrict access to certain locations by an individual, and secure specific information fields from certain roles. 

Only the administrators of the system will be able to manage the user access rights associated with each user account, customize views to enable or disable the available modules, manage user group settings through the User Management module.

Question from: Stephanie Hohenberg

#18 Does the Log frame Tab include image of the LF or is it an interactive one with provision of timely review?

LogFrame is interactive. Please, check the following webinar recording where it is demonstrated how the LogFrame can be designed (i.e. how logframe levels are selected, how result items are created, how indicators are attached, etc.) and later adjusted to the organization’s needs without any coding or re-programming.

https://www.youtube.com/watch?v=j9YinE-GaNM&t=2007s 

For more details, please, also check Question # 11.

Question from: Arif, Tahmid
Company: ILO
Position: Programme Officer (M&E)
Country: Bangladesh