Evaluación del Sistema de Mejora de la Calidad

Project Description and Background to Evaluation

The Quality Improvement System (SMC) is a project implemented by VIVA since 2009 in the cities of La Paz and El Alto, with the purpose of promoting quality improvement in organizations working with children and adolescents at risk.   This tool interacts with organizations to achieve quality improvement in six areas: Technical assistance to staff, good governance, Child Protection, Financial Transparency, Planning and Project Design and Integral Development of Children. Each of these areas has been identified as key to ensuring a high impact on actions in favor of Children and Adolescents at Risk. The SMC works on two levels of quality improvement: the fundaments level and Quality level. These are achieved through the application of the so-called Improvement Cycle.

Purpose/objective

Three objectives were considered in this evaluation. First to measure the level of achievement in the indicators of purpose and results defined in the project logical framework. Secondly to analyze the effectiveness and efficiency of strategies and/or processes that were implemented by the project. Finally to identify and analyze, which processes were generated for the intervention sustainability.

Within the scope of the evaluation, the following aspects were considered: 

• Effectiveness of methodologies and strategies implemented by the project. 
• Effectiveness of actions implemented with the target population, demonstrating the effect on them.  
• Relevance and pertinence of training modules.  
• Relevance of goals and outcomes established in the design and within the current context, where the project was implemented.  
• Effectiveness in achieving products: Beneficiaries at the expected time, activities in the periods established, investment in accordance with activities. Good practices and lessons learned.

Methodology

Qualitative and quantitative methods for the gathering and analysis of information were used. The data collected were in line to indicators. Likewise it was used to illustrate experiences and opinions. It helped to understand and analyze the sequence of events, cause and effect, how perceptions affect decisions; and make an analysis of narrative speeches. Furthermore personal interviews were used to collect information that was not available in files or reports. This also helped to learn experiences and perceptions. Documentation review was also conducted. This was one of the main methods of the evaluation to collect secondary data.

Key findings

• There is evidence that children receive better care in Projects from staff. Children perceive that their teachers are more patient, treat them with more affection and respect.
  
  
• In the centers, the Child Protection requires 100 of support that means empowering, raising awareness, sensitizing the team that is in direct contact with the children. In this regard the work done by SMC is carried out with the Centers as an "invaluable help" because it was observed the staff’ change of attitude when interacting with the children and the same among coworkers.
  
  
• An example is the experience of a SEDEGES project that had a high percentage of dropouts children because of the bad treatment they received in the center. After the implementation of the protection module and additional issues, the SMC team witnessed that the number of children running away became zero. The impact reached a point, where today there are children, who do not want to leave the centers because they receive better treatment there than at home. Although it is recommended to incorporate modules of Family Reintegration, it should be considered the rotation of staff and children.
  
  
• SEDEGES has requested VIVA to become advisors in the Departmental Plan 2014-2018. The aim is to integrate the SMC modules in its Strategic Planning with the purpose of making it a standard for all organizations working with children and adolescents. An achievement of great impact is that SEDEGES wants to incorporate into their organizational culture the Improvement System Quality that means as part of the work in all foster homes and make them a model of work for other organizations.
  
  
• 16 interviewed Projects said they value SMC, as a tool to "put order in the house» not only at the Administrative Management level but also at a level, where there is a direct impact on the lives of people who work with children. These people feel empowered, motivated and challenged to do their work with greater diligence and respect for children.
  
  
• Directors (as) of projects see a change of attitude in their educators and / or tutors.
  
  
• The Support Quality Network is an instance that works and where members of different projects are currently implementing the Quality Improvement System. Likewise there are professionals and technicians in different areas that help to strengthen the quality issue. These professionals are responsible for giving updated courses, workshops and provide advice on thematic associated with the system.

Recommendations

• It is necessary to strengthen the implementation process. This stage should not be left unattended and without supervision. The SCM technical team must accompany the Projects to ensure proper implementation.
  
  
• For the number of projects implemented by SMC, the number of visits to workshops, monitoring and evaluation of process implemented as well as budget seem to be small. It should be estimated the cost of implementing the SMC per Child Care Center, considering an introductory visit to the SMC, workshops, diagnostic visits and / or evaluation, follow-up visit, monitoring visits, materials, etc. This would give a real approach not only for the strategic planning of new interventions of SMC, but also it would give a real response to centers’ demands of increased monitoring. In other words it would allow to plan and projects according to resources.
  
  
• People doing field work in coordination with SMC staff must define a minimum number of visits to each project. In addition they must define the number of projects, where SMC will be implemented each year as well as the maximum time to implement the SMC, because there are projects that are working three modules in two years.
  
  
• The implementation of the 6 modules can be done over a period of 1 year. The 16 projects, who participated in the evaluation, agree on that. They also said they could include this activity in their Annual Work Plan (POA) in order to ensure their implementation.
  
  
• Projects that do not meet all the requirements should not get the certificates. Also they should not be moved to another if they have not finished the first module. This weakens the System standards and cause to lose thoroughness, which must be part of the SMC.
  
  
• Projects that completed 6 modules and were certified with the seal of quality must renew the certification annually. So they do not have the impression they "already attained it" and "there is nothing more to do".  Organizational culture must be also generated in the Projects and annual evaluation about the system performance may be pushed the sustainability of this culture. On the other hand this would contribute to a continuous improvement.