.

The Indonesia version of this article was originally  posted on the Knowledge Sector Initiative website and has been translated by the SMERU team.

.

It was October 2014 and something felt different for Nila Warda, or familiarly known as Lala, and the other members of the SMERU survey team. They were on their way to the rural areas in Kabupaten Timor Tengah Selatan, with their steps seeming lighter than usual. Instead of carrying a big backpack filled with paper questionnaires, this time each team member only had with them a smart tablet when visiting respondents’ homes. That was the first time SMERU officially utilized a smart tablet, or what is known as computer-assisted personal interviewing (CAPI), in its primary data collection activities.

****

In the previous year, The SMERU Research Institute had just concluded their study on teacher absenteeism. With 8,000 students from 900 schools as the respondents, the survey produced a mountain of paper questionnaires, piled up in the corners of SMERU’s office. The data entry and cleaning processes also took forever to complete. The overwhelming resources that must be spent for such big-scale research have been a concern for SMERU researchers. On the other hand, as a research institution, SMERU is constantly required to develop its capacity so that it can produce bigger impact from its research. It is this line of thought that served as the basis for utilizing CAPI.

The idea of utilizing CAPI was introduced by Asep Suryahadi, Director of The SMERU Research Institute, in a staff meeting held in June 2014. Asep suggested the utilization of digital questionnaire on tablets to avoid the classic problems that frequently occur when conducting big-scale research. The idea came to Asep after reflecting on the experience of other research institutions, such as Statistics Indonesia (BPS) and SurveyMETER, which had started using CAPI as their main approach in their data collection.

Though interesting, this idea was not immediately accepted. Some voiced their concern about whether there would be an ethical issue, i.e., using a smart tablet—which is an expensive gadget—when conducting interviews with poor or even very poor households that often become the main respondents in many of SMERU’s studies.

Most staff members, however, agreed that they needed to implement CAPI immediately. They believed that CAPI offers several advantages. The first one is that the allocation of budget and other resources to copy the data in the paper questionnaires into the digital form will automatically be no longer required. Next, direct supervision can be carried out on the same day of the survey so that the research team can assure the quality of the survey results. Thirdly, with CAPI, SMERU staff will be more comfortable working at the office; gone are the needs for transforming the office rooms into emergency storage rooms for the post-study paper questionnaires. Most SMERU staff members agreed that the advantages offered by the new method outweigh the potential risks that might arise. It was then decided that SMERU was to utilize CAPI for the first time in their study on the livelihood of poor women under the Empowering Indonesian Women for Poverty Reduction (Maju Perempuan Indonesia untuk Penanggulangan Kemiskinan/MAMPU) program.

****

Prior to the study, in July 2014, Rahmitha, the leader of the MAMPU research project, conducted a comparative study at several institutions and with key informants, including those from BPS and Firman Witoelar from SurveyMETER, as they had already incorporated CAPI in their data collection process. From the comparative study, the SMERU team learned that the use of CAPI requires technical know-how, including mastering CSPro [1] program, which can be used to create survey applications to be deployed on Android (via smartphones or tablets). Even though at first there were doubts, the SMERU team started the development of a CAPI system which was adjusted to their specific needs. Several SMERU staff members, such as Nila Warda and Hafiz Arfyanto, were assigned to learn the CSPro features that are especially meant for developing digital questionnaires in a relatively short time. Making use of all the available resources, including learning about the ins and outs of CAPI through online media, in three months the team could develop a CAPI system ready for a field test.

In September 2014, SMERU conducted a pilot study in Kabupaten Bandung, West Java, using tablets. At the time, the SMERU quantitative research staff acted as the enumerators. The result of the pilot study was not satisfactory. Various technical issues arose and the adaption process by the enumerators was less than ideal. However, the concern that the survey team would receive negative responses from the poor households proved to be groundless. After the study, the team still discovered some peculiarities in the data they collected. Again, this was because of the technical errors occurring when using the tablets. Considering the results, the SMERU team were optimistic with CAPI, seeing the positive potential it offers despite some technical issues that needed to be resolved.

Several months followed; the team continued developing the system and learning some more about CSPro. Finally, in October 2014, the SMERU team conducted their first survey using CAPI for their MAMPU study. Using 24 Android tablets, the local enumerators and regional coordinators from SMERU conducted a survey in Pangkajene Kepulauan and Timor Tengah Selatan. Before going to the field, the enumerators took part in an intensive training program that lasted for three days. During the training, they learned to work with the tablet and digital questionnaire. One of the challenges facing them was the adaptation process which varied between enumerators. In general, the survey was successful, with some drawbacks. For example, when there was an error with the application system on the tablet, the enumerators tended to panic, as there was no procedure for handling technical issues. In November 2014, the team went to several other study locations, namely Deli Serdang, Cilacap, and Kubu Raya. The survey went well and the result was good.

Then it was time to check whether the data was of good quality. Back in Jakarta, the data cleaning process was underway. It took around four or five months, which is relatively fast, to clean the data of about 1,500 households. To compare, SMERU’s study on migrant workers’ children (the Children Left Behind/CLB study), which used paper questionnaires, had only around 400 households as the respondents and it required the same amount of time to complete the data cleaning process. We should also take into consideration the time saved, as there was no more data entry. Also, no more revisitings to the respondents’ homes were required after the team returned to Jakarta. Revisiting a respondent is usually necessary if there is a possibility that an enumerator has filled in the questionnaire not based on the information from the respondent. CAPI has made it possible to detect inconsistency in the answers to questions in a questionnaire more easily.

****

The utilization of CAPI continued in the next studies, such as the study on clean water (IUWASH) in Bogor and the second wave of the MAMPU study. The new study gave rise to new challenges and room to further improve SMERU’s CAPI system. Hafidz was one of the SMERU staff members who played an important role in developing the system. He invested much time in learning and improving his understanding of CSPro. As a result, in the second wave of the MAMPU study, led by Niken Kusumawardhani, the data cleaning process took only a short time, i.e., three or four months. "CAPI also helps minimize human error from the enumerators’ side. This is a truly useful approach, especially for a large-scale longitudinal survey," Niken said.

SMERU’s use of tablets continued to expand. Besides replacing paper questionnaires, tablets are used as a recording device and GPS locator in the PKH study led by Rika Kumala Dewi. Unlike the MAMPU study, the PKH study was a qualitative one so that it did not rely so much on questionnaires. The features were also useful during the MAMPU study, especially to locate respondents’ homes during the second wave of the survey. With the tablets, this was much easier to do and tended to cost less because there was no need for investing in extra hardware.

The data collecting process was faster; there was no more need for data entry; and it was easier to run the data cleaning process. These have positively impacted the quality of the reports and the study donors’ level of satisfaction.

Local enumerators also enjoyed the positive impact. Firman Silalahi, one of the local enumerators for the MAMPU study in Deli Serdang, said that at first it was difficult to work using the system simply because he was not used to using a tablet to conduct a survey. However, after making adjustments, the impact was nothing but positive. Firman compared his involvement in the teacher absenteeism study and the MAMPU study. In the former, he still worked with paper questionnaires and in the latter, he utilized a tablet. He said, “Once we know how to work with the tablet, our interaction with the respondents remains effective. When using the tablet, the respondents tend to be more enthusiastic and serious in answering the questions.” Firman also felt that enumerators were helped with the features in CAPI, which could remind them should any information in the questionnaire contradict any previous piece of information.

Aside from the practical use mentioned above, the CAPI system provides opportunities for SMERU to conduct a longitudinal quantitative study [2] at the household level, the first in the history of SMERU. Previously, such study tended to be difficult to conduct because it required complex data management and needed GPS locators to track back the position of the respondents’ homes. Digitalization of the data management helps ease the process. The MAMPU study proved that the team could retrace 94% of the respondents. The success of conducting a large-scale longitudinal study was an achievement for SMERU. Only a few research institutions in Indonesia have managed to conduct a longitudinal study with a very high revisiting rate.

One important thing to note from SMERU’s experience in using the system is the support from the Knowledge Sector Initiatives (KSI). The purchase of the tablets could only be possible with the core funding from KSI. The ease of allocating the fund for the purchase is an example of KSI’s commitment to the development of research capacity of local research institutions in Indonesia. Going forward, SMERU and KSI expect to remain committed to developing research potentials, not only limitedto the development of the CAPI system, but also the adaptation of technology that may be available in the future. SMERU is enthusiastic about sharing this experience with other institutions so that research activities in Indonesia will develop further and be able to provide vital input for the formulation and evaluation of public policies.

[1] CSPro was in fact not new to SMERU researchers. All this time, however, the utilization of this software was limited to formulating paper questionnaires. To develop a questionnaire in the form of an Android application for an Android-powered tablet requires new technical knowledge, such as the understanding of the logic file, and other technical matters.

[2] The longitudinal study has a great potential to influence policymaking, as it can help policymakers see the impact of a policy or the present condition more clearly. This is possible because in a longitudinal study, the research team revisit the same households in each batch of the survey in a certain period of time. In many advanced countries, longitudinal studies are very reliable to obtain accurate information needed in policymaking.