Data Export

Export of data may be requested at any time when the Survey Solutions server is running (during or after the survey) and it doesn’t cause any interruption of the synchronization (of mobile devices) or online work (CAWI surveys). In case the server is busy processing other export requests, the new export jobs are put into a queue and processed sequentially. This article describes how to request a data export from Survey Solutions.

System generated files

Every archive of data exported from the Survey Solutions Headquarters contains among the user data the following system-generated data files:

• assignment__actions
• interview__actions
• interview__comments
• interview__diagnostics
• interview__errors

The above data files are saved in the same data format as the main survey data, for example, if the main data is requested in Stata format, the system-generated data files will also be exported in Stata format.

The data files are also accompanied by an automatically generated text file named export__readme.txt. This file documents which version of the Survey Solutions software was used to export the data, when the export was done, and placement of variables into the data files. Here is an example of such a file that may be accompanying, e.g. an agricultural survey.

Paradata file format

Every survey collected in Survey Solutions is supplemented with a paradata file. No actions are needed from the questionnaire designers or headquarters team to collect it, it is produced automatically by the Survey Solutions software.

The paradata files describe the process of data collection. They explain how the data was entered, detailing all edits, who and when undertaken them. These files may be large and most conveniently processed using specialized statistical packages, rather than general purpose tools.

Interview diagnostics file

interview__diagnostics file is a special service file, automatically produced with every data export, that contains useful information for assessing the quality of interviews:

• number of invalid entities (invalid answers and errors in static texts),
• number of questions with comments,
• duration of the interview etc.

For a description of available fields and their types, refer to this article.

This is especially useful for user-written monitoring systems that work alongside Survey Solutions and continually monitor the quality of incoming data by periodically exporting the data and building customized reports. Supervisors already had access to all the same information interactively through interview details and interview history pages. The interview statistics API endpoint has been updated to obtain the same information for API consumers.

Audio Audit Files

Survey Solutions has a feature of audio audit: a capability of recording an interview and provide the recording for the audit of the interviewing process. The recording covers all the duration of the interview from the moment it starts to completion, except the time when:

• the tablet is off;
• the user switches to a different app;
• the interview is not active (Interviewer has switched to a different interview or switched to the dashboard);
• the audio recording is done as an answer to an audio question;
• when the interview is completed over the web (CAWI mode).

Audio audit files are exported as part of the binary data. The naming convention used in forming this data archive is as following:

How does Survey Solutions identify records and establish connections between data?

Survey Solutions works with various surveys done in different conditions, so trying to come up with one naming scheme that would be equally suitable for all applications is a challenging task. A particular survey may come up with a numbering like 2 digits of the province code, followed by 3 digits of the district code, followed by the household id of 4 digits. Yet it is not a good identifier in the case where several surveys are conducted on the same server, or there may be repetitive visits, and other twists.

Organization of meta-data storage in the exported data

Survey Solutions export file is a zip-archive with multiple data files corresponding to various levels of data and system-generated accompanying data reflecting the comments, errors, statuses and other technical information about interviews and assignments. When the corresponding setting is activated in the export interface, the data is accompanied with the meta-data reflecting the structure of the questionnaire. The meta-data is saved to the subfolder Questionnaire of the export archive.

The questionnaire subfolder contains the following information:

Merging files of different data levels

Survey Solutions produces export data files separately for each data level. This can be for example households, persons, plots, crops, etc. The process is completely automatic and not configurable by the user.

If you need to combine information from different data levels (for example bring household characteristics to the personal level) you can utilize a statistical package to post-process your data. This is a basic operation supported by most packages, such as SAS, SPSS, Stata, R and others, though the exact terms may differ by package. Note that doing this in spreadsheet applications like Excel introduces unnecessary complications.

Missing values

Survey Solutions recognizes two situations of missingness:

1. the value is missing because the logic of the questionnaire is disabling (skipping) this question. There is nothing that the interviewer can do to enter a value in such a field and in some cases even to see the question and its options.
2. the value is missing despite the question is enabled (not skipped). This could be because of any of the following:

• interviewer hasn’t reached this point in the interview;
• interviewer missed the question, didn’t ask it and hence no answer was recorded;
• none of the options was suitable to accommodate the respondent’s answer and hence the interviewer didn’t record the answer;
• other similar situations.

In the first case the missingness will be reflected in the export data file with a system missing, in the second case special reserved values will be used:

Stata export file format

When Survey Solutions v5.18 is writing an exported data file in Stata format it is producing it according to Stata 14 data file specification, available from StataCorp LLC. 

Stata versions prior to version 14.0 (e.g. 12.0, 13.1, etc) cannot open these files because these older versions of the software are not aware of the newer file format. The message that the user will see on the screen varies, and may look, for example like the following: