Investigating the Feasibility of a Factorial Survey in a CATI Björn Andernach Reinhard Schunck - PDF

Description
Investigating the Feasibility of a Factorial Survey in a CATI Björn Andernach Reinhard Schunck SFB 882 Working Paper Series No. 28 June 2014 DFG Research Center (SFB) 882 From Heterogeneities to Inequalities

Please download to get full document.

View again

of 35
All materials on our website are shared by users. If you have any questions about copyright issues, please report us to resolve them. We are always happy to assist you.
Information
Category:

Economy & Finance

Publish on:

Views: 41 | Pages: 35

Extension: PDF | Download: 0

Share
Transcript
Investigating the Feasibility of a Factorial Survey in a CATI Björn Andernach Reinhard Schunck SFB 882 Working Paper Series No. 28 June 2014 DFG Research Center (SFB) 882 From Heterogeneities to Inequalities Björn Andernach and Reinhard Schunck Investigating the Feasibility of a Factorial Survey in a CATI SFB 882 Working Paper Series, No. 28 DFG Research Center (SFB) 882 From Heterogeneities to Inequalities Research Project B3 Bielefeld, June 2014 SFB 882 Working Paper Series General Editors: Martin Diewald and Thomas Faist ISSN This publication has been funded by the German Research Foundation (DFG). SFB 882 Working Papers are refereed scholarly papers. Submissions are reviewed by peers in a two-stage SFB 882 internal and external refereeing process before a final decision on publication is made. The Working Paper Series is a forum for presenting works in progress. Readers should communicate comments on the manuscript directly to the author(s). The papers can be downloaded from the SFB 882 website SFB 882 From Heterogeneities to Inequalities University of Bielefeld Faculty of Sociology PO Box D Bielefeld Germany Phone: +49-(0) or +49-(0) Web: DFG Research Center (SFB) From Heterogeneities to Inequalities Whether fat or thin, male or female, young or old people are different. Alongside their physical features, they also differ in terms of nationality and ethnicity; in their cultural preferences, lifestyles, attitudes, orientations, and philosophies; in their competencies, qualifications, and traits; and in their professions. But how do such heterogeneities lead to social inequalities? What are the social mechanisms that underlie this process? These are the questions pursued by the DFG Research Center (Sonderforschungsbereich (SFB)) From Heterogeneities to Inequalities at Bielefeld University, which was approved by the German Research Foundation (DFG) as SFB 882 on May 25, In the social sciences, research on inequality is dispersed across different research fields such as education, the labor market, equality, migration, health, or gender. One goal of the SFB is to integrate these fields, searching for common mechanisms in the emergence of inequality that can be compiled into a typology. More than fifty senior and junior researchers and the Bielefeld University Library are involved in the SFB. Along with sociologists, it brings together scholars from the Bielefeld University faculties of Business Administration and Economics, Educational Science, Health Science, and Law, as well as from the German Institute for Economic Research (DIW) in Berlin and the University of Erlangen-Nuremberg. In addition to carrying out research, the SFB is concerned to nurture new academic talent, and therefore provides doctoral training in its own integrated Research Training Group. A data infrastructure project has also been launched to archive, prepare, and disseminate the data gathered. Research Project B3 Interactions Between Capabilities in Work and Private Life: A Study of Employees in Different Work Organizations This research project primarily addresses capabilities in working and private life and the interrelations between them. Adapting Sen s approach, capabilities are the ability to achieve one s life goals. The project adopts a comprehensive view that identifies multidimensional states of inequality. Crucial is the recognition that pursuing one s interests in one life domain may even constrain goal attainment in other life domains. The same personal circumstances and employment conditions may be perceived and evaluated differently against the background of heterogeneous life goals. The concept of employment relationships allows us to gain an overview of a wide range of different gratifications and different demands and stresses, against the background of different psychological contracts. On the level of employees, we therefore firstly study the heterogeneity of different employment relationships in companies situated in various business sectors. Secondly, we assess these employees in terms of their embedment in various forms and phases of life. Thus, also the situation and views of a partner will be considered. In a next step this project examines how heterogeneities (e.g. gender, age, life style preferences, education) become social inequalities with a particular focus on the role of the organizational context. As possible mechanisms different individual interests within companies and private bonds being negotiated in different ways are investigated. Health also plays a role in these interdependencies influencing the prospects for successful multiple engagement in both life domains. It is a hard indicator of maladjustment. In this project detailed studies of employees and characteristics of their companies are carried out. Companies play a dual role, first as negotiation partners and second as opportunity structures. Various actors within the companies and companies institutional and sector-specific context are considered. Proceeding from a sample of 100 work organizations, an extended linked employer-employee design will be used to study an average of 65 employees in each organization. If employees have life partners, they will also be surveyed with a short version of the instrument. By combining these data with information from the same employees and their companies from the German Institute for Employment Research (IAB), we can achieve a unique density of information for large case numbers. The longitudinal design initiated during the first funding period allows distinguishing causal effects more clearly and to adequately study processes of discrimination and self-selection. The Authors Björn Andernach is a member of the project Interactions between Capabilities in Work and Private Life: A Study of Employees in Different Work Organizations in the Collaborative Research Center 882 (SFB 882) at Bielefeld University. His research interests are social inequality, social classes, social closure at the level of occupations, interdependencies of work and family life and life course research. Contact: Reinhard Schunck is an assistant professor at the Faculty of Sociology, Bielefeld University, and principal investigator of the research project B3 Interactions Between Capabilities in Work and Private Life: A Study of Employees in Different Work Organizations in the Collaborative Research Center 882 (SFB 882). He received his PhD at the Bremen International Graduate School of Social Sciences (BIGSSS). His research interests span social stratification and inequality, the life course, health, migration and integration, and quantitative methods. Among his recent publications are Transnational Activities and Immigrant Integration in Germany. Concurrent or Competitive Processes? (2014) Springer International Publishing and Within- and Between-Estimates in Random Effects Models. Advantages and Drawbacks of Correlated Random Effects and Hybrid Models (2013) in: The Stata Journal, 13(1): Contact: Investigating the Feasibility of a Factorial Survey in a CATI Björn Andernach* Reinhard Schunck* * Collaborative Research Center 882 From Heterogeneities to Inequalities, Bielefeld University Abstract Factorial surveys are a common method for studying social norms, attitudes, and hypothetical decision situations in the social sciences. Although they are usually applied in interview settings which allow for a visual representation of the factorial survey, they are also regularly used in computer assisted telephone interviews (CATI). However, we know little about the applicability of a factorial survey in an interview mode that does not allow for a visual presentation of the factorial survey. This paper investigates potential problems that may arise in implementing a factorial survey in a CATI by investigating how respondents of different age and educational backgrounds deal with factorial surveys of different degrees of complexity. To asses potential problems we rely on respondents self-reported response difficulties, a measure of response latency, and response consistency. We do not find that older respondents are experiencing or reporting more difficulties in processing the factorial survey. Respondents with higher levels education appear to produce more consistent responses. Keywords: vignettes; factorial survey; CATI; telephone interviews; response latency; response consistency; age; education Corresponding author: Björn Andenach, 1 Introduction Studying social norms and belief systems thru factorial surveys (also called factorial vignettes) has become increasingly popular in the field of social science (Wallander 2009). Factorial surveys confront respondents with a description of a set of hypothetical situations or objects (vignette decks), in which the attributes (dimensions) vary in their value (level), and ask respondents to evaluate them or to form a (hypothetical) decision on their basis (Beck and Opp 2001; Jasso and Opp 1997; Rossi and Nock 1982). The aim of a factorial survey is to identify the attributes relative importance in the evaluation or decision (Sauer et al. 2011). The main advantage factorial surveys have over other survey instruments is their multifactorial, quasi-experimental character (Rossi and Anderson 1982). The factorial survey allows to separately asses the relative importance of attributes that may be highly confounded in reality (Auspurg et al. 2009). The quasi-experimental design obtained by using factorial surveys provides the advantage of internal validity, achieved through a randomized, multifaceted design. This is particularly advantageous when combined with a representative (general population) survey although external validity may be limited due to the hypothetical nature of the factorial survey (Rossi and Anderson 1982; Sniderman and Grob 1996; Atzemüller and Steiner 2010). Because of their multifactorial design (Rossi and Anderson 1982), factorial surveys may comprise complex decision or rating tasks and may put greater cognitive burden on respondents (Sauer et al. 2011) than simpler instruments, such as direct questions. Since visual presentation, makes information processing easier (Cohen, Horowitz and Wolfe 2009), factorial surveys are mostly implemented in interview modes that allow for the visual presentation of the vignettes, such as computer assisted personal interviews (CAPI), computer assisted self-interviews (CASI), computer assisted web interviews (CAWI), or paper and pencil interviews (PAPI). To our knowledge, previous research on potential problems related to factorial surveys (e.g. Rossi and Anderson 1982; Auspurg, Hinz and Liebig 2009; Sauer et al. 2011) is concerned only with interview modes that allow for the visual representation of vignettes. However, much data in survey research is collected through computer assisted telephone interviews (CATI). CATI is not only cheaper than other interviewing modes, in particular CAPI, but it may also be better suited to collect representative general population samples through, for instance, employing randomdigit dialing (Gabler and Häder 2002; Link et al. 2008). Even though factorial surveys are regularly implemented in CATI (e.g. Hunter et al. 2009; Jorm, Wright and Morgan 2007; Wright, Jorm and Mackinnon 2012; Sikorski et al. 2012; Tindale et al. 2011; Denton et al. 2010; Lubman et al. 2007; Sorenson 2006 and Cotton et al. 2006), little is known about potential problems that 2 may arise due to this interview mode. 1 CATI differs from the above modes crucially, because it does not allow for the visual representation of vignettes and it is therefore questionable whether previous research on the feasibility of factorial surveys in other interview modes can be transferred to CATI. This paper addresses potential problems that may arise in implementing a factorial survey in a CATI by investigating how respondents of different age and educational backgrounds deal with factorial surveys of different degrees of complexity. The data for this study comes from a telephone survey on work-life-balance in Germany. Complexity of the factorial survey was modified by two split ballots: Respondents were randomly assigned to conditions that differed in the number of dimensions (4, 5, or 6) and the number of vignettes per deck (8 or 12). To assess potential problems in processing the factorial survey, we analyzed three types of measures: subjectively reported response difficulty, response time, and response consistency. Cognitive Load in Factorial Surveys Factorial surveys are a well-established method in opinion and attitude surveys (Wallander 2009). Although factorial surveys may be more realistic, due to their multifactorial design (Rossi and Anderson 1982), the combination of several different dimensions within one fictive description can also be seen as complex, putting high cognitive burden on respondents (Sauer et al. 2011). The complexity of factorial surveys depends on the number of dimensions that a vignette consists of and the number of vignettes respondents have to process (Rossi and Anderson 1982; Auspurg, Hinz and Liebig 2009; Auspurg et al. 2009; Sauer et al. 2011). Previous research on the effects of the complexity of factorial surveys in CAPI, CASI, and PAPI indicates that no difficulties arise in processing factorial surveys, if the number of dimensions per vignette is below 12 and the number of vignettes does not exceed 20 (Sauer et al. 2011). Yet, CATI studies differ from other interview modes in that they do not allow for a visual presentation of the vignettes. A visual representation gives respondents the possibility to reread the vignettes and, in most cases, full control over the progress of the factorial survey. If the vignettes are presented through speech this is likely to increase in the cognitive load and the burden on the working memory, resulting in fewer items (chunk of information) (Miller 1956) 1 Although some studies report the implementation of pilot studies to test whether the research design is suitable for telephone interviews (e.g. Hunter et al. 2009), the results of these pilot studies are not discussed. 3 that can be processed. 2 Moreover, auditory memory appears to be inferior to visual memory (Cohen et al. 2009). To get respondents to produce an informed reaction (e.g. a judgment or a rating), it is however a basic necessity that they understand the pieces of information provided by the vignette, recall them, and integrate them into a coherent representation of what the vignette intends to portray. 3 The demands on respondents working memory (Daneman and Merikle 1996), which dynamically determines their ability to comprehend, store, and recall information and thus constitutes the basis for making evaluations and decisions based on these information (Cowan 2005), increases with (vignette) information complexity and may affect the outcomes (Swait and Adamowicz 2001). Current research on working memory capacity indicates that people may be able to process only about three to five pieces of information (Mathy and Feldman 2012; Li et al. 2013; Cowan 2012 et al.; Cowan 2001). Generally, the higher the number of pieces of information that have to be processed, the more difficult will it become for respondents to form an informed evaluation of the depicted situation. We therefore assume that the higher the number of dimensions, the more difficult will it be to process the factorial survey (H1). The ability to process information also depends on cognitive ability. People with lower cognitive ability may have difficulties integrating successively encountered information into a coherent representation (Daneman and Merikle 1996). Since cognitive capacity relates to schooling (Brody 1997; Rindermann and Neubauer 2004; Falch and Sandgren Massih 2011), lower levels of education may be associated with increased difficulties to process information. We therefore assume that less educated respondents will exhibit more difficulties in processing the factorial survey (H2). The individual ability to process the factorial survey may, moreover, depend on the respondent s age. First, cognitive capacity declines with age (Park 1999; Salthouse 1996). Second, aging may also impair hearing and speech understanding (Murphy, Daneman and Schneider 2006, Pichora-Fuller and Souza 2003), making it difficult to encode verbally presented the information. This problem is particularly pertinent in CATI interviews. We therefore assume that older respondents will exhibit more difficulties in processing the factorial survey (H3). 2 It should be noted, however, that the interviewers were instructed to re-read the vignettes if necessary. 3 It may also be argued that there is a central capacity limit, i.e. that the limit for visual and auditory information does not differ (Cowan 2000). While this may be the case, it is unnecessary to recall information when one is able to look at the vignette while performing the rating task. 4 Data and Methods To test these hypotheses data was collected on 331 randomly chosen respondents in a CATI through list-directed dialing. The universe comprises all households in Germany who had their telephone number listed in a telephone book. Within households, eligible respondents were identified through the birthday method (Salmon and Nichols 1983; Rizzo, Brick and Park 2004). Because this study was part of pretest for a larger research project, eligibility also depended on employment status so that all respondents in the sample are working. Moreover, since in this pretest the response rate was not of primary concern, it turned out low (AAPOR Response Rate 1 = 5.8%, AAPOR 2010). After listwise deletion of missing values and exclusion of respondents with extreme values on response time (see variables section) and those who did not complete all vignettes the finale sample comprises 291 respondents. Descriptive statistics on the sample are shown in table 3. Table 1: Vignette Dimensions and Levels Dimensions Earnings (gross, monthly, in Euros) Levels 1000, 2000, 3000, 4000, 5000 Job (prestige according to MPS) Unskilled worker (MPS = 31) Train conductor (MPS = 50.1) Retailer / shopkeeper (MPS = 78) Architect (MPS = 111.7) Doctor (MPS = 191.3) Marital status Partner / no partner Children 0, 1, 2, 3 Health Very good, good, satisfactory, bad, very bad Close friends 0, 2, 4, 6, 8 Note: The dimensions health and close friends were only included 68.28% and 32.63% of the cases respectively. Job prestige is operationalized through the magnitude prestige scale (Christoph 2005; Wegner 1985) Vignettes The vignettes described hypothetical constellations of work and family variables and asked the respondents to rate those constellations according to their hypothetical life satisfaction on an eleven point scale. The vignette dimensions and their values are presented in table 1. A sample vignette would read Imagine you are a trained retail salesman, you have a partner, and 2 children; your health is excellent, you earn 2000 euro pre-tax and you have 2 close friends. How satisfied would you be? 5 Vignette investigation plan The study implemented six split ballots, following the experimental design shown in table 2. The product of the number of levels per dimension leads to a vignette universe of 5,000 possible vignettes. In the first step, 60 decks, with 12 vignettes per deck, were drawn thru stratified random sampling (stratification dimension: earnings) out of the vignette universe. 4 In the second step, either one (number of close friends) or two the dimensions (number of close friends and health) were deleted in 20 randomly chosen decks, leaving 20 decks with 4, 5, and 6 dimensions respectively. In a third step, 30 decks were randomly chosen out of the full 60 decks and 4 vignettes were randomly deleted within each of the chosen decks. This procedure lead to 20 decks with 4, 5, or 6 dimensions each and to 30 decks with either 8 or 12 vignettes. The
Related Search
Similar documents
View more...
We Need Your Support
Thank you for visiting our website and your interest in our free products and services. We are nonprofit website to share and download documents. To the running of this website, we need your help to support us.

Thanks to everyone for your continued support.

No, Thanks