T h e g r o w i n g c h a l len g e o f s u r v ey f r a u d a n d h o w t o a ddr e s s i t

A s a r e s e ar c h ag e nc y wo r k i ng a c r o s s m u l t i pl e c o u nt r i e s , ve r t i c a l l y , a n d c a t eg o r i es ( C o ns u m e r and E nt e r pr i s e ) , o ne o f t he m o s t i m po r t ant t hi ng s w e do i s e n s u re w e hav e l e g i ti ma te a n d e n g a g e d p a r ti c i p a n ts i n th e fi e l d w o r k w e c o n d u c t e v e r y d a y .

T he c hal l e ng e o f e ns u r i ng s am pl e i nt e g r i t y and i n t u r n, da t a qu al i t y , is n o t ne w – ev er s i n c e w e al l s t ar t e d u s i ng o nl i ne s am pl e , w e hav e pl a c e d a bar r i e r be t w e e n o u r s e l v e s as r e s e ar c he r s and o u r t ar g e t r e s po nde nt s . W hi l e t hi s m aj o r t e c hno l o g i c a l e nhanc e m e nt e n a b l e d o u r i n d u s tr y to g r o w , r e d u c e c o s t p e r i n te r v i e w a n d i n c r e a s e fi e l d w o r k e ffi c i e n c i e s , i t al s o i nc r e as e d t he r i s k o f e r r o ne o u s r e s po ns e s , “ l o w qu al i t y ” ( i . e. , d i s e n ga ge d ) r e s po nde nt s and fr au d u l e nt par t i c i pant s .

“ O ld” m et h o d s u s ed t o be en o u g h t o i den t i f y qu a li t y i s s u e s

A s an ag e nc y , w e c o n t i nu al l y f o c u s o n t hi ng s wi t hi n o u r c o nt r o l , s u c h as i m pr o v i n g s u r v e y f o r ma ts , g u a r d i n g s u r v e y l e n g th , p i l o t te s ti n g fi e l d w o r k to c a p tu r e a n y d a ta c o l l e c ti o n i s s u e s , and r u nni ng nu m e r o u s da t a c he c k s t o i de nt i f y qu al i t y i s s u e s , i nc l u di ng m o n i t o ri n g s u r v e y c o m pl e t i o n pa t t e r ns and i de nt i f y i ng “ s p e e ds t e r s ” and “ s t r ai g ht - l i n e rs .” W e a l s o w o r k c l o s e l y wi t h pane l c o m pani e s who e m pl o y o t h e r t o o l s , s u c h as c r o s s - c he c k i ng r e s po nde nt i de nt i t y du r i ng pane l r e c r u i t m e nt , r e - v al i da t i n g pane l par t i c i pant s o v e r t i m e , and u s i ng d i g i ta l fi n g e r p r i n ti n g to e n s u r e a s i n g l e I P a d d r e s s c a n n o t b e u s e d m o r e t han o n c e f o r t he s am e s u r v e y . I n s ho r t , t he c o m bi nat i o n o f a r i g o r o u s r e s e ar c he r and c o ns c i e nt i o u s pane l c o mp a n y c o u l d s u ffi c i e n tl y mi ti g a te th e r i s k u p u n ti l r ec en t l y .

W i der i n du s t r y r e co g ni z e s t h e i s s u e o f s u r v ey f r a u d

H o w e v e r , t he r e i s a g r o wi ng pr o bl e m i n o ne a r e a w e hav e f ar l e s s c o nt r o l o v e r – f a ke r e s po nde nt s and s u r v e y fr au d c o m i n g fr o m s u r v e y f ar m s and “ bo t s .” A nd it is n o t j u s t u s fl a g g i n g th i s p r o b l e m, t h e M R S a n n o u n c e d j u s t a f e w d a y s a g o th a t i t w i l l b e w o r k i n g

to g e th e r w i th E S O M A R , T h e I n s i g h ts A s s o c i a ti o n a n d S a mp l e C o n to “ t r a c k a n d a d d r e s s s am pl e fr au d and t he r i s k s i t po s e s t o da t a qu a l i t y .” A s J a n e F r o s t, C h a i r o f th e M R S m e nt i o ne d “ fr au du l e nt ac t i v i t y i s be c o m i n g i nc r e as i ng l y s o phi s t i c a t e d, par t i c u l a r l y i n o n l i n e r e s e a r c h , ” a n d i n th e s a me p r e s s r e l e a s e M e l a n i e C o u r tr i g h t, C E O o f th e I n s i g h ts A s s o c i a t i o n s ai d t he i s s u e s ar e “ g l o bal and pe r s i s te n t a n d mu s t b e a d d r e s s e d .”

B o t s a nd s u r v e y f a rms a r e b e co mi ng m o r e so p h i st i c a t ed

B o t h m y s e l f and m y c o l l e ag u e s , al l o f who m ha v e 2 0 + y e ar s ’ e x pe r i e nc e i n t he i ndu s t r y , h a v e n e v e r s e e n s u c h s e r i o u s s a mp l e q u a l i ty i s s u e s a s w e a r e f a c i n g c u r r e n tl y . T h e p r o b l e m ma n i f e s ts i ts e l f i n c l a s s i c w a y s , s u c h a s i n c o n s i s te n c i e s i n s u r v e y d a ta . H o w e v e r, it is no t a c as e o f r e s po nde nt s s t r ai g ht - l in in g o r t a k in g s ur v e y s q uic k l y – t he bo t s o pe r a t i ng wi t hi n t he s u r v e y f ar m s c an s u bm i t what l o o k l i k e v al i d r e s po ns e pa t t e r ns i n a r e as o nabl e ( i . e. , n ot t oo s h or t ) p e ri o d . T he y c an al s o a ns we r a ll qu e s t i o n f o r m a t s and wr i t e wha t l ook s l i k e , o n fi r s t v i e w , v a l i d r e s p o n s e s to o p e n e n d e d q u e s ti o n s . F o r th e l a tte r o f t h e s e , it is on l y whe n l o o k i ng at t he wi d e r da t a s e t y o u r s e l f whe n y o u c an s t ar t t o s e e s i m i l ar l o o k i ng r e s p o n s e s u s i n g v a r i a n t s o f th e s a me s e t o f w o r d s . T h e s e ar e no t ne c e s s ar i l y pi c k e d u p b y p a n e l q u a l i ty c h e c k s a s th e y l o o k v a l i d o n fi r s t v i e w , s o th e o n u s h a s i n c r e a s e d o n r e s e a r c h e r s to e s ta b l i s h n e w me th o d s to s p o t th e m a n d r e m o v e th e m f r o m th e s a mp l e .

T h e e x t en t o f t h e pr o bl em

T he s u r v e y f ar m s a re wi de s pr e ad e no u g h and s o phi s t i c a t e d e no u g h t o m i m i c l e g i t i m a t e r e s po nde nt s fr o m m u l t i pl e c o u nt r i e s and c an g e ne r a t e (o n e w o u l d a s s u me ), e n o u gh i n t he f o r m o f s u r v e y i n c e n ti v e s to ma k e i t w o r th w h i l e f o r th e p e r p e tr a to r s . F o r s o me o f th e o n l i n e s u r ve y s w e hav e c o ndu c t e d r e c e nt l y wi t h a r an g e o f w e l l - k no wn pane l pr o v i de r s , u p t o 2 0 % o f t he r e s po nde nt s hav e c l e ar l y be e n g e ne r a t e d b y bo t s , a nd it is g e t t i ng har de r and har de r to s p o t th e m.

S o , w h a t c a n w e d o a b o u t i t? W h i l e th e r e a r e w e l c o me o n g o i n g e ffo r ts f r o m th e i n d u s tr y me n ti o n e d a b ove , w e m u st t ak e p r ag m a t i c s t e ps whi l e t he i ndu s t r y ( and t e c hno l o g y ) ca t ch e s u p .

S o lu t i o n 1 : U s e a r a n g e o f qu e s t i o n f o r m a t s , e spec i a l ly t h o s e r e qu i r i n g lo g i c

 W hi l e “ bo t s ” c an ans w e r m o s t qu e s t i o ns i n an o nl i ne s u r v e y i n what a ppe ar s a v al i d way , t he y s t r u g g l e wi t h m o r e c o m pl i c a t e d f o r m a t s , s u c h as “ 1 0 0 % c al c u l a t o r ” and “ c hi p al l o c at i o n ” qu e s t i o ns ( whe r e t he r e s po nd e nt has t o al l o c a t e pr o p o r t i o ns o f a d e fin e d a m o u n t to d i ffe r e n t q u e s ti o n o p ti o n s ), a s w e l l a s “ d r a g a n d d r o p ” a n d “ s l i d e r ” q u e s ti o n f o r m a ts .

 A n addi t i o nal qu al i t y m e as u r e i s t he u s e o f “ r e d he r r i ng ” qu e s t i o ns , w he r e m u l t i pl e ans w e r o p t i o ns ar e pr o v i de d i nc l u di ng a n o b vi ou s a n s w e r to a s i mp l e q u e s ti o n . A l s o , t r i c k qu e s t i o ns i n t he s u r v e y t hat r e qu i r e a l a t e r s u r v e y r e s po ns e t o m a t c h an e ar l i e r o ne c an al s o pr o v e he l pfu l . B o t s c o m pl e t i ng s u r v e y s i n a “ r a ndo m ” f as hi o n ma y mi s s o b v i o u s o r m a tc h i n g a n s w e r s .

S o lu t i o n 2 : C o n si der m o r e r o b u s t s a m pl e s o u r c e s

 A l te r n a ti v e s a mp l e s o u r c e s a n d r e c r u i tme n t m e th o d s c a n o ffe r a n e v e n mo r e e ffe c ti v e s o l u ti o n . P o o r (o r c l e a r l y f r a u d u l e n t) s am pl e s ar e m o r e pr e v al e nt i n o u r e xp e r i e n c e i n B 2 B s u r ve ys ( p os s i b l y be c au s e o f s u r v e y f a r m o p e r a to r s ta r g e ti n g th e mo r e l u c r a ti v e r e w a r d s a v a i l a b l e f o r c o mp l e ti n g th e s e s u r v e y s ). A s s u c h , w e hav e f o u nd u s i ng a P ho ne t o W e b ( P 2 W ) appr o a c h i s a v i abl e s o l u t i o n i n m any c as e s f o r B 2 B s u r v e y s . U s i n g th i s a p p r o a c h , o u r fi e l d te a m co n d u ct s an i ni t i al w av e o f pho ne r e c r u i t m e nt t o t ar g e t an d qu al i f y po t e nt i al s u r v e y par t i c i pant s and o nl y af t e r t he y a r e v a l i d a te d a n d s c r e e n e d i s a l i n k s e n t f o r th e o n l i n e s u r v e y . T h e d i ffe r e n c e i n r e s u l ts v s . c o n v e n ti o n a l o n l i n e B 2 B s a mp l e (w h e r e w e h a v e t ri a l e d bo t h s am pl e s o u r c e s b e f o r e r o l l i n g o u t f u l l fi e l d w o r k ), h a s b e e n s ta r tl i n g – q u i t e li t e r a l ly d a y a n d n i g h t. W h i l e t h e P 2 W m e th o d i s mo r e e x p e n s i v e th a n th e o n l i n e s a mp l e , th e d i ffe r e n c e i n q u a l i ty a n d th e r e f o r e c o n fi d e n c e i n th e r e s u l ts mo r e th a n ma k e s u p fo r th e c o s t d i ffe r e n c e . F a k e r e s p o n d e n ts a n d s p u r i o u s d a ta a r e o f n o u s e t o a n y o n e , r e g a r d l e s s o f t h e c o s t.

L o o k i ng ahe ad, it is e nc o u r ag i ng t o s e e s t e ps b e i ng t ak e n t o ad dr e s s t h e pr o bl e m , i nde e d s o me p a n e l p r o v i d e r s h a v e a l r e a d y s ta r te d to i n te g r a te n e w f r a u d d e te c ti o n to o l s . H o w e v e r , whi l e w e wai t f o r t hi s t o be c o m e m o r e m ai ns t r e am , w e s t r o ng l y adv o c at e u s i ng t he m e as u r e s ab