FeatureTranscriptionAnalysisAgreementOriginalSTP < WS06 < TWiki

Skip to content.

Find topic

WS06 topics

Members' area

Tools

Help!

-- KarenLivescu - 15 Dec 2005

STP results (percent agreement)

Intra-Switchboard Transcription Project (STP) agreement (Colleen vs. Coulston): n = 18

feature	timeInAgreement	timeNotInAgreement	obsAgreement	expAgreement	Kappa
dg1	92.2	22.0	0.808	0.247	0.744
dg2	105.5	8.7	0.924	0.697	0.750
glo	100.8	13.3	0.884	0.511	0.762
nas	106.3	7.8	0.932	0.810	0.640
pl1	93.6	20.5	0.820	0.219	0.770
pl2	105.5	8.7	0.924	0.697	0.750
rd	102.9	11.2	0.902	0.807	0.492
vow	87.3	26.8	0.765	0.335	0.647
TOTAL	794.1	118.9	0.870	0.540	0.717

feature	sed	obsAgreement	expAgreement	kappa
dg1	0.1659	0.8341	0.2110	0.7898
dg2	0.2912	0.7088	0.3175	0.5733
glo	0.1423	0.8450	0.3783	0.7507
nas	0.1250	0.8750	0.4276	0.7816
pl1	0.1493	0.8507	0.1774	0.8185
pl2	0.2912	0.7088	0.3175	0.5733
rd	0.2197	0.6177	0.2615	0.4824
vow	0.2495	0.7505	0.1895	0.6921
total	0.1881	0.7966	0.2404	0.7322

STP final transcriptions vs. our transcribers

Hybrid Phone/Feature - LL: n = 4

feature	timeInAgreement	timeNotInAgreement	obsAgreement	expAgreement	Kappa
dg1	9.8	2.5	0.798	0.249	0.731
dg2	11.5	0.8	0.937	0.678	0.804
glo	11.5	0.8	0.934	0.528	0.860
nas	11.7	0.6	0.950	0.878	0.589
pl1	10.1	2.2	0.825	0.233	0.771
pl2	11.5	0.8	0.937	0.678	0.806
rd	10.8	1.5	0.875	0.720	0.556
vow	9.4	2.9	0.762	0.328	0.646
TOTAL	86.3	12.1	0.877	0.537	0.736

feature	sed	obsAgreement	expAgreement	kappa
dg1	0.2105	0.7895	0.1783	0.7438
dg2	0.3000	0.7000	0.2878	0.5788
glo	0.2321	0.7768	0.3307	0.6665
nas	0.0000	1.0000	0.5118	1.0000
pl1	0.1711	0.8289	0.1701	0.7939
pl2	0.2759	0.7241	0.3032	0.6041
rd	0.3833	0.6250	0.2753	0.4826
vow	0.2707	0.7293	0.1989	0.6621
total	0.2298	0.7717	0.2261	0.7050

Hybrid Phone/Feature - XC: n = 4

feature	timeInAgreement	timeNotInAgreement	obsAgreement	expAgreement	Kappa
dg1	10.1	2.2	0.824	0.255	0.763
dg2	11.7	0.6	0.948	0.674	0.840
glo	11.6	0.7	0.941	0.523	0.876
nas	11.7	0.6	0.954	0.870	0.648
pl1	10.4	1.9	0.847	0.240	0.798
pl2	11.7	0.6	0.948	0.674	0.840
rd	10.9	1.4	0.883	0.708	0.600
vow	9.5	2.8	0.769	0.335	0.653
TOTAL	87.4	10.9	0.889	0.535	0.762

feature	sed	obsAgreement	expAgreement	kappa
dg1	0.2114	0.7886	0.1793	0.7424
dg2	0.1852	0.8148	0.3676	0.7072
glo	0.2712	0.7458	0.3045	0.6344
nas	0.0714	0.9286	0.4745	0.8641
pl1	0.1842	0.8158	0.1789	0.7757
pl2	0.1852	0.8148	0.3649	0.7084
rd	0.3167	0.7083	0.3178	0.5725
vow	0.3286	0.6714	0.1845	0.5971
total	0.2365	0.7672	0.2313	0.6972

All-Feature - LL: n = 5

feature	timeInAgreement	timeNotInAgreement	obsAgreement	expAgreement	Kappa
dg1	10.4	2.9	0.779	0.242	0.709
dg2	11.5	1.8	0.866	0.667	0.597
glo	10.4	2.9	0.785	0.424	0.627
nas	12.3	1.0	0.927	0.771	0.682
pl1	10.6	2.7	0.799	0.214	0.744
pl2	11.5	1.8	0.865	0.666	0.596
rd	12.2	1.1	0.916	0.882	0.286
vow	10.0	3.3	0.754	0.335	0.631
TOTAL	89.0	17.4	0.837	0.525	0.656

feature	sed	obsAgreement	expAgreement	kappa
dg1	0.1582	0.8418	0.1952	0.8034
dg2	0.5143	0.4857	0.2229	0.3382
glo	0.3053	0.6947	0.2755	0.5787
nas	0.1351	0.8649	0.4207	0.7667
pl1	0.1667	0.8333	0.1453	0.8050
pl2	0.5143	0.4857	0.2196	0.3410
rd	0.6346	0.4038	0.1579	0.2921
vow	0.2537	0.7463	0.1794	0.6908
total	0.2705	0.7325	0.2052	0.6634

All-Feature - XC: n = 5

feature	timeInAgreement	timeNotInAgreement	obsAgreement	expAgreement	Kappa
dg1	10.2	3.1	0.765	0.249	0.687
dg2	11.6	1.7	0.870	0.667	0.609
glo	10.8	2.5	0.810	0.434	0.664
nas	12.5	0.8	0.942	0.782	0.734
pl1	10.5	2.8	0.788	0.222	0.728
pl2	11.6	1.7	0.870	0.667	0.609
rd	12.0	1.3	0.901	0.850	0.341
vow	10.3	3.0	0.774	0.334	0.660
TOTAL	89.4	17.0	0.840	0.526	0.663

feature	sed	obsAgreement	expAgreement	kappa
dg1	0.1975	0.8025	0.1996	0.7533
dg2	0.5143	0.4857	0.2106	0.3485
glo	0.2667	0.7333	0.2968	0.6208
nas	0.0811	0.9189	0.4456	0.8538
pl1	0.1866	0.8134	0.1577	0.7785
pl2	0.5143	0.4857	0.2090	0.3498
rd	0.5385	0.5288	0.2067	0.4061
vow	0.2556	0.7444	0.1724	0.6911
total	0.2689	0.7363	0.2140	0.6644