2. SecurePhone : a mobile phone with biometric authentication and e-signature support for dealing secure transactions on the fly IST-2002-506883Secure contracts signed by mobile Phone

3. A Talking Elevator, WS2006 UdS, SecurePhone 2 Presentation Outline SecurePhone concept and use Project aim 1: secure exchange Project aim 2: multi-modal Biometric Recogniser Performance on PDA Implementation constraints and performance on SIMcard PDA selection Conclusion

4. A Talking Elevator, WS2006 UdS, SecurePhone 3 What is a SecurePhone? A SecurePhone is a new mobile communications device that enables users to exchange text/audio documents during a phone call to draw legally valid transactions. It combines biometric identity verification with e-signing in a system using front-edge technologies (wireless networking, double-key cryptography). Biometric recognizer enables strong authentication by comparing live biometric features with models trained on enrollment data which were previously stored on the PDA or on the device’s SIM card. Biometric authentication grants access to built-in e-signing facilities, also integrated on the PDA/SIM.

5. A Talking Elevator, WS2006 UdS, SecurePhone 4 User interface User Interface implementation includes: biometric data management modules ̶capture and pre-processing of enrollment or “live” test data on PDA ̶training of GMM models (“templates”) for identity verification on PC (or PDA) ̶Verification on PDA (now)/SIM (underway) SharedDoc module ̶interactive modification and exchange of a text file, the “e-contract” ̶exchange of audio files

6. A Talking Elevator, WS2006 UdS, SecurePhone 5 Project aim 1: secure exchange Secure PKI (personal key infrastructure) Deal secure m-contracts during a mobile phone call secure: private key stored on SIM card dependable: multi-modal: voice, face, signature user-friendly: familiar, intuitive, non-intrusive flexible: legally binding text/audio transactions dynamic: mobile (anytime, anywhere)

7. A Talking Elevator, WS2006 UdS, SecurePhone 6 PK technology in SecurePhone Public key encryption technology is used for e-signature, i.e. to enforce data integrity and non-repudiation; in P2B, public-key technology is used for authentication over networks and/or form e-signing. SIM card is used as a tamper-proof device for e-signing and storing the user’s e-signature private key (“strong signature” if the corresponding digital certificate is e-signed by a valid CA). Standard e-signature certificates and procedures are used for certificate verification and management, so documents e-signed by means of the SecurePhone have the same legal validity as documents e-signed by other means.

8. A Talking Elevator, WS2006 UdS, SecurePhone 7 Biometric verification architectures Biometric templates can be stored ̶on the SIM card (ToC) ̶on the PDA/host (ToH) ̶on a Trusted Third Party (TTP) server (ToS) Matching/verification can be performed ̶by an applet running on the SIM card (MoC) ̶by an application running on the PDA/host (MoH) ̶by an application running on a TTP server (MoS) Only ToC + MoC meets requirements on ̶security ̶privacy and user acceptance

9. A Talking Elevator, WS2006 UdS, SecurePhone 8 Person-to-Person (P2P) user scenario During a phone call, two SecurePhone end users (actors) agree on drawing a distance contract by setting up a direct m-transaction One actor (proposer) sends an e-document (e-contract, i.e. text/audio file) to the other actor (endorser) In case of text files, the e-contract can be interactively modified and transmitted back and forth until a formal agreement on its contents is reached To finalize the m-transaction, the endorser e-signs the e-contract and sends it to the proposer as evidence of formal acceptance of the terms contained therein Depending on the e-contract type, the proposer may also be requested to e-sign the e-contract

10. A Talking Elevator, WS2006 UdS, SecurePhone 9 Person-to-Business (P2B) scenario Scenario compatible with SecurePhone architecture, but not implemented in the project SecurePhone user accesses the server of service provider using his browser Server sets up an SSL/TLS communication channel with strong client authentication Browser triggers local authentication, which releases private key The e-signing of web-based forms is accepted by service provider as evidence of agreed e-contracts

11. A Talking Elevator, WS2006 UdS, SecurePhone 10 In both P2P and P2B, the user (i.e. a host application) needs to locally authenticate in order to “unlock” cryptographic functions and access the private key securely stored on the SIM card PIN- or password-based authentication is admissible yet weak and unsatisfactory for security-critical applications (e-commerce, e-health- e-government) Local authentication strengthened in order to increase user’s trust in the system by combining ̶WYK:a token that only the user knows (signature) ̶WYH:a token that only the user holds (PDA with SIM card) ̶WYA:biometric identity Project aim 2: biometric verification

12. A Talking Elevator, WS2006 UdS, SecurePhone 11 User verification system User requests PDA to verify their identity PDA requests user to read prompt (face in box) sign signature Feature processing applied to each modality [silence removal, histogram equalisation, MFCC or Haar wavelets, online CMS, delta features, etc.] for each modality S(i)=log p(Xi|C)-log p(Xi|I) if S(i) < θ(i) for any (i) please repeat else fused-score = log p(S|C) - log p(S|I) if fused-score > φ user accepted else user rejected Press to start/stop speaking 7 9 8 5 1 start/stop

13. A Talking Elevator, WS2006 UdS, SecurePhone 12 Multi-modal biometric verification preprocessing modelling preprocessing facevoicesignature accept user release private key reject user fusion client & impostor joint-score models user profile

14. A Talking Elevator, WS2006 UdS, SecurePhone 13 Voice verification (SU / GET ENST) Fixed 5-digits prompt – conceptually neutral, easily extendable, requires few Gaussians 22 KHz sampling Online energy based non-speech frame removal MFCCs with online CMS and first-order time difference features – slow to compute, but fixed point faster than floating point Features modelled by 100-Gaussian GMM pdf, with UBM for model initialisation and score normalisation Training on data from 2 indoor and 2 outdoor recordings from one session, testing on similar data from another session

15. A Talking Elevator, WS2006 UdS, SecurePhone 14 Face verification (BU) Static face recognition – 10 grey-scale images selected at random, 160x192 pixels Histogram equalisation and z-score normalisation of features Haar low-low-4 (or low-high) wavelet features – f ast to compute Features modelled by only 4 Gaussian GMM pdf – UBM used for model initialisation and score normalisation Training on data from 2 indoor and 2 outdoor recordings from one session, testing on similar data from another session

16. A Talking Elevator, WS2006 UdS, SecurePhone 15 Signature verification (GET INT) Shift normalisation, but no rotation or scaling 2D coordinates (100 Hz) augmented by time difference features, curvature, etc. – total 19 features Note:no pressure or angles available, since obtained from PDA’s touch screen, not from writing pad Fast to compute Features modelled by 100 Gaussian GMM pdf – UBM used for model initialisation and score normalisation Training and testing on data from one session

17. Fusion (GET INT) For each modality S(i) = log p(Xi|C) - log p(Xi|I) LLR score fusion was tested by: Optimal linear weighted sum: Fused-score = sum over i of w(i) * S(i) GMM scores modelling, i.e. modelling both client and impostor joint score pdf’s by diagonal covariance GMMs: Fused-score = log p(S|C) - log p(S|I)

18. PDAtabase After initial development with many databases, CSLU/BANCA- like database recorded on Qtek2020 PDA for realistic conditions (sensors, environment) 60 English subjects: 24 for UBM, 18 for g1, 18 for g2. Accept/reject threshold optimised on g1, then evaluated on g2, vice versa Video (voice + face): 6 x 5-digit, 10-digit and phrase prompts; 2 sessions, with 2 inside and 2 outside recordings per session Signatures in one session, 20 expert impostorisations for each Virtual couplings of audio-visual with signature data (independent) Automatic test script allows to test many possible configuration User just provides executables for feature modelling, scores generation and scores fusion

19. A Talking Elevator, WS2006 UdS, SecurePhone 18 Performance on PDA DET curves for prompts T1 (5 digits, left), T2 (10 digits, middle) and T3 (short phrases, right) in PDAtabase

20. A Talking Elevator, WS2006 UdS, SecurePhone 19 Performance on PDA Fusion results (% WER, FAR and FRR) for the best fusion method (Min-Max + GMM), for the 3 prompt types in the PDAtabase EER R=1 WER (FAR/FRR) R=0.1 WER (FAR/FRR) R=10 WER (FAR/FRR) T1 2.392.40 (1.57/3.24)1.87 (4.97/1.56)1.02 (0.43/6.95) T2 1.541.60 (0.89/3.32)1.37 (3.05/1.20)0.63 (0.25/4.37) T3 2.302.37 (1.61/3.14)2.03 (4.54/1.78)0.92 (0.38/6.34)

21. A Talking Elevator, WS2006 UdS, SecurePhone 20 Implementation constraints PDA main processor is much slower than PC, but does speech preprocessing in real time for 22 kHz signals Note: speech signal taken directly from mic, therefore > 8 kHz Only data on the SIM card is secure, so all biometric models must be stored and processed on the SIM, which has very limited computational resources SIM model storage limited to 40 K: text-dependent prompts Note: text-independent prompts or varied text-dependent prompts are more secure, but would require 200-400 K GMM based verification is well suited to integer computation Enrolment can use only one short indoor session

22. A Talking Elevator, WS2006 UdS, SecurePhone 21 Performance on SIMcard SIM processor very slow: single verification takes 53 minutes! Most time goes to voice and signature processing: these use a large number of frames and models with a lot of Gaussians. Not acceptable for any practical application. Drastic measures needed: global processing. By using means and standard deviations across all parameters for all frames in the utterance/signature, the number of frames is reduced to one. Since the data are much simpler, only a few Gaussian mixtures are needed for modelling Single verification now under 1 second, but performance for T1 is now 10.5% EER.

23. A Talking Elevator, WS2006 UdS, SecurePhone 22 Remarks on PDA selection No suitable off-the-shelf products at moment of selection fulfilled all SecurePhone requirements Limitations of Qtek 2020: –Class B GPRS  impossible to transmit voice and data simultaneously –Camera is on the rear  difficulties with video acquisition and text prompt reading –Proprietary video SDK, not freely available  problems with low-level raw image data recording Now available: Qtek 9000 solves first two problems, solution to last problem may be usuable with Qtek 9000!

24. A Talking Elevator, WS2006 UdS, SecurePhone 23 Conclusion The SecurePhone combines secure communication with user authentication is user-friendly and respects privacy does not require special hardware enables m-business with legal validity can easily be extended to other applications delivers proof-of-concept has very high performance on PDA, performance on SIM must still be improved.

25. A Talking Elevator, WS2006 UdS, SecurePhone 24 Secure contracts signed by mobile Phone IST-2002-506883 http://www.secure-phone.info