NO2ID

From: David Moss
Sent: 16 February 2011 09:42
To: Mark Lockie
Subject: Drowning in a sea of false positives

http://www.planetbiometrics.com/ : this independent portal is put together by a savvy team of industry-leading journalists with more than 20 years of biometric reporting experience

Dear Mark

I have submitted a review of the article 'India releases biometric enrolment report'. The article was published on Planet Biometrics on 16 December 2010.

Could you explain please why the review is not being displayed on the website?

Thank you, in advance.

For convenience, a copy of the review is appended below.

Yours
David Moss

In the Conclusion section of their proof of concept report (PoC), UIDAI say: “the biometric accuracy levels necessary for deduplication of all residents of India are achievable”.

This follows their claim in the Results section of the report that “we can be confident that biometric matching can be used on a wider scale to realize the goal of creating unique identities”.

In fact, nothing in the report suggests that it would be feasible to prove that each electronic identity on UIDAI’s population register is unique. Not with a billion people on the register. Far from it, we can be confident from the figures quoted in the PoC report that deduplication could never be achieved.


It just takes a simple two-step argument.

Step 1 – uniqueness
If UIDAI are to prove that each electronic identity is unique, then each set of biometrics must be compared to every other set of biometrics and shown to be different from every other set of biometrics. UIDAI know that. As they say in the Results section: “the matching analysis was done on two sets of 20,000 biometrics, for a total of 40,000. However, the number of comparisons was several orders of magnitude more than 40,000, since each set of fingerprints would be matched against every other set of fingerprints in the data set”.

How many combinations of 2 can be chosen from 40,000? Answer, 40,000 X 39,999 / 2 = 799,980,000. UIDAI are right. 40,000 is a number of the order of 10 to the power 4, whereas the number of comparisons which have to be made is of the much higher order of 10 to the power 9.

The population of India is not 40,000. It is more like 1.2 billion or 1.2 X 10 to the power 9. In which case, the number of comparisons between biometrics that would need to be made to prove uniqueness is 7.2 X 10 to the power 17.

Step 2 – false positives
In a perfect world, those 7.2 X 10 to the power 17 comparisons could be performed by computer and it could be proved thereby that there are no duplicates, i.e. each electronic identity is unique.

In the real world, problems arise. UIDAI say quite rightly that we must expect the occasional false positive. That is, on occasion, it will look as though two people have the same biometrics.

There may be all sorts of reasons for that. The equipment used may not be entirely reliable. An exhausted UIDAI agent may by mistake register Mr Clark’s biometrics against Mr Smith’s name. Mr Clark may have naughtily enrolled twice, once in his real name and once as Mr Smith. Or, conceivably, Mr Clark and Mr Smith may be genuinely two different people who happen to have the same biometrics.

When a false positive arises, it has to be investigated by a team of human beings. It can’t be resolved by computer.

How many false positives do we expect? In the Results section of their report, UIDAI say “we will look at the point where the [false positive identification rate] (i.e. the possibility that a person is mistaken to be a different person) is 0.0025 %”. That is, they expect to get 2.5 false positives on average for every 100,000 comparisons.

Given that we have to make 7.2 X 10 to the power 17 comparisons, how many false positives do we expect? Answer: 1.8 X 10 to the power 13. That’s 18,000,000,000,000 false positives for people to investigate and resolve.


And that’s the end of the argument. To prove uniqueness, every single Indian would have to investigate and resolve 15,000 false positives. By the time they had finished, many of them would be dead, many more Indians would have been born, and the process would have to start again. Using UIDAI’s own figures, we can be confident that the proof of uniqueness is not achievable.

This problem was identified by Professor John Daugman, the man who invented the technology of scanning people’s irises to determine a unique biometric. The problem is referred to as “drowning in a sea of false positives”.

UIDAI cannot possibly deliver what they promise. Their own figures prove it. If India is relying on unique identification, then India has a serious problem.

The biometric matching analysis of 40,000 people showed that the accuracy levels achieved using both iris and ten fingerprints were more than an order of magnitude better compared to using either of the two individually.

Happy to inform that this query has now been answered in high detail by one of our readers.

I hope it makes sense and, if the reasoning is sound, demonstrates that India shouldn't be drowning in a sea of false positives.

See the comment section at the end of this link:

http://www.planetbiometrics.com/article-details/i/417/

1 According to Murali Chirala, “Mr. David Moss makes two fundamental errors in ...” and “Mr. Moss confuses FPIR ... with FMR ...” and “Mr. Moss has misinterpreted FPIR and FNIR for FMR and FNMR ...” and there is “... a massive discrepancy in Mr. Moss’ calculations” and “Mr. Moss incorrectly assumes that ...”

2 He’s pretty stupid, this man Moss.

3 We can safely ignore him, he’s irrelevant.

4 Even so, the question remains, for an intelligent third party, are all the records on the Central ID Repository (CIDR) being built by UIDAI biometrically unique? Are they? Or aren’t they? Yes? Or no?

5 Let’s call this intelligent third party “Churali Mirala”, or “CM” for short. CM wants to understand exactly what Murali Chirala is saying, because this question of uniqueness is important. After all, if the Unique Identification Authority of India isn’t offering unique identification, what is it offering?

6 CM takes a good look at Murali Chirala’s second paragraph, where there is talk of “two different galleries”, each with 20,000 records.

7 A gallery here, CM assumes, is a mini CIDR, a set or database of biometric records.

8 What is in the records, CM asks? In UIDAI’s proof of concept trial, each record identified one Indian using 12 items, viz. ten fingerprints and two irisprints. Since that is what we are talking about, UIDAI’s proof of concept trial report, CM assumes that by a “record”, Murali Chirala means 12 fields/columns containing ten fingerprints and two irisprints.

9 Murali Chirala talks of “40,000 searches” on each gallery. Why 40,000? Murali Chirala has a rather elliptical way of writing. We have to “unpack” his words, so to speak. Where did 40,000 come from? CM makes a guess. He assumes that the ten fingerprints are treated as one search term/probe and the two irisprints are treated as a second probe. (CM has been doing some background reading and understands that when you do a matching exercise in the academic field of biometrics you “probe” the “gallery”, which is also sometimes called the “background”.) Good. So that makes two searches for each Indian in the gallery. 40,000 searches in all. CM would like Murali Chirala to confirm that this guess is correct. He doesn’t want to be thought to be as stupid as David Moss. But if the guess isn’t correct, then CM can’t understand what the elliptical Murali Chirala is talking about with his 40,000 searches.

10 And what are these searches, CM asks? There’s not much point searching a gallery against itself, CM thinks, because then the false negative identification rate would have to be zero. And it isn’t. Then CM remembers that the participants in the proof of concept trial had to make two visits to be registered. CM assumes that the gallery was built from the biometrics registered at the first visit and that it is the biometrics registered at the second visit which are being used as the probes. Again, CM would like Murali Chirala to confirm that this guess is correct. He still doesn’t want to be thought to be as stupid as David Moss. But if the guess isn’t correct, then CM still can’t understand what the elliptical Murali Chirala is talking about with his 40,000 searches.

11 Obviously Murali Chirala’s answers would be wasted on David Moss. No question. But Murali Chirala still nevertheless owes answers to any intelligent third party who is interested to know whether records on the CIDR are unique.

12 More questions for Murali Chirala. 60,000 participants attended the second registration session. And yet the gallery comprises only 20,000 Indians. Why, CM wants to know, why doesn’t the gallery have 60,000 people in it? How were the 20,000 chosen? What was wrong with the other 40,000?

13 Also, Murali Chirala, please confirm CM’s assumption that the 20,000 probes chosen from the second registration session used to match against the gallery created from the first registration session were chosen because UIDAI thought they were the same 20,000 people, they’re not 20,000 probes chosen at random, say, from the 60,000 available.

14 We’re still in the second paragraph of Murali Chirala’s post. And there’s another question which requires his attention. He really is the most elliptical of writers.

15 There are two galleries. One of them is “seeded” and the other one isn’t. What is the difference? It seems from what Murali Chirala says that the seeded gallery is required to calculate the false negative identification rate, you can’t do that with an unseeded gallery. CM makes a guess. If Murali Chirala, say, is not matched and the operation returns a negative, the only way UIDAI can know that that is a false negative is if they know in advance that Murali Chirala is in the gallery. So seeding the gallery must involve checking it first, before matching operations begin, to see who’s in it. Perhaps Murali Chirala could confirm.

16 There is an equivalent problem with false positives. Suppose UIDAI probe the unseeded gallery with Murali Chirala’s biometrics and get three matches. Perhaps Murali Chirala is registered on the CIDR three times. They could be legitimate matches. How do UIDAI know that at least two of the positives are false? Only if they already know that all the records in the gallery are unique.

17 But hang on a minute. Isn’t that where we started? That’s exactly what we need to know. Is each record on the CIDR unique? The answer from Murali Chirala so far is that, yes, each record is unique if each record is unique. It seems that whatever Murali Chirala is talking about, the correct way to measure false positive and negative identification rates, it doesn’t help us to establish whether these blessed records are or are not unique.

18 We need answers to these questions raised by Murali Chirala’s second paragraph before we can move on to his third paragraph, which we look forward to doing, soon.

19 Your comments would be appreciated Murali Chirala. It took you 37 days, from 16 February to 25 March to respond last time. That’s fine as long as we’re only dealing with the halfwit David Moss. But an intelligent third party, perhaps an Indian taxpayer wondering if his or her tax money is being wasted, will look forward to an answer from you, Murali Chirala, much more quickly than that, please.

20 Let’s say that CM, the intelligent third party, while waiting for Murali Chirala’s response, reads ahead and finds this: “Mr. Moss incorrectly assumes that the decision threshold on the 1-to-1 comparison score is kept constant independently of the gallery size. In fact this threshold is adjusted in the real operational system to keep FPIR constant independently of the gallery size G. This fact alone removes the possibility of the sea of false positives”.

21 Is Murali Chirala right when he says that the possibility of a sea of false positives has been removed?

22 Probably.

23 After all, this sea of false positives argument is devised by David Moss, already acknowledged as the biometrics village idiot.

24 Or is it? CM checks and, lo, he finds this*, talking about the UK ID card scheme: “academic John Daugman, a former member of the Biometrics Assurance Group (BAG) which reviewed the scheme, says its reliance on fingerprints and facial photos to verify a person's identity will cause the system to collapse under the weight of mismatched identifications”. And this: “Daugman, an expert on iris recognition, says fingerprints and facial photos are not distinctive enough to be able to tell the UK's 45-million-strong adult population apart”. And this: “Daugman said that even if the error rate was as low as one in a million, the 10 to the power of 15 comparisons needed to verify the IDs of 45 million people would result in one billion false matches”. And this: “He told silicon.com: ‘The use of fingerprints will cause deduplication to drown in false matches’ ...”.

25 And who, pray, is John Daugman?

26 Only the king of biometrics based on the iris, the inventor and patentholder of the iris-matching algorithm and a professor at the Cambridge Computer Lab, that’s who**. Search for “john daugman” “biometrics” and Google gets you 7,930 hits.

27 Professor Daugman is not going to be stupid. Not like David Moss. Clearly Murali Chirala disagrees with him. But then, if you search for “Murali Chirala” “biometrics”, Google gets you one hit.

28 It’s hardly a slam dunk argument, you can’t resolve biometrics problems by Google hits, but CM thinks he’s got to make up his own mind about the sea of false positives and not just take Murali Chirala’s word for it that there is not even possibly a drowning problem. Once again, he hopes that Murali Chirala will soon answer.

----------
* http://www.silicon.com/management/publi ... -39294213/

** http://www.cl.cam.ac.uk/~jgd1000/

29 Considering paragraphs 20 to 28 above, CM, the intelligent Indian taxpayer investigating whether UIDAI is wasting public money, wonders why, if there is a sea of false positives, no explorer has discovered it and charted it.

30 One potential answer, considering paragraphs 1 to 19 above, is that no explorer has looked for it. If the sea of false positives is there, it would be revealed the minute anyone tried to prove that all records on the CIDR are biometrically unique. But they don’t try to prove that. Instead, if Murali Chirala is right, they try to calculate the false positive and negative identification rates. And that is a different job.

31 You’d think that CM would now test the truth of this hypothesis.

32 But no, you would be wrong.

33 CM’s eye has been caught by something else. Something else explosive. What looks like a fundamental error in Murali Chirala’s argument. An error that would bring down the whole house of cards. At least it would if UIDAI have made the same error. A hole at the heart of UIDAI’s case. A problem so glaring that it has even occurred to the troglodytic David Moss (in his own dim and benighted way, at least).

34 Every other reader of paragraph 20 above will also have spotted it. It’s a hand grenade with the pin removed. It’s a time-bomb, ticking down fast to zero.

35 Let’s take this slowly, just to check our logic as we go.

36 We’re going to look at matching, comparing someone’s biometrics against the template stored on the CIDR, we’re going to look at the “decision threshold” and the “comparison score” mentioned by Murali Chirala. About time too! That’s what biometrics is all about -- comparing/matching and verifying that Murali Chirala is Murali Chirala. How come it’s taken until paragraph 36 to get onto the subject? Never mind, here we are at last.

37 What we will find is that Murali Chirala is nearly right when he says “... this fact alone removes the possibility of the sea of false positives”. To be quite correct, Murali Chirala should have said “... this fact alone removes the possibility of FINDING the sea of false positives”.

38 We will find that UIDAI and NIST (see below) and others are using a peculiar concept of identity nothing like what normal people understand by the word and muddying the waters by confusing the two usages.

39 And, finally, we will discover that when it comes to biometrics NIST (see below) are arguably a busted flush and so is any organisation like UIDAI that follows NIST’s prescriptions.

40 Heady stuff. Hold on to your hat.

41 That’s what we will find and we’re going to start with NIST, a US organisation, the National Institute of Standards and Technology, and their May 2004 report ‘Matching Performance for the US-VISIT IDENT System Using Flat Fingerprints’, a report some of us have read at least a dozen times, please see http://dematerialisedid.com/PDFs/ir_7110...

42 Murali Chirala has great confidence in NIST. He says: “... while the [UIDAI proof of concept trial] does not provide sufficient number of identification searches to estimate [FPIR] and FNIR in a gallery of size of 1.2B directly, and extrapolating too far out has its own dangers in terms of statistical significance, methodology is available to extrapolate the [UIDAI proof of concept trial] results using trends established earlier in other large well established biometric performance evaluations performed by NIST and other agencies”.

43 Lots of other people share that confidence, including UIDAI, who were advised by NIST in the early days, please see UIDAI’s paper ‘Biometrics Design Standards For UID Applications’ at http://uidai.gov.in/UID_PDF/Committees/B...

44 That confidence is misplaced.

India has many federal and state programs to help people living in poverty, but today it's nearly impossible to be sure that funds and benefits are actually being delivered to those who need them. The ID project is an attempt to cut down on fraud and graft by increasing accountability and transparency. It's also meant to provide access to banking and the formal economy that many people lack.

Government biometrics programs have been tried before and failed, in India and elsewhere. The United Kingdom's universal ID program, for instance, got bogged down by both costs and privacy concerns and didn't offer tangible benefits to the average citizen. But the UIDAI's universal ID program, or Aadhaar, as it's called, seems to be off to a fast start.

From: David Moss
Sent: 04 May 2011 01:41
To: 's.hassler@ieee.org'
Subject: Joshua J Romero, May 2011, 'Fast Start for World's Biggest Biometrics ID Project'

Susan Hassler
Editor-in-Chief
IEEE Spectrum

Dear Ms Hassler

I refer you to Joshua J Romero's May 2011 article 'Fast Start for World's Biggest Biometrics ID Project'.

The article provokes a number of questions. The IEEE offers training for Certified Biometrics Professionals. The best way to get at those questions is to consider what the IEEE's CBP examiners would say about Mr Romero's article which I imagine would be something like the following.

Your response is sought.

Yours sincerely
David Moss
http://DematerialisedID.com

cc AxXiom for Liberty
Constitutional Alliance

--------------------------------------------------------------------------------

Examiners' Report
Mr Romero makes the following observation but then fails to investigate it:

"When a duplicate record is detected, it is flagged for manual verification. In the pilot study, the system generated false positive errors in 0.0025 percent of cases. That rate would generate 25 erroneous records daily, if the project makes the goal of 1 million IDs per day by October 2011 set by Finance Minister Pranab Mukherjee."

At first, 25 false positives out of every one million biometric comparisons sounds eminently manageable. In this case, first impressions are cruelly deceptive.

Given that the population of India is about 1.2 X 10^9, if the Unique Identification Authority of India claim to offer unique identification -- as their name suggests -- then they must perform 7.2 X 10^17 biometric comparisons to prove uniqueness. And if the false positive identification rate is to remain constant at 2.5 X 10^(-5) -- as Mr Romero suggests -- then they must expect to perform 1.8 X 10^13 "manual verifications". That is so impractical as to be impossible. It follows that UIDAI can never prove uniqueness and, to be honest, they should therefore not promise it.

The situation is, of course, much worse than that. UIDAI's false positive identification rate was derived from a test gallery of only 20,000 people. As the number of people on their population register rises, the number of false positives must be expected to increase. At least, it must be expected to increase if the matching threshold remains constant. With a constant matching threshold, there would be even more than 1.8 X 10^13 "manual verifications" to perform and UIDAI's job becomes even more impossible.

Mr Romero unfortunately fails to mention the matching threshold at all. That is a serious omission from his paper.

With a constant matching threshold, UIDAI will drown in a sea of false positives as the gallery size approaches 1.2 X 10^9. One way to avoid that would be to increase the matching threshold in line with the size of the gallery so that it becomes gradually harder and harder to make two sets of biometrics match. But then false negatives would increase correspondingly, causing UIDAI to drown in a sea of false negatives.

Either way, UIDAI drown. Mr Romero fails to entertain the possibility that UIDAI's project is impossible of success. That failure could cause India to waste a fortune, which would not please Finance Minister Pranab Mukherjee. It wouldn't please the 1.2 billion people of India either. They may be expected quite legitimately to riot.

We cannot at this moment recommend that Mr Romero should be awarded the CBP qualification. Fail.

--------------------------------------------------------------------------------