Because that data can tell a story that perhaps you’d prefer to keep private or is none of anyone else’s business.  For instance:

Certain telephone numbers are used for a single purpose, such that any contact reveals basic and often sensitive information about the caller. Examples include support hotlines for victims of domestic violence and rape, including a specific hotline for rape victims in the armed services.

Similarly, numerous hotlines exist for people considering suicide, including specific services for first responders, veterans, and gay and lesbian teenagers. Hotlines exist for suffers of various forms of addiction, such as alcohol, drugs, and gambling.

Or if you’re a government agency:

Similarly, inspectors general at practically every federal agency—including the NSA—have hotlines through which misconduct, waste, and fraud can be reported, while numerous state tax agencies have dedicated hotlines for reporting tax fraud. Hotlines have also been established to report hate crimes, arson, illegal firearms and child abuse. In all these cases, the metadata alone conveys a great deal about the content of the call, even without any further information.

So to those who buy into the “you have nothing to fear if you’ve done nothing wrong” nonsense … yes, you do have something to fear.  Misuse of this data.  This isn’t about what you may have done wrong.  This is about snooping and drawing conclusions about you based on numbers you’ve dialed.

And, it can get more intimate than that:

Two people in an intimate relationship may regularly call each other, often late in the evening. If those calls become less frequent or end altogether, metadata will tell us that the relationship has likely ended as well—and it will tell us when a new relationship gets underway. More generally, someone you speak to once a year is less likely to be a close friend than someone you talk to once a week.

Consider the following hypothetical example: A young woman calls her gynecologist; then immediately calls her mother; then a man who, during the past few months, she had repeatedly spoken to on the telephone after 11p.m.; followed by a call to a family planning center that also offers abortions. A likely storyline emerges that would not be as evident by examining the record of a single telephone call.

Likewise, although metadata revealing a single telephone call to a bookie may suggest that a surveillance target is placing a bet, analysis of metadata over time could reveal that the target has a gambling problem, particularly if the call records also reveal a number of calls made to payday loan services.

If a government employee suddenly begins contacting phone numbers associated with a number of news organizations and then the ACLU and then, perhaps, a criminal defense lawyer, that person’s identity as a prospective whistleblower could be surmised.

There is no reason to collect metadata unless you plan on doing something with it.  It is just a large list of telephone numbers. Intelligence production comes from analysis. What is outlined above are just of a few of the things that can be found with that information. As Ed Felten, a professor of computer science at Princeton University argues:

[T]he distinction between call “contents” and “metadata” isn’t always clear. Sometimes, the mere fact that someone called a particular number reveals extremely sensitive personal information.

Unfortunately the Supreme Court decided in 1979 that phone records are not protected by the Fourth Amendment because consumers do not have a reasonable expectation of privacy in their calling records.  In light of the NSA’s collection habits and the obvious privacy issues that sort of collection can bring as illustrated above, it’s time to revisit that decision and come down on the side of citizenry and their right to privacy.  If the NSA or other collection agencies feel the need to monitor certain phone numbers or collect metadata on them, let them justify themselves to a court.