A doctoral candidate in Virginia developed a highly accurate (as far as we can tell) and probably one of a kind map of North Korea (Wall Street Journal article here). This may become important in light of other developments, including North Korea’s announcement of having done a second, successful underground test of an atomic bomb (see AP story here).

Earlier this year, researchers for the Open Security Foundation used seemingly unrelated newspaper articles to learn details of the Heartland Systems data breach, one of the biggest data hacking incidents yet known (Wired story here), before the breach was made public.

Both of these items reminded me of an old story about one of the first people to study serial murder. This was a detective (whose name I should be able to remember but can’t just now. Sorry!) who began studying newspapers from all over California in order to find similar murders that were not thought to be linked, as likely as not because they were in different jurisdictions so that the investigators involved did not even know about them. He discovered quite a few links no one else had noticed this way.

This sort of research to link up scattered, seemingly unrelated information is called open source intelligence gathering and we may not be far from the time when you can get a degree in it and (hopefully) lots of high-paying jobs. The term should not be confused with open source software or artificial intelligence. This intelligence is the kind that concerns intelligence agencies like the CIA. And the open just means not hidden.

Wikipedia defines open source intel as a “form of intelligence collection management that involves finding, selecting, and acquiring information from publicly available sources and analyzing it to produce actionable intelligence. In the Intelligence Community (IC), the term “open” refers to overt, publicly available sources (as opposed to covert or classified sources); it is not related to open-source software or public intelligence” (Open Source Intelligence). Or, more understandably: OSI is the art of finding things out that are not explicitly published, without spies or wiretaps or other such arcane and often illegal methods. It’s taking bits and pieces of information that are publicly available and combining them to discover something more interesting, something that might be non obvious or even secret. This is a form of data mining that the Internet has made much much easier than it used to be, though that is only part of the equation.

The interesting thing is that not only does the internet make it easier to access open sources (see for example how mandatory legal disclosures are more and more moving online, bypassing those impossible to read little newspaper pages where they’ve been for decades here), it also makes it easier to get distributed teams of people working on a problem, purely for the love of probing a mystery or solving a problem.

In the first example above, Curtis Melvin collates information provided by people from all over the world (sometimes known to him, sometimes anonymous), checks what he can and has used that information and Google Earth to make what may be the most accurate map of North Korea in existence. Agencies like the CIA may have more accurate ones but it seems at least possible that, if they do, it’s because they built on his work. There just aren’t enough people who’ve been to North Korea (and come back) for the pool of those really knowledgeable to be all that big.

This goes beyond concepts like Web Mining (which is similar but limited to web pages) and Crowd Sourcing (which is generally thought of more in a business context, though it doesn’t have to be) to create new information that certain people (Kim Jong who?) might prefer stayed hidden. Sometimes this is attributed to the power of the Internet but don’t forget the drive, curiosity and ingenuity of the people doing the work.

I think we’re at the beginning of seeing what this sort of distributed mania can do with all the info that’s shimmering out there in the matr in the Internet.

If I were a sneakier sort of person, I would say that what Melvin has built is the beginnings of a good escape map.

See what open source intel can do?

A few years ago when I took a course in web development, I had an assignment to survey a number of different websites all from the same industry. Since I worked for a newspaper at the time, I chose the newspaper industry. After spending many hours on this assignment, my conclusion was that the newspaper industry was completely devoid of creativity or any thought, whatsoever, about the needs of the consumer. I found the web pages for all the different papers to be essentially the same, offering the same news in the same format, with the same crappy navigation system using the same old web 1.0 (or maybe 0.8) technology.

Since then, we have had a few years for web technology to develop and for companies to learn the ropes of the new system. For the most part, though, newspaper web sites haven’t improved beyond adding some video and maybe a search function. Circulations are down, advertising is way down but there doesn’t seem to be a lot of creativity going toward finding solutions.

Or does there?

I’m beginning to think the story might be more complicated than that. There might even be grounds for hope. Maybe. Don’t go buying stock in news media yet, but don’t write them off forever, either. I mentioned in a previous post (April Random Roundup) that my old employer the Rochester Democrat & Chronicle was planning on launching a pay-walled version of the web site. I expressed just a tiny amount of skepticism as to whether it was a good idea. Since then, there’s been a change. The site was late launching and then the D&C’s masters at Gannett pulled the plug on the whole idea. Apparently, there’s some thought of corporate rolling out a pay-site solution company wide. At least, that was the excuse I heard. It’s just dumb enough to be true, too.

I started thinking along more positive lines when I looked at the pay site (shortly before it was taken down). One thing that stood out to me was that the design was far better than standard newspaper web sites. Instead of the usual disorganized jumble of headlines, section links and ads (Oh! They have tabs now! So Gannett properties are up to the 20th century finally!) there was a simple, tasteful layout with a few headlines and a series of very easy to use drop-downs for navigation (It also had the same old boring content but that’s another story).

This reminded me that even a failed experiment is often worth trying, for the lessons learned and skills acquired if for nothing else. Gannett could have allowed the D&C pay site to proceed and tracked it as an important experiment. It could have been used as a way to try out lots of new ideas and to train people in applying technologies that, frankly, newspapers have mostly ignored in the childish hope that they would go away. The difficulty of getting upper management to even consider new things frustrated me numerous times when I worked there. When they finally decided to give something new a decent shot (even if it was the wrong thing to try), corporate stepped on them with all the deep insight and careful timing usually expected from lifelong bureaucrats (In case you’re wondering, that was an example of a literary device known as irony).

For the record, the idea that it would be better to try the whole thing corporate wide is idiotic. That’s not an experiment it’s policy. If it goes wrong, it goes wrong everywhere. And everyone learns the same lessons from it (history shows that lesson to be, “Those people at corporate are morons!”).

The stifling of innovation, not just from the D&C but from every other site like it that was thinking of trying some kind of experiment of its own, is emblematic of what’s wrong with the entire industry. It’s funny, really, that an industry that depends so heavily on writing should be so hostile to creativity.

But I said there was room for hope and I meant it. In recent days there have been numerous stories about efforts by various news agencies and companies to try things they haven’t before. Interestingly, the analysis is almost always similar to my original take on the D&C pay site: misguided and too late anyway. That might even be true of each individual story. But taken together, it shows signs that the newspaper industry has finally started earnestly looking for new ways to tackle the systemic problems that have been dragging them down. Here are a few:

An open letter by Mark Cuban

He has a number of ideas about how papers can improve their revenue. Some of these ideas are familiar to those of us who’ve been screaming (mostly metaphorically) at newspapers for years but they are still not mainstream. He has this wild theory that creative ways of using technology to give customers value might get them to spend money. Interesting stuff.

A New York Times story about electronic readers for newspapers

There’s been a lot of buzz about this one lately. Several companies are apparently going to put out a clipboard-sized flat screen device that people can read their newspapers on. Most analysis I have seen has been highly skeptical (See here and here for example) and with good reason. But the whole idea is a century ahead of the thinking of most newspaper executives (which puts it around the year 1953 but that’s not the point!)

An interesting one about the Chicago Trib actually asking customers for comment on online content

Some journalistic types appear to have considered it a breach of ethics to do this (maybe because it was done before stories were published, though that might just be a cover for indignation at the possibility of being judged. Again, for purposes of this post, the reason doesn’t really matter). The more interesting thing is that newspapers have been very slow to embrace the idea of working with online readers rather than merely shoveling content at them. Despite the reaction to this one, expect to see more of it in the future. This is probably more important than delivering content to fancy new gadgets.

Last on today’s list, Rupert Murdoch wants to charge money for access to the news

This is another one of those doomed-to-fail experiments but maybe it needs to fail spectacularly (as only someone of Murdoch’s caliber can) for people to get the point that locking up content and charging for access only works if people are reasonably sure the content is worth money. If you can just click a couple links to read the same thing on some free site, there’s no chance at all of making money off of it.

Anyway, despite negative reviews for so much of the stuff in the list, taken as a whole what I see is a burst of innovation and experimentation (outside Gannett) that will likely go on for years and produce some real successes in with the failures. Just the act of criticizing someone else’s experiment will help other people at other papers come up with their own ideas. They may be just as hare-brained as the originals but with new twists and new combinations of features, until something clicks. Notice that I’m not saying that newspaper-sized e-readers or pay-walls are the answer. I’m just saying that the atmosphere of experimentation, of competing to produce ideas, is healthy.

This is where the hope for newspapers (and newspaper readers) starts: With humans cooking up crackpot schemes – that sometimes work. Keep your fingers crossed.

I work for someone who often talks about “disruptive technology” and how hard it is to keep it alive. He believes that not only is the project we are building disruptive in the context of the technology world but also in the company itself. One definition of disruptive technology is found at the old standby, Wikipedia “A disruptive technology or disruptive innovation is a technological innovation that improves a product or service in ways that the market does not expect, typically by being lower priced or designed for a different set of consumers.”

The term came to my mind in a completely different context, though, when I was reading an article [at DefenseTech] about the U.S. Army and the developing – and struggling – doctrine of hybrid war. I was already familiar with the somewhat different concept of asymmetric warfare, in which a very weak opponent (such as Al Qaeda in Iraq) uses guerrilla or terrorist tactics to go after a much more powerful foe (such as the United States) [See http://en.wikipedia.org/wiki/Asynchronous_warfare for more on asymmetric warfare]. But the term hybrid war was new to me.

According to the article, hybrid war is fought against (surprise!) hybrid enemies who “come equipped with high-end, precision guided weapons, yet fight in distributed networks of small units and cells more akin to guerrillas.” This put me in mind of the Afghani Mujahideen of the 1980s, who used U.S. supplied stinger missiles against the invading Soviets. This kind of warfare is not fought with the traditional tank columns and carrier groups but can still do terrible damage. It is made possible both by modern weaponry and by the cleverness and determination of small group leaders.

There are thoughtful leaders in the military who understand that this change in the nature of the enemy is not just a serious threat to people those enemies target. It requires deep changes in military procurement, tactics and training. Hopefully, it is starting to sound like maybe I wasn’t completely off base relating the concepts of hybrid war to the market-oriented definition of disruptive innovation (another term for disruptive technology and a little closer to where I think I’m going with this).

The article mentions the disillusion of an Army officer who went through extensive training that seemed geared more toward World War 2 weapons and tactics than anything currently likely in the 21st century. The old saying goes that armies train to fight the last war, but WW2 came before Korea and Vietnam, not to mention Iraq (1 and 2) and Afghanistan. To be fair, battles of vast amounts of technology (tanks, planes, artillery) were considered a serious issue during the Cold War because that was also the way Soviet forces were organized. But the Soviet Union doesn’t even exist anymore, so what’s the point?

Now, zigging (rather than zagging) back to the business world, one of the characteristics of disruptive technology is that it serves a need other than that of the traditional dominant customers. It includes not just new products like the iPhone but new methods of dealing with customers such as, when it was new, was exemplified by FedEx. And, as my boss has pointed out, this kind of change can be fragile. The old guard came up through the ranks serving the dominant customers and thinks that, in so doing, it has learned what is needed for success.

Buggy whip makers didn’t abandon their product because they saw it had become obsolete. It’s a fair bet that not one switched to making carburetors. They hung on until bankruptcy or until they were too old to do business anymore and there were no successors in sight. They did what had always worked. That’s what people do, in businesses or armies.

This is not a small problem. Think about the civilizations that have come and gone, enduring for centuries or even millenia, with only very small technological or social change. Egypt and China, for example, remained more or less stable (barring the occasional civil war) for thousands of years before finally being brought down by external forces. Without those external forces they might have lasted indefinitely.

We think of our time as being one of enormous, high speed change but is it really? How many companies are out there doing as the Army does, fighting battles of the past with outdated tools, while better disruptive alternatives fight to be heard? How many outdated assumptions (wars are fought with tanks because that’s what the other side uses) are holding us back?

Or to put another twist on things, how many hybrid enemies have our institutions failed to recognize? When put that way, the problem seems even more urgent, doesn’t it?

Sorry. I don’t have any answers just now. It’s going to take some thinking.