Social Data-mining, Part III: Social Data-mining and Datawunder

Datawunder is first a data filtering tool, and then a data visualization tool. It is intended to be a data querying application above all else.

The objective is not just to find patterns in the data that one is actively looking to derive. Datawunder also aims to provide you with patterns previously unlooked for, or those not considered.

The original creator can show the world his derived patterns. Another individual can come along and derive his own conclusion(s) from the very same data, so on and so forth… In this way, each use can present a different point-of-view to the world, all from a single datasource.

Once the view is made public, it is accessible to all and sundry (with an internet connection, of course). Anyone can see the data, anyone can use the data, and anyone can modify the data. This last part is the core of ‘social data-mining’!

There are several features tailored toward ‘Social Data-mining’:

• Comments

The comments system in Datawunder is an important part of the whole. Users can leave (traditional) plain-text comments about the default view or a particular snapshot of a view. Additionally, they can also create their own ‘snapshot’ from the original view to present their own point-of-view as a comment.

This would be a great way of pointing out a particular fact or observation or counterpoint. The user making the comment can also give the ‘snapshot’ a title and description of its own, thus differentiating it from the original (parent) view.

Similarly, each new comment can also incorporate its own snapshot, either of the original view, or a snapshot itself. These will all be presented in a conversation-style thread.

Each comment will also have the full range of sharing options i.e. Facebook, Twitter etc. so that users can share their comments with others.

• Snapshots

One of the most important features coming to Datawunder is ‘Snapshots’. “What is a snapshot?” you may ask. A snapshot is basically a ‘saved’ selection of data, or a pre-configured/pre-filtered selection of data that can be saved.

The objective of a snapshot is to present a certain point-of-view or a particular observation or an alternate conclusion based on the data in the view. Use Datawunder to filter your data, and then save that selection as a ‘snapshot’. Write a small description about it and share it with the world.

Now, in most cases, a snapshot implies finality. This is not so in Datawunder. Snapshots can be edited and modified based on the user’s requirements.

A user can take an existing snapshot, add and remove filters and save it as a new snapshot as a comment, which can be shared via social networking services.

 

Social Data-mining, Part II: Share, share, share…

“Knowledge is power!”, or so the saying goes. Knowledge shared is knowledge doubled!

Sharing (of information) is an integral aspect of Datawunder. Upload an Excel spreadsheet, create a view from it and share it with others. Alternatively, you can filter the contents of your view, save the selection as a snapshot and share that snapshot too. Invite debate and commentary by creating a slideshow from multiple snapshots. You can share that too, of course.

There are two ways in which views from Datawunder can be shared:

 

• Direct View Sharing

Using this method, a user can completely integrate a Datawunder view into another web page. The entire view, with complete functionality i.e. filtering, searching, row selection etc. can be accessed from that page itself.

This method would be the preferred method used by media channels, such as journalists, bloggers etc. when dealing with their self-created views.

The procedure is quite simple. Buttons linking to most popular social networking services will be present right from the view page itself. As such, the user can share the view with just a click.

The page will also contain a link back to the original view page.

 

• Link to View/Snapshot

This method mostly deals with the commenting system. Users can link to their views or snapshots (in the comments section) via their walls/feeds/timelines on major social networking sites.

This can also be done via buttons incorporated into the comments thread.

 

The future of the internet is all about sharing, and Datawunder is no different in this.

 

What is Social Datamining?

“Data mining is the process of extracting or deriving patterns from large quantities of data.”

- Internet, the

It combines methods from statistics and artificial intelligence along with database management. Data mining is an increasingly important tool for modern businesses to transform data into ‘business intelligence’, thereby providing an informational advantage. It is currently used in a wide range of profiling practices, such as marketing, surveillance, fraud detection, and scientific discovery.

The ‘Social’ aspect of data-mining involves the use and analysis of publicly available/open data to derive patterns in order to offer a certain opinion or point-of-view. The information serves to back-up this opinion.

This information is then spread across the internet through news, blogs, websites, etc. Anyone can draw their own patterns and observations from the data, comment on them, and debate the conclusions. At its very core, social data-mining involves the sharing of such data.

‘Data journalism’ is a prime example of social data-mining. It is a field that is growing constantly and gaining recognition by the day. Data journalists use such publicly available data, especially data made available by governments, to draw some startling conclusions. These are published over the internet and attract significant attention.

Similarly, Datawunder hopes to enable users to accomplish the same i.e. extracting patterns and drawing conclusions, while sharing this information in the cloud.

 

Data that makes sense…

While Datawunder can work with all types of spreadsheet content, it is – by nature of its coding – more suited to some types than others. This is related to the way Datawunder groups similar or identical sets of data.

Therefore, datasets with frequently repeated values are obviously more suitable for Datawunder to work with. As such, logs/log files and pre-dominantly text-based files are inherently tailored to work with Datawunder.

On the other hand, spreadsheets with a lot of individual or distinct values, while still usable by Datawunder – will not provide the same clarity or brevity that Datawunder aims for. In fact, the reverse is probably true – Datawunder will appear infinitely more cluttered in such a scenario.

Datawunder will also be of more value when used with larger spreadsheets i.e. those above 100 rows/20,000 records. This is where Datawunder’s grouping function really comes into play.

Similarly, spreadsheets with inbuilt statistical functions or mathematical formulas and data charts will work with Datawunder, but it is uncertain what ‘value’ the user will derive from this.

Bottom line: using the right type of spreadsheet can make a world of difference to the overall Datawunder experience!

Reminder! Phase 1 of Datawunder beta ends today…

Phase 1 of the Datawunder beta is almost at an end. All the feedback received from our beta testers will enable us to create and deliver a better final product for you.

We hope that with a larger base of people using and testing Datawunder, a larger amount of feedback will be received, enabling us to fix problems sooner, roll out new features at a rapid pace, as well as taking into consideration user requests for features.

As part of the beta, we have a small survey we’d like you to participate in.

As an incentive, we will be giving out an iPod Nano to the user who provides the best feedback for each phase of the beta.

 

View: Global Remittance Outflows [1985-2009]

Last time, I posted a little bit about ‘Global Remittance Inflows’. This time around, it’s all about the opposite – ‘remittance outflows’.

The view provides us with the total amount of remittance outflows’ from each country during the period 1985-2009.

Which nation contributes the most remittance income? Is this because it pays higher wages or employs more individuals?

How do developed nations fare in the disbursement of remittance income?

Have there been changes in the outward flow of income over the 25-year period?

 

Global Remittance Outflows [1985-2009]

View: Global Remittance Inflows [1985-2010]

A remittance is a transfer of money by a foreign worker to his or her home country.

“Money sent home by migrants constitutes the second largest financial inflow to many (developing) countries, even exceeding international aid.”

The global remittance economy is a booming one; the view provides information on the total amount of remittance ‘inflows’ each country has received during the period 1985-2010.

• Which country receives the largest remittance income?
• Do developing countries receive the largest slice of the global remittance pie?
• How long has a country been in the remittance business? How much has its share changed?

Global Remittance Inflows [1985-2010]

View: Changes in Corruption Levels

Have you ever wondered about the level of corruption in a country? What do the citizens of each country think?

Transparency International, an international corruption watchdog organisation, has compiled data on the change in corruption levels in each country, as perceived by the general public in that country.

Which countries were thought to have reduced corruption levels? Has there been a change in the eyes of the public?

Changes In Corruption Levels

Teaser…

The inclusion of pie-charts into Datawunder was an extremely important milestone – we can now actually call it data visualization – in the development of the service.

However, pie-charts are but the tip of the iceberg; we have several new features in the pipeline. The next update or two will add more statistical functions to Datawunder. As such, users can choose from several methods of visualizing their statistical data. These ‘alternate views’ will also be dynamic; and users will be able to switch between them on the fly.

Bar Charts will be added in the next update to Datawunder. They will be similar to pie-charts i.e. they will be ‘clickable’, interactive and capable of filtering data.

Table/Grid Views: Further down the line, we also plan to introduce ‘table’ or ‘grid views’ to Datawunder.

Improvements to Datawunder

Along with the addition of pie-charts, a few changes have been made to the standard Datawunder ‘view’.

1. Changes to search:

The search functions of Datawunder have also been modified:

• Global Search:  No changes have been made to the functionality of global search, but improvements under the hood ensure that it now returns results faster than before.

• Row Search:  The method of searching individual rows has now changed. You will notice a new search icon in the top-right corner of every row. Clicking this icon will open a drop-down search box.

 

2. Architectural Changes:

There have been a few underlying improvements to Datawunder; as a result, it is now faster than ever. Views open sooner than before, and the filtering of selected data has been given a noticeable speed boost.

These improvements should make using Datawunder a much smoother experience.