Wolfram Data Drop - new service Wolfram Research
Translation of Stephen Wolfram's post " The Wolfram Data Drop Is Live! ". I express my deep gratitude to Kirill Guzenko for his help in translating.
Where should the data from the Internet of things go? We have a great technology in Wolfram Language for interpretation, visualization, query analysis and other interesting things. But the question is, how should all this data from network-connected devices and all other sources get to where you can do all of the above with them? Today we are launching what, in my opinion, is a great solution to this problem: Wolfram Data Drop .
')
When I first started thinking about Data Drop, I viewed it mostly as a convenient means of moving various data. But now that Data Drop has been created, I understand that this is much more than just a service for moving data. In fact, this is an important step in our ongoing journey of integrating computing and the real world.
So what is Wolfram Data Drop? At the functional level, it is a universal data aggregator designed to receive and organize data obtained from various sensors, devices, programs, people or whatever you are. While the data is stored in the cloud in such a way that can be used for computing and processing as if they are stored on the device itself (seamless integration of calculations and data).
Our goal is that data from Wolfram Data Drop can be accessed as easily as possible from anywhere. You can use web API , email , Twitter , web form , Arduino , Raspberry Pi and the like. How are we going to gradually add more and more different opportunities for communication with other data collection systems, both software and hardware. It doesn't matter where the data comes from. The idea is that Wolfram Data Drop collects this data using the same algorithm: in “databin” with a specific ID.
Below is an example of how this works. On my desk now is this little device:
Every 30 seconds it receives data from tiny sensors that are at the right end, and then sends data via wifi and web API to Wolfram Data Drop databin with ID “3pw3N73Q”. Like all databin, this databin has its own page on the Internet: wolfr.am/3pw3N73Q .
This page is the control center , through which you can, for example, download the original data. But the most interesting fact is that databin is fully integrated into Wolfram Language. The basic concept of the Wolfram Language is that it is based on knowledge, contains within itself a huge amount of knowledge about the world, calculations; how it works
For example, the Wolfram Language has current data on stock prices , earthquakes, and many other things . But now he can know about things like the environmental conditions on my desk, which were obtained through Wolfram Data Drop. In our case, we get the data from the small device shown above.
Here is how it works. Below is the Wolfram Language character object called databin:
You can perform various actions on it. For example, below are graphs of time series data from databin:
And here are the histograms of values:
But the initial data presented in the form of a table:
What is really cool: databin, which can hold data from anywhere, is only part of the language. And we can perform calculations with this data as with anything else.
Here, for example, the minimum and maximum temperature functions on my desk are given (MinMax is a new Wolfram Language feature):
It can be converted to other units of measurement ( % corresponds to the result calculated at the previous step):
Let's now imagine the function of pressure on time. Here she is:
Of course, Wolfram Knowledgebase has an archive of weather data . So, in the Wolfram Language, we can simply request pressure at our location for a period of time that is contained in the databin, and the results will be surprisingly similar:
If we want, we can do a lot of interesting things with the data: to analyze time series , use machine learning algorithms, create a model and many other things. We can visualize data in some intricate way. You can create structured queries or queries in natural language .
It should be noted that the data we receive from the databin, go along with the units of measurement . Here is an example of a fundamental feature of Wolfram Data Drop: it is not just storage of source data; the data is stored and contain the real meaning that is given to them. This means that you can immediately understand where this data should be used.
To do this, we use our high-tech system: Wolfram Data Framework ( WDF ). Developed in close connection with Wolfram | Alpha , it turned out our system of standards for symbolic representation of data from the real world. And each databin in Wolfram Data Drop can use WDF to determine the meaning of the data (“ data semantics signature ”), which determines how this data will be interpreted, as well as how our automatic import system and natural language recognition system will process new data that comes in .
The beauty of all this lies in the fact that since the data is in Wolfram Data Drop, it can be universally available and interpreted for Wolfram Language and any system using this language. So, for example, any public databin in Wolfram Data Drop is available at any time via Wolfram | Alpha, as well as through any intelligent helpers that use Wolfram | Alpha. Tell Wolfram | Alpha the name databin, and the system will create an automatic report analyzing the data contained:
With WDF, Wolfram Data Drop instantly works with 10,000 kinds of different physical dimensions . But Data Drop is not limited to numbers and numeric values. Anything can be uploaded to the system. And since the Wolfram Language is a symbolic language, it can handle all of this with the same methods.
The Wolfram Data Drop automatically records time and, when possible, location data. In both cases, there are specific standards for presenting this data in WDF. As well as chemical compounds , cities , classes , networks , or thousands of other things . You can also work with images loaded into Wolfram Data Drop.
Somewhere in our quality assurance department, there is a Raspberry Pi with a connected camera that takes pictures of two of our fish every 10 minutes and writes to the databin in Wolfram Data Drop:
Using the Wolfram Language, it is easy to collect all these images into an interactive 3D cube image :
Or process images to get a map of where the fish spent their time:
We can carry out any kind of data analysis in Wolfram Language. But for me the most exciting fact is how easy it is to put data from the real world into the system via Wolfram Data Drop.
In our office, various databin are commonly used. They are very easy to create and integrate with existing control and monitoring systems. We now have a databin for the server (HVAC), for temperature sensors on the roof of our headquarters, for refrigerators in the rest room, for monitoring the Internet network, and for Data Drop itself. Every day some new solutions appear and are being introduced.
A lot of databin was created for personal use. For me, for example, collecting and analyzing data is a long-standing hobby. And, in fact, I have collected personal data about myself for more than a quarter of a century. I can also say that in March 2015 there will be a step forward on a historical scale. After all, using Data Drop, it has become much easier to collect data, as a result of which the amount of information I will collect will increase dramatically. I will be a man with at least twenty-five databin ... and this is not the limit.
A very important fact is that everything in Wolfram Data Drop is stored in WDF , that is, all data is presented in a certain form, which means it is possible to work with different databin, compare, combine different values and get meaningful results of calculations.
As long as you work with small amounts of data, Wolfram Data Drop is completely free and open, that is, anyone or anything can instantly download data there . Official users will be able to work with much larger amounts of data at a rate that we believe will continue to increase.
Wolfram Data Drop databin can be both public and private. It may also be open for changes or require authorization. Anyone can access Wolfram Data Drop using our main cloud - Wolfram Cloud . However, organizations that have their own Wolfram Private Cloud will soon also be able to have their own private Data Drop, which operates within their own infrastructure.
So what does a typical workflow with Wolfram Data Drop look like? It depends on what you do. And even with a single databin, judging by my experience, several different workflows are required.
It is very convenient to be able to take some kind of databin and immediately make calculations online in the Wolfram Language session, examine the data, record the results of work and create reports in a separate file.
However, it is often much more convenient when working with databin occurs automatically. For example, you can create a scheduled task, and the system will send an email to databin. You can place data on the Wolfram Cloud so that someone using CloudCDF can work with the received data over the Internet. You can make the report automatically generated and sent whenever someone visits the page.
But the scope is not limited to the Internet. From the moment the report entered the Wolfram Cloud, it immediately becomes available on mobile and wearable gadgets. The same applies to desktops.
You don't have to do the report yourself. Instead, you only need to have a program in Wolfram Language that will monitor the databin, then, for example, send a notification / alarm / perform any other action in case some combination of conditions is executed.
You can make databin public, thereby effectively publish some data in this way. Or you can make the databin private, accessible only to a data source or some third party you have designated. You can create an API that will receive data from the databin in the original / processed form, and you can call all this not only via the Internet, but also through any programming language or any software system.
A certain databin can receive data only from one source, or from one device, or collect data from different sources and act as an aggregator. Since each piece of data is supplied with detailed metadata, you can always tell where it came from.
For several years, we have been pretty closely connected with companies that are involved in various plug-in devices, in particular through our Connected Devices Project . And many times I had a similar dialogue: the company talks about a wonderful device that it makes, which measures something very interesting. Then I ask - and what happens to the data that this device registers? And, often, they answered that this is a question that worries them a lot, but they do not plan to hire individual teams to create cloud servers, applications and other infrastructure for working with data.
One of the reasons that we created Wolfram Data Drop was the desire to share the best solution with these companies. They get the data, then just send it to Data Drop, and from there the data already goes to our cloud (or private cloud), where they can be easily analyzed, visualized, received and distributed via the Internet page, API, applications or whatever that either.
I guess a large number of device companies will use Wolfram Data Drop. They will be able to receive data in different ways, sometimes through the web API. Sometimes through direct connection to the Wolfram Language system, for example, on the Raspberry Pi . Sometimes through Arduino , or through Electric Imp , or through any other hardware platform compatible with Data Drop. Data can be transmitted via phones or other mobile devices. And sometimes already recorded data from another cloud can serve as a source.
We do not work with the lowest-level problems of receiving data from the device through wires, wifi, Bluetooth and whatever else.
There are various ways to access data between connected devices. Developers and researchers can access the Wolfram Cloud directly, or via the Wolfram Language desktop / online environment. Customer-focused companies associated with devices can work, for example, through their own private Wolfram Private Cloud . They can also access Data Drop using the Wolfram mobile app or their own program. Or through some kind of wearable device.
Sometimes a company needs to collect data from different devices. Suppose for monitoring a network or doing some research. And, again, their users may want to work directly with the Wolfram Language, or through the portal / application.
When I first started thinking through the concept of Wolfram Data Drop, I assumed that the data would mainly come from some kind of automated devices. But now that we have Data Drop, I realized that it would be very useful to work with the data that comes from people. This is a great way to collect answers, say, in a classroom, or collect feedback for a crowdsourcing project, store various personal notes and the like. Once the data semantics signature for the databin has been determined, Wolfram Data Drop can automatically generate a data presentation form that can be uploaded to the network or to a mobile device.
Forms can work with images, text, and many other things. In the case of text, our natural language interpretation system can retrieve data and interpret it in WDF and lead to standard form.
Now that we have Wolfram Data Drop, I continue to find various uses for this system and I don’t understand at all - how could I have done without this system before? As with everything in Wolfram Language, this is the history of automation: Wolfram Data Drop automates the entire erratic set of actions associated with collecting and processing real-world data from various sources.
The ultimate goal for me is the possibility of collecting and analyzing data of any kind without the need to create / use any other systems. For example, last week I finished using Wolfram Data Drop to collect performance data on our cloud. In a normal situation, this task would be too complicated and cumbersome, I would not even consider the possibility of doing it myself. But using Data Drop, installation and configuration took me only a few minutes, and now I can receive the data that interests me.
I look forward to the implementation of the things that I am going to implement in Wolfram Data Drop, exactly like to see how people use this system. You can already try out the beta version that we launched today and send us feedback (along with Data Drop databin, of course). I hope it will take a little time before the databins are woven into the infrastructure of the world: one more step forward in our long-term mission - to make the world computable ...
Where should the data from the Internet of things go? We have a great technology in Wolfram Language for interpretation, visualization, query analysis and other interesting things. But the question is, how should all this data from network-connected devices and all other sources get to where you can do all of the above with them? Today we are launching what, in my opinion, is a great solution to this problem: Wolfram Data Drop .
')
When I first started thinking about Data Drop, I viewed it mostly as a convenient means of moving various data. But now that Data Drop has been created, I understand that this is much more than just a service for moving data. In fact, this is an important step in our ongoing journey of integrating computing and the real world.
So what is Wolfram Data Drop? At the functional level, it is a universal data aggregator designed to receive and organize data obtained from various sensors, devices, programs, people or whatever you are. While the data is stored in the cloud in such a way that can be used for computing and processing as if they are stored on the device itself (seamless integration of calculations and data).
Our goal is that data from Wolfram Data Drop can be accessed as easily as possible from anywhere. You can use web API , email , Twitter , web form , Arduino , Raspberry Pi and the like. How are we going to gradually add more and more different opportunities for communication with other data collection systems, both software and hardware. It doesn't matter where the data comes from. The idea is that Wolfram Data Drop collects this data using the same algorithm: in “databin” with a specific ID.
Below is an example of how this works. On my desk now is this little device:
Every 30 seconds it receives data from tiny sensors that are at the right end, and then sends data via wifi and web API to Wolfram Data Drop databin with ID “3pw3N73Q”. Like all databin, this databin has its own page on the Internet: wolfr.am/3pw3N73Q .
This page is the control center , through which you can, for example, download the original data. But the most interesting fact is that databin is fully integrated into Wolfram Language. The basic concept of the Wolfram Language is that it is based on knowledge, contains within itself a huge amount of knowledge about the world, calculations; how it works
For example, the Wolfram Language has current data on stock prices , earthquakes, and many other things . But now he can know about things like the environmental conditions on my desk, which were obtained through Wolfram Data Drop. In our case, we get the data from the small device shown above.
Here is how it works. Below is the Wolfram Language character object called databin:
You can perform various actions on it. For example, below are graphs of time series data from databin:
And here are the histograms of values:
But the initial data presented in the form of a table:
What is really cool: databin, which can hold data from anywhere, is only part of the language. And we can perform calculations with this data as with anything else.
Here, for example, the minimum and maximum temperature functions on my desk are given (MinMax is a new Wolfram Language feature):
It can be converted to other units of measurement ( % corresponds to the result calculated at the previous step):
Let's now imagine the function of pressure on time. Here she is:
Of course, Wolfram Knowledgebase has an archive of weather data . So, in the Wolfram Language, we can simply request pressure at our location for a period of time that is contained in the databin, and the results will be surprisingly similar:
If we want, we can do a lot of interesting things with the data: to analyze time series , use machine learning algorithms, create a model and many other things. We can visualize data in some intricate way. You can create structured queries or queries in natural language .
It should be noted that the data we receive from the databin, go along with the units of measurement . Here is an example of a fundamental feature of Wolfram Data Drop: it is not just storage of source data; the data is stored and contain the real meaning that is given to them. This means that you can immediately understand where this data should be used.
To do this, we use our high-tech system: Wolfram Data Framework ( WDF ). Developed in close connection with Wolfram | Alpha , it turned out our system of standards for symbolic representation of data from the real world. And each databin in Wolfram Data Drop can use WDF to determine the meaning of the data (“ data semantics signature ”), which determines how this data will be interpreted, as well as how our automatic import system and natural language recognition system will process new data that comes in .
The beauty of all this lies in the fact that since the data is in Wolfram Data Drop, it can be universally available and interpreted for Wolfram Language and any system using this language. So, for example, any public databin in Wolfram Data Drop is available at any time via Wolfram | Alpha, as well as through any intelligent helpers that use Wolfram | Alpha. Tell Wolfram | Alpha the name databin, and the system will create an automatic report analyzing the data contained:
With WDF, Wolfram Data Drop instantly works with 10,000 kinds of different physical dimensions . But Data Drop is not limited to numbers and numeric values. Anything can be uploaded to the system. And since the Wolfram Language is a symbolic language, it can handle all of this with the same methods.
The Wolfram Data Drop automatically records time and, when possible, location data. In both cases, there are specific standards for presenting this data in WDF. As well as chemical compounds , cities , classes , networks , or thousands of other things . You can also work with images loaded into Wolfram Data Drop.
Somewhere in our quality assurance department, there is a Raspberry Pi with a connected camera that takes pictures of two of our fish every 10 minutes and writes to the databin in Wolfram Data Drop:
Using the Wolfram Language, it is easy to collect all these images into an interactive 3D cube image :
Or process images to get a map of where the fish spent their time:
We can carry out any kind of data analysis in Wolfram Language. But for me the most exciting fact is how easy it is to put data from the real world into the system via Wolfram Data Drop.
In our office, various databin are commonly used. They are very easy to create and integrate with existing control and monitoring systems. We now have a databin for the server (HVAC), for temperature sensors on the roof of our headquarters, for refrigerators in the rest room, for monitoring the Internet network, and for Data Drop itself. Every day some new solutions appear and are being introduced.
A lot of databin was created for personal use. For me, for example, collecting and analyzing data is a long-standing hobby. And, in fact, I have collected personal data about myself for more than a quarter of a century. I can also say that in March 2015 there will be a step forward on a historical scale. After all, using Data Drop, it has become much easier to collect data, as a result of which the amount of information I will collect will increase dramatically. I will be a man with at least twenty-five databin ... and this is not the limit.
A very important fact is that everything in Wolfram Data Drop is stored in WDF , that is, all data is presented in a certain form, which means it is possible to work with different databin, compare, combine different values and get meaningful results of calculations.
As long as you work with small amounts of data, Wolfram Data Drop is completely free and open, that is, anyone or anything can instantly download data there . Official users will be able to work with much larger amounts of data at a rate that we believe will continue to increase.
Wolfram Data Drop databin can be both public and private. It may also be open for changes or require authorization. Anyone can access Wolfram Data Drop using our main cloud - Wolfram Cloud . However, organizations that have their own Wolfram Private Cloud will soon also be able to have their own private Data Drop, which operates within their own infrastructure.
So what does a typical workflow with Wolfram Data Drop look like? It depends on what you do. And even with a single databin, judging by my experience, several different workflows are required.
It is very convenient to be able to take some kind of databin and immediately make calculations online in the Wolfram Language session, examine the data, record the results of work and create reports in a separate file.
However, it is often much more convenient when working with databin occurs automatically. For example, you can create a scheduled task, and the system will send an email to databin. You can place data on the Wolfram Cloud so that someone using CloudCDF can work with the received data over the Internet. You can make the report automatically generated and sent whenever someone visits the page.
But the scope is not limited to the Internet. From the moment the report entered the Wolfram Cloud, it immediately becomes available on mobile and wearable gadgets. The same applies to desktops.
You don't have to do the report yourself. Instead, you only need to have a program in Wolfram Language that will monitor the databin, then, for example, send a notification / alarm / perform any other action in case some combination of conditions is executed.
You can make databin public, thereby effectively publish some data in this way. Or you can make the databin private, accessible only to a data source or some third party you have designated. You can create an API that will receive data from the databin in the original / processed form, and you can call all this not only via the Internet, but also through any programming language or any software system.
A certain databin can receive data only from one source, or from one device, or collect data from different sources and act as an aggregator. Since each piece of data is supplied with detailed metadata, you can always tell where it came from.
For several years, we have been pretty closely connected with companies that are involved in various plug-in devices, in particular through our Connected Devices Project . And many times I had a similar dialogue: the company talks about a wonderful device that it makes, which measures something very interesting. Then I ask - and what happens to the data that this device registers? And, often, they answered that this is a question that worries them a lot, but they do not plan to hire individual teams to create cloud servers, applications and other infrastructure for working with data.
One of the reasons that we created Wolfram Data Drop was the desire to share the best solution with these companies. They get the data, then just send it to Data Drop, and from there the data already goes to our cloud (or private cloud), where they can be easily analyzed, visualized, received and distributed via the Internet page, API, applications or whatever that either.
I guess a large number of device companies will use Wolfram Data Drop. They will be able to receive data in different ways, sometimes through the web API. Sometimes through direct connection to the Wolfram Language system, for example, on the Raspberry Pi . Sometimes through Arduino , or through Electric Imp , or through any other hardware platform compatible with Data Drop. Data can be transmitted via phones or other mobile devices. And sometimes already recorded data from another cloud can serve as a source.
We do not work with the lowest-level problems of receiving data from the device through wires, wifi, Bluetooth and whatever else.
There are various ways to access data between connected devices. Developers and researchers can access the Wolfram Cloud directly, or via the Wolfram Language desktop / online environment. Customer-focused companies associated with devices can work, for example, through their own private Wolfram Private Cloud . They can also access Data Drop using the Wolfram mobile app or their own program. Or through some kind of wearable device.
Sometimes a company needs to collect data from different devices. Suppose for monitoring a network or doing some research. And, again, their users may want to work directly with the Wolfram Language, or through the portal / application.
When I first started thinking through the concept of Wolfram Data Drop, I assumed that the data would mainly come from some kind of automated devices. But now that we have Data Drop, I realized that it would be very useful to work with the data that comes from people. This is a great way to collect answers, say, in a classroom, or collect feedback for a crowdsourcing project, store various personal notes and the like. Once the data semantics signature for the databin has been determined, Wolfram Data Drop can automatically generate a data presentation form that can be uploaded to the network or to a mobile device.
Forms can work with images, text, and many other things. In the case of text, our natural language interpretation system can retrieve data and interpret it in WDF and lead to standard form.
Now that we have Wolfram Data Drop, I continue to find various uses for this system and I don’t understand at all - how could I have done without this system before? As with everything in Wolfram Language, this is the history of automation: Wolfram Data Drop automates the entire erratic set of actions associated with collecting and processing real-world data from various sources.
The ultimate goal for me is the possibility of collecting and analyzing data of any kind without the need to create / use any other systems. For example, last week I finished using Wolfram Data Drop to collect performance data on our cloud. In a normal situation, this task would be too complicated and cumbersome, I would not even consider the possibility of doing it myself. But using Data Drop, installation and configuration took me only a few minutes, and now I can receive the data that interests me.
I look forward to the implementation of the things that I am going to implement in Wolfram Data Drop, exactly like to see how people use this system. You can already try out the beta version that we launched today and send us feedback (along with Data Drop databin, of course). I hope it will take a little time before the databins are woven into the infrastructure of the world: one more step forward in our long-term mission - to make the world computable ...
Source: https://habr.com/ru/post/253873/
All Articles