Dropbox Crawler

Personal cloud storage is becoming more and more popular - Dropbox is certainly the best known example. Cloud storage already generates a huge amount of Internet traffic. Because of that, understanding how people interact with such applications is essential for designing efficient cloud storage systems.

We have been doing research on the usage of Dropbox (see our results here). As a next step, we need to know what type of files people store in the service. This would allow us to understand the impact of some technologies on the system performance and on network traffic, among other things. For that, we need volunteers to provide us some basic statistics (size, type etc) about files stored in their folders.

Be part of the crowd: Help our research

To Run our crawler, you may try do load it directly from our page, clicking here. Or you may download the Jar package and run it (double click on most OS or java -jar HelpOurResearch.jar)

Lauch Java Application

What will be captured?

What we do:

We will read all your DropBox Folder; We will collect basic statistics (log format can be viewed in the following); We will send these statistics to our web server.

What we DO NOT do:

We do not copy any file content; We do not copy file or folder name; We do not copy any personal information; We do not install or store anything in your computer.

Client source code

Download the Java Source Code to Capture Files Information The Project may be used direct in NetBeans, version 7.2.1

Policy

We ensure that:

All data we collect are anonymized. We do not copy any file content. We do not collect any personal information and file/dir names.

We also will make our data publicity in a near future. Thus, anyone will be able to use this important data source.

Format

All files are in a simple format. Each line has files attributes, separeted by #.

The following columns are found in these traces:

############################################################################
#     #     # Short description      # Unit  # Long description            #
############################################################################
#  1  #     # Lenght                 # -     # File Size in Bytes
#  2  #     # Modified               # -     # Last modification on file (Unix date/time format)
#  3  #     # MIME                   # -     # File Mime Type using Magic Java Unit
#  4  #     # EXTENSION              # -     # File extension (substring after the last "." on the string)
#  5  #     # MD5                    # -     # MD5 hash code of the initial/final 8 bytes of the file.
#  6  #     # MD5 of the name        # -     # MD5 hash code of file name string.
############################################################################

More information

You may find Dropbox information on our previous work

Drago, I. and Mellia, M. and Munafò, M. M. and Sperotto, A. and Sadre, R. and Pras, A. (2012) Inside Dropbox: Understanding Personal Cloud Storage Services. Proceedings of the 12th ACM Internet Measurement Conference - IMC'12, Boston, Nov. 2012

As described in the paper, the data was captured at 4 vantage points in 2 European countries. The first 4 files were collected from March 24, 2012 to May 5, 2012. A second dataset was collected in Campus 1 in June and July 2012 to complement the analysis.

The data was captured using Tstat: An open source monitoring tool developed at Politecnico di Torino. Tstat exports flow data containing more than 100 metrics. The source code of Tstat can be obtained from here. More information about the DN-Hunter version of Tstat, needed for some experiments, can be found here. Note that all IP addresses in the datasets are anonymized