Simple file hosting on Google App Engine
Surely each of you in your life found a convenient file hosting service, and after some time found out that his eyes hurt from the amount of advertising, the conditions are far from being so loyal and in general it would be time to find something new. There are two options for further actions - either to find a new, not yet promoted file sharing service and use it until it deteriorates, or organize your own solution. For the second option, in turn, you can purchase hosting (you really have to fill the cones until you find a bona fide hoster with quality services) or use the cloud service.
Quite an interesting find was PaaS hosting from Google - Google App Engine (hereinafter referred to as GAE), which makes it possible to store up to 5 GB of files with 1 GB of incoming and 1 GB of outgoing traffic per day, and among other things, it uses the High Replication model, that is, your data will be stored on several servers all over the world at once!
A special feature of GAE is a somewhat non-standard interface for working with files, which is why I made my own service, which I will discuss in this article.
GAE makes it possible to create applications in Java, Go and Python. Since the first two languages ​​are almost unfamiliar to me, I wrote my service in Python. Previously, it was possible to use only version 2.5, which created certain difficulties, but recently added support for 2.7, so there is no need to recall outdated approaches.
To start using GAE, download and install Python version 2.7, as well as the appengine SDK for your operating system. The narration will be conducted on the example of the Linux SDK version (there is a user-friendly graphical interface in Windows and Mac, so it will be easy to understand, although you can do everything from the console, as described below).
')
To make it easier in the future with the settings, it is recommended to register the application in advance at http://appengine.google.com by clicking the “Create application” button. You will be prompted to enter a unique identifier for the application, according to which it will be available as a subdomain of appspot.com, as well as a name that can be changed in the future, unlike the identifier. All other settings can not touch, in our case they do not have much value. To get started, create a folder with a name that corresponds to your ID in which all files related to the application will be stored.
Any application consists, at a minimum, of the file app.yaml, which contains the name of the application, the version of the executable environment, and a description of URL handlers and various errors. A URL handler consists of a regular expression, with which links are checked, as well as descriptions of necessary files. I note that in the names of scripts for handlers, now we need to put an extension (which is not an extension, but an object in the application) not '.py', but '.app'
For our application, the app.yaml file looks like this:
App Engine has a special Blobstore for storing files, and Datastore for data (I note that the non-relational data storage model is used; there is a framework for data access that has its own framework similar to Django's ORM). In our application, each file is assigned a FileRecord object in the Datastore with a built-in unique integer identifier (meaning retrieving the object key by id), which contains the BlobReferenceProperty object:
For more specific URL parsing and, in general, to initialize the application, an app object is created at the end of the file, which is a representative of WSGIApplication:
Depending on the received URL, the application starts the corresponding handler. In this case, the handlers from app.yaml are partially duplicated, since it contains the settings for authorization.
The application works as follows:
To test the application, run the debug server in the console:
If you want the server to be accessible over the network, add the --address 0.0.0.0 parameter after dev_appserver.py.
If there are no problems, you can start downloading the application:
Then you will be asked for your login and password to your Google account, after which the application will be downloaded and will be available at appid.appspot.com , where appid is your unique identifier.
Application source code on code.google.com (to use, simply edit the app.yaml file): http://code.google.com/p/fileshare-appengine
Advantages over existing file hosting services:
UPDATE: If you get an error like UnicodeDecodeError at the stage of loading the script in Windows, it may help to delete all keys that have Cyrillic from the HKEY_CLASSES_ROOT / Mime / Database / ContentType registry key
UPDATE2: Now the source can be viewed at http://code.google.com/p/fileshare-appengine/source/browse/
UPDATE3: as well as on GitHub https://github.com/SergeyBurma/fileshare-appengine
Quite an interesting find was PaaS hosting from Google - Google App Engine (hereinafter referred to as GAE), which makes it possible to store up to 5 GB of files with 1 GB of incoming and 1 GB of outgoing traffic per day, and among other things, it uses the High Replication model, that is, your data will be stored on several servers all over the world at once!
A special feature of GAE is a somewhat non-standard interface for working with files, which is why I made my own service, which I will discuss in this article.
GAE makes it possible to create applications in Java, Go and Python. Since the first two languages ​​are almost unfamiliar to me, I wrote my service in Python. Previously, it was possible to use only version 2.5, which created certain difficulties, but recently added support for 2.7, so there is no need to recall outdated approaches.
To start using GAE, download and install Python version 2.7, as well as the appengine SDK for your operating system. The narration will be conducted on the example of the Linux SDK version (there is a user-friendly graphical interface in Windows and Mac, so it will be easy to understand, although you can do everything from the console, as described below).
')
To make it easier in the future with the settings, it is recommended to register the application in advance at http://appengine.google.com by clicking the “Create application” button. You will be prompted to enter a unique identifier for the application, according to which it will be available as a subdomain of appspot.com, as well as a name that can be changed in the future, unlike the identifier. All other settings can not touch, in our case they do not have much value. To get started, create a folder with a name that corresponds to your ID in which all files related to the application will be stored.
Any application consists, at a minimum, of the file app.yaml, which contains the name of the application, the version of the executable environment, and a description of URL handlers and various errors. A URL handler consists of a regular expression, with which links are checked, as well as descriptions of necessary files. I note that in the names of scripts for handlers, now we need to put an extension (which is not an extension, but an object in the application) not '.py', but '.app'
For our application, the app.yaml file looks like this:
application: fileshare # fileshare version: 1 runtime: python27 api_version: 1 threadsafe: true handlers: - url: /favicon\.ico static_files: favicon.ico upload: favicon\.ico - url: /delete/.* # script: main.app login: admin # . - url: /get/.* # , , login: admin login: required, Google . script: main.app - url: .* # , . script: main.app login: admin App Engine has a special Blobstore for storing files, and Datastore for data (I note that the non-relational data storage model is used; there is a framework for data access that has its own framework similar to Django's ORM). In our application, each file is assigned a FileRecord object in the Datastore with a built-in unique integer identifier (meaning retrieving the object key by id), which contains the BlobReferenceProperty object:
class FileRecord(db.Model): blob = blobstore.BlobReferenceProperty() For more specific URL parsing and, in general, to initialize the application, an app object is created at the end of the file, which is a representative of WSGIApplication:
app = webapp2.WSGIApplication( [('/', MainHandler), ('/upload', UploadHandler), ('/delete/([^/]+)?', DeleteHandler), ('/get/([^/]+)?', GetHandler), ], debug=False) Depending on the received URL, the application starts the corresponding handler. In this case, the handlers from app.yaml are partially duplicated, since it contains the settings for authorization.
The application works as follows:
- When accessing '/', the user is authorized, then all the records are loaded and a form for downloading the file is displayed.
- When accessing '/ upload', which occurs automatically when uploading a file, an object is created in the Blobstore, which is associated with the object in the Datastore, then in case of success, the transition is made to '/'.
- When referring to '/ delete /', the object number is extracted from the URL, after which the object is deleted.
- When accessing '/ get /', the object number is also extracted from the URL, after which the file associated with it is sent for download.
To test the application, run the debug server in the console:
python google_appengine/dev_appserver.py < > If you want the server to be accessible over the network, add the --address 0.0.0.0 parameter after dev_appserver.py.
If there are no problems, you can start downloading the application:
python google_appengine/appcfg.py update < > Then you will be asked for your login and password to your Google account, after which the application will be downloaded and will be available at appid.appspot.com , where appid is your unique identifier.
Application source code on code.google.com (to use, simply edit the app.yaml file): http://code.google.com/p/fileshare-appengine
Advantages over existing file hosting services:
- No need to watch ads, wait a minute, etc.
- The file can be downloaded for a direct link.
- Very high reliability.
- Only you manage what is uploaded to your file sharing service. It will not be closed due to another user.
- Within the specified limits of resource use, everything is free.
- Even if the allocated resources are few, you can make another application (these are already personal speculations, there is nothing in the rules specifically for this case, so it’s better to find out. UPDATE: it’s still prohibited by the rules).
- Not quite clear about the maximum file size - in the Russian documentation 2 GB is written, and in English - 32 MB.
- It turned out that Russian file names are not supported. I do not quite understand why.
UPDATE: If you get an error like UnicodeDecodeError at the stage of loading the script in Windows, it may help to delete all keys that have Cyrillic from the HKEY_CLASSES_ROOT / Mime / Database / ContentType registry key
UPDATE2: Now the source can be viewed at http://code.google.com/p/fileshare-appengine/source/browse/
UPDATE3: as well as on GitHub https://github.com/SergeyBurma/fileshare-appengine
Source: https://habr.com/ru/post/137797/
All Articles