Development of IM for the competition of Pavel Durov using Xamarin

Good day.

As many probably know, Pavel Durov is developing a new clone of the What's App and other popular instant messengers based on his own protocol MTProto.

Recently, the American company released an iOS client for this protocol called Telegram. In parallel with this, a competition is held to develop an Android client .
Recently, the second stage was completed, the people sent their crafts and myself. I will say right away that I did not pass the second stage.
')
Unlike many participants, for development I used the language C # and Xamarin about what I want to tell more in detail below, since on Xamarin in the runet of information we will say just a little.

Instead of entry

I'm a doughter. I work with dotnet from the second version, when I was still a student, I know well the features and features of this platform. Not so long ago, I wanted to develop on mobile devices, the query "Android C #" and brought me to Xamarin - MonoDroid. But so far I have only written with him, it was the first serious project on Android, and I want to tell about it. Understanding this article requires knowledge of C #, .NET, and at least a primitive understanding of Android.

What is it - in a nutshell

Xamarin is a company (as well as a mobile platform) created by Nat Friedman and Miguel de Icaza , the author of GNOME and Mono. So Xamarin is a logical development for Mono.
Xamarin allows you to write native applications for Anroid and iOS in C # and that's fine. I personally believe that there is a future for a hybrid cross-platform. And Mac. And actively vote for MonoBerry , which may sometime be included in the Xamarin.

Task

In the tasks of the competition was the implementation of the provided protocol MTProto (first stage) and the creation of a full-fledged application (second stage). At the third stage, refinement.

The protocol as a whole is an RPC implementation with all sorts of buns like advanced encryption and all sorts of different things.

Decision

Hereinafter I will tell you how I solved these problems. So let's get started.

Getting Xamarin

Xamarin costs $ 2000. Yes it is. If you want to write in your favorite studio, its price is $ 999 per platform. If you have enough of a good environment MonoDevelop - its cost is $ 299 for the platform. In correspondence with the authors, I was able to beg for a discount of up to $ 799 for the platform.
How can you get xamarin? Well, for starters, you can download it from torrents . Xamarin offers an academic license for $ 99 for a platform that gives all Business opportunities except mail support. And yes, if your wife is a graduate student this also works.

Creating solution structure

As I mentioned, Xamarin creates native code for each mobile OS. This means that each OS will have its own build, but the code between them should be divided. The creators of Xamarin offer as many as three ways to do this, but for Visual Studio the simplest is the excellent Project Linker utility, which is built into the environment itself.
A couple of mouse clicks and a cross-platform solution is ready:



All files are connected as links, any changes in the main project will be displayed in all associated projects.



The utility is placed from the "Extension Manager" studio.

Solution structure

The main assemblies are MTProto.Core and Talks.Backend . These are common builds under .net 4.5 and covered with Unit tests.
Mono.Stub is a few specific classes from Mono, in particular, I use BigInteger from there.
Folder Droid - contains Androdov clones of projects, Dataflow is the TPL.Dataflow source code from github. I actively use in my project asynchronous capabilities C # 5.0.
In the Platfrom folder are specific implementations for each platform. So far this is only Android.

MTProto.Core

This is the implementation of the protocol. The protocol as a whole is RPC with advanced encryption and some additional features, such as forming a container or sending / receiving files in slices. Thus, to implement the IM client, we need to learn how to perform an RPC request, receive a response, and also process incoming system messages and status updates.

Of the features: async all the way and Dataflow.

async all the way

C # 5.0 has introduced a couple of keywords that extremely simplify the development of asynchronous code based on Task Asynchronous Pattern (TAP). They are very well described in MSDN .
All IO operations should be asynchronous, it should be as a commandment.

public async Task RunAsync() { await _cl.LoadSettings().ConfigureAwait(false); if (await _cl.CheckAndGenerateAuth().ConfigureAwait(false)) { await _cl.RunAsync().ConfigureAwait(false); } if ((_cl.Settings.DataCenters == null) || (_cl.Settings.DataCenters.Count == 0)) { await _cl.GetConfig().ConfigureAwait(false); } _db = await TalksDatabase.GetDatabase().ConfigureAwait(false); _ldm = new LocalDataManager(_db); _cl.ProcessUpdateAsync = ProcessUpdateAsync; } 

Stephen Cleary, one of the leading experts in asynchronous programming in C #, wrote several principles for using async-await, which have already become de facto standards for its use. If you have not read, I advise.

The essence of the “async all the way” approach is that all methods on the call tree are asynchronous from event to directly IO operation (in this case).

For example, if you need to asynchronously process a click on a button:

 async void button_Click(object sender, EventArgs e) { _button.Enabled = false; await _presenter.SendMessage(); } 

Then you make asynchronous all methods on the call tree in Presenter:



As you can see, not all methods are marked with keywords async-await. The general practice is this: if you do not need to do anything after an asynchronous call and if you have one asynchronous call, then it makes sense to simply return it as a Task from the method.
Another practice (also described in the Cleary article) is that asynchronous methods inside libraries should not capture the context and attempt to return to it after execution. Those. all asynchronous calls must contain .ConfigureAwait(false) done to prevent deadlocks. Read more about this in the article above.

Dataflow

TPL.Dataflow is a library designed to implement a Data Flow design pattern or processing pipeline. The source code of the library is available on github, which allows using it on mobile devices. Those wishing to learn more about the capabilities of this library send to MSDN .

In a nutshell, the library allows you to build a pipeline consisting of blocks for storing or processing data, linking them according to some condition. Initially, in my project there were two such conveyors: for the input and for the output packages. After refactoring, I decided to leave only one for incoming packets.

It looks like this:


and the creation process looks like this:

 BufferBlock<byte[]> _inputBufferBytes = new BufferBlock<byte[]>(); BufferBlock<InputTLCombinatorInstance> _inputBuffer = new BufferBlock<InputTLCombinatorInstance>(); ActionBlock<byte[]> _inputBufferParcer; ActionBlock<TLCombinatorInstance> _inputUpdates; ActionBlock<TLCombinatorInstance> _inputSystemMessages; TransformBlock<InputTLCombinatorInstance, RpcAnswer> _inputAnswers; BufferBlock<RpcAnswer> _inputAnswersBuffer = new BufferBlock<RpcAnswer>(); BufferBlock<RpcAnswer> _inputRejectedBuffer = new BufferBlock<RpcAnswer>(); BufferBlock<InputTLCombinatorInstance> _inputUnsorted = new BufferBlock<InputTLCombinatorInstance>(); // -- //   _inputBufferParcer = new ActionBlock<byte[]>(bytes => ProcessInputBuffer(bytes)); _inputSystemMessages = new ActionBlock<TLCombinatorInstance>(tlci => ProcessSystemMessage(tlci)); _inputUpdates = new ActionBlock<TLCombinatorInstance>(tlci => ProcessUpdateAsync(tlci)); _inputAnswers = new TransformBlock<InputTLCombinatorInstance, RpcAnswer>(tlci => ProcessRpcAnswer(tlci)); // from [_inputBufferBytes] to [_inputBufferTransformer] _inputBufferBytes.LinkTo(_inputBufferParcer); // from [_inputBufferTransformer] to [_inputBuffer] //_inputBufferTransformer.LinkTo(_inputBuffer); // if System then from [_inputBuffer] to [_inputSystemMessages] _inputBuffer.LinkTo(_inputSystemMessages, tlciw => _systemCalls.Contains(tlciw.Combinator.Name)); // if Updates then from [_inputBuffer] to [_inputUpdates] _inputBuffer.LinkTo(_inputUpdates, tlciw => tlciw.Combinator.ValueType.Equals("Updates")); // if rpc_result then from [_inputBuffer] to [_inputRpcAnswers] _inputBuffer.LinkTo(_inputAnswers, tlciw => tlciw.Combinator.Name.Equals("rpc_result")); // if rpc_result then from [_inputBuffer] to [_inputRpcAnswers] //_inputBuffer.LinkTo(_inputUnsorted); // and store it [_inputAnswers] to [_inputAnswersBuffer] to process it _inputAnswers.LinkTo(_inputAnswersBuffer); _inputRejectedBuffer.LinkTo(_inputAnswersBuffer); 

As you can see, input byte arrays are sorted out, classified and decomposed into buffers, from where they are sorted out as needed. In particular, updates and systemMessages are processed immediately upon arrival in ActionBlock , and rpcAnswers is first converted using TransformBlock and then added to BufferBlock . Classification of the packet type occurs inside the BufferBlock based on the conditions of linking blocks.

Immediately after calling the method, we create WriteOnceBlock - a block where you can write only 1 value:

 WriteOnceBlock<RpcAnswer> answer = new WriteOnceBlock<RpcAnswer>(e => e); 

And link it to the RPC response buffer:

 _inputAnswersBuffer.LinkTo(answer, new DataflowLinkOptions { MaxMessages = 1 }, i => i.SessionId == _em.SessionId); 

And then we wait asynchronously until the answer comes:

 return await answer.ReceiveAsync(TimeSpan.FromSeconds(60)).ConfigureAwait(false); //       

Separately, I want to note that up to this point I have not written a single line of code for Android. All development and testing was carried out for the usual build under .net 4.5

Talks.Backend

Client backend I decided to implement the client according to the MVP design pattern with IoC, and initially I aimed at the Passive View variation, when the view does not contain any logic, but in the end I came to understand that the Supervising Controller would work much better.

What problems arose before me when creating the backend? Access to the notebook, access to the database, access to the file system (for storing photos). The rest of the backend is an ordinary MVP implementation: a set of Presenters and IView's

Access to the notebook

To access the address book, the Xamarin team has already thought of everything for us. They developed the Xamarin.Mobile library which encapsulates a set of functions on a mobile device - notebook, GPS, camera, in a cross-platform manner. In addition, with full support for async-await.

Thus, access to the address book is extremely simple:

 #if __ANDROID__ public async Task GetAddressbook(Android.Content.Context context) { contacts = new AddressBook(context); #else public async Task GetAddressbook() { contacts = new AddressBook(); #endif if (!await contacts.RequestPermission()) { Trace.WriteLineIf(clientSwitch.TraceInfo, "Permission for contacts denied", "[ContactsPresenter.PopulateAddressbook]"); _view.SendSmallMessage("CONTACTS PERMISSON DENIED"); return; } else { _icv.PlainContacts = new ListItemCollection<ListItemValue>( (from c in contacts where (c.Phones.Count() > 0) select new ListItemValue(c)).ToList()); } } 

The compilation __ANDROID__ introduced because a context is required to get a list of contacts on Android, but not on other OSes.

One of the drawbacks of Passive View for a cross-platform solution is visible here. On the instructions we needed to group contacts by the first letter of the last name. For Android, this is done through the creation of the ListItemCollection class, which performs grouping, a classic example of this is available on the Internet. On iOS, a completely different approach to creating such a grouping is that on WinPhone - I do not know. So it is appropriate to receive and group contacts directly in the View.

This is the main problem in hybrid cross-platform development in my opinion. It should be clearly understood where you need to abstract from the platform, and where you should not. I think it comes with experience.

Database access

Access to the database Xamarin recommends through a simple ORM SQLite.Net. Once I tried to ignore these recommendations and work with the database directly through the driver, but in the end I realized that it was better to listen to the advice of more experienced developers.

I don’t see much point in describing how to work with SQLite.Net. I’ll just say that in order to test a build with SQlite.Net connected, you need to have sqlite binaries in your project, which are available on the official website www.sqlite.org/download.html

Separately, I note that SQLite.Net fully supports TAP and async-await.
I recommend expanding the SQLite.SQLiteAsyncConnection class with a set of Generic classes to simplify database access:

 #region Public Methods public Task<List<T>> GetItemsAsync<T>() where T : IBusinessEntity, new() { return Table<T>().ToListAsync(); } public Task<T> GetItemAsync<T>(int id) where T : IBusinessEntity, new() { return GetAsync<T>(id); } public async Task<bool> CheckRowExistAsync<T>(int id) where T : IBusinessEntity, new() { string tblName = typeof(T).Name; return await ExecuteScalarAsync<int>("select 1 from " + tblName + " where Id = ?", id).ConfigureAwait(false) == 1; } public async Task<int> SaveItemAsync<T>(T item) where T : IBusinessEntity, new() { if (await CheckRowExistAsync<T>(item.Id)) { return await base.UpdateAsync(item).ConfigureAwait(false); } else { return await base.InsertAsync(item).ConfigureAwait(false); } } public Task<int> DeleteItemAsync<T>(int id) where T : IBusinessEntity, new() { return DeleteAsync(new T() { Id = id }); } #endregion 

It is also worth remembering that the rules for access to the file system on each OS are different.

Therefore, the path to the database can be obtained as follows:

 public static string DatabaseFilePath { get { var sqliteFilename = "TalksDb.db3"; #if SILVERLIGHT // Windows Phone expects a local path, not absolute var path = sqliteFilename; #else #if __ANDROID__ // Just use whatever directory SpecialFolder.Personal returns string libraryPath = Environment.GetFolderPath(Environment.SpecialFolder.Personal); #else // we need to put in /Library/ on iOS5.1 to meet Apple's iCloud terms // (they don't want non-user-generated data in Documents) string documentsPath= Environment.GetFolderPath(Environment.SpecialFolder.MyDocuments); // Documents folder string libraryPath = Path.Combine(documentsPath, "..", "Library"); // Library folder #endif var path = Path.Combine(libraryPath, sqliteFilename); #endif return path; } } 

File system access

One of the objectives of the competition was the receipt and storage of images. To solve this problem, I took and refined the cross-platform disk cache class from here . In general, working with the file system is one of the least portable parts, since the requirements for working with files on all operating systems are different. Part of the features of file systems are described in the official docks of Xamarin.

Talks.Droid

Androyd version of the application. Ideally, by the time you create a project for a specific platform, you can have a fully working and tested backend. In my version it did not work out this way, but in the future I will strive for this.
The main difficulties begin here.

At the core of the application is the Bound Service to which the App class is binded - a singleton implementing the “application”. This is done in order for any Activity to access the service using App.Current.MainService .

Inside the service, a Model creates a separate thread, as well as there is a class with which the Activity retrieves its Presenters, like this:

 _presenter = App.Current.MainService.CreatePresenter<ChatListPresenter>(typeof(ChatListPresenter), this); 

It should be remembered that Xamarin forms AndroidManifest on its own and does not directly edit it. All Activity parameters are recorded as attributes:

  [Activity(Label = "Settings", Theme = "@style/Theme.TalksTheme")] [MetaData("android.support.PARENT_ACTIVITY", Value = "talks.ChatListActivity")] public class SettingsActivity : SherlockActivity, IView 

In general, the Activity code is not much different from the java version, CamelCase, and some getters / setters are wrapped in properties.

  protected override void OnCreate(Bundle bundle) { base.OnCreate(bundle); // Set our view from the "main" layout resource SetContentView(Resource.Layout.MessagesScreen); AndroidUtils.SetRobotoFont(this, (ViewGroup)Window.DecorView); _presenter = App.Current.MainService.CreatePresenter<MessagePresenter>(typeof(MessagePresenter), this); _presenter.PlatformSpecificImageResize = AndroidResizeImage; this.ChatId = Intent.GetIntExtra("userid", 0); userName = Intent.GetStringExtra("username"); _button = FindViewById<ImageButton>(Resource.Id.bSendMessage); _button.Click += button_Click; _button.Enabled = false; _message = FindViewById<EditText>(Resource.Id.etMessageToSend); _message.TextChanged += message_TextChanged; _lv = FindViewById<ListView>(Resource.Id.lvMessages); _lv.Adapter = new Adapters.MessagesScreenAdapter(this, this.Messages); } 

Login Activity

Login Activity must contain 3 items - enter the phone, get the code and enter it, register. For convenience, this is done using fragments.



The easiest way to do this with fragments. However, the use of fragments and MVP is not at all obvious !

In the end, I came to the fact that I did one presenter, and LoginActivity just wrapped the IView implementations:

 PhoneFragment _pf = null; CodeFragment _cf = null; SignUpFragment _suf = null; public string PhoneNumber { get { if (_pf != null) { return _pf.Phone; } else { return ""; } } } public string AuthCode { get { return _cf.Code; } } public string Name { get { return _suf.FirstName; } } public string Surname { get { return _suf.Surname; } } 

Photo / video shooting

An interesting and not obvious point. One of the tasks was to receive a photo / video from the camera to send it to the interlocutor or to install it as an avatar.

This is done through the menu using Xamarin.Mobile.

  public override bool OnMenuItemSelected(int featureId, Xamarin.ActionbarSherlockBinding.Views.IMenuItem item) { switch (item.ItemId) { // Respond to the action bar's Up/Home button case Android.Resource.Id.Home: NavUtils.NavigateUpFromSameTask(this); return true; case Resource.Id.messages_action_takephoto: _presenter.TakePhoto(this); return true; case Resource.Id.messages_action_gallery: _presenter.PickPhoto(this); return true; case Resource.Id.messages_action_video: _presenter.TakeVideo(this); return true; } return base.OnMenuItemSelected(featureId, item); } 

However, the event responsible for selecting the menu item returns bool, therefore we cannot apply the async-await construction to it. This is solved very simply, it should be remembered that async-await is just a syntactic sugar that eventually generates all the same Continuation. And nothing forbids us to write it as before:

 #if __ANDROID__ /// <summary> ///     /// </summary> /// <param name="context"></param> /// <returns></returns> public bool TakePhoto(Android.Content.Context context) { var picker = new MediaPicker(context); #else public bool TakePhoto() { var picker = new MediaPicker(); #endif if (picker.IsCameraAvailable) { picker.TakePhotoAsync(new StoreCameraMediaOptions { Name = String.Format("{0:dd_MM_yyyy_HH_mm}.jpg", DateTime.Now), Directory = "TalksPictures" }) .ContinueWith((prevTask) => { if (prevTask.IsCanceled) { _imv.SendSmallMessage("User canceled"); return; } if (PlatformSpecificImageResize != null) { string path = PlatformSpecificImageResize(prevTask.Result); //   DomainModel.Message msg = new DomainModel.Message(r.Next(Int32.MaxValue), 0, _imv.ChatId, _imv.PendingMessage, "", 0); _imv.AddMineMessage(msg); } }) .ContinueWith((prevTask) => { if (!prevTask.IsCanceled) { Console.WriteLine("User ok"); } }, TaskScheduler.FromCurrentSynchronizationContext()); return true; } return false; } 

Xamarin in addition to cross-platform offers the Component Store, where the ports of popular Android and / or iOS libraries and components, both free and paid, are located. In particular, ActionBar.Scherlok is present there and Android.Support.v7 recently appeared, and you can install components directly from the environment, like in NuGet, which is very convenient



Thus, in two clicks you can get ActionBar support on devices with Android 2.3 and higher.

Publication

The application is published according to the scheme approved by Google.

This involves quite a lot of action. But especially for us, the team from Xamarin made a wizard built into VS that allows you to prepare an application for publication in several steps.

and ready

True, I could not immediately create a KeyStore using this wizard. Something with the key lifetime was. I had to create pens.

Testing

A quick note on testing. Testing on an emulator is awful and impossible. This should be discarded as soon as possible. The cheapest androyd now costs 3,000 rubles, a Chinese tablet can be found at a similar price. With the start of the contest, I immediately bought my wife Fly with Android 4.0.1, since I only had an old HTC with 2.3.

About testing and development for iOS is more difficult. Of course, the best option is to take the cheapest MacBook, that will be enough.
But buying a pair of iPhone and iPAD for testing ... I do not know, not the best option. Now I am considering the possibility of MacInCloud and if everything is fine there, I will describe the whole process in detail.

Total

Now it is difficult to sum up. During the development process, I learned well the features of the Android platform,
developed a good, covered with tests and, most importantly, a cross-platform backend.
They say there will be a contest for WinPhone and iPad ahead. Well, I can only draw interfaces.

Bug work

"Note to self" as they say. Just comments on the future that I did wrong.
1. Lack of design. I double-refactored MTProto.Core almost completely. The reason for this is that I did not sit down with a piece of paper and did not draw completely how this core should look like. Many decisions were made spontaneously and without calculation for the future.
2. Poor understanding of the Android platform. For a long time I tried to understand how to organize interaction with the Android service. Frankly, I still do not know a better way to ensure this interaction. It is necessary to understand that the guides d.android.com are useless here, the service for the android is specific, and we need to get rid of the platform and do something cross-platform.
3. Stubbornness and greed. I had the opportunity to attract another programmer and maybe together we would have shown a better result. But I myself, all by myself.