Optimized distributed large-scale analytics over decentralized data sources with imperfect communication

Tremendous amounts of data are generated by sensors and connected devices with high velocity in a variety of forms and large volumes. These characteristics, defined as big data, need new models and methods to be processed in near real-time. The nature of decentralized large-scale data sources requires distributed algorithms in which it is assumed that the data sources are capable of processing their own data and collaborating with neighbor sources. The network objective is to make an optimal decision, while the data are processed in a distributed manner. New technologies, like next generation of wireless communication and 5G, introduce practical issues such as imperfect communication that should be addressed. In this paper, we study a generalized form of distributed algorithms for decision-making over decentralized data sources. We propose an optimal algorithm that uses optimal weighting to combine the resource of neighbors. We define an optimization problem and find the solution by applying the proposed algorithm. We evaluate the performance of the developed algorithm by using both mathematical methods and computer simulations. We introduce the conditions in which the convergence of proposed algorithm is guaranteed and prove that the network error decreases considerably in comparison with some of the known modern methods.