DISTRIBUTED SYSTEMS AND BIG DATA

All the enterprise software systems and successful applications that we have today are wide-area distributed architecture which features a large quantity of data and codebase. This course aims to systematically present the main topics related to the design of these systems and to provide basic information on the related technologies  

In this rapidly changing scenario, the main objective is to help the students improve their skills to apply the fundamental algorithms and patterns to design large scale complex solutions. 

Knowledge and Understanding

Today there are many different continuously evolving solutions thanks to the research enterprise’s effort to meet the requirements of an ever-growing customer base. Thus, the goal is to give the students not only tools to meet today’s needs in the work environment but also to deal with the emerging challenges in the area of distributed software architecture maturing their abilities to adapt to novel solutions and technologies.

Among the main topics of the course:

• Models and tools for software development, delivery, and deployment.
• Working with *as a service (_aaS) approach.
• Algorithms and patterns for resource sharing, concurrency control, and distributed data management.
• Strategies for handling interoperability, QoS, Scalability,  Fault Tolerance.

•  direct and indirect communication 

• Transaction and Replica Management.
• Microservices and the relevant main patterns and technologies.
• Virtualization and containerization.
• monitoring and analysis of time series
• dtata pipelines for processing of Big Data.

Applied knowledge and Understanding

The intent of the course is to push the students to design software solutions and to improve their ability to autonomously engage in the design, development, and deployment, individually or in teams, of today’s real-world large distributed software architecture. The course uses practical activities and homework to achieve:

• The ability to apply the right strategies to guarantee adequate QoS in terms of performance, scalability, availability, safety, and robustness in distributed systems.
• Practical experience and know-how in the design and continuous development and deployment of microservices with container-based platforms (Docker) and Kubernetes orchestrators.
• Know how to design systems for big data management, integrating data pipelines, data lakes and data warehouses
• capability to evaluate the behaviour of a distributed application, utilise metering tools and evaluate performance and QoS

Making judgements

the student will be able to choices of the software architecture and the algorithms to design complex applications able to guarantee adequate QoS properties

Communication skills

the student will be able to rigorously present the knowledge acquired and with mastery of the terminology of Distributed Systems

Learning skills.

in the heterogenous and rapidly evolving scenario of DS  it is necessary to be able to independently learn new methodologies, patterns and algorithms emerging in the sector.

The objective is therefore to provide the student with the skills and ability to adapt their skills to new emerging design methods, evolving with the integration of new solutions and technologies.

Course is structured into lectures, lab exercises, seminars and presentation of real case studies.

knowledge of multithreading programming. 

Knowledge of network architectures and protocols, and in particular of the TCP/IP suite. DNS. 

knowledge of mySQL and web programming. Java and/or BSD sockets. 

Knowledge of software design techniques and fundamental GoF patterns. 

Knowledge of internetworking, web programming and firewall and security. 

Basic knowledge relating to the management of UNIX-Like systems, basic Unix Bash programming, SSH, nmap, iptables, Iperf.

attendance is not mandatory but strongly encouraged.

attendance (70%) is necessary only for students who intend to take ongoing tests

Distributed systems requirements: interoperability, separation of concerns & Transparency, Quality of Service (QoS), Scalability, Availability, and Fault Tolerance, Data consistency. CAP theorem.

Software design paradigms and models in large scale distributed systems - Patterns for distributed software architectures, components, application servers and dependency injection,  Service Oriented Architecture (SOA and XaaS). Localization: Naming, Addressing & Discovery, multitier systems and vertical and horizontal distribution; scaleup & scale out, message oriented systems.

Communication, interoperability – C/S and remote procedure calls (gRPC). Group communication, Multicast and Broadcast. Messaging systems Kafka). web socket (outline),

software architectures of large distributed systems: monolithic and microservice architectures. notes on monolithic enterprise systems (notes Maven and Spring).

Microservice architectures, hexagonal architecture, containerization. Docker. Docker Compose

DevOps. Orchestration. Kubernetes.

Big Data. Characteristics of Big Data. Big Data Storage. Partitioning, sharding. Saga. Data Stream processing.

Distributed Algorithms Synchronous Systems vs. asynchronous, clock models:,Lamport clock and vector clock,  event ordering and concurrency, global snapshot; dependability and Fault tolerance. Fault classification: crash and byzantine, agreement algorithms in the presence of various fault topologies, fault detector. cooperative algorithms. Election. Mutual exclusion, deadlock detection. Distributed transactions, serializability, recoverability and commitment. Consistency and Replication. Distribution of replicas. Synchronous vs Saga Transactions. Load Sharing. notes on distributed file systems

• “Distributed Systems: Concepts and Design”, G. Coulouris, J. Do llimore, T. Kindberg, G. Blair, 2011 – 5th edition”, Addison Wesley, 2011.

• “Microservices Patterns, with examples java”, Chris Richardson, Madison

for technologies and freamworks you can refer to the websites (as it will be during the course)

other textbooks you can consult 

• “Distributed Computing: Principles, Algorithms, and Systems”, Kshemkalyani, M Singhal,

• “Pattern Oriented Software Architecture” vol. 1-4 Buschmann, Schmidt et alii (particolare riferimento a vol. 2 e 4)

• “Distributed and Cloud Computing”, K. Hwang, J.Dongarra, G.C. Fox, Morgan Kaufmann, 2011.

• "Cloud Computing: Theory and Practice, Second Edition", Dan Marinescu, Morgan Kaufmann, 2018

• K.P. Birman, Guide to Reliable Distributed Systems: Building High-Assurance Applications and Cloud-Hosted Services”, Springer, 2012.

• “Kafka: The Definitive Guide” 2nd ed. G. Shapira, T. Palino, Rajini Sivaram, Krit Petty  O'Reilly Media; 2021.

• “Cloud Native DevOps with Kubernetes” 2nd ed, Justin Domingus, John Arundel
  

• “Kubernetes: Up and Running: Dive into the Future of Infrastructure”, K Hightower, O’Reilly 2017

learning assessment  

oral + project

the project is not mandatory: it can be requested 40 days before the exam date, will be assigned 30 days before that date. the student has to deliver the solution 1 week before that date. 

a profile of the features and requirements of the project can be found on studium.

The oral test focuses on the discussion of the project, and questions relating to the topics of the course program

the student can only take the oral exam, without project, but with a minimum sufficient evaluation of the project (i.e. it is evaluated with a default value of 18/30 which will be averaged with the oral evaluation and will allow final evaluation of 25/30 at most)

---

learning assessment  in-itinere

they are planned as 1 or 2 homeworks concerning design and DevOp to microservices and performance and QoS analysis.

the test will consist of 1/2 times of verification, homework discussion and oral test on the course topics; the homework must be delivered by January while the oral test must take place within the 1st session which ends in February.

---

If the teaching is given in mixed or remote mode, the necessary variations may be introduced on the ongoing test compared to what was previously declared; this test may not even be carried out