System Design – Application like Twitter

Whenever we thing of designing a system we need to understand the requirement. We need to look at existing applications in market. Application cannot be created in one night. It needs proper planning on requirement, resource availability, time duration ,funds for the project etc.

Lets talk about twitter. What is twitter?

Twitter we can say is social networking site where people communicate with each other with short messages. We call these messages as tweet. These tweets can contain text, videos, photos or links. It allows people to share there thoughts with other people.

Functional requirement:

  • Tweet
  • Re-tweet
  • Follow
  • Search

Non-Functional requirement:

  • Heavy reading
  • Fast rendering
  • Always available
  • Fast tweet
  • Scalable (Number of users, Daily tweet, Monthly tweet etc.)
  • Low latency

Different type of users:

  • Well known people ( Celebrity, Player, Politicians etc.) They have huge followers
  • Active users
  • Passive users
  • Online users ( Available at that point)
  • Inactive/Deleted

Components that we can think off:

  • Different services like
    • User service to get sign up/login or get user details
    • Analytics service to do analysis
    • Graph service
    • User live (notification) service
  • Different databases
    • RDBMS/No SQL
    • In memory (Redis)
    • Cassendra or Hbase for images
    • CDN for distributing data
  • Search functionality
    • Elastic search
  • Load balancer
  • Kafka
    • Used to build real-time streaming data pipelines and real-time streaming applications

System should be designed such that it is loosely coupled. Initially  if team size is small, you can think of monolithic architecture but you need to keep in mind that later it will be decoupled.

 

Different type of databases

What are different type of databases?

1. Relational databases
-SQL language
-They are compliant with ACID

Examples – SQL Server, Oracle Database, MySQL, PostgreSQL

2. Cloud databases
-Database that’s designed to run in the cloud.
-cloud databases offer flexibility and scalability, along with high availability
Examples: Azure SQL Database, Amazon Relational Database

3. NoSQL databases
-Non-relational database
-To store unstructured or semi-structured data
-One advantage of NoSQL databases is that developers can make changes to the database on the fly, without affecting applications that are using the database
Examples: MongoDB,Apache Cassandra,CouchDB, and CouchBase

4. Key-value databases
-NoSQL databases
-Key-value databases are highly scalable and can handle high volumes of traffic
-Caching values

Examples: Amazon DynamoDB, Redis

5. Columnar databases
-store data in columns rather than rows
– it essentially ignores all of the data that doesn’t apply to the query, because you can retrieve the information from only the columns you want
-Ever increasing data

Examples: Google BigQuery, Cassandra, HBase, MariaDB

6. Document databases
-document stores, use JSON-like documents to model data instead of rows and columns
-Document databases are simple and scalable, making them useful for mobile apps that need fast iterations.

Examples: MongoDB, Amazon DocumentDB, Apache CouchDB

7.Time series databases (Application pushing matrix)
-database optimized for time-stamped, or time series, data

Examples: InfluxDB, Druid,Open TS DB

8. Data ware house/Offline reporting
When you have lot of information for analysis
Example : Hadoop (Amazon/Uber data)

9. Search relevant text
Elastic search (search engine)
Solr

10. Distributed database
Distribute images to different server across globe
Example : CDN

11.Blob storage
Storing video’s/images (amazon/Netflix)
Example Amazon S3

Dispose Vs Finalize

What are unmanaged resources?

File handler, Database connection, socket connection are example of unmanaged resource.

.Net garbage collector is designed to handle managed code. We need to implement IDisposable interface when we are using unmanaged resource directly.

There are 2 ways to call dispose method:

  1. Using : If object is implementating IDisposable interface we can use Using keyword, which will call objects Dispose method.
  2. Try/Catch/Finally: We can call Dispose in finally.

IDisposable and the inheritance hierarchy
A base class with subclasses that should be disposable must implement IDisposable as follows.
You should use this pattern whenever you implement IDisposable on any type that isn’t sealed.

It should provide one public, non-virtual Dispose() method and a protected virtual Dispose(Boolean disposing) method.

The Dispose() method must call Dispose(true) and should suppress finalization for performance.

The base type should not include any finalizers.

To implement Dispose method for custom class, you need to implement IDisposable interface.
IDisposable interface expose Dispose method where code to release unmanaged resource will be written.

Finalize

Finalize method is called destructor of the class. Finalize method can not be called explicitly in the code. Only Garbage collector can call the the Finalize when object become inaccessible. Finalize method cannot be implemented directly it can only be implement via declaring destructor.

Rule

For a class owning managed resources, implement IDisposable (but not a finalizer).
For a class owning at least one unmanaged resource, implement both IDisposable and a finalizer.

To avoid code duplication, Dispose() and the finalizer should be implemented like this (in pseudo-code):

public void Dispose() {
Dipose(true);
GC.SuppressFinalize(this);
}

GC.SuppressFinalize() simply prevents the finalizer from being called. Since the finalizer’s only task is to free unmanaged data, it doesn’t need to be called if Dispose() was already called (and already freed all unmanaged data by calling the finalizer).
Using GC.SuppressFinalize() give a small performance improvement but nothing more.

The default dispose implementation pattern used in the previous sections create a method called Dispose(bool). This method is protected virtual and is meant to be overridden by child classes – in case they need to dispose some data of their own.

In C#, an implementation must:

first check whether it already has been disposed
then dispose everything
and then call the base method

The base method is called last to ensure that child classes are disposed before their parent classes.

Garbage collector in .Net

Garbage collector is .net way of memory management of managed resources.

GC is not able to release memory from unmanaged resource like File handlers, window handlers, network sockets, database connections etc.

System.GC.Collect() forces garbage collector to run. This is not recommended but can be used if situations arise

Threading in C#

What is Process?

Process is nothing but executing program. Process has a unique process ID. We can view the process within which program is being executed using windows task manager.

Here we can see different programs running under different process.

Another example is calculator program. When we start calculator program, calculator process will start.

What is Threading?

Thread is light weight process. A single process can have multiple threads.

Threads are the smallest unit of execution scheduled by the operating system.
Any program or application has a minimum of one thread be it web application or a console window application.

There are 2 types of threads:

  • Foreground Thread
  • Background Thread

Main purpose of threading is do parallel code execution. By default foreground thread is created. Even if main application quits, then also foreground thread will continue to execute.

In case of background thread,when main application quits, background thread quits.

Namespace: System.Threading

Purpose: To make Application responsive.

How to debug Thread?

  • Name the Threads
  • When we are debugging 1 thread another thread is paused/halt
  • If we want to debug 1 thread and pause another thread using freeze option and start it using Thaw option.
  • Use break point option- conditional debugging.

Thread Safe:

If object is behaving abnormally in multi threaded environment, it is not thread safe.

Example: Divide by number program in loop where we set Num1 = 0 and Num2 = 0 after Num1/Num2.
Here  DividebyZero exception is raised.

So user needs to ensure that only 1 thread is accessing the critical section.

Different types of locks:

  • Lock(Monitor)
  • Mutex
  • Semaphore

Mutex:

  • Mutex works across multiple programs/Process.
  • We can have global mutex.
  • Mutex are objects that are owned by owned by thread at a time. Other threads are forced to wait until first thread releases it.

Example

There is 1 room having 1 key to it.Now 3 persons want to use it. Person who has key can access the room.

Similarly Mutex is object owned by thread so there is ownership in mutex. Mutex allow 1 thread to access the resource.

Semaphore:

  • A Semaphore restricts the number of simultaneous users of a shared resource up to a maximum number.
  • Thread can request access to the resource(decrementing the semaphore) & can signal that they have finished using the resource(incremeting the semaphore)
  • Semaphore is signalling mechanism. It allows a number of thread to access shared resources.

Example:

A  semaphore is like nightclub, it has certain capacity enforced by bouncer. Once it is full no more people can enter & a queue is build up outside. Then for each perosn that leaves, one person enters from the head of the queue.

TPL: Task parallel Library

Problems around Threading:

We want to run the logic parallely and it should use the CPU power to maximum. If my machine is 2 core processor then it should run half the iterations on 1 processor and another half on second processor. However that doesn’t happen and everything runs on 1 processor.

This can be verified using perfmon tool.

Single processor ->
Thread 1 and Thread 2 – It switches time between threads. it is called time slicing or context switching.

Two Processors-> Expectation is Processor 1 should execute Thread 1 and Processor 2 should execute Thread 2.
Processor 1 -> Thread 1
Processor 2 -> Thread 2

Namespace: System.Threading.Task

Purpose: To utilize CPU power to maximum. TPL encapsulates multi-core execution from end user.

How TPL does thread pooling?

Designing an online booking system

When we want to design something, there are multiple things that needs to be considered.

  1. Use case: How many users will be using it?
  2. Traffic: What is the estimated traffic?
  3. Database – SQL/NoSQL

Primary design goals:

  • Highly concurrent
  • Database ACID compliant
  • Distributed message queue to push notifications





We need following things when we think about a online booking system software:

  1. User Interface
  2. Database
  3. Caching
  4. Logging
  5. Load balancing
  6. Payment API
  7. Notifications

Thinking in terms of Objects:

  • Movie
  • City
  • Theatre
  • Show
  • User
  • Booking
  • Seat





What does these objects consist of?

  • Movie : MovieID ,Name, Description
  • City: CityID, CityName
  • Theatre: TheatreID, Name, Address, CityID
  • Show: ShowID,TheatreID, CityID, MovieID, Date, Timing
  • User: UserID, Name, Email, Phone
  • Booking: BookingID, UserID, ShowID, Amount, Date
  • Seat: SeatID, BookingID, SeatStatus

 

Popular online booking system: Book My Show.



Object Oriented Programming Concept

What is OOP?
Object oriented programming is a way of thinking. It is one of the software development approach. Thinking real world items/things in terms of object.

Different ways of programming

Programming started with the invention of computers. Programming is nothing but giving instructions to the computer or machine.

  • Machine language (bits)
  • Assembly Language (Example ADD A,C)
  • Structured programming( if- else, functions) Emphasis on functionality and not on data. Sharing data globally
  • Object Oriented Programming

Example :






Badminton court online booking.

There are basically 2 categories of people: 1 Technical 2. Non-Technical. It is not necessary that person who gives you requirement as technical back ground. It becomes difficult for non-technical person to interact with technical person.

Technical person thinks in terms of language to use(Asp.net, Angular js), database (oracle, Mysql, SQL), hosting environment etc.
Non technical person thinks in terms of users, number of courts, availability, security guards, light supply etc.

In real life we talk about objects and not functionality. In day-to-day life we come across multiple objects like TV, Fan,watch, Car,AC etc.All these are objects and we are thinking in terms of object. So there was need to correlate/represent these objects with software to be developed.

Court ——————-> Court object

User ———————> User Object

Booking —————–> Booking object

Way of simulating real life thinks into code is nothing but object oriented programming.


Object Oriented Programming Pillars

    • Objects : Object is instance of a class.
      • Attributes and functionality(behavior)
      • Example Court
        • Court attributes: Size/Dimension, surface(grass, tiles, soil, cement),  color
        • Functionality: DoCourtBooking()/ DoCourtCleaning()
      • Attributes are variables or data-members and Functionality is function.
    • Class :
      • Class is blueprint of object. It is specification of the object. How object will look like.
      • Example Student:
        • Attributes: Name, marks,RollNo
        • Functionality: SetMarks(),GetMark(), ExtracurricullarPerformance()
        • Change in attribute results in change in behavior. As the students mark changes, his performance improves.
      • Example 2 – Transaction class
        • Attribute: TransactionID, Amount, TransactionYear
        • Functionality: AddTransaction(), UpdateTransaction(), DeleteTransaction()




  • Abstraction:
    • It is way of showing the necessary details and ignoring the rest.
    • Example Student class:
      • Attributes: There are multiple attributes of student like his Name, RollNo, height, weight, address, hobbies, friends, Age, Marks, shoe size etc. Out of this we need to select few which are required for our application.
      • Here we select Name, RollNo, Marks – This is one view point of abstraction to pick required attributes
      • Behaviour: Communction skills, punctionality, Sports, curricularPerformance etc.
      • Here we pick curricularPerformance  and extra- curricularPerformance
    • For developers pick relevant attributes and function
    • Second part of abstraction – API development
      • We don’t know the implementation part nor bother about it.
  • Encapsulation:
    • Data and functionality is encapsulated within class.
    • Controlling the access of data using access specifier.
    • Capsule :
    • Correlation: Medicine Capsule:
      • In order to prevent any reaction with atmospheric ingredient, medicine content is added in capsule for protection.
    • Similarly in software application we need to protect data.
    • First we put attributes and function within class
    • Second we use access specifier to achieve encapsulation.
    • Different access specifier: Private, protected, internal, public.
    • Ideally attributes are marked private and required functions are made public.
    • Example: AC
      • We have remote as interface to increase temp, decrease temp, on/off
  • Polymorphism
    • Having many forms
    • Same function applied on different object gives you different results.
    • Static polymorphism:
      • Function overloading
      • Operator Overloading
    • Runtime polymorphism
      • Virtual method
    • Example Washing vegetables:
      • Washing potatoes, brinjal and tomatoes.
      • Same command of washing is different for each vegetable based on its class.
  • Inheritance
    • Deriving a class from another class is called inheritance.
    • Creating object using the properties of parent is called inheritance
    • In real life there is relation ship. object works in terms of relation ship
      • Country – Citizen, Parent- Child, Teacher – student
    • Inheritance is way to simulate relationship.
    • Re-usability + Extensibility




Coding best practices !!!

What is good code?

Code can be called as good code when it is

  • Understandable
  • Readable
  • Reusable
  • Maintainable
  • Reviewed



How to write good code? What are coding best practices?

In order to write good code, we need to follow/define coding best practices which should be followed by Junior Developer to Sr. Developer.

Below are some of the coding best practices i have come across:

  1. Naming convention: Decide which name convention to follow for variables/Methods/Classes etc
  2. Code refactoring: Remove dead code, duplicate code/Make reuse of code
  3. Simple code design: Keep design simple so that it is understandable to everyone and easy to maintain. Keep method/class size small.
  4.  Pair programming: In case of complex code or when there are multiple scenarios to be implemented/tested, its better to do pair programming
  5. Continuous Integration: Make sure latest code is available to everyone working on project so that there are no integration issues later.It is always good practice to have common code base.
  6. Test driven development:Use tools like Nunit
  7. Making use of code review tools: Use tools like Fxcop, Stylecop or resharper for code reviews.




Test first vs Debugging approach:

It’s always good to have test driven development so that we have test cases ready and can be used by everyone. Debugging efforts are wasted and only helps for person who is working on it  to write quality code. If same person continues to work on that module, he will not face much problem in identifying and verifying the scenarios.However there is no guarantee that all the scenarios will be tested.




Asp.net web API

ASP.Net Web API is a powerful platform/framework for creating HTTP enabled service APIs (web services) that exposes data services to more clients including mobile devices, tablets, browsers and traditional desktop applications.

ASP.Net Web API uses all the features of HTTP like URIs, request/response headers, various content formats caching, versioning and there is no need to define any extra configuration settings for different devices like WCF rest services.

ASP.Net Web API communicate with HTTP/HTTPs

The basic CRUD operations are mapped to the HTTP protocols in the following manner:
GET: This will be used to retrieve the required data from the remote resource.
POST: This will create a new entry for the current data that is being sent to the server.
PUT: This will update the current representation of the data on the remote server.
DELETE: This will delete the specified data from the remote server.

Features of ASP.NET WEB API
-It can be hosted with in the applicaion or on IIS.
-It has automatic support for OData.
-It supports convention-based CRUD Actions

Basics of HTTP protocol

What is HTTP protocol?
HTTP stand for Hypertext transfer protocol.
It is application level protocol
It is client server protocol

Request/Response

Requests are sent by one entity like web browser and response is sent by another entity like web server.
Between this request and response there are numerous entities, collectively designated as proxies, which perform different operations and act as gateways or caches.
In reality, there are more computers between a browser and the server handling the request: there are routers, modems, and more.

How does HTTP protocol works?

HTTP uses the TCP transport protocol to make this connection. By default, web traffic uses TCP port 80

Basics of HTTP protocol:
It is simple – Human readable
It is extensible
It is stateless but not sessionless
It mostly uses TCP transport protocol