Data Warehose concept

RDBMS record-oriented structure

The record-based structure is the most common choice for data warehouse applications.

In this structure, data is stored in physical records using the common physical location of

data values as the logical connection across all data points of the individual record. For

example, to find a customer address, the system must first locate the customer using an

indexed value such as customer number, then scan across the record to the position for

address. As a result, the smallest addressable unit of storage is an entire record, and all

physical I/O functions will always move complete records or sets of complete records.

Since entire records are physically written and read as single units, selecting any field

requires reading the entire record, and changing a single value requires reading and

rewriting the entire record. If the new value requires more space than the original,

extensive additional I/O is needed. The modified record may be written into an overflow

area, forcing either multiple read operations at query time or reorganizing and rewriting the

previous area.

Columnar database column-based structure

The newer column structure has gained interest as the indexing and data transfer

problems associated with record structures have proved problematic for analytics

applications. The column-based DBMS stores all of the values from one column of a table

in a contiguous data set. This allows the reading and/or writing of parts of records. It

conserves I/O bandwidth by transferring only the values that may be used in the query.

Since most data warehouse applications use only a few columns from a table during a

typical single access, the resulting bandwidth savings can be substantial.

Writing new records requires updating each of the column sets that belong to the record.

Reading a complete record requires locating the correct value from each column set and

assembling those into the original record format. Therefore, full record operations show

little improvement over record-based systems and, in some cases, even can perform more

slowly.

Correlation DBMS value-based structure

The most recent innovation, value-based storage (VBS)—used in the correlation database

management system (CDBMS)—also avoids the indexing problems of record structures,

but in addition, it has exploratory and analytical features not available with either recordoriented

or column-based structures. The value-based model stores each unique value

once, then uses extended metadata to maintain the information needed to manage record

formatting. VBS minimizes bandwidth consumption, as the actual data value storage is so

small that the entire set of all data values can frequently stay in memory, affecting I/O

bandwidth only when the system is started.

Relational Database (RDBMS)

There are many commercial variations of record-based databases available. All share the

common characteristic of storing related values in a physically contiguous record. The

most commonly used form used for data warehousing and analytics is the RDBMS, which

has dominated the entire database market for years due to its strengths in processing and

storing transactions and in generating reports.

Columnar Database

A columnar database stores data much differently than does a RDBMS. In this structure,

data is stored in sets by column—that is, all of the values stored in column one are stored

in one set, all the values in column two in another set, and so on. In addition to the values,

the information needed to reinsert them into the proper position in the original record

format is stored with each set.

Correlation Database (CDBMS)

The newest entry in the data warehouse platform arena is the correlation database

(CDBMS) with its value-based storage (VBS) model. This structure stores data values

independently from the logical schema and uses extended metadata to maintain the

information needed to reconstruct the original records. VBS offers performance equal to or

better than the columnar structures, plus additional features not provided by either

RDBMSs or columnar databases. In VBS, each unique data value is stored only once,

making the database extremely compact and fast.

Although in my point of view  relational databases (RDBMS) are the most common choice for data warehouse implementations, their record-based structure is far from ideal. There are many commercial variations of record-based databases available. All share the

common characteristic of storing related values in a physically contiguous record. The

most commonly used form used for data warehousing and analytics is the RDBMS, which

has dominated the entire database market for years due to its strengths in processing and

storing transactions and in generating reports.

 As data volumes grow and users demand more sophisticated analytical capabilities, the deficiencies of the RDBMS to data storage become more conspicuous.

Conclusion

Despite their limitations, rrecord-based relational databases have long been the prevailing

data structure used in data warehouse systems. However, as data volumes have

increased and analytical needs have become more sophisticated, the shortcomings of the

RDBMS—most significantly, the design compromise required between optimizing for

performance versus query flexibility—have spurred the development of alternative data

structures.

Columnar databases offer faster query performance and require less disk storage space

than do RDBMSs, but they force their own compromise between optimizing for new record

insertion versus record selection and retrieval.

The newest data warehouse database, correlation DBMS, uses a value-based storage

(VBS) model that eliminates design compromises. The upfront physical design effort is

minimized since the CDBMS is optimized for both query performance and flexibility

automatically as data is loaded. The correlation DBMS provides load speeds comparable

to RDBMS and columnar systems, minimizes disk space requirements, optimizes query

performance, and enables unique types of queries—such as associative and incremental

queries—that are impractical or impossible with other data structures.

Concepts about Internet

CSMA/CD

There is no central control management when computers transmit on Ethernet.

For this purpose the Ethernet employs CSMA to coordinate transmission among multiple attached computers.

CSMA is a coordination scheme that defines how to take turns using a shared cable.

A computer listen to codes i.e. it sense the carrier. If the cable is idle it starts transmitting and if the cable is in use then it waits.

If simultaneous transmission occurs, the frames interfere with each other  and this phenomenon is called collision.

COLLISION DETECTION:

As explained earlier, the signals from two computers will interfere with each other and the overlapping of frames is called a collision. It does not harm to the hardware but data from both frames is grabbled.

ETHERNET CD:

To detect the collision , Ethernet interfaces include hardware to detect transmission. It performs two operations:

It monitors outgoing signals.

Grabbled signal is interpreted as a collision.

After collision  is detected computers stop transmission . Internet uses CSMA/CD to coordinate transmission.

All wiring technologies use identical Ethernet specifications .e.g they use same frame format. They use same CSMA/CD algorithms. They can mix different technologies in a Ethernet.

For Example :

In a bus topology all computers are attached to single  long cable and any computer can send data to any other computer.

For this purpose , coordination is required to decide which computer has to use the line at what time. In this situation , they use CSMA/CD to sense the carrier if line is idle , they transmit their data to other computer attached on this line.

Recovery From Collision :

Computer that detects sends  special signal to force all other interfaces to detect collision.

Computer then waits for other  to be idle before transmission . But if both computers wait for same length of time, frames will collide again. So the standard specifies maximum delay and both computers choose random delay, which is lesser . After waiting, computers use carrier sense to avoid subsequence collision.

After waiting, computers use carrier sense to avoid subsequence collision.

The computer with shorter delay will go first and other computer may transmit later.

Exponential Back Off:

Even with random delays, collision may occur especially likely with busy segments. Computers double delay with each subsequent collision. It reduces likely hood of sequence of collision.

Data Stuffing

Sometimes the special character (i-e soh and eot) may appear in data as a part of data they will be misinterpreted as framing data.

The solution to this problem is Byte stuffing.

In general  to distinguish between data being sent and control information such as frame delimiters network systems arrange for the sending side to change the data  slightly  before it is sent because systems usually insert data or bytes to change data of transmission , the technique is known as Data Stuffing.

There are two types of data stuffing:

Byte Stuffing

Bit Stuffing

Byte stuffing refers stuffing with character  oriented hardware and bit stuffing refers to bit oriented hardware.

Byte stuffing translates each reserved byte into two unreserved bytes . For example:

It can use esc as prefix followed  by  x for soh, y for eot and z for eco.

The receiver then replaces each occurrence of esc y and esc z by the corresponding single character .