WEEK 1
1 MAP-REDUCE
HADOOP
K-V pair
Spark
Resilient distributed dataset
Page Rank
Rank(B) = sum (Rank(A) / DEST_COUNT(A))
Week2
Socket
JVM_A -> JVM_B
b: server socket
a: bSocket.accept().
a.getInputStream()
a,getOutputStream
a: Socket
a.getInputs
a.getOutput
Serialization / Deserialization
- custom ser/des
- USE XML
- JAVA SER/DES (transient not serialize )
- interface def.lang (IDL, protocol buffer, you need write. )
Remote method invocation
registery
Week3
2 Distributed parallelism
SPMD model
XL(I) = RANK(NODE_x) * LENGH(NODE_x) + I
point-topoint commn(send / recv)
Message ordering,
- same sender
- same receiver
- same type
- same tag
- same order for 2 msgs
Deadlock
| NODE | 0 | 1 |
| STEP_1 | SEND(NODE_1) | SEND(NODE_0) |
| STEP_2 | RECV(NODE_0) | RECV(NODE_1) |
Non-blocking communications
Collective communication
BROADCAST. send data to all nodes.