Introduction
Yao-Yuan Chuang
1
Outline
High Performance Computing
Scientific Computing
Parallel Computing
UNIX, GNU, and Linux
Cluster Environment
Bash
Linux Commands
Software development process
2
High-Performance Computing
The term high performance computing (HPC) refers to
the use of (parallel) supercomputers and computer clusters,
that is, computing systems comprised of multiple (usually
mass-produced) processors linked together in a single
system with commercially available interconnects.
This is in contrast to mainframe computers, which are
generally monolithic in nature.
Because of their flexibility, power, and relatively low cost,
HPC systems increasingly dominate the world of
supercomputing. Usually, computer systems in or above the
teraflop-region are counted as HPC-computers.
3
Scientific Computing
“Scientific computing (or computational science) is the
field of study concerned with constructing mathematical
models and numerical solution techniques and using
computers to analyze and solve scientific and engineering
problems.
In practical use, it is typically the application of computer
simulation and other forms of computation to problems in
various scientific disciplines.
Scientists and engineers develop computer programs,
application software, that model systems being studied and
run these programs with various sets of input parameters.
Typically, these models require massive amounts of
calculations (usually floating-point) and are often executed
on supercomputers or distributed computing platforms.
4
Problem Domains for Computational
Science/Scientific Computing
Numerical simulations
Model fitting and data analysis
Numerical simulations have different objectives depending on the
nature of the task being simulated:
Reconstruct and understand known events (e.g., earthquake, tsunamis
and other natural disasters).
Predict future or unobserved situations (e.g., weather, sub-atomic
particle behavior).
Appropriately tune models or solve equations to reflect observations,
subject to model constraints (e.g. oil exploration geophysics,
computational linguistics)
Use graph theory to model networks, especially those connecting
individuals, organizations, and websites.
Optimization
Optimize known scenarios (e.g., technical and manufacturing
processes, front end engineering).
5
Supercomputer
A supercomputer is a computer
that leads the world in terms of
processing capacity, particularly
speed of calculation, at the time
of its introduction. The term
"Super Computing" was first used
by New York World newspaper in
1920 to refer to large custombuilt tabulators IBM made for
Columbia University.
6
IBM Roadrunner, LLNL, USA, 1.026 PFLOPS
7
Moore’s Law
Moore's Law is the
empirical observation
made in 1965 that the
number of transistors
on an integrated circuit
for minimum
component cost
doubles every 24
months.
The most popular
formulation is of the
doubling of the number
of transistors on
integrated circuits
every 18 months.
8
FLOPS
In computing, FLOPS (or flops) is an acronym
meaning FLoating point Operations Per Second.
This is used as a measure of a computer's performance,
especially in fields of scientific calculations that make
heavy use of floating point calculations; similar to
million instructions per second (MIPS).
The standard SI prefixes can be used for this purpose,
resulting in such units as megaFLOPS (MFLOPS, 106),
gigaFLOPS (GFLOPS, 109), teraFLOPS (TFLOPS, 1012),
petaFLOPS (PFLOPS, 1015) and exaFLOPS (EFLOPS,
1018).
As of 2008 the fastest supercomputer's performance
tops out at one petaflop.
9
www.top500.org
10
www.linuxhpc.org
11
www.winhpc.org
12
Intel’s 80 Core Chip
Performance numbers*
at 4.27 Ghz:
peak performance:
* 1.37 SP TFLOPS
Explicit PDE solver:
* 1 SP TFLOPS
matrix multiplication:
•0.51 SP TFLOPS
•2 independent FMAC units – 2 Single
Precision FLOPS per cycle
13
Compiler
A programming language is
an artificial language that can
be used to control the behavior
of a machine, particularly a
computer.
A compiler is a computer
program (or set of programs)
that translates text written in a
computer language (the source
language) into another
computer language (the target
language). The original
sequence is usually called the
source code and the output
called object code. Commonly
the output has a form suitable
for processing by other
programs (e.g., a linker), but it
may be a human-readable text
file.
14
Debug and Performance Analysis
Debugging is a methodical process of finding and reducing
the number of bugs, or defects, in a computer program or a
piece of electronic hardware thus making it behave as
expected.
In software engineering, performance analysis (a field of
dynamic program analysis) is the investigation of a
program's behavior using information gathered as the
program runs, as opposed to static code analysis. The usual
goal of performance analysis is to determine which parts of
a program to optimize for speed or memory usage.
A profiler is a performance analysis tool that measures the
behavior of a program as it runs, particularly the frequency
and duration of function calls. The output is a stream of
recorded events (a trace) or a statistical summary of the
events observed (a profile).
15
Optimization
In computing, optimization is the process of
modifying a system to make some aspect of it
work more efficiently or use less resources. For
instance, a computer program may be optimized
so that it executes more rapidly, or is capable of
operating within a reduced amount of memory
storage, or draws less battery power in a portable
computer. The system may be a single computer
program, a collection of computers or even an
entire network such as the Internet.
16
(http://en.wikipedia/org/wiki/Optimization_%28computer_science%29)
Parallel Computing
Parallel computing is the simultaneous execution of the
same task (split up and specially adapted) on multiple
processors in order to obtain results faster.
The idea is based on the fact that the process of solving a
problem usually can be divided into smaller tasks, which
may be carried out simultaneously with some coordination.
Flynn's taxonomy
whether all processors execute the same instructions at the
same time (single instruction/multiple data -- SIMD)
each processor executes different instructions (multiple
instruction/multiple data -- MIMD).
17
(http://en.wikipedia/org/wiki/Parallel_computing)
Terminology in parallel computing
Efficiency
is the execution time using a single processor divided by the quantity
of the execution time using a multiprocessor and the number of
procesors.
Parallel Overhead
the extra work associated with parallel version compared to its
sequential code, mostly the extra CPU time and memory space
requirements from synchronization, data communications, parallel
environment creation and cancellation, etc.
Synchronization
the coordination of simultaneous tasks to ensure correctness and
avoid unexpected race conditions.
Speedup
also called parallel speedup, which is defined as wall-clock time of best
serial execution divided by wall-clock time of parallel execution.
Amdahl's law can be used to give a maximum speedup factor.
Scalability
a parallel system's ability to gain proportionate increase in parallel
speedup with the addition of more processors. Also, see this Parallel
Computing Glossary
Task
a logically high level, discrete, independent section of computational 18
work. A task is typically executed by a processor as a program
Amdahl's law
Amdahl's law, named after computer architect Gene
Amdahl, is used to find the maximum expected
improvement to an overall system when only part of
the system is improved. It is often used in parallel
computing to predict the theoretical maximum
speedup using multiple processors.
1
n Pk
k 0 S
k
Pk is a percentage of the instructions that can be
improved (or slowed),
Sk is the speed-up multiplier (where 1 is no speed-up
and no slowing),
k represents a label for each different percentage and
speed-up, and
n is the number of different speed-up/slow-downs
resulting from the system change.
19
Parallelism
One of the simplest methods used to accomplish
increased parallelism is to begin the first steps of
instruction fetching and decoding before the prior
instruction finishes executing. This is the simplest form
of a technique known as instruction pipelining, and is
utilized in almost all modern general-purpose CPUs.
Pipelining allows more than one instruction to be
executed at any given time by breaking down the
execution pathway into discrete stages.
Another strategy of achieving performance is to execute
multiple programs or threads in parallel. This area of
research is known as parallel computing. In Flynn's
taxonomy, this strategy is known as Multiple
Instructions-Multiple Data or MIMD.
A less common but increasingly important paradigm of
CPUs (and indeed, computing in general) deals with
data parallelism. A vector processor, or array
processor, is a CPU design that is able to run
mathematical operations on multiple data elements
simultaneously.
20
Parallelism
In computing, multitasking is a method by which
multiple tasks, also known as processes, share
common processing resources such as a CPU.
Multiprocessing is a generic term for the use of two
or more central processing units (CPUs) within a single
computer system. It also refers to the ability of a
system to support more than one processor and/or the
ability to allocate tasks between them
A multi-core microprocessor is one that combines two
or more independent processors into a single package,
often a single integrated circuit (IC). A dual-core
device contains two independent microprocessors. In
general, multi-core microprocessors allow a computing
device to exhibit some form of thread-level parallelism
(TLP) without including multiple microprocessors in
separate physical packages. This form of TLP is often
known as chip-level multiprocessing.
21
Computer Cluster
A computer cluster is a group of tightly coupled
computers that work together closely so that in
many respects they can be viewed as though
they are a single computer. The components of a
cluster are commonly, but not always, connected
to each other through fast local area networks.
Clusters are usually deployed to improve
performance and/or availability over that
provided by a single computer, while typically
being much more cost-effective than single
computers of comparable speed or availability.
22
High-Availability (HA) Clusters
High-availability clusters (a.k.a
Failover clusters) are implemented
primarily for the purpose of
improving the availability of services
which the cluster provides. They
operate by having redundant nodes,
which are then used to provide
service when system components
fail. The most common size for an
HA cluster is two nodes, which is the
minimum requirement to provide
redundancy. HA cluster
implementations attempt to manage
the redundancy inherent in a cluster
to eliminate single points of failure.
There are many commercial
implementations of High-Availability
clusters for many operating systems.
The Linux-HA project is one
commonly used free software HA
package for the Linux OS.
23
Load-balancing Clusters
Load-balancing clusters operate by having all workload
come through one or more load-balancing front ends, which
then distribute it to a collection of back end servers.
Although they are primarily implemented for improved
performance, they commonly include high-availability
features as well. Such a cluster of computers is sometimes
referred to as a server farm.
There are many commercial load balancers available
including Platform LSF HPC, Sun Grid Engine, Moab Cluster
Suite and Maui Cluster Scheduler. The Linux Virtual Server
project provides one commonly used free software package
for the Linux OS.
For example, web farm of Google and Yahoo
24
High-performance Clusters
High-performance clusters are implemented primarily to
provide increased performance by splitting a computational
task across many different nodes in the cluster, and are
most commonly used in scientific computing.
One of the most popular HPC implementations is a cluster
with nodes running Linux as the OS and free software to
implement the parallelism. This configuration is often
referred to as a Beowulf cluster.
HPCs are optimized for workloads which require jobs or
processes happening on the separate cluster computer
nodes to communicate actively during the computation.
These include computations where intermediate results
from one node's calculations will affect future calculations
on other nodes.
25
Beowulf (computing)
Beowulf is a design for high-performance parallel
computing clusters on inexpensive personal computer
hardware. Originally developed by Thomas L. Sterling and
Donald Becker at NASA, Beowulf systems are now deployed
worldwide, chiefly in support of scientific computing.
A Beowulf cluster is a group of usually identical PC
computers networked into a small TCP/IP LAN, and have
libraries and programs installed which allow processing to
be shared among them.
There is no particular piece of software that defines a
cluster as a Beowulf. Commonly used parallel processing
libraries include MPI (Message Passing Interface) and PVM
(Parallel Virtual Machine). Both of these permit the
programmer to divide a task among a group of networked
computers, and recollect the results of processing.
26
OpenMosix
openMosix is a free cluster
management system that provides
single-system image (SSI)
capabilities, e.g. automatic work
distribution among nodes. It allows
program processes (not threads) to
migrate to machines in the node's
network that would be able to run
that process faster (process
migration). It is particularly useful
for running parallel and intensive
input/output (I/O) applications. It is
released as a Linux kernel patch, but
is also available on specialized
LiveCDs and as a Gentoo Linux
kernel choice.
openMosix is currently considered
stable on 2.4 kernel for the x86
architecture.
27
Pthread
POSIX Threads is a POSIX standard for threads. The
standard defines an API for creating and manipulating
threads.
POSIX or "Portable Operating System Interface for uniX"
is the collective name of a family of related standards
specified by the IEEE to define the application programming
interface (API) for software compatible with variants of the
Unix operating system.
A thread in computer science is short for a thread of
execution. Threads are a way for a program to split itself
into two or more simultaneously (or pseudo-simultaneously)
running tasks. Threads and processes differ from one
operating system to another, but in general, the way that a
thread is created and shares its resources is different from
the way a process does.
28
Message Passing Interface
The Message Passing Interface (MPI) is a languageindependent computer communications descriptive application
programmer interface (API) with defined semantics and flexible
interpretations; it does not define the protocol by which these
operations are to be performed in the sense of sockets for TCP/IP
or other layer-4 and below models in the ISO/OSI Reference
Model.
It is consequently a layer-5+ type set of interfaces, although
implementations can cover most layers of the reference model,
with sockets+TCP/IP as a common transport used inside the
implementation. MPI's goals are high performance, scalability, and
portability.
Productivity of the interface for programmers is not one of the key
goals of MPI, and MPI is generally considered to be low-level. It
expresses parallelism explicitly rather than implicitly. MPI is
considered successful in achieving high performance and high
portability, but is often criticized for its low-level qualities.
29
TCP/IP
Application
Transport layer
(TCP)
Network layer(IP)
Data link
Physical layer
30
OpenMP
The OpenMP (Open Multi-Processing) application
programming interface (API) supports multi-platform
shared memory multiprocessing programming in C/C++
and Fortran on many architectures, including Unix and
Microsoft Windows platforms. It consists of a set of
compiler directives, library routines, and environment
variables that influence run-time behavior.
Jointly defined by a group of major computer hardware and
software vendors, OpenMP is a portable, scalable model
that gives programmers a simple and flexible interface for
developing parallel applications for platforms ranging from
the desktop to the supercomputer.
(http://en.wikipedia.org/wiki/Openmp)
31
Distributed Shared Memory
Distributed Shared Memory (DSM), in computer science,
refers to a wide class of software and hardware
implementations, in which each node of a cluster has
access to a large shared memory in addition to each node's
limited non-shared private memory.
Software DSM systems can be implemented within an
operating system, or as a programming library. Software
DSM systems implemented in the operating system can be
thought of as extensions of the underlying virtual memory
architecture. Such systems are transparent to the
developer; which means that the underlying distributed
memory is completely hidden from the users. In contrast,
Software DSM systems implemented at the library or
language level are not transparent and developers usually
have to program differently. However, these systems offer a
more portable approach to DSM system implementation.
32
Distributed Computing
Distributed computing is a method of computer
processing in which different parts of a program run
simultaneously on two or more computers that are
communicating with each other over a network.
Distributed computing is a type of parallel processing. But
the latter term is most commonly used to refer to
processing in which different parts of a program run
simultaneously on two or more processors that are part of
the same computer.
While both types of processing require that a program be
parallelized -- divided into sections that can run
simultaneously, distributed computing also requires that the
division of the program take into account the different
environments on which the different sections of the
program will be running. For example, two computers are
likely to have different file systems and different hardware
33
components.
(http://en.wikipedia/org/wiki/Distributed_computing)
BOINC
The Berkeley Open Infrastructure for Network
Computing (BOINC) is a middleware system for volunteer
computing, originally developed to support SETI@home,
but intended to be useful for other applications as well.
Currently BOINC is being developed by a team based at the
University of California, Berkeley led by David Anderson,
who also leads SETI@home. As a "quasi-supercomputing"
platform BOINC has over 435,000 active computers (hosts)
worldwide processing on average 500 TFLOPS as of January
30, 2007
boinc.berkeley.edu
34
Grid Computing
Grid computing or grid clusters are a technology closely
related to cluster computing. The key differences between
grids and traditional clusters are that grids connect
collections of computers which do not fully trust each other,
and hence operate more like a computing utility than like a
single computer. In addition, grids typically support more
heterogeneous collections than are commonly supported in
clusters.
Grid computing is optimized for workloads which consist of
many independent jobs or packets of work, which do not
have to share data between the jobs during the
computation process. Grids serve to manage the allocation
of jobs to computers which will perform the work
independently of the rest of the grid cluster. Resources such
as storage may be shared by all the nodes, but
intermediate results of one job do not affect other jobs in
progress on other nodes of the grid.
35
Topics in Parallel Computing
General High-performance computing
Parallelism Data parallelism • Task parallelism
Theory Speedup • Amdahl's law • Flynn's Taxonomy • Cost
efficiency • Gustafson's Law • Karp-Flatt Metric
Elements Process • Thread • Fiber • Parallel Random Access
Machine
Multiprocessing Multitasking • Memory coherency • Cache
coherency • Barrier • Synchronization • Distributed computing •
Grid computing
Programming Programming model • Implicit parallelism • Explicit
parallelism
Hardware Computer cluster • Beowulf • Symmetric
multiprocessing • Non-Uniform Memory Access • Cache only
memory architecture • Asymmetric multiprocessing • Simultaneous
multithreading • Shared memory • Distributed memory • Massively
parallel processing • Superscalar processing • Vector processing •
Supercomputer
Software Distributed shared memory • Application checkpointing
APIs Pthreads • OpenMP • Message Passing Interface (MPI)
36
Problems Embarrassingly parallel • Grand Challenge
UNIX
Unix (officially trademarked as UNIX®) is a computer operating
system originally developed in the 1960s and 1970s by a group of
AT&T employees at Bell Labs including Ken Thompson, Dennis
Ritchie and Douglas McIlroy. Today's Unix systems are split into
various branches, developed over time by AT&T as well as various
commercial vendors and non-profit organizations.
The present owner of the trademark UNIX® is The Open Group, an
industry standards consortium. Only systems fully compliant with
and certified to the Single UNIX Specification qualify as "UNIX®"
(others are called "Unix system-like" or "Unix-like").
During the late 1970s and early 1980s, Unix's influence in
academic circles led to large-scale adoption (particularly of the
BSD variant, originating from the University of California, Berkeley)
of Unix by commercial startups, the most notable of which is Sun
Microsystems. Today, in addition to certified Unix systems, Unixlike operating systems such as Linux, Mac OS X and BSD
derivatives are commonly encountered.
37
UNIX
38
(http://en.wikipedia/org/wiki/Unix)
GNU
The GNU project was publicly announced by Richard
Stallman.
GNU was to be a complete Unix-like operating system
composed entirely of free software. Software
development work began in January 1984. By the
beginning of the 1990s, the project had produced or
collected most of the necessary components of this
system, including libraries, compilers, text editors,
and a Unix shell.
Thus the GNU mid-level portions of the operating
system were almost complete. The upper level could
be supplied by the X Window System, but the lower
level, which consisted of a kernel, device drivers, and
daemons, was still mostly lacking. In 1990, the GNU
project began developing the GNU Hurd kernel, based
on the Mach microkernel.
39
(http://en.wikipedia/org/wiki/GNU)
Linux
Linus Torvalds, creator of the Linux kernel.
In 1991, work on the Linux kernel began by Linus Torvalds
while attending the University of Helsinki. Torvalds
originally created the Linux kernel as a non-commercial
replacement for the Minix kernel; he later changed his
original non-free license to the GPLv2, which differed
primarily in that it also allowed for commercial
redistribution.
Although dependent on the Minix userspace at first, work
from both Linux kernel developers and the GNU project
allowed Linux to work with GNU components. Thus Linux
filled the last major gap in running a complete, fully
functional operating system built from free software.
40
(http://en.wikipedia/org/wiki/Linux)
Kernel
In computing, the kernel is the central component of
most computer operating systems (OSs). Its
responsibilities include managing the system's
resources and the communication between hardware
and software components.
As a basic component of an operating system, a
kernel provides the lowest-level abstraction layer for
the resources (especially memory, processors and I/O
devices) that applications must control to perform
their function. It typically makes these facilities
available to application processes through interprocess communication mechanisms and system calls.
While monolithic kernels will try to achieve these goals
by executing all the code in the same address space
to increase the performance of the system,
microkernels run most of their services in user space,
aiming to improve maintainability and modularity of
the codebase.
41
Linux Distribution
A Linux distribution, often simply distribution
or distro, is a member of the Linux family of
Unix-like operating systems comprised of the
Linux kernel, the non-kernel parts of the GNU
operating system, and assorted other software.
Linux distributions take a variety of forms, from
fully-featured desktop and server operating
systems to minimal environments (typically for
use in embedded systems, or for booting from a
floppy).
CentOS • Debian • Fedora • Gentoo • Knoppix •
Mandriva Linux • Red Hat Enterprise Linux •
Slackware • SUSE Linux • Ubuntu • more… •
42
Unix Shell
A Unix shell, also called "the command line", provides
the traditional user interface for the Unix operating
system and for Unix-like systems.
bash – Bourne Again SHell, (mostly) sh-compatible and
csh-compatible, standard shell on Linux systems and
Mac OS X.
csh – C shell. Written by Bill Joy for BSD systems.
ksh – Korn shell, standard shell on many proprietary
Unix systems, powerful successor to the Unix Bourne
shell (sh), written by David Korn,
rc – originally written for Plan 9.
sh – Bourne shell, only shell present on all UNIX and
Unix-like systems, written by Stephen Bourne for
Version 7 Unix.
tcsh – TENEX C shell, standard shell on BSD systems.
zsh – Z shell.
43
(http://en.wikipedia/org/wiki/Unix_shell)
Bash
Bash is a Unix shell written for the GNU Project. The name
of the actual executable is bash. Its name is an acronym
for Bourne-again shell, a pun ('Bourne again' / 'born again')
on the name of the Bourne shell (sh), an early and
important Unix shell written by Stephen Bourne and
distributed with Version 7 Unix circa 1978. Bash was
created in 1987 by Brian Fox. In 1990 Chet Ramey became
the primary maintainer.
Bash is the default shell on most Linux systems as well as
on Mac OS X and it can be run on most Unix-like operating
systems. It has also been ported to Microsoft Windows
within the Cygwin POSIX emulation environment for
Windows and to MS-DOS by the DJGPP project. Released
under the GNU General Public License, Bash is free software.
44
(http://en.wikipedia/org/wiki/Bash)
Math Cluster
45
Math Cluster
控制節點
計算節點
對外IP：140.127.223.11 (eth1)
對內IP：192.168.0.254 (eth0)
192.168.0.1 ~7 (node1 ~ node7)
帳號
hpc001 …
46
Remote Login:SSH Client
Don’t forget change the password after first time login
47
Kernel Specific Commands
Kernel specific
date – Print or set the system date and/or
time
dmesg – Print the kernel message buffer
ipcrm – Remove a message queue,
semaphore set or shared memory id
ipcs – Provide information on IPC facilities
uname – Print assorted system statistics
48
(http://en.wikipedia/org/wiki/Linux_commands)
General User Commands
dd – Convert and copy a file (Disk Dump)
dirname – Strip non-directory suffixes from a
path
echo – Print to standard output
env – Show environment variables; run a
program with altered environment variables
file (or stat) – Determine the type of a file
nohup – Run a command with immunity to
hangups outputting to non-tty
sh – The Bourne shell, the standard Unix shell
uptime – Print how long the system has been
running
49
(http://en.wikipedia/org/wiki/Linux_commands)
Processes and tasks management
anacron – Periodic command scheduler
at – Single-time command scheduler
chroot – Change the system root directory for all child
processes
cron – Periodic command scheduler
crontab – Crontab file editor
daemonic – Interface to daemon init scripts
htop – Interactive ncurses-based process viewer that allows
scrolling to see all processes and their full command lines
kill – Send a signal to process, or terminate a process (by
PID)
killall – Terminate all processes (in GNU/Linux, it's kill by
name)
50
(http://en.wikipedia/org/wiki/Linux_commands)
Processes and tasks management
nice – Alter priorities for processes
pgrep – Find PIDs of processes by name
pidof – GNU/Linux equivalent of pgrep
pkill – Send a signal to process, or terminate a process (by
name). Equivalent to Linux killall
ps – Report process status
renice – Alter the priorities of an already running process
sleep – Delay for specified time
time – Time a command
timex – Time process shell execution, measure process data
and system activity
top – Produce a dynamic list of all resident processes
wait – Wait for the specified process
51
(http://en.wikipedia/org/wiki/Linux_commands)
User management and support
chsh – Change user shell
finger – Get details about user
id – Print real/effective UIDs/GIDs
last – show listing of last logged in users
lastlog – show last log in information for users
locale – Get locale specific information
localedef – Compile locale definitions
logname – Print user's login name
man – Manual browser
mesg – Control write access to your terminal
passwd – Change user password
52
(http://en.wikipedia/org/wiki/Linux_commands)
User management and support
su – Start a new process (defaults to shell) as a
different user (defaults to root)
sudo – execute a command as a different user.
users – Show who is logged on (only users names)
w – Show logged-in users and their current tasks
whatis – command description from whatis
database
whereis – locates the command's binary and
manual pages associated with it
which (Unix) – locates where a command is
executed from
who – Show who is logged on (with some details)
write – Send a message to another user
53
(http://en.wikipedia/org/wiki/Linux_commands)
Filesystem Utilities
info – The GNU alternative to man
man – The standard unix documentation system
chattr – Change file attributes on a Linux second
extended file system
chgrp – Change the group of a file or directory
chmod – Change the permissions of a file or
directory
chown – Change the owner of a file or directory
cd – Change to another directory location
cp – Copy a file or directory to another location
54
Filesystem Utilities
df – Report disk space
dircmp – Compare contents of files between two directories
du – Calculate used disk space
fdupes – Find or remove duplicate files within a directory
find – Search for files through a directory hierarchy
fsck – Filesystem check
ln – Link one file/directory to another
ls – List directory contents
lsattr – List file attributes on a Linux second extended file
system
lsof – list open files
55
Filesystem Utilities
mkdir – Make a directory
mkfifo – Make a named pipe
mount – Mount a filesystem
mv – Move or rename a file or directory
pwd – Print the current working directory
rm – Delete a file or directory tree
readlink – Display value of a symbolic link, or display
canonical path for a file
rmdir – Delete an empty directory
touch – Create a new file or update its modification time
tree – Print a depth-indented tree of a given directory
unlink – System call to remove a file or directory
56
Archivers and compression
afio – Compatible superset of cpio with added functionality
ar – Maintain, modify, and extract from archives. Now
largely obsoleted by tar
bzip2 – Block-sorting file compressor
compress – Traditional compressor using the LZW algorithm
cpio – A traditional archiving tool/format
gzip – The gzip file compressor
p7zip – 7zip for unix/linux
pack, pcat, unpack – included in old versions of ATT Unix.
Uses Huffman coding, obsoleted by compress.
pax – POSIX archive tool that handles multiple formats.
tar – Tape ARchiver, concatenates files
uncompress – Uncompresses files compressed with
compress.
zcat – Prints files to stdout from gzip archives without
unpacking them to separate file(s)
57
Text Processing
awk – A pattern scanning and processing language
banner – Creates ascii art version of an input string for
printing large banners
cat – Concatenate files to standard output
cksum – Print the CRC checksum and bytecount of a file
(see also MD5)
cmp – Compare two files byte for byte
cut – Remove sections from each line of a file or standard
input
diff – Compare two text files line by line
egrep – Extended pattern matching (synonym for "grep -E")
fgrep – Simplified pattern matching (synonym for "grep -F")
fold – Wrap each input line to fit within the given width
grep – Print lines matching a pattern
head – Output the first parts of a file
iconv – Convert the encoding of the specified files
58
join – Join lines of two files on a common field
less – Improved more-like text pager
Text Processing
more – Pager
nroff – Fixed-width (non-typesetter) version of the standard
Unix typesetting system
patch – Change files based on a patch file
sed – Stream EDitor
sort – Sort lines of text files
split – Split a file into pieces
tail – Output the tail end of files
tee – Read from standard input, write to standard output
and files
uudecode – Decodes a binary file that was used for
transmission using electronic mail
uuencode – Encodes a binary file for transmission using
electronic mail
59
wc – Word/line/byte count
Text Editors
GNU Emacs – Freely programmable full-screen text editor
and general computing environment (using builtin Elisp, a
simple dialect of the Lisp programming language)
Joe – a screen-oriented text editor using a Wordstar-style
command set
Jove – a screen-oriented text editor using an Emacs-style
command set
pico – PIne's message COmposition editor (simple, easy to
use screen editor)
vi – "Visual" (screen-oriented) text editor (originally ex in
screen-oriented "visual" mode)
vim – Vi IMproved, portable vi-compatible editor with
multiple buffers, screen splitting, syntax highlighting and a
lot of other features not found in standard ex/vi
XEmacs – Popular version of emacs that is derived from
GNU Emacs
60
Communication
ftp, sftp – File transfer protocol, secure FTP
NFS – Network filesystem
OpenVPN – virtual private (encrypting) networking software
Postfix — mail transfer agent
rsh, SSH, telnet – Remote login
Samba – SMB and CIFS client and server for UNIX
Sendmail – popular E-Mail transport software
talk – Talk to another logged-in user
uustat – a Basic Networking Utilities (BNU) command that
displays status information about several types of BNU
operations
uux – Remote command execution over UUCP
61
Network monitoring and security
Ethereal and tethereal – a feature rich protocol analyzer (now
called Wireshark, see below)
John the Ripper – password cracking software
Nessus – a comprehensive open-source network vulnerability
scanning program
Netstat – displays a list of the active network connections the
computer
Nmap – free port scanning software
SAINT – System Administrator’s Integrated Network Tool –
Network Vulnerability Scanner.
SATAN – the Security Administrator Tool for Analyzing Networks –
a testing and reporting tool that collects information about
networked hosts
Snoop – Solaris packet sniffer
Snort – an open source network intrusion detection system
tcpdump – a computer network debugging tool that intercepts and
displays TCP/IP packets being transmitted or received
Wireshark – a protocol analyzer, or "packet sniffer", similar to
tcpdump, that adds a GUI frontend, and more sorting and filtering62
options (formerly named Ethereal)
Programming Tools
bash – Bourne Again SHell, (mostly) sh-compatible and cshcompatible, standard shell on Linux systems and Mac OS X.
csh – C shell. Written by Bill Joy for BSD systems.
ksh – Korn shell, standard shell on many proprietary Unix systems,
powerful successor to the Unix Bourne shell (sh), written by David
Korn,
rc – originally written for Plan 9.
sh – Bourne shell, only shell present on all UNIX and Unix-like
systems, written by Stephen Bourne for Version 7 Unix.
tcsh – TENEX C shell, standard shell on BSD systems.
zsh – Z shell.
awk – Standard Unix pattern scanning and text processing tool.
perl – Perl scripting language.
PHP – PHP scripting language.
Python – Python scripting language.
63
Compilers
as – GNU assembler tool.
c99 – C programming language.
cc – C compiler.
dbx – (System V and BSD) Symbolic debugger.
f77 – Fortran 77 compiler.
gcc – GNU Compiler Collection C frontend (also known as GNU C Compiler)
gdb – GNU symbolic debugger.
ld – Program linker.
lex – Lexical scanner generator.
ltrace – (Linux) Trace dynamic library calls in the address space of the
watched process.
m4 – Macro language.
make – Automate builds.
nm – List symbols from object files.
size – return the size of the sections of an ELF file.
strace – (Linux) or truss (Solaris) Trace system calls with their arguments
and signals. Useful debugging tool, but does not trace calls outside the
kernel, in the address space of the process(es) being watched.
64
Desktops/Graphical User Interfaces
CDE – Common Desktop Environment, most commonly
found on proprietary UNIX systems
Enlightenment – an open source window manager for the X
Window System
FVWM and its variant FVWM95, which has been modified to
behave like Windows 95 Also FVWM-Crystal that aims to be
eye candy
GNOME – GNU Network Object Model Environment
IceWM – ICE Window Manager
JWM – Joe's Window Manager
KDE – K Desktop Environment
XFce – a desktop environment for Unix and other Unix-like
platforms
65
Package Management
apt – Front-end for dpkg or rpm
debconf – Debian package configuration management
system
dpkg – The Debian package manager
drakconf – Front-end configuration utility for Mandriva Linux
emerge – A frontend to portage
pacman – A package manager used primarily by Arch Linux
portage – The Gentoo Linux package manager
rpm – Originally the package manager for Red Hat Linux,
now used by several distributions including Mandriva Linux
Synaptic – GTK+ frontend for the apt package manager.
Primarily used by Ubuntu Linux, Debian Sarge, and other
Debian-based systems; but usable on any system using apt.
urpmi – Front-end to rpm, used by Mandriva Linux
YaST - System management utility mainly used by SuSE
yum - Front-end for rpm, used by Fedora
66
Web Browsers
Dillo – Extremely light-weight web browser
ELinks – Enhanced links
Epiphany – Light-weight GNOME web browser
Galeon – Light-weight old GNOME web browser
Konqueror – KDE web browser
Links – Console based web browser
lynx – Console based web browser
Mozilla Application Suite – Graphical cross platform web
browser & email client
Mozilla Firefox – Extensible Web browser
Opera – Web browser and e-mail client (Proprietary
Software)
w3m – Console based web browser
67
Desktop Publishing
groff – Traditional typesetting system
LaTeX – Popular TeX macro package for higherlevel typesetting
lp – Print a file (on a line printer)
Passepartout – Desktop publishing program
pr – Convert text files for printing
Scribus – Desktop publishing program
TeX – Macro-based typesetting system
troff – The original and standard Unix typesetting
system
68
Databases
DB2
Firebird
MySQL
Oracle
PostgreSQL
Progress Software
SQLite
Sybase
69
Math Tools
maxima – Symbol manipulation program.
octave – Numerical computing language (mostly
compatible with Matlab) and environment.
R – Statistical programming language.
units – Unit conversion program.
bc – An arbitrary precision calculator language
with syntax similar to the C programming
language.
cal – Displays a calendar
dc – Reverse-Polish desk calculator which
supports unlimited precision arithmetic
fortune – Fortune cookie program that prints a
random quote
70
Unix command line
File and file system management: cat | cd | chmod | chown |
chgrp | cp | du | df | file | fsck | ln | ls | lsof | mkdir | mount | mv
| pwd | rm | rmdir | split | touch
Process management: at | chroot | crontab | kill | killall | nice |
pgrep | pidof | pkill | ps | sleep | time | top | wait | watch
User Management/Environment:env | finger | id | mesg |
passwd | su | sudo | uname | uptime | w | wall | who | whoami |
write
Text processing:awk | cut | diff | ex | head | iconv | join | less |
more | paste | sed | sort | tail | tr | uniq | wc | xargs
Shell programming:echo | expr | printf | unset
Printing:lp
Communications:
inetd | netstat | ping | rlogin | traceroute
Searching:
find | grep | strings
Miscellaneous:
banner | bc | cal | man | size | yes
71
Software development process
A software development process is a structure imposed
on the development of a software product. Synonyms
include software lifecycle and software process. There
are several models for such processes, each describing
approaches to a variety of tasks or activities that take place
during the process.
Software Elements Analysis
Specification
Software architecture
Implementation (or coding)
Testing
Documentation
Software Training and Support
Maintenance
72
Software Elements Analysis
The most important task in creating a software
product is extracting the requirements.
Customers typically know what they want, but
not what software should do, while incomplete,
ambiguous or contradictory requirements are
recognized by skilled and experienced software
engineers.
Frequently demonstrating live code may help
reduce the risk that the requirements are
incorrect.
73
Specification
Specification is the task of precisely describing the software
to be written, possibly in a rigorous way.
In practice, most successful specifications are written to
understand and fine-tune applications that were already
well-developed, although safety-critical software systems
are often carefully specified prior to application
development.
Specifications are most important for external interfaces
that must remain stable.
74
Software Architecture
The architecture of a software system refers to an abstract
representation of that system. Architecture is concerned
with making sure the software system will meet the
requirements of the product, as well as ensuring that future
requirements can be addressed. The architecture step also
addresses interfaces between the software system and
other software products, as well as the underlying hardware
or the host operating system.
75
Implementation and Testing
Implementation (or coding): Reducing a design to code
may be the most obvious part of the software engineering
job, but it is not necessarily the largest portion.
Testing: Testing of parts of software, especially where code
by two different engineers must work together, falls to the
software engineer.
76
Documentation and Training
An important (and often overlooked) task is documenting
the internal design of software for the purpose of future
maintenance and enhancement. Documentation is most
important for external interfaces.
Common types of computer hardware/software
documentation include online help, FAQs, HowTos, and user
guides. The term RTFM is often used in regard to such
documentation, especially to computer hardware and
software user guides.
RTFM is an initialism for the statement "Read The Fucking
Manual." This instruction is sometimes given in response to
a question when the person being asked believes that the
question could be easily answered by reading the relevant
"manual" or instructions.
Some people prefer the backronym "Read The Fine
Manual." Alternatively, the "F" can be dropped entirely and
the initialism rendered as "RTM" (Read The Manual), or the
more polite "RTMP" (Read The Manual Please).
77
Software Training and Support
A large percentage of software projects fail because the
developers fail to realize that it doesn't matter how much
time and planning a development team puts into creating
software if nobody in an organization ends up using it.
People are occasionally resistant to change and avoid
venturing into an unfamiliar area, so as a part of the
deployment phase, its very important to have training
classes for the most enthusiastic software users (build
excitement and confidence), shifting the training towards
the neutral users intermixed with the avid supporters, and
finally incorporate the rest of the organization into adopting
the new software. Users will have lots of questions and
software problems which leads to the next phase of
software.
78
Maintenance
Maintaining and enhancing software to cope with newly
discovered problems or new requirements can take far
more time than the initial development of the software. Not
only may it be necessary to add code that does not fit the
original design but just determining how software works at
some point after it is completed may require significant
effort by a software engineer. About ⅔ of all software
engineering work is maintenance, but this statistic can be
misleading. A small part of that is fixing bugs. Most
maintenance is extending systems to do new things, which
in many ways can be considered new work. In comparison,
about ⅔ of all civil engineering, architecture, and
construction work is maintenance in a similar way.
79
Software Developing Models
Waterfall model
Spiral model
Model driven development
User experience
Top-down and bottom-up design
Chaos model
Evolutionary prototyping
Prototyping
V model
Extreme Programming
Hysterical raisins
80
Waterfall Model
The waterfall model is a
sequential software
development model (a process
for the creation of software) in
which development is seen as
flowing steadily downwards
(like a waterfall) through the
phases of requirements
analysis, design,
implementation, testing
(validation), integration, and
maintenance. The origin of the
term "waterfall" is often cited
to be an article published in
1970 by W. W. Royce
81
Spiral Model
The new system requirements are defined in as much detail
as possible. This usually involves interviewing a number of
users representing all the external or internal users and
other aspects of the existing system.
A preliminary design is created for the new system.
A first prototype of the new system is constructed from the
preliminary design. This is usually a scaled-down system,
and represents an approximation of the characteristics of
the final product.
A second prototype is evolved by a fourfold procedure: (1)
evaluating the first prototype in terms of its strengths,
weaknesses, and risks; (2) defining the requirements of the
second prototype; (3) planning and designing the second
prototype; (4) constructing and testing the second
prototype.
82
Spiral Model
At the customer's option, the entire project can be aborted
if the risk is deemed too great. Risk factors might involve
development cost overruns, operating-cost miscalculation,
or any other factor that could, in the customer's judgment,
result in a less-than-satisfactory final product.
The existing prototype is evaluated in the same manner as
was the previous prototype, and, if necessary, another
prototype is developed from it according to the fourfold
procedure outlined above.
The preceding steps are iterated until the customer is
satisfied that the refined prototype represents the final
product desired.
The final system is constructed, based on the refined
prototype.
The final system is thoroughly evaluated and tested.
Routine maintenance is carried out on a continuing basis to
prevent large-scale failures and to minimize downtime.
83
Top-Down and Bottom-up
Top-down and bottom-up are strategies of information
processing and knowledge ordering, mostly involving software,
and by extension other humanistic and scientific system theories
(see systemics).
In the top-down model an overview of the system is formulated,
without going into detail for any part of it. Each part of the system
is then refined by designing it in more detail. Each new part may
then be refined again, defining it in yet more detail until the entire
specification is detailed enough to validate the model. The topdown model is often designed with the assistance of "dark boxes"
that make it easier to bring to fulfillment but insufficient and
irrelevant in understanding the elementary mechanisms.
In bottom-up design, first the individual parts of the system are
specified in great detail. The parts are then linked together to
form larger components, which are in turn linked until a complete
system is formed. This strategy often resembles a "seed" model,
whereby the beginnings are small, but eventually grow in
complexity and completeness.
84
Software prototyping
Identify basic requirements
Develop Initial Prototype
The initial prototype is developed that includes only user
interfaces.
Review
Determine basic requirements including the input and output
information desired. Details, such as security, can typically
be ignored.
The customers, including end-users, examine the prototype
and provide feedback on additions or changes.
Revise and Enhancing the Prototype
Using the feedback both the specifications and the prototype
can be improved. Negotiation about what is within the scope
of the contract/product may be necessary. If changes are
introduced then a repeat of steps #3 ands #4 may be
needed.
85
Extreme Programming
Extreme Programming (XP) is a software engineering
methodology, the most prominent of several agile software
development methodologies. Like other agile methodologies,
Extreme Programming differs from traditional
methodologies primarily in placing a higher value on
adaptability than on predictability. Proponents of XP regard
ongoing changes to requirements as a natural, inescapable
and desirable aspect of software development projects;
they believe that being able to adapt to changing
requirements at any point during the project life is a more
realistic and better approach than attempting to define all
requirements at the beginning of a project and then
expending effort to control changes to the requirements.
86
Software Architecture
Software architecture is commonly organized in views,
which are analogous to the different types of blueprints
made in building architecture. Some possible views
(actually, viewpoints in the 1471 ontology) are:
Functional/logic view
Code view
Development/structural view
Concurrency/process/thread view
Physical/deployment view
User action/feedback view
Several languages for describing software architectures
have been devised, but no consensus has yet been reached
on which symbol-set and view-system should be adopted.
The UML was established as a standard "to model systems
(and not just software)," and thus applies to views about
software architecture.
87
© Copyright 2024 ExpyDoc
