ECE
5730 Foundations of Neural Networks
Spring
2022
version
20 April 2022
Instructor
Dr. Damon
A. Miller, Associate Professor of Electrical and Computer Engineering, Western
Michigan University, College of Engineering and Applied Sciences, Floyd Hall,
Room A-240, 269.276.3158, 269.276.3151 (fax), damon.miller@wmich.edu, www.homepages.wmich.edu/~miller/, https://wmich.webex.com/meet/damon.miller.
Office Hours
Dr. Miller is
available for in-person office hours as posted in his schedule. Appointments at other times are requested
by email to damon.miller@wmich.edu.
Catalog
Description
ECE 5730 Foundations
of Neural Networks,
3 hrs. Biological and artificial neural
networks from an electrical and computer engineering perspective. Neuron
anatomy. Electrical signaling, learning, and memory in biological neural
networks. Development of neural network circuit models. Artificial neural
systems including multilayer feedforward neural networks, Hopfield networks,
and associative memories. Electronic implementations and engineering
applications of neural networks.
Prerequisite
Abilities
You must be able to work independently on research
projects and to write a professional quality written reports describing your
project work.
Copyright
Information
Materials prepared by
Dr. Miller are © 2022 Damon A. Miller. Other copyrights apply to materials such
as text and images from books, datasheets, etc. Consult source documents for
copyright information. Any lecture videos are for use in ECE 2100 only and must
not be distributed in any way.
Acknowledgments
ECE faculty
member(s), particularly J. Gesink, contributed to the course syllabus. Dr.
Miller also thanks Instructional Designer M. Strock and the Educational
Technology Department for contributions to this syllabus as ported from an ECE
2100 syllabus.
Course Objectives
Student
will develop:
1. an understanding
of the characteristics of intelligent systems;
2. an ability to
develop numerical solutions of ordinary differential equations;
3. an understanding
of basic neuron cell structure, anatomy, and functionality;
4. an understanding
of neuron interactions via synaptic function;
5. an understanding
of current knowledge of neural mechanisms that enable high level information
processing in biological organisms;
6. an ability to
develop computer models of biological neuron(s) and biological neural networks;
7. an ability to
design, analyze, and simulate circuits to model biological neuron(s) and
biological neural networks;
8. an understanding
of common artificial neural network (ANN) architectures;
9. an understanding
of adaptation and ‘learning’ in ANNs;
10. an understanding
of classifier design, including the role of discriminant functions;
11. an ability to
design and evaluate a multilayer feedforward neural network approximator or
classifier;
12. a basic
understanding of dynamical systems;
13. an ability to
perform a Lyapunov stability analysis;
14. an understanding
of discrete and continuous feedback networks;
15. an understanding
of associative memories;
16. an understanding
of unsupervised learning techniques (ABET: a).
17. an ability to
utilize computer simulations to study artificial neural networks;
18. an understanding
of application areas for artificial neural networks, including pattern
recognition, image processing, and signal processing;
19. effective and
ethical research methods with particular attention to proper citation
techniques; and
20. an ability to
produce a concise summary of work performed using a standard journal paper
format.
Textbook
and Materials
Required:
1.
Jacek
M. Zurada, Artificial Neural Systems,
PWS Publishing, Boston, 1992 (ISBN 0-314-93391-3). Available from the author, instructions for
securing a copy to be provided in class.
2.
W.
Otto Friesen and J. A. Friesen, NeuroDynamix II: Concepts of Neurophysiology Illustrated by
Computer Simulations, Oxford University Press, 2010 (ISBN 978-0-19-537183-3).
3.
Scott
Freeman et al., Biological Science,
Pearson, 7th edition, 2019 (ISBN-13: 9780135276815), available as an EBook.
4.
Linear
Technology, LTspice®, available at no cost at http://www.linear.com/designtools/software/.
You are responsible for ensuring access to a working copy.
SPICE
EXAMPLES
a.
VCCS example (problem 4.43 from
Nilsson and Reidel, Electric Circuits, 8th ed.)
b.
CCCS and CCVS example (problem 4.51 from
Nilsson and Reidel Electric Circuits, 8th ed.)
c.
VCVS example (simple operational
amplifier model)
d.
Chua’s “Simple” Chaotic Circuit (need the National
Semiconductor LM741 model available as part of laboratory six in the course
schedule below.
5.
The MathWorks, MATLAB®. The
student version is a tremendous value as this package includes many add-ons
that must be purchased separately for use in a professional version. [An
alternative programming language can be used with permission of the course
instructor].
References:
(see Dr. Miller, might be put on reserve in ECE Department Office, check-out
with WMU ID)
1. E. M. Izhikevich, Dynamical Systems in Neuroscience: The Geometry of Excitability and Bursting,
The MIT Press, Cambridge, Massachusetts, 2007.
2. Simon Haykin, Neural Networks: A Comprehensive Foundation,
IEEE Press, 1st edition, 1994.
3. A. S. Sedra and K.
C. Smith, Microelectronic Circuits,
Oxford University Press, 5th edition, 1998.
4. M. J. Maron, Numerical Analysis: A Practical
Approach, Macmillan Publishing Co., Inc., 1982.
5. J. G. Nicholls, A. R. Martin, B. G. Wallace,
P. A. Fuchs, From Neuron to Brain,
Sinauer Associates, Inc., 2000.
6. E. Scheinerman, Invitation to Dynamical Systems,
Prentice Hall, 1996.
7. F. Severance, System Modeling and Simulation, Wiley,
2001.
8. D. A. Miller, R. Arguello, and
G. W. Greenwood, “Evolving Artificial Neural Network Structures: Experimental Results for
Biologically-Inspired Adaptive Mutations,” Proceedings
of the 2004 Congress on Evolutionary Computation, June 2004.
8. C. M. Bishop, Neural Networks for Pattern Recognition,
Oxford University Press, 1995.
9. Scott Freeman, Biological Science, Pearson, 2nd edition, 2005.
Online References:
1.
W.
H. Press, S. A. Teukolsky, W. T. Vetterling,
and B. P. Flannery, Numerical Recipes in
C: The Art of Scientific Computing,
Cambridge University Press, 2nd edition, 1992.
Available online at http://apps.nrbook.com/c/index.html.
2.
C.
R. Nave, HyperPhysics
website, http://hyperphysics.phy-astr.gsu.edu/hbase/hframe.html, outstanding physics
tutorial/reference.
3.
http://www.nature.com/scitable/topicpage/what-is-a-cell-14023083
4.
http://www.cell.com/pictureshow
5.
Richard
F. Olivo, Biological Sciences 330/331
(Neurophysiology) website, Smith College, http://www.science.smith.edu/departments/NeuroSci/courses/bio330/, See the links for videos shown in
class.
6.
A.
L. Hodgkin and A. F. Huxley, “A quantitative description of membrane current
and its application to conduction and excitation in nerve,” J. Physiol., no. 117, pp. 500-544, 1952.
Available at http://jp.physoc.org/cgi/content/full/538/1/2.
8.
Donovan
Suires, Instrumentation
Electronics for an Integrated Electrophysiology Data Acquisition and
Stimulation System (2013). Masters Theses. 447.
https://scholarworks.wmich.edu/masters_theses/447
9.
Alexandra
C. Ferguson, Optimization and
Experimental Application of Current Stimuli to Leech Pressure-Sensitive
Mechanosensory Cells (2017). Masters Theses. 1131.
https://scholarworks.wmich.edu/masters_theses/1131.
10.
Lucas
M. Essenburg, Intracellular
Electrometer (2019). Masters Theses. 5099.
https://scholarworks.wmich.edu/masters_theses/5099.
Course
Policies
Academic
Honesty
General:
Students are responsible for making
themselves aware of and understanding the University policies and procedures
that pertain to Academic Honesty. These policies include cheating, fabrication,
falsification and forgery, multiple submission, plagiarism, complicity and
computer misuse. The academic policies addressing Student Rights and
Responsibilities can be found in the Undergraduate Catalog at http://catalog.wmich.edu/index.php?catoid=35 and the Graduate
Catalog at http://catalog.wmich.edu/index.php?catoid=39. If there is reason
to believe you have been involved in academic dishonesty, you will be referred
to the Office of Student Conduct. You will be given the opportunity to review
the charge(s) and if you believe you are not responsible, you will have the opportunity
for a hearing. You should consult with your instructor if you are uncertain
about an issue of academic honesty prior to the submission of an assignment or
test.
Students and instructors are responsible
for making themselves aware of and abiding by the “Western Michigan University
Sexual and Gender-Based Harassment and Violence, Intimate Partner Violence, and
Stalking Policy and Procedures” related to prohibited sexual misconduct under
Title IX, the Clery Act and the Violence Against
Women Act (VAWA) and Campus Safe. Under this policy, responsible employees
(including instructors) are required to report claims of sexual misconduct to
the Title IX Coordinator or designee (located in the Office of Institutional
Equity). Responsible employees are not confidential resources. For a
complete list of resources and more information about the policy see http://www.wmich.edu/sexualmisconduct.
In addition, students are encouraged to
access the Code of Conduct, as well as resources and general academic policies
on such issues as diversity, religious observance, and student disabilities:
·
Office
of Student Conduct http://www.wmich.edu/conduct
·
Division
of Student Affairs http://www.wmich.edu/students/diversity
·
Registrar’s
Office http://www.wmich.edu/registrar/calendars/interfaith
·
Disability
Services for Students http://www.wmich.edu/disabilityservices.
—
section provided by the WMU Faculty Senate with minor link reformatting
Plagiarism:
For an in-depth
exploration of plagiarism, see http://libguides.wmich.edu/plagiarism
COVID-19
Statement
Safety requirements are in place to
minimize exposure to the Western Michigan University community. These
guidelines apply to all in-person and hybrid classes held inside a WMU building
to ensure the safety of all students, faculty, and staff during the pandemic.
Noncompliance is a violation of the class requirements and the Student Code. https://wmich.edu/conduct/code
Facial coverings (masks), over
both the nose and mouth, are required for all students while in- class, no
matter the size of the space. Following this recommendation can minimize the
transmission of the virus, which is spread between people interacting in close
proximity through speaking, coughing, or sneezing. During specified classes in
which facial coverings (masks) would prevent required class elements, students
may remove facial coverings (masks) with instructor permission, in accordance
with the exceptions in the Facial Covering (mask) Policy ("such as playing
an instrument, acting, singing, etc."). https://wmich.edu/policies/facial-covering-mask
Facial
coverings (masks) must remain in place throughout the class. Any student who
removes the mandatory facial covering (mask) during class will be required to
leave the classroom immediately.
Students who are unable to wear a
facial covering (mask) for medical/disability reasons must contact Disability
Services for Students before they attend class. https://wmich.edu/disabilityservices
—
section provided by the WMU Faculty Senate, highlight added
NO FOOD OR DRINK IN LECTURE OR LAB.
ONLY STUDENTS WITH A GREEN BADGE STATUS
ARE PERMITTED IN LECTURE OR LAB. YOU MUST BE ABLE TO DEMONSTRATE YOUR BADGE
STATUS.
Accommodations
If you have a documented disability
and verification
from the Disability Services for Students (DSS), and wish to discuss academic
accommodations, please contact your instructor as soon as possible. It is the
student’s responsibility to provide documentation of disability to DSS and meet
with a DSS counselor to request special accommodation before classes start.
Grading
Basis
1. Projects (70%) will be assigned on
a regular basis.
Some
project results will be reported using the IEEE journal paper format; see http://ieeeauthorcenter.ieee.org/wp-content/uploads/Transactions-instructions-only.pdf for details.
You may not use any sources other than
those provided in class or in this syllabus when preparing your project report without prior approval from the course
instructor. You may be asked to
demonstrate your project.
LATE
PROJECTS WILL NOT BE ACCEPTED AND ARE DUE AT THE BEGINNING OF CLASS.
Unless otherwise noted projects are completed individually.
2. Homework: 30%
OUTSTANDING WORK might earn extra credit.
Scale:
0-60 E | 60-65 D | 65-70 DC | 70-75 C | 75-80 CB | 80-85 B | 85-90 BA | 90-100
A |
Midterm
grades are not assigned.
Grade
Appeals
If you have a
question regarding graded course materials (e.g. exam problems, homework
problems, laboratory reports, etc.), contact Dr. Miller within TWO business days of receiving the
grade for the assignment in question.
Late
Assignments will
not be accepted without a documented excuse. If an emergency prevents you from
submitting an assignment on-time, contact your instructor PRIOR to the
assignment due date or as soon as you can, via email. Failure to adhere to this policy will result
in zero credit for the assignment.
PARTIAL CHECK LIST
FOR SUBMITTED ASSIGNMENTS
1. Each problem must
include: (a) author's name, (b) name/title of the assignment, and (c) date of
completion.
2. Use only one side
of the paper and include a brief and concise statement of the problem prior to
its solution. Begin each problem on a new page.
3. Number the pages.
4. Staple each
problem in the upper left corner as needed.
5. Entitle graphs,
label and include axes, include key symbols for multiple curve graphs, and give
brief notes of explanation where appropriate.
6. USE
A WHITE BACKGROUND FOR ALL LTspice® schematics and
waveform plots. Reports must not be handwritten, though you must include copies
of your hand-written lab notebook as an appendix.
7.
Briefly
but clearly annotate your document in a way which will provide the document
reader with information such as which part of the assignment is this?
a.
what
is being done and why?
b.
how
was it done and what are the results?
c.
how
was this equation obtained and how was it used?
d.
sample
calculations and definitions of symbols/parameters where appropriate; and
e.
BOX AND LABEL
ANSWERS.
In
case of conflict, information in this syllabus supersedes all other course
documents.
Tentative
Course Schedule
The schedule will
be frequently updated as the semester progresses.
Yellow highlight indicates item requires future
attention.
|
# |
date |
topic |
assignments |
|
PART
I: BIOLOGICAL NEURONS |
|||
|
WEEK
1 |
|||
|
1 |
1/10 |
Course introduction (syllabus) Acquiring course materials Plagiarism What are intelligent systems? [ECE5730CourseOverviewLeechStim.pptx] |
Read
syllabus Watch Chris Urmson:
How a driverless car sees the road HW
1: 1.
Read
the plagiarism tutorial found at http://lib.usm.edu/plagiarism_tutorial/whatis_plagiarism.html and turn in signed statement to
that you completed the quiz at http://lib.usm.edu/plagiarism_tutorial/acceptable_use1.html (if you did!). 2.
Solve
the circuit analysis power problems (4) handed out in class. Project
1 Simulation of a Simple Neuron |
|
2 |
1/12 |
Review of Electric
Circuit Fundamentals Discuss Project 1 Discuss HW #1 Discuss Project 2 |
Read Freeman Skim A. L. Hodgkin
and A. F. Huxley, “A quantitative description of membrane current and its
application to conduction and excitation in nerve,” J. Physiol., vol. 117, pp. 500-544, available at https://link.springer.com/article/10.1007/BF02459568 Project
2 Translation of
F&F “Modeling: Electricity Lessons”
to LTspice® DUE 1/28. Use HW format. Read F&F I.3 Physical Basis for the Resting Potential |
|
3 |
1/14 |
cells Lipids,
Membranes, and the First Cells [Freeman CH 6] [ECE5730CellMembrane.pptx] |
|
|
WEEK
2 |
|||
|
|
1/17 |
NO CLASS: MLK DAY RECESS |
|
|
4 |
1/19 |
Lipids,
Membranes, and the First Cells [Freeman CH 6] [ECE5730CellMembrane.pptx] Neuron Types/Neuron Anatomy [Freeman CH 43] [ECE5730NeuronTypesAnatomyMembranePotential.pptx] Membrane
Potentials |
HW #1 DUE Project 1 DUE |
|
5 |
1/21 |
Membrane
Potentials [Nicholls et al.] Physical Basis for the Resting
Potential [F&F,
I.3] [ECE5730NeuronAnatomyMembranePotential.pptx] |
|
|
WEEK
3 |
|||
|
6 |
1/24 |
Membrane
Potentials [Nicholls et al.] [ECE5730NeuronAnatomyMembranePotential.pptx] Patch-Clamp
Recording [Freeman CH 43] [ECE5730PatchClamp.pptx] Discuss
Project 2 |
Project
3 Translation of
F&F Patch-Clamp Recording to LTspice® DUE 2/4. Use HW format. |
|
7 |
1/26 |
Discuss Project 3 Action Potentials [Freeman] [ECE5730ActionPotentials.pptx] |
Read F&F I.4 Basis of the Nerve Impulse |
|
8 |
1/28 |
Earthworm Action
Potentials (see video on the Olivo website) Squid Giant Axon
Experiments (see videos on the Olivo Bio 330 website) |
Project 2 DUE |
|
WEEK
4 |
|||
|
9 |
1/31 |
Basis
of the Nerve Impulse [Nicholls et al.] Hodgkin-Huxley
Equations |
Read F&F I.5 Properties of Neurons |
|
|
2/2 |
NO
CLASS: |
Read F&F I.6 Electrophysiology of Neuronal Interactions |
|
10 |
2/4 |
Discuss Project 4 Animal Nervous Systems: The
Synapse [Freeman CH 43.3] [ECE5730Synapses.pptx] |
Project
4 Translation of
F&F Physical Basis for the Resting
Potential to MATLAB® DUE 2/11. Use HW format. Videos are
available at |
|
WEEK
5 |
|||
|
11 |
2/7 |
Project 3
Presentations |
Project 3 DUE |
|
12 |
2/9 |
Discuss Project 5 Properties
of Neurons [F&F I.5] Electrophysiology
of Neuronal Interactions [F&F
1.6] |
Project
5 Translation of
F&F I.4 Basis of the Nerve Impulse to
MATLAB® DUE 2/21. Use HW format. |
|
13 |
2/11 |
The
Vertebrate Nervous System and
Human Brain [Freeman 43.4] [ECE5730VertebrateNervousSystem.pptx] Project 4 Presentations |
Project 4 DUE |
|
WEEK
6 |
|||
|
14 |
2/14 |
Discuss Project 6 “Video of Hubel
& Wiesel's experiments on visual cortex” Neuronal
Oscillators [F&F I.7] |
|
|
|
2/16 |
Instructor Absence |
|
|
|
2/18 |
Instructor Absence |
|
|
WEEK
7 |
|||
|
15 |
2/21 |
Optimal Control
Applied to Neural Stimulation [Ferguson] Instrumentation for
Neural Stimulation [Essenburg] Project 5
Presentations |
Project 5 DUE |
|
16 |
2/23 |
Discuss Project 7 |
|
|
17 |
2/25 |
Discuss Project 6 |
Read Zurada: Preface,
CH 1 Artificial Neural System:
Preliminaries, |
|
PART
II: ARTIFICIAL NEURAL NETWORKS |
|||
|
WEEK
8 |
|||
|
18 |
2/28 |
Introduction
to Artificial Neural Systems [Zurada] 2.1 Biological Neurons and Their
Artificial Models |
Read How
I Built an AI to Sort 2 Tons of Lego Pieces
by Jacques Mattheij,
example application of neural networks to pattern recognition problem HW
2: Zurada: CH 2:
1, 4, 14. (Use homework
format) |
|
19 |
3/2 |
2.1
Biological Neurons and Their Artificial Models 2.2 Models of Artificial Neural
Networks [Zurada] |
Read Zurada CH 3 Single
Layer Perceptron Classifiers A3 Time-Varying and Gradient Vectors,
Jacobian, and Hessian Matrices |
|
|
3/4 |
NO CLASS: SPIRIT DAY RECESS |
|
|
WEEK
9 |
|||
|
20 |
3/14 |
2.3 Neural Processing 2.4 Learning and Adaptation |
|
|
21 |
3/16 |
2.4 Learning and Adaptation |
|
|
22 |
3/18 |
2.4 Learning and Adaptation 3
Single-Layer Perceptron Classifiers [Zurada] |
HW 2 DUE HW
3: Zurada: CH 3:
3, 5, 6, 7, 8, 13 (use MATLAB® to plot the error surface in 3D and to
prepare a contour plot as in Fig. P3.13 of [Zurada].
(Use homework
format) DUE 3/28 |
|
WEEK
10 |
|||
|
23 |
3/21 |
3
Single-Layer Perceptron Classifiers [Zurada] |
|
|
24 |
3/23 |
3
Single-Layer Perceptron Classifiers [Zurada] |
|
|
25 |
3/25 |
HW 3 Discussion |
|
|
WEEK
11 |
|||
|
26 |
3/28 |
4
Multilayer Feedforward Networks [Zurada] |
HW #3 DUE |
|
27 |
3/30 |
4
Multilayer Feedforward Networks [Zurada] |
Review
the perspective of Michael Jordan as described in “Machine-Learning Maestro
Michael Jordan on the Delusions of Big Data and Other Huge Engineering
Efforts” at http://spectrum.ieee.org/robotics/artificial-intelligence/machinelearning-maestro-michael-jordan-on-the-delusions-of-big-data-and-other-huge-engineering-efforts Read
the online article How
a Pioneer of Machine Learning Became One of Its Sharpest Critics by Kevin Hartnett about the
perspective of Judea Pearl. Read Deep Learning Reinvents the
Hearing Aid by DeLiang Wang |
|
28 |
4/1 |
4
Multilayer Feedforward Networks [Zurada] |
Project
6: Design of a Multilayer
Feedforward Neural Network Classifier and Approximator project files: Read
Zurada CH 5 Single-Layer
Feedback Networks |
|
WEEK
12 |
|||
|
29 |
4/4 |
Project 6 A Bayesian perspective of training [Bishop] |
|
|
30 |
4/6 |
MFNN Hardware |
|
|
31 |
4/8 |
Dynamical Systems Stability/Lyapunov
Functions [Scheinerman] [Zurada] |
|
|
WEEK
13 |
|||
|
32 |
4/11 |
Project 6 Dynamical Systems Stability/Lyapunov
Functions [Scheinerman] [Zurada] |
|
|
33 |
4/13 |
HW #3 |
|
|
34 |
4/15 |
Single-Layer
Feedback Networks [Zurada CH 5] |
|
|
WEEK
14 |
|||
|
35 |
4/18 |
Project
6 [Zurada CH 6] |
Project 6 DUE Project
7: Study of an Associative Memory Browse the October 2021 edition of IEEE Spectrum: “The
Turbulent Past and Uncertain Future of Artificial Intelligence,” particularly
“How Deep Learning Works,” “Deep Learning’s Diminishing Returns,” and “7
Revealing Ways AIs Fail”. Read “Andrew Ng: Unbiggen AI” here. |
|
36 |
4/20 |
Associative Memories [Zurada CH 6] |
|
|
37 |
4/22 |
What is “Deep
Learning?” |
|
|
WEEK
15 |
|||
|
38 |
THU |
Final Exam 12:30PM-2:30PM |
Project 7 DUE |