ECE 5730 Foundations of Neural Networks

Spring 2022
version 20 April 2022

Instructor

Dr. Damon A. Miller, Associate Professor of Electrical and Computer Engineering, Western Michigan University, College of Engineering and Applied Sciences, Floyd Hall, Room A-240, 269.276.3158, 269.276.3151 (fax), damon.miller@wmich.edu, www.homepages.wmich.edu/~miller/, https://wmich.webex.com/meet/damon.miller.

Office Hours
Dr. Miller is available for in-person office hours as posted in his schedule. Appointments at other times are requested by email to damon.miller@wmich.edu.

Catalog Description

ECE 5730 Foundations of Neural Networks, 3 hrs.  Biological and artificial neural networks from an electrical and computer engineering perspective. Neuron anatomy. Electrical signaling, learning, and memory in biological neural networks. Development of neural network circuit models. Artificial neural systems including multilayer feedforward neural networks, Hopfield networks, and associative memories. Electronic implementations and engineering applications of neural networks.

 

Prerequisite Abilities

You must be able to work independently on research projects and to write a professional quality written reports describing your project work.

 

Copyright Information

Materials prepared by Dr. Miller are © 2022 Damon A. Miller. Other copyrights apply to materials such as text and images from books, datasheets, etc. Consult source documents for copyright information. Any lecture videos are for use in ECE 2100 only and must not be distributed in any way.

 

Acknowledgments
ECE faculty member(s), particularly J. Gesink, contributed to the course syllabus. Dr. Miller also thanks Instructional Designer M. Strock and the Educational Technology Department for contributions to this syllabus as ported from an ECE 2100 syllabus.

 

Course Objectives

Student will develop:

1.      an understanding of the characteristics of intelligent systems;

2.      an ability to develop numerical solutions of ordinary differential equations;

3.      an understanding of basic neuron cell structure, anatomy, and functionality;

4.      an understanding of neuron interactions via synaptic function;

5.      an understanding of current knowledge of neural mechanisms that enable high level information processing in biological organisms;

6.      an ability to develop computer models of biological neuron(s) and biological neural networks;

7.      an ability to design, analyze, and simulate circuits to model biological neuron(s) and biological neural networks;

8.      an understanding of common artificial neural network (ANN) architectures;

9.      an understanding of adaptation and ‘learning’ in ANNs;

10.   an understanding of classifier design, including the role of discriminant functions;

11.   an ability to design and evaluate a multilayer feedforward neural network approximator or classifier;

12.   a basic understanding of dynamical systems;

13.   an ability to perform a Lyapunov stability analysis;

14.   an understanding of discrete and continuous feedback networks;

15.   an understanding of associative memories;

16.   an understanding of unsupervised learning techniques (ABET: a).

17.   an ability to utilize computer simulations to study artificial neural networks;

18.   an understanding of application areas for artificial neural networks, including pattern recognition, image processing, and signal processing;

19.   effective and ethical research methods with particular attention to proper citation techniques; and

20.   an ability to produce a concise summary of work performed using a standard journal paper format.

 

Textbook and Materials

 

Required:

1.      Jacek M. Zurada, Artificial Neural Systems, PWS Publishing, Boston, 1992 (ISBN 0-314-93391-3).  Available from the author, instructions for securing a copy to be provided in class.

2.      W. Otto Friesen and J. A. Friesen, NeuroDynamix II:  Concepts of Neurophysiology Illustrated by Computer Simulations, Oxford University Press, 2010 (ISBN 978-0-19-537183-3).

3.      Scott Freeman et al., Biological Science, Pearson, 7th edition, 2019 (ISBN-13: 9780135276815), available as an EBook.

4.      Linear Technology, LTspice®, available at no cost at http://www.linear.com/designtools/software/.
You are responsible for ensuring access to a working copy.

SPICE EXAMPLES

a.      VCCS example (problem 4.43 from Nilsson and Reidel, Electric Circuits, 8th ed.)

b.      CCCS and CCVS example (problem 4.51 from Nilsson and Reidel Electric Circuits, 8th ed.)

c.      VCVS example (simple operational amplifier model)

d.      Chua’s “Simple” Chaotic Circuit (need the National Semiconductor LM741 model available as part of laboratory six in the course schedule below.

5.      The MathWorks, MATLAB®.  The student version is a tremendous value as this package includes many add-ons that must be purchased separately for use in a professional version. [An alternative programming language can be used with permission of the course instructor].

 

References:
(see Dr. Miller, might be put on reserve in ECE Department Office, check-out with WMU ID)

1.  E. M. Izhikevich, Dynamical Systems in Neuroscience:  The Geometry of Excitability and Bursting, The MIT Press, Cambridge, Massachusetts, 2007.

2.  Simon Haykin, Neural Networks: A Comprehensive Foundation, IEEE Press, 1st edition, 1994.

3.  A. S. Sedra and K. C. Smith, Microelectronic Circuits, Oxford University Press, 5th edition, 1998.

4.  M. J. Maron, Numerical Analysis:  A Practical Approach, Macmillan Publishing Co., Inc., 1982.

5.  J. G. Nicholls, A. R. Martin, B. G. Wallace, P. A. Fuchs, From Neuron to Brain, Sinauer Associates, Inc., 2000. 

6.  E. Scheinerman, Invitation to Dynamical Systems, Prentice Hall, 1996.

7.  F. Severance, System Modeling and Simulation, Wiley, 2001.

8.  D. A. Miller, R. Arguello, and G. W. Greenwood, “Evolving Artificial Neural Network Structures:  Experimental Results for Biologically-Inspired Adaptive Mutations,” Proceedings of the 2004 Congress on Evolutionary Computation, June 2004.

8.  C. M. Bishop, Neural Networks for Pattern Recognition, Oxford University Press, 1995.

9.  Scott Freeman, Biological Science, Pearson, 2nd edition, 2005.

 

 

 

Online References:

1.      W. H. Press, S. A. Teukolsky, W. T. Vetterling, and B. P. Flannery, Numerical Recipes in C:  The Art of Scientific Computing, Cambridge University Press, 2nd edition, 1992.  Available online at http://apps.nrbook.com/c/index.html.

2.      C. R. Nave, HyperPhysics website, http://hyperphysics.phy-astr.gsu.edu/hbase/hframe.html, outstanding physics tutorial/reference.

3.      http://www.nature.com/scitable/topicpage/what-is-a-cell-14023083

4.      http://www.cell.com/pictureshow

5.      Richard F. Olivo, Biological Sciences 330/331 (Neurophysiology) website, Smith College, http://www.science.smith.edu/departments/NeuroSci/courses/bio330/, See the links for videos shown in class.

6.      A. L. Hodgkin and A. F. Huxley, “A quantitative description of membrane current and its application to conduction and excitation in nerve,” J. Physiol., no. 117, pp. 500-544, 1952. Available at http://jp.physoc.org/cgi/content/full/538/1/2.

7.      NeuroDynamix II website

8.      Donovan Suires, Instrumentation Electronics for an Integrated Electrophysiology Data Acquisition and Stimulation System (2013). Masters Theses. 447.
https://scholarworks.wmich.edu/masters_theses/447

9.      Alexandra C. Ferguson, Optimization and Experimental Application of Current Stimuli to Leech Pressure-Sensitive Mechanosensory Cells (2017). Masters Theses. 1131.
https://scholarworks.wmich.edu/masters_theses/1131.

10.   Lucas M. Essenburg, Intracellular Electrometer (2019). Masters Theses. 5099.
https://scholarworks.wmich.edu/masters_theses/5099.

 

Course Policies

Academic Honesty

General:

Students are responsible for making themselves aware of and understanding the University policies and procedures that pertain to Academic Honesty. These policies include cheating, fabrication, falsification and forgery, multiple submission, plagiarism, complicity and computer misuse. The academic policies addressing Student Rights and Responsibilities can be found in the Undergraduate Catalog at http://catalog.wmich.edu/index.php?catoid=35 and the Graduate Catalog at http://catalog.wmich.edu/index.php?catoid=39. If there is reason to believe you have been involved in academic dishonesty, you will be referred to the Office of Student Conduct. You will be given the opportunity to review the charge(s) and if you believe you are not responsible, you will have the opportunity for a hearing. You should consult with your instructor if you are uncertain about an issue of academic honesty prior to the submission of an assignment or test.

 

Students and instructors are responsible for making themselves aware of and abiding by the “Western Michigan University Sexual and Gender-Based Harassment and Violence, Intimate Partner Violence, and Stalking Policy and Procedures” related to prohibited sexual misconduct under Title IX, the Clery Act and the Violence Against Women Act (VAWA) and Campus Safe. Under this policy, responsible employees (including instructors) are required to report claims of sexual misconduct to the Title IX Coordinator or designee (located in the Office of Institutional Equity). Responsible employees are not confidential resources. For a complete list of resources and more information about the policy see http://www.wmich.edu/sexualmisconduct.

 

In addition, students are encouraged to access the Code of Conduct, as well as resources and general academic policies on such issues as diversity, religious observance, and student disabilities:

·        Office of Student Conduct http://www.wmich.edu/conduct

·        Division of Student Affairs http://www.wmich.edu/students/diversity

·        Registrar’s Office http://www.wmich.edu/registrar/calendars/interfaith

·        Disability Services for Students http://www.wmich.edu/disabilityservices.

 

— section provided by the WMU Faculty Senate with minor link reformatting

 

Plagiarism: For an in-depth exploration of plagiarism, see http://libguides.wmich.edu/plagiarism

 

COVID-19 Statement

 

Safety requirements are in place to minimize exposure to the Western Michigan University community. These guidelines apply to all in-person and hybrid classes held inside a WMU building to ensure the safety of all students, faculty, and staff during the pandemic. Noncompliance is a violation of the class requirements and the Student Code. https://wmich.edu/conduct/code

Facial coverings (masks), over both the nose and mouth, are required for all students while in- class, no matter the size of the space. Following this recommendation can minimize the transmission of the virus, which is spread between people interacting in close proximity through speaking, coughing, or sneezing. During specified classes in which facial coverings (masks) would prevent required class elements, students may remove facial coverings (masks) with instructor permission, in accordance with the exceptions in the Facial Covering (mask) Policy ("such as playing an instrument, acting, singing, etc."). https://wmich.edu/policies/facial-covering-mask

Facial coverings (masks) must remain in place throughout the class. Any student who removes the mandatory facial covering (mask) during class will be required to leave the classroom immediately.

Students who are unable to wear a facial covering (mask) for medical/disability reasons must contact Disability Services for Students before they attend class. https://wmich.edu/disabilityservices

— section provided by the WMU Faculty Senate, highlight added

NO FOOD OR DRINK IN LECTURE OR LAB.

 

ONLY STUDENTS WITH A GREEN BADGE STATUS ARE PERMITTED IN LECTURE OR LAB. YOU MUST BE ABLE TO DEMONSTRATE YOUR BADGE STATUS.

Accommodations

If you have a documented disability and verification from the Disability Services for Students (DSS), and wish to discuss academic accommodations, please contact your instructor as soon as possible. It is the student’s responsibility to provide documentation of disability to DSS and meet with a DSS counselor to request special accommodation before classes start.

 

Grading Basis

1.      Projects (70%) will be assigned on a regular basis. 

 

Some project results will be reported using the IEEE journal paper format; see http://ieeeauthorcenter.ieee.org/wp-content/uploads/Transactions-instructions-only.pdf for details. You may not use any sources other than those provided in class or in this syllabus when preparing your project report without prior approval from the course instructor.  You may be asked to demonstrate your project.

 

LATE PROJECTS WILL NOT BE ACCEPTED AND ARE DUE AT THE BEGINNING OF CLASS. Unless otherwise noted projects are completed individually.

2.      Homework: 30%

OUTSTANDING WORK might earn extra credit.

 

Scale: 0-60 E | 60-65 D | 65-70 DC | 70-75 C | 75-80 CB | 80-85 B | 85-90 BA | 90-100 A |

Midterm grades are not assigned.

Grade Appeals
If you have a question regarding graded course materials (e.g. exam problems, homework problems, laboratory reports, etc.), contact Dr. Miller within TWO business days of receiving the grade for the assignment in question.

Late Assignments will not be accepted without a documented excuse. If an emergency prevents you from submitting an assignment on-time, contact your instructor PRIOR to the assignment due date or as soon as you can, via email.  Failure to adhere to this policy will result in zero credit for the assignment.

PARTIAL CHECK LIST FOR SUBMITTED ASSIGNMENTS

 

1.      Each problem must include: (a) author's name, (b) name/title of the assignment, and (c) date of completion.

2.      Use only one side of the paper and include a brief and concise statement of the problem prior to its solution. Begin each problem on a new page.

3.      Number the pages.

4.      Staple each problem in the upper left corner as needed.

5.      Entitle graphs, label and include axes, include key symbols for multiple curve graphs, and give brief notes of explanation where appropriate.

6.      USE A WHITE BACKGROUND FOR ALL LTspice® schematics and waveform plots. Reports must not be handwritten, though you must include copies of your hand-written lab notebook as an appendix.

7.      Briefly but clearly annotate your document in a way which will provide the document reader with information such as which part of the assignment is this?

a.      what is being done and why?

b.      how was it done and what are the results?

c.      how was this equation obtained and how was it used?

d.      sample calculations and definitions of symbols/parameters where appropriate; and

e.      BOX AND LABEL ANSWERS.

 

In case of conflict, information in this syllabus supersedes all other course documents.

 

Tentative Course Schedule
The schedule will be frequently updated as the semester progresses.

Yellow highlight indicates item requires future attention.

 

#

date

topic

assignments

PART I: BIOLOGICAL NEURONS

WEEK 1

 1 

1/10

Course introduction (syllabus)

 

Acquiring course materials

 

Plagiarism

 

What are intelligent systems?

[ECE5730CourseOverviewLeechStim.pptx]

Read syllabus
Read F&F I.1 Fundamentals of Electricity
Read F&F I.2 Patch-Clamp Recording
(as you read, be sure to insure that all formulas agree with the passive sign convention)
Read Freeman Chapter 6 Lipids, Membranes, and the First Cells
Read
http://www.nature.com/scitable/topicpage/what-is-a-cell-14023083
Browse:  http://www.cell.com/pictureshow

Watch Chris Urmson: How a driverless car sees the road

HW 1:
DUE 1/19

1.      Read the plagiarism tutorial found at http://lib.usm.edu/plagiarism_tutorial/whatis_plagiarism.html and turn in signed statement to that you completed the quiz at http://lib.usm.edu/plagiarism_tutorial/acceptable_use1.html (if you did!).

2.      Solve the circuit analysis power problems (4) handed out in class.

 

Project 1 Simulation of a Simple Neuron
(Use homework format, be sure to include copies of your code) DUE 1/19

 2 

1/12

Review of Electric Circuit Fundamentals

 

Discuss Project 1

 

Discuss HW #1

 

Discuss Project 2

Read Freeman
Chapter 43 Animal Nervous Systems and
Chapter 44 Animal Sensory Systems
Chapter 45 Animal Movement

 

Skim A. L. Hodgkin and A. F. Huxley, “A quantitative description of membrane current and its application to conduction and excitation in nerve,” J. Physiol., vol. 117, pp. 500-544, available at https://link.springer.com/article/10.1007/BF02459568

 

Project 2 Translation of F&F “Modeling: Electricity Lessons” to LTspice® DUE 1/28. Use HW format.

 

Read F&F I.3 Physical Basis for the Resting Potential

 3 

1/14

cells

 

Lipids, Membranes, and the First Cells [Freeman CH 6]

[ECE5730CellMembrane.pptx]

 

WEEK 2

 

1/17

NO CLASS: MLK DAY RECESS

 

 

 

 4 

1/19

Lipids, Membranes, and the First Cells [Freeman CH 6]

[ECE5730CellMembrane.pptx]

 

Neuron Types/Neuron Anatomy

[Freeman CH 43]

[ECE5730NeuronTypesAnatomyMembranePotential.pptx]

 

Membrane Potentials
[Freeman CH 43 and Ch 45 in 2nd ed.]
[ECE5730NeuronAnatomyMembranePotential.pptx]

HW #1 DUE

Project 1 DUE

 5 

1/21

Membrane Potentials
[Freeman CH 43 and Ch 45 in 2nd ed.]

[Nicholls et al.]

Physical Basis for the Resting Potential

[F&F, I.3]

[ECE5730NeuronAnatomyMembranePotential.pptx]

 

WEEK 3

 6 

1/24

Membrane Potentials
[Freeman CH 43 and Ch 45 in 2nd ed.]

[Nicholls et al.]

[ECE5730NeuronAnatomyMembranePotential.pptx]

 

Patch-Clamp Recording

[Freeman CH 43]
[F&F, I.2]

[ECE5730PatchClamp.pptx]

 

Discuss Project 2

Project 3 Translation of F&F Patch-Clamp Recording to LTspice® DUE 2/4. Use HW format.

 7 

1/26

Discuss Project 3

 

Action Potentials

[Freeman]

[ECE5730ActionPotentials.pptx]

Read F&F I.4 Basis of the Nerve Impulse

 8 

1/28

Earthworm Action Potentials

(see video on the Olivo website)
(also see [Squires])

 

Squid Giant Axon Experiments

(see videos on the Olivo Bio 330 website)

Project 2 DUE

WEEK 4

 9 

1/31

Basis of the Nerve Impulse
[F&F, I.4]
(includes “voltage-clamp” method)

[Nicholls et al.]

 

Hodgkin-Huxley Equations
discuss H&H paper

Read F&F I.5 Properties of Neurons

 

2/2

NO CLASS:
CAMPUS CLOSURE

Read F&F I.6 Electrophysiology of Neuronal Interactions

10 

2/4

Discuss Project 4

 

Animal Nervous Systems: The Synapse

[Freeman CH 43.3]

[ECE5730Synapses.pptx]

Project 4 Translation of F&F Physical Basis for the Resting Potential to MATLAB® DUE 2/11. Use HW format.

Videos are available at
http://www.science.smith.edu/departments/neurosci/courses/bio330/videos.html [Olivo website]

WEEK 5

11 

2/7

Project 3 Presentations

 

Project 3 DUE

12 

2/9

Discuss Project 5

 

Properties of Neurons

[F&F I.5]

 

Electrophysiology of Neuronal Interactions

[F&F 1.6]

Project 5 Translation of F&F I.4 Basis of the Nerve Impulse to MATLAB® DUE 2/21. Use HW format.

13 

2/11

The Vertebrate Nervous System and Human Brain

[Freeman 43.4]

[ECE5730VertebrateNervousSystem.pptx]

 

Project 4 Presentations

Project 4 DUE
Read F&F I.7 Neuronal Oscillators

WEEK 6

14 

2/14

Discuss Project 6

 

“Video of Hubel & Wiesel's experiments on visual cortex”
[https://www.science.smith.edu/departments/neurosci/courses/bio330/vision/VisualCortexHiRes.mp4

 

Neuronal Oscillators

[F&F I.7]

Project 6 Translation of F&F I.5 Properties of Neurons (Fig. I-5.1) to MATLAB® DUE 2/25. Use HW format.

 

2/16

Instructor Absence

 

 

2/18

Instructor Absence

 

WEEK 7

15 

2/21

Optimal Control Applied to Neural Stimulation [Ferguson]

Instrumentation for Neural Stimulation [Essenburg]

 

Project 5 Presentations

Project 5 DUE

16 

2/23

Discuss Project 7

Project 7 Translation of F&F Neuronal Oscillators to MATLAB® DUE 3/14. Use HW format.

 

17 

2/25

Discuss Project 6

Read Zurada: Preface, CH 1 Artificial Neural System: Preliminaries,
CH 2 Fundamental Concepts and Models of Artificial Neural Systems,
A1 Vectors and Matrices,
A6 Analytic Geometry in Euclidian Space in Cartesian Coordinates

PART II: ARTIFICIAL NEURAL NETWORKS

WEEK 8

18 

2/28

Introduction to Artificial Neural Systems

[Zurada]

 

2.1 Biological Neurons and Their Artificial Models
[Zurada]

Read How I Built an AI to Sort 2 Tons of Lego Pieces by  Jacques Mattheij, example application of neural networks to pattern recognition problem

 

HW 2:  Zurada:  CH 2:  1, 4, 14.
Verify the results of Figure 2.15 using LTspice® and MATLAB® (numerically solve the differential equations that describe the circuit)

(Use homework format)
DUE 3/18

19 

3/2

2.1 Biological Neurons and Their Artificial Models
[Zurada]

 

2.2 Models of Artificial Neural Networks

[Zurada]

Read Zurada CH 3 Single Layer Perceptron Classifiers
CH 4 Multilayer Feedforward Networks
A2 Quadratic Forms and Definite Matrices

A3 Time-Varying and Gradient Vectors, Jacobian, and Hessian Matrices

 

3/4

NO CLASS: SPIRIT DAY RECESS

 

WEEK 9

20 

3/14

2.3 Neural Processing
[Zurada]

 

2.4 Learning and Adaptation
[Zurada]

 

21 

3/16

2.4 Learning and Adaptation
[Zurada]

 

22 

3/18

2.4 Learning and Adaptation
[Zurada]

 

3 Single-Layer Perceptron Classifiers

[Zurada]

HW 2 DUE

 

HW 3:  Zurada:  CH 3:  3, 5, 6, 7, 8, 13 (use MATLAB® to plot the error surface in 3D and to prepare a contour plot as in Fig. P3.13 of [Zurada].

(Use homework format)

DUE 3/28

WEEK 10

23 

3/21

3 Single-Layer Perceptron Classifiers

[Zurada]
LAST DAY TO WTIHDRAW

 

24 

3/23

3 Single-Layer Perceptron Classifiers

[Zurada]

 

25 

3/25

HW 3 Discussion

 

WEEK 11

26 

3/28

4 Multilayer Feedforward Networks

[Zurada]

HW #3 DUE

27 

3/30

4 Multilayer Feedforward Networks

[Zurada]

Review the perspective of Michael Jordan as described in “Machine-Learning Maestro Michael Jordan on the Delusions of Big Data and Other Huge Engineering Efforts” at http://spectrum.ieee.org/robotics/artificial-intelligence/machinelearning-maestro-michael-jordan-on-the-delusions-of-big-data-and-other-huge-engineering-efforts

 

Read the online article How a Pioneer of Machine Learning Became One of Its Sharpest Critics by Kevin Hartnett about the perspective of Judea Pearl.
Note: This link is provided to share this article only and does not imply endorsement of any views expressed on that webpage.

 

Read Deep Learning Reinvents the Hearing Aid by DeLiang Wang

28 

4/1

4 Multilayer Feedforward Networks

[Zurada]
Project 6

Project 6: Design of a Multilayer Feedforward Neural Network Classifier and Approximator
(use IEEE report format)
DUE 4/18

 

project files:
approx1t.dat
approx1v.dat

 

Read Zurada CH 5 Single-Layer Feedback Networks
Read Zurada CH 9 Neural Networks Implementation up to section 2 only
A4 Solution of Optimization Problems
A5 Stability of Nonlinear Dynamical Systems

WEEK 12

29 

4/4

Project 6

 

A Bayesian perspective of training [Bishop]

 

30 

4/6

MFNN Hardware
[Zurada]

 

31 

4/8

Dynamical Systems

Stability/Lyapunov Functions

[Scheinerman]

[Zurada]

 

WEEK 13

32 

4/11

Project 6

 

Dynamical Systems

Stability/Lyapunov Functions

[Scheinerman]

[Zurada]

 

33 

4/13

HW #3

 

34 

4/15

Single-Layer Feedback Networks

[Zurada CH 5]

 

WEEK 14

35 

4/18

Project 6
Associative Memories

[Zurada CH 6]

Project 6 DUE

 

Project 7:  Study of an Associative Memory
(use IEEE report format) DUE 4/28

 

Browse the October 2021 edition of IEEE Spectrum:  “The Turbulent Past and Uncertain Future of Artificial Intelligence,” particularly “How Deep Learning Works,” “Deep Learning’s Diminishing Returns,” and “7 Revealing Ways AIs Fail”.

 

Read “Andrew Ng: Unbiggen AI” here.

36 

4/20

Associative Memories

[Zurada CH 6]

 

37 

4/22

What is “Deep Learning?”
Course Wrap-Up

 

WEEK 15

38 

THU
4/28

 

Final Exam

12:30PM-2:30PM
Project 7 Presentations

Project 7 DUE