ECE 5730 Foundations of Neural Networks

Fall 2018
version 4 December 2018

Instructor

Dr. Damon A. Miller, Associate Professor of Electrical and Computer Engineering, Western Michigan University, College of Engineering and Applied Sciences, Parkview Campus, Room A-240, 269.276.3158, 269.276.3151 (fax), damon.miller@wmich.edu, www.homepages.wmich.edu/~miller/.

Office Hours

Guaranteed office hours are posted on Dr. Miller’s door and at http://homepages.wmich.edu/~miller/. Please respect my office hours.  Other times are available by appointment.

WMU Catalog Description

ECE 5730 Foundations of Neural Networks, 3 hrs.  Biological and artificial neural networks from an electrical and computer engineering perspective. Neuron anatomy. Electrical signaling, learning, and memory in biological neural networks. Development of neural network circuit models. Artificial neural systems including multilayer feedforward neural networks, Hopfield networks, and associative memories. Electronic implementations and engineering applications of neural networks.

 

Prerequisite Abilities

 

You must be able to work independently on research projects and to write professional quality written reports describing your results. You must also be able to write computer programs using a language such as C, MATLAB®, Mathematica®, etc.

 

Fall 2018 Semester

 

Dr. Miller will tailor a course plan for each individual student based on the catalog description and individual research objectives.

 

Textbook and Materials

1.  ALL:  Jacek M. Zurada, Artificial Neural Systems, PWS Publishing, Boston, 1992 (ISBN 0-314-93391-3).  Available from the author, instructions for securing a copy to be provided in class.

2.  NEUROPHYSIOLOGY GROUP: W. Otto Friesen and J. A. Friesen, NeuroDynamix II:  Concepts of Neurophysiology Illustrated by Computer Simulations, Oxford University Press, 2010 (ISBN 978-0-19-537183-3).

3.  NEUROPHYSIOLOGY GROUP: Scott Freeman, Biological Science, Prentice Hall, 2nd edition, 2005 (ISBN 0-13-140941-7):  chapters 6 (“Lipids, Membranes, and the First Cells”), 45 (“Electrical Signals in Animals”), and 46 (“Animal Sensory Systems and Movement”) only.
Later editions of this text are acceptable provided these chapters are present.

 

References

1.      Simon Haykin, Neural Networks: A Comprehensive Foundation, Prentice Hall, 1st ed., 1994. On reserve in ECE Office.

2.      E. Scheinerman, Invitation to Dynamical Systems, Prentice Hall, 1996.

3.      F. Severance, System Modeling and Simulation, Wiley, 2001.

4.      D. A. Miller, R. Arguello, and G. W. Greenwood, “Evolving Artificial Neural Network Structures:  Experimental Results for Biologically-Inspired Adaptive Mutations,” Proceedings of the 2004 Congress on Evolutionary Computation, June 2004.

5.      J. G. Nicholls, A. R. Martin, B. G. Wallace, P. A. Fuchs, From Neuron to Brain, Sinauer Associates, Inc., 2000.

Course Policies

Academic Honesty

General:

“Students are responsible for making themselves aware of and understanding the University policies and procedures that pertain to Academic Honesty. These policies include cheating, fabrication, falsification and forgery, multiple submission, plagiarism, complicity and computer misuse. The academic policies addressing Student Rights and Responsibilities can be found in the Undergraduate Catalog at [http://catalog.wmich.edu/index.php] and the Graduate Catalog at [http://catalog.wmich.edu/index.php]. If there is reason to believe you have been involved in academic dishonesty, you will be referred to the Office of Student Conduct. You will be given the opportunity to review the charge(s) and if you believe you are not responsible, you will have the opportunity for a hearing. You should consult with your instructor if you are uncertain about an issue of academic honesty prior to the submission of an assignment or test.

Students and instructors are responsible for making themselves aware of and abiding by the “Western Michigan University Sexual and Gender-Based Harassment and Violence, Intimate Partner Violence, and Stalking Policy and Procedures” related to prohibited sexual misconduct under Title IX, the Clery Act and the Violence Against Women Act (VAWA) and Campus Safe. Under this policy, responsible employees (including instructors) are required to report claims of sexual misconduct to the Title IX Coordinator or designee (located in the Office of Institutional Equity). Responsible employees are not confidential resources. For a complete list of resources and more information about the policy see www.wmich.edu/sexualmisconduct.

In addition, students are encouraged to access the Code of Conduct, as well as resources and general academic policies on such issues as diversity, religious observance, and student disabilities:

·        Office of Student Conduct www.wmich.edu/conduct

·        Division of Student Affairs www.wmich.edu/students/diversity

·        University Relations Office http://www.wmich.edu/registrar/calendars/interfaith

·        Disability Services for Students www.wmich.edu/disabilityservices

— provided by the WMU Faculty Senate Professional Concerns Committee

Plagiarism:

Plagiarism WILL NOT BE TOLERATED.  For an in-depth exploration of plagiarism, see http://lib.usm.edu/plagiarism_tutorial.html

Grading Basis

1.      Projects (70%) will be assigned on a regular basis.  Some project results will be reported using the IEEE journal paper format; see http://ieeeauthorcenter.ieee.org/wp-content/uploads/Transactions-instructions-only.pdf for details. You may not use any sources other than those provided in class or in this syllabus when preparing your project report without prior approval from the course instructor.  You may be asked to demonstrate your project. LATE PROJECTS WILL NOT BE ACCEPTED AND ARE DUE AT THE BEGINNING OF CLASS. All projects are to be completed individually.

2.      Homework (30%)

Note: OUTSTANDING WORK might earn extra credit.

 

Scale: 0-59 E | 60-64 D | 65-69 DC | 70-74 C | 75-79 CB | 80-84 B | 85-89 BA | 90-100 A |
Numeric scores are rounded to the nearest integer.

Grade Appeals:  If you have a question regarding grading of any course materials, see Dr. Miller within FIVE business days of receiving the grade for the assignment in question.  If you disagree with the assessment of that assignment at that meeting, you must submit a written description of your concern to Dr. Miller via his ECE Department mailbox (not email) within five business days of that meeting.

 

HOMEWORK is assigned in class with a due date. Each problem must be worked on separate page(s).  LATE HOMEWORK will not be accepted, except under extraordinary circumstances. Homework is to be completed individually.

 

Homework should normally be done on 8 1/2'' by 11'' sheets. “Engineer's Pad” sheets are preferred.  Solutions must be done in a neat, structured, logical, and orderly manner with frequent brief notations enabling the grader to readily verify the author's source of information, steps taken, sources of formula, equations, and methods used. USE THE PARTIAL CHECK LIST FOR SUBMITTED HOMEWORK BELOW.  Papers failing to meet these guidelines may not be graded and may be returned, with or without an opportunity for resubmission with a penalty.

 

PARTIAL CHECK LIST FOR SUBMITTED HOMEWORK

 

1.      Each problem must include: (a) author's name, (b) name/title of the assignment, and (c) date of completion.

2.      Use only one side of the paper and include a brief and concise statement of the problem prior to its solution. Begin each problem on a new page.

3.      Number the pages and DOUBLE SPACE the text.

4.      Staple each problem in the upper left corner as needed.

5.      Entitle graphs, label and include axes, include key symbols for multiple curve graphs, and give brief notes of explanation where appropriate.

6.      Briefly but clearly annotate your document in a way which will provide the document reader with information such as

a.      which part of the assignment is this?

b.      what is being done and why?

c.      how was it done and what are the results?

d.      how was this equation obtained and how was it used?

e.      sample calculations and definitions of symbols/parameters where appropriate; and

f.       BOX AND LABEL ANSWERS.

 

In case of conflict, information in this syllabus supersedes all other course documents.

 

Other

Students are expected to attend all lectures and to be on time (homework is collected at the beginning of class). Electronic devices are to be turned off (unless there is a safety issue) during lecture unless arrangements have been made with the instructor.

Course Schedule
Will be updated as the semester progresses

 

Class #

Date

Topic

[some topics may be verbatim from course references]

Assignment

WEEK 1

1                  

8/31

Course introduction

EJ/JW/RM
Secure a copy of the Zurada, Friesen and Friesen, and Freeman texts listed above.

 

NK/CP
Secure a copy of the Zurada text.

 

ALL

Read Zurada: Preface, CH 1 Artificial Neural System: Preliminaries,
CH 2 Fundamental Concepts and Models of Artificial Neural Systems,
A1 Vectors and Matrices,
A6 Analytic Geometry in Euclidian Space in Cartesian Coordinates

WEEK 2

2                  

9/4

Introduction to Artificial Neural Systems
Biological Neurons and Their Artificial Models [Zurada 2.1]

Models of Artificial Neural Networks

[Zurada 2.2]

HW 1:  Zurada:  CH 2:  1, 4, 14.
Verify the results of Figure 2.15 using LTspice and MATLAB® (numerically solve the differential equations that describe the circuit)

(Use homework format)
HW #1 Example MATLAB® Code
DUE 9/21 in Dr. Miller’s department mailbox

3                  

9/6

Models of Artificial Neural Networks

[Zurada 2.2]

 

Learning and Adaptation

[Zurada 2.4]

 

Neural Network Learning Rules

[Zurada 2.5]


LE: 6730 Presentation (12:25PM)
ZR: 6730 Presentation

(12:35PM)

 

WEEK 3

4                  

9/11

ZR: 6730 Presentation (cont’d)

Neural Network Learning Rules

[Zurada 2.5]

 

Classification Model, Features, and Feature Recognition
[Zurada 3.1]

 

Discriminant Functions

[Zurada 3.2]

 

Machine Learning and Minimum Distance Classification
[Zurada 3.3]

Read Zurada CH 3 Single Layer Perceptron Classifiers
CH 4 Multilayer Feedforward Networks
A2 Quadratic Forms and Definite Matrices

A3 Time-Varying and Gradient Vectors, Jacobian, and Hessian Matrices

5                  

9/13

Nonparametric Training Concept
[Zurada 3.4]

 

Training and Classification Using the Discrete Perceptron…

[Zurada 3.5]

 

Single-Layer Continuous Perceptron Networks for Linearly Separable Classifications

[Zurada 3.6]

 

Multicategory Single-Layer Perceptron Networks

[Zurada 3.7]

HW 2:  Zurada:  CH 3:  3, 5, 6, 7, 8, 13 (use MATLAB® to plot the error surface in 3D and to prepare a contour plot as in Fig. P3.13 of [Zurada]).

(Use homework format)

DUE 9/28 via Dr. Miller’s mailbox

WEEK 4

6                  

9/18

GC: 6730 Presentation (11:30AM)

LE: 6730 Presentation

(11:40AM)

ZR: 6730 Presentation

(11:50AM)

 

Linearly Nonseparable Patter Classification
[Zurada 4.1]

 

Delta Learning Rule for Multiperceptron Layer

[Zurada 4.2]

 

7                  

9/20

Generalized Delta Learning Rule

[Zurada 4.3]

 

Feedforward Recall and Error Back-Propagation Training

[Zurada 4.4]

 

Learning Factors

[Zurada 4.5]

 

Discuss Project 1

EBPT training example

Project 1:  Design of a Multilayer Feedforward Neural Network Classifier and Approximator
(use IEEE report format)
DUE 10/12

project files:
class1t.dat
class2t.dat
class1v.dat
class2v.dat

approx1t.dat
approx1v.dat

 

DUE 10/4

 

9/21

 

HW #1 DUE via Dr. Miller’s mailbox

WEEK 5

8                  

9/25

ZR: 6730 Presentation
(11:30AM)
GC: 6730 Presentation
(11:40AM)

 

Discuss Project 1

Read Zurada CH 5 Single-Layer Feedback Networks
Read Zurada CH 9 Neural Networks Implementation up to section 2 only
A4 Solution of Optimization Problems
A5 Stability of Nonlinear Dynamical Systems

 

Review the perspective of Michael Jordan as described in “Machine-Learning Maestro Michael Jordan on the Delusions of Big Data and Other Huge Engineering Efforts” at http://spectrum.ieee.org/robotics/artificial-intelligence/machinelearning-maestro-michael-jordan-on-the-delusions-of-big-data-and-other-huge-engineering-efforts

 

Read the online article How a Pioneer of Machine Learning Became One of Its Sharpest Critics by Kevin Hartnett about the perspective of Judea Pearl.
Note: This link is provided to share this article only and does not imply endorsement of any views expressed on that webpage.

 

Read Deep Learning Reinvents the Hearing Aid by DeLiang Wang

9                  

9/27

Neural Networks Implementation

[Zurada CH 9]

 

 

9/28

 

HW #2 DUE via Dr. Miller’s mailbox

WEEK 6

10               

10/2

LE: 6730 Presentation
(11:30AM)

GC: 6730 Presentation
(11:40AM)

 

Review progress on Project 1

 

11               

10/4

Dynamical Systems:
Fixed Points and Local Stability
See Scheinerman and Zurada texts

 

 

10/5

 

PROJ PROGRESS REPORT #1 DUE via Dr. Miller’s mailbox

WEEK 7

12               

10/9

ZR: 6730 Presentation
(11:30AM)

LE: 6730 Presentation
(11:40AM)

GC: 6730 Presentation
(11:50AM)

 

Single-Layer Feedback Networks

[Zurada CH 5]

Read Zurada CH 6 Associative Memories

13               

10/11

Associative Memories
[Zurada CH 6]

Project 2:  Study of an Associative Memory
(use IEEE report format)

DUE 10/26

 

10/12

 

PROJ #1 DRAFT DUE via Dr. Miller’s mailbox

WEEK 8

14               

10/16

GC: 6730 Presentation
(11:30AM)

ZR: 6730 Presentation
(11:40AM)

 

 

10/18

 

FALL BREAK

WEEK 9

15               

10/23

ZR: 6730 Presentation
(11:30AM)

Course Planning

 

16               

10/25

Project consultation with Dr. Miller as needed

 

Neurophysiology Group (ER and JW):
Read F&F I.1 Fundamentals of Electricity
Read F&F I.2 Patch-Clamp Recording
Read F&F I.3 Physical Basis for the Resting Potential
(as you read, be sure to insure that all schematics agree with the passive sign convention)

Read Freeman Chapter 6 Lipids, Membranes, and the First Cells

Read http://www.nature.com/scitable/topicpage/what-is-a-cell-14023083
Browse:  http://www.cell.com/pictureshow

 

Neurophysiology Group (ER and JW):

Project 3:
F&F I.1 Fundamentals of Electricity and Simulation of a Simple Neuron
(Use homework format, be sure to include copies of your code)
Project 3 Example MATLAB® Code

DUE 11/6

 

RM:  Project 3:  Use TensorFlow to implement Projects 1 and 2.
DUE 11/6.

 

NK and CP: Prepare one-page summary of proposed project. Include references and milestones. Present in class 10/30.

 

10/26

 

PROJ #1 DUE via Dr. Miller’s mailbox

PROJ #2 DUE via Dr. Miller’s mailbox

WEEK 10

17               

10/30

LE: 6730 Presentation

(11:30AM)

GC: 6730 Presentation

(11:40AM)

NK: 5730 Presentation
(11:50AM)
CP: 5730 Presentation
(12:00PM)

 

Passive Sign Convention
Discuss Project 3

NK: Provide update on work to use multilayer feedforward neural network to classify skin lesions (present in class 11/13)

 

CP: Provide update on work to use radial basis function neural network to interpolate across lookup tables needed for control system (present in class 11/13)

18               

11/1

Neuron membranes: structure, permeability, channels, active and passive ion transport

[Freeman CH 6]

Neurophysiology Group (ER and JW):
Read Freeman
Chapter 45 Electrical Signals in Animals and
Chapter 46 Animal Sensory Systems and Movement

Read F&F I.4 Basis of the Nerve Impulse

Skim A. L. Hodgkin and A. F. Huxley, “A quantitative description of membrane current and its application to conduction and excitation in nerve,” J. Physiol., vol. 117, pp. 500-544, available at https://link.springer.com/article/10.1007/BF02459568

WEEK 11

19               

11/6

ZR: 6730 Presentation
(11:30AM)

 

Patch-Clamp Recording [F&F, I.2]
[Freeman]

Neurophysiology Group PROJ #3 DUE
RM PROJ #3 DUE via Dr. Miller’s mailbox

Project 4 F&F I.2 Patch-Clamp Recording (Ion Channels)

DUE 11/16 via Dr. Miller’s mailbox

 

11/8

 

NO CLASS

WEEK 12

20               

11/13

LE: 6730 Presentation

(11:30AM)
GC: 6730 Presentation

(11:50AM)

NK: 5730 Presentation
(12:00PM)
CP: 5730 Presentation
(12:10PM)

CP: Investigate inversion of trained radial basis function neural network. Present in class 11/27.
NK: Continue development of lesion classification system. Present in class 11/27.

21               

11/15

Membrane Potential

[Freeman]

[Nicholls et al.]

 

Neuron Anatomy
[Freeman]

 

Action Potentials

[Freeman]

 

View videos on historical giant squid axons available here

Videos are available at
http://www.science.smith.edu/departments/neurosci/courses/bio330/videos.html [Olivo website]


Neurophysiology Group (ER and JW):

Project 5 Resting Potential and the Nerve Impulse
Topics: Physical Basis for the Resting Potential {F&F. I.3], H&H equation simulation,  Basis of the Nerve Impulse [F&F, I.4]
(Use homework format)

DUE 11/29 via Dr. Miller’s mailbox

 

11/16

 

Neurophysiology Group PROJ #4 DUE via Dr. Miller’s mailbox

WEEK 13

22               

11/20

ZR: ECE 6730 Presentation

(11:30AM)

 

Review HW/projects

Neurophysiology Group (ER and JW):
Project 6 Complete the “Neurodynamics II Modeling: Neuron Lessons” in section I.5 of F&F (starts on page 93).

Due 11/29

(Use homework format)

23               

 

 

THANKSGIVING

WEEK 14

24               

11/27

LE: 6730 Presentation

(11:30AM)

GC: 6730 Presentation

(11:40AM)

CP: 5730 Presentation
(11:50PM)

NK: 5730 Presentation
(12:00PM)

NK:RM:CP: Prepare draft of final report for your project. DUE 12/4

25               

11/29

View videos on historical giant squid axons available here

Electrical Signals in Animals

[Freeman CH 45]

 

Hodgkin-Huxley Equations

discuss H&H paper

Neurophysiology Group PROJ #6 DUE
Project 7  Complete the “Neurodynamix II Modeling: Synapse Lessons”

(F&F I.6 , starts on page 111)

Complete the “Neurodynamix II Modeling:  Circuit Lessons”

(F&F I.7, starts on page 134)

(Use homework format)

DUE 12/12 at final exam

WEEK 15

26               

12/4

Review draft reports

NK:RM:CP: Final draft report for your project due.

 

12/6

NO CLASS

NK:RM:CP: Final report for your project DUE 12/12 at final exam. Present result at final.

WEEK 16

27               

12/12 WED.

10:15AM-12:15PM

FINAL EXAM

(verify this day/time on your own)

NK:RM:CP: Final report due. Present results.
ECE 6730 presentations

 

Credits

 

Parts adapted/adopted from syllabi by J. Gesink and J. Kelemen.

© 2018 Damon A. Miller

Return to Dr. Miller’s homepage