ECE 5730 Foundations of Neural Networks

 

Summer I 2016
version 28 June 2016

The online version of this syllabus at http://homepages.wmich.edu/~miller/ECE5730.html has hyperlinks and will be updated as needed.

Instructor

Dr. Damon A. Miller, Associate Professor of Electrical and Computer Engineering, Western Michigan University, College of Engineering and Applied Sciences, Parkview Campus, Room A-240, 269.276.3158, 269.276.3151 (fax), damon.miller@wmich.edu, www.homepages.wmich.edu/~miller/.

Office Hours

Guaranteed office hours are posted on Dr. Miller’s door and at http://homepages.wmich.edu/~miller/. Please respect my office hours.  Other times are available by appointment.

WMU Catalog Description

ECE 5730 Foundations of Neural Networks, 3 hrs.  Biological and artificial neural networks from an electrical and computer engineering perspective. Neuron anatomy. Electrical signaling, learning, and memory in biological neural networks. Development of neural network circuit models. Artificial neural systems including multilayer feedforward neural networks, Hopfield networks, and associative memories. Electronic implementations and engineering applications of neural networks.

 

Prerequisite Abilities

 

You must be able to work independently on research projects and to write a professional quality written reports describing your project work.

 

Reading Assignments

 

You must keep up with your reading. Exam/quiz questions may be developed from any assigned reading material. Much of the course material will require expanding your vocabulary; keep a list of new terms and their definitions. As you read, think of questions to ask in class.

 

ECE 5730 Course Learning Outcomes

 

Graduates of ECE 5730 will exhibit (with most relevant ABET learning outcomes identified):

1.      an understanding of the characteristics of intelligent systems (ABET: a,c);

2.      an ability to develop numerical solutions of ordinary differential equations (ABET: a,e,k);

3.      an understanding of basic neuron cell structure, anatomy, and functionality (ABET: a);

4.      an understanding of neuron interactions via synaptic function (ABET: a);

5.      an understanding of current knowledge of neural mechanisms that enable high level information processing in biological organisms (ABET: a);

6.      an ability to develop computer models of biological neuron(s) and biological neural networks (ABET: a,b,e,k);

7.      an ability to design, analyze, and simulate circuits to model biological neuron(s) and biological neural networks (ABET: a,b,c,e,k);

8.      an understanding of common artificial neural network (ANN) architectures (ABET: a);

9.      an understanding of adaptation and ‘learning’ in ANNs (ABET: a,e);

10.   an understanding of classifier design, including the role of discriminant functions (ABET: a,e) ;

11.   an ability to design and evaluate a multilayer feedforward neural network approximator or classifier (ABET: a,e);

12.   a basic understanding of dynamical systems (ABET: a);

13.   an ability to perform a Lyapunov stability analysis (ABET: a,e);

14.   an understanding of discrete and continuous feedback networks (ABET: a);

15.   an understanding of associative memories (ABET: a);

16.   an understanding of unsupervised learning techniques (ABET: a);

17.   an ability to utilize computer simulations to study artificial neural networks (ABET: b,e,k);

18.   an understanding of application areas for artificial neural networks, including pattern recognition, image processing, and signal processing (ABET: a,c,i);

19.   effective and ethical research methods with particular attention to proper citation techniques (ABET: f,k); and

20.   an ability to produce a concise summary of work performed using a standard journal paper format (ABET: k).

 

Textbook and Materials

Required:

1.  Jacek M. Zurada, Artificial Neural Systems, PWS Publishing, Boston, 1992 (ISBN 0-314-93391-3).  Available from the author, instructions for securing a copy to be provided in class.

2.  W. Otto Friesen and J. A. Friesen, NeuroDynamix II:  Concepts of Neurophysiology Illustrated by Computer Simulations, Oxford University Press, 2010 (ISBN 978-0-19-537183-3).

3.  Scott Freeman, Biological Science, Prentice Hall, 2nd edition, 2005 (ISBN 0-13-140941-7):  chapters 6 (“Lipids, Membranes, and the First Cells”), 45 (“Electrical Signals in Animals”), and 46 (“Animal Sensory Systems and Movement”) only.  Any version of this text is acceptable provided these chapters are present.

4.  Linear Technology, LTspice IV, available at no cost at http://www.linear.com/designtools/software/.  You are responsible for ensuring access to a working copy.

5.  The MathWorks, MATLAB® & SIMULINK®.  The student version is a tremendous value as this package includes many add-ons that must be purchased separately for use in a professional version.

References:
(see Dr. Miller, might be put on reserve in ECE Department Office, check-out with WMU ID)

1.  E. M. Izhikevich, Dynamical Systems in Neuroscience:  The Geometry of Excitability and Bursting, The MIT Press, Cambridge, Massachusetts, 2007.

2.  Simon Haykin, Neural Networks: A Comprehensive Foundation, IEEE Press, 1st edition, 1994.

3.  A. S. Sedra and K. C. Smith, Microelectronic Circuits, Oxford University Press, 5th edition, 1998.

4.  M. J. Maron, Numerical Analysis:  A Practical Approach, Macmillan Publishing Co., Inc., 1982.

5.  J. G. Nichols, A. R. Martin, B. G. Wallace, P. A. Fuchs, From Neuron to Brain, Sinauer Associates, Inc., 2000. 

6.  E. Scheinerman, Invitation to Dynamical Systems, Prentice Hall, 1996.

7.  F. Severance, System Modeling and Simulation, Wiley, 2001.

 

Online References:

1.       W. H. Press, S. A. Teukolsky, W. T. Vetterling, and B. P. Flannery, Numerical Recipes in C:  The Art of Scientific Computing, Cambridge University Press, 2nd edition, 1992.  Available online at http://apps.nrbook.com/c/index.html.

2.      C. R. Nave, HyperPhysics website, http://hyperphysics.phy-astr.gsu.edu/hbase/hframe.html, outstanding physics tutorial/reference.

3.      http://www.nature.com/scitable/topicpage/what-is-a-cell-14023083

4.      http://www.cell.com/pictureshow

5.      Richard F. Olivo, Biological Sciences 330/331 (Neurophysiology) website, Smith College, http://www.science.smith.edu/departments/NeuroSci/courses/bio330/, See the links for videos shown in class.

4.  A. L. Hodgkin and A. F. Huxley, “A quantitative description of membrane current and its application to conduction and excitation in nerve,” J. Physiol., no. 117, pp. 500-544, 1952. Available at http://jp.physoc.org/cgi/content/full/538/1/2.

5.  NeuroDynamix II website

6.  D. Squires, Instrumentation Electronics for an Integrated Electrophysiology Data Acquisition and Stimulation System, Master of Science in Electrical Engineering Thesis, Western Michigan University, December 2013, available at http://scholarworks.wmich.edu/masters_theses/447/

 

Course Policies

Academic Honesty

General:

“You are responsible for making yourself aware of and understanding the University policies and procedures that pertain to Academic Honesty. These policies include cheating, fabrication, falsification and forgery, multiple submission, plagiarism, complicity and computer misuse. (The academic policies addressing Student Rights and Responsibilities can be found in the Undergraduate Catalog at http://catalog.wmich.edu/content.php?catoid=24&navoid=974 and the Graduate Catalog at http://catalog.wmich.edu/content.php?catoid=25&navoid=1030.) If there is reason to believe you have been involved in academic dishonesty, you will be referred to the Office of Student Conduct. You will be given the opportunity to review the charge(s) and if you believe you are not responsible, you will have the opportunity for a hearing. You should consult with your instructor if you are uncertain about an issue of academic honesty prior to the submission of an assignment or test.”— provided by the Professional Concerns Committee of the WMU Faculty Senate, bold face added, links updated.

Plagiarism:

Plagiarism WILL NOT BE TOLERATED.  See the Plagiarism Tutorial at http://www.lib.usm.edu/legacy/plag/plagiarismtutorial.php to learn about plagiarism and how to properly cite sources.

Grading Basis

1.      Projects (60%) will be assigned on a regular basis.  Some project results will be reported using the IEEE journal paper format; see http://www.ieee.org/documents/transactions_journals.pdf for details. You may not use any sources other than those provided in class or in this syllabus when preparing your project report without prior approval from the course instructor.  You may be asked to demonstrate your project. LATE PROJECTS WILL NOT BE ACCEPTED AND ARE DUE AT THE BEGINNING OF CLASS. All projects are to be completed individually.

2.      Examinations:  10%

3.      Homework and Quizzes (announced or unannounced):  30%

OUTSTANDING WORK might earn extra credit.

 

Scale: 0-60 E | 60-65 D | 65-70 DC | 70-75 C | 75-80 CB | 80-85 B | 85-90 BA | 90-100 A |

EXAMINATIONS AND QUIZZES will be closed-notes closed-book unless otherwise noted. You must have a WMU issued ID with you at the exam.

 

Only under extremely unusual circumstances will make-up examinations and quizzes be considered.  Religious observances will be accommodated with advanced notice.  If an emergency prevents you from attending a scheduled examination or quiz, contact your instructor PRIOR to the test or as soon as you can reach a telephone, e-mail terminal, etc. If the instructor cannot be reached directly, leave a message with the department (276-3150).  Failure to adhere to this policy will result in zero credit for the exercise.

 

Use of Calculators

When a calculator is allowed on a quiz/exam, without exception only models accepted by the Fundamentals of Engineering Examination may be used; see http://ncees.org/exams/calculator-policy/ for a list of approved calculators.

HOMEWORK is assigned in class. Students must maintain a homework folder that is brought to each class. Assignments will be randomly collected from the homework folder perhaps without prior warning. Homework due dates will be given in class. Each homework problem must be worked on separate page(s).  LATE HOMEWORK will not be accepted, except under extraordinary circumstances. Homework is to be completed individually.

 

Homework should normally be done on 8 1/2'' by 11'' sheets. “Engineer's Pad” sheets are preferred.  Solutions must be done in a neat, structured, logical, and orderly manner with frequent brief notations enabling the grader to readily verify the author's source of information, steps taken, sources of formula, equations, and methods used. USE THE PARTIAL CHECK LIST FOR SUBMITTED HOMEWORK BELOW.  Papers failing to meet these guidelines may not be graded and may be returned, with or without an opportunity for resubmission with a penalty.

 

PARTIAL CHECK LIST FOR SUBMITTED HOMEWORK

 

1.      Each problem must include: (a) author's name, (b) name/title of the assignment, and (c) date of completion.

2.      Use only one side of the paper and include a brief and concise statement of the problem prior to its solution. Begin each problem on a new page.

3.      Number the pages and DOUBLE SPACE the text.

4.      Staple each problem in the upper left corner as needed.

5.      Entitle graphs, label and include axes, include key symbols for multiple curve graphs, and give brief notes of explanation where appropriate.

6.      Briefly but clearly annotate your document in a way which will provide the document reader with information such as

a.      which part of the assignment is this?

b.      what is being done and why?

c.      how was it done and what are the results?

d.      how was this equation obtained and how was it used?

e.      sample calculations and definitions of symbols/parameters where appropriate; and

f.       BOX AND LABEL ANSWERS.

 

In case of conflict, information in this syllabus supersedes all other course documents.

 

Other

Students are expected to attend all lectures (note possibility of unannounced quizzes) and to be on time (homework is collected at the beginning of class). Electronic devices are to be turned off (unless there is a safety issue) during lecture unless arrangements have been made with the instructor.


 

Course Schedule
(a tentative schedule for the semester was provided in class; the online schedule will be frequently updated as the semester progresses)

class #

date

topic

[some topics may be verbatim from course references]

assignment

BIOLOGICAL NEURAL NETWORKS

[F&F] is the primary reference for this course section.

WEEK 1

         1          

5/10

course introduction (syllabus)

 

how to order [Zurada]

 

plagiarism

 

What is an intelligent system?

 

electric circuit fundamentals

 

Bring a laptop to the first class with LTspice and MATLAB installed.

 

Let Dr. Miller know ASAP if you are unable to procure a copy of Biological Science by Freeman to get a bookstore course pack order count

 

Read syllabus
Read F&F I.1
Read Freeman chapter 6
Read
http://www.nature.com/scitable/topicpage/what-is-a-cell-14023083
Browse:  http://www.cell.com/pictureshow

Project 1:  F&F I.1 and Simulation of a Simple Neuron
(Use homework format, be sure to include copies of your code)

DUE 5/19

 

Project 1 Example MATLAB® Code

 

HW 1:
DUE 5/19

1.      Complete the plagiarism tutorial found at http://www.lib.usm.edu/legacy/plag/plagiarismtutorial.php and turn in signed statement to that you completed tutorial (If you did!).

2.      Find the power of the circuit elements in problems 4 and 5 of the ECE 2100 Spring 2015 Exam handed out in class.

electric circuit fundamentals

 

operational amplifiers

         2          

5/12

Discuss Project 1

 

Introduction to LTspice

 

Introduction to MATLAB®

FOR NEW USERS:

 

Matlab Workshop I

Matlab Workshop II
(Mackey and Shen)

 

Numerical Computing with MATLAB by C. Moler

Read F&F 1.2 and 1.3

Read F&F 1.4

cell membranes

WEEK 2

         3          

5/17

Discuss Project 1

 

“Patch-Clamp Recording” [F&F, I.2]

 

membrane potentials

 

“Physical Basis for the Resting Potential” [F&F, I.3]

Read Freeman chapters 45 and 46
Read F&F 1.5


Skim A. L. Hodgkin and A. F. Huxley, “A quantitative description of membrane current and its application to conduction and excitation in nerve,” J. Physiol., vol. 117, pp. 500-544, available at
http://jp.physoc.org/cgi/reprint/117/4/500?ssource=mfc&rss=1

Project 2:  F&F I.2 (Ion Channels)
DUE 5/24
(Use homework format)

Videos are available at
http://www.science.smith.edu/departments/neurosci/courses/bio330/videos.html [Olivo website]

 

neuron anatomy

 

“Basis of the Nerve Impulse” [F&F, I.4, 2010] including the “voltage-clamp method”

 

Earthworm Action Potentials

(see video on the Olivo website)

 

See [Squires]

squid giant axon experiments

(see videos on the Olivo Bio 330 website)

         4          

5/19

Hodgkin-Huxley Equations

 

“Electrical Signals in Animals”

{Freeman, CH 45]

Read F&F I.6

Project 3:  F&F I.3, the Hodgkin-Huxley Model, and F&F 1.4
(
Resting Potential and the Nerve Impulse)
DUE 5/31
(Use homework format)

Project 3 Example MATLAB® Code

 

Videos are available at http://www.science.smith.edu/departments/NeuroSci/courses/bio330/squid.html [Olivo website]

“Properties of Neurons”
(Neuron Functional Characteristics)
[F&F, I.5]

WEEK 3

         5          

5/24

QUIZ #1

 

IN-CLASS WORK ON PROJECT 3

Read F&F I.7

 

Project 4:  F&F I.5
Complete the “Neurodynamics II Modeling: Neuron Lessons” in section I.5 of F&F (starts on page 93).

Due 5/31
(Use homework format)

         6          

5/26

“Properties of Neurons”
(Neuron Functional Characteristics)
[F&F, I.5]

 

“Electrophysiology of Neuronal Interactions”

(synapses)
[F&F 1.6]

 

Human CNS

Read Zurada CH 1, 2, A1, A6

 

Project 5:  F&F I.6 and 1.7

Complete the “Neurodynamix II Modeling:   Synapse Lessons”
(starts on page 111)
Complete the “Neurodynamix II Modeling:   Circuit Lessons”
(starts on page 134)

(Use homework format)
DUE 6/7

“Neuronal Oscillators”
[F&F I.7]

ARTIFICIAL NEURAL NETWORKS

[Zurada] is the primary source for this course section.

WEEK 4

         7          

5/31

“Neural Computation”
[Zurada, CH 1]

Read Zurada CH 3, CH 4, A2, A3

HW 2:  Zurada:  CH 2:  1, 4, 14
Verify the results of Figure 2.15 using LTspice and MATLAB® (numerically solve the differential equations that describe the circuit)

(Use homework format)
DUE 6/7

“Learning and Adaptation”

[Zurada, CH 2]

         8          

6/2

“Learning and Adaptation”

[Zurada, CH 2]

“Single Layer Perceptron Classifiers”
[Zurada, CH 3]

 HW 3:  Zurada:  CH 3:  3, 5, 6, 7, 8, 13 (use MATLAB® to plot the error surface in 3D and to prepare a contour plot as in Fig. P3.13 of [Zurada].
(Use homework format)

DUE 6/14

WEEK 5

 

6/6

LAST DAY TO WITHDRAW

         9          

6/7

“Multilayer Feedforward Networks”

[Zurada, CH 4]

 

Discuss Project 4

Read Zurada CH 5, A4, A5

 

Project 6:  Design of a Multilayer Feedforward Neural Network Classifier and Approximator
(use IEEE report format)
DUE 6/21

project files:
class1t.dat
class2t.dat
class1v.dat
class2v.dat

approx1t.dat
approx1v.dat

       10        

6/9

EBP Hints [Haykin]

IN-CLASS WORK ON PROJECT 6

 

WEEK 6

       11        

6/14

dynamical systems

[Scheinerman]

 

Lyapunov functions

[Scheinerman]

[Zurada, A5]

 

Optimization

[Zurada, A4]

Read Zurada CH 6 (sections 1-4)

 

HW 4:  Final Exam Presentation Topic Proposal
DUE 6/21
Select a topic directly related to material covered in this course that you will present to the class on 6/23.  Provide a one paragraph summary of what you plan to present and related reference(s).   Target a 5-10 minute presentation.

 

Some example databases to use:

1.      IEEE Xplore
http://libguides.wmich.edu/az.php?a=i

2.      Biosis Previews
http://libguides.wmich.edu/az.php?a=b

 

Some ideas:

1.      Current understanding of ion channels, synaptic function, learning, memory, etc.

2.      Current application of an artificial neural network.

3.      Review the perspective of Michael Jordan as described in “Machine-Learning Maestro Michael Jordan on the Delusions of Big Data and Other Huge Engineering Efforts” at http://spectrum.ieee.org/robotics/artificial-intelligence/machinelearning-maestro-michael-jordan-on-the-delusions-of-big-data-and-other-huge-engineering-efforts

 

 

Hopfield networks

[Zurada, CH 5]

 

       12        

6/16

associative memories
[Zurada, CH 6]

Project 7:  Study of an Associative Memory
(use IEEE report format)
DUE 6/23

WEEK 7

       13        

6/21

Review HW #3

 

Final Exam Presentation Topic Approval

 

Review Project 6
(individual basis)

 

IN-CLASS QUESTIONS ON PROJECT 7

 

       14        

6/23

PROJECT REVIEW/
IN-CLASS WORK

(individual basis)

 

WEEK 8

       15        

6/28

ALL RESUBMISSIONS DUE BY 5 PM IN MY MAILBOX

 

Final Exam

(Class Presentations)

 

 

Credits

 

Parts adapted/adopted from syllabi by J. Gesink and J. Kelemen.

© 2016 Damon A. Miller

Return to Dr. Miller’s homepage