Serendipity35

Last year, MIT researchers announced that they had built “liquid” neural networks, inspired by the brains of small species. This is a class of flexible, robust machine-learning models that learn on the job and can adapt to changing conditions. That is important for safety-critical tasks, like driving and flying.

The flexibility of these “liquid” neural nets are great but they are computationally expensive as their number of neurons and synapses increase and require inefficient computer programs to solve their underlying, complicated math.

Now, the same team of scientists has discovered a way to alleviate this bottleneck by solving the differential equation behind the interaction of two neurons through synapses. This unlocks a new type of fast and efficient artificial intelligence algorithm and is orders of magnitude faster, and scalable.

What I find interesting is that the equation that needed to be solved to do this has not had a known solution since 1907. That was the year that the differential equation of the neuron model was introduced. I recall when I was a student and when I was teaching at a university (in the humanities) hearing the complaints of students battling away in a course on differential equations.

These models are ideal for use in time-sensitive tasks like pacemaker monitoring, weather forecasting, investment forecasting, or autonomous vehicle navigation. On a medical prediction task, for example, the new models were 220 times faster on a sampling of 8,000 patients. 

Specifically, the team solved, “the differential equation behind the interaction of two neurons through synapses… to unlock a new type of fast and efficient artificial intelligence algorithms.” “The new machine learning models we call ‘CfC’s’ [closed-form Continuous-time] replace the differential equation defining the computation of the neuron with a closed form approximation, preserving the beautiful properties of liquid networks without the need for numerical integration,” MIT professor and CSAIL Director Daniela Rus said. By solving this equation at the neuron-level, the team is hopeful that they’ll be able to construct models of the human brain that measure in the millions of neural connections, something not possible today. The team also notes that this CfC model might be able to take the visual training it learned in one environment and apply it to a wholly new situation without additional work, what’s known as out-of-distribution generalization. That’s not something current-gen models can really do and would prove to be a significant step toward the generalized AI systems of tomorrow.

Source  https://news.mit.edu/2022/solving-brain-dynamics-gives-rise-flexible-machine-learning-models-1115

Albert Einstein was definitely a subject matter expert, but he is not regarded as a good professor. Einstein first taught at the University of Bern but did not attract students, and when he pursued a position at the Swiss Federal Institute of Technology in Zurich, the president raised concerns about his lackluster teaching skills. Biographer Walter Isaacson summarized, “Einstein was never an inspired teacher, and his lectures tended to be regarded as disorganized.” It's a bit unfair to say that "Einstein Was Not Qualified To Teach High-School Physics" - though by today's standards he would not be considered qualified. It probably is fair to say that "Although it’s often said that those who can’t do teach, the reality is that the best doers are often the worst teachers."

Beth McMurtrie wrote a piece in The Chronicle called "What Would Bring the Science of Learning Into the Classroom?" and her overall question was: Why doesn't the scholarship on teaching have as much impact as it could have in higher education classroom practices?

It is not the first article to show and question why higher education appears not to value teaching as much as it could or should. Is it that quality instruction isn't valued as much in higher education as it is in the lower grades? Other articles show that colleges and most faculty believe the quality of instruction is a reason why students select a school.

Having moved from several decades in K-12 teaching to higher education, I noticed a number of things related to this topic. First of all, K-12 teachers were likely to have had at least a minor as undergraduates in education and would have taken courses in pedagogy. For licensing in all states, there are requirements to do "practice" or "student teaching" with monitoring and guidance from education professors and cooperating teachers in the schools.

When I moved from K-12 to higher education at NJIT in 2001, I was told that one reason I was hired to head the instructional technology department was that I had a background in pedagogy and had been running professional development workshops for teachers. It was seen as a gap in the university's offerings. The Chronicle article also points to "professional development focused on becoming a better teacher, from graduate school onward, is rarely built into the job."

As I developed a series of workshops for faculty on using technology, I also developed workshops on better teaching methods. I remember being surprised (but shouldn't have been) that professors had never heard of things like Bloom's taxonomy, alternative assessment, and most of the learning science that had been common for the past 30 years.

K-12 teachers generally have required professional development. In higher education, professional development is generally voluntary. I quickly discovered that enticements were necessary to bring in many faculty. We offered free software, hardware, prize drawings and, of course, breakfasts, lunches and lots of coffee. Professional development in higher ed is not likely to count for much when it comes to promotion and tenure track. Research and grants far outweigh teaching, particularly at a science university like NJIT.

But we did eventually fill our workshops. We had a lot of repeat customers. There was no way we could handle the approximately 600 full-time faculty and the almost 300 adjunct instructors, so we tried to bring in "champions" from different colleges and departments who might later get colleagues to attend.

I recall more than one professor who told me that they basically "try to do the thing my best professors did and avoid doing what the bad ones did." It was rare to meet faculty outside of an education department who did any research on teaching. We did find some. We brought in faculty from other schools who were researching things like methods in engineering education. I spent a lot of time creating online courses and improving online instruction since NJIT was an early leader in that area and had been doing "distance education" pre-Internet.

Discipline-based pedagogy was definitely an issue we explored, even offering specialized workshops for departments and programs. Teaching the humanities and teaching the humanities in a STEM-focused university is different. Teaching chemistry online is not the same as teaching a management course online.

Some of the best parts of the workshops were the conversations amongst the heterogeneous faculty groups. We created less formal sessions with names that gathered professors around a topic like grading, plagiarism and academic integrity, applying for grants, writing in the disciplines, and even topics like admissions and recruiting. These were sessions where I and my department often stepped back and instead offered resources to go further after the session ended.

It is not that K-12 educators have mastered teaching, but they are better prepared for the classroom from the perspective of discipline, psychology, pedagogy, and the numbers of students and hours they spend in face-to-face teaching. College faculty are reasonably expected to be subject matter experts and at a higher level of expertise than K-12 teachers who are expected to be excellent teachers. This doesn't mean that K-12 teachers aren't subject matter experts or that professors can't be excellent teachers. But the preparations for teaching in higher and the recognition for teaching excellence aren't balanced in the two worlds.

I learned about Gordon Moore's 1965 prediction about 10 years after he proposed it. He said that by paying attention to an emerging trend, he extrapolated that computing would dramatically increase in power, and decrease in relative cost, at an exponential pace. His idea is known as Moore’s Law. Moore's law sort of flips Murphy's law by saying that everything gets better.

Moore was an Intel co-founder and his idea was "law" in the electronics industry. Moore helped Intel to make the ever faster, smaller, more affordable transistors that are in a lot more than just computers today. The 2021 chip shortage globally reminded us that cars and appliances and toys and lots of other electronics rely on microchips.

Moore's law is the observation that the number of transistors in a dense integrated circuit (IC) doubles about every two years. (Originally, Moore said it would happen every year but he revised it in 1975 when I was introduced to it to say that it would happen every two years.)

Though the cost of computer power for consumers falls, the cost for chip producers rises. The R&D, manufacturing, and testing costs keep increasing with each new generation of chips. And so, Moore's second law (also called Rock's law) was formulated saying that the capital cost of a semiconductor fabrication also increases exponentially over time. This extrapolation says that the cost of a semiconductor chip fabrication plant doubles every four years.

Huang's Law is new to me. Up front, I will say that this newer "law" is not without questions about its validity. It is based on the observation that advancements in graphics processing units (GPU) are growing at a rate much faster than with traditional central processing units (CPU).

This set Huang's Law as being in contrast to Moore's law. Huang's law states that the performance of GPUs will more than double every two years. The observation was made by Jensen Huang, CEO of Nvidia, in 2018. His observation set up a kind of Moore versus Huang.  He based it on Nvidia’s own GPUs which he said were "25 times faster than five years ago." Moore's law would have expected only a ten-fold increase.

Huang saw synergy between the "entire stack" of hardware, software and artificial intelligence and not just chips as making his new law possible.

If you are not in the business of producing hardware and software, how do these "laws" affect you as an educator or consumer? They highlight the rapid change in information processing technologies. The positive growth in chip complexity and reduction in manufacturing costs would mean that technological advances can occur. Those advances are then factors in economic, organizational, and social change.

When I started teaching computers were not in classrooms. They were only in labs. The teachers who used them were usually math teachers. It took several years for other disciplines to use them and that led to teachers wanting a computer in their classroom. Add 20 years to that and the idea of students having their own computer (first in higher ed and about a decade later in K-12) became a reasonable expectation. During the past two years of pandemic-driven virtual learning, the 1:1 ratio of student:computer became much closer to being ubiquitous.

Further Reading
investopedia.com/terms/m/mooreslaw.asp
synopsys.com/glossary/what-is-moores-law.html
intel.com/content/www/us/en/silicon-innovations/moores-law-technology.html